6

In my web application I generate links in the following format:

https://example.com/^token^

Each link according to my specs is sent via email and should not allow robots to scrape it. In order to avoid visits via robot I placed the following robots.txt:

User-agent: *
Disallow: /

And on page's <header></header> tag I placed:

    <meta name="robots" content="noindex">
    <meta name="robots" content="nofollow">

The question is how I can ensure that a link is opened only when user has clicked upon it and not a random bot/spider scrapes it? Does the length and the randomness of the ^token^ in url factor into bot visit prevention?

In my application the ^token^ is a cryptographically-random 5 byte value that once generates is hex-econded. So in my case, if the token length takes a significant role on non-scrappines of this link, which is the recommended length of the random token?

My application is in Laravel and I use nginx and php-fpm combo in order to serve the content to the browser.

Improve this question
7
  • 3
    Is your token long enough and random enough that bots can't guess it without scraping it from emails? In other words, it isn't just a serially incrementing id number?
    – Stephen Ostermiller ♦
    Dec 8, 2020 at 11:22
  • It is now generated via 5-byte length cryptographically-secure random number that is hex-encoded. The problem though is how long it should be the random token.
    – Dimitrios Desyllas
    Dec 8, 2020 at 11:24
  • 2
    Are the bots random, or associated with the email provider? Do you get hits on the tokens before you send out an email (e.g. log entries for non-existent tokens), or only after?
    – lights0123
    Dec 8, 2020 at 19:47
  • Are you trying to prevent targeted attacks (e.g. if you are selling a popular latest gen console and people are trying to guess how to get to the offer page) or just casual search engines or non-targeted attacks?
    – user3067860
    Dec 9, 2020 at 13:14
  • @user3067860 No it is a invitation link for a contract signature into a B2B business app. Once contract signed then a new account is created.
    – Dimitrios Desyllas
    Dec 9, 2020 at 13:43

4 Answers 4

Reset to default
13

In short: you can't.

Answer depends on the definition of "bots", but if a bot can crawl the e-mail, the bot can hit the link. A bot doesn't necessarily respect robots.txt, or mentioned html meta tags.

__

However (a bit outside of the scope):

  • You should have the <token> be a long format random as stated on a comment to your question.

  • You could have the link protected by a captcha or any other Turing test.

  • You could also explain what you've been observing in order for us to understand what you would like to avoid.

Improve this answer
10

Incorrectly used meta tags

Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two.

<meta name="robots" content="noindex,nofollow">

Robots.txt and robots meta tags are mutually exclusive

Disallowing pages in robots.txt prevents robots from downloading the page and seeing the meta tags. There are approximately zero bots that wouldn't obey robots.txt but would obey meta tags. You should choose one or the other, but not both.

Robots.txt won't prevent search engines from indexing URLs

If Google finds enough links to a URL, it may include that URL in its search index, even if that URL is disallowed in robots.txt. If your fear is that some of these URLs will get indexed in search engines, you should allow crawling in robots.txt but disallow indexing via the meta tag.

Meta tags won't prevent bots from hitting the URLs

If your fear is that bots will mess up your stats or cause other undesired effects when bots hit them, then you should use robots.txt. Search engines might still index a URL occasionally, but most bots will obey robots.txt and not even request the URLs

No way to prevent indexing and bot hits with robots.txt and meta tags

If you want to prevent indexing of the URLs and bot hits to the URLs, you are out of luck. There is no way to use robots.txt and meta tags to do both at the same time.

Are your tokens long enough?

A five byte token gives you 256^5 or 1.1E12 (1.1 trillion) possible URLs. If you send out a million email messages, that leaves a 1 in 1 million chance of getting an in-use token for each guess. If you send out a billion email messages, the odds of getting a in-use token rise to 1 in 1 thousand. You'd certainly want to increase the length of your token after sending out 1 million emails.

You could also get a lot more security without increasing the length of your URLs. Five bytes hex encoded is 10 characters. It would be smarter to use 10 characters that are randomly chosen from:

  • 10 digits
  • 26 lowercase letters
  • 26 uppercase letters
  • 4 URL safe symbols (-._~)

If you did that you could increase the possible number of tokens to 66^10 or 1.5E18 (1.5 quintillion). That would give you enough token space no matter how many emails you sent out.

Other ways to increase security

You could also employ any of the following tactics to further ensure that bots don't get access to this content:

  • Use server side configuration that gives an error code if a suspected robot hits these URLs. You could detect robots based on:
    • User agent
    • What other things that IP address is requesting
    • Can it pass captcha?
  • Require that users log in to see this content when clicking from an email.
  • Expire old tokens: require that clicks happen in the hours or days after email is sent out.
Improve this answer
7
  • Also I can allow whitelist User Agents as well Is non provided or User agent is a non-browser one then I can prompt to a captcha challenge as well ;)
    – Dimitrios Desyllas
    Dec 9, 2020 at 8:54
  • “No way to prevent indexing and bot hits” - not entirely true, you can detect some bots via IP or User-agent and block access to the page.
    – DisgruntledGoat
    Dec 9, 2020 at 12:30
  • @DisgruntledGoat I updated that heading to be clearer about the context (robots.txt and meta tags)
    – Stephen Ostermiller ♦
    Dec 9, 2020 at 12:51
  • I'm not sure about your calculations. This looks like a setting where the birthday paradox applies, so if you have 1e12 possible token a collision starts to become probable (about 50%) once you send out 1e6 emails.
    – Federico Poloni
    Dec 9, 2020 at 12:57
  • @FedericoPoloni I did not calculate the likelihood of a collision, only the probability of bot or hacker guessing a used token. As you point out, avoiding collisions is a harder problem that requires either a larger token space or checking to see if a token is already in use after picking it.
    – Stephen Ostermiller ♦
    Dec 9, 2020 at 13:44
7

Not only can you not ensure it, it's highly likely that the link will be retrieved by programs not under the user's control. Many spam and phishing filters will pre-fetch any web pages linked to in an email to scan them for possible threats (I've had as many as five non-user hits recorded for a single link).

The solution is to make sure nothing sensitive is displayed on the page, and when the link is opened, nothing is changed on the server. Sensitive content should be placed behind a login barrier or equivalent. Actions that make changes (such as a password reset) should require the user to take an action on the page, such as clicking a "submit" button.

Improve this answer
4
  • We had this issue once on a project: Two links in a mail were used to approve/decline something (”Do you want to approve this action..."). The virus scanner on the users machines (corporate network, so everybody had the same virus scanner) checked always both links and as the approve link was the first one in the mail, everything got approved automatically... At the end, the approval wasn't done when opening the link but when clicking on a button on the opening page.
    – Reeno
    Dec 9, 2020 at 22:11
  • 2
    All non-safe actions should not come from a HTTP GET, but something else (usually a POST), which won't happen by normally following a link.
    – Paŭlo Ebermann
    Dec 9, 2020 at 23:25
  • @PaŭloEbermann, the requirement on GET is actually stronger than that: it needs to be both safe and idempotent: making a GET request repeatedly should act no differently than making it once.
    – Mark
    Dec 10, 2020 at 0:13
  • 1
    Are there safe and non-idempotent HTTP-methods? As far as I remember, safe-ness automatically implies idempotence. (I.e. a GET should not modify anything on the server side, except possibly log files and similar. Of course this is still the same when done multiple times.)
    – Paŭlo Ebermann
    Dec 10, 2020 at 0:15
2

Have a second, follow-up page which requires the token, but requires interaction before the token is considered to be used/invalidated.

When the link is used, it brings the user to a page where they must confirm something. Expire the token when they perform this action, rather than when the link is used.

As others have provided, if you provide a URL, expect it to be activated without the process continuing.

Do still

  • invalidate old tokens
  • invalidate tokens after a wider count of uses (10 instead of 1)
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.