Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp.

94.254.xxx.xxx - - [02/Jul/2011:22:25:46 +0200] "GET /some-action/unique-token-123abc HTTP/1.1" 200 410 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"

Now, Googlebot somehow found out about this unique link and tried to access the exact same URL one week later. - - [10/Jul/2011:09:56:01 +0200] "GET /some-action/unique-token-123abc HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

(status code is 302 because the token had expired)

Let me emphasize that this is a unique URL that was visible exactly once, for only 2 seconds, before the user clicked it and proceeded to visit that page. It was not sent in an email or published anywhere public.

What is going on here, how is it possible that Google found this unique URL?

2 Answers 2


It's hard to say for sure but here are likely scenarios:

  • The user has a browser toolbar or extension installed that reports the URLs they visit to Google.

  • Someone linked to that URL and Google found it by crawling the page with that link on it.

  • If you're talking about the Google Toolbar only sends back URLs to Google in case you enable the "PageRank" feature, but we never used that data to discover new URLs. If about some other toolbar that was released by us, let me know please.
    – methode
    Jul 12, 2011 at 20:08

I just realized that the user must have found an outbound link on this authenticated page, and then leaked the private URL as Referer when clicking through to some other website. This is the only possible explanation, and should really have been obvious from the start.

Once leaked, the private URL may have been exposed to Google in a number of ways, e.g. the target site might have published their access logs publicly. Note: none of the outbound links were using Google Analytics so this does not indicate that Googlebot are using referrer URLs from Analytics.

Lesson relearned: never ever put sensitive data in URLs unless you use https, in which case the browser would have left Referer empty.

  • 1
    You're right: putting sensitive data in URLs can be dangerous. Whenever you can, you should pass unique user IDs between pages using POST requests (which don't send variables as part of the URL, like GET requests), or with cookie/session variables.
    – Nick
    Jul 12, 2011 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.