Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 9 down vote favorite
4

Here is one of my customers, performing some action after having logged in to his account. The unique token is simply an encrypted user id + timestamp.

94.254.xxx.xxx - - [02/Jul/2011:22:25:46 +0200] "GET /some-action/unique-token-123abc HTTP/1.1" 200 410 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"

Now, Googlebot somehow found out about this unique link and tried to access the exact same URL one week later.

66.249.71.179 - - [10/Jul/2011:09:56:01 +0200] "GET /some-action/unique-token-123abc HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

(status code is 302 because the token had expired)

Let me emphasize that this is a unique URL that was visible exactly once, for only 2 seconds, before the user clicked it and proceeded to visit that page. It was not sent in an email or published anywhere public.

What is going on here, how is it possible that Google found this unique URL?

share|improve this question

2 Answers

up vote 4 down vote accepted

I just realized that the user must have found an outbound link on this authenticated page, and then leaked the private URL as Referer when clicking through to some other website. This is the only possible explanation, and should really have been obvious from the start.

Once leaked, the private URL may have been exposed to Google in a number of ways, e.g. the target site might have published their access logs publicly. Note: none of the outbound links were using Google Analytics so this does not indicate that Googlebot are using referrer URLs from Analytics.

Lesson relearned: never ever put sensitive data in URLs unless you use https, in which case the browser would have left Referer empty.

share|improve this answer
1  
You're right: putting sensitive data in URLs can be dangerous. Whenever you can, you should pass unique user IDs between pages using POST requests (which don't send variables as part of the URL, like GET requests), or with cookie/session variables. – Nick Jul 12 '11 at 16:35
up vote 6 down vote

It's hard to say for sure but here are likely scenarios:

  • The user has a browser toolbar or extension installed that reports the URLs they visit to Google.

  • Someone linked to that URL and Google found it by crawling the page with that link on it.

share|improve this answer
If you're talking about the Google Toolbar only sends back URLs to Google in case you enable the "PageRank" feature, but we never used that data to discover new URLs. If about some other toolbar that was released by us, let me know please. – methode Jul 12 '11 at 20:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.