Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other irrelevant crap.

Lately I have noticed that some web sites require cookies to be turned on, even if you are a bot. What is the purpose of this? Why would a search engine robot need to accept cookies? Or is this just because the guy who designed the web site screwed up?

share|improve this question
    
Are you suggesting that when cookies are not set then no content is served? AFAIK most bots (including Googlebot) do not set cookies. So this would seem to be blocking access to bots. –  w3d Jan 12 at 1:04
    
Thats what I am saying. Some sites do not show the content to the bot unless cookies are accepted. Example: New York Times –  Tyler Durden Jan 13 at 0:16
add comment

2 Answers

There are two other possibilities that spring to mind:

  1. Websites can tell that you are pretending to be Googlebot. Google issued a procedure for verifying Googlebot that these websites might be using. They may be requiring cookies only because you are not really Googlebot
  2. The sites don't care about having their pages included in the search indexes. When you don't care about being indexed, it doesn't really matter if you treat the search bots badly.
share|improve this answer
    
I kinda thought at first that since Google's procedure to verify the Googlebot involved DNS lookups it was perhaps too slow/expensive to do in real-time? But I guess lookups would only be needed on a select few user agents and the results would be cached - so maybe it is OK? The related blogpost. –  w3d Jan 24 at 20:10
    
It would have to be implemented with caching (per IP address) to use it real time, as you suggest. With that in place, and only doing the needed queries for the Googlebot user agent, it would be fine to use real time from a performance standpoint. –  Stephen Ostermiller Jan 24 at 21:57
add comment

Tracking comes to mind. Cookies would enable the site to ID a bot as it crawls the site and allow them to separate it from human traffic when analyzing browsing habits.

share|improve this answer
    
But AFAIK most bots (including Googlebot) do not set cookies, so I can't see how this would enable the site to ID it? –  w3d Jan 14 at 1:11
    
Googlebot wouldn't set cookies. It would get cookies. Then they could be identified and tracked because that cookie would identify it for its entire session. –  John Conde Jan 14 at 2:16
1  
@JohnConde I'm pretty sure that Googlebot doesn't support any cookies including session cookies. –  Stephen Ostermiller Jan 15 at 3:34
    
@StephenOstermiller You're right. And I'm sure that website knows this. –  John Conde Jan 15 at 12:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.