2

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other irrelevant crap.

Lately I have noticed that some web sites require cookies to be turned on, even if you are a bot. What is the purpose of this? Why would a search engine robot need to accept cookies? Or is this just because the guy who designed the web site screwed up?

Improve this question
2
  • Are you suggesting that when cookies are not set then no content is served? AFAIK most bots (including Googlebot) do not set cookies. So this would seem to be blocking access to bots.
    – MrWhite
    Jan 12, 2014 at 1:04
  • Thats what I am saying. Some sites do not show the content to the bot unless cookies are accepted. Example: New York Times
    – Tyler Durden
    Jan 13, 2014 at 0:16

2 Answers 2

Reset to default
2

There are two other possibilities that spring to mind:

  1. Websites can tell that you are pretending to be Googlebot. Google issued a procedure for verifying Googlebot that these websites might be using. They may be requiring cookies only because you are not really Googlebot
  2. The sites don't care about having their pages included in the search indexes. When you don't care about being indexed, it doesn't really matter if you treat the search bots badly.
Improve this answer
2
  • I kinda thought at first that since Google's procedure to verify the Googlebot involved DNS lookups it was perhaps too slow/expensive to do in real-time? But I guess lookups would only be needed on a select few user agents and the results would be cached - so maybe it is OK? The related blogpost.
    – MrWhite
    Jan 24, 2014 at 20:10
  • It would have to be implemented with caching (per IP address) to use it real time, as you suggest. With that in place, and only doing the needed queries for the Googlebot user agent, it would be fine to use real time from a performance standpoint.
    – Stephen Ostermiller ♦
    Jan 24, 2014 at 21:57
0

Tracking comes to mind. Cookies would enable the site to ID a bot as it crawls the site and allow them to separate it from human traffic when analyzing browsing habits.

Improve this answer
4
  • But AFAIK most bots (including Googlebot) do not set cookies, so I can't see how this would enable the site to ID it?
    – MrWhite
    Jan 14, 2014 at 1:11
  • Googlebot wouldn't set cookies. It would get cookies. Then they could be identified and tracked because that cookie would identify it for its entire session.
    – John Conde ♦
    Jan 14, 2014 at 2:16
  • 1
    @JohnConde I'm pretty sure that Googlebot doesn't support any cookies including session cookies.
    – Stephen Ostermiller ♦
    Jan 15, 2014 at 3:34
  • @StephenOstermiller You're right. And I'm sure that website knows this.
    – John Conde ♦
    Jan 15, 2014 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.