8

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record (including his user-agent, referer (if any), and so on, in my database. I plan to check each user to see if they have the cookie I've set, and then create a visit record each subsequent time they come by, to include if they click any links on my site.

The site is very very new (only about a week old as of today), but I've been tracking my visitors in a text log and I see that I can identify visiting bots by their user-agents (the Google and Bing bots/crawlers have come by, along with a couple of others).

It occurs to me that they probably won't accept cookies. Does anyone know this for sure? Or do they also collect cookies in addition to site information?

Improve this question
4
  • Are you trying to restrict your site crawling by using cookies? Bots will always crawl your site regardless of do they accept cookies or not.
    – Oleg
    Commented Mar 20, 2014 at 6:28
  • No, I am not trying to restrict anything. Cookies are just to detect who has been to the site previously and to try to track what they do on the site. I am just curious if anyone knows if crawlers or bots are known to accept cookies, so I can perhaps plan for that.
    – Cyberherbalist
    Commented Mar 20, 2014 at 7:26
  • I don't know sure answer, but I'd not rely on accepting cookies by bots.
    – Oleg
    Commented Mar 20, 2014 at 7:51
  • 2
    Google's crawler doesn't, as they don't want it to index very session specific content. See this blogpost for details: googlewebmastercentral.blogspot.co.uk/2008/03/…. I expect other search engines have a similar policy.
    – Jeremy Halliwell
    Commented Mar 20, 2014 at 8:18

1 Answer 1

Reset to default
6

Most bots don't accept cookies (including Googlebot), however, some bots do.

You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. However, this information is not used in the search results.

When Googlebot visits your site it's like they have never visited before, which goes for any real users who have disabled their cookies.

Further reading: Does googlebot keep session when crawling?

One way to handle the situation of generating a unique visitor ID (guid) is... if the cookie doesn't exist (either a new user or the cookie hasn't been set for whatever reason) then attempt to set a cookie with your real guid, but use a hash of the user agent and IP address as the guid for the current request (which is distinguishable from the real guid). It's not perfect (it's not entirely unique), but it at least allows you to more easily track the bots and importantly avoids counting 1000's of unreal visitors (if you are relying solely on a cookie to count visitors).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.