The title basically says it all, but the reason for my question is this...

I make a few checks before I load in any ads, and one of these checks is to see if the browser has local storage as my "click bomb" defense is based on this feature. I have also heard that if your ads are served fewer times with a better click through rate then you get paid more per click. Please don't quote me on any of that knowledge without first verifying it. But if that is in fact the case then not serving up ads on hundreds of thousands of bot hits per day has to be the better solution right? So I am just looking for a catch all that is common to all spiders and can be detected via js without attempting to sort through every possible user agent. I have, I think done a pretty thorough search, but inevitably someone is going to come back with a link that I missed, and trust me I appreciate it.

  • It depends on the web crawler. I see no reason why it couldn't have local storage. Its design to act like an actual client.
    – Ramhound
    Nov 10 '14 at 16:02
  • I am not sure how your click bomb detection works. It would be interesting to know. Are you using a tool for this or something you wrote? I block spiders, and it is part of my research, but it is not such an easy task. It can be rather nuanced. User agents are junk data and cannot be relied upon. You can look to see some simple things such as if an image is requested, a JS is requested (both are not definitive), if the requests come at regular intervals and a speed that is not human, see if they read robots.txt, and so on. There is much more to it than that of course- but it is a start.
    – closetnoc
    Nov 10 '14 at 23:08

GoogleBot supports localStorage, according to this service: https://www.whatismybrowser.com/

I found it, ironically, in the google-crawled version of the site. https://twitter.com/dcorbacho/status/702150914023538688

  • I would expect Googlebot to clear local storage between requests though. You can't rely on local storage for saving state between requests with any web crawler. Feb 24 '16 at 15:56

The LocalStorage object exists in the js world. Initially web crawlers didn't understand/run javascript, but just rendered markup as static content. But gradually Google and Bing (among others) are working hard to achieve crawlers that understand JS as the browser does. I've found here a similar question.

However this sounds like a flaw/security issue. Could you imagine what a malware-crawler could do with this? Maybe steal identities/sensitive information/etc. So really I think if crawlers could do that, it would be in a limited way.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy