3

The title basically says it all, but the reason for my question is this...

I make a few checks before I load in any ads, and one of these checks is to see if the browser has local storage as my "click bomb" defense is based on this feature. I have also heard that if your ads are served fewer times with a better click through rate then you get paid more per click. Please don't quote me on any of that knowledge without first verifying it. But if that is in fact the case then not serving up ads on hundreds of thousands of bot hits per day has to be the better solution right? So I am just looking for a catch all that is common to all spiders and can be detected via js without attempting to sort through every possible user agent. I have, I think done a pretty thorough search, but inevitably someone is going to come back with a link that I missed, and trust me I appreciate it.

2
  • It depends on the web crawler. I see no reason why it couldn't have local storage. Its design to act like an actual client.
    – Ramhound
    Commented Nov 10, 2014 at 16:02
  • I am not sure how your click bomb detection works. It would be interesting to know. Are you using a tool for this or something you wrote? I block spiders, and it is part of my research, but it is not such an easy task. It can be rather nuanced. User agents are junk data and cannot be relied upon. You can look to see some simple things such as if an image is requested, a JS is requested (both are not definitive), if the requests come at regular intervals and a speed that is not human, see if they read robots.txt, and so on. There is much more to it than that of course- but it is a start.
    – closetnoc
    Commented Nov 10, 2014 at 23:08

2 Answers 2

3

GoogleBot supports localStorage, according to this service: https://www.whatismybrowser.com/

I found it, ironically, in the google-crawled version of the site. https://twitter.com/dcorbacho/status/702150914023538688

1
  • I would expect Googlebot to clear local storage between requests though. You can't rely on local storage for saving state between requests with any web crawler. Commented Feb 24, 2016 at 15:56
1

The LocalStorage object exists in the js world. Initially web crawlers didn't understand/run javascript, but just rendered markup as static content. But gradually Google and Bing (among others) are working hard to achieve crawlers that understand JS as the browser does. I've found here a similar question.

However this sounds like a flaw/security issue. Could you imagine what a malware-crawler could do with this? Maybe steal identities/sensitive information/etc. So really I think if crawlers could do that, it would be in a limited way.

2
  • 1
    Not true. Google definitely processes JavaScript. Local storage may be included in that (I don't think they have said either way, though).
    – John Conde
    Commented Sep 23, 2015 at 17:31
  • You're right. I didnt know when this started to happen :) googlewebmastercentral.blogspot.mx/2014/05/…
    – mario ruiz
    Commented Sep 24, 2015 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.