3

During analyzing traffic generated by bots to our website, we found the strange issue. We found that AdsBot-Google crawling domain which we stopped to advertise in the middle of 2016.

Right now there are robots.txt to disallow all bots to crawl our site, but ads bot anyway ignore these settings.

Where can I look for further investigations? Is it safe to just block this bots with 403 http status?

1 Answer 1

3

This crawler is unlikely to be hurting your site in any way. I'd continue to allow it. You would need to do so if you want to resume advertising.

According to http://www.botreports.com/user-agent/adsbot-google.shtml it does not honor user-agent: * directives in robots.txt. You need to give the more specific:

User-agent: AdsBot-Google
Disallow: /

Another thing to check is that the bot is actually from Google. Google has a IP address lookup method for checking that bots are actually from Google: https://support.google.com/webmasters/answer/80553?hl=en It could be that some other crawler has appropriated the user agent for nefarious purposes.

If you find you do want to block and robots.txt still isn't working, a 403 error is fine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.