I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my bosses are worried about.

The White List:

crawler_name - crawler_host
Googlebot - .googlebot.com
Yahoo! Slurp - crawl.yahoo.net
MSNBot - search.msn.com

If the HTTP_REQUEST_HEADER contains the crawler_name and the hostname (reverse DNS lookup of the IP) contains the crawler_host of any of the above, then we leave them request as many pages as they want.

Is this list good enough? Will this cover the main search engine spiders? Or might we accidentally block one?

I've tested it using the "Fetch as GoogleBot" feature in Google Webmaster Tools and it's working as expected.
According to Microsoft "Bing operates three crawlers today: bingbot, adidxbot, msnbot". That's fine, I can add in bingbot and adidxbot but will their resolved hostnames still contain "search.msn.com"?

  • As an aside you can use Google Webmaster tools to request your preferred crawl rate if that's something you want to do. Jun 8, 2011 at 8:38
  • I think we've that already done
    – Adam Lynch
    Jun 8, 2011 at 8:41
  • If I ever start web scraping or ddosing, the first thing I would do is spoof the googlebot headers.
    – captncraig
    Jun 9, 2011 at 13:35
  • @CMP I do a lookup on the IP to get the hostname and then do another lookup in the reverse order and compare the two IPs. Pretty sure that stops this
    – Adam Lynch
    Jun 9, 2011 at 13:46

1 Answer 1


This is the wrong strategy. Also, headers are trivial to spoof.

Honestly, anti-crawler protections are very fragile and generally unwise. You may end up blocking legitimate users (who'll be annoyed), or your code might end up being forgotten about, become stale, and block crawlers you want to allow.

You can verify whether a bot belongs to Google, however - advice here: http://www.google.com/support/webmasters/bin/answer.py?answer=80553

I'm not sure if the same is possible with other crawlers. Frankly, this isn't a strategy I'd employ.

  • What would be your strategy then?
    – Adam Lynch
    Jun 17, 2011 at 8:33
  • To not implement the feature at all. I struggle to think of a legitimate use case that necessitates such code. What is the reason you have for implementing this?
    – Matty
    Jun 17, 2011 at 8:35
  • Protecting our content. Scrapers will try and steal our content
    – Adam Lynch
    Jun 17, 2011 at 8:41
  • If you're not able to make your content members' only, I wouldn't worry about it. This isn't the answer you're looking for, but scrapers come with the territory of being online. Anti-bot protections aren't really much use here, and at best wouldn't eliminate scraping entirely but would risk annoying users or excluding the SE bots you want. Google's pretty smart about scraped content too.
    – Matty
    Jun 17, 2011 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.