Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 5 down vote favorite

I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my bosses are worried about.

The White List:

crawler_name - crawler_host
Googlebot - .googlebot.com
Yahoo! Slurp - crawl.yahoo.net
MSNBot - search.msn.com

If the HTTP_REQUEST_HEADER contains the crawler_name and the hostname (reverse DNS lookup of the IP) contains the crawler_host of any of the above, then we leave them request as many pages as they want.

Is this list good enough? Will this cover the main search engine spiders? Or might we accidentally block one?

Edit:
I've tested it using the "Fetch as GoogleBot" feature in Google Webmaster Tools and it's working as expected.
According to Microsoft "Bing operates three crawlers today: bingbot, adidxbot, msnbot". That's fine, I can add in bingbot and adidxbot but will their resolved hostnames still contain "search.msn.com"?

share|improve this question
As an aside you can use Google Webmaster tools to request your preferred crawl rate if that's something you want to do. – paulmorriss Jun 8 '11 at 8:38
I think we've that already done – Adam Lynch Jun 8 '11 at 8:41
If I ever start web scraping or ddosing, the first thing I would do is spoof the googlebot headers. – CMP Jun 9 '11 at 13:35
@CMP I do a lookup on the IP to get the hostname and then do another lookup in the reverse order and compare the two IPs. Pretty sure that stops this – Adam Lynch Jun 9 '11 at 13:46

1 Answer

up vote 2 down vote

This is the wrong strategy. Also, headers are trivial to spoof.

Honestly, anti-crawler protections are very fragile and generally unwise. You may end up blocking legitimate users (who'll be annoyed), or your code might end up being forgotten about, become stale, and block crawlers you want to allow.

You can verify whether a bot belongs to Google, however - advice here: http://www.google.com/support/webmasters/bin/answer.py?answer=80553

I'm not sure if the same is possible with other crawlers. Frankly, this isn't a strategy I'd employ.

share|improve this answer
What would be your strategy then? – Adam Lynch Jun 17 '11 at 8:33
To not implement the feature at all. I struggle to think of a legitimate use case that necessitates such code. What is the reason you have for implementing this? – Matty Jun 17 '11 at 8:35
Protecting our content. Scrapers will try and steal our content – Adam Lynch Jun 17 '11 at 8:41
If you're not able to make your content members' only, I wouldn't worry about it. This isn't the answer you're looking for, but scrapers come with the territory of being online. Anti-bot protections aren't really much use here, and at best wouldn't eliminate scraping entirely but would risk annoying users or excluding the SE bots you want. Google's pretty smart about scraped content too. – Matty Jun 17 '11 at 9:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.