I've a system which blacklists users who request too many pages too fast if they are not in my white list. We are just worried about the main search engines and to be honest Google is the only one my bosses are worried about.
The White List:
crawler_name - crawler_host
Googlebot - .googlebot.com
Yahoo! Slurp - crawl.yahoo.net
MSNBot - search.msn.com
If the HTTP_REQUEST_HEADER contains the crawler_name and the hostname (reverse DNS lookup of the IP) contains the crawler_host of any of the above, then we leave them request as many pages as they want.
Is this list good enough? Will this cover the main search engine spiders? Or might we accidentally block one?
I've tested it using the "Fetch as GoogleBot" feature in Google Webmaster Tools and it's working as expected.
According to Microsoft "Bing operates three crawlers today: bingbot, adidxbot, msnbot". That's fine, I can add in bingbot and adidxbot but will their resolved hostnames still contain "search.msn.com"?