Possible Duplicate:
How do spambots work?

I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for users that access an abnormal number of pages per minute.

I don't want search engine spiders to ever receive the error. My inclination is to set a robots.txt crawl-delay which will ensure spiders access a number of pages per minute under my 503 threshold.

Is this an appropriate solution? Do all major search engines support the directive? Could it negatively affect SEO? Are there any other solutions or recommendations?

marked as duplicate by John Conde Oct 17 '12 at 23:32

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • The answers in that question explain how to handle bad bots – John Conde Oct 17 '12 at 23:33
  • @JohnConde I don't believe this is a duplicate at all. I'm not dealing with SpamBots who are submitting forms on my site. I'm dealing with Scrapers who are extracting data from product pages on my site. The "duplicate" you cited doesn't address Crawl-Delay or implementing hit limits in any way. – skibulk Oct 18 '12 at 1:00
  • Google doesn't use the "crawl-delay" directive so you'll need to throttle them by other means. – Fiasco Labs Oct 18 '12 at 4:05