I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
In my logs, I keep noticing that 18.104.22.168, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?
Possible Duplicate: How do spambots work? I want to prevent web scrapers from abusing 1,000,000 on my website. I'd like to do this by returning a "503 Service Unavailable" error code for ...