All Questions
Tagged with iptables web-crawlers
2 questions
3
votes
1
answer
760
views
New hostile web crawler from China throttling server
My server has been crashes several times the last 24 hours by a new extremely aggressive hostile distributed web crawler.
It is hitting over 1000 pages per minute and seems to be crawling all content ...
- 171
5
votes
5
answers
4k
views
Which tools can help limit maximum page view per IP to limit scrapers and bots?
I would like to prevent scrapers grabbing all my content except Google, Bing and other search engines. I am thinking of going with Fail2ban and limiting hits from an IP maybe at around 1000 per day. ...
- 199