1

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I be looking for other clues as well? I would whitelist the biggest scrapers like Google, Bing and ban the rest. I understand there may be some casualties if hundreds of users are behind an IP, but the ban would be temporary.

1

Yes. You can. I do not know the tool you are using, but here are some clues.

  • Not a search engine (duh!).
  • High number of requests over a longer period of time- more than a user.
  • Does not request an image, CSS, or JS (some do, most don't).
  • Changes agent name and OS over a longer period of time.
  • Comes from a web-hosting IP address block/range.

Check for access over various periods of time; one hour, one day, one week, one month. The reason for this is that stealth bots will make a high number of requests over a longer period of time. If the number of requests are higher than that of a user, then block them. Bots will stick out big-time. There will be no confusion on this one.

Some bots will request images to escape automated blocking, but may or may not request CSS and JS (javascript). Some will. If you have a request for a page but none of the above, it is a bot. If you have a request for a page and only one or two of the three, then it is a bot. Create a bogus JS (with virtually do nothing code) file and request it in your page. Keep it very small so that you are not effecting download speeds. You just want to see if it is requested.

Some bots will change the user agent (browser) and OS, some won't. If over a longer period of time these elements change, it is a bot.

Bots sometimes can come from subscriber lines (telcos) but most will not. If you check the IP address and it belongs to a web hosting company, then it is a bot. Subscriber lines are also used. Over a period of time, you will learn where bots come from. It will become a no-brainer for the most part.

If you are not sure if an IP address or access is a bot, come back here and ask making sure to post the original request. I should be able to help. I do this all day every day.

I have a much longer list of criteria, but those determinations get complicated and are protected processes. Sorry. But the list above should be more than enough for anyone to determine a bot %99 of the time.

2
  • Oh excellent response. Been thinking about creating some open source initiative for this. I notced for example that Gutenberg.org blocks access from non residential ips straight away. So either such an open source db, or rules like the ones you mentioned could be maintained on github perhaps, – giorgio79 Apr 9 '15 at 17:13
  • @giorgio79 Go for it! I used to have a PERL script called tinyTrap that worked very well, but rolled that into a variety of other highly detailed mechanisms that run live on my site. In my case, I do not block access, but change the content and report on these bots (hackers too). I could do a better job of course. It is a work in progress. I do research for others so I cannot give it all away. But the list I gave is strong enough to really do a good job blocking bots. Let me know what you come up with! If you have questions, I am here every day- but not all day- mid-day and evenings mostly. – closetnoc Apr 9 '15 at 17:22
1

Alas if your data is worth getting it will be gotten. When we decided to handle IP addresses, the scrapers went horizontal on us and came back with thousands of IP addresses, each accessing us well under the limit we set. I am myself hunting for the newest hints on how to stop robots that do not respect any protection.

4
  • Were you blocking IP address with your firewall or something of the sort? – Alexis Wilke Oct 27 '15 at 3:14
  • Something if the sort. But if an address never returns it is just a waste of processing to block it. – mplungjan Oct 27 '15 at 5:02
  • Well... I have seen many bots hitting my websites hundred of times. So blocking is really cheap in comparison to regenerate a full request in Apache, and then my code... – Alexis Wilke Oct 27 '15 at 6:52
  • 1
    Great if it works for you. Our robots came in the hundred thousand unique IPs in no particular range and made one or two requests per IP. Only visible as robots after the fact. To block them would be useless and slow every future request from legitimate users down. – mplungjan Oct 27 '15 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.