9

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.

Obviously we've had a robots.txt file that disallows all right from the start. However, observing the robots.txt file is something only well behaved robots do. Recently we've had some issues with less well behaved robots. I've configured Apache to ban a few user-agents but it is pretty easy to get around that.

So, the question is, is there some way to configure Apache (perhaps by installing a some module?) to detect robot-like behavior and respond? Any other ideas?

At the moment all I can do is ban IP addresses based on manual inspection of the logs and that is simply not a viable long term strategy.

Improve this question
1
  • Given that you're asking about Apache modules (check out mod_evasive) and your solution may end up involving some custom log parsing and iptables rules, this topic may be a better candidate for ServerFault.com if you have questions about blocking specific bot behaviors.
    – danlefree
    Commented Sep 6, 2011 at 11:37

3 Answers 3

Reset to default
7

You can link to a hidden page that, when visited, captures the useragent and IP address of the bot and then appends one or both of them to a .htaccess file which blocks them permanently. It's automated so you don't have to do anything to maintain it.

Improve this answer
5
  • That is a very interesting idea, although in case they are using a very generic user-agent, you wouldn't want to blanket lockout any user-agent automatically.
    – Kris
    Commented Sep 2, 2011 at 12:43
  • I would stick to IP addresses then. Plus if you use I addresses and see a pattern from a block of IPs you can then easily block all of them with one simple rule instead of maintaining a long list of individual IPs.
    – John Conde ♦
    Commented Sep 2, 2011 at 13:06
  • 3
    You may use the described method from casually stopping bad bots from crawling your site. But 1) It is possible to bypass (bad bots - and their masters - may learn how to identify honeypots and know how to avoid them); and 2) This method also may also block legitimiate human users that have been re-allocated IPs that has been blacklisted as belonging to misbehaving bots. If you have a legal or regulatory obligation to not have your site indexed or automatically seached, you must use proper authentication and only give authenticated users access. Everything else is not secure.
    – Free Radical
    Commented Jan 6, 2013 at 20:24
  • Nice idea. But, If I implemented it, I bet I'd keep accidentally hitting the honeypot myself and keep getting blocked from my own site.
    – JW01
    Commented Jan 6, 2014 at 19:02
  • @JW01 All you have to do to avoid that is not visit the page that handles this. Since there is no content on it that should be simple to do.
    – John Conde ♦
    Commented Jan 6, 2014 at 19:03
2

You can piggyback on work other people have done in identifying bad IPs by using an Apache module which interfaces with Project Honeypot's IP blacklist. If you're doing this on a large scale, it would probably be polite to offer to run a honeypot.

Improve this answer
2
  • I was amazed when I added Project Honeypot's IP blacklist on my site. Years of anguish ended so simply by blocking the baddies. I think you can detect search engine bots with it too. So, plus 1 for that.
    – JW01
    Commented Jan 6, 2014 at 18:57
  • But the crux of the matter is: If you have public pages, expect them to be indexed. So, some kind of authentication is needed. See answer by Michael Hampton.
    – JW01
    Commented Jan 6, 2014 at 19:06
2

As Gisle Hannemyr mentioned in a comment, the best way to do this is to require logins of all users, and do not provide the restricted content to anyone who isn't logged in.

If you can't require logins for some reason, there are still a couple of fallbacks you can use (disclaimer: both of them are either partly or completely my fault):

  1. The OWASP ModSecurity Core Rule Set contains a number of rules designed to detect automation, even when the bot has taken steps to disguise itself as a browser (e.g. faking its User-Agent string). If you are in full control of your server, such as a VPS, dedicated server, or something larger than that, then you can use these rules with ModSecurity.

    This rule set also contains other rules meant to stop a wide variety of inappropriate activity; if you haven't looked at it, you definitely should.

  2. If you aren't in full control of your server (i.e. you're on shared web hosting) and your host doesn't allow you to use your own ModSecurity rules, you can try something at the application level, such as my own Bad Behavior. I started this project in 2005 to fight blog spam and content scrapers such as those that concern you. It can be added to any PHP-based web site.

    I should also note that many of Bad Behavior's rules have been incorporated into the ModSecurity Core Rule Set, so as long as you've enabled those rules, running both would be rather redundant. These rules are annotated in the Core Rule Set as originating from Bad Behavior.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.