up vote 3 down vote favorite
share [g+] share [fb]

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.

Obviously we've had a robots.txt file that disallows all right from the start. However, observing the robots.txt file is something only well behaved robots do. Recently we've had some issues with less well behaved robots. I've configured Apache to ban a few user-agents but it is pretty easy to get around that.

So, the question is, is there some way to configure Apache (perhaps by installing a some module?) to detect robot-like behavior and respond? Any other ideas?

At the moment all I can do is ban IP addresses based on manual inspection of the logs and that is simply not a viable long term strategy.

link|improve this question
80% accept rate
Given that you're asking about Apache modules (check out mod_evasive) and your solution may end up involving some custom log parsing and iptables rules, this topic may be a better candidate for ServerFault.com if you have questions about blocking specific bot behaviors. – danlefree♦ Sep 6 '11 at 11:37
feedback

2 Answers

up vote 4 down vote

You can link to a hidden page that, when visited, captures the useragent and IP address of the bot and then appends one or both of them to a .htaccess file which blocks them permanently. It's automated so you don't have to do anything to maintain it.

link|improve this answer
That is a very interesting idea, although in case they are using a very generic user-agent, you wouldn't want to blanket lockout any user-agent automatically. – Kris Sep 2 '11 at 12:43
I would stick to IP addresses then. Plus if you use I addresses and see a pattern from a block of IPs you can then easily block all of them with one simple rule instead of maintaining a long list of individual IPs. – John Conde♦ Sep 2 '11 at 13:06
feedback
up vote 0 down vote

You can piggyback on work other people have done in identifying bad IPs by using an Apache module which interfaces with Project Honeypot's IP blacklist. If you're doing this on a large scale, it would probably be polite to offer to run a honeypot.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.