I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had a robots.txt file that disallows all right from the start. However, observing the robots.txt file is something only well behaved robots do. Recently we've had some issues with less well behaved robots. I've configured Apache to ban a few user-agents but it is pretty easy to get around that.
So, the question is, is there some way to configure Apache (perhaps by installing a some module?) to detect robot-like behavior and respond? Any other ideas?
At the moment all I can do is ban IP addresses based on manual inspection of the logs and that is simply not a viable long term strategy.