I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?
you might be better off using your firewall instead of relying on a php script, since after all the php script will only affect php page loads and not static files like images. Plus, having php log all connections and check for >800 from the same ip for every request is going to add some serious overhead to your server, maybe even more so than the requests themselves!
If you have a linux server you can use IPTables:
or you would setup fail2ban to block excessive GETs, you should be safe from blocking real users at a limit of 800/2sec
On Windows Server... well I don't think you can set connections/sec limits, but I think you can set bandwidth quotas in the QoS services which would effectively limit the bots. Or there are plenty of 3rd party tools that would let you accomplish this.
I've been thinking about this some more, and it seems the most efficient way to log each ip request and check previous hits would be to concatenate the ip address & current time as the filename and just append a single character, so you are just incrementing the filesize by 1 byte each request. Here is a test script I wrote that you can play around with to get the general idea:
Run this with $requests < $limits and you'll see everything is fine no matter how many times you refresh. Change $requests > $limits and you'll see it stops once the filesize reaches 401 bytes. Refresh again and you'll see now you are instantly banned!
Thanks for the great code. I have modified it in such a way that it will ban anything without a session for 1 whole hour if this trap is activated. I am calling this the Extreme Flood Trap. I know I will be banning even the good bots which leads me to my next question. Will serving the Searchbots with 403 for an hour have bad consequences for the site in the long run? Here is the code so far -
What do you think?? bad choice??