Number of page requests by any Bot in 5 secs

Question

I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?

Answer 1

you might be better off using your firewall instead of relying on a php script, since after all the php script will only affect php page loads and not static files like images. Plus, having php log all connections and check for >800 from the same ip for every request is going to add some serious overhead to your server, maybe even more so than the requests themselves!

If you have a linux server you can use IPTables:

http://blog.bodhizazen.net/linux/prevent-dos-with-iptables/

or you would setup fail2ban to block excessive GETs, you should be safe from blocking real users at a limit of 800/2sec

http://go2linux.garron.me/linux/2011/05/fail2ban-protect-web-server-http-dos-attack-1084.html

On Windows Server... well I don't think you can set connections/sec limits, but I think you can set bandwidth quotas in the QoS services which would effectively limit the bots. Or there are plenty of 3rd party tools that would let you accomplish this.

Edit

I've been thinking about this some more, and it seems the most efficient way to log each ip request and check previous hits would be to concatenate the ip address & current time as the filename and just append a single character, so you are just incrementing the filesize by 1 byte each request. Here is a test script I wrote that you can play around with to get the general idea:

$limit      = 400;
$requests   = 1000;
$log_file   = '/tmp/ip_'.$_SERVER['REMOTE_ADDR'].'_'.time();
$ban_file   = '/tmp/ban_'.$_SERVER['REMOTE_ADDR'];

for($i = 0; $i < $requests; $i++){
    clearstatcache();
    if(file_exists($ban_file)){
        echo "<h1>you've been banned</h1>"; 
        exit;
    }
    $log = fopen($log_file, "a"); 
    fwrite($log,'0');
    fclose($log);       
    if(filesize($log_file) > $limit){
        $log = fopen($ban_file, "w"); 
        fwrite($log,NULL);
        fclose($log);
    }
    else{
        echo filesize($log_file).'<br/>';
    }
}
echo 'final '.filesize($log_file).'<br/>';

Run this with $requests < $limits and you'll see everything is fine no matter how many times you refresh. Change $requests > $limits and you'll see it stops once the filesize reaches 401 bytes. Refresh again and you'll see now you are instantly banned!

It is important to have the clearstatcache(); before each file checks, otherwise PHP will cache the initial filesize and file_exists results and keep reporting as 1 byte & file doesnt exist and never exceed your limit or see the ban file. Also you will need to run a cronjob script periodically to delete the old ip counter files so they don't fill up too much space.

Answer 2

Thanks for the great code. I have modified it in such a way that it will ban anything without a session for 1 whole hour if this trap is activated. I am calling this the Extreme Flood Trap. I know I will be banning even the good bots which leads me to my next question. Will serving the Searchbots with 403 for an hour have bad consequences for the site in the long run? Here is the code so far -

$limit      = 400;
$log_file   = 'ip_'.$_SERVER['REMOTE_ADDR'].'_'.time();
$ban_file   = 'ban_'.$_SERVER['REMOTE_ADDR'];

clearstatcache();
if(file_exists($ban_file)){  
    $banlimit = file_get_contents($ban_file)+3600; 
    if(time() < $banlimit){
        if (!tep_session_id()) {
            header('HTTP/1.1 403 Forbidden');
            exit;
        }
    }
}

//creats a file and addes 1 byte of data on each page request
$log = fopen($log_file, "a"); 
fwrite($log,'0');
fclose($log);    

//if the size of the log file is greater than the Request limit size then create a ban file
if(filesize($log_file) > $limit){
    $log = fopen($ban_file, "w"); 
    fwrite($log,time());
    fclose($log);
}

What do you think?? bad choice??