Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am writing a script that will block any bot that requests page(s) for example X times in the past 5 secs. I need to find the X here. Do you guys know some approx values I can use?

share|improve this question
    
Since I did not get any suggestions, I am setting it to 800 requests in 2 secs. This means my script will block all the bots if there are 800 requests or more in the past 2 secs. –  Manish Pradhan Oct 3 '12 at 14:58
add comment

2 Answers

up vote 5 down vote accepted

you might be better off using your firewall instead of relying on a php script, since after all the php script will only affect php page loads and not static files like images. Plus, having php log all connections and check for >800 from the same ip for every request is going to add some serious overhead to your server, maybe even more so than the requests themselves!

If you have a linux server you can use IPTables:

http://blog.bodhizazen.net/linux/prevent-dos-with-iptables/

or you would setup fail2ban to block excessive GETs, you should be safe from blocking real users at a limit of 800/2sec

http://go2linux.garron.me/linux/2011/05/fail2ban-protect-web-server-http-dos-attack-1084.html

On Windows Server... well I don't think you can set connections/sec limits, but I think you can set bandwidth quotas in the QoS services which would effectively limit the bots. Or there are plenty of 3rd party tools that would let you accomplish this.

Edit

I've been thinking about this some more, and it seems the most efficient way to log each ip request and check previous hits would be to concatenate the ip address & current time as the filename and just append a single character, so you are just incrementing the filesize by 1 byte each request. Here is a test script I wrote that you can play around with to get the general idea:

$limit      = 400;
$requests   = 1000;
$log_file   = '/tmp/ip_'.$_SERVER['REMOTE_ADDR'].'_'.time();
$ban_file   = '/tmp/ban_'.$_SERVER['REMOTE_ADDR'];

for($i = 0; $i < $requests; $i++){
    clearstatcache();
    if(file_exists($ban_file)){
        echo "<h1>you've been banned</h1>"; 
        exit;
    }
    $log = fopen($log_file, "a"); 
    fwrite($log,'0');
    fclose($log);       
    if(filesize($log_file) > $limit){
        $log = fopen($ban_file, "w"); 
        fwrite($log,NULL);
        fclose($log);
    }
    else{
        echo filesize($log_file).'<br/>';
    }
}
echo 'final '.filesize($log_file).'<br/>';

Run this with $requests < $limits and you'll see everything is fine no matter how many times you refresh. Change $requests > $limits and you'll see it stops once the filesize reaches 401 bytes. Refresh again and you'll see now you are instantly banned!

It is important to have the clearstatcache(); before each file checks, otherwise PHP will cache the initial filesize and file_exists results and keep reporting as 1 byte & file doesnt exist and never exceed your limit or see the ban file. Also you will need to run a cronjob script periodically to delete the old ip counter files so they don't fill up too much space.

share|improve this answer
    
Thanks for taking the time to reply. The request table only has about max 80 rows at any given time. The script above has 800 set so that I want to trigger the condition only if there really is a need for it. I am hosting a site elsewhere and cant really play with the firewall. Thanks anyways. –  Manish Pradhan Oct 3 '12 at 17:55
1  
The problem here is that you can't really have peristant vars across requests. You could with a session but a bot is likely to ignore your session cookie, making each request appear as a new client. Writing to a database or flat file 400+times per second AND checking that file every requests is a lot of work! If you do go the MySQL route, set your storage engine to MEMORY so it will use a ram storage instead of writing to the harddrive since you need speed and dont need permanent storage across reboots –  WebChemist Oct 3 '12 at 21:48
    
can you modify the apache conf files? If so, take a look at the limitipconn mod mentioned here: stackoverflow.com/questions/8248982/… to cut down the number of requests. And then add a usleep(100000) at the top of each page to make each request delay 1/10th of a second - not enough for your visitors to notice but enough to slow down a bot requesting a page every 1/400 of a second. –  WebChemist Oct 3 '12 at 21:50
    
added test code for you to play around with, let me know if that works in the real world if you do use it –  WebChemist Oct 3 '12 at 23:32
add comment

Thanks for the great code. I have modified it in such a way that it will ban anything without a session for 1 whole hour if this trap is activated. I am calling this the Extreme Flood Trap. I know I will be banning even the good bots which leads me to my next question. Will serving the Searchbots with 403 for an hour have bad consequences for the site in the long run? Here is the code so far -

$limit      = 400;
$log_file   = 'ip_'.$_SERVER['REMOTE_ADDR'].'_'.time();
$ban_file   = 'ban_'.$_SERVER['REMOTE_ADDR'];

clearstatcache();
if(file_exists($ban_file)){  
    $banlimit = file_get_contents($ban_file)+3600; 
    if(time() < $banlimit){
        if (!tep_session_id()) {
            header('HTTP/1.1 403 Forbidden');
            exit;
        }
    }
}

//creats a file and addes 1 byte of data on each page request
$log = fopen($log_file, "a"); 
fwrite($log,'0');
fclose($log);    

//if the size of the log file is greater than the Request limit size then create a ban file
if(filesize($log_file) > $limit){
    $log = fopen($ban_file, "w"); 
    fwrite($log,time());
    fclose($log);
}

What do you think?? bad choice??

share|improve this answer
    
According to developers.google.com/webmasters/control-crawl-index/docs/… 4xx (client errors) - Google treats all 4xx errors in the same way and assumes that no valid robots.txt file exists. It is assumed that there are no restrictions. This is a "full allow" for crawling. Note: this includes 401 "Unauthorized" and 403 "Forbidden" HTTP result codes. –  Manish Pradhan Oct 4 '12 at 17:22
    
Good use of time() in the ban file for a temporary ban, I was going to suggest that then I saw you already implemented it. For Google, you can create a google webmaster account and control the crawl rate of your site in order to not get hammered. Pretty sure bing and yahoo have similar tools. I don't think session is necessary since a real user will not hammer your site like this and it would mean the ban would be ineffective against a bot that did use sessions –  WebChemist Oct 4 '12 at 19:10
    
Also I noticed you removed the folder path from ip and ban files, You should have a set folder location for these, otherwise unless you use a framework with a bootstrap file (ie all pages flow through /index.php?page=), it will scatter the ip files in the different directories where your pages are which would make them ineffective. –  WebChemist Oct 4 '12 at 19:16
    
Actually I just pasted the test code on here. I would be using more refined codes for real implementation. I wanted this script only for bots without sessions as I have a number of other scripts that will protect my site for those conditions. Thanks for your help. This has been an educational thread. –  Manish Pradhan Oct 5 '12 at 13:52
    
no problem my friend. I enjoy a good challenge, and I might just need to add something like this to my own site one day. –  WebChemist Oct 5 '12 at 17:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.