I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots are generating over 1 GB of traffic per hour for my sites. The real human traffic for all of these sites is less than 100 MB / month.

I have done extensive research on both robots.txt and .htaccess file to block these bots but all methods failed.

I have also put code in the robots.txt files to block access to the scripts directories, but these bots (Google, MS Bing, and Yahoo) ignore the rules and run the scripts anyways.

I do not want to completely block the Google, MS Bing, and Yahoo bots, but I want to limit there crawl rate. Also, adding a Crawl-delay statement in the robots.txt file does not slow down the bots. My current robots.txt and .htacces code for all sites are stated below.

I have setup both Microsoft and Google webmaster tools to slow down the crawl rate to the absolute minimum, but they are still hitting these sites at a rate of 10 hits / second.

In addition, every time I upload a file that causes an error, the entire VPS webserver goes down within seconds such that I cannot even access the site correct the issue due to the onslaught of hits by these bots.

What can I do to stop the on-slot of traffic to my websites?

I tried asking my web hosting company (site5.com) many times about this issue in the past months and they cannot help me with this problem.

What I really need is to prevent the Bots from running the rss2html.php script. I tried both sessions and cookies and both failed.


User-agent: Mediapartners-Google
User-agent: Googlebot
User-agent: Adsbot-Google
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: MSNBot
User-agent: bingbot
User-agent: Slurp
User-Agent: Yahoo! Slurp
# Directories
User-agent: *
Disallow: /
Disallow: /cgi-bin/
Disallow: /ads/
Disallow: /assets/
Disallow: /cgi-bin/
Disallow: /phone/
Disallow: /scripts/
# Files
Disallow: /ads/random_ads.php
Disallow: /scripts/rss2html.php
Disallow: /scripts/search_terms.php
Disallow: /scripts/template.html
Disallow: /scripts/template_mobile.html


ErrorDocument 400 http://english-1329329990.spampoison.com
ErrorDocument 401 http://english-1329329990.spampoison.com
ErrorDocument 403 http://english-1329329990.spampoison.com
ErrorDocument 404 /index.php
SetEnvIfNoCase User-Agent "^Yandex*" bad_bot
SetEnvIfNoCase User-Agent "^baidu*" bad_bot
Order Deny,Allow
Deny from env=bad_bot
RewriteEngine on
RewriteCond %{HTTP_user_agent} bot\* [OR]
RewriteCond %{HTTP_user_agent} \*bot
RewriteRule ^.*$ http://english-1329329990.spampoison.com [R,L]
RewriteCond %{QUERY_STRING} mosConfig_[a-zA-Z_]{1,21}(=|\%3D) [OR]
# Block out any script trying to base64_encode crap to send via URL
RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [OR]
# Block out any script that includes a <script> tag in URL
RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
# Block out any script trying to set a PHP GLOBALS variable via URL
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
# Block out any script trying to modify a _REQUEST variable via URL
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
# Send all blocked request to homepage with 403 Forbidden error!
RewriteRule ^(.*)$ index.php [F,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !^/index.php
RewriteCond %{REQUEST_URI} (/|\.php|\.html|\.htm|\.feed|\.pdf|\.raw|/[^.]*)$  [NC]
RewriteRule (.*) index.php
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
# Don't show directory listings for directories that do not contain an index file (index.php, default.asp etc.)
Options -Indexes
<Files http://english-1329329990.spampoison.com>
order allow,deny
allow from all
deny from 108.
deny from 123.
deny from 180.
deny from


function botcheck(){
 $spiders = array(
   array('Yahoo! Slurp','help.yahoo.com')
 $useragent = strtolower($_SERVER['HTTP_USER_AGENT']);
 foreach($spiders as $bot) {
     $ipaddress = $_SERVER['REMOTE_ADDR']; 
     $hostname = gethostbyaddr($ipaddress);
     $iphostname = gethostbyname($hostname);
     if (preg_match("/$bot[1]/i",$hostname) && $ipaddress == $iphostname){return true;}
if(botcheck() == false) {
  // User Login - Read Cookie values
     $username = $_COOKIE['username'];
     $password = $_COOKIE['password'];
     $radio_1 = $_COOKIE['radio_1'];
     $radio_2 = $_COOKIE['radio_2'];
     if (($username == 'm3s36G6S9v' && $password == 'S4er5h8QN2') || ($radio_1 == '2' && $radio_2 == '5')) {
     } else {
       $selected_username = $_POST['username'];
       $selected_password = $_POST['password'];
       $selected_radio_1 = $_POST['group1'];
       $selected_radio_2 = $_POST['group2'];
       if (($selected_username == 'm3s36G6S9v' && $selected_password == 'S4er5h8QN2') || ($selected_radio_1 == '2' && $selected_radio_2 == '5')) {
         setcookie("username", $selected_username, time()+3600, "/");
         setcookie("password", $selected_password, time()+3600, "/");
         setcookie("radio_1", $selected_radio_1, time()+3600, "/");
         setcookie("radio_2", $selected_radio_2, time()+3600, "/");
       } else {
        header("Location: login.html");

I also added the following to the top tof the rss2html.php script

// Checks to see if this script was called by the main site pages, (i.e. index.php or mobile.php) and if not, then sends to main page
   if(isset($_SESSION['views'])){$_SESSION['views'] = $_SESSION['views']+ 1;} else {$_SESSION['views'] = 1;}
   if($_SESSION['views'] > 1) {header("Location: http://website.com/index.php");}
  • When did you update your robots.txt? It can take time for the robots to read the updated version.
    – ilanco
    May 18, 2012 at 20:45
  • a few days ago. What I really need is to prevent the Bots from running the rss2html.php script. I tried both sessions and cookies and both failed.
    – Sammy
    May 18, 2012 at 20:46
  • How is rss2html.php being used by your site? via PHP include, redirects, Ajax....?
    – cHao
    May 18, 2012 at 21:00
  • The rss2html.php files is called via a file_get_contents() command
    – Sammy
    May 18, 2012 at 21:06
  • file_get_contents...? Seems rather odd. The file is on another server or something?
    – cHao
    May 19, 2012 at 11:03

11 Answers 11


You could set your script to throw a 404 error based on the user agent string provided by the bots - they'll quickly get the hint and leave you alone.

   $agent = $_SERVER['HTTP_USER_AGENT'];

   header("HTTP/1.1 301 Moved Permanently");
   header("Location: http://www.google.com/");

Pick through your logs and reject Bingbot, etc. in a similar manner - it won't stop the requests, but might save some bandwidth - give googlebot a taste of it's own medicine - Mwhahahahaha!


Looking at your code, I think your problem is here:

if (preg_match("/$bot[1]/i",$hostname) && $ipaddress == $iphostname)

If they are malicious bots then they could be coming from anywhere, take that $ipaddress clause out and throw a 301 or 404 response at them.

Thinking right up by the side of the box

  1. Googlebot never accepts cookies, so it can't store them. In fact, if you require cookies for all users, that's probably going to keep the bot from accessing your page.
  2. Googlebot doesn't understand forms - or - javascript, so you could dynamically generate your links or have the users click a button to reach your code (with a suitable token attached).

    <a href="#" onclick="document.location='rss2html.php?validated=29e0-27fa12-fca4-cae3';">Rss2html.php</a>

    • rss2html.php?validated=29e0-27fa12-fca4-cae3 - human
    • rss2html.php - bot
  • 1
    Not as quickly as you might think. I've seen bots hit the same nonexistent page for months, occasionally even years after the page was deleted. Depends on how well behaved the bot is and what it's after.
    – cHao
    May 18, 2012 at 20:48
  • The way is works for human visitors is that the index.php file calls the rss2html.php script. The bots avoid the index.php script and directly run the rss2html.php script. How can I protect that rss2html.php file if it was not accessed via the index.php script?
    – Sammy
    May 18, 2012 at 20:52
  • Try renaming rss2html.php to something else and updating your index.php to reference new name. May 18, 2012 at 20:54
  • I tried the renaming the file, but it fails after a few days. How can I add the code I am using to this thread? I want to show you what I have tried.
    – Sammy
    May 18, 2012 at 20:56
  • 1
    ok - I know a trick you can pull :) - put the rss2html.php script code outside of your website (will update answer)
    – web_bod
    May 18, 2012 at 21:19

If rss2html.php is not being used directly by the client (that is, if it's PHP always using it rather than it being a link or something), then forget trying to block bots. All you really have to do is define a constant or something in the main page, then include the other script. In the other script, check whether the constant is defined, and spit out a 403 error or a blank page or whatever if it's not defined.

Now, in order for this to work, you'll have to use include rather than file_get_contents, as the latter will either just read in the file (if you're using a local path), or run in a whole other process (if you're using a URL). But it's the method that stuff like Joomla! uses to prevent a script from being included directly. And use a file path rather than a URL, so that the PHP code isn't already parsed before you try to run it.

Even better would be to move rss2html.php out from under the document root, but some hosts make that difficult to do. Whether that's an option depends on your server/host's setup.

  • 1
    Chao, thanks. Currently, I am rewriting my code to convert the file_get_contents to include instead.
    – Sammy
    May 20, 2012 at 19:33

I've solved the same issue with the script available at http://perishablepress.com/blackhole-bad-bots/. With this blackhole approach I collected a list of malicious ip, and then using .htaccess denied them. (Which is not mandatory, since the script itself does the banning. But I need to reduce the server load by avoiding php parsing for known unwanted ips) in three days my traffic came down from 5GB per day to 300MB, which is quiet expected.

Check this page also for full list of htaccess rules to block many known junk bots. http://www.askapache.com/htaccess/blocking-bad-bots-and-scrapers-with-htaccess.html


PHP Limit/Block Website requests for Spiders/Bots/Clients etc.

Here I have written a PHP function which can Block unwanted Requests to reduce your Website-Traffic. Good for Spiders, Bots and annoying Clients.

CLIENT/Bots Blocker

DEMO: http://szczepan.info/9-webdesign/php/1-php-limit-block-website-requests-for-spiders-bots-clients-etc.html


/* Function which can Block unwanted Requests
 * @return array of error messages
function requestBlocker()
        Version 1.0 11 Jan 2013
        Author: Szczepan K
        me[@] szczepan [dot] info
        A PHP function which can Block unwanted Requests to reduce your Website-Traffic.
        God for Spiders, Bots and annoying Clients.


        # Before using this function you must 
        # create & set this directory as writeable!!!!
        $dir = 'requestBlocker/';

        $rules   = array(
                #You can add multiple Rules in a array like this one here
                #Notice that large "sec definitions" (like 60*60*60) will blow up your client File
                        //if >5 requests in 5 Seconds then Block client 15 Seconds
                        'requests' => 5, //5 requests
                        'sek' => 5, //5 requests in 5 Seconds
                        'blockTime' => 15 // Block client 15 Seconds
                        //if >10 requests in 30 Seconds then Block client 20 Seconds
                        'requests' => 10, //10 requests
                        'sek' => 30, //10 requests in 30 Seconds
                        'blockTime' => 20 // Block client 20 Seconds
                        //if >200 requests in 1 Hour then Block client 10 Minutes
                        'requests' => 200, //200 requests
                        'sek' => 60 * 60, //200 requests in 1 Hour
                        'blockTime' => 60 * 10 // Block client 10 Minutes
        $time    = time();
        $blockIt = array();
        $user    = array();

        #Set Unique Name for each Client-File 
        $user[] = isset($_SERVER['REMOTE_ADDR']) ? $_SERVER['REMOTE_ADDR'] : 'IP_unknown';
        $user[] = isset($_SERVER['HTTP_USER_AGENT']) ? $_SERVER['HTTP_USER_AGENT'] : '';
        $user[] = strtolower(gethostbyaddr($user[0]));

        # Notice that I use files because bots do not accept Sessions
        $botFile = $dir . substr($user[0], 0, 8) . '_' . substr(md5(join('', $user)), 0, 5) . '.txt';

        if (file_exists($botFile)) {
                $file   = file_get_contents($botFile);
                $client = unserialize($file);

        } else {
                $client                = array();
                $client['time'][$time] = 0;

        # Set/Unset Blocktime for blocked Clients
        if (isset($client['block'])) {
                foreach ($client['block'] as $ruleNr => $timestampPast) {
                        $elapsed = $time - $timestampPast;
                        if (($elapsed ) > $rules[$ruleNr]['blockTime']) {
                        $blockIt[] = 'Block active for Rule: ' . $ruleNr . ' - unlock in ' . ($elapsed - $rules[$ruleNr]['blockTime']) . ' Sec.';
                if (!empty($blockIt)) {
                        return $blockIt;

        # log/count each access
        if (!isset($client['time'][$time])) {
                $client['time'][$time] = 1;
        } else {


        #check the Rules for Client
        $min = array(
        foreach ($rules as $ruleNr => $v) {
                $i            = 0;
                $tr           = false;
                $sum[$ruleNr] = 0;
                $requests     = $v['requests'];
                $sek          = $v['sek'];
                foreach ($client['time'] as $timestampPast => $count) {
                        if (($time - $timestampPast) < $sek) {
                                $sum[$ruleNr] += $count;
                                if ($tr == false) {
                                        #register non-use Timestamps for File 
                                        $min[] = $i;
                                        $tr = true;

                if ($sum[$ruleNr] > $requests) {
                        $blockIt[]                = 'Limit : ' . $ruleNr . '=' . $requests . ' requests in ' . $sek . ' seconds!';
                        $client['block'][$ruleNr] = $time;
        $min = min($min) - 1;
        #drop non-use Timestamps in File 
        foreach ($client['time'] as $k => $v) {
                if (!($min <= $i)) {
        $file = file_put_contents($botFile, serialize($client));

        return $blockIt;


if ($t = requestBlocker()) {
        echo 'dont pass here!';
} else {
        echo "go on!";

It's likely that your site is being indexed by fake google bot(s). You can try adding a check and serve 404 for all fake google bot requests.

Here is an article that explains how to verify Googlebot: http://googlewebmastercentral.blogspot.com/2006/09/how-to-verify-googlebot.html

Also you could check your records against known fake bots: http://stopmalvertising.com/security/fake-google-bots.html

  • Thanks, but actually I tried that too, determining the user agents and sending the bots to a login page. This also failed.
    – Sammy
    May 18, 2012 at 20:54
  • Sounds like you're missing the point - relying on user agent for determining the bot authenticity isn't sufficient. May 18, 2012 at 20:56

You should really make sure in a first place that any page requested from a useragent, of whichever abusive crawler you have, will be served a static page.

An apache mod_rewrite with a condition or equiv with your http server. For apache, something like this:

RewriteCond  %{HTTP_USER_AGENT}  ^GoogleBot [OR]
RewriteCond  %{HTTP_USER_AGENT}  ^OtherAbusiveBot
RewriteRule  ^/$                 /static_page_for_bots.html  [L]
  • thanks, but I do not want to completely block the Google, MS Bing, and Yahoo bots, but I want to limit there direct hits on the rss2html.php script file. I just need to add something to the beginning of the rss2html.php script that will prevent it from running if it was not accessed via the index.php script. The bots are currently running the rss2html.php script bypassing the index.php file.
    – Sammy
    May 18, 2012 at 21:48
  • This doesnt block them.. you simply serve a cached version of your php.. this is very easy for a server todo, it's one less php instance/one less apache child process. => Cost(static file) < Cost(php instance).
    – smassey
    May 18, 2012 at 21:56
  • how would I cache the pages? Since the pages are RSS, will the cached pages be refreshed often enough to provide fresh data?
    – Sammy
    May 18, 2012 at 22:05
  • Of course... Write a cronjob which does it for you. If you say they hit the server 10req/s if you cache the pages for 1minute you've saved your server 599 extra php instances (which certainly include db connections/queries).. And once a minute is much more than what I would vote for: 10/15min.
    – smassey
    May 18, 2012 at 22:41

To continue on smassey's post, you can put several conditions:

RewriteCond  %{HTTP_USER_AGENT}  ^GoogleBot [OR]
RewriteCond  %{HTTP_USER_AGENT}  ^OtherAbusiveBot
RewriteRule  ^rss2html\.php$     /static.html  [L]

This way, the bots still access your pages, but just not that one. As it's strange that the (legitimate) bots are not keeping to rules, do you have any referers that push bots to your page from other sources (domainname forwarding, ...)

// Checks to see if this script was called by the main site pages,
// (i.e. index.php or mobile.php) and if not, then sends to main page
if (isset($_SESSION['views'])) {$_SESSION['views'] = $_SESSION['views']+ 1;} else {$_SESSION['views'] = 1;}
if ($_SESSION['views'] > 1) {header("Location: http://website.com/index.php");}

This script does not do what the comment says, in fact it does the complete opposite. This will always let the bots through, since the session variable will never be set when the bot requests your script. All it will potentially do is prevent legitimate requests (from index.php or mobile.php) from calling the script more than once.

In order to prevent a bot from accessing your script you should only allow access if a session variable (or cookie) is actually set. Assuming of course that the (malicious) bot does not accept cookies. (We know that the real Googlebot doesn't.)

As has already been mentioned, placing rss2html.php above the web root (outside of the public webspace) would prevent a bot from accessing the script directly - but you say this causes other problems? Or, place it in a directory and protect that directory with .htaccess. Or you might even be able to protect just the file itself in .htaccess from direct requests?


Go set up your domain on Cloudflare (free service for this). They block malicious bots at the domain level before they hit your server. Takes about 20 minutes, never have to monkey with the code.

I use this service on all my sites and all client sites. They identify malicious bots based on a number of techniques including leveraging project Honey pot.


Old question, but current answers don't appear to have addressed the "error"(s) in the robots.txt file...

I have also put code in the robots.txt files to block access to the scripts directories, but these bots (Google, MS Bing, and Yahoo) ignore the rules and run the scripts anyways.

Yes, they are actually obeying the robots.txt file as written. This robots.txt file does not prevent these bots (Google, MS Bing, and Yahoo) from crawling the scripts directories!

When a bot parses the robots.txt file, it only matches against at most one group of directives, identified by the User-agent directive(s). The "special" User-agent: * (all bots) group only applies if the bot does not match against any other - more specific - group. It is not always processed, as appears to be the assumption here. All the named bots in this robots.txt file are permitted to crawl everything.

For "Google, MS Bing and Yahoo" they each have their own specific group that allows unrestricted crawling of everything (ie. Disallow: ). Consequently they don't see the directives in the User-agent: * group that blocks specific URLs.

User-agent: *
Disallow: /
Disallow: /cgi-bin/
Disallow: /ads/

The first Disallow: / directive blocks "all other" bots from crawling everything. The more specific Disallow directives that follow are entirely superfluous.

What I really need is to prevent the Bots from running the rss2html.php script. I tried both sessions and cookies and both failed.

To prevent all (good) bots from crawling rss2html.php then all you would need is the following and no specific groups for the other bots:

User-agent: *
Disallow: /scripts/rss2html.php

What you need to do is install a SSL Certificate on your server for apache/nginx/email/ftp. Enable HSTS and also you need to edit your ssl.conf file so that SSLv2 SSLv3 TLSv1 are disabled and doesn't allow incoming connections. Strengthen your server the right way and you will not have any issues from bots.

  • 2
    It is not clear to me what problem SSL/TLS is solving in this case. It looks like you are begging the question and arriving at the result. Please explain how this solution controls the problem.
    – user40589
    Sep 3, 2018 at 12:41
  • SSL/TLS certificates only encrypt the traffic to the server. They don't filter it. Nov 27, 2020 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.