Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed 164 GB this month.

By comparison, Googlebot consumed 10 GB and visitors (viewed traffic) consumed 25 GB. This means rogue bots are consumming over 6X the bandwidth of visitors. For other sites which I run (about a dozen) the normal ratio is 25%, so for 25GB of viewed traffic, bots take about 6GB in TOTAL.

The question therefore is: How to identify which bot(s) are causing this huge amount of request and how to stop them or slow them down if they are useful?

Obviously, most bots that visit the site are important including the Googlebot, Yahoo Slurp, MSNBot, etc, including the AdSense/DoubleClick bots, so I cannot simply block all bots.

The reason I am investigating this is that I am reaching the limit of bandwidth and exceeded CPU usage for my host, so I was sent a notice.

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted
  1. Create a page that captures IP addresses of anyone who visits it. Add those IPs to an htaccess file that blocks it. (see example here)

  2. Link to that page in the footer of your website using 1px transparent image

  3. Block that page in robots.txt so good robots won't find it

Note: whitelisting good IPs and/or useragents is also a good idea (IP Addresses of Search Engine Spiders)

share|improve this answer
    
Thanks! Neat trick :) –  Itai Sep 28 '11 at 13:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.