Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site?

My first thought is that allowing bots onto the site can potentially bring real traffic to it. Is there any reason to allow bots that are not known to be sending real traffic onto a site, and how do you spot these "good" bots?

share|improve this question
1  
+1: good question; however it's hard to answer your question because there are so many bots. –  Zistoloen Jun 13 at 8:24
    
@Zistoloen: Yes, I'm aware it is a hard question; in fact, reason I asked is because a non-major search engine I know of that has indexed billions of pages was complaining about how they were unable to access large amounts of the web because sites were attempting to block non-major search engines. –  blunders Jun 13 at 12:44
1  
Related: en.wikipedia.org/wiki/Spider_trap –  Mooing Duck Jun 13 at 17:18
    
@blunders Thanks for taking the time. I would have edited it myself if I was able to parse the question :) –  DisgruntledGoat Jun 13 at 19:03
    
@DisgruntledGoat: No problem, thanks for the edits! –  blunders Jun 13 at 19:50

2 Answers 2

up vote 11 down vote accepted

Within the realm of normal bots, it all depends on what you appreciate and only you can decide that. Of course there is Google, Bing/MSN/Yahoo!, Baidu, and Yandex. These are the the major search engines. There are also the various SEO and backlink sites. Right or wrong, I allow a couple of the big ones have access to my site, but generally, they are useless sites. I block archive.org not only in robots.txt, but by domain name and IP address. This is because they ignore robots.txt big time! This is something that you need to get a feel for. Do not get fooled by agent names. Often they are forged by bad people. Now days, I am getting thousands of page requests from sources claiming to be Baidu, but are not. Get to know these spiders by domain names and IP address blocks and learn to deal with them on that level. The good ones obey robots.txt.

But I must warn you, there are a TON of stealth bots, rogue bots, scrapers, and so on that you will want to search your log analysis frequently and block. This 5uck5! But it has to be done. They largest threat from them these days are low quality links to your site. My updated anti-bot security code I implemented this year has dropped 7700 low quality links automatically. Of course, my code still needs work, but you get the point. The bad bots still steal site potential.

It won't be long before you get the hang of it.

share|improve this answer

I had problems with Baidu bots slowing down my server while the search engine was sending almost no traffic. These bots do not respect the robots.txt file so to block Baidu bots just paste the following into your htccess file.

# User-agent: Baiduspider
# Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
# Baiduspider+(+http://www.baidu.com/search/spider.htm)

# IP range
# 180.76

RewriteCond %{REMOTE_ADDR} ^180\.76\. [OR]
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC]
RewriteRule .* - [F,L]

I've also had problems with Bing/Microsoft spiders crawling too fast, unlike Baidu they do respect the robots.txt file so;

User-agent: bingbot
Crawl-delay: 1

User-agent: msnbot
Crawl-delay: 1
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.