Why deny access to website for msnbot/bingbot?

Question

I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for hacking and spamming.

But why are there so many sites recommending to ban msnbot/bingbot?
Since it's a search engine, even if only with a marginal market share, I would except one would want this bot to crawl one's sites.

What is it that msnbot does that makes people ban it?

Answer 1

msnbot is quite prolific when it comes to spidering servers and if you have a lot of pages to index it can quite easily cripple your server. As traffic from MSN is considerably less than what Google can give it's quite common just to deny the msnbot via .htaccess, iptables or robots.txt. With Googlebot you can limit the speed quite easily in http://google.com/webmasters

Answer 2

I don't think people should ban bing bot.

Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/ where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video: http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jz

Googlebot is just as notorious in excessive crawling of sites as msnbot. Also the better (traffic/linkage) your site gets, the more googlebot crawls. Just look how fast stackexchange questions get indexed after being posted. You can see how much these bots hit up your server if you check your access logs.

I also discovered that msnbot supports robots.txt Crawl-delay parameter. http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

Answer 3

One of my clients was doing $10,000 monthly from Bing shopping alone. Organics from Bing was even more. Banning them would cause a big loss of revenue. Anyone suggesting it must have their own personal reasons. Bing generates visits so if you want to decrease your traffic go a head and ban Bing. Otherwise like Anthony said you can work with their Webmaster Tools to better your site for Bing.com

Answer 4

MSNBot is extremely aggressive and has sucked up over 2.5GB of bandwidth from many of my sites in less than a month (that's 2.5GB+ for each site). Microsoft really needs to straighten that out but probably never will. Until then, I'm treating MSNBot as the malicious program it is and banning it from my systems.