6

Here is a weird one that I am not sure what to do. Today our companies e-commerce site went down. I tailed the production log and saw that we were receiving a ton of request from this range of IP's 157.55.98.0/157.55.100.0. I googled around and come to find out that it is a MSN Web Crawler.

So essentially MS web crawler overloaded our site causing it not to respond. Even though in our robots.txt file we have the following;

Crawl-delay: 10

So what I did was just banned the IP range in iptables.

But what I am not sure to do from here is how to follow up. I can't find anywhere to contact Bing about this issue, I don't want to keep those IPs blocked because I am sure eventually we will get de-indexed from Bing. And it doesn't really seem like this has happened to anyone else before.

Any Suggestions?

Update, My Server / Web Stats

Our web server is using Nginx, Rails 3, and 5 Unicorn workers. We have 4gb of memory and 2 virtual cores. We have been running this setup for over 9 months now and never had an issue, 95% of the time our system is under very little load. On average we receive 800,000 page views a month and this never comes close to bringing / slowing down our web server.

Taking a look at the logs we were receiving anywhere from 5 up to 40 request / second from this IP range.

In all my years of web development I have never seen a crawler hit a website so many times.

Is this new with Bing?

Improve this question
6
  • 2
    To make this a programming question, you could ask how to fix your site so it performs well enough to be on the public internet.
    – Greg Hewgill
    Dec 7, 2011 at 20:22
  • You are asking the right question at the wrong place ;)
    – dragosrsupercool
    Dec 7, 2011 at 22:00
  • If some crawler breaks your site there is probably more a problem with your site than with the crawler (besides the fact that the crawler doesn't respect your robots.txt which you should never trust they do :-) ).
    – PeeHaa
    Dec 7, 2011 at 22:33
  • It's possible that your robots.txt file's directives are not being honored because of a formatting problem - can you post the entire contents of the file for review?
    – danlefree
    Dec 7, 2011 at 23:18
  • @PeeHaa, thanks for the response, yeah I guess I can't trust that a crawler will respect my robots.txt file. But I updated my question with some server stats. If you have any other insight I would greatly appreciate it.
    – akaDanPaul
    Dec 7, 2011 at 23:24

3 Answers 3

Reset to default
4

Sign up with Bing webmaster tools and fill out their crawl speed chart. Set it for fastest crawling during your off hours and a much reduced rate during your busiest times.

If Bing is knocking over your website, you need to rethink your web server capacity. The best test is to see if you can survive Google, Bing, Yahoo and Baidu all hitting your system at once. If it remains in service during the onslaught, then you're ready for a live customer load.

Yes, Bing can hit you pretty hard if you haven't given them a limit. It was causing me serious issues here two months ago. I just tuned the system up to handle it and it was a good thing, otherwise Black Friday would have resulted in a very Blue Monday after viewing the server stats.enter image description here

Improve this answer
5
  • I have done that - and not work at all....
    – Aristos
    Sep 17, 2013 at 21:33
  • Did you install the file that identifies your website to BWT and check that they've verified it? If Bing can't id the site, the crawl rate histogram will do nothing at all for limiting traffic.
    – Fiasco Labs
    Sep 18, 2013 at 3:45
  • Yes, my sites are verified, I just check it. I have cut them now with firewall... to calm down, however Bing support are very friendly, I have contact with them, they suggest me to add on the robots.txt this line crawl-delay: 10 that is not work ether, and now they ask me for the logs, that I have all ready send them, to look them out.
    – Aristos
    Sep 18, 2013 at 8:24
  • "The best test is to see if you can survive Google, Bing, Yahoo and Baidu all hitting your system at once. If it remains in service during the onslaught, then you're ready for a live customer load." << Customers generate money, if Google, Bing, Yahoo and Baidu are sending ZERO traffic, they should be blocked in my opinion. Using crawlers abusing a site as a way to load test a website in my opinion makes zero sense.
    – blunders
    Jun 20, 2014 at 20:04
  • 1
    @blunders - and since Bing, Google and Yandex are the majority traffic sources on our website, we have to survive all of them scanning our website simultaneously. Guess what happens to Baidu here --> Scrapheap. Heh, statement still holds true that your website will need to withstand being indexed by the web crawlers you choose to let in or it is no website at all.
    – Fiasco Labs
    Jun 23, 2014 at 5:00
2

Use PHP plus Regex. Forget the Robots.txt. Several bad bots don't respect it...

if (preg_match('/(?i)bingbot/',$_SERVER['HTTP_USER_AGENT']))
{
exit(); 
}

And you tell for Bing: The door is closed for you!

Improve this answer
4
  • 2
    Good choice if you don't depend on Bing/Live/MSNSearch for incoming traffic. This will completely deindex your website with them and do a pretty good job of reducing web server loading.
    – Fiasco Labs
    Dec 20, 2012 at 5:50
  • Thanks Fiasco. In my opinion, BingBot is an evil bot, as they work like a Web Ripper. If the webmaster has dependence of Bing for make revenue, so he need to consider buying more and more resource to work on it. Bing was banned on all my 95 sites. Good luck for all you.
    – Mike Niner
    Dec 20, 2012 at 18:07
  • 1
    My comment was almost, but not quite tongue in cheek. I've had both Yahoo and Bing hit my site at once and nearly take the site to its knees. The loading was worse than Yandex which in the past has caused me grief. Yandex actually has upgraded their internal operations to work more like Google and not strain stuff so terribly. Baidu and Bing are on equal terms now for being overaggressive and requiring server tuning to handle the extra traffic.
    – Fiasco Labs
    Dec 20, 2012 at 18:14
  • If denying based on user agent is desired, you'd better deny them in your server configuration.
    – Koen.
    Nov 29, 2013 at 19:45
2

There are two ways of controlling the Bingbot; see http://www.bing.com/webmaster/help/crawl-control-55a30302 for details.

If you don't want to use their control panel just use a robots.txt file.

"If we find a crawl-delay: directive in your robots.txt file then it will take always precedence over the information from this feature."

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.