1

I have a site with multiple subdomains. On certain hours of the day, Bingbots would gather at my site and do a massive scan like this:

01:23:11 a.example.com GET /index HTTP/1.1 200 Bot.A
01:23:11 b.example.com GET /index HTTP/1.1 200 Bot.A
01:23:11 c.example.com GET /index HTTP/1.1 200 Bot.A
01:23:11 d.example.com GET /index HTTP/1.1 200 Bot.A
01:23:12 e.example.com GET /index HTTP/1.1 200 Bot.A
01:23:12 f.example.com GET /index HTTP/1.1 403 Bot.A
01:23:12 g.example.com GET /index HTTP/1.1 403 Bot.A
01:23:22 h.example.com GET /index HTTP/1.1 200 Bot.B
01:23:22 i.example.com GET /index HTTP/1.1 200 Bot.B
01:23:22 j.example.com GET /index HTTP/1.1 200 Bot.B
01:23:22 k.example.com GET /index HTTP/1.1 200 Bot.B
01:23:23 l.example.com GET /index HTTP/1.1 200 Bot.B
01:23:23 m.example.com GET /index HTTP/1.1 403 Bot.B
01:23:23 n.example.com GET /index HTTP/1.1 403 Bot.B

As the bots are scanning across multiple subdomains, the Crawl-delay: 1 directive in robots.txt would have no effect on such behaviour. The server defence mechanism would kick in and block these crawlers by issuing 403 errors.

Is there a way to spread out BingBot's crawling evenly? The default crawl pattern on Bing's webmaster tool doesn't seem to be followed.

  • Check the requester domain name or IP address. I bet it is not Microsoft. Can you update us with one or more domain names and/or IP addresses? – closetnoc Sep 24 '14 at 3:25
  • @closetnoc, I am quite sure those IPs are from Microsoft as I have been running whois and host on them. Example a whois on 207.46.13.100 says "NetName: MICROSOFT-GLOBAL-NET". A PTR lookup says "msnbot-207-46-13-100.search.msn.com". Also, the webmaster tool reported these 403 errors. – Question Overflow Sep 24 '14 at 6:05
  • Okay. I am a bit surprised but I believe you. I have seen a lot of spoofing these days and rogue spiders/bot use Bing as an agent name a lot. I thought Bing was like Google where the requests would come about every 2 +/- seconds unless stated otherwise. You may want to up your crawl rate to 2 seconds or remove it all together to see what happens. – closetnoc Sep 24 '14 at 15:24
4

This is Vincent from Bing Webmaster Tools and I noticed your post.

First of all, I'm sorry to hear about the problem you are having with our crawler's crawl activity across your subdomains. I am sure we can do better.

Couple of things:

I noticed you mentioned crawl pattern setting in Webmaster Tools wasn't working for. The reason is that when using a Crawl-delay: directive in robots.txt and - this directive always gets precedence over any Crawl Control settings in the Bing Webmaster Tools, which is why this isn't working as expected (see the note in http://www.bing.com/webmaster/help/crawl-control-55a30302).

On the other hand, since having to mitigate this through several subdomain-specific robots.txt with different crawl-delay: directives isn't optimal I don't have a good self-service solution here which is why I suggest you contact Bing Webmaster Support and share the domain/sub-domain information so they can pass it to the right team to take a closer look (they may ask for server logs to help with the investigation).

To contact Webmaster Support go to http://go.microsoft.com/fwlink/p/?linkid=261881, fill out the required fields, and in the "What type of problem do you have?" dropdown, select "Under-Crawling or Over-Crawling inquiry" and describe the problem you are seeing. Even if they don't come back with a personalized response immediately (it can take 24-48 hrs.), this should at least get the ball rolling.

  • 1
    Welcome to Pro Webmasters - thanks for your input here! – dan Sep 24 '14 at 23:36
  • Hi Vincent, thank you for your attention. Just to clarify some details. The webmaster crawl pattern has always been at the default crawl rate with default uniform pattern. I was having 403 errors from BingBot before I decided to add in the crawl delay directive, hoping to mitigate the problem. But the situation did not improve. I will file a report and see how it goes from there. Having said that, it would be good if BingBot can be optimized to crawl one subdomain at a time to avoid overloading websites with multiple subdomains since there is no good way to control this with robots.txt – Question Overflow Sep 25 '14 at 3:17
1

In my experience, MSN Bots have always been too unruly for robots.txt and they bring no discernable traffic advantage. For the past three years I have become very intolerant of their terrible behaviour. And yes, they are from Microsoft.

  • 403 them with RewriteCond %{HTTP_USER_AGENT} in .htaccess

  • Use fail2ban or mod_security to block or rate-limit-block them

  • Both of the above

  • Collect the log data and null route them

They do not deserve access.

  • I have not had any trouble with the bot, but unless you are listed in Bing, they are useless and waste a lot of resources I grant you that! Bing is cutting it's own throat with the notion of being choosy. Google did not become #1 by being choosy. It got to be #1 by indexing as much as it could day after day after day and giving newer and smaller sites a shot to compete on merit on scale rather than waiting till they got big and had a lot of backlinks. It is a foolish strategy. At least that is how I see it. – closetnoc Sep 24 '14 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.