1

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about 30000 requests a day) among all the crawlers. Google has less requests a day.

Do you also experience heavy traffic from bingbot? Why doesn't it behave when I set crawling rate to a minimum?

1

Are you sure it is really a Bing bot? I have seen bots that ignore disallows that turned out to be from somewhere other than expected when doing a reverse IP lookup using dig.

dig -x 157.55.39.142

; <<>> DiG 9.9.5-3-Ubuntu <<>> -x 157.55.39.142
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30244
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;142.39.55.157.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
142.39.55.157.in-addr.arpa. 71  IN      PTR     msnbot-157-55-39-142.search.msn.com.

;; Query time: 57 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Jun 12 18:18:06 EDT 2014
;; MSG SIZE  rcvd: 104

The reverse lookup shows the IP is indeed related to Bing's search bots.

They do state that the crawl control directive in a robots.txt file will take precedence over a configured setting on their end.

http://www.bing.com/webmaster/help/crawl-control-55a30302

If that doesn't seem to be the case you may want to try using their controls to adjust their crawl for your site.

  • It is bingbot, the requests are coming from the 157.55.... range and reverse lookup points to bingbot. I haven't tried the robots.txt method yet (I thought using their controls is enough), but I'll give it a try. – Tom Jun 13 '14 at 5:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.