3

Bingbot is making these bogus requests to resources that has never been a thing on our domain. We see the same behavior in client domains leading us to the conclusion this is not an inheritage from a previous domain owner, but simply bogus. No other crawlers are doing this.

Does anyone know why Bingbot is doing this? And should we tackle these requests with a HTTP 404, 410, or maybe something just as bogus like HTTP 418?

Examples of requests:

  • /volleyplayersit
  • /volley-mercato
  • /mayflower-seafood-restaurant-reidsville-online-menu
  • /kfc-kentucky-fried-chicken-reidsville-online-menu
  • /conceptevents/quinceanera-dresses

A raw request from hostname msnbot-xxx-xx-xx.search.msn.com:

GET /kfc-kentucky-fried-chicken-reidsville-online-menu HTTP/1.1
Host: ...
Cache-Control: no-cache
Pragma: no-cache
Accept: */*
Accept-Encoding: gzip, deflate
from: bingbot(at)microsoft.com
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36
X-HTTPS: 1

Note: I am fairly certain that this is not related to any spam infections or CDN Proxy glitching.

6
  • This happens to me on my web site. There's a set of about a dozen URLs that never existed on my domain (I'm first owner, its an Israeli domain, and the URLs are in Spanish), and Bing comes around every two weeks asking for two of them at random. Seems a common enough problem for a Google search to come up with several results, including from MS forums from a few years back where MS said they're working on it. When I searched for a solution at the time, the only one I could find was excluding the relevant URLs in robots.txt, which does work for me.
    – Uri Raz
    Feb 12, 2023 at 19:13
  • @UriRaz Thank you. Are you able to post some of those links for reading? I managed to find people having the same problems. But no one really explaining why this happens or how to tackle it.
    – tim
    Feb 13, 2023 at 6:49
  • 1
    Have you reverse lookup'd the referring IP's? I've experienced this in the past and it turned out to be coming from a hacked website. This answer of mine might be relevant webmasters.stackexchange.com/a/136613/122139 Feb 14, 2023 at 19:51
  • 1
    @MikeCiffone Yes, reverse dns lookup resolves to hostnames like msnbot-xxx-xx-xx.search.msn.com.
    – tim
    Feb 15, 2023 at 8:19
  • Presumably your site has always returned a 404 HTTP status for these "bogus" URLs? Is Bingbot crawling the same "bogus" URLs over time, or do they continue to vary?
    – MrWhite
    Feb 16, 2023 at 22:33

4 Answers 4

2

A 404 would be the conventional response and technically correct one but a 410 would be better as its considered permanent - unlike 404 - and tells the search engine it should be removed from its engine.

A 418 is only suitable for use on February 30-31, except for testing. Configuration and testing of 418 should only ever be done on 1 April or days not ending in "y".

1
  • LOL :-) Thanks for providing an awesome answer. I will await a few more answers before accepting one to see if someone also have information on why Bingbot is doing this.
    – tim
    Feb 13, 2023 at 6:27
1

The best page I've seen is https://answers.microsoft.com/en-us/bing/forum/all/bing-crawlers-never-drop-crawling-a-non-existing/9c56f5f2-4d80-4095-87b2-99ffdb346785

I can share my observation that Bing keeps requesting URLs that never existed, or takes a URL with caps and requests it with all lower case, and that disallowing the URLs on robots.txt does make Bing stop requesting them.

Bing being proprietary software, only Microsoft can explain why it's doing so. My guess is Microsoft doesn't keep full URLs, but rather breaks them into host and path, and uses a hash with too many collisions or a probabilistic data structure, like Bloom filter, to reconnect them. My guess is it saves them enough computing resources (including electricity to keep them running) for them to pay the price of spurious 404s and webmaster complaints.

0

Guessing here again. Bingbot will be crawling a different set of sites to others bots. And perhaps, on one of those sites they have put links to your website. The latter is quite a common occurrence.

If this is the case, if you respond with 404, Bingbot may continue exploring them. As @DavidGo's suggestion, a 410 might be more appropriate.

3
  • I have had this thought myself. This theory would allow anyone evil to just post a bunch of made up links for competitors allover the web and they would be crawled by Bingbot. Logic tells me Bingbot hopefully seeks a little trust in what they would waste resources in crawling. I'm not able to confirm or deny.
    – tim
    Feb 15, 2023 at 8:12
  • @tim, there are many many posts here of people complaining about such links. It's not just Bingbot, it's also Google. On my websites, Google has these links. They are small in number so I haven't busted my gut yet. Feb 15, 2023 at 10:20
  • I haven't came across this problem with Googlebot, Yahoo! Slurp, Petalbot, DuckDuckBot, or Yandex Bot so this might not be one of those cases you are referring to.
    – tim
    Feb 16, 2023 at 6:43
0

If you don't have much traffic from Bing (most people don't). Consider blocking Bing altogether. We found Bing was making 40k+ requests to one of our domains, causing server issues and even some loss of Google traffic. Blocking Bing Bot fixed everything.

Letting Bing Bot crawl has very little upside but a massive possible downside with bogus requests, excessive crawling and other issues.

2
  • This can be a short term solution to problems with mass crawling. I would first try setting Crawl-delay in robots.txt.
    – tim
    May 3, 2023 at 10:12
  • @tim Not worth it. Bing would only respect Crawl delay if set for all bots.
    – JM John
    May 4, 2023 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.