0

I have several sites with dynamic links to products, where the product attributes are encoded into the URL. e.g.

I've created error handlers to catch any attempts to load non-existent products -- perhaps typos or discontinued products in static links to these products.

To my surprise, I'm getting lots of error messages of attempts to load products that have never existed, and they're all coming from bingbots, e.g.

I'm seeing misspellings of product codes, incorrect case, and lots of seemingly random product codes. Again, these are all coming from bingbots, and they're products that have never existed (nor had links to them that could have been crawled).

Does bing attempt to crawl random variations on dynamic URLs? I contacted Microsoft/Bing for support, and perhaps I needed to ask them to bump it up a tier, but they didn't seem to understand what I was asking.

ETA 04/30/20: There was an error in my original analysis. The incorrect product codes coming from bingbots seem to be limited to the ones with case issues. The more random strings seems to be coming from users.

1

Do you have IP addresses for the requests & can confirm they definitely originated from Bing?

HTTP referers can be easily spoofed (e.g. curl -e) making it appear that Bing was responsible for a request, when in reality it was some other bot etc. If that's the case, you should examine the traffic from the IP generating those requests in case it's trying anything dodgy... it could be trying to detect security holes (e.g. SQL injection opportunities) by sending mangled URLs, for example.

It seems unlikely that Bing would "invent" URLs for it to crawl... (unlikely, but not impossible!)

| improve this answer | |
  • Yes, I'm using the request IP, not referer. (Referer isn't set.) However, I discovered an error in my analysis after posting here. The hits from bingbots seem to be limited to the ones with case issues (uppercase, lowercase product codes). Seems MS is translating them at some point, which doesn't seem quite as unlikely as inventing them. :) The ones with unrelated text appear to be coming from other sources... possibly a user altering the URL. There aren't any control characters in most of these, so SQL injection doesn't appear to be the goal on these. – Mark Apr 30 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.