2

Years ago Bingbot managed to register a bunch or URLs from some bad bot trap technology that used random query parameters for honey pots e.g.

  • /?c39cfn3
  • /?d37hvn3
  • /?28ce031

For years now we have had a regex rule returning HTTP 301 Moved Permanently for these requests redirecting to /. But Bingbot is still today requesting thousands of them every day and it seems it has no plan of stopping anytime soon. This is like 95% of all the redirect activity.

How do we once and for all stop Bingbot from continuously requesting these URLs that are permanently moved? (Without disallowing it completely or remove the root / resource from index.)

A request:

GET /?orneyt= HTTP/1.1
Host: ...
Cache-Control: no-cache
Pragma: no-cache
Accept: */*
Accept-Encoding: gzip, deflate
from: bingbot(at)microsoft.com
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36
X-HTTPS: 1

And the response:

HTTP/1.1 301 Moved Permanently
Location: /
...
Improve this question
2

2 Answers 2

Reset to default
1

Redirects are being checked regularly, so it's not a good option. In any case, I would start with the fundamentals, which is to check that there are no links in the content pointing out to those resources.

If those resources exist, then I would recommend using a 403 (Forbidden) response code which declares that access is limited to them and they shouldn't do so.

If those resources don't exist, I would recommend using a 410 (Gone) response code which declares that those resources are no longer available and that this condition is likely to be permanent.

Improve this answer
5
  • 1
    Agreed, or even 404.
    – Rohit Gupta
    Feb 10 at 11:23
  • This would suggest that Bing treats a HTTP 301 as a temporary redirect (HTTP 302). I can try HTTP 410 and see if there is any reduction by time. What I am afraid of is that it's gonna affect the indexing of the root / itself.
    – tim
    Feb 11 at 21:05
  • @RohitGupta Won't a 404 encourage Bingbot to revisit the URL again?
    – tim
    Feb 11 at 21:10
  • @tim, you are correct, 410 may be the best.
    – Rohit Gupta
    Feb 12 at 11:16
  • I wonder if the problem is more to do with the "?" in the URL, signifying its a parameter. What impact would a 404 or 410 have on the homepage associated with the URL. I posit a 400 (malformed request syntax/deciptive request routing) may be more appropriate. A 403 might also work but implies a refusal to show the content rather that this request is a client error or deceptive request routing.
    – davidgo
    Feb 13 at 0:36
0

I have recceived a reply from the Microsoft engineering team:

We use 410 and 404 as a content signal to avoid selecting such content in the future. It may take more than one crawls as many 404 and 410 comes back alive.

With that said I can only say 404 and 410 seems to be the right approach. I can confirm that after switching from 301 to 410 I have a month later started to see a reduction in the number of crawled pages.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.