1

I've recently noticed that Bing is crawling THOUSANDS of essentially duplicated pages on my site, and have updated my robots.txt as such:

User-agent: *
Crawl-delay: 1
Allow: /
Disallow: /report-js-error
Disallow: /*sort1by
Disallow: /*sort2by
Disallow: /*sort3by
Disallow: /*sort4by
Disallow: /*sort5by
Disallow: /*sort6by
Disallow: /*sort7by
Disallow: /*sort9by
Disallow: /*sort10by

As that's how I construct URLs if users have sorted tables on the page.

e.g.

www.example.com/results/sort1by4/sort9by3/

Sorts table 1 by column 4 and table 9 by column 3.

I'd like Bing NOT to crawl these pages, as they're numerous. A page with 3 tables with 10 columns each ends up with 1000 variations...

So I've created my robots.txt as above. The catch is, Bing fetched the robots.txt file yesterday, but is still crawling those URLs. If I copy and paste one of them into the Bing Webmasters robots.txt test page, it successfully tells me such URLs are blocked. But it's still crawling them.

I've added the meta tag value noindex to the pages as well now, but that only stops them from being indexed, not crawled...

2
  • How long has it been since you updated robots.txt? May 27, 2023 at 22:17
  • A couple of days, and bing re-fetched it within a few hours. It does seem in the last 12 hours that the requests are starting to drop off, I'd say it took about 24 hours after fetching it. I guess they have lots of crawlers and it takes a while for the new information to propagate between them, or similar.
    – Codemonkey
    May 28, 2023 at 6:41

1 Answer 1

1

Bingbot and Googlebot are distributed crawlers. They run on scores of computers spread through different data centers. Just because one of these instances has downloaded your robots.txt file, it doesn't mean that it instantly shares it every other instance.

They promise that each instance will fetch your robots.txt file every 24 hours and obey the most recent version of robots.txt that it has locally. That means that when you update your robots.txt file, it will take a full day before crawlers totally obey the new rules in it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.