4

We host a large number of sites that have a large number of catalog pages. We would like to slow down several bots because the traffic is quite excessive from those bots. Specifically, we get quite a bit of traffic from BingBot and SEMRushBot that we'd like to slow down. The information pages for both bots say that they obey the crawl-delay directive. However, despite changing the crawl delays for both, I am seeing no change in traffic even after several days. Is there something wrong with my file? (I put 60 into the SEMRushBot delay but I read that they only delay for a max of 10 seconds. I have seen no change at all after adding them to the robots.txt file).

User-agent: *
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: yahoo-mmcrawler
Disallow: /m/
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Crawl-delay: 60
User-agent: Bingbot
Crawl-delay: 10
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
User-agent: dotbot
Crawl-delay: 1
User-agent: Goodzer
Crawl-delay: 1
User-agent: rogerbot
Crawl-delay: 5
User-agent: Baiduspider
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: Linguee Bot
Disallow: /
User-agent: Seekport Crawler
Disallow: /
User-agent: GrapeshotCrawler
Crawl-Delay: 1
User-agent: istellabot
Disallow: /
User-agent: SeznamBot
Disallow: /
Sitemap:
Improve this question
1
  • Have you verified that they have re-checked your robots.txt? See if your logs have requests for the file from those user agents.
    – DisgruntledGoat
    Commented Jul 2, 2020 at 22:59

2 Answers 2

Reset to default
4

Your robots.txt file does not do what (I think) you intend it to do because you do not use blank lines between the sections. The robots.txt standard says (emphasis added):

The format logically consists of a non-empty set or records, separated by blank lines. The records consist of a set of lines of the form:

<Field> ":" <value>

In this memo we refer to lines with a Field "foo" as "foo lines".

The record starts with one or more User-agent lines, specifying which robots the record applies to, followed by "Disallow" and "Allow" instructions to that robot. For example:

User-agent: webcrawler
User-agent: infoseek
Allow:    /tmp/ok.html
Disallow: /tmp
Disallow: /user/foo

So when you have multiple sections for different user agents, there need to be blank lines between them. You can also assign multiple user agents to the same rule. Your robots.txt should be:

User-agent: *
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/

User-agent: yahoo-mmcrawler
Disallow: /m/

User-agent: SemrushBot
Crawl-delay: 60

User-agent: Bingbot
Crawl-delay: 10
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/

User-agent: dotbot
User-agent: Goodzer
Crawl-delay: 1

User-agent: rogerbot
Crawl-delay: 5

User-agent: Baiduspider
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: Linguee Bot
User-agent: Seekport Crawler
User-agent: GrapeshotCrawler
User-agent: istellabot
User-agent: SeznamBot
Disallow: /

Keep in mind that if you provide a separate section for a bot, it no longer follows the User-agent: * directives. So Semrushbot, dotbot, Goodzer, and rogerbot are allowed to crawl your entire site including /nobots/. yahoo-mmcrawler is similarly allowed to crawl almost everything.

Even a crawl delay of 1 second significantly slows down the rate at which crawlers can fetch documents to the point that your server shouldn't notice additional load from them. A typical server can handle hundreds or even thousands of requests every second. I'd recommend putting a default Crawl-delay: 1 and not try to customize it for different bots. Most bots ignore it completely, but those that honor it should still be able to crawl hundreds of pages from your site in a reasonable time period. I think this robots.txt file would better match your intent and slow crawlers enough to prevent server overload:

User-agent: *
Crawl-delay: 1
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/

User-agent: yahoo-mmcrawler
Crawl-delay: 1
Disallow: /nobots/
Disallow: /products/features/
Disallow: /product/features/
Disallow: /product/reviews/
Disallow: /webservices/ajax/
Disallow: /m/

User-agent: Baiduspider
User-agent: MJ12bot
User-agent: AhrefsBot
User-agent: YandexBot
User-agent: YandexImages
User-agent: Linguee Bot
User-agent: Seekport Crawler
User-agent: GrapeshotCrawler
User-agent: istellabot
User-agent: SeznamBot
Disallow: /
Improve this answer
1
  • Thank you for the response. I took a quick look and our dynamic file generation is stripping out the lines I have in the file. I have been pulled onto something else at the moment but I'll give it a try soon.
    – ChipnCharge
    Commented Jul 7, 2020 at 16:27
3

Thanks for the question and welcome to this great community of like-minded individuals. You're query is perplexing however it is solvable. I see a couple of things that can be improved/changed in your robots.txt file per Bing's documentation on best practices.

  1. You say you host a large number of sites. Make sure that you have separate robots.txt files stored at the root folder of each site that you host.
  2. Bing recommends that any crawl-delay directive be made in the generic directive section for all bots to minimize the chance of code mistakes that can affect how a site is indexed by a particular search engine. I see that your crawl-delay directives are made for specific bots. Personally I would ignore this recommendation as you do not want to slow down Googlebot, as that's the only bot that matters ;)
  3. Also note that any crawl-delay directives set are applicable only on the web server instance hosting the robots.txt file.
  4. If you want to add customized directives for specific bots that are not appropriate for all bots, such as crawl-delay:, add them in a custom section after the first, generic section, changing the User-agent reference to a specific bot.
  5. And finally, as it relates to the crawl delay value, Bing recommends using the lowest value possible, if you must use any delay, in order to keep the index as fresh as possible with your latest content. Bing recommends against using any value higher than 10, as that will severely affect the ability of the bot to effectively crawl your site for index freshness.

enter image description here

Also note that Bing has a Crawl Control tool which allows you to exert control over the speed at which Bingbot makes requests. This tool can be found inside Bing's webmaster account.

I hope this helps. Good luck on your journey to everlasting search glory!

Improve this answer
2
  • Thank you for the response. I have been pulled onto something else and haven't had a chance to try #2. That may work for my situation.
    – ChipnCharge
    Commented Jul 7, 2020 at 16:03
  • I agree with this answer, and personally I would really NOT expect most bots respecting the crawl-delay directive (recommendation) in robots.txt. I usually put it in my robots.txt, but well knowing it's worth little to nothing. I'd just block out all those seo/stats bots and set google/bing/yandex in thir Webmaster Tools, that works :)
    – bobble bubble
    Commented Apr 22, 2022 at 10:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.