1

I had a very simple robots.txt file setup for a site I maintain. After a spike of traffic that the ISP put down to crawlers they suggested I add a crawl delay directive which is fair enough. So I ended up with this file

User-agent: *
Disallow: /a-page-i-wanted-to-ignore
Crawl-delay: 1

I still receive spikes in traffic that are causing downtime. The ISP told me in relation to the robots file I had setup (shown above) that "the crawl delay is configured only for the '/a-page-i-wanted-to-ignore ' page and not for the entire website."

I wanted to check, is that correct? If a command like crawl-delay placed under a 'disallow' line will it be specific to the disallow clause ?

2

The ISP told me that this directive has only defined the crawl delay for the page /a-page-i-wanted-to-ignore

That does not make sense. The crawl-delay directive has nothing to do with the preceding Disallow directive.

The page /a-page-i-wanted-to-ignore shouldn't be crawled at all - since it's included in the Disallow directive, so if the crawl-delay should apply to that page it's in direct contradiction to what the Disallow directive does. Maybe if it followed an Allow directive - but no, it does not work that way.

Your crawl-delay directive applies to the group User-agent: *, ie. all bots (that are not defined elsewhere in the robots.txt file). That is, all URLs for "all bots". And that's "all bots" that obey the robots.txt file. Many "bad bots" don't.

However, the crawl-delay directive is only supported by a few bots, so it may not have much of an impact at all. The crawl-delay directives is supposedly supported by "Yahoo!, Bing and Yandex" - but the lesser known bots (that are perhaps causing you problems) probably don't support this at all.

the ISP put down to crawlers

You need to find out more precisely (from the access logs) the traffic that was hitting your site during this time... user-agents, IP addresses, the URLs being requested, response codes/size...

4
  • Understood about that page being ignored. But to be sure about crawl-delay (for the bots that don't ignore it) - this is a universal command? It's not possible to specify a crawl delay on certain directories/files ? What I was told is incorrect ?
    – AdamJones
    Nov 22 '20 at 13:14
  • What do you mean by "universal command"? crawl-delay is not part of any standard. I believe it is a group-member directive that applies to the stated User-agent(s) (that define the group). "It's not possible to specify a crawl delay on certain directories/files ?" - That's correct, not possible. "What I was told is incorrect ?" - yes, entirely incorrect. Were you "told" this, or did you get a written/email response? If the later then I would be interested to see it.
    – MrWhite
    Nov 22 '20 at 14:17
  • By universal command, I was asking if crawl-delay is a document wide directive within the robots.txt file or if it is used within some kind of context, which is what I understood from what I was told by my ISP. I will update the question to clarify exactly what they said.
    – AdamJones
    Nov 22 '20 at 16:02
  • No, crawl-delay is a group-member directive (as mentioned). As opposed to a non-group-member ("universal") directive like Sitemap. The "group-member" directives work within the context of a "group". The group is defined by the User-agent directives that precede it, which define the bots that should honour the group-member directives that follow. Disallow and Allow are also group-member directives, just like Crawl-delay. Any one bot only matches at most one group.
    – MrWhite
    Nov 22 '20 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.