1

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than stop, until next time.

My problem

It is unclear to me from whenuntil when, it stops (24 hours?).

It is further unclear to me what it does after I have read the following statement here:

By setting a crawl delay of 10 seconds you’re only allowing these search engines to access 8,640 pages a day.

My question

crawl-delay: x; has someone here managed to understand how the math works there?

  • And yes, I know that this feature might be good, generally, only to very large websites of millions of users or something similar. – user58733 Feb 23 at 11:13
  • Related: webmasters.stackexchange.com/questions/128101/… – user58733 Mar 1 at 3:53
2

The crawl delay setting is the minimum number of seconds between page fetches allowed. It is not the "number of webpages per one second" as you postulate.

For example, with a crawl delay of 10 a crawler should:

  1. Crawl page A
  2. Wait at least 10 seconds
  3. Crawl page B

That would mean that crawler could crawl at most 6 pages per minute (sixty seconds divided by 10), 360 pages per hour, or 8,640 pages per day. The smaller the crawl delay, the more crawling is allowed.

Many web crawlers don't support the crawl-delay directive. For example Googlebot ignores it and instead backs off crawling if your site is slowing down (dynamic reactive crawl-delay).

As a webmaster, I never set the crawl-delay. Not enough crawlers support it to make a difference in server load. When it is set to a large value, it can hurt your SEO for search engines whose bots do follow it.

|improve this answer|||||
  • Hello again Stephen ; I am sorry to disturb you on this matter but please only tell me if crawl-delay: 1 should bring crawling of up to 1 webpage per second, hence 30 webpages in a minute. – user58733 Feb 28 at 14:33
  • A crawled lay of one would allow crawling of it most 60 documents in one minute because there are 60 seconds in one minute. – Stephen Ostermiller Feb 28 at 16:08
  • If your server is super fast and serves pages instantaneously, the fastest a crawler could go is 60 pages per second. It also isn't clear if the wait time is supposed to be between the start of a request to the start of the next, or whether the delay should be between the end a request and the start of the next. Different crawlers may interpret it differently because the specific meaning is not codified in any spec as far as I know. – Stephen Ostermiller Feb 28 at 19:06
  • Crawlers are always free to crawl at a slower rate. Crawlers are always free to go slower and wait longer that the specified delay if they want. – Stephen Ostermiller Feb 29 at 1:12
  • Related: webmasters.stackexchange.com/questions/128101/… – user58733 Mar 1 at 3:53
0

This answer is just an edited version of Stephen Ostermiller's answer which I have accepted and suggest to up vote (as I myself did). I publish it here because it is easier for me to read by means of web accessibility.

The crawl delay setting is the minimum number of seconds between page fetches allowed.
So, it is not the "number of webpages per one second" as you have postulated.

For example, with a crawl delay of 10, a crawler should:

  1. Crawl page A
  2. Wait at least 10 seconds
  3. Crawl page B

That would mean that crawler could crawl at most 6 pages per minute (sixty seconds divided by 10), 360 pages per hour, or 8,640 pages per day.

The smaller the crawl delay, the more crawling is allowed.

Side notes

  • Many web crawlers don't support the crawl-delay directive. For example Googlebot ignores it and instead backs off crawling if your site is slowing down (dynamic reactive crawl-delay).

  • As a webmaster, I never set the crawl-delay. Not enough crawlers support it to make a difference in server load. When it is set to a large value, it can hurt your SEO for search engines whose bots do follow it.

Update

Stephen later added in comments:

If your server is super fast and serves pages instantaneously, the fastest a crawler could go is 60 pages per second.

It also isn't clear if the wait time is supposed to be between the start of a request to the start of the next, or whether the delay should be between the end a request and the start of the next.

Different crawlers may interpret it differently because the specific meaning is not codified in any spec as far as I know.

I replied:

Stephen, I think these are very important issues --- if I understand you correctly, one cannot know what will be the outcome of crawl-delay: 1 (30 seconds, 60 seconds or something in between these numbers) as it depends on application performance. Well, I do webmastering since 2008 but I have never came across such data. I think it is very important to be added in to your answer and I will give bounty if this is the price for us having a good session about this for all newcomers in the future.

Stephen replied:

Crawlers are always free to crawl at a slower rate. Crawlers are always free to go slower and wait longer that the specified delay if they want.

I currently add here:

My problem is that the crawl-delay: 1 30 seconds versus 60 seconds issue is still not fully dealt with for others and for ourselves; I personally have read in several different documents and came out confused and I believe Stephen removed all my confusion besides in that particular issue

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy