crawl-delay: x --------- has someone here managed to understand how the math works there?

Question

From prior research of reading different articles, I understand that a robots.txt crawl delay such as crawl-delay: 1 allows a given search engine to crawl x number of webpages per one second, and than stop, until next time.

My problem

It is unclear to me from when → until when, it stops (24 hours?).

It is further unclear to me what it does after I have read the following statement here:

By setting a crawl delay of 10 seconds you’re only allowing these search engines to access 8,640 pages a day.

My question

crawl-delay: x; has someone here managed to understand how the math works there?

Answer 1

The crawl delay setting is the minimum number of seconds between page fetches allowed. It is not the "number of webpages per one second" as you postulate.

For example, with a crawl delay of 10 a crawler should:

1. Crawl page A
2. Wait at least 10 seconds
3. Crawl page B

That would mean that crawler could crawl at most 6 pages per minute (sixty seconds divided by 10), 360 pages per hour, or 8,640 pages per day. The smaller the crawl delay, the more crawling is allowed.

Many web crawlers don't support the crawl-delay directive. For example Googlebot ignores it and instead backs off crawling if your site is slowing down (dynamic reactive crawl-delay).

As a webmaster, I never set the crawl-delay. Not enough crawlers support it to make a difference in server load. When it is set to a large value, it can hurt your SEO for search engines whose bots do follow it.

Answer 2

_{This answer is just an edited version of Stephen Ostermiller's answer which I have accepted and suggest to up vote (as I myself did). I publish it here because it is easier for me to read by means of web accessibility.}

The crawl delay setting is the minimum number of seconds between page fetches allowed.
So, it is not the "number of webpages per one second" as you have postulated.

For example, with a crawl delay of 10, a crawler should:

1. Crawl page A
2. Wait at least 10 seconds
3. Crawl page B

That would mean that crawler could crawl at most 6 pages per minute (sixty seconds divided by 10), 360 pages per hour, or 8,640 pages per day.

The smaller the crawl delay, the more crawling is allowed.

Side notes

• Many web crawlers don't support the crawl-delay directive. For example Googlebot ignores it and instead backs off crawling if your site is slowing down (dynamic reactive crawl-delay).

• As a webmaster, I never set the crawl-delay. Not enough crawlers support it to make a difference in server load. When it is set to a large value, it can hurt your SEO for search engines whose bots do follow it.

Update

Stephen later added in comments:

If your server is super fast and serves pages instantaneously, the fastest a crawler could go is 60 pages per second.

It also isn't clear if the wait time is supposed to be between the start of a request to the start of the next, or whether the delay should be between the end a request and the start of the next.

Different crawlers may interpret it differently because the specific meaning is not codified in any spec as far as I know.

I replied:

Stephen, I think these are very important issues --- if I understand you correctly, one cannot know what will be the outcome of crawl-delay: 1 (30 seconds, 60 seconds or something in between these numbers) as it depends on application performance. Well, I do webmastering since 2008 but I have never came across such data. I think it is very important to be added in to your answer and I will give bounty if this is the price for us having a good session about this for all newcomers in the future.

Stephen replied:

Crawlers are always free to crawl at a slower rate. Crawlers are always free to go slower and wait longer that the specified delay if they want.

I currently add here:

My problem is that the crawl-delay: 1 30 seconds versus 60 seconds issue is still not fully dealt with for others and for ourselves; I personally have read in several different documents and came out confused and I believe Stephen removed all my confusion besides in that particular issue