This answer is just an edited version of Stephen Ostermiller's
answer which I have accepted and suggest to up vote (as I myself did).
I publish it here because it is easier for me to read by means of web
The crawl delay setting is the minimum number of seconds between page fetches allowed.
So, it is not the "number of webpages per one second" as you have postulated.
For example, with a crawl delay of
10, a crawler should:
- Crawl page A
- Wait at least 10 seconds
- Crawl page B
That would mean that crawler could crawl at most 6 pages per minute (sixty seconds divided by 10), 360 pages per hour, or 8,640 pages per day.
The smaller the crawl delay, the more crawling is allowed.
Many web crawlers don't support the
crawl-delay directive. For example Googlebot ignores it and instead backs off crawling if your site is slowing down (dynamic reactive crawl-delay).
As a webmaster, I never set the
crawl-delay. Not enough crawlers support it to make a difference in server load. When it is set to a large value, it can hurt your SEO for search engines whose bots do follow it.
Stephen later added in comments:
If your server is super fast and serves pages instantaneously, the
fastest a crawler could go is 60 pages per second.
It also isn't clear if the wait time is supposed to be between the
start of a request to the start of the next, or whether the delay
should be between the end a request and the start of the next.
Different crawlers may interpret it differently because the specific
meaning is not codified in any spec as far as I know.
Stephen, I think these are very important issues --- if I understand
you correctly, one cannot know what will be the outcome of
crawl-delay: 1 (30 seconds, 60 seconds or something in between these
numbers) as it depends on application performance. Well, I do
webmastering since 2008 but I have never came across such data. I
think it is very important to be added in to your answer and I will
give bounty if this is the price for us having a good session about
this for all newcomers in the future.
Crawlers are always free to crawl at a slower rate. Crawlers are
always free to go slower and wait longer that the specified delay if
I currently add here:
My problem is that the
crawl-delay: 1 30 seconds versus 60 seconds issue is still not fully dealt with for others and for ourselves; I personally have read in several different documents and came out confused and I believe Stephen removed all my confusion besides in that particular issue