The exact interpretation of crawl delay is not specified anywhere. Crawlers could reasonably interpret it in two ways:
- They have to wait between starting requests
- They have to wait between the end of one request and the start of the next
For example, here how a crawler might fetch three pages according to rule #1
0 1 2 3 seconds
| | |*page3*
And here is how the same fetch might look under rule #2
0 1 2 3 4 5 seconds
| | **page2********
| | | | | **page3*
Both of the crawlers are obeying some interpretation of the directive, but the first may have overlapping requests and the second may take a lot longer to crawl.
Wikipedia notes this in the crawl-delay section of their robots.txt article. It says that Bingbot obeys interpretation #1 and Yandexbot obeys interpretation #2.
With a crawl delay of
1, the fastest a crawler observing crawl delay could hit your site would be 60 times per minute. Any crawler (eg Bingbot) observing rule #1 might hit this speed.
If a bot is observing rule #2, it won't be able to crawl as quickly. The number of requests in a time period will depend on how quickly your site can deliver pages (including network transmission time) to that crawler. For:
t in seconds,
- Crawl delay
cd in seconds
- Average page speed
aps in seconds
the formula for the average number of pages that can be fetched in a time period is:
t / ( cd + aps)
So if you have a crawl delay of
1 and it takes on average a second to serve a page on your site, here is how the major search engines will behave:
- Googlebot ignores the crawl delay and fetches as many pages as it wants as long it it doesn't look like your site is slowing down because of it.
- Bingbot will fetch at most sixty pages in a minute.
- Yandexbot will fetch at most thirty pages in a minute.
It is also worth noting that the crawl-delay puts a maximum cap on the number of pages crawled. Crawlers may choose to crawl fewer page if they desire.