What does crawl-delay: 1 mean if it is interpreted as is?

Question

I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler.

I understood from this answer by Stephen Ostermiller that:

With a crawl delay of 10 a crawler should:

• Crawl a page A

• Wait at least 10 seconds

• Crawl page B

That would mean that a crawler could crawl at most 6 pages per minute (60 seconds divided by 10); 360 pages per hour, or 8,640 pages per day.

The smaller the crawl delay → the more crawling is allowed.

Well, from that I understand that, a crawling of up to one webpage per second and then a delay of a second will bring the crawling of 30 webpages per minute.

and yet, when corresponding in comments, I understood from Stephen it will be close to or near 60 crawlings ("fetches").

Well, 60 crawlings or at most 30 crawlings?

In other words,

What does crawl-delay: 1 mean if it is interpreted as is?

Answer 1

The exact interpretation of crawl delay is not specified anywhere. Crawlers could reasonably interpret it in two ways:

1. They have to wait between starting requests
2. They have to wait between the end of one request and the start of the next

For example, here how a crawler might fetch three pages according to rule #1

0         1         2         3  seconds
|*page1**          
|         |*page2*********
|         |         |*page3*

And here is how the same fetch might look under rule #2

0         1         2         3         4         5  seconds
|*page1**          
|         |        **page2********
|         |         |         |         |   **page3*

Both of the crawlers are obeying some interpretation of the directive, but the first may have overlapping requests and the second may take a lot longer to crawl.

Wikipedia notes this in the crawl-delay section of their robots.txt article. It says that Bingbot obeys interpretation #1 and Yandexbot obeys interpretation #2.

With a crawl delay of 1, the fastest a crawler observing crawl delay could hit your site would be 60 times per minute. Any crawler (eg Bingbot) observing rule #1 might hit this speed.

If a bot is observing rule #2, it won't be able to crawl as quickly. The number of requests in a time period will depend on how quickly your site can deliver pages (including network transmission time) to that crawler. For:

• Time t in seconds,
• Crawl delay cd in seconds
• Average page speed aps in seconds

the formula for the average number of pages that can be fetched in a time period is:

t / ( cd + aps)

So if you have a crawl delay of 1 and it takes on average a second to serve a page on your site, here is how the major search engines will behave:

• Googlebot ignores the crawl delay and fetches as many pages as it wants as long it it doesn't look like your site is slowing down because of it.
• Bingbot will fetch at most sixty pages in a minute.
• Yandexbot will fetch at most thirty pages in a minute.

It is also worth noting that the crawl-delay puts a maximum cap on the number of pages crawled. Crawlers may choose to crawl fewer page if they desire.

Answer 2

Crawl-delay is used to stop bot from crawling web pages or website very frequently. However, it should be used when your website have number of pages and is a big size website. Crawl-delay can be used in social bookmarking sites, sites like twitter, facebook etc. which are very frequently updated.

User-agent: Googlebot
Crawl-delay: 120

Its means is that Googlebot should crawl webpages in delay of 120 second not frequently.