2

I have a problem of understanding what crawl-delay: 1 means if it isn't ignored and actually being interpreted as is by a given web crawler.

I understood from this answer by Stephen Ostermiller that:

With a crawl delay of 10 a crawler should:

  • Crawl a page A

  • Wait at least 10 seconds

  • Crawl page B

That would mean that a crawler could crawl at most 6 pages per minute (60 seconds divided by 10); 360 pages per hour, or 8,640 pages per day.

The smaller the crawl delay → the more crawling is allowed.

Well, from that I understand that, a crawling of up to one webpage per second and then a delay of a second will bring the crawling of 30 webpages per minute.

and yet, when corresponding in comments, I understood from Stephen it will be close to or near 60 crawlings ("fetches").

Well, 60 crawlings or at most 30 crawlings?

In other words,

What does crawl-delay: 1 mean if it is interpreted as is?

  • Why are you assuming 1 second crawling? That is going to depend on the speed of your site and the speed of the network. Are you trying to determine the most crawling that could happen (very fast page loads), or how limiting it could be for bots that want to go faster? As I noted in the other question, that depends on how crawlers interpret when they have to wait from in any case. – Stephen Ostermiller Feb 29 at 17:45
  • Why are you assuming 1 second crawling? by principle. Are you trying to determine the most crawling that could happen (very fast page loads), or how limiting it could be for bots that want to go faster? I try to understand what will happen when a crawler "respects" crawl-delay: 1 as is; nothing more (although it is clear to me that there could be some "disrespects" or biases as you have explained). – user58733 Mar 1 at 3:52
3
+50

The exact interpretation of crawl delay is not specified anywhere. Crawlers could reasonably interpret it in two ways:

  1. They have to wait between starting requests
  2. They have to wait between the end of one request and the start of the next

For example, here how a crawler might fetch three pages according to rule #1

0         1         2         3  seconds
|*page1**          
|         |*page2*********
|         |         |*page3*

And here is how the same fetch might look under rule #2

0         1         2         3         4         5  seconds
|*page1**          
|         |        **page2********
|         |         |         |         |   **page3*

Both of the crawlers are obeying some interpretation of the directive, but the first may have overlapping requests and the second may take a lot longer to crawl.

Wikipedia notes this in the crawl-delay section of their robots.txt article. It says that Bingbot obeys interpretation #1 and Yandexbot obeys interpretation #2.

With a crawl delay of 1, the fastest a crawler observing crawl delay could hit your site would be 60 times per second. Any crawler (eg Bingbot) observing rule #1 might hit this speed.

If a bot is observing rule #2, it won't be able to crawl as quickly. The number of requests in a time period will depend on how quickly your site can deliver pages (including network transmission time) to that crawler. For:

  • Time t in seconds,
  • Crawl delay cd in seconds
  • Average page speed aps in seconds

the formula for the average number of pages that can be fetched in a time period is:

t / ( cd + aps)

So if you have a crawl delay of 1 and it takes on average a second to serve a page on your site, here is how the major search engines will behave:

  • Googlebot ignores the crawl delay and fetches as many pages as it wants as long it it doesn't look like your site is slowing down because of it.
  • Bingbot will fetch at most sixty pages in a minute.
  • Yandexbot will fetch at most thirty pages in a minute.

It is also worth noting that the crawl-delay puts a maxium cap on the number of pages crawled. Crawlers may choose to crawl fewer page if they desire.

| improve this answer | |
  • Is there any way to simplify the answer? I think I misunderstood what is going on; I understand crawl-delay command can be ignored or interpreted but that is not what I am asking about; I only mean to ask about what happens when crawl-delay: 1 does get "respected" fully (i.e not ignored or interpreted but is executed as is). – user58733 Mar 1 at 3:56
  • The simple answer is "it depends." "As is" is open to interpretation. – Stephen Ostermiller Mar 1 at 12:08
  • Stephen - I am sorry, I am still confused about the 30-60 range of seconds issue if we assume that no special "interpretation" is done and a crawler works with a crawl-delay: 1 directive "as is". – user58733 Mar 1 at 12:09
  • The answer is still "it depends." Some crawlers may crawl 60 pages per second with a crawl delay of 1. Some crawlers will ignore the directive. Other crawlers will limit themselves further basing the delay on the time between page fetches. – Stephen Ostermiller Mar 1 at 16:59
  • I am sorry, your last comment made me more confused than I was before regarding this specific matter of crawl-delay: 1. – user58733 Mar 1 at 17:04
0

Crawl-delay is used to stop bot from crawling web pages or website very frequently. However, it should be used when your website have number of pages and is a big size website. Crawl-delay can be used in social bookmarking sites, sites like twitter, facebook etc. which are very frequently updated.

User-agent: Googlebot
Crawl-delay: 120

Its means is that Googlebot should crawl webpages in delay of 120 second not frequently.

| improve this answer | |
  • Googlebot doesn't support the crawl-delay directive. See “Crawl-delay” rule ignored by googlebot. It is an optional extenson to robots.txt. Googlebot is one of the bots that doesn't use it. – Stephen Ostermiller Mar 1 at 11:39
  • It’s important to say that the Google bot does not take into consideration the crawl-delay setting. That is why you should not worry that such a directive can influence your Google standings and you can safely use it, in case there are other aggressive bots you want to stop. It is highly unlikely to experience issues due to Google bot crawling, but if you want to lower its crawl-rate you can do this only from the Google Search Console – DontWorry Mar 3 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy