I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?

My goal is to allow crawlers not to run over 10 requests per second, so if they run one every 0.1sec this will fulfill my requirement.

Putting it to 1sec is too slow as in has to be done on a large website.


1 Answer 1


As far as I know the Crawl-delay directive is only applicable to Bingbot. Googlebot does not use it. So this answer is going to focus on what Bing allows.

Bing says:

The crawl-delay directive accepts only positive, whole numbers as values.

So the smallest crawl delay you can set is 1. It does not support decimals such as 0.1.

If you want your site to be crawled at 10 requests per second, you should omit the Crawl-delay entirely and let bots decide on their own how fast they want to crawl your site.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.