I would like to start using the Crawl-delay on the robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over 10 requests per second, so if they run one every 0.1sec this will fulfill my requirement.
Putting it to 1sec is too slow as in has to be done on a large website.
As far as I know the Crawl-delay directive is only applicable to Bingbot. Googlebot does not use it. So this answer is going to focus on what Bing allows.
The crawl-delay directive accepts only positive, whole numbers as values.
So the smallest crawl delay you can set is 1. It does not support decimals such as 0.1.
If you want your site to be crawled at 10 requests per second, you should omit the Crawl-delay entirely and let bots decide on their own how fast they want to crawl your site.
