I would like to start using the Crawl-delay on the
robots.txt and saw that the value is in seconds. Is there a way to set it up below 1 second. Maybe 0.1?
My goal is to allow crawlers not to run over 10 requests per second, so if they run one every 0.1sec this will fulfill my requirement.
Putting it to 1sec is too slow as in has to be done on a large website.