Does anyone know if there is a max limit to Crawl-Delay time? I read people set to 10 to keep bots down to 8,600 crawls a day, but I would like to see the limit even more.
I set mine to 30, which would mean 2,150 crawls a day (if I did the math right).
I know that bots can ignore the crawl time, but simply want Google's bots to stop hitting the site so much.
I don't believe that Google acts on the crawl-delay in robots.txt (See https://support.google.com/webmasters/thread/251817470/putting-crawl-delay-in-robots-txt-file-is-good?hl=en), also in the quiz at https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget#improve_crawl_efficiency. Also, consider that Google will crawl from multiple IPs, presumably meaning from multiple servers and possibly even multiple datacenters - this could make it difficult for Google to manage to set a crawl rate based on a time.
https://developers.google.com/search/docs/crawling-indexing/reduce-crawl-rate and https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget#improve_crawl_efficiency has a lot of relevant info (and even info in the quiz) that may be useful to you. There is a trade-off, but reducing the content available to Google may be a viable solution.
According to the documentation, returning a 503 or 429 error code when Google is hitting your site too frequently should result in Google getting the hint and slowing down - but this should be used sparingly. (I have not tried it, but I wonder if setting up something to return a 429 - the HTTP code for to many requests) only when Google goes over your acceptable threshold could tame the beast). Doing this will certainly be a lot more work then adding something to .htaccess and would be platform dependent, but I could see knocking up a trivial check at the start of each page - possibly querying and logging a memcache or database or even just checking the age of a file you write could do this without too much overhead.
