New answers tagged crawl-rate
0
votes
Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?
As far as I know the Crawl-belay directive is only applicable to Bingbot. Googlebot does not use it. So this answer is going to focus on what Bing allows.
Bing says:
The crawl-delay directive ...
- 96.4k
Top 50 recent answers are included