New answers tagged

0 votes

Can I set Cawl-delay in robots.txt to less than one second (like 0.1)?

As far as I know the Crawl-belay directive is only applicable to Bingbot. Googlebot does not use it. So this answer is going to focus on what Bing allows. Bing says: The crawl-delay directive ...

Top 50 recent answers are included