Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace?

I like being indexed by all these bots but with Yandex and Baidu it is just a bit overwhelming for my app which is very very database heavy.

Eventually I will cache stuff but right now i'm in beta. I do enjoy the traffic. Just need to slow these bots down a bit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.