Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace?
I like being indexed by all these bots but with Yandex and Baidu it is just a bit overwhelming for my app which is very very database heavy.
Eventually I will cache stuff but right now i'm in beta. I do enjoy the traffic. Just need to slow these bots down a bit.