My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots.

My site generates over 1000 posts each day, so it might be normal for the bots crawling my site often.

However, is it possible to decrease the frequency for the bots crawling my site? Or just tell the bots to craw my new posts? It seems they are crawling my site more and more often and my server becomes slow when they visit my site.

3 Answers 3


I think this help document from Google should be solving my problem:

Change the crawl rate:

  • On the Webmaster Tools Home page, click the site you want.
  • Click the gear icon , then click Site Settings.
  • In the Crawl rate section, select the option you want.

The new crawl rate will be valid for 90 days.

  • This is specific to Google's crawler, it won't apply to other crawlers. Commented Jan 7, 2014 at 10:16
  • Yes, but Google bots are the major ones crawling my site. The other 1/3 are from Bing. And Bing also has similar setting in its webmaster tool. So after I changed the settings for Google & Bing, my site is doing ok now. FYR.
    – Joe Huang
    Commented Jan 7, 2014 at 10:28
  • That is good to know! Commented Jan 7, 2014 at 13:12

You should also specify a crawl delay in robots.txt for all the other search engines (Yandex and Baidu can be quite aggressive in their crawling). Add this:

User-agent: *
Crawl-delay: 5

The crawl delay is in seconds. Make sure not to go too high - 5-10 seconds max should lighten the server load considerably. If you have 1000 new pages per day you want search engines to be able to find them all.

However the best method for Google (and possibly Bing) is still using their webmaster tools, as Google ignores the crawl-delay directive in robots.txt.


The de facto standard for telling robots what to crawl is robots.txt.

Make the URLs of your posts fit a pattern such that you can generate a robots.txt that selects just the new postings. You can generate it automatically with a script, e.g. on the fly (CGI script) or in a batch job you run every hour, or in some other way.

  • 1
    Putting pages into robots.txt will often prevent them from being indexed as well as crawled. I wouldn't list pages in robots.txt unless you don't mind having them drop from the Google index. Commented Jan 6, 2014 at 16:33
  • You can allow Google in your robots.txt. Commented Oct 25, 2017 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.