Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots.

My site generates over 1000 posts each day, so it might be normal for the bots crawling my site often.

However, is it possible to decrease the frequency for the bots crawling my site? Or just tell the bots to craw my new posts? It seems they are crawling my site more and more often and my server becomes slow when they visit my site.

share|improve this question
add comment

migrated from serverfault.com Jan 6 at 9:12

This question came from our site for professional system and network administrators.

3 Answers

up vote 2 down vote accepted

You should also specify a crawl delay in robots.txt for all the other search engines (Yandex and Baidu can be quite aggressive in their crawling). Add this:

User-agent: *
Crawl-delay: 5

The crawl delay is in seconds. Make sure not to go too high - 5-10 seconds max should lighten the server load considerably. If you have 1000 new pages per day you want search engines to be able to find them all.

However the best method for Google (and possibly Bing) is still using their webmaster tools, as Google ignores the crawl-delay directive in robots.txt.

share|improve this answer
add comment

I think this help document from Google should be solving my problem:

Change the crawl rate:

  • On the Webmaster Tools Home page, click the site you want.
  • Click the gear icon , then click Site Settings.
  • In the Crawl rate section, select the option you want.

The new crawl rate will be valid for 90 days.

share|improve this answer
    
This is specific to Google's crawler, it won't apply to other crawlers. –  reinierpost Jan 7 at 10:16
    
Yes, but Google bots are the major ones crawling my site. The other 1/3 are from Bing. And Bing also has similar setting in its webmaster tool. So after I changed the settings for Google & Bing, my site is doing ok now. FYR. –  Joe Jan 7 at 10:28
    
That is good to know! –  reinierpost Jan 7 at 13:12
add comment

The de facto standard for telling robots what to crawl is robots.txt.

Make the URLs of your posts fit a pattern such that you can generate a robots.txt that selects just the new postings. You can generate it automatically with a script, e.g. on the fly (CGI script) or in a batch job you run every hour, or in some other way.

share|improve this answer
1  
Putting pages into robots.txt will often prevent them from being indexed as well as crawled. I wouldn't list pages in robots.txt unless you don't mind having them drop from the Google index. –  Stephen Ostermiller Jan 6 at 16:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.