How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

Question

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page descriptions in search results might not get updated fast enough.

This is clearly a balancing act; while I understand that one can determine that the site is not being crawled fast enough by looking at the cached page version of a search result, I don't understand how to measure that a site is being too heavily crawled and slowed down.

I have a few different types of CMS and hosting, and I was wondering if there was a generalized way to figure this out on all of them...maybe using server logs or web page performance test of some sort.

Answer 1

Generally, if you have a popular website, you need to factor in to the server's performance capabilities, a) your customers/viewers that will be on simultaneously and b) being hit simultaneously by Yahoo, Bing, and Google updates, and possibly Yandex and Baidu, depending on your viewing target area.

Slowing down Bing and Yahoo via the robots.txt entry wasn't a good idea as it started affecting the number of indexed pages. They tend to hit your website harder for traffic that Google does, and Baidu and Yandex are worse at moderating their traffic when they pay your website a visit.

So I eliminated the entry in the robots.txt file (Google doesn't use it anyway) and use the Bing and Google webmaster tools settings to do some traffic shaping. What I finally settled on was to allow more than usual traffic on off-peak hours and ask for slightly less traffic during normal customer high traffic hours to keep the site responsive for them.

My information on traffic comes from my hosting provider's implementation of cPanel. They provide a traffic graph for daily, weekly and yearly bandwidth which I used to tune the settings. You can get a program like Weblog Expert to extract traffic statistics out of your access logs as well.