Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However it's a fairly large site and I wouldn't of thought that the crawl rate would've been decreased.

What we have noticed though is a sizeable increase in page load times, in my mind this would be the cause. Can anyone else confirm if the crawl rate is directly correlated to page load time? Seems logical, longer page load time, less pages crawled.

Any decent documentation on this would be appreciated, I don't normally have any input on SEO so this is new to me.

share|improve this question

1 Answer 1

I can confirm that Google crawls more when your site is faster and less when you site is slower.

In Google Webmaster Tools, the "Crawl rate" graphs will illustrate this perfectly. "Time spent downloading a page" will almost always be inversely proportional to the "Pages crawled per day" and "Kilobytes downloaded per day" graphs. Your Googlebot crawl allotment seems to be in terms of the total amount of time that Googlebot is willing to spend downloading from your site.

Here is a first hand report from a webmaster that improved his site performance which led to a lot more Googlebot crawling. The article includes this screenshot of the crawl rate graphs from Google Webmaster Tools that nicely illustrate the inverse relationship:

Inverse relationship of page speed to crawl rate in Google Webmaster Tools

share|improve this answer
    
Cheers, can you confirm that a page won't be crawled until the entire page has loaded. When doing page tests we're seeing calls to external content that's taking ages to load. In my mind this is what's dragging the crawl rate down? Or will a page be crawled regardless of if it's finished rendering yet? I wouldn't think that's likely but I just want to be clear on this before I start pointing fingers! –  Sam Pegler Oct 14 '13 at 10:56
    
The graph that is depicted here is just download of the page itself without any of external images, CSS, or JavaScript. Googlebot may crawl the external resources to generate page previews, but it doesn't need to fetch those for the main purpose of indexing the content on your website. So when improving site speed for Googlebot crawl purposes, you only to focus on speeding up the delivery of the HTML itself. –  Stephen Ostermiller Oct 14 '13 at 11:01
    
Do you have a link that runs through how googlebot crawls the site? –  Sam Pegler Oct 14 '13 at 11:08
    
Here is a reference about what the graphs in webmaster tools represent: webmasters.stackexchange.com/questions/18160/… If you have other questions about how Google crawls, you should ask a new question. –  Stephen Ostermiller Oct 14 '13 at 11:16
    
Brilliant, that was just what I was looking for. Thanks for the help. –  Sam Pegler Oct 14 '13 at 11:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.