2

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However it's a fairly large site and I wouldn't of thought that the crawl rate would've been decreased.

What we have noticed though is a sizeable increase in page load times, in my mind this would be the cause. Can anyone else confirm if the crawl rate is directly correlated to page load time? Seems logical, longer page load time, less pages crawled.

Any decent documentation on this would be appreciated, I don't normally have any input on SEO so this is new to me.

Improve this question

1 Answer 1

Reset to default
1

I can confirm that Google crawls more when your site is faster and less when you site is slower.

In Google Webmaster Tools, the "Crawl rate" graphs will illustrate this perfectly. "Time spent downloading a page" will almost always be inversely proportional to the "Pages crawled per day" and "Kilobytes downloaded per day" graphs. Your Googlebot crawl allotment seems to be in terms of the total amount of time that Googlebot is willing to spend downloading from your site.

Here is a first hand report from a webmaster that improved his site performance which led to a lot more Googlebot crawling. The article includes this screenshot of the crawl rate graphs from Google Webmaster Tools that nicely illustrate the inverse relationship:

Inverse relationship of page speed to crawl rate in Google Webmaster Tools

Improve this answer
5
  • Cheers, can you confirm that a page won't be crawled until the entire page has loaded. When doing page tests we're seeing calls to external content that's taking ages to load. In my mind this is what's dragging the crawl rate down? Or will a page be crawled regardless of if it's finished rendering yet? I wouldn't think that's likely but I just want to be clear on this before I start pointing fingers!
    – Sam Pegler
    Commented Oct 14, 2013 at 10:56
  • The graph that is depicted here is just download of the page itself without any of external images, CSS, or JavaScript. Googlebot may crawl the external resources to generate page previews, but it doesn't need to fetch those for the main purpose of indexing the content on your website. So when improving site speed for Googlebot crawl purposes, you only to focus on speeding up the delivery of the HTML itself.
    – Stephen Ostermiller ♦
    Commented Oct 14, 2013 at 11:01
  • Do you have a link that runs through how googlebot crawls the site?
    – Sam Pegler
    Commented Oct 14, 2013 at 11:08
  • Here is a reference about what the graphs in webmaster tools represent: webmasters.stackexchange.com/questions/18160/… If you have other questions about how Google crawls, you should ask a new question.
    – Stephen Ostermiller ♦
    Commented Oct 14, 2013 at 11:16
  • Brilliant, that was just what I was looking for. Thanks for the help.
    – Sam Pegler
    Commented Oct 14, 2013 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.