Google Crawl Stats: What's included in "time spent downloading a page" metric?

Question

Background

I'm the owner of a quite large site (millions of pages). I'm trying to optimize the site for "time spent downloading a page"-metric in google search console. Currently my stats in Search Console reports something like this:

Obviously this is quite slow. As the task for me of reducing this is quite big, I wanna make sure I spend my time optimizing the right things.

My setup

My core site (serving the HTML) is hosted in Germany, as my primary user group is European. The primary pages responding with 200 OK are all in sitemaps, and those are page cached responding in less than 10ms. I have a lot of images on my site too (millions). Those are delivered via a cloudflare CDN, so they are cached too + geographically distributed.

Questions

In my quest for lowering the server response time of my site I have the following questions?

• Is there any source/knowledge out there of what is actually included in this metric in Search Console?
• Is it only pages responding with 200 OK status code or would it also include response times 301s, 410s 404s and 5XXs?
• Is it only including response times of HTML pages or would it also include response times of e.g. images and what about even async JSON requests that google some times executes when visiting my pages with JS enabled?
• Is it including only requests to TLD or is it also including any request to subdomains too?
• What's the consequence of the physical distance to googles primary crawling servers? As I understand from googling around and also looking at the google IP's crawling my site, google spends around 95% of it's crawling efforts accessing my site from the US. Would those extra ~300ms of network from oversea be reflected in the numbers too or are they subtracted? Or in other words, would it be worth for me serving the requests from a US based hosting?

Hope someone has some insight knowledge on this.

Answer 1

Is there any source/knowledge out there of what is actually counted in this metric?

Back in the day, John Mueller wrote on Google+ that it was a simple HTTP request time. Basically, how long it takes GoogleBot to complete the HTTP request.

Is it only pages responding with 200 OK status code or would it also include response times 301s, 410s 404s and 5XXs?

Googlebot crawls every URL in your site not only 200 OK as you can see errors report in Search Console. There is not an official answer to that, so the time could apply to all status codes (as they crawl all of them) or only 200 OK (which are the URLs that can be indexed).

Is it only including response times of HTML pages or would it also include response times of e.g. images and what about even async JSON requests that google some times executes when visiting my pages with JS enabled?

Same answer as the first question: just a simple HTTP request, nothing related to rendering the page.

Is it including only requests to TLD or is it also including any request to subdomains too?

It will depend on the type of domain property you set up on Search Console. URL-prefix property will exclude all subdomains, domain property will crawl all subdomains.

What's the consequence of the physical distance to googles primary crawling servers? As I understand from googling around and also looking at the google IP's crawling my site, google spends around 95% of it's crawling efforts accessing my site from the US. Would those extra ~300ms of network from oversea be reflected in the numbers too or are they subtracted? Or in other words, would it be worth for me serving the requests from a US based hosting?

You are right, Googlebot mainly works from the US but you can check from where your site is being crawled using a reverse DNS lookup based on the traces of Googlebot in your log files.

However, you are giving too much credit to a graph that wasn't migrated to the new version of Google Search Console so, probably, Google knows it doesn't give qualify nor actionable information. For that, you should be using Google Lighthouse in order to improve performance and quality that will help directly to the load time of your site.