0

I'm searching for documents related to crawl and indexing. I don't know how to decrease the response time of Google crawl And what impacts on it? enter image description here

Hope you can help me!

1 Answer 1

1

I would look at all the various elements that are downloaded for a given page. In Google Search Console, it says this about avg. response time:

Average page response time for a crawl request to retrieve the page content. Does not include retrieving page resources (scripts, images, and other linked or embedded content) or page rendering time.

There is a good thread on Google which talks about some of these issues.

I would first focus on the page itself. Is there any extra content that doesn't need to be there. Any way that you could "minimize" the initial .html file. This will probably not do a whole lot but according to Google's definition, it is focusing on the response of getting that initial page.

From there, I would look at all the other pieces of the page: videos, images, scripts, css, etc. All of those elements will, in essence, contribute to a longer response time.

My question when looking at pages is this: Is this reaaaallly needed to communicate the message of the page? Sometimes, yes, you need all the wiz-bang to get your point across. Other times, it might not be needed. It all depends.

There are also a lot of other factors that could compound response. Your location compared to where Google is. If you are in another country and Google's crawlers come out of the US, it would take longer than if your site/assets were being served from a closer server. This is where you would see some good gains with a CDN (for both page and assets). If a CDN isn't an option, you might need to look at your servers and see if things can be optimized in order to serve assets faster.

Another is the connection speed. Google doesn't explicitly say what connection speed they use for googlebot. But if they are leveraging a slower connection speed, then it would be good to test that out as well. If you are curious of your own user speeds, you can access the Core Web Vitals report for your specific site to see what the average connection speeds are of users.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.