It was early 2018, almost two years ago, when I retired a bunch of pages from my site that were no longer needed/relevant/maintained/etc. I did that by returning a 410-Gone HTTP status. To my surprise, Googlebot still keeps coming back to those pages at very regular intervals, just to get 410s again and again and again. For 2 years. I have checked and those pages are not in the index anymore, but I'm curious as to why the crawler still thinks the page "might" be available? Does anyone know of a good blog/discussion that explains how 410s really work?
1
-
1Would there be any internal or external pages still trying to link to them?– Bronwyn VJan 29, 2020 at 6:44
Add a comment
|
2 Answers
Once Googlebot finds a working URL, it will never forget it. It will always come back to crawl it periodically. It doesn't matter what status it returns. It doesn't matter if it still has any links to it or not. I have URLs that have been 301 permanent redirecting for 15 years. Googlebot still crawls them occasionally.
The logic appears to be that old URLs occasionally get resurrected. Googlebot might as well check sometimes. The number of links into the URL determines how often it will get crawled. If you manage to remove every internal and external link to the URL, it will get crawled far less frequently.
Googlebot even has a special crawl mode for old URLs with no links. Sometimes I will find Googlebot requesting hundreds of such old URLs from my site over the course of a day. When Googlebot is in this crawl mode, it appears to crawl the URLs in order of length. The shortest URLs get crawled first.
This shouldn't be anything to worry about:
- As long as the URL is returning a "410 Gone" status and you allow it to be crawled, there is no danger that Google will index the URL. Googlebot is simply checking to make sure the page is indeed still gone.
- 404 and 410 status URLS don't usually consume many server resources. They are very small responses. They don't usually require database lookups. They don't usually trigger additional resource requests for JS, CSS, and images.
- Google expects to find 404 and 410 errors on properly configured websites. Serving these error pages to Googlebot won't hurt the rest of your site in any way. It doesn't matter if there are 100 or 10 million error URLs according to Google's John Mueller
answered Jan 29, 2020 at 9:29
Regarding a good blog/discussion, there is one here
https://www.searchenginejournal.com/google-404-status/254429/
“If a 404 error goes to a page that doesn’t exist, should I make them a 410?”
John Mueller answered:
“From our point of view, in the mid term/long term, a 404 is the same as a 410 for us. So in both of these cases, we drop those URLs from our index.
We generally reduce crawling a little bit of those URLs so that we don’t spend too much time crawling things that we know don’t exist.
The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we’re talking on the order of a couple days or so.
So if you’re just removing content naturally, then that’s perfectly fine to use either one. If you’ve already removed this content long ago, then it’s already not indexed so it doesn’t matter for us if you use a 404 or 410.”
And from the article link official specs for 410 error code
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11
