1

We have a sitemap that is regenerated daily based on the records in our database. There are about 55 million records and each record is accessible as a separate page. However sometimes records are deleted and after 1 to 2 weeks the Google Search Console complains for a couple (but not all) of the deleted items that their URL returns a 404. This is shown as a warning under the console's Sitemap errors section with a link to the sitemap where this URL used to be, but is now removed.

I suspect the cause of this is that Google doesn't fetch the sitemap every day. It seems to cache it for a couple weeks, despite our http-response headers:

Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Expires: 0

Google then complains when it checks cached URLs that have been removed both from the site and the latest version of the sitemap. Can someone confirm that Google always caches sitemaps?

Looking at the dates that the pieces of our sitemap are processed, it seems our entire sitemap also takes about 2 weeks to process. Is it possible to tell Google to fetch and use the latest version of a sitemap page daily?

I've read this similar question, but that question is asking where Google gets its old urls from. I know exactly where it comes from (Google told me so). I understand the 404s are probably not a big deal, but if possible I'd like to prevent them.

  • 2
    I am not sure why so many people seem to freak out over 404 messages. It seems to be an epidemic to stress over what is a normal function in how the web works. Freaketh thyslf not. 404s are to be expected. It is not a problem. Nothing to fix. – closetnoc May 22 '17 at 15:21
  • Forget 1 to 2 weeks. Googlebot will sometimes crawl and old URL a decade after it last had a page up. – Stephen Ostermiller May 22 '17 at 17:02
  • 1
    @StephenOstermiller but do these crawls lead to warnings in the sitemap part of the Google Search Console? – THelper May 22 '17 at 17:03
  • 1
    @closetnoc We indeed overwrite the existing sitemap with a new one every day. There is a sitemap index page that refers to the subpages containing 45.000 urls each. But your remark about 'modified date' got me thinking. Should I see such a last modified date in the http-response-header when retrieving the sitemap index page? Because right now I don't see that. – THelper May 23 '17 at 12:20
  • 1
    Okay. I understand a bit more. BTW- 45k was my limit per sitemap. Google uses the sitemaps primarily as an auditing tool. The primary benefit of a sitemap is for very large sites such as yours. G does not process sitemaps quickly much of the time. You are right about caching in that G will download the file than refer to it periodically only to check the site for updated sitemaps periodically. I imagine it would be very common to see 404s for a large dynamic site where pages come and go. Seeing a 404 for pages that should not exist is nothing to worry about. – closetnoc May 23 '17 at 15:39
2

It isn't that Google is caching the sitemap file itself. When Google downloads the sitemap, it parses it and adds the URLs to a database. It then decides whether it needs to crawl those URLs soon. Google also queries its database to show you the information in search console. It is Google's database that has this out of date information.

There is no way to force Google to re-fetch sitemaps more often than it already does. When you have 1000+ sitemap files, there is no way that Google is going to fetch them all every day. Like most files on your website, Google is going to fetch them every couple weeks. Only very high Pagerank pages (such as your home page) tend to get fetched by Google more often. If you have a high Pagerank site you have more important pages to link to other than sitemap files in the hopes of getting them downloaded more frequently.

My suggestion would be to return "410 Gone" status for the removed pages rather than "404 Not Found". You may still see warnings about it, but you should then be able to differentiate the intentional removal from an unintentional problem.

Google also treats 410 status differently. It removes the page itself from the index immediately upon crawling as opposed to 404 where it gives it a 24 hour grace period. Google comes back to re-try crawling 410 URLs much less frequently than URLs with 404 status.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.