How frequently does Google fetch sitemaps?

I am now working with a high traffic website that normally has 30 new posts per minute. It currently it provides a sitemap which only includes the 100 newest posts (3 minutes).

  1. Is this method is enough? Do bots fetch sitemaps every 3 minutes?

  2. Does the site need sitemaps to list all 5M posts? How would providing these sitemaps change traffic and PageRank?

  3. Will Googlebot remove URLs that were previously listed in sitemaps but now are not?

  • I would suggest also using the <lastmod> tag for such a case (if you're not already)
    – Noam
    Aug 15, 2013 at 12:24

2 Answers 2

  1. 3mins is absolutely fine, you should check to see how often Google is reading the sitemap I bet its much longer than that.
  2. You should break down your sitemaps into segments and have the main sitemap link to the others as this becomes easier to manage and Google has a better time processing smaller sitemaps.
  3. Google will only remove the urls if the page is removed from the site and returns an error status. Your find that it lists more than what your sitemap has but not vice versa.

I would try to provide at least a week's worth of new content in the sitemaps if the goal is to get Google to discover and index new content. Googlebot is not likely to fetch sitemap files every 3 minutes.

Many sites provide a sitemap that contains ALL of the URLs that are available on the site. In addition to your sitemap of new content, I would suggest creating a set of comprehensive sitemap files as well.

Sitemaps only provide URL discovery. They don't help the pages in them rank. See The sitemap paradox. If Googlebot can find all 5M pages on the site through normal crawling, there is no ranking benefit to a sitemap. You do get additional stats about pages included in sitemaps through Google webmaster tools.

If Googlebot can find links to a page, it will continue to index that page even if the page is no longer in a sitemap.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.