How frequently does Googlebot fetch sitemaps? Does it depend on PageRank?

Question

How frequently does Google fetch sitemaps?

I am now working with a high traffic website that normally has 30 new posts per minute. It currently it provides a sitemap which only includes the 100 newest posts (3 minutes).

1. Is this method is enough? Do bots fetch sitemaps every 3 minutes?

2. Does the site need sitemaps to list all 5M posts? How would providing these sitemaps change traffic and PageRank?

3. Will Googlebot remove URLs that were previously listed in sitemaps but now are not?

Answer 1

1. 3mins is absolutely fine, you should check to see how often Google is reading the sitemap I bet its much longer than that.
2. You should break down your sitemaps into segments and have the main sitemap link to the others as this becomes easier to manage and Google has a better time processing smaller sitemaps.
3. Google will only remove the urls if the page is removed from the site and returns an error status. Your find that it lists more than what your sitemap has but not vice versa.

Answer 2

I would try to provide at least a week's worth of new content in the sitemaps if the goal is to get Google to discover and index new content. Googlebot is not likely to fetch sitemap files every 3 minutes.

Many sites provide a sitemap that contains ALL of the URLs that are available on the site. In addition to your sitemap of new content, I would suggest creating a set of comprehensive sitemap files as well.

Sitemaps only provide URL discovery. They don't help the pages in them rank. See The sitemap paradox. If Googlebot can find all 5M pages on the site through normal crawling, there is no ranking benefit to a sitemap. You do get additional stats about pages included in sitemaps through Google webmaster tools.

If Googlebot can find links to a page, it will continue to index that page even if the page is no longer in a sitemap.