Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
I recently came across the phrase 'crawling budget'.
Is keeping a small number of links in the sitemap and rotating them regularly programmatically more effective than posting everything at once? Currently I have around 300k pages to be indexed, and expecting more.
No, you shouldn't try rotating a small number of URLs through your sitemap. A sitemap is most useful when it has all the URLs from your site. Even if Googlebot doesn't crawl all the URLs in your sitemap right away, it will come back and crawl the rest. It won't speed up the process to only show Googlebot a few of the links at a time. If your sitemap doesn't have all the URLs in, you won't get the benefits of the extra statistics about how many of those URLs are indexed from Google Search Console.
Googlebot has huge crawl budgets. It is very rare for a site to have so many pages that it exceeds its crawl budget. Googlebot is usually willing to crawl thousands of pages from any site, no matter how new and no matter how little reputation it has. For a more established site (a couple years old with a decent number of external links), Googlebot should be more than willing to crawl hundreds of thousands of pages.
Unfortunately, XML sitemaps don't help your pages get indexed or ranked well in Google. Googlebot is usually willing to crawl pages that it finds only in the sitemap, but Google usually chooses not to index such pages. Even in cases in which they get indexed, they won't rank well for any competitive keywords. See The Sitemap Paradox.
To get Google to index hundreds of thousands of pages you need to:
