1

I recently came across the phrase 'crawling budget'.

Is keeping a small number of links in the sitemap and rotating them regularly programmatically more effective than posting everything at once? Currently I have around 300k pages to be indexed, and expecting more.

1

No, you shouldn't try rotating a small number of URLs through your sitemap. A sitemap is most useful when it has all the URLs from your site. Even if Googlebot doesn't crawl all the URLs in your sitemap right away, it will come back and crawl the rest. It won't speed up the process to only show Googlebot a few of the links at a time. If your sitemap doesn't have all the URLs in, you won't get the benefits of the extra statistics about how many of those URLs are indexed from Google Search Console.

Googlebot has huge crawl budgets. It is very rare for a site to have so many pages that it exceeds its crawl budget. Googlebot is usually willing to crawl thousands of pages from any site, no matter how new and no matter how little reputation it has. For a more established site (a couple years old with a decent number of external links), Googlebot should be more than willing to crawl hundreds of thousands of pages.

Unfortunately, XML sitemaps don't help your pages get indexed or ranked well in Google. Googlebot is usually willing to crawl pages that it finds only in the sitemap, but Google usually chooses not to index such pages. Even in cases in which they get indexed, they won't rank well for any competitive keywords. See The Sitemap Paradox.

To get Google to index hundreds of thousands of pages you need to:

  • Link to each page from several other pages on your site. Each of those hundreds of thousands of pages should have links to several others.
  • Give it time (usually at least a couple years).
  • Work on getting external links into your site, especially deep into some of the hundreds of thousands of URLs.
1
  • You helped me a very lot. Thank you!
    – bakunet
    Apr 24 at 9:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.