Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
Google sitemaps can contain max 50k URLs, and you are allowed to have max 500 sitemap index files, leading me to believe you can have at most 25 million URLs in sitemap files.
What if you have more URLs than that, what are your options? Say I have a billion URLs (like an ecommerce site the fraction of the size of Amazon)? That is 40 times the understood limit of 25 million, so what can/should you do?
You don't actually need a sitemap.
Do I need a sitemap? If your site's pages are properly linked, Google can usually discover most of your site. Proper linking means that all pages that you deem important can be reached through some form of navigation, be that your site's menu or links that you placed on pages. Even so, a sitemap can improve the crawling of larger or more complex sites, or more specialized files.
You need to make sure that all pages are internally linked - point 2 applies to you.
You might not need a sitemap if:
- Your site is "small". By small, we mean about 500 pages or fewer on your site. (Only pages that you think need to be in search results count toward this total.)
- Your site is comprehensively linked internally. This means that Google can find all the important pages on your site by following links starting from the home page.
- You don't have many media files (video, image) or news pages that you want to show in search results. Sitemaps can help Google find and understand video and image files, or news articles, on your site. If you don't need these results to appear in Search you might not need a sitemap.
From developer.google.com Learn about sitemaps
So, I would suggest that you have the xml sitemaps for the most important pages. And ensure that all other pages are linked in some fashion. One way to do this is to have html pages, that just have links to other parts of your website. Normally called sitemaps (and meant for users to see the structure of your site).
The first option I can think of is to make sure that all products are part of categories and subcategories and make sure that they are part of the XML sitemap itself, not the products. In theory, this should be a drastically lower number and they will take crawler bots to all products.
Another option is to use HTML sitemaps, which don't have such restrictions and will help crawling bots reach all products.
Another possibility is to use a general category that unites absolutely all products in the project so that the crawler bot reaches all possible products.
