1

Google sitemaps can contain max 50k URLs, and you are allowed to have max 500 sitemap index files, leading me to believe you can have at most 25 million URLs in sitemap files.

What if you have more URLs than that, what are your options? Say I have a billion URLs (like an ecommerce site the fraction of the size of Amazon)? That is 40 times the understood limit of 25 million, so what can/should you do?

2 Answers 2

1

You don't actually need a sitemap.

Do I need a sitemap? If your site's pages are properly linked, Google can usually discover most of your site. Proper linking means that all pages that you deem important can be reached through some form of navigation, be that your site's menu or links that you placed on pages. Even so, a sitemap can improve the crawling of larger or more complex sites, or more specialized files.

You need to make sure that all pages are internally linked - point 2 applies to you.

You might not need a sitemap if:

  • Your site is "small". By small, we mean about 500 pages or fewer on your site. (Only pages that you think need to be in search results count toward this total.)
  • Your site is comprehensively linked internally. This means that Google can find all the important pages on your site by following links starting from the home page.
  • You don't have many media files (video, image) or news pages that you want to show in search results. Sitemaps can help Google find and understand video and image files, or news articles, on your site. If you don't need these results to appear in Search you might not need a sitemap.

From developer.google.com Learn about sitemaps

So, I would suggest that you have the xml sitemaps for the most important pages. And ensure that all other pages are linked in some fashion. One way to do this is to have html pages, that just have links to other parts of your website. Normally called sitemaps (and meant for users to see the structure of your site).

0

The first option I can think of is to make sure that all products are part of categories and subcategories and make sure that they are part of the XML sitemap itself, not the products. In theory, this should be a drastically lower number and they will take crawler bots to all products.

Another option is to use HTML sitemaps, which don't have such restrictions and will help crawling bots reach all products.

Another possibility is to use a general category that unites absolutely all products in the project so that the crawler bot reaches all possible products.

2
  • How you do HTML sitemaps is that microdata or something? I will have to search that...
    – Lance
    Mar 17, 2023 at 17:49
  • This is just a simple page (dynamic) on a website that lists all of the pages on that site in a hierarchical or structured manner. Its purpose is to help users and search engines navigate a website more easily, by providing an overview of the site's organization and content. Simple as that. By the way, you can combine it with SiteNavigationElement schema markup to indicate that a certain section of your HTML sitemap is intended to help visitors navigate your website. I haven't implemented It for that purpose, so couldn't share any feedback.
    – Maniac
    Mar 18, 2023 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.