1

I need to put in place a system that will generate sitemap files (https://www.sitemaps.org) for a huge website with content changing dynamically. These are example figures, but I'm thinking of something at and above these orders of magnitude:

  • 10,000,000 pages.
  • 1,000s of pages added daily.
  • 1,000s of pages modified daily.

My ongoing sitemap goals after search engines index everything initially are:

  • New pages to be discovered/indexed ASAP.
  • Modified pages to be discovered/indexed ASAP.
  • Non-modified pages to be re-crawled rarely.
  • Try to help search engines save bandwidth, e.g. by putting all the new/modified pages in one sitemap file.

I'll add that I suspect sites such as Wikipedia or StackOverflow are in a similar position.

Are there any good algorithms for such a use case?

3

Sitemaps aren't going to help much. Google rarely indexes pages just because they are in a sitemap. Stack Overflow had problems getting all questions indexed when it was only using XML sitemaps. See The Sitemap Paradox. Telling which pages are modified with sitemaps doesn't work either: Google Says They Mostly Ignore the lastmod Tag In XML Sitemaps

I don't know much about your 10 million page site, but chances are that you are only going to get a small fraction of your pages indexed at all. Google indexes only the number of pages that it thinks a site with your reputation deserves. I'd estimate that you need a link from some other unique domain for each 1,000 pages you want to get indexed. To get all your pages indexed, you'd need 10,000 different sites that link to yours. There are plenty of sites that big that have done that, but it usually takes years to achieve.

If you want to get as many pages ranked as possible, the best strategy is usually to link deep pages to other deep pages. That is a large reason why this site has a list of "related questions" on each question page.

Sitemaps can help you get new pages crawled relatively quickly. Adding new pages to a sitemap is generally a good way to get Googlebot to come take a look at them. Google doesn't index most things Googlebot crawls. If you want to get your new pages indexed, they need to be linked from one of your very popular pages. That is a big part of the reason why this site has a list of new questions on the home page.

Modified pages don't usually require urgent crawling and re-indexing. Unless the content of the page changes dramatically, it will usually be fine ranking for whatever keywords were on it before. Google usually finds modifications even on less popular pages within a few weeks. The only way I know that you can request a particular page be re-crawled soon is to use the "fetch as Google" feature in Google search console. You are limited to a small number of those requests per day, so that strategy doesn't work for thousands (or even tens) of pages modified every day.

There is no way to prevent Googlebot from re-crawling unmodified pages. Googlebot tends to re-crawl pages based on how popular they are. Your home page may get crawled several times a day while deep pages may be every few weeks. There is nothing I know of that you can do to modify this behavior of Googlebot. Googlebot uses a significant amount of bandwidth. If you want search engine traffic, you need to plan for that and budget for that. Luckily bandwidth and hosting are fairly cheap.

1

I have a similar situation and I just created two files.

  1. A sitemap file that loads 50,000 URLs at a time based on an offset in the querystring.
  2. A sitemap index file that counts up the total records and creates the URLs that point to the above file with offsets of 0, 50k, 100k, 150k... until it reaches the total number of potential URLs.

I then submit the second file to Google as a sitemap index.

For my own sanity and organization, I do this a few different times on the site. We have three main sections, so I create a sitemap index for each section. I also create a dedicated, not dynamic, sitemap for URLs that contain the main, gateway, pages of the site.

Sitemaps and indexes are limited to 50,000 URLs, so you will have to adjust this method to accommodate larger sites than mine. But the logic, I think, scales.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.