3

I am working on a rapidly growing social network and generating a sitemap with millions of links is becoming unfeasible.

I was wondering if we could just include the last X created / updated user profiles in our sitemap instead? This way sitemap would not include all the records, but only those that were created in the last day or week.

As long as Google visits sitemap often enough to identify changes between two versions, this should be sufficient for it to discover all new links without us needing to generate and persist a large sitemap.

The only risk I see is if Google would somehow penalize website for removing links from the sitemap?

2

Having an XML sitemap is optional for SEO. They don't help rankings and they don't usually get pages included in them indexed. Including a new pages in your sitemap is unlikely to get Google to index them. See The Sitemap Paradox.

If you want your profile pages indexed, you need to find a way to link to them. For example you could link a profile to their friends' profiles. That way Googlebot would find them through links and would be able to judge their popularity with PageRank (link juice.)

It still could be a good idea to put your new pages into an XML sitemap. Doing so will get your new pages crawled quickly. However, without any links to them, Google is usually going to decide not to index those new pages.

Google is never going to penalize a site for removing links from your sitemap. XML sitemaps don't have enough influence over search engines for them to matter enough. Incomplete sitemaps are fine and they won't hurt your site in any way. URLs that are removed from your sitemap won't get de-indexed because they are no longer included in your sitemap. See SEO Myth: Google will only index the pages listed in your XML sitemap

Another sitemap strategy that could work for you is generating a new sitemap with only the new pages and publishing it on a new sitemap URL. If you kept all the old sitemaps, you would end up with all of your pages included, while not having to regenerate your full sitemaps. At most your would just have to generate a new sitemap index file that lists all of the sitemaps.

2

You can submit RSS feeds as sitemaps. They are designed to only include recent urls.

You may even want to push rss feed changes via pubsubhubub:

https://developers.google.com/search/docs/guides/submit-URLs

https://pubsubhubbub.appspot.com/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.