I have a dynamic site which has many (well, less than 50) users. Each user is allowed to create as many pages as they want. I know that there is a limit to how many pages you can be listed in sitemap.xml, and for now I am under that limit, but I would like to know what the best way to enable search engines to know (all?) about my user's pages while still being under that limit even if I have more than the limit.
Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap. Sitemaps can be compressed using gzip, reducing bandwidth consumption. Multiple sitemap files are supported, with a Sitemap index file serving as an entry point for a total of 1000 Sitemaps.
Note that you can specify multiple sitemaps in your robots.txt file:
If you're bumping into the limitation, perhaps divide your users up somehow. Let's say each of your users has no more than 10,000 urls a piece, you could break it up into files and add this to your
Sitemap: http://www.example.com/sitemaps/users-001-005.xml Sitemap: http://www.example.com/sitemaps/users-006-010.xml Sitemap: http://www.example.com/sitemaps/users-011-015.xml Sitemap: http://www.example.com/sitemaps/users-016-020.xml Sitemap: http://www.example.com/sitemaps/users-021-025.xml Sitemap: http://www.example.com/sitemaps/users-026-030.xml Sitemap: http://www.example.com/sitemaps/users-031-035.xml Sitemap: http://www.example.com/sitemaps/users-036-040.xml Sitemap: http://www.example.com/sitemaps/users-041-045.xml Sitemap: http://www.example.com/sitemaps/users-046-050.xml
Now, you'd still have to be mindful of the 10MB limit for each individual sitemap, but this is an approach to handle the "too many" urls problem.