Proper sitemap.xml setup

Question

I have a dynamic site which has many (well, less than 50) users. Each user is allowed to create as many pages as they want. I know that there is a limit to how many pages you can be listed in sitemap.xml, and for now I am under that limit, but I would like to know what the best way to enable search engines to know (all?) about my user's pages while still being under that limit even if I have more than the limit.

Answer 1

Sitemap Limits:

Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap. Sitemaps can be compressed using gzip, reducing bandwidth consumption. Multiple sitemap files are supported, with a Sitemap index file serving as an entry point for a total of 1000 Sitemaps.

Note that you can specify multiple sitemaps in your robots.txt file:

If you're bumping into the limitation, perhaps divide your users up somehow. Let's say each of your users has no more than 10,000 urls a piece, you could break it up into files and add this to your robots.txt file:

Sitemap: http://www.example.com/sitemaps/users-001-005.xml
Sitemap: http://www.example.com/sitemaps/users-006-010.xml
Sitemap: http://www.example.com/sitemaps/users-011-015.xml
Sitemap: http://www.example.com/sitemaps/users-016-020.xml
Sitemap: http://www.example.com/sitemaps/users-021-025.xml
Sitemap: http://www.example.com/sitemaps/users-026-030.xml
Sitemap: http://www.example.com/sitemaps/users-031-035.xml
Sitemap: http://www.example.com/sitemaps/users-036-040.xml
Sitemap: http://www.example.com/sitemaps/users-041-045.xml
Sitemap: http://www.example.com/sitemaps/users-046-050.xml

Now, you'd still have to be mindful of the 10MB limit for each individual sitemap, but this is an approach to handle the "too many" urls problem.

See the CNN and Google robots.txt files to see multiple sitemaps in action.