I have a dynamic site which has many (well, less than 50) users. Each user is allowed to create as many pages as they want. I know that there is a limit to how many pages you can be listed in sitemap.xml, and for now I am under that limit, but I would like to know what the best way to enable search engines to know (all?) about my user's pages while still being under that limit even if I have more than the limit.
Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap. Sitemaps can be compressed using gzip, reducing bandwidth consumption. Multiple sitemap files are supported, with a Sitemap index file serving as an entry point for a total of 1000 Sitemaps.
Note that you can specify multiple sitemaps in your robots.txt file:
If you're bumping into the limitation, perhaps divide your users up somehow. Let's say each of your users has no more than 10,000 urls a piece, you could break it up into files and add this to your
Sitemap: http://www.example.com/sitemaps/users-001-005.xml Sitemap: http://www.example.com/sitemaps/users-006-010.xml Sitemap: http://www.example.com/sitemaps/users-011-015.xml Sitemap: http://www.example.com/sitemaps/users-016-020.xml Sitemap: http://www.example.com/sitemaps/users-021-025.xml Sitemap: http://www.example.com/sitemaps/users-026-030.xml Sitemap: http://www.example.com/sitemaps/users-031-035.xml Sitemap: http://www.example.com/sitemaps/users-036-040.xml Sitemap: http://www.example.com/sitemaps/users-041-045.xml Sitemap: http://www.example.com/sitemaps/users-046-050.xml
Now, you'd still have to be mindful of the 10MB limit for each individual sitemap, but this is an approach to handle the "too many" urls problem.
See the CNN and Google
robots.txt files to see multiple sitemaps in action.
2Note: the 10 MB limit is per sitemap, not total Jul 9, 2010 at 0:10
1It may be worth linking to and giving an example of the Sitemap Index file: sitemaps.org/protocol.php#index This is used by many sites as well, and has the benefits of drill-down in google webmaster tools. Couple examples: nanaimo.ca/sitemap.xml barnonedrinks.com/sitemap.xml Jul 9, 2010 at 9:02
Has anyone actually seen a 10MB sitemap? Yeowzerz!– Tim PostAug 8, 2010 at 14:21
I haven't seen a 10 MB sitemap, but here's one with 50,000 urls: gstatic.com/dictionary/static/sitemaps/…– artlungAug 8, 2010 at 14:44
(found in a few minutes by browsing some sitemaps for cnn and google and a few other large news/archive sites)– artlungAug 8, 2010 at 14:46