Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a dynamic site which has many (well, less than 50) users. Each user is allowed to create as many pages as they want. I know that there is a limit to how many pages you can be listed in sitemap.xml, and for now I am under that limit, but I would like to know what the best way to enable search engines to know (all?) about my user's pages while still being under that limit even if I have more than the limit.

share|improve this question
add comment

1 Answer

up vote 11 down vote accepted

Sitemap Limits:

Sitemap files have a limit of 50,000 URLs and 10 megabytes per sitemap. Sitemaps can be compressed using gzip, reducing bandwidth consumption. Multiple sitemap files are supported, with a Sitemap index file serving as an entry point for a total of 1000 Sitemaps.

Note that you can specify multiple sitemaps in your robots.txt file:

If you're bumping into the limitation, perhaps divide your users up somehow. Let's say each of your users has no more than 10,000 urls a piece, you could break it up into files and add this to your robots.txt file:

Sitemap: http://www.example.com/sitemaps/users-001-005.xml
Sitemap: http://www.example.com/sitemaps/users-006-010.xml
Sitemap: http://www.example.com/sitemaps/users-011-015.xml
Sitemap: http://www.example.com/sitemaps/users-016-020.xml
Sitemap: http://www.example.com/sitemaps/users-021-025.xml
Sitemap: http://www.example.com/sitemaps/users-026-030.xml
Sitemap: http://www.example.com/sitemaps/users-031-035.xml
Sitemap: http://www.example.com/sitemaps/users-036-040.xml
Sitemap: http://www.example.com/sitemaps/users-041-045.xml
Sitemap: http://www.example.com/sitemaps/users-046-050.xml

Now, you'd still have to be mindful of the 10MB limit for each individual sitemap, but this is an approach to handle the "too many" urls problem.

See the CNN and Google robots.txt files to see multiple sitemaps in action.

share|improve this answer
1  
Note: the 10 MB limit is per sitemap, not total –  Andreas Bonini Jul 9 '10 at 0:10
    
Thanks Andreas, I updated the answer for clarity. –  artlung Jul 9 '10 at 0:11
1  
It may be worth linking to and giving an example of the Sitemap Index file: sitemaps.org/protocol.php#index This is used by many sites as well, and has the benefits of drill-down in google webmaster tools. Couple examples: nanaimo.ca/sitemap.xml barnonedrinks.com/sitemap.xml –  JasonBirch Jul 9 '10 at 9:02
    
Has anyone actually seen a 10MB sitemap? Yeowzerz! –  Tim Post Aug 8 '10 at 14:21
    
I haven't seen a 10 MB sitemap, but here's one with 50,000 urls: gstatic.com/dictionary/static/sitemaps/… –  artlung Aug 8 '10 at 14:44
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.