Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the best practices for sitemaps esp. for large sites?

Also, is there anyway to make google (and other search engines) only crawl the pages in the sitemap?

Thanks

Update:
Also, any idea how stackoverflow handle this? I'd like to know but unfortunately (also understandingly) they have blocked access to their sitemap.

share|improve this question
1  
How big is the site? There is a size limit for both robots.txt and the sitemap. Amazingly, many exceed both, which is why I'm asking. –  Tim Post Aug 22 '10 at 18:27
    
@Tim, it's not really big for now (everything can fit in one sitemap), but I'm trying to plan ahead. –  Mee Aug 23 '10 at 7:24
add comment

1 Answer

up vote 12 down vote accepted

Include all pages. The purpose of the XML sitemap is to tell the search engines about all of your content. Not just the new stuff.

From the sitemaps.org website (emphasis mine):

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.

If you have a lot of content you can use multiple XML sitemaps.

If you have content that you don't want to have crawled or indexed you need to specifically tell the search engines not to crawl and index those pages. Use a robots.txt file to block any pages or directories that you do not wish to have crawled. You can also use a meta tag for that as well. But you cannot specify in an XML sitemap not to crawl unlisted pages.

share|improve this answer
    
Thanks for your answer, I'll include everything in the sitemap. –  Mee Aug 24 '10 at 0:00
    
Do you have a lib that can handle 50+k of pages? –  acidzombie24 Oct 1 '10 at 1:11
    
Are those 50k+ pages in a database? –  John Conde Oct 1 '10 at 2:08
    
You do not need to place every page of your site in a sitemap. A sitemap is useful for informing search engines about pages that are available for crawling. If the search engine can already see every crawlable page, and you're not adding information about "last modified", then there's zero reason to have one. –  Django Reinhardt Mar 28 at 13:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.