For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the best practices for sitemaps esp. for large sites?

Also, is there anyway to make google (and other search engines) only crawl the pages in the sitemap?


Also, any idea how stackoverflow handle this? I'd like to know but unfortunately (also understandingly) they have blocked access to their sitemap.

  • 1
    How big is the site? There is a size limit for both robots.txt and the sitemap. Amazingly, many exceed both, which is why I'm asking.
    – Tim Post
    Aug 22, 2010 at 18:27
  • @Tim, it's not really big for now (everything can fit in one sitemap), but I'm trying to plan ahead.
    – Mee
    Aug 23, 2010 at 7:24

1 Answer 1


Include all pages. The purpose of the XML sitemap is to tell the search engines about all of your content. Not just the new stuff.

From the sitemaps.org website (emphasis mine):

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.

If you have a lot of content you can use multiple XML sitemaps.

If you have content that you don't want to have crawled or indexed you need to specifically tell the search engines not to crawl and index those pages. Use a robots.txt file to block any pages or directories that you do not wish to have crawled. You can also use a meta tag for that as well. But you cannot specify in an XML sitemap not to crawl unlisted pages.

  • Thanks for your answer, I'll include everything in the sitemap.
    – Mee
    Aug 24, 2010 at 0:00
  • Do you have a lib that can handle 50+k of pages?
    – user1585
    Oct 1, 2010 at 1:11
  • Are those 50k+ pages in a database?
    – John Conde
    Oct 1, 2010 at 2:08
  • You do not need to place every page of your site in a sitemap. A sitemap is useful for informing search engines about pages that are available for crawling. If the search engine can already see every crawlable page, and you're not adding information about "last modified", then there's zero reason to have one. Mar 28, 2014 at 13:19
  • 2
    This answer seems somewhat conflicting with webmasters.stackexchange.com/a/5151/30596. Quoting @John Mueller from Google, Using a Sitemap file won't reduce our normal crawling of your site. It's additional information, not a replacement for crawling. Similarly, not having a URL in a Sitemap file doesn't mean that it won't be indexed.
    – user
    Jun 14, 2015 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.