15

For a website with dynamic content (new content is constantly being added), should I only include the newest content in the sitemap or should I include everything (with a sitemap index)? What are the best practices for sitemaps esp. for large sites?

Also, is there anyway to make google (and other search engines) only crawl the pages in the sitemap?

Thanks

Update:
Also, any idea how stackoverflow handle this? I'd like to know but unfortunately (also understandingly) they have blocked access to their sitemap.

Improve this question
2
  • 1
    How big is the site? There is a size limit for both robots.txt and the sitemap. Amazingly, many exceed both, which is why I'm asking.
    – Tim Post
    Commented Aug 22, 2010 at 18:27
  • @Tim, it's not really big for now (everything can fit in one sitemap), but I'm trying to plan ahead.
    – Mee
    Commented Aug 23, 2010 at 7:24

1 Answer 1

Reset to default
13

Include all pages. The purpose of the XML sitemap is to tell the search engines about all of your content. Not just the new stuff.

From the sitemaps.org website (emphasis mine):

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.

If you have a lot of content you can use multiple XML sitemaps.

If you have content that you don't want to have crawled or indexed you need to specifically tell the search engines not to crawl and index those pages. Use a robots.txt file to block any pages or directories that you do not wish to have crawled. You can also use a meta tag for that as well. But you cannot specify in an XML sitemap not to crawl unlisted pages.

Improve this answer
8
  • Thanks for your answer, I'll include everything in the sitemap.
    – Mee
    Commented Aug 24, 2010 at 0:00
  • Do you have a lib that can handle 50+k of pages?
    – user1585
    Commented Oct 1, 2010 at 1:11
  • Are those 50k+ pages in a database?
    – John Conde ♦
    Commented Oct 1, 2010 at 2:08
  • You do not need to place every page of your site in a sitemap. A sitemap is useful for informing search engines about pages that are available for crawling. If the search engine can already see every crawlable page, and you're not adding information about "last modified", then there's zero reason to have one.
    – Django Reinhardt
    Commented Mar 28, 2014 at 13:19
  • 2
    This answer seems somewhat conflicting with webmasters.stackexchange.com/a/5151/30596. Quoting @John Mueller from Google, Using a Sitemap file won't reduce our normal crawling of your site. It's additional information, not a replacement for crawling. Similarly, not having a URL in a Sitemap file doesn't mean that it won't be indexed.
    – user
    Commented Jun 14, 2015 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.