Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

If a page is missing in sitemap.xml (but crawable through HTML hyperlinks), may this cause it not to be indexed? (I mean it would be indexed without sitemap, but not indexed with sitemap because it is missing in the sitemap.)

share|improve this question
add comment

4 Answers

up vote 3 down vote accepted

If the page is crawlable, then it will be indexed unless you have explicitly marked it as noindex. The sitemap.xml is another way for Google to discover pages on your website as Ilmari Karonen mentioned in the answer. The only thing that will happen is that the page will not be found via the sitemap but by other means.

share|improve this answer
add comment

No, leaving a URL out of your sitemap will not prevent Google from indexing it.

To quote Google's sitemaps FAQ:

"Sitemaps provide an extra way for us to understand what pages you have on your site, and can help speed up the discovery of new and updated pages on your site. But neither HTML nor XML Sitemaps replace the normal crawling process."

In fact, having an up-to-date partial sitemap may, in some cases, even speed up the discovery of pages that are not included in the sitemap. For example, if you create a new page X and link to it from a page Y in your sitemap, and update the "lastmod" date of page Y in your sitemap to reflect the fact that the new link was added, then Google will notice the update and schedule page Y for crawling, which will cause them to notice the link to page X and add it to their crawl queue as well.

That said, one can also think of scenarios where a partial sitemap could slow down the indexing of pages not listed in the sitemap. For example, let's say your site was divided into two distinct sections A and B, with few if any links between them, and your sitemap only covered section A. Since Google uses the last modification dates in the sitemap to prioritize recrawling, it's at least plausible that, if you kept updating the content in section A (and your sitemap of it) frequently enough that Googlebot stayed busy recrawling it, pages in section B might not get recrawled as often as they would be in the absence of the sitemap.

Of course, this is kind of an artificial scenario, and easily avoided. Still, it at least does show that, theoretically, it's possible for a partial sitemap to slow down the indexing of pages not included in it.

share|improve this answer
add comment

As you don't need to provide an XML sitemap to be indexed by Google, then I think it's fair to assume that leaving a page off the XML sitemap won't cause it to be exlcuded either.

On the flipside there is not any evidence that XML sitemaps do anything for your Google ranking. It is merely implied that giving Google a helping hand by providing an XML sitemap, should help.

share|improve this answer
add comment

The lastmod label with an expired date (p.ex 06-10-2013) is a possible cause of not indexing sitemaps.

I have a very big website and my sitemaps in google where not indexed at all. Google only was indexing small parts of my sitemaps. Today, I have changed all my lastmod labels, actualizing them at today.

There is any possibility that the lastmod was the cause of wrong indexing pages.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.