Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have recently created a sitemap.xml file and uploaded it to my Google Webmasters Tools account. Google didn't report any issues or errors with the uploaded sitemap of my site.

Now my question is:

  • How do I know if my sitemap is working within Google Webmaster Tools?

The reason I ask is I don't know what I'm suppose to being seeing or looking for, and it feels like I've uploaded an useless file.

share|improve this question
add comment

migrated from stackoverflow.com May 27 '12 at 17:07

This question came from our site for professional and enthusiast programmers.

4 Answers

A Sitemap file helps search engines to discover new and updated URLs on your website. In particular, if your website is fairly large, then this can help them to be able to focus on the new & updated content, instead of having to blindly crawl through everything to see if anything has changed. That can result in new content being found much faster, which can be quite noticeable especially if the site is larger or more complex.

With Google in particular (I work at Google; I don't know how other search engines handle these), it also does the following:

  • Find the number of indexed URLs for your website: These statistics are recalculated daily and very accurate. You can find these in the Sitemaps detail page.
  • Discover canonicalization issues: If the numbers there don't match up, that's frequently a sign that you're specifying URLs in the Sitemap file that don't match what we find during our crawling. That's usually a sign that you need to work on canonicalization.
  • Help with canonicalization: When we find multiple URLs on your site that show identical content, we will give any URL that's listed in a Sitemap an extra edge, even if you don't use other canonicalization methods.
  • Find badly-indexed parts of your site: These counts are supplied per Sitemap file, so you can create separate Sitemap files for logical sections of your site, to discover areas where Google isn't indexing as much as you'd like.
  • Prioritize crawl errors: In the crawl errors section, URLs that were specified in Sitemaps files are listed separately. Since you specifically supplied these URLs, we assume that you want them indexed, and that any crawl errors there are important.

Additionally, you can use several extensions in Sitemaps files (eg for images, video, News, or internationalization), should you choose to do that. These extensions are all optional.

For most websites, the most visible element of Sitemaps files is that you can see the indexed URL count. It can take a day or so to appear, so if you just submitted a Sitemap for the first time, you may need to be a bit patient. While other ways (eg a site:-query) are very, very rough approximations, this count is extremely acccurate.

Edited to add: another thing that I personally find extremely useful with regards to Sitemaps is that if you're not generating them directly with your CMS, you invariably find out a lot about how your website is crawlable, and what kind of URLs are discovered during that process. I've seen many cases where crawling a website with a tool on your side (eg a Sitemaps generator) will bubble up issues that you might miss otherwise, be that session-IDs in URLs, duplicate content through URL differences, infinite spaces (such as endless calendars), or even parts of a site that aren't linked at all.

share|improve this answer
    
Great info. I have also heard that with some high traffic heavily crawed sites it's better not to use a sitemap because Google does a better job crawling and if there is anything missing from your sitemap it might stop getting indexed. –  Chris_O May 29 '12 at 22:18
2  
@Chris_O You're mashing different problems together. In that example, it's not that Google is doing a "better" job crawling on its own; the sitemap itself is faulty. That's isn't a direct line of argument to "don't use a sitemap at all." The solution to that situation is to fix the sitemap. Additionally, sitemaps are informative, not directives. Something missing from a sitemap doesn't mean Google won't find it on its own, or disregard it. –  Su' May 30 '12 at 4:36
    
The site in question has over 40k indexed pages and new content gets indexed in less than 5 minutes(with no site map). Based on your answer we're going to start building them and break them into years. –  Chris_O May 30 '12 at 6:29
    
@john-mueller hi mr. m. - i once submitted a test sitemap with 1000 URLs, we got back an index count of about 700, now we tested all 1000 URLs via the site:www.complete.org/url/to/the/page.html, we got back an count way below the 700 urls (more in the 200 region). what does this mean? –  Franz May 30 '12 at 15:37
2  
@Franz There are sometimes details involved which make it hard to reproduce the indexed URL count with site:-queries. For example, there are situations where we might combine multiple URLs and only show one of them for a site:-query. So if you see a difference there, it's usually not worth worrying about. –  John Mueller Jun 2 '12 at 20:13
show 2 more comments

If you're not getting any errors then you can assume Google has parsed it and is aware of the contents. But that doesn't mean they will crawl and/or index those pages. Sitemaps are just another way to tell search engines about your pages. They are not obligated to crawl and index any or all of those pages. The same applies to them finding pages through links or URL submissions.

share|improve this answer
1  
As added information, it's pretty easy to see if Google understands the sitemap. Check under Optimizations -> Sitemaps. It will tell you how many URLs have been submitted in the sitemap. The sitemap is successfully read the contents and found this many links in it. The second number is how many have been indexed and as John points out, completely up to Google as to whether they find none, some, or most of your content worthwhile. It will always be less than the submitted amount, Google just finds some pages to be not-so-hot and doesn't bother with them. –  Fiasco Labs May 27 '12 at 18:36
add comment

Google usually does a good job of crawling your website if you have a good number of quality links. If your spending a lot of time looking at the number of pages you have indexed I would suggest its better to improve your site and get some quality links.

share|improve this answer
add comment

Maybe a sitemap validation tool like this, though I've never "not" seen the sitemap start to kick in.

share|improve this answer
    
The file has already been validated. That's not the question. –  Su' May 27 '12 at 19:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.