Google Webmaster Tools URLs in the index differ from the sitemap. How comes?

Question

Google Webmaster Tools says ~14000 URLs in the index but as you can see in the following picture there are only ~6500 from the sitemap. How comes?

I mean: my website is not properly "crawl-able". So google can only use the sitemap (and that's where I'm lost actually)

For example: if you google (in France) this: "tondeur de moutons lot et gronne" my website is second. But you can't access directly to this page from my website (= you have to go through the "search page" and type by hand "tondeur".

So what I mean is that there are very few pages (~ 100) that a bot can "guess" without the whole sitemap.

• google may not have crawled the full website but it says "14000 URLs are in the index"
• if so, why google can display some results that are only in the sitemap?
• if google has indexed ~6500 URLs from the sitemap, why does it say "14635"?

So... well err I'm lost... any idea?

Answer 1

Just because a page isn't listed in the sitemap doesn't mean Google won't index it. Google will follow links on pages listed in a sitemap even if that page isn't itself listed on the sitemap. So if you have pages that are not listed on any of your sitemaps that are linked to by a pages that is listed on a sitemap then Google will index it and that probably is the case here. It doesn't have to be deep links either. Just one link here and there adds up quick especially if the links use query variables for any kind of search, etc. Somewhere you have links to pages not listed in your sitemap that you (apparently) do not know about. Maybe you can use Google to help identify what those pages are and figure out why you don't know why you don't have them in your sitemap.

Answer 2

Sitemap has nothing to do with PageRank. If you find the contrary somewhere, please cite ;-)

Sitemap may help bots to index your site easier. That's all! (Of course, it also help you to know how many pages you have etc. but that's another story.)

You says "my website is not properly "crawl-able". So google can only use the sitemap". Do you mean that there are no links at all in any page in your website, therefore bots can not follow links? So, it's very important to fix it first.

Ok, let come back to your answer. Even if your site is not crawlable, even there is no link at all, bots can still guess the URLs. E.g. if you have 3 url /page/1, /page/2, /page/4, googlebot could guess and try to index /page/3 even if /page/3 is not in sitemap and no page has a link to it. Of if you have /page/1?p=1, /page/1?p=2, /page/2?p=1, googlebot could guess you'll have /page/2?p=2 and try to crawl that page. The easiest way is to turn on web server access log then look at urls crawled by googlebot.