3

Last months I have noticed a strange behaviour in Google's Webmaster Tools.

Although I provide a valid Sitemap for my website, which is normally processed and without any issues from Google, while the number of Submitted Webpages keep rising (30.923 pages), the Indexed Pages don't rise anymore (26.819 pages since the end of January).

Checking the Crawler Stats, everything seems normal and there are no Crawl Errors.

There was no significant changes to the webpage these last months.

Anyone noticed a similar issue and any possible suggestion?

  • What is your number of indexed pages when you search with site:example.com on Google? – Zistoloen May 19 '14 at 10:43
  • @Zistoloen, the result I get is: 69.500 – GeGe May 20 '14 at 8:37
1

Submitting content with a Sitemap doesn't guarantee it will be indexed, though it is an aid to getting it crawled, among other things.

A common reason for a mismatch between what's submitted and indexed is duplication - either multiple versions of pages are being submitted, or else the pages are unique but Google judges them to be insufficiently unique to warrant indexing them.

You might try:

  1. Verifying that all URLs submitted by the XML Sitemap are unique (e.g., you don't have the same URL multiple times due to parameters that don't affect page content), and valid (not redirects, 404s, etc.)
  2. Checking that your page content is unique, or pretty much so. Look for things like large amounts of "boilerplate" copy used across numerous pages, or repeats of the same content in different categories.
  3. See if your site's taxonomy creates duplication, like archive and tag pages in WordPress for example.
  4. Run crawler software on your site to verify that content is actually accessible. If it finds less content than you expect, look for reasons, e.g., JavaScript-based navigation, nofollow links, etc. (Google Webmaster Tools will have warning notifications if, say, robots.txt is blocking access).
|improve this answer|||||
1

I think it's a problem on Google's end.

I keep a spreadsheet monitoring my GWT sitemap stats on a site and I have seen the exact same behavior. The number of pages indexed on Jan 20 was different from the number when I checked Jan 10, as it nearly always was every few days I checked for the last three and a half years, but since Jan 20, the number has not changed even though the number of pages submitted has increased and I have even added another sitemap.

Like you, all other indicators seem normal, and the site is a fairly dynamic, large, well-indexed site (well over 100K pages) that, until the last three months, had previously never had the same number of Indexed pages for more than a week.

I've been wondering if it was a bug but I couldn't find anyone else on the Internet talking about it and as I've often seen different aspects of GWT be temporarily slow or outdated, I kept expecting it to fix itself and haven't looked into it further. I'm guessing it doesn't effect everyone or we would have heard about it elsewhere by now (though anyone could confirm this if your Indexed numbers have changed since mid-January), but I think you've provided evidence that it's more than just me.

|improve this answer|||||
1

Welcome to the world of Google. If you submit a sitemap, the number of pages indexed never seems to match the number submitted in Google Webmaster Tools. It has been this way for years. I am not sure why, but I can tell you this much. Google seems to apply an algorithm that manages the speed of which the pages are fetched from a sitemap. The more pages in a sitemap, the faster it goes. As it begins to near the end, it slows down more and more. I suspect that it stops almost entirely short of the total number of pages at least giving the appearance that the entire sitemap is not indexed. And that may be the case. It just may be that the number of pages fetched from the sitemap each day reaches such a low number that the final few pages (often measured in a few thousand) takes a long time. For my site, this can range from 2000 to 4000 which seems to be exactly what you are experiencing.

Another consideration along the way is that the sitemap is read and all of the entries are put into the queue to be fetched. As Google slows down fetching pages from the sitemap, other pages that are deemed important, discovered via other means, perform well in searches, and so on are moved to the top of the queue and are a part of the mix. So if Google slows down to about 1000-2000 pages fetched each day, fewer and fewer are the remaining pages from the sitemap as each day goes along making way for other items in the queue.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.