Why is the number of pages (of my website) indexd by googlebot decreasing?

Question

I am monitoring the number of pages indexed by Gbot, for one of my websites .

I noticed that since last week, the number of pages has dropped by about 60 or so. Now to be fair, I have not added that much over the last week (actually I dont think I added any new content). But at the very least, I would have thought that the number of indexed pages would be static (if I was not adding anything new) - so I dont understand why the numbers have dropped.

Does anyone have an official explanation for this?. I have checked the Google webmaster site, and have not come accross anything that addresses this issue directly.

Answer 1

It looks like very recently Google has had a "glitch" with their indexing, it appears a lot of sites were seeing pages drop out of Google's index read here for more details:

http://searchengineland.com/is-google-broken-sites-big-small-seeing-indexing-problems-53701

There is always the possibility that there are other issues impacting your site's ability to be indexed, but since this is a wide spread issue that Google seems to have confirmed I'd start there.

Answer 2

First, Google Bot can detect if pages were not updated and not crawl them in the first place (this should not reduce the amount of indexed pages). He can detect that in multiple ways, for example using timestamps or etags.

After crawling, there is no promise that it will get indexed - it may, or it may not. There are quite a lot of factors that determine if Google will index the pages the bot crawled.

From time to time Google will also purge their index and remove pages that they don't think should be there. Things like content-duplication, junk/spam content and other factors are used to decide this. No one (except Google) actually knows in advance when these kind of purges happen, or what factors are used. But they do happen from time to time.

Also, read the content under the title "Indexing Stuff" at the link for even more information, http://www.google.com/support/forum/p/Webmasters/thread?tid=2ad71287c04eb280

Answer 3

This sounds like Googlebot is no longer able to find your pages. If a page is still on your site, Google won't remove it from the index, but if many pages are returning 404 errors then they would generally be removed. In Webmaster Tools, check the "fetch as Googlebot" tool on some of your pages.

Duplicate content on the same site (e.g. the URLs page.html and page.html?x=nothing) may be ignored/removed but duplicate content across multiple sites (e.g. if you copied content from Wikipedia) would never be removed from the index per se, it simply will not rank anywhere.