How do I find out why Google is now indexing fewer pages from my site?
Here is what Google's help says although I don't find it that helpful:
A steady increase in the number of crawled and indexed pages indicates that Google can regularly access your content, and that your site is being indexed.
If you see a sudden drop in the number of indexed pages, it may mean that your server is down or overloaded, or that Google is having trouble accessing your content.
To see the percentage of your site that’s accessible to Google, compare the number of crawled pages with the number of pages blocked by robots.txt. Sudden spikes or dips that appear in several charts can indicate problems with site configuration, redirects, or security.
Starting on June 3rd, Google also says: Googlebot can't access your site
Over the last 24 hours, Googlebot encountered 179 errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is 5.0%.
In January 2013 I updated my site from Drupal 6 to Drupal 7, and some links were changed. I've tried to solve the issue by following Google's instructions:
After you think you've fixed the problem, use Fetch as Google to verify that Googlebot can properly access your site.
- I've clicked "fetch as Google" and it says "success" URL and linked pages submitted to index.
- I've clicked on those site errors, and it says:
- DNS OK
- Server Connectivity OK
- Robots.txt Fetch OK
Should I go and check out every single page of my site with "fetch as Google"? What if I find a link is broken because I no longer use it?
robots.txt
file?