3

How do I find out why Google is now indexing fewer pages from my site?

Google webmaster tools last check

Here is what Google's help says although I don't find it that helpful:

A steady increase in the number of crawled and indexed pages indicates that Google can regularly access your content, and that your site is being indexed.

If you see a sudden drop in the number of indexed pages, it may mean that your server is down or overloaded, or that Google is having trouble accessing your content.

To see the percentage of your site that’s accessible to Google, compare the number of crawled pages with the number of pages blocked by robots.txt. Sudden spikes or dips that appear in several charts can indicate problems with site configuration, redirects, or security.

Starting on June 3rd, Google also says: Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 179 errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is 5.0%.

In January 2013 I updated my site from Drupal 6 to Drupal 7, and some links were changed. I've tried to solve the issue by following Google's instructions:

After you think you've fixed the problem, use Fetch as Google to verify that Googlebot can properly access your site.

  • I've clicked "fetch as Google" and it says "success" URL and linked pages submitted to index.
  • I've clicked on those site errors, and it says:
    • DNS OK
    • Server Connectivity OK
    • Robots.txt Fetch OK

Should I go and check out every single page of my site with "fetch as Google"? What if I find a link is broken because I no longer use it?

Improve this question
3
  • Please can you include the contents of your robots.txt file?
    – zigojacko
    Sep 11, 2013 at 10:49
  • Please check your robot.txt file.. it may preventing crawler to crawl the pages.
    – user37371
    Mar 28, 2014 at 17:03
  • What about duplicate content, iframes, irrelevant pages, pages that were redirected.. These could be reflected on "ever crawled" but not necessarily be pages with actual content.
    – riseagainst
    Mar 28, 2014 at 19:25

3 Answers 3

Reset to default
2

Use a program such as Xenu's Link Sleuth to spider your site yourself. This may help you to find issues that you weren't aware of.

Any broken links you do find can either be removed or redirected using .htaccess (or there may be a feature in Drupal, I'm not familiar with Drupal) to a page that does exist.

Improve this answer
1
  • Drupal at least had a Global Redirect that one could use to redirect any page to any other page. It also had the Link Checker tool that would do what your Sleuth link does, internally.
    – Alexis Wilke
    Sep 3, 2017 at 19:10
1

Make sure you've set up a Google Sitemap set up, and registered with Google Webmaster Tools. This Drupal plugin should help you get started: https://drupal.org/project/xmlsitemap

The sitemap will tell Google about all of the pages that are available on your site, so that it knows to crawl them.

Improve this answer
1

Google will not keep pages that are similar in nature from the same domain indexed. It is natural to see this decline if there is no unique value at these URL's.

I'd check the URL structure change between Drupal 6 and Drupal 7 and see if those URL's serve anything of value or are referenced in your robots directive.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.