Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

How may I find why Google indexes less webpages in my site?

I´ve recently find this: Google webmaster tools last check

And I wonder why could make Google index less pages. I´ve then tried the Google help, and it wasn´t that helpful:

A steady increase in the number of crawled and indexed pages indicates that Google can regularly access your content, and that your site is being indexed.

If you see a sudden drop in the number of indexed pages, it may mean that your server is down or overloaded, or that Google is having trouble accessing your content.

To see the percentage of your site that’s accessible to Google, compare the number of crawled pages with the number of pages blocked by robots.txt. Sudden spikes or dips that appear in several charts can indicate problems with site configuration, redirects, or security.

Now, on June 3rd Google says: Googlebot can't access your site

Over the last 24 hours, Googlebot encountered 179 errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is 5.0%.

Now, I´ve recently (january 2013) updated my site from Drupal 6 to Drupal 7, and some links were changed. I´ve tried to solve the issue, and after that I´ve followed Google´s instructions:

After you think you've fixed the problem, use Fetch as Google to verify that Googlebot can properly access your site.

I´ve clicked "fetch as Google" and it says "success" URL and linked pages submitted to index. I´ve clicked on those site errors, and it says: DNS OK Server Connectivity OK Robots.txt Fetch OK

So, how may I find issues? Should I go and check out every single page of my site? What if I find a link is broken because I no longer use it?

share|improve this question
    
Please can you include the contents of your robots.txt file? –  zigojacko Sep 11 '13 at 10:49
    
Please check your robot.txt file.. it may preventing crawler to crawl the pages. –  gugnu Mar 28 at 17:03
    
What about duplicate content, iframes, irrelevant pages, pages that were redirected.. These could be reflected on "ever crawled" but not necessarily be pages with actual content. –  guisasso Mar 28 at 19:25
add comment

3 Answers 3

Use a program such as Xenu's Link Sleuth to spider your site yourself. This may help you to find issues that you weren't aware of.

Any broken links you do find can either be removed or redirected using .htaccess (or there may be a feature in Drupal, I'm not familiar with Drupal) to a page that does exist.

share|improve this answer
add comment

Make sure you've set up a Google Sitemap set up, and registered with Google Webmaster Tools. This Drupal plugin should help you get started: https://drupal.org/project/xmlsitemap

The sitemap will tell Google about all of the pages that are available on your site, so that it knows to crawl them.

share|improve this answer
add comment

Google will not keep pages that are similar in nature from the same domain indexed. It is natural to see this decline if there is no unique value at these URL's.

I'd check the URL structure change between Drupal 6 and Drupal 7 and see if those URL's serve anything of value or are referenced in your robots directive.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.