Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return a 200, all those 15,000 pages won't index in Google. They aren't appearing in search results.

These are constant errors Google Webmaster Tools keeps reporting and I'm not sure what the problem is. We've thought of a DNS issue, but it shouldn't be a DNS issue, because if it were, no page would be indexed (I have 10,000 perfectly indexed).

Regarding URL parameters, my pages do not share a similarity in URL parameters that can make it obvious to me what could be causing the error.

share|improve this question
    
I am experiencing a similar problem. –  c s h May 10 '13 at 22:52
    
Its unlikely this will help but could we see a url of a page that is not indexed? –  bybe May 10 '13 at 23:10
    
Hi @c-s-h any luck on your end? And sorry we cannot post URLs :( - and we're still having a rough time with this. –  Edgar Quintero May 17 '13 at 20:44
    
@EdgarQuintero Yes not much luck. –  c s h May 19 '13 at 2:34
    
@EdgarQuintero Google reports are about 3-4 days old usually, look at the time at which the crawl happened. Also try to curl your urls to see if you see a 404 or 200. –  Sairam Sep 8 '13 at 5:25

2 Answers 2

Google offers a fetch as Googlebot option as part of Google Webmaster Tools.

Use that tool to fetch your pages from Google's servers. That will show you if it is a problem with connectivity from Google or with the Googlebot user agent as opposed to downtime of your website when nobody can reach it periodically.

share|improve this answer

I suggest you temporary put noindex for those 404. It would very much harmful to your site where 2/3 is not indexed.

Then slowly identify what's the real problem. Since you can't disclose your link then no way we can tell what are the real problem.

But I sure you can find plenty of sources on "why google didn't indexed my pages".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.