4

My server went down for a couple days.(2-3 august)
15 days later, Google Webmaster Tools notified me of increase in 404 errors.

The problem is that Google reports 70,000 error pages while it has ever indexed maximum 7000 pages, as shows the index status:

enter image description here

While this is the crawl error page:

enter image description here

How can I understand which pages Google cannot find? I can see maximum 1000 errors but these 1000 errors are old and were present before too. And how is that possible that it cannot find 70,000 pages if it had ever indexed maximum 7000?

Note I have tried to run Xenu link checker to check for internal broken links and none were found.

Note2: In correspondence with the spike in errors (21th august) there were a spike in crawled pages too:

enter image description here

  • My guess is that the 70K+ perhaps consists of all the non-canonical versions of those URLs as well? Since your "server was down", it had no way of knowing the canonical URLs, so it perhaps attempted to crawl... example.com/page as well as example.com/page?foo=1 and example.com/page?foo=2, etc. – MrWhite Aug 28 '16 at 18:36
  • that could be, but i doubt i have so many non-canonical urls. Anyway do i have any way to debug this? Since the 1000 errors that G shows are old and doesnt help. How can i know which are the other 69'000 errors? – giò Aug 28 '16 at 20:58
  • If you sort by "Detected" (ascending) do you see any "newer" records? If you "Download" the list, can you download more than the first 1000? Google Search Console (formerly Webmaster Tools) is unfortunately a bit limited when it comes to processing larger reports. – MrWhite Aug 28 '16 at 21:06
2

This could happen in the case you have a lot of URL parameters which usually result in a lot of pages being created -Googlebot automatically ensures that not all of these are indexed. However, when your site went down all of these could have shown up as 404 errors.

How does one analyse and clear these?

There is a priority column in the crawl errors report. Sort from 1 to the last and start fixing from 1. Usually we have seen that solving the first 1 has an effect of solving a lot of other errors - In your case 100-1000s of errors will disappear if you solve the first (Since you have 70,000 errors)

We faced a similiar issue recently and this helped. Let us know how that went.

  • Thanks for your input. Actually i don't have such pages, and I am thinking now that the server down was not the issue here since the spike in errors happened 20 days after the down – giò Aug 29 '16 at 9:04
  • Google webmasters shows a lot of data with around 3 days lag. Errors appear and disappear witha much bigger lag. So 20 days seems not far off. – HopeKing Aug 29 '16 at 10:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.