Google Webmaster Tools showing provably wrong 404 crawl errors

Question

We have a modestly successful ecommerce site that employs GWT.

While making changes to URLs we always tried to replace the old link with a new one, using a 301 redirect to tell search crawlers where to look and preserve the ranking.

However, due to a technical problem, all the redirects had to be removed, leaving only the current links of all products and categories alive. Since that day, 404 errors are piling up.

Now, we had hoped that with a delay of 3 months, Google would have picked up on all redirects and removed the old URLs from the index, but it seems that is not so. After reading up on the meaning of 404s for search engines, we would be fine with the way it's working now.

However: For most, if not all of the 404 errors cropping up after every night (about 4000), we can click them and see where they were linked. We see 2 types of sources:

• Another web page from our site that does not exist anymore either
• Our sitemap

Now, opening up the sitemap GWT refers to and manually looking for the 404 always shows that the URL is not there. Almost every day we resubmit the sitemap in the hopes that this would fix things, but it seems that GWT is using an older, cached version of our sitemap that still contains the nonexistent URLs.

Is there any way out of this except waiting? It has been 3 months of redirect and 4 months since the URL changes, and we really expected that Google would have picked up on that.

Answer 1

We get these questions all the time.

You have nothing to worry about. 404's happen and are a natural part of doing business. But what many people do not know is that it really does take a while for a page to disappear using a 404 error. The reason is this:

404 errors are temporarily gone where 410 errors are gone. If a 410 error can be produced, it would take much less time. However, 404 errors are much easier.

In light of this, Google will retry each page with a 404 error a number of times. How long it takes is based upon how often Google would normally visit the page. Additionally, if any new links are found anywhere on the Internet, Google will retry the page again. It is possible that any page that has been removed will be retried periodically for years.

As a warning, do not Mark as Fixed any 404 error where the page has been removed. You are in effect telling Google that the page should exist, that you fixed the problem, and to try again. If you fix a proper 404 error, you are in a very real effect, starting the retry process all over again and it will take that much longer to remove the page.