0

We have a modestly successful ecommerce site that employs GWT.

While making changes to URLs we always tried to replace the old link with a new one, using a 301 redirect to tell search crawlers where to look and preserve the ranking.

However, due to a technical problem, all the redirects had to be removed, leaving only the current links of all products and categories alive. Since that day, 404 errors are piling up.

Now, we had hoped that with a delay of 3 months, Google would have picked up on all redirects and removed the old URLs from the index, but it seems that is not so. After reading up on the meaning of 404s for search engines, we would be fine with the way it's working now.

However: For most, if not all of the 404 errors cropping up after every night (about 4000), we can click them and see where they were linked. We see 2 types of sources:

  • Another web page from our site that does not exist anymore either
  • Our sitemap

Now, opening up the sitemap GWT refers to and manually looking for the 404 always shows that the URL is not there. Almost every day we resubmit the sitemap in the hopes that this would fix things, but it seems that GWT is using an older, cached version of our sitemap that still contains the nonexistent URLs.

Is there any way out of this except waiting? It has been 3 months of redirect and 4 months since the URL changes, and we really expected that Google would have picked up on that.

1

We get these questions all the time.

You have nothing to worry about. 404's happen and are a natural part of doing business. But what many people do not know is that it really does take a while for a page to disappear using a 404 error. The reason is this:

404 errors are temporarily gone where 410 errors are gone. If a 410 error can be produced, it would take much less time. However, 404 errors are much easier.

In light of this, Google will retry each page with a 404 error a number of times. How long it takes is based upon how often Google would normally visit the page. Additionally, if any new links are found anywhere on the Internet, Google will retry the page again. It is possible that any page that has been removed will be retried periodically for years.

As a warning, do not Mark as Fixed any 404 error where the page has been removed. You are in effect telling Google that the page should exist, that you fixed the problem, and to try again. If you fix a proper 404 error, you are in a very real effect, starting the retry process all over again and it will take that much longer to remove the page.

  • Thanks for explaining. That leaves the last problem: GWT states that the links are contained in the sitemap. Although a recent sitemap without the links has been submitted multiple times, it still says that the sitemap links the broken links. It looks as if the submitted and displayed sitemap is not actually used to determine which links to crawl. – 0xCAFEBABE Oct 20 '15 at 8:54
  • @0xCAFEBABE Yeah. Excuse me while I grin. Google is notoriously slow especially when it comes to GWT and the sitemap. You really have to determine if that is actually true or not on your own. Google, it seems, will fetch a sitemap, but will not audit against it for a while. I have seen lag times of several weeks. Not sure what the process is exactly, but it can be slow. Keep in mind that Google uses the sitemap mostly for audit purposes if it can crawl your site properly and while it may have fetched the new sitemap, it has not begun to audit it yet and is still reading the old one. – closetnoc Oct 20 '15 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.