Google Webmaster Tools Crawl Errors shows old URLs which have been removed from sitemaps

Question

My site is several years old and there are been pages which were removed a long time ago, have no links from the outside and the sitemaps were rebuilt several times after this, but Google Webmaster Tools shows that Google still crawls them and it counts the missing pages as 404 errors.

Also, for testing purposes I installed one more WordPress in a subdirectory and removed it after a couple of days. The sitemap has been rebuilt since then. This test site doesn't have any links from outside, but still, in crawl errors I find pages from there listed.

How can I resolve this problem and tell Google that these pages no longer exist?

Answer 1

If those pages got indexed while they were live then they would come up as errors. Some things you can do:

1) 301 redirect to the equivalent pages on the main site

2) 301 to closely related pages on the main site

3) Leave as is because its better to have a 404 error then redirect everything to the homepage.

I would also create a custom 404 page and include some navigation links on it that make sense. This will help people get to the right place quickly.

Have you tried "Fetch As Google" options? Sometimes it will make old errors go away if you have already fixed them. Also, keep in mind that sometimes it just takes a while for the errors to go away.

Last tip: when you copied your WP install to a sub directory you should have blocked the crawlers from accessing it in your robots.txt file. For a single page you can also do a noindex/nofollow.

Answer 2

You need to 410 those pages, then mark them as fixed in WMT. They'll return soon enough, but with a 410 error. When that happens, think of it as an advisory note. It will drop off the list in a few 3 months.

A 404 is sort of like a 302, both have little practical use and should be avoided. Google handles them as a temporary issue and will continue trying to crawl the url. When you see these in your crawl report, take action to get rid of them.

A 410 is sort of like a 301. Use it when the content -- which may not be the same as a url, because it may have been incorporated into another page -- is permanently gone. When you see a 302 or 404 in your crawl report, you should either 301 that url to the url where that content now exists, or 410 the url so Google knows it's really permanently gone.