Webmaster tools show old URLs are crawled

Question

We renewed our website last year (2012) in December. After that we have put 301 redirection in place for the required pages. We removed some pages in old site when we moved to new site.

For example : In the old site we had http://www.mydomain.com/article/ and in new website we removed it.

Pages that don't exist in new site we redirect it to 404 page. Since then, the Webmaster -> Crawler Errors -> Not Found report shows these old URLs with a Response code of 404. Also the Linked from section displays internal site links that refer to this page. When I checked these pages, the link is not found.

I understand by reading this Google link http://support.google.com/webmasters/bin/answer.py?hl=en&answer=2409439 that these error don't harm our SEO, but I would like to know the exact reason why these happening.

Answer 1

Once Googlebot crawls a URL and gets a document there, it will never forget it. One site that I worked on changed its URL structure over 10 years ago. They put 301 redirects in place to redirect each URL to a SEO friendly URL. To this day, Googlebot will occasionally crawl all the old URLs, just to check to see.

If you don't want these URLs to appear in the 404 report there are two things that you can do.

• Redirect them to a different related page with 301 redirects. Note that you can't redirect them to the home page or to a 404 page, or Google still considers them 404.
• Use the HTTP status "410 Gone" which tells Google that you removed them on purpose. Google treats "410 Gone" very similarly to "404 Not Found" but it crawls the "gone" URLs less frequently and shouldn't put them in the 404 report.

Google keeps track of where it found links to pages. You are seeing this in the report of page that link to the 404 pages. This is more like a list of pages that currently link to or have linked to the page. Google is saying, "We are crawling this page because at some point in the past we found a link to it here." Again, googlebot just doesn't seem to forget, even when the links are gone.