Google reports 404 errors on URLs that haven't existed for six months and are no longer linked to on my site

Question

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found).

But when I look at the source pages that are accused of having these broken links, there are no such links in them.

As of now, I have 6 404 errors in sitemaps and 28 in the rest of my site.

E.g. the address http://www.webdesk.co.il/articles.php?id=13 does not exist, yet Google indicates that I have 3 pages that link to this address:

http://www.webdesk.co.il/sitemapgen.php
http://www.webdesk.co.il/articles.php?t=css
http://www.webdesk.co.il/articles.php?id=17 //this address dont even exist

If you open each of these pages, you'll see that they don't have these links in them.

I thought this may be a result of an old crawling that was not yet updated, but those pages haven't existed for over than 6 months and Google indicates that all the crawling results are up to date (Dec 22 2012).

Why do I keep getting these 404 errors by Google?

Answer 1

Google does seem to hang on to old urls for a very long period of time, my guess is that:-

• These links exist on another site either because of a typo or other error.

• The content used to be there a very long time ago perhaps an old version of the site created by a former owner of the domain for example.

• An old link which looks like a new link because of htaccess redirection e.g. one of my sites still gets links through to its naked IP but these show up in the logs as the domain because of url redirection settings (took ages to track that down).

• The link exists in the html content but not in the visible part of the page - this most often happens because of errors in visual html editors or commented content.

The best plan is to have your server respond to requests for these urls with a 410 status code otherwise google will keep looking for them.

410 Gone Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed and the resource should be purged. Upon receiving a 410 status code, the client should not request the resource again in the future. Clients such as search engines should remove the resource from their indices. Most use cases do not require clients and search engines to purge the resource, and a "404 Not Found" may be used instead.

Answer 2

Here is what Google's John Mueller (who works on Webmaster Tools and Sitemaps) has to say about 404 errors that appear in Webmaster tools:

HELP! MY SITE HAS 939 CRAWL ERRORS!!1

I see this kind of question several times a week; you’re not alone - many websites have crawl errors.

1. 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
2. In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
3. What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
4. You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it. http://support.google.com/webmasters/bin/answer.py?answer=2467403
5. We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages. http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
6. There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors. http://support.google.com/webmasters/bin/answer.py?answer=181708
7. Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.