Why are there Google crawl errors for unknown urls?

Question

I have been getting crawl errors in Google's WebMaster tools for urls like the following:

http://madisonps.com/Media/eurl.axd/812e5d58643d804993276b6417a3c5ef

These urls are not part of our site and I cannot figure out where they are coming from. I have researched several other posts here and from Google search, but I still cannot find a solution that will prevent search bots from creating these types of urls.

Has anyone had experience with similar urls who can shed some light on this recurring problem and present solution?

Answer 1

In Webmaster Tools, Google will tell you where they found the URL.

• In the error report, click on the URL
• In the resulting popup, click on the "Linked From" tab

Google will then display the URL of the page that contains that link.

---

Unfortunately, there is no way to prevent these junk URLs from appearing as 404. They come from a variety of sources:

• Scraper sites that steal your content and modify slightly, corrupting the links
• Search engine result pages that truncate URLs resulting in error URLs ending in "..."
• Googlebot's lousy interpretation of JavaScript where it assumes that many of the Strings in the source code look enough like links that it should crawl them

---

Here is what Google's John Mueller has to say about these errors:

1. 404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html
2. In some cases, crawl errors may come from a legitimate structural issue within your website or CMS. How you tell? Double-check the origin of the crawl error. If there's a broken link on your site, in your page's static HTML, then that's always worth fixing. (thanks +Martino Mosna)
3. What about the funky URLs that are “clearly broken?” When our algorithms like your site, they may try to find more great content on it, for example by trying to discover new URLs in JavaScript. If we try those “URLs” and find a 404, that’s great and expected. We just don’t want to miss anything important (insert overly-attached Googlebot meme here). http://support.google.com/webmasters/bin/answer.py?answer=1154698
4. You don’t need to fix crawl errors in Webmaster Tools. The “mark as fixed” feature is only to help you, if you want to keep track of your progress there; it does not change anything in our web-search pipeline, so feel free to ignore it if you don’t need it. http://support.google.com/webmasters/bin/answer.py?answer=2467403
5. We list crawl errors in Webmaster Tools by priority, which is based on several factors. If the first page of crawl errors is clearly irrelevant, you probably won’t find important crawl errors on further pages. http://googlewebmastercentral.blogspot.ch/2012/03/crawl-errors-next-generation.html
6. There’s no need to “fix” crawl errors on your website. Finding 404’s is normal and expected of a healthy, well-configured website. If you have an equivalent new URL, then redirecting to it is a good practice. Otherwise, you should not create fake content, you should not redirect to your homepage, you shouldn’t robots.txt disallow those URLs -- all of these things make it harder for us to recognize your site’s structure and process it properly. We call these “soft 404” errors. http://support.google.com/webmasters/bin/answer.py?answer=181708
7. Obviously - if these crawl errors are showing up for URLs that you care about, perhaps URLs in your Sitemap file, then that’s something you should take action on immediately. If Googlebot can’t crawl your important URLs, then they may get dropped from our search results, and users might not be able to access them either.

Answer 2

Webmaster tools tell you where they are linked from. So you can check it out.

If they are linked from your website, then you must fix them. Otherwise Google doesn't care about them and you can't get any penalty.