Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

My question is about definition of Crawl errors in Google Webmaster Tools. Crawl errors is devided into few sections.

Let's first consider HTTP section.

I assume that all broken links in this section was somehow found by crawler, this is not the links from sitemap. If all this links was found by scanning pages from sitemap for links, why it doesn't mention what was the source page, like in sitemap section with column Linked From. Please correct me if I am wrong.

Sitemap section.

Looks like all those links came from my sitemap. But there is Linked From column, I already know, that all those broken links is from sitemap, so in order to fix the error, I should revise my sitemap. Am I wrong?

Not followed section.

I don't know what does it mean. Looks like it accumulates all links that caused redirect, but for some reason Google considers all those redirect as wrong redirect. Do you know if there are any set of rules how to determine wrong redirect. Actually I found were was my mistake, I tried to normalize URL and redirect it to the right URL, but I did normalization in a wrong way.

Not found section.

This section like HTTP section but with 404 errors. This section has Linked From column. But very often Linked From has unavailable. What does it mean, Google can not say me how it found this non existing page. How this section related to sitemap section. Does this section contains all 404 links from sitemap too. But there is too many 404 links, much more than in sitemap. I tried to take a look what we have in Linked From, and I saw that this link came from sitemap two month ago. But why Google keeps it indexed, the link is already dead, new sitemap doesn't have it. If there is any expire date for old links?

Unreachable section.

Looks like this section for 500 errors. This section doesn't contain Linked From column. There are too many completely meaningless links, I really don't know where this stuff came from, and without Linked From I am not able to figure out how to deal with it.

Sorry for such a big topic, but I just want to make it clear, what every section stands for, because it's extremely crucial in order to deal with all those problems. Hopefully it will be useful not just for me.

Thanks!

share|improve this question
1  
I looked at the help file for those reports google.com/support/webmasters/bin/… and it answers most of your questions. Those it doesn't answer - is there an expire date? I don't think anyone knows how long before Google gives up on bad links. You could put in a redirect for those pages in the meantime. The meaningless links - we can't help without knowing more information. –  paulmorriss Dec 1 '11 at 16:26
    
i have the same issue. 500 errors on links that dont come from me and have invalid parameters. –  user14392 Mar 31 '12 at 5:47
add comment

1 Answer 1

up vote 3 down vote accepted

I think you are mostly correct with your assumptions.

The first HTTP section shows all 4xx errors apart from 404 errors (which are far more common so get their own page). I get 400 (bad request) errors from CodeIgniter disallowing certain characters in URLs. 403 (forbidden) are here too.

The sitemap section just tells you any URL in a sitemap that can't be found. The "linked from" section is useful so that if a page no longer exists you can remove links to it.

I've never seen the not followed section but it sounds like it lists any URL that's linked to with rel=ofollow.

Not found is every 404 error. It will probably list pages in your sitemap because obviously those should be linked to on your site (in other words, a page only in the sitemap and not linked anywhere would not appear here).

Unreachable is 5xx errors like you said, which are server errors. "Linked from" is not shown here because that doesn't matter, no page should ever return a 5xx error.

share|improve this answer
    
I agree that pages ideally shouldn't return 5xx errors, but sometimes the "linked from" is useful in this context (at least for determining what you're going to do about it) - for example if you're being sent invalid query parameters from a dropdown box, you might want confirm that these aren't coming from your site. –  Zhaph - Ben Duguid Dec 2 '11 at 10:51
    
Not sure how I can explain this any better, but there is no "ideally", 5xx errors should never happen. It's a server problem, nothing to do with who's linking to you or what parameters they send. –  DisgruntledGoat Dec 2 '11 at 12:11
    
I'll try and explain myself better then: A 5xx error can be caused by poor programming: If I expect only a fixed set of variables to be sent from to my page, I might (foolishly) not bother handling the situation where the page receives the wrong values (i.e. one's that aren't in my dropdown). If I know that some random site is trying to link to a page without the correct values I can then make a call about how/what to fix - and that fix might be to return a 404 or 403, or I could decide that I have higher priority things to fix and this dodgy link can be ignored for now. –  Zhaph - Ben Duguid Dec 2 '11 at 12:32
    
Incorrect parameters would produce PHP errors (if you're using PHP) which aren't server errors. Usually the page still returns 200 (OK) in those instances AFAIK. Perhaps certain frameworks like .net output 500 errors but they shouldn't really. –  DisgruntledGoat Dec 2 '11 at 18:04
    
Yep, it was more .NET I was thinking of where an exception in the code will result in an internal server error. I'd not be happy if that resulted in a "200 Success" though: "Response status codes beginning with the digit "5" indicate cases in which the server is aware that it has encountered an error or is otherwise incapable of performing the request. [T]he server should include an [...] explanation of the error situation[...]. [U]ser agents should display any included entity to the user. These response codes are applicable to any request method." bit.ly/t9zYJb –  Zhaph - Ben Duguid Dec 2 '11 at 22:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.