Majority of the links in the screenshot below are valid! Such as:



And so on... Even if I tried "Fetch as Googlebot" the result is 200 OK.

enter image description here

Please note: I do not block any content for google bot in my robots.txt, neither in my HTML meta tags.

  • I think this belongs on Webmasters.Stackexchange.com
    – Alan
    Jul 27, 2011 at 4:53
  • 1
    Look at your server logs and see if the Googlebot visited on that date/time and if your server really issued a 500 error. At least then you'll know it's legitimate and not some weird Webmaster Tools problem.
    – Sparky
    Jul 27, 2011 at 4:57
  • I guess NOT, actually ALL my real blogs whose link format is similar to this: wayneye.com/Blog/[Blog-Title] Are not indexed by Google, you could search google "site:wayneye.com", you will see it never indexed my real blog links... BTW, all those links will return HTTP 301 to redirect to wayneye.com/ViewBlog.aspx?BlogID=xx But many of my links that indexed by google also return HTTP 301, while they were indexed. I was really confused, please help!
    – Wayne Ye
    Jul 29, 2011 at 6:58
  • What does any of that have to do with my suggestion that you should look at your server logs?
    – Sparky
    Jul 29, 2011 at 15:08

1 Answer 1


Possible causes for the 500 error you're seeing:

  1. Your robots.txt file starts with an illegal character. When validated here, the file contains a character that's hidden when viewed as UTF-8 in a text editor such as TextMate, but visible when viewed as ISO-8859-1 (see image below). To fix this, delete the first line, retype it, save the file, and revalidate it.

    enter image description here

  2. Your site may have simply returned a 500 error when Google last crawled it. (i.e. The server was down.) If this is the case, the errors you should not be repeated when Google next crawls your site, assuming it stays up.

  • Thanks a lot for the answer! I fixed robots.txt, however, my service provider does not allow me to view the IIS log.... Since ALL my blog posts that URL like wayneye.com/Blog/[Blog_Title] does NOT indexed by google, so I don't think it is because of "server down", instead, I strongly suspect there are something wrong however I couldn't find out...
    – Wayne Ye
    Aug 3, 2011 at 2:15
  • Even I tried "Fetch as google bot" in Webmaster tool, it returned OK for wayneye.com/Blog/[Blog_Title]
    – Wayne Ye
    Aug 3, 2011 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.