Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Majority of the links in the screenshot below are valid! Such as:



And so on... Even if I tried "Fetch as Googlebot" the result is 200 OK.

enter image description here

Please note: I do not block any content for google bot in my robots.txt, neither in my HTML meta tags.

share|improve this question

migrated from stackoverflow.com Jul 28 '11 at 17:28

This question came from our site for professional and enthusiast programmers.

I think this belongs on Webmasters.Stackexchange.com – Alan Jul 27 '11 at 4:53
Look at your server logs and see if the Googlebot visited on that date/time and if your server really issued a 500 error. At least then you'll know it's legitimate and not some weird Webmaster Tools problem. – Sparky Jul 27 '11 at 4:57
I guess NOT, actually ALL my real blogs whose link format is similar to this: wayneye.com/Blog/[Blog-Title] Are not indexed by Google, you could search google "site:wayneye.com", you will see it never indexed my real blog links... BTW, all those links will return HTTP 301 to redirect to wayneye.com/ViewBlog.aspx?BlogID=xx But many of my links that indexed by google also return HTTP 301, while they were indexed. I was really confused, please help! – Wayne Ye Jul 29 '11 at 6:58
What does any of that have to do with my suggestion that you should look at your server logs? – Sparky Jul 29 '11 at 15:08

Possible causes for the 500 error you're seeing:

  1. Your robots.txt file starts with an illegal character. When validated here, the file contains a character that's hidden when viewed as UTF-8 in a text editor such as TextMate, but visible when viewed as ISO-8859-1 (see image below). To fix this, delete the first line, retype it, save the file, and revalidate it.

    enter image description here

  2. Your site may have simply returned a 500 error when Google last crawled it. (i.e. The server was down.) If this is the case, the errors you should not be repeated when Google next crawls your site, assuming it stays up.

share|improve this answer
Thanks a lot for the answer! I fixed robots.txt, however, my service provider does not allow me to view the IIS log.... Since ALL my blog posts that URL like wayneye.com/Blog/[Blog_Title] does NOT indexed by google, so I don't think it is because of "server down", instead, I strongly suspect there are something wrong however I couldn't find out... – Wayne Ye Aug 3 '11 at 2:15
Even I tried "Fetch as google bot" in Webmaster tool, it returned OK for wayneye.com/Blog/[Blog_Title] – Wayne Ye Aug 3 '11 at 2:20

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.