Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue.

But I also use Google Page Speed test, site is working correctly than getting 91/100 score. how can I fix it?

share|improve this question
Presumably your robots.txt is returned OK if requested from the browser? – w3dk Aug 19 '13 at 18:40

Check if your robots.txt bear any empty Disallow like below:

 User-agent: *
 Disallow: /cgi-bin/ 

 Sitemap: http://www.yoursite.com/sitemap.xml

It should be updated to:

 User-agent: * 
 Disallow: /cgi-bin/ 

 Sitemap: http://www.yoursite.com/sitemap.xml
share|improve this answer
An empty Disallow field should simply be ignored by the Google bot (as invalid), it certainly shouldn't result in an "Unreachable" error!? (You should also have a blank line before the Sitemap field, in order to separate it from the preceding group.) – w3dk Aug 19 '13 at 18:43
my problem has solved. – ahmetlutfu Aug 20 '13 at 7:39
@w3d: Googlebot considers an empty Disallow: as invalid?! From the robots.txt specification perspective, it’s a perfectly valid line with the meaning "allow everything". – unor Aug 20 '13 at 14:04
@unor: Well spotted - you're right, it's not invalid and Google doesn't state this. However, Google does formally define the path value as pathvalue = "/" path (which implies it cannot be empty) and later states that "When no path is specified, the directive is ignored" (which will have the same result as allowing). I would add, however, that 99% of the time, an empty Disallow field is in error (or simply unnecessary). – w3dk Aug 20 '13 at 18:54

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.