Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue.

But I also use Google Page Speed test, site is working correctly than getting 91/100 score. how can I fix it?

share|improve this question
4  
Presumably your robots.txt is returned OK if requested from the browser? –  w3d Aug 19 '13 at 18:40

1 Answer 1

Check if your robots.txt bear any empty Disallow like below:

 User-agent: *
 Disallow:
 Disallow: /cgi-bin/ 

 Sitemap: http://www.yoursite.com/sitemap.xml

It should be updated to:

 User-agent: * 
 Disallow: /cgi-bin/ 

 Sitemap: http://www.yoursite.com/sitemap.xml
share|improve this answer
    
An empty Disallow field should simply be ignored by the Google bot (as invalid), it certainly shouldn't result in an "Unreachable" error!? (You should also have a blank line before the Sitemap field, in order to separate it from the preceding group.) –  w3d Aug 19 '13 at 18:43
    
my problem has solved. –  ahmetlutfu Aug 20 '13 at 7:39
1  
@w3d: Googlebot considers an empty Disallow: as invalid?! From the robots.txt specification perspective, it’s a perfectly valid line with the meaning "allow everything". –  unor Aug 20 '13 at 14:04
1  
@unor: Well spotted - you're right, it's not invalid and Google doesn't state this. However, Google does formally define the path value as pathvalue = "/" path (which implies it cannot be empty) and later states that "When no path is specified, the directive is ignored" (which will have the same result as allowing). I would add, however, that 99% of the time, an empty Disallow field is in error (or simply unnecessary). –  w3d Aug 20 '13 at 18:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.