1

My robots.txt is the typical that allows all bots everywhere

User-agent: *
Disallow: /

When I test it with Google tool everything is OK.

robots.txt no errors

But in Google Search Console I get an error that my site can't be accessed.

robots.txt unable to access page

Can somebody help?

|improve this question
  • 3
    User-agent: * Disallow: / does not "allow all bots everywhere". It blocks all bots from accessing your home page. – Maximillian Laumeister Dec 5 '18 at 0:06
  • 3
    More than just the home page. It Blocks the whole site. – Stephen Ostermiller♦ Dec 5 '18 at 1:53
  • OMG i was so tired that i put disallow instead of allow. Thank you If i let allow / disallow line empty ,is it ok ? I think so. – user2016496 Dec 5 '18 at 10:38
1

When I test it with Google tool everything is ok.

The tool is simply telling you that the file is syntactically valid. Whether it blocks (or allows) traffic is another matter.

But in Google Search Console I get an error that my site can't be accessed.

User-agent: *
Disallow: /

As already mentioned in comments, this blocks everything and allows nothing! To allow everything (the default action) you should remove the / (slash) on the Disallow: line:

 User-agent: *
 Disallow:

Or simply remove all the directives (or even the file itself).

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.