0

I want to knew if the following robots.txt file block the whole pages from being indexed or not.

User-agent: *
Disallow:
Disallow: /cgi-bin/
Improve this question

1 Answer 1

Reset to default
3

You can easily test in your Google Webmaster Tools account. But the answer is, no. It does not block your whole site. You block the /cgi-bin/ directory in line three. Line two essentially explicitly allows your entire site to be crawled. See the robots.txt documentation for more on this.

Improve this answer
1
  • 2
    Just to add, line two (Disallow: ) is not required (the default action is to allow).
    – MrWhite
    Commented Apr 22, 2015 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.