I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do not want accessed. While I am not looking to get into a debate about that I do have a question:

If I have a structure like this:

    /Sub-Folder 1/
    /Sub-Folder 2/
    (Note: There are no pages within /Folder/ only other folders.)

If I Disallow: /Folder/, it will prevent "good" robots from accessing the directory and any contents within the sub-folders.

While we know that bad robots will see the /Folder/ will they be able to see and access the sub-folders and the pages within the subfolders if they are not listed in the robots.txt?

(Note: I do not fully understand how good or bad robots crawl a site beyond using a robots.txt and the links within the site.)

2 Answers 2


It depends. Pull up that URL in your browser. Do you see those folders listed? If so, they can find them.

To prevent these folders from being listed just add an htaccess file in that directory with this line in it:

Options -Indexes

This will prevent the listing of that directory's contents from appearing.

  • Thanks for the info, unfortunatly I use a hosted CMS and do not have access to htaccess but when I type in the address mysite.com/folder/ I get a 404 error on my site so I hope this means they cannot see them.
    – L84
    Commented Jun 28, 2012 at 19:53
  • If you can't see it, they can't see it.
    – John Conde
    Commented Jun 28, 2012 at 20:03

robots.txt only gives instructions, it doesn't enforces any kind of security.
It exist in order to tell search engines what should/shouldn't be listed, nothing more.

Lets say you have and admin.php script located under sub-folder1. A bad bot could "guess" and try to load it. If isn't password protected, you are in problems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.