Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do not want accessed. While I am not looking to get into a debate about that I do have a question:

If I have a structure like this:

/Folder/
    /Sub-Folder 1/
    /Sub-Folder 2/
    (Note: There are no pages within /Folder/ only other folders.)

If I Disallow: /Folder/, it will prevent "good" robots from accessing the directory and any contents within the sub-folders.

While we know that bad robots will see the /Folder/ will they be able to see and access the sub-folders and the pages within the subfolders if they are not listed in the robots.txt?

(Note: I do not fully understand how good or bad robots crawl a site beyond using a robots.txt and the links within the site.)

share|improve this question

2 Answers 2

up vote 1 down vote accepted

It depends. Pull up that URL in your browser. Do you see those folders listed? If so, they can find them.

To prevent these folders from being listed just add an htaccess file in that directory with this line in it:

Options -Indexes

This will prevent the listing of that directory's contents from appearing.

share|improve this answer
    
Thanks for the info, unfortunatly I use a hosted CMS and do not have access to htaccess but when I type in the address mysite.com/folder/ I get a 404 error on my site so I hope this means they cannot see them. –  Lynda Jun 28 '12 at 19:53
    
If you can't see it, they can't see it. –  John Conde Jun 28 '12 at 20:03

robots.txt only gives instructions, it doesn't enforces any kind of security.
It exist in order to tell search engines what should/shouldn't be listed, nothing more.

Lets say you have and admin.php script located under sub-folder1. A bad bot could "guess" and try to load it. If isn't password protected, you are in problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.