I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do not want accessed. While I am not looking to get into a debate about that I do have a question:
If I have a structure like this:
(Note: There are no pages within /Folder/ only other folders.)
Disallow: /Folder/, it will prevent "good" robots from accessing the directory and any contents within the sub-folders.
While we know that bad robots will see the /Folder/ will they be able to see and access the sub-folders and the pages within the subfolders if they are not listed in the robots.txt?
(Note: I do not fully understand how good or bad robots crawl a site beyond using a robots.txt and the links within the site.)