robots.txt ignored

Question

I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories.

By default it had Disallow: /images/ and the rule works, can't access the folder. I tried to add Disallow: /images/thumbnails and that one didn't work. I commented out Disallow: /images/ just to test. The folder IS STILL inaccessible. There is no other instance of robots.txt on my domain.

Why are my changes on robots.txt being ignored ? Is there some other file that could override it ?

Answer 1

The file robots.txt will only work if it is located in the root folder of your web site and not in any sub folder.

Having said that robots.txt is only for search engine robots and has no effect on preventing people browsing folders. If people are able to do that on your server then it is poorly configured and should be set to disallow indexes. This can be done either in an .htaccess file or preferably in the apache httpd.conf file

Answer 2

Robots.txt is not used for preventing users to browse directories. Directory browsing is handled by the web server settings.

All robots.txt does is asking search engines to not index the content within the directories that you specify.