I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt.

As per the following google group post, the user specifies that by setting file permission to 640, it's possible to deny access to robots.txt file by the world but still enable search engine robots to read them.

Is that true? If not how it's possible to deny general public access to robots.txt but still allow Search engine robots to read them.


As long as your robots.txt is accessible by search engines on your server, any user can directly access it by typing its address.

You could block its access via .htaccess using specific user-agent filters, though, namely authorizing only search engines user agents to access it.

As long as no one is fooling around with a fake user agent to blind guess your robots.txt file, you should be safe.

The real question is: why don't you want your users to see it?

If any informaton is too important to be visible on the web, it should not be published online (unless behind a password-protected wall).

  • Hi,could u pls give an link where I can read more about user-agent filters authorizing only SE user agents to access it. – dkjain Dec 15 '12 at 2:27
  • Here are two links: StackOverflow AskApache – Alex Dec 16 '12 at 12:05

No, it's not true. Whether a bot or a person is accessing your site they all go through the webserver software (Apache for example) which accesses the robots.txt file in the same way for every request. It will either always be able to read it, or never. If it's never then something's probably configured wrong.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.