I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing
As per the following google group post, the user specifies that by setting file permission to 640, it's possible to deny access to robots.txt file by the world but still enable search engine robots to read them.
Is that true? If not how it's possible to deny general public access to robots.txt but still allow Search engine robots to read them.