6

We make use of the robots.txt file to prevent Google (and other search spiders) from crawling certain pages/directories in our domain. Some of these directories/files are secret, meaning they aren't linked (except perhaps on other pages encompassed by the robots.txt file). Some of these directories/files aren't secret, we just don't want them indexed.

If somebody browses directly to www.mydomain.com/robots.txt, they can see the contents of the robots.txt file. From a security standpoint, this is not something we want publicly available to anybody. Any directories that contain secure information are set behind authentication, but we still don't want them to be discoverable unless the user specifically knows about them.

Is there a way to provide a robots.txt file but to have it's presence masked by John Doe accessing it from his browser? Perhaps by using PHP to generate the document based on certain criteria? Perhaps something I'm not thinking of? We'd prefer a way to centrally do it (meaning a <meta> tag solution is less than ideal).

Improve this question
3
  • 2
    Horrible idea. This is security through obscurity. – Rook Feb 14 '11 at 22:12
  • 1
    Security through obscurity would be the equivalent of having "secret" directories that "no one knows about" and that's all the security you have in place. All of our "secret" directories are locked by authentication and are completely secure as they stand. We just wish to mask the directories as well as to not be seen when looking directly at the robots.txt file. There is nothing "horrible" about that idea. – Michael Irigoyen Feb 14 '11 at 22:16
  • 2
    Having unique user names and passwords controlled by LDAP and a stringent password change policy used in conjunction with a controlled access list is security through obscurity how exactly? And what does this question even have to do with SQL injections? Perhaps you need to step back, breathe, and understand not everyone on the internet isn't as enlightened as you seem to be. It's as if you have a personal vendetta against this simple question, which is quite immature and very unprofessional. I'm willing to listen if you're willing to communicate professionally. Otherwise, just please stop. – Michael Irigoyen Feb 14 '11 at 23:04
13

If you don't want something visible to the web, then don't make it visible. In your case, using robots.txt for excluding directories is security by non-obscurity. Instead of publicly saying "Hey, there's the place with all the precious jewels and valuable metals, don't go in there!", just say nothing and simply do not advertise the presence of those directories at all.

This take some discipline on your part, making sure that none of those directories are referred to in any way on your publicly accessible pages.

Otherwise it's impossible to do so via robots.txt. If you make that a dynamically generated page based on the visitor, then what's to stop someone from setting their browser's user agent string to "Googlebot" and getting the full list of your excluded directories? There's no 100% reliable method of detecting who's at the other end of a connection - at best you can guess and hope for the best. And guessing just doesn't cut it when it comes to security.

Password protected pages/direcotries can't be crawled, so those would be safe even if they're not explictly listed in the robots.txt for exclusion - Googlebot and its brethren won't have login credentials, so can't get into the pages to spider them.

Improve this answer
2
  • You answer has merit. However, as I stated in my question, all "secured" directories are secured by authentication. I'm in no way practicing security through obscurity. That being said, we do not use .htaccess for authentication. Our organization uses a centralized login system (phpCAS to be specific). So, even though Google couldn't get past the login page, it would still index the fact that there is a login page in that directory, thus someone would be able to determine that the directory exists. – Michael Irigoyen Feb 14 '11 at 22:24
  • I'm accepting this answer because it contains several good and well explained points that leads to the conclusion that it is not technically possible to hide the contents of the robots.txt file from everyone except the spiders. – Michael Irigoyen Feb 14 '11 at 23:08
5

If you don't want robots crawling those directories but don't want to announce them in your robots.txt file use the x-robots-tag HTTP header to block them.

Put this in a .htaccess file in any directory you don't want indexed:

Header set x-robots-tag: noindex

That will tell robots to ignore the contents of the files in that directory. This way no robots.txt file is necessary and you have your security through obscurity and a legitimate way of telling the search engines to stay out.

Improve this answer
0

You could use mod rewrite to redirect robots.txt to a php script, and check the user agent, but it could be bypassed by faking the user agent.

You have nothing that differs a normal user from a search engine, except for the information that it sends, and all those can be faked.

Improve this answer
0

You can check the User Agent HTTP Header. It isn't fool proof, but you could tell from that if the user is using Firefox, Explorer, etc. Of course, the user could use curl and get around that.

Improve this answer
2
  • 1
    Then you can use the user-agent switcher firefox plugin to bypass this terrible excuse for a security system. – Rook Feb 14 '11 at 22:13
  • I agree. User agents are not the way to go as they can be spoofed by a variety of methods. Now, if spiders and crawlers passed other data that was non-spoofable, this idea might have merit. – Michael Irigoyen Feb 14 '11 at 22:21
0

robots.txt is about as good an idea as favicon.ico. But it's the only standard, so you have to work within it.

You can put all your restricted pages in a single heirarchy (e.g., /restricted containing /restricted/login etc.) and name just the root of the hierarchy in robots.txt. A user can find out that some pages are there, but won't know what the pages actually are.

This could be a subdomain with Disallow: /.

Another thing you could try is not listing your restricted pages, but instead detecting user agents/IPs accessing robots.txt (which only web crawlers should do) and later denying them access to the pages.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.