Possible Duplicate:
How to restrict the download of all files in a folder?

I want to make a private file available for download but not use username/password protection. I want to put the file into a directory called something like download. How can I ensure:

  • the file does not become part of search engine results, and
  • the file cannot be accessed by bots that might guess the directory name?

First of all you can add a proper robots.txt entry to tell willing bots to ignore and not index your specific directory.

Bots should follow your robots.txt intentions, but some people (or bots) will read them on purpose to get to know what you want to hide.

So you will have to make sure your webserver does not automatically create an index of all files in your directory and create hard-to-guess filenames. This is security by obscurity and bad.

When returning the file you could also think of adding the relatively new X-Robots-Tags as a last resort and add directives like "noindex,noarchive". This way google bot even if it finds the file should not index and cache it. But others not knowing the tag will.

Programmatically you could also write code that returns the file for download only if the accessing client is no bot. Problem here: Bot-detection is a never ending story and could be faked - as almost everything can what comes with the HTTP header.

Maybe you can limit the access by IP address ranges as another measure to limit the range of clients with access or to exclude known bot ip addresses (but those often change).

  • Do malicious bots comply with robots.txt files? – CJ7 Sep 24 '12 at 9:20
  • As I wrote: no. It's nice to read it to know where to dig further. But if you don't want or can set up proper protection mechanisms and go the security by obscurity path, you have not much options than add at least some different layers. – initall Sep 24 '12 at 9:35
  • Just to add, a disallow directive in robots.txt does not necessarily prevent the file from being indexed - it prevents it from being crawled (by the good bots). If the file is linked to then it could still get indexed (a URL-only link in the SERPs). Also, if you include an X-Robots-Tag response header then you must not disallow in robots.txt. – MrWhite Sep 24 '12 at 9:53
  • @w3d My idea with the X-Robots-Tag is: Let's say you have a wrong disallow statement or deleted your robots.txt file by accident and the bot accesses your file - than again you have a small chance it might follow your directives you send with the headers and ignore it. As CJ7 wants to make it not visible to the public, combining disallow and X-Robots-Tag is perfectly ok. And yes, deeplinks are a problem. This overall approach (no protection by credentials) is a problem. – initall Sep 24 '12 at 10:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.