Possible Duplicate:
Blocking Search Engines to Crawl Parts of a Website?

In general, I want my site to be indexed, but there are certain files I don't wish to be searchable from Google, Yahoo, etc...

I know you can block your site from being indexed using a robots file, but what if I only want certain files to not be indexed?


For this you can use robots.txt too. Then you would write several disallow statements. Here are some examples:

Disallow: /tmp
Disallow: /logs

For more information see the specification of the robots file.

But be aware that not every spider respects this file. So maybe you need other methods like e.g. the nofollow meta tag if you really want to be sure.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.