Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
Blocking Search Engines to Crawl Parts of a Website?

In general, I want my site to be indexed, but there are certain files I don't wish to be searchable from Google, Yahoo, etc...

I know you can block your site from being indexed using a robots file, but what if I only want certain files to not be indexed?

share|improve this question

marked as duplicate by danlefree Dec 26 '11 at 1:36

This question was marked as an exact duplicate of an existing question.

For this you can use robots.txt too. Then you would write several disallow statements. Here are some examples:

Disallow: /tmp
Disallow: /logs

For more information see the specification of the robots file.

But be aware that not every spider respects this file. So maybe you need other methods like e.g. the nofollow meta tag if you really want to be sure.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.