Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, site2.example.com, etc. However, because all of these sites live in the same directory, they share a robots.txt file.

I would like to set up a test/demo site on this same codebase(i.e. demo.example.com), but I do not want it to be indexed by search engines. Is there any way that I can configure robots.txt to disallow an entire subdomain while not affecting other subdomains that live in the same physical location?

share|improve this question
you can't use a robots.txt to do this directly but you could use a rewrite rule to serve a different version of robots.txt base on incoming URL. I have not given his as an answer as I don't know how to make the rewrite rule. someone else will... – ollybee Apr 19 '11 at 20:51
Thanks for this comment - it pointed me in the right direction. I submitted the rule that is working for me as an answer. – Sean Walsh Apr 20 '11 at 17:07
Make sure to set your answer as the accepted answer, so people know you're set! :) – Jacob Hume Apr 20 '11 at 19:15
@Jacob - Yah, Stack Exchange won't allow me to accept my own answer for another 22 hours. Thanks, though! – Sean Walsh Apr 20 '11 at 19:35

You can configure the subdomain site to send an additional X-Robots-tag HTTP header, assuming that each subdomain is set up as it's own virtual site.

That will allow you to tell spiders not to index that subdomain. The catch is that I do not know how many crawlers currently support this header. A quick look indicates that Google and Yahoo! do.

share|improve this answer
Thank you for the answer - it is something I will keep in mind for the future, but it's not an option for this situation. – Sean Walsh Apr 20 '11 at 17:05
up vote 3 down vote accepted

I ended up using mod_rewrite to accomplish what I need:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^demo\.example\.com
RewriteRule ^robots\.txt$ robots-demo\.txt

RewriteCond %{HTTP_HOST} !^demo\.example\.com
RewriteRule ^robots\.txt$ robots-othersites\.txt
share|improve this answer
Oh, by the way, you can accept your own answer if it solved your problem. – mc10 Apr 20 '11 at 21:03
Yep, gotta wait another 19 hours before I can. :) – Sean Walsh Apr 20 '11 at 22:15

You can use a meta robots tag, which is supported by all of the major search engines today:

<meta name="robots" content="noindex" />

I'm not exactly sure how to do this with robots.txt, though.

share|improve this answer
Thank you for the answer, but unfortunately it won't work in my case. All the sites share the same templates and I do not want to do a conditional each time a page is rendered to check for the subdomain. – Sean Walsh Apr 20 '11 at 17:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.