1

We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, site2.example.com, etc. However, because all of these sites live in the same directory, they share a robots.txt file.

I would like to set up a test/demo site on this same codebase(i.e. demo.example.com), but I do not want it to be indexed by search engines. Is there any way that I can configure robots.txt to disallow an entire subdomain while not affecting other subdomains that live in the same physical location?

Improve this question
2
  • 1
    you can't use a robots.txt to do this directly but you could use a rewrite rule to serve a different version of robots.txt base on incoming URL. I have not given his as an answer as I don't know how to make the rewrite rule. someone else will...
    – ollybee
    Apr 19, 2011 at 20:51
  • Thanks for this comment - it pointed me in the right direction. I submitted the rule that is working for me as an answer.
    – Sean Walsh
    Apr 20, 2011 at 17:07

3 Answers 3

Reset to default
3

You can configure the subdomain site to send an additional X-Robots-tag HTTP header, assuming that each subdomain is set up as it's own virtual site.

That will allow you to tell spiders not to index that subdomain. The catch is that I do not know how many crawlers currently support this header. A quick look indicates that Google and Yahoo! do.

Improve this answer
1
  • Thank you for the answer - it is something I will keep in mind for the future, but it's not an option for this situation.
    – Sean Walsh
    Apr 20, 2011 at 17:05
3

I ended up using mod_rewrite to accomplish what I need:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^demo\.example\.com
RewriteRule ^robots\.txt$ robots-demo\.txt

RewriteCond %{HTTP_HOST} !^demo\.example\.com
RewriteRule ^robots\.txt$ robots-othersites\.txt
Improve this answer
1
  • Yep, gotta wait another 19 hours before I can. :)
    – Sean Walsh
    Apr 20, 2011 at 22:15
2

You can use a meta robots tag, which is supported by all of the major search engines today:

<meta name="robots" content="noindex" />

I'm not exactly sure how to do this with robots.txt, though.

Improve this answer
1
  • Thank you for the answer, but unfortunately it won't work in my case. All the sites share the same templates and I do not want to do a conditional each time a page is rendered to check for the subdomain.
    – Sean Walsh
    Apr 20, 2011 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.