Block Google (and other) from indexing a domain

Question

We want to open a new domain for certain purposes (call them PR). The thing is we want the domain to point to the same website we currently have.

We do not want this new domain to appear on search engines (specifically Google) at all.

Options we've ruled out:

• Robots.txt can't be used - it will work the same on both domains, which isn't what we want.
• The rel=canonical doesn't block - only suggests to index a similar page instead. The original page might end up being indexed.

Is there a way to handle this?

Answer 1

If you simply want our site accessible at the new domain, and it is acceptable that the URL reverts to the old domain upon navigating to the site, then your need will be satisfied by using 301 redirects to map the new domain to your existing site.

Otherwise, you can serve a special robots.txt file only for your new domain. To do that, place something like this in your htaccess file:

RewriteCond %{HTTP_HOST} ^yournewdomain\.com$
RewriteCond %{REQUEST_URI} ^/?robots.txt$
RewriteRule .* /robots-yournewdomain.txt [L]

Answer 2

rel=canonical is just a suggestion, but it is a strong suggestion. Google would certainly honor it in a situation like this. Other search engines might not do so very well. I use rel=canonical to keep development servers from being indexed even when they are publicly accessible. Googlebot will still crawl the entire site on my development server, but it won't actually index it.

For anything else to work, you will need to be able to have some difference in the code between the two sites. You could serve two different robots.txt files. You could use .htaccess and mod_rewrite to choose the correct file for the correct domain. Here is a blog entry that walks you through the mechanics of doing so: http://blog.cherouvim.com/robotstxt-control-for-host-aliases-via-mod_rewrite/

Alternately you could put a meta noindex tag on the pages of the new site. I don't know what software powers your site, but it could be possible to modify it so that you just noindex just the new site. For example, if you site is powered by PHP, you could put the meta noindex inside a statement like

if ($_SERVER['HTTP_HOST'] == 'new.example.com'){