Remove third/nth level domains from google Index

Question

Somehow google has indexed some third(and fourth!) level domains that I had attached to my server temporarily, eg. my.domain.root.com. I now have these redirected properly where I would like them to go, however with a carefully crafted search one can still find them and I'd rather they not be exposed. My google foo skills have failed me in finding an answer, so I come to you wonderful folks: Is there a way/How do I remove sub-level domains from google search results?

I have the site in google webmaster tools and verified, but all the URL removal requests I can perform append the url to the base url, not prefixed.

And finally, how can I prevent this in the future?

Answer 1

You mention that the domains are now redirecting to their correct destinations. If the redirect in place is a permanent 301 redirect, then the source URLs will eventually drop out of Google's Index.

But if you want to stop it happening in future without having to redirect, then take John's advice.

Answer 2

You can use the X-Robots-Tag HTTP header. This prevents pages from being in a search engine's index when you set the value to "noindex".

In Apache you could put this in your conf file or .htaccess file in the root directory of the subdomains you do not want indexed:

Header set X-Robots-Tag "noindex"

Answer 3

To remove the subdomains from Google via Google's removal tool you need to add them exactly to Webmaster Tools as separate sites, it is not enough to just add the root domain as the site removal wont work (as you have found out).

Answer 4

Just put a "go away" robots.txt file in each virtual server, using an Alias if you have a shared docroot.

In text file goaway-robots.txt put:

User-agent: *
Disallow: /

...then in each virtual server section add:

Alias /robots.txt %{DOCUMENT_ROOT}/goaway-robots.txt

You can still have a more Google-friendly default robots.txt on your main site.

For more information about using robots.txt, look here: http://www.robotstxt.org/robotstxt.html.

For more information about Apache's "Alias" directive, look here: http://httpd.apache.org/docs/current/mod/mod_alias.html#alias.