My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (mostly development), and are usually deleted once they outlive their use.
The setup is as follows
public_html / maindomain_com subdomain1_maindomain_com subdomain2_maindomain_com . . . subdomain30_maindomain_com subdomain31_maindomain_com . .
Recently it has come to our attention that some of the search engines have indexed some of our defunct and some of our active subdomains.
What would be the best way of preventing search engines and crawlers from crawling and indexing the main site and its subdomains?
I am aware that it's possible to create a
robots.txt file with
noindex, nofollow, etc directives, but I'd like to avoid it, if possible (going through ~30 subdomains, setting up a new
robots.txt when a new subdomain is created, etc).
I'm also aware that it's possible to use:
<IfModule mod_headers.c> Header set X-Robots-Tag "noindex, noarchive, nosnippet, nofollow, noodp, noydir" </IfModule>
.htaccess. What I'm unsure of is whether I could place this directive inside of main domain's
.htaccess, in hopes of preventing crawling for both the main domain and its subdomains.
Is there a better way of doing this? I've come across a StackOverflow answer that
Alias can be used in
robots.txt. Can this be used for
.htaccess as well?