My company is renting a CentOS server, with one main domain and ~30 subdomains (the number varies, but it's usually around 30) hosted on it. The subdomains themselves are used for various purposes (mostly development), and are usually deleted once they outlive their use.

The setup is as follows

public_html / 

Recently it has come to our attention that some of the search engines have indexed some of our defunct and some of our active subdomains.

What would be the best way of preventing search engines and crawlers from crawling and indexing the main site and its subdomains?

I am aware that it's possible to create a robots.txt file with noindex, nofollow, etc directives, but I'd like to avoid it, if possible (going through ~30 subdomains, setting up a new robots.txt when a new subdomain is created, etc).

I'm also aware that it's possible to use:

<IfModule mod_headers.c>
  Header set X-Robots-Tag "noindex, noarchive, nosnippet, nofollow, noodp, noydir"

in .htaccess. What I'm unsure of is whether I could place this directive inside of main domain's .htaccess, in hopes of preventing crawling for both the main domain and its subdomains.

Is there a better way of doing this? I've come across a StackOverflow answer that Alias can be used in httpd.conf for robots.txt. Can this be used for .htaccess as well?

  • Crawling and indexing are two different things. It is usually impossible to totally prevent both. You can either prevent all crawling and accept that a URL or two occasionally gets indexed, or you can allow crawling and prevent indexing. Feb 12, 2021 at 8:34
  • Robots.txt doesn't support nofollow. Google used to support noindex in robots.txt but no longer does. The only relevant robots.txt directive that is available is Disallow:. Feb 12, 2021 at 8:38
  • Why do you want a .htaccess solution rather than something for httpd.conf? Feb 12, 2021 at 11:28
  • @StephenOstermiller Thank you for the first two remarks, I'll keep them in mind. As for the .htaccess solution, it seemed to be the easiest to implement in case a crawlable domain is added to the server. But in this case, it's only one main domain, and xy subdomains. Feb 12, 2021 at 11:54
  • Do all of these subdomains have the same document root are are you serving them out of different directories? Feb 12, 2021 at 12:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.