3

Google's Robots meta tag and X-Robots-Tag HTTP header specifications page (Practical implementation of X-Robots-Tag with Apache section) gives an example of noindexing certain filetypes, ie:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

But I can't find any explanation on how to apply an X-Robots-Tag directive to all subdomains, save certain exceptions, eg docs.example.com.

PS - In my case there is only one root folder (using WordPress multisite, which shares file structure across the network).

  • Do all your subdomains point to the same directory on the filesystem? ie. do they all point to the document root of the main domain? – MrWhite Sep 25 '18 at 0:47
  • 1
    Yes, this is a WordPress multisite installation so the file system is all the same! Thanks for asking good question. I will update my question accordingly. – Drewdavid Sep 25 '18 at 1:55
2

Since all the subdomains point to the same place on the filesystem then you can do something like the following at the top of your .htaccess file in the document root:

SetEnvIf Host ^docs\. INDEX
SetEnvIf Host ^foo\. INDEX
Header set X-Robots-Tag "noindex, nofollow" env=!INDEX

This uses mod_setenvif to conditionally set an environment variable INDEX for the subdomains you wish to allow indexing, ie. that you do not want to set the X-Robots-Tag header. The subdomains are identified by checking the Host HTTP request header. The noindex X-Robots-Tag header is then only set if this environment variable is not set.

Note that this will currently set the noindex X-Robots-Tag on the domain apex (ie. example.com) and www subdomains as well. To exclude the main domain then add another exception:

SetEnvIf Host ^(www\.)?example\.com INDEX
-1
<IfModule mod_headers.c>
 <IfModule mod_setenvif.c>
    SetEnvIf Request_URI "^doc\." x_tag=yes
    Header set X-Robots-Tag "noindex, nofollow" env=x_tag
 </IfModule>
</IfModule>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.