Different robots.txt for two different domains point to same folder

Question

I have the following two domains:

example.com

test.example.com

Both point to same folder which is public_html. What I want is a different robots.txt file for each domain. So when someone browse example.com/robots.txt then a different file is shown. And when someone go to test.example.com/robots.txt then a different file is shown.

How can I do this using URL rewriting in .htaccess?

Answer 1

Remeber that no one from Internet can see your directory tree, so stackoverflow.com/ and test.stackoverflow.com are completely diffrent sites for us and search robots.

You can do that by checking the HTTP host in query:

RewriteCond %{HTTP_HOST} ^test
RewriteRule robots.txt someotherrobots.txt

Answer 2

Expanding on the earlier answer, you could use a single mod_rewrite rule to handle different robots.txt files for any number of subdomains.

For example:

RewriteEngine On

# Rewrite "robots.txt" to "robots-<subdomain>".txt
RewriteCond %{HTTP_HOST} ^(?!www)([a-z]+)\.example\.com
RewriteRule ^robots\.txt$ robots-%1.txt [L]

Any request for <subdomain>.example.com/robots.txt (except the www subdomain, as stated using the negative lookahead (?!www)) would be served by the corresponding /robots-<subdomain>.txt file. The www subdomain and the domain apex (ie. example.com) would be served by the default robots.txt file.

You could take this a step further and have the subdomain's robots-<subdomain>.txt file served "if it exists" and fallback to the default robots.txt file if it doesn't, by adding a filesystem check.

For example:

# Rewrite "robots.txt" to "robots-<subdomain>".txt if it exists
RewriteCond %{HTTP_HOST} ^(?!www)([a-z]+)\.example\.com
RewriteCond %{DOCUMENT_ROOT}/robots-%1.txt -f
RewriteRule ^robots\.txt$ robots-%1.txt [L]

Prevent direct access to robots-<subdomain>.txt

To prevent direct access to the /robots-<subdomain>.txt file you can add an external redirect before the above rewrite. For example:

# Redirect direct requests to "robots-<subdomain>.txt" to "robots.txt" on <subdomain>
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^robots-([a-z]+)\.txt$ https://$1.example.com/robots.txt [R=301,L]

The above would redirect a direct request for /robots-<subdomain>.txt (regardless of the hostname being requested) to https://<subdomain>.example.com/robots.txt. The check against the REDIRECT_STATUS env var ensures that only direct requests are redirected and not internally rewritten requests to /robots-<subdomain>.txt. Note that this uses the filename to determine the correct subdomain. eg. https://foo.example.com/robots-bar.txt would redirect to https://bar.example.com/robots.txt - changing the hostname, but ensures the same file is served.

To simply redirect to /robots.txt and keep the requested hostname. eg. https://foo.example.com/robots-bar.txt redirects to https://foo.example.com/robots.txt (which will result in robots-foo.txt being served by the above rewrite). Then the redirect can be simplified to:

# Redirect direct requests to "robots-<subdomain>.txt" to "robots.txt" on same host
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^robots-[a-z]+\.txt$ /robots.txt [R=301,L]

Summary

RewriteEngine On

# Redirect direct requests to "robots-<subdomain>.txt" to "robots.txt" on <subdomain>
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^robots-([a-z]+)\.txt$ https://$1.example.com/robots.txt [R=301,L]

# Rewrite "robots.txt" to "robots-<subdomain>".txt
RewriteCond %{HTTP_HOST} ^(?!www)([a-z]+)\.example\.com
RewriteRule ^robots\.txt$ robots-%1.txt [L]