While developing, each instance lives on client.dev.example.com
Any "polite" bot will request the robots.txt
file at http://client.dev.example.com/robots.txt
. So this request must serve the necessary response. There is nothing you can do in a "parent" robots.txt
file to influence other domains/subdomains, since it's simply never requested for this hostname.
If all your subdomains actually point to the same area of the filesystem then you could simply serve the same robots.txt
file.
Alternatively, you could have a robots-disallow.txt
file with each site and conditionally serve this when accessing the dev site, based on the hostname in the request. At least this way you can still distribute the same codebase for the live site without alteration.
For example, in each sites .htaccess
file, you could do something like the following near the top:
RewriteEngine On
RewriteCond %{HTTP_HOST} ^[a-z-]+\.dev\.
RewriteRule ^robots\.txt$ robots-disallow.txt [L]
This specifically looks for the .dev.
subdomain (after the client
subdomain) in the request. If found then it internally rewrites any request for robots.txt
to robots-disallow.txt
. Where robots-disallow.txt
consists of something like:
User-agent: *
Disallow: /
You can also have a robots.txt
file which contains the "live" settings and is only served on the live site (when .dev.
does not occur in the request).
If you have access to the server config, then you could potentially do this once for all sites in the main server config instead of .htaccess
. You'll need to tweak the RewriteRule
directive if you do, for example:
RewriteRule ^/robots\.txt$ /robots-disallow.txt [L]
You could also remove the robots-disallow.txt
file from each website and store this once elsewhere on your server (providing it is accessible) and rewrite to this file instead (using the absolute filesystem path) - you can only do this in the server config, not .htaccess
.