9

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and example). Normally I'd modify /robots.txt and add Disallow: /, but due to shared code base I cannot modify /robots.txt without affecting all (sub)domains.

Any ideas how to go about it?

11

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in .htaccess. Something like:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^(www\.)?example\.com$ [NC]
RewriteRule ^robots\.txt$ robots-disallow.txt [L]

The above states that for all requests to robots.txt where the host is anything other than www.example.com or example.com, then internally rewrite the request to robots-disallow.txt. And robots-disallow.txt will then contain the Disallow: / directive.

If you have other directives in your .htaccess file then this directive will need to be nearer the top, before any routing directives.

  • I was thinking of same solution. I wasn't sure if there is something else out there, but at the end of the day, if that is it, then that what will gets my job done) – alexus Aug 18 '16 at 15:13
  • 1
    If both subdomains/hosts point to the very same webspace/code base then there's nothing in the robots.txt "standard" that can control this, if that is what you are suggesting. The bot is simply going to request sub.example.com/robots.txt, so you would need to do something to serve a different response depending on the subdomain. You don't need to use mod_rewrite, but it is a technique I've seen used several times. If robots.txt is dynamically generated then you could change the response in the server-side code (eg. PHP). – MrWhite Aug 18 '16 at 15:36
  • An alternative to using robots.txt might be to prevent indexing, rather than crawling, by sending an X-Robots-Tag: noindex HTTP response header when such subdomains are accessed (which could also be done in .htaccess). Although I think preventing crawling is probably preferable. (?) – MrWhite Aug 18 '16 at 15:40
0

robots.txt works only if it is present in the root.

You need to upload a separate robots.txt for each subdomain website, where it can be accessed from http://subdomain.example.com/robots.txt.

Add the code below in to robots.txt

User-agent: *
Disallow: /

And another way is you can insert a Robots <META> tag in all pages.

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
  • 3
    But the OP already states: "Normally I'd modify /robots.txt and add Disallow: /, but due to shared code base I cannot modify /robots.txt without affecting all (sub)domains." – MrWhite Aug 28 '16 at 11:58
0

I'd remove the meta tag from the html page and dynamically build it depending on your subdomain. e.g. we use subdomain dev. for development. So in the pageload event we have this:

  ' Check if domain is DEV or PROD and set robots accordingly meta tag in head
  Dim metatag As HtmlMeta = New HtmlMeta
  metatag.Attributes.Add("name", "robots")
  If CurrentURL.Contains("dev.advertise-it") Then
    metatag.Attributes.Add("content", "NOINDEX, NOFOLLOW")
  Else
    metatag.Attributes.Add("content", "INDEX, FOLLOW")
  End If
  Page.Header.Controls.Add(metatag)
  • Although this does not prevent crawling, which would seem to be the OPs requirement. – MrWhite Mar 26 '18 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.