I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site directly.

When I do a google search for "site:direct.example.com" it comes up with results. It means google had also crawled and indexed that sub-domain too. The problem is, since example.com & direct.example.com both consists of same content, it will end up in a content duplication. (I think it's not good for SEO).

So what I want is google bot to not to crawl and index "direct.example.com". I tried to use robots.txt to do the trick, but I failed since both uses the same robots.txt. What should I do to entirely prevent my sub-domain from Indexing? Is there any other options to over come this problem?

Thank you.

  • What type of content is being indexed? May 7, 2013 at 15:26
  • Do you use the direct subdomain? If not, you could just disable it. May 7, 2013 at 15:37

1 Answer 1


The simplest solution would be to disable the 'direct' subdomain. If however you want to use that subdomain you would have to use a more creative approach.

One way to do it is to have a dynamic robots.txt. When the web spider requests robots.txt we redirect it to our dynamic robots page. If the subdomain matches our criteria we send a 'disallow' otherwise we just present the normal robots.txt.

If you use Apache your rewrite rule might look something like this:

RewriteRule /robots\.txt$ /var/www/myweb/robots.php

The php file is generic.

    header('Content-type: text/plain');

    if ($_SERVER['HTTP_HOST'] == 'direct.example.com') { 
        echo "User-agent: *\n";
        echo "Disallow: /\n";
    } else {              

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.