Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site directly.

When I do a google search for "site:direct.example.com" it comes up with results. It means google had also crawled and indexed that sub-domain too. The problem is, since example.com & direct.example.com both consists of same content, it will end up in a content duplication. (I think it's not good for SEO).

So what I want is google bot to not to crawl and index "direct.example.com". I tried to use robots.txt to do the trick, but I failed since both uses the same robots.txt. What should I do to entirely prevent my sub-domain from Indexing? Is there any other options to over come this problem?

Thank you.

share|improve this question
    
What type of content is being indexed? –  bybe May 7 '13 at 15:26
    
Do you use the direct subdomain? If not, you could just disable it. –  nathangiesbrecht May 7 '13 at 15:37

1 Answer 1

up vote 1 down vote accepted

The simplest solution would be to disable the 'direct' subdomain. If however you want to use that subdomain you would have to use a more creative approach.

One way to do it is to have a dynamic robots.txt. When the web spider requests robots.txt we redirect it to our dynamic robots page. If the subdomain matches our criteria we send a 'disallow' otherwise we just present the normal robots.txt.

If you use Apache your rewrite rule might look something like this:

RewriteRule /robots\.txt$ /var/www/myweb/robots.php

The php file is generic.

<?php
    header('Content-type: text/plain');

    if ($_SERVER['HTTP_HOST'] == 'direct.example.com') { 
        echo "User-agent: *\n";
        echo "Disallow: /\n";
    } else {              
        include('robots.txt');    
    }
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.