I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act as an CDN and serve static content in order to improve speed. However, all of my content can be accessed via subdomain aswell, so Google has indexed it all and now I am dealing with duplicated content.

How could I deny access to crawlers for the subdomain baring in mind that I do not have different subfolder for subdomain, so I can't create a separate robots.txt file?

1 Answer 1


One of the simplest things that you can do is set the rel canonical link tag on each of your pages. The canonical can point to itself on your main domain. When the page is served through the CDN, it would point to the live site. This would prevent search engines from indexing the HTML pages served through the CDN.

CDNs act like caching proxy servers. When a request hits them, they check to see if they have the resource. If they don't the have to fetch it from your website before they cache it and serve it. When they do that fetch, they should send some indication to your web server that the fetch is for the CDN. This may be a specific user agent string or some other header. Check with your CDN, or log an entire request sent through the CDN at a new URL that hasn't been fetched yet. Once you know this information, you could configure your server to serve a different robots.txt file that will be used only by the CDN.

  • I cannot edit pages on my subdomain because it is just a mirror site of the main site. It is pointing to the same root folder as the main domain. Maybe the only thing to do is to set something in htaccess and to create different robots.txt for subdomain requests, but I don't know what exactly May 11, 2014 at 10:46
  • 1
    The canonical tag can be put on your main site and it will do the right thing for pages fetched from the subdomain as well. May 11, 2014 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.