Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act as an CDN and serve static content in order to improve speed. However, all of my content can be accessed via subdomain aswell, so Google has indexed it all and now I am dealing with duplicated content.

How could I deny access to crawlers for the subdomain baring in mind that I do not have different subfolder for subdomain, so I can't create a separate robots.txt file?

share|improve this question

One of the simplest things that you can do is set the rel canonical link tag on each of your pages. The canonical can point to itself on your main domain. When the page is served through the CDN, it would point to the live site. This would prevent search engines from indexing the HTML pages served through the CDN.

CDNs act like caching proxy servers. When a request hits them, they check to see if they have the resource. If they don't the have to fetch it from your website before they cache it and serve it. When they do that fetch, they should send some indication to your web server that the fetch is for the CDN. This may be a specific user agent string or some other header. Check with your CDN, or log an entire request sent through the CDN at a new URL that hasn't been fetched yet. Once you know this information, you could configure your server to serve a different robots.txt file that will be used only by the CDN.

share|improve this answer
I cannot edit pages on my subdomain because it is just a mirror site of the main site. It is pointing to the same root folder as the main domain. Maybe the only thing to do is to set something in htaccess and to create different robots.txt for subdomain requests, but I don't know what exactly – user3474818 May 11 '14 at 10:46
The canonical tag can be put on your main site and it will do the right thing for pages fetched from the subdomain as well. – Stephen Ostermiller May 11 '14 at 12:40

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.