CDN virtual subdomain causes duplicated content

Question

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act as an CDN and serve static content in order to improve speed. However, all of my content can be accessed via subdomain aswell, so Google has indexed it all and now I am dealing with duplicated content.

How could I deny access to crawlers for the subdomain baring in mind that I do not have different subfolder for subdomain, so I can't create a separate robots.txt file?

Answer 1

One of the simplest things that you can do is set the rel canonical link tag on each of your pages. The canonical can point to itself on your main domain. When the page is served through the CDN, it would point to the live site. This would prevent search engines from indexing the HTML pages served through the CDN.

CDNs act like caching proxy servers. When a request hits them, they check to see if they have the resource. If they don't the have to fetch it from your website before they cache it and serve it. When they do that fetch, they should send some indication to your web server that the fetch is for the CDN. This may be a specific user agent string or some other header. Check with your CDN, or log an entire request sent through the CDN at a new URL that hasn't been fetched yet. Once you know this information, you could configure your server to serve a different robots.txt file that will be used only by the CDN.