2

I have a CloudFront distribution with a configured alternative name (e.g. cdn.example.com). Is it possible to restrict access to the CDN via d1234.cloudfront.net name?

I don't want it to be indexed if we place a link to d1234.cloudfront.net by mistake.

Improve this question
  • Are you able to use canonical tags to fix this? – jrtapsell Jul 9 '18 at 12:26
  • @jrtapsell We use cannonical tags. However, they do not prevent Google from indexing that other domain and showing it in search results. We already had that problem. – Ivan Nikitin Jul 9 '18 at 14:11
  • So for d1234.cloudfront.net you are serving a canonical URL on cdn.example.com? – jrtapsell Jul 9 '18 at 15:29
  • @jrtapsell Exactly. Yet, Google indexes both domains and shows them both in the search results. Probably, lower our rankings in the process since it sees duplicated content. – Ivan Nikitin Jul 9 '18 at 16:13
1

The solution is uploading a separate robots.txt file for CloudFront URL. This will not block the CloudFront URL but will keep it away from google search

Create robots_cf.txt file that contains this text:

User-agent: *
Disallow: /

Now add this rule to .htaccess file (say Cloudfront URL is aaabbbcccddd.cloudfront.net):

RewriteCond %{HTTP_HOST} ^aaabbbcccddd.*$ [NC]    
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]    
RewriteRule ^robots\.txt /robots_cf.txt [NC,L]

Eventually, Google will get this and stop crawling your CloudFront URL.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.