I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront CDN.
I've spent a lot of time testing and trying to figure out why google doesn't crawl these files and reports a "resource block" status.
Currently I serve these files from several hostnames like: cdn1.example.com, cdn2.example.com, …
cdn1, cdn2 and the others are CNAME's to the cloudfront distribution name.
Test: I've tried to use directly the cloudfront distribution (no CNAME) but the problem persists.
Currently my robots.txt looks like this:
# Google AdSense User-agent: Mediapartners-Google Disallow: #Google images User-agent: Googlebot-Image Disallow: / User-agent: * Disallow: /homepage Disallow: /index.php* Disallow: /uncategorized* Disallow: /tag/* Disallow: *feed Disallow: */page/* Disallow: *author* Disallow: *archive* Disallow: */category* Disallow: *tag=* Disallow: /test* Allow: /
And examples of files blocked in one example page:
I've even tried to allow everything in robots.txt but I always have the same result.
I've also been looking carefully at CloudFront settings in Amazon and see nothing that could be related (I don't use and never used the option :"Restrict Viewer Access (Use Signed URLs or Signed Cookies)".
Right now I've spent a lot of time looking into this and have no more ideas.
Can someone can think of a reason why Googlebot would be blocked from crawling files hosted in Amazon CloudFront?