This is a design limitation:
The file size must be between 1,000 and 10,000,000 bytes.
Compressing files is resource-intensive, so the designers of CloudFront placed an upper bound on the size of files that CloudFront will spend resources compressing on-...
So, the solution seems to be that Amazon cloudfront also evaluates my robots.txt and somehow uses different syntax rules from google.
The working version of my robots.txt is the following:
will browsers recognize individual files as being the same as pre-cached ones, if they are served from different subdomains?
No. Two identical files served from different locations are different files as far as the browser (cache) is concerned.
The URL is the key by which the file is cached by the browser.
As media1, media2 etc all serve the same files, ...
Amazon IAM ( http://aws.amazon.com/iam/ ) service gives you the ability to create accounts for students with separate password/credentials and with access to only specified AWS resources.
You can create one account for all students there, or create a group called "Students" and assign all your students to that group.
After doing that you need to specify an ...
It does matter. Using bucket-name.s3.amazonaws.com means you are referencing the S3 storage direct and not CloudFront. The requests will be served from wherever the S3 servers are (America, I assume) instead of an edge location closer to the user. Obviously this means you don't get the speed benefits; I believe S3 bandwidth is also more expensive than CF.
The advantage of CloudFront is that it serves your content geographically near to your users. A dedicated server is only in one location, but enables you to do lots of different things. You can't easily compare the two things.
If an attacker is able to put themselves between your origin server and CloudFront edge location - it won't be so difficult for the attacker to capture your HTTP traffic and extract your information. Theoretically, it can be done on the side of your ISP.
It is called as a "Man-in-the-middle" attack:
I guess, the better, more scalable approach is to find out what system your client is using, and educate that network engineer.
But if you really need a static IP to serve for Cloudfront content for your really important client, and if a different subdomain is acceptable to you, you could set up a proxy on your side for him.
That said, you will lose the ...
You should run WebPagetest on your web page and see where most of the time to render you web page takes place. WebPagetest produces a waterfall diagram that allows you to see where the time to load your page is spent.
Here is part of a WebPagetest waterfall diagram and report:
For example, if your web page make lots of references to static content and ...
Are you having the content on the CDN accessed by using the jr27a8vopf32a.cloudfront.net host name?
The usual way is to not expose the cloudfront server name, but add a CNAME entry to your DNS to point requests for CDN content to something like media.mysite.com
media.mysite.com CNAME jr27a8vopf32a.cloudfront.net
Cloudfront is notified of this CNAME ...
There are several advantages of using your own domain for a CDN:
You have full control over it, and can switch CDN providers more easily while retaining the same URLs.
The domain can contain keywords relevant to your site. For example if you sell widgets then widgetimages.com is a good domain to use.
If you use a subdomain, the images are now related to ...
The problem was that although I modified the search.widget.com CNAME to point to drvevtora80lk.cloudfront.net I forgot to add search.widget.com as an alternate name within my CloudFront distribution, added that and it now works.
My assets.example.com CloudFront distribution needed a robots.txt file after all. After adding one following these instructions, googlebot is able to fetch & render my static assets successfully.
It no longer reports them as Temporarily unreachable, and instead shows Complete.
Create a robots.txt in a bucket.
Create another origin for your cloudfront distribution.
Set your bucket's priority higher then your website.
Invalidate your site's robots.txt on Cloudfront.
After doing the above, Google will read the sites robots.txt when crawling your site and will get to see the different robots.txt when following links from your cdn.
That is pretty much what I have done;
Images on CDN with CNAME record.
Verified the CDN domain in Google Webmasters.
Using CDN urls in sitemap.
Added sitemap listing in robots.txt as well as in Google Webmasters.
No robots restriction on CDN domain.
And Google is indexing my images just fine. If I search for site:mysitedomain.com on Google; all images ...
Copy the css code that is used in the page, and place it in-line in a style tag on the page.
This forces the css, and there is no performance hit, and in some cases, it will run faster as there is no request routines for a css page.
Also, the fonts need to be the standard fonts, or locally installed, but I prefer to use the standard fonts the browser ...
My issue was because my CloudFront origin was not in us-east, and when you auto-select your bucket's origin in CloudFront, it doesn't add the location prefix to the url. So make sure you add your s3 location like so:
You unfortunately do not provide the name involved so people can not really troubleshoot things except by throwing guesses.
Here is my generic (not exhaustive and not all possible corner cases) quick but logical list of steps to execute in order to check for timeouts, and you should complete, in that order, each step successfully before going to the next ...
Double redirects are less efficient:
The browser makes a GET request for each
Each hostname (or subdomain) requires a separated DNS lookup.
In your chain you will have:
3 GET requests
Two DNS lookups (HTTP and HTTPS use the same DNS lookup for example.com)
If it is convenient, it would be better to configure http:// www.example.com to redirect directly ...
CloudFront supports multiple origin servers, and uses path patterns to determine which origin server to forward the requests to... so multiple, independent backend systems, even systems that aren't inside AWS, can all "own" one or more paths under a single hostname, with one of them being the default and owning all the paths not explicitly ...
Google should only return image results under Google image search. So if you search for site:example.com and click on the image search afterwards, it should just return your results.
Google will index whatever is embedded in the page regardless of whether you're using a CDN or not, and so 'View Image' will display it on the CDN while 'Visit Page' will ...
Found out the problem:
The CloudFront reads the robots.txt and prevents serving the content, but it parses some how different from what robots should, I guess.
For instance, the following content on robots.txt:
When Googlebot gets it itself, it indexes it;
When CloudFront reads it, it doesn't consider ...
Google does not block external resources from being indexed via using a robots.txt in the root of the main site. Using a sub domain, a cdn or other is classed as an external domain therefor the only way to block the content is using a header response on the file served by the CDN itself, or by using a robots.txt on the cdn or sub domain.
I am said network engineer, well maybe not that specific one but just another example and I come up against this question all of the time...
There are a number of firewalls out there that when at Enterprise scale, have issues with DNS based ACL's. There are Security and Performance Issues to be considered.
Can you trust the DNS won't be subverted or ...
When you added CDN, did you also added site-wide SSL? If you did, your current site in WMT is HTTP and that won't be crawled anymore as you have move away from HTTP.
So go ahead, add a new site in WMT as https:// in front of it and it will show you all the impressions and everything back to normal.
No, it will no hurt your SEO if your HTML page are static. The CDN will check for new versions from time to time (it depends on the configuration).
If your pages are not static, then you may want to set a no-cache header to your html pages. The CDN will not store a copy and users will always get a fresh version. If you don't do this, your users may have a ...
There is no way to get Googlebot to add parameters. Google does not support such functionality.
Googlebot does send a User-Agent header:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
However, you should not being doing anything differently for Googlebot than you do for users. Google calls that practice cloaking. Google will ...
If you have the names of the files you can paste those into CloudFront's invalidation web interface.
On Linux and Mac, you can get all the filenames easily by selecting the files in the file browser, copying them (Ctrl+C or Cmd+C) then pasting into a text editor. Then use find/replace to get them in the correct format for AWS. That doesn't work on Windows ...
The browser has no way to know that two distinct URLs are the same file.
I think the underlying question you are left with is: How do I host my files on multiple domains without getting as many downloads as there are domains (wow. is that English? sorry I'm French)
You can handle that by choosing wisely where your files are coming from. Two commons methods:...