5

My website is a static blog of visualizations and I am serving it via AWS S3.

Some of the visualizations use large CSVs, ranging from 1 megabytes to 20 megabytes.

I've set up Cloudfront to automatically gzip files, but for some reason there is a maximum size of 10 megabytes.

As a result, the page that depends on a 20 megabyte CSV takes around 5 seconds to load because Cloudfront isn't gzipping it.

I've checked, and if this file was gzipped then it would only be around 2 megabytes.

Is there any reason Cloudfront doesn't gzip files past 10 megabytes, or is there any sort of workaround I can use for automatically serving a compressed version of this file without too much hassle?

Improve this question
3
  • what is your origin server running?
    – L Martin
    Commented Jan 2, 2018 at 15:27
  • @LMartin: it's just an S3 bucket, so whatever Amazon uses I suppose
    – Chron Bag
    Commented Jan 2, 2018 at 15:39
  • Is this limit tweakable by AWS support? Has anyone tried to contact them about this issue?
    – Yves M.
    Commented Jan 15, 2018 at 12:55

2 Answers 2

Reset to default
11

This is a design limitation:

The file size must be between 1,000 and 10,000,000 bytes.

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html

Compressing files is resource-intensive, so the designers of CloudFront placed an upper bound on the size of files that CloudFront will spend resources compressing on-the-fly.

There's not an "automatic" workaround.

If your files are that large, and compressible, it's probably in your interest to store them compressed in S3 anyway, to reduce your storage costs.

Gzip the files with gzip -9 (which may actually result in slightly smaller files than generated by CloudFront -- gzip has varying levels of compression, with -9 being the most aggressive, and the level used by CloudFront does not appear to be documented), then remove the .gz extension, and upload the files to S3, setting Content-Encoding: gzip in the upload metadata.

Improve this answer
3
  • Note that with a file compressed directly in S3, the un-compressed version of the fille will not be available (if the client doesn't support gzip (unusual these days), or if you use curl without any parameters). CloudFront will not un-compress the file on the fly.
    – Yves M.
    Commented Jan 15, 2018 at 9:42
  • @YvesM. that is a valid point. I agonized over it a few years back when I set up a system that stored nearly everything in S3 gzipped but aside from curl's default behavior of not decompressing the payload unless you specify --compressed, this setup has never caused me any trouble "in the wild." I do this even for files like .xlsx, which benefit very little from gzipping because over hundreds of thousands of files, even a few bytes saved on storage and downloads seems like a winner.
    – Michael - sqlbot
    Commented Jan 15, 2018 at 12:50
  • I creative workaround would be Lambda@Edge, if this were an issue -- you can rewrite the path before sending the request to the origin, so you could theoretically change /foo to /uncompressed/foo for requests that lack the Accept-Encoding: gzip header and serve the uncompressed version, etc. (storing both versions at differing paths). Lambda@Edge could also be used to compress on the fly but only of you knew that the gzipped version was guaranteed to be < 1MB, which is the upper limit for "generated" responses, and this would definitely add some latency.
    – Michael - sqlbot
    Commented Jan 15, 2018 at 12:54
1

My website is a static blog

Since your site is static, it is an excellent candidate for s3_website, which will automatically gzip files locally before uploading, and will also handle setting the content-type and cache invalidation on CloudFront. It's a 'no-brainer' once you get it set up and I highly recommend it. It's also free and open-source.

You need both Ruby and Java installed to run it.

To get you started, here is a sample config file s3_website.yml that I use for an S3 bucket + Cloudfront delivered static site which is served via HTTPS with HTTP/2 enabled:

# S3 Credentials
s3_id: <redacted>
s3_secret: <redacted>

# Site URL
s3_bucket: www.example.com

# Config options
s3_endpoint: eu-west-1
index_document: index.html
cache_control:
  "assets/*": public, no-transform, max-age=1209600, s-maxage=1209600
  "*": no-cache, no-store
gzip:
  - .html
  - .css
  - .js
  - .ico
  - .svg

# CloudFront
cloudfront_distribution_id: AABBCC123456
cloudfront_wildcard_invalidation: true
cloudfront_invalidate_root: true
cloudfront_distribution_config:
  default_cache_behavior:
    min_ttl: <%= 60 * 60 * 24 %>
  http_version: http2
Improve this answer
2
  • Unfortunately s3_website doesn't work with AWS GovCloud. This is what I tried first.
    – Chron Bag
    Commented Jan 16, 2018 at 14:40
  • Ahh, that's a shame. You should consider adding that to the question as there are many GovCloud users out there. And maybe a fix can be implemented if you file a bug report for s3_website on GitHub? (this assumes it's a bug and not blocked intentionally by AWS, which may be what's happening)
    – Tom Brossman
    Commented Jan 17, 2018 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.