2

I have a website with some some content which changes frequently.

I would like to benefit from good caching when this doesn't change (so would ideally use a longer Cache-Control header), but would also like to ensure the latest information is used (so ideally would use a shorter Cache-Control header). These obviously conflict.

There are two benefits to caching data:

  1. Reduce any call at all and use version straight from cache (if resource is used within cache control header timeframe).

  2. Use 304 - Not Modified in response to a request for expired content which it turns out is still the latest version, and which is still in the browser cache (based on ETag or Last-Modified header in request). This still means you need to request the data and receive a 304 response meaning a round trip and latency issues (the performance hit of which is not to be underestimated!), so not as good as first option, but does at least save on the full download and ensures you have latest version.

So my questions are:

When an HTML page or content (CSS, JavaScript, images, fonts... etc.) is expired past it's cache control header time, how long do browsers keep the content around to make use of 304 responses?

Is there anything I can do at server or application level to ensure data is available for 304 responses for a longer time? I.e. Is it possible to have data cached in the browser for a long time (say 30 days) but re-requested within a much shorter time (say after 3 hours) to benefit from 304 responses more? I don't think there are any "keep in cache" HTTP headers, other than the Cache-Control header which I can only use for the shorter time, so am guessing this is purely down to Browser implementation? I'm assuming browsers periodically clear out their cache of expired data to keep disk usage down and asking if I have any influence over which of my website resources I'm happy to clear out more aggressively and which I'd prefer to advise the browser to keep round for longer on the hope that a 304 can be used later.

P.S. I'm aware you can add a timestamp or other unique code to the filename, and long cache-control header, and update this unique part on file change to achieve what I want. This means that, to the browser, it looks like a new resource and hence is fetched. However, for various reasons, this is not always the easiest to implement (and can't be done on the default "index.html" page without server side redirects).

2

When an HTML page or content (CSS, JavaScript, images, fonts... etc.) is expired past it's cache control header time, how long do browsers keep the content around to make use of 304 responses?

While browsers may keep the content stored on disk in a temporary or cache directory beyond the expiry date/time, the browser will consider it out-of-date (not 'fresh') and will still refer to it and revalidate it before attempting to re-download the file. Unfortunately the specification does not cover how browsers should deal with expires files, and the actual clean-up of the files on disk will depend on the browser and user preferences (such as a cached files diskspace limit) and can vary but could be when the browser is closed.

Mark Amery has written an excellent answer (Browser caching - prevent request that returns 304) which while addressing a different question, explains clearly and accurately the relationship of HTTP 304 responses and browser caching behaviour.

Is there anything I can do at server or application level to ensure data is available for 304 responses for a longer time? I.e. Is it possible to have data cached in the browser for a long time (say 30 days) but re-requested within a much shorter time (say after 3 hours) to benefit from 304 responses more? I don't think there are any "keep in cache" HTTP headers, other than the Cache-Control header which I can only use for the shorter time, so am guessing this is purely down to Browser implementation?

I would suggest simply specifying a Cache-Control: public, max-age=1080, must-revalidate (18 minutes, from your comment) and the browser will certainly revalidate the content after that time, but may also at its own discretion revalidate during this time or when a user hits refresh to reload a page.

If you really wanted you could also try using ETag matching with an MD5 hash rather simply than the checking the last modified date/time to ensure files are not re-downloaded unless the content of the file has changed. There can be a slight performance hit calculating ETag's for each request if you get a lot of traffic, and many people oppose this method either for the slight performance degradation, and because many people don't configure it to use MD5 (you'll have to use a server-side script for this) it would offer no real benefit over the last modified date/time check and could behave erraticly when files are distributed from multiple servers. Having said that, configured correctly, ETag comparison combined with long expiry times can help ensure less downloads of files is required. To ensure you're not poorly affected by performance, before setting up any ETags make sure you webserver configuration is setup to use persistent connections so that requests can be chained together on one connection very quickly rather than requiring multiple connections since all modern browsers will utilise this if your server is setup for it.

I'm assuming browsers periodically clear out their cache of expired data to keep disk usage down and asking if I have any influence over which of my website resources I'm happy to clear out more aggressively and which I'd prefer to advise the browser to keep round for longer on the hope that a 304 can be used later.

You can indeed specify a different cache time for different resources on your website. For example if you have a sub-folder (for example, lib) in which you store third party libraries/code such as jQuery, requireJS or FontAwesome for example, then you could include the version in the folder name (e.g. http://www.example.com/lib/fontawesome-4.3.0/) and then set the maximum or at least a very long cache time for everything within this lib folder as you will not be making modifications directly to third party code/files, and future versions would be introduced under new folders with the version name included. Static resources (.js,.css,.jpg,.png etc) that are used for your website template that may change from time to time could have a shorter cache time set.

# Cache third party files for 1 year (31536000 seconds)
<Directory "/var/www/public_html/lib">
  <IfModule mod_expires.c>
    Header set Cache-control max-age=31536000
  </IfModule>
</Directory>

# Cache template files for 1 day (86400 seconds)
<Directory "/var/www/public_html/tpl">
  <IfModule mod_expires.c>
    Header set Cache-control max-age=86400
  </IfModule>
</Directory>

P.S. I'm aware you can add a timestamp or other unique code to the filename, and long cache-control header, and update this unique part on file change to achieve what I want. This means that, to the browser, it looks like a new resource and hence is fetched. However, for various reasons, this is not always the easiest to implement (and can't be done on the default "index.html" page without server side redirects).

You're absolutely right, these are not easy to implement in an effective and efficient manner and requires a lot more effort/code than using the HTTP protocol headers as they were intended. The only scenario I know of where this kind of solution is sometimes preferred is for cross platform software such as content management systems where they wish to consistently support all the various webserver environments. For example, if their software was written in PHP, they might implement a solution that can be achieved entirely in PHP rather than utilising .htaccess rules.


Added on 04-Jun-2015:

Regarding the browser behaviour of expired ('stale') cached files:

"A stored response is considered "fresh", as defined in Section 4.2, if the response can be reused without "validation" (checking with the origin server to see if the cached response remains valid for this request). A fresh response can therefore reduce both latency and network overhead each time it is reused. When a cached response is not fresh, it might still be reusable if it can be freshened by validation (Section 4.3) or if the origin is unavailable (Section 4.2.4)."

Source: RFC 7234: HTTP Caching, last paragraph of Introduction.

Regarding the behaviour of the must-revalidate directive:

"The "must-revalidate" response directive indicates that once it has become stale, a cache MUST NOT use the response to satisfy subsequent requests without successful validation on the origin server."

Source: RFC 7234: HTTP Caching, 5.2.2.1. must-revalidate.

  • Ok but my understanding was a browser would still potentially use the cache after expiry, but revalidate (as it's expired and so can no longer be assumed to be valid) and potentially get a 304 back. You're basically saying browsers only 304 when an explicit refresh is done WITHIN the expiry windowing and never 304 AFTER expiry. Will do some more testing on that to confirm. I also don't particularly want to set the 30 day expiry and assume browsers "might" revalidate unless I had some more guarantees on that (e.g. an RFC that said it should be revalidated at least every x days). – Barry Pollard Jun 3 '15 at 19:05
  • Guess what I'm looking for, and what doesn't exist, is a header something like: "Header set Cache-control max-age=259200, revalidate-after=1080". Guess it doesn't exist as it would be prone to abuse as everyone would just set really long expiries. – Barry Pollard Jun 3 '15 at 19:09
  • You are correct, the browser will still use the cache after expiry and revalidate before re-downloading however there are no guarantees or requirements in the standards for how browsers should manage their caching. If the file is still in the cache, then the browser will attempt to revalidate it. I've corrected the part of my earlier answer that suggested expired cached files would be disregarded, apologies for confusion on that aspect. – richhallstoke Jun 4 '15 at 9:43
  • 1
    Best you can do is set max-age=1080 and hope the browser will keep a copy stored as long as 25900 seconds after being downloaded. Specify must-revalidate to ensure that after 1080 seconds further requests will revalidate (and buy you another 1080 seconds of validated cache copy). – richhallstoke Jun 4 '15 at 9:43
  • If the only website users access is yours, the expired files would never leave the cache as most modern browsers seem to only delete old files from cache to make space for new files to be cached. Therefore the more frequent they access your site the more likely stale (expired) files will remain stored in the cache directory and be available for revalidation. – richhallstoke Jun 4 '15 at 9:47
0

When you deal with static pages (such as files with the html extensions), header configuration can only be applied to server configuration files, and depending on the server, the configuration options might be limited.

For more flexibility without messing up the server, use a server-side scripting language such as PHP.

Also, for cache-control, you want to use the "must-revalidate" option so that when the cache expires, then the browser will recheck the server for the resource needed.

As for the 304, when the browser requests the resource again, it will issue a value to a server environment variable if-modified-since or if-none-match depending on whether you use the eTags or last-modified header. The server (or script executed by the server) then issues a 304 if it believes the variable contains the required data. Search for if-modified-since and if-none-match on google, and there will be more details on how those work.

  • I think "must-revalidate" means you should revalidate even within the cache expiry time. I'm happy to use cache within the shorter time (3 hours in my example) without revalidating, but wanted to know if there was a way to keep it available to cache after expiry time so it could be used again if revalidated. My concern is that a short expiry time will equate to a higher likelihood of cache purging. – Barry Pollard Jun 4 '15 at 7:18
  • I test my website headers through redbot.org with must-revalidate attached to the cache-control header, the caching results are what are expected: This response cannot be served by a cache once it becomes stale. I also ran a test on yahoo.ca where it returned the cache-control header without must-revalidate and the info returned was This response may still be served by a cache once it becomes stale. – Mike Jun 4 '15 at 15:07
  • Thanks. More interesting reading here: stackoverflow.com/questions/18148884/… – Barry Pollard Jun 4 '15 at 16:18
-1

Although an answer has been accepted here, the advice given is (IMHO) not very good.

The RFC is very clear about what a cache should do with stale content (its not the job of the browser to cache content, the cache is considered a seperate component). It should not serve up content which is considered stale.

Except for very large files, returning a 304 header without new caching information is no faster than returning new content. If the content appears multiple times on a page, the resulting load time can be much slower than loading from an empty cache! Unfortunately its difficult to configure most webservers to include new caching instructions on a 304 response.

There are multiple solutions to the problem.

can't be done on the default "index.html" page without server side redirects

No, you can't change the URL, but if your caching policy is based on access time then you can update the file timestamp. Alternatively just remove the if-none-match and if-modified-since request headers. But for the reasons above, you'll get little benefit.

For videos, pdfs and sitemap.xml.gz allowing 304 responses is probably a good thing - use expiresByType or LoactionMatch or a directory specific config if its in a seperate dir.

  • Thanks for taking the time, but this answer doesn't really answer the question at all in my opinion. Additionally it IS possible to return caching headers with 304s (at least in Apache and Nginx) and the RFC says that "use other header fields provided in the 304 (Not Modified) response to replace all instances of the corresponding header fields in the stored response." While I agree 304s can be almost as slow as a full 200 for small pages,I still do not think that's a reason not to use them. – Barry Pollard Oct 26 '15 at 14:12
  • Could you please share how to send new caching information with a 304 response in Apache? – symcbean Oct 26 '15 at 15:32
  • You don't need to do anything special. Once caching headers are set up they work for 304s in same way as 200s. For example go to www.apache.org with Chrome and with Developer tools open (and make sure Disable cache is not ticked). Pick an asset which will 304 (e.g. asf_logo.png), note the Expires time, then press F5 to reload the page and look at the 304 response which will have an Expires time a few seconds further on from the original one. – Barry Pollard Oct 26 '15 at 16:04
  • That didn't answer my question. It only seems to be possible to issue new caching information in apache if it is done regardless of the state of the content using mod_headers or using mod_expires with an expiry time relative to the access time. – symcbean Oct 26 '15 at 17:37
  • Yes you basically reapply the same caching. Say you've said "cache it for 3 hours" on the original 200 request, then you visit after 4 hours and luckily it's still in the browser cache so you get a 304 which then says the cached version is still up to date and now is valid again for another 3 hours. Not sure why you feel it's necessary to provide completely new caching information on a 304 response? Could you explain that for my understanding? – Barry Pollard Oct 26 '15 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.