is expired past it's cache control header time, how long do browsers
keep the content around to make use of 304 responses?
While browsers may keep the content stored on disk in a temporary or cache directory beyond the expiry date/time, the browser will consider it out-of-date (not 'fresh') and will still refer to it and revalidate it before attempting to re-download the file. Unfortunately the specification does not cover how browsers should deal with expires files, and the actual clean-up of the files on disk will depend on the browser and user preferences (such as a cached files diskspace limit) and can vary but could be when the browser is closed.
Mark Amery has written an excellent answer (Browser caching - prevent request that returns 304) which while addressing a different question, explains clearly and accurately the relationship of HTTP 304 responses and browser caching behaviour.
Is there anything I can do at server or application level to ensure
data is available for 304 responses for a longer time? I.e. Is it
possible to have data cached in the browser for a long time (say 30
days) but re-requested within a much shorter time (say after 3 hours)
to benefit from 304 responses more? I don't think there are any "keep
in cache" HTTP headers, other than the Cache-Control header which I
can only use for the shorter time, so am guessing this is purely down
to Browser implementation?
I would suggest simply specifying a
Cache-Control: public, max-age=1080, must-revalidate (18 minutes, from your comment) and the browser will certainly revalidate the content after that time, but may also at its own discretion revalidate during this time or when a user hits refresh to reload a page.
If you really wanted you could also try using
ETag matching with an MD5 hash rather simply than the checking the last modified date/time to ensure files are not re-downloaded unless the content of the file has changed. There can be a slight performance hit calculating ETag's for each request if you get a lot of traffic, and many people oppose this method either for the slight performance degradation, and because many people don't configure it to use MD5 (you'll have to use a server-side script for this) it would offer no real benefit over the last modified date/time check and could behave erraticly when files are distributed from multiple servers. Having said that, configured correctly, ETag comparison combined with long expiry times can help ensure less downloads of files is required. To ensure you're not poorly affected by performance, before setting up any ETags make sure you webserver configuration is setup to use persistent connections so that requests can be chained together on one connection very quickly rather than requiring multiple connections since all modern browsers will utilise this if your server is setup for it.
I'm assuming browsers periodically clear out their cache of expired
data to keep disk usage down and asking if I have any influence over
which of my website resources I'm happy to clear out more aggressively
and which I'd prefer to advise the browser to keep round for longer on
the hope that a 304 can be used later.
You can indeed specify a different cache time for different resources on your website. For example if you have a sub-folder (for example, lib) in which you store third party libraries/code such as jQuery, requireJS or FontAwesome for example, then you could include the version in the folder name (e.g.
http://www.example.com/lib/fontawesome-4.3.0/) and then set the maximum or at least a very long cache time for everything within this lib folder as you will not be making modifications directly to third party code/files, and future versions would be introduced under new folders with the version name included. Static resources (.js,.css,.jpg,.png etc) that are used for your website template that may change from time to time could have a shorter cache time set.
# Cache third party files for 1 year (31536000 seconds)
Header set Cache-control max-age=31536000
# Cache template files for 1 day (86400 seconds)
Header set Cache-control max-age=86400
P.S. I'm aware you can add a timestamp or other unique code to the
filename, and long cache-control header, and update this unique part
on file change to achieve what I want. This means that, to the
browser, it looks like a new resource and hence is fetched. However,
for various reasons, this is not always the easiest to implement (and
can't be done on the default "index.html" page without server side
You're absolutely right, these are not easy to implement in an effective and efficient manner and requires a lot more effort/code than using the HTTP protocol headers as they were intended. The only scenario I know of where this kind of solution is sometimes preferred is for cross platform software such as content management systems where they wish to consistently support all the various webserver environments. For example, if their software was written in PHP, they might implement a solution that can be achieved entirely in PHP rather than utilising .htaccess rules.
Added on 04-Jun-2015:
Regarding the browser behaviour of expired ('stale') cached files:
"A stored response is considered "fresh", as defined in Section 4.2, if
the response can be reused without "validation" (checking with the
origin server to see if the cached response remains valid for this
request). A fresh response can therefore reduce both latency and
network overhead each time it is reused. When a cached response is not
fresh, it might still be reusable if it can be freshened by validation
(Section 4.3) or if the origin is unavailable (Section 4.2.4)."
Source: RFC 7234: HTTP Caching, last paragraph of Introduction.
Regarding the behaviour of the must-revalidate directive:
"The "must-revalidate" response directive indicates that once it has become stale, a cache MUST NOT use the response to satisfy subsequent requests without successful validation on the origin server."
Source: RFC 7234: HTTP Caching, 220.127.116.11. must-revalidate.