optimal cache setting for dynamic page

Question

My webpages on my website may appear static to the user, but they're actually dynamic since data is retrieved from a database and may be changed at anytime. Additionally, users may customize some preferences and that also changes the output as well.

At this present time, I have set the client side caching to nothing with the following HTTP headers:

Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
Cache-control: private, no-cache, no-store, must-revalidate

I also added a connection: close since the assets to the page are from a different subdomain.

For some reason, I feel there could be more optimal values for cache-control and expires. I also feel I could use etag validation but I'm not 100% sure. I want to use optimal values based on the worst-user perspective. and I think the worst user may consist of hitting the refresh button 100 times a minute.

I know for sure if I set the caching values too high, then when users choose customized settings (aka cookie value changes), they will have to refresh the browser or wait the number of seconds the cache is set for in order for the settings to take effect. With no caching set, the customized settings can be saved immediately.

I'm just trying to strike the right balance.

Should I enable caching, and if so, how long? and have other dynamic sites enabled caching for their dynamic html pages?

Answer 1

There are a couple of options, but they can be tricky to implement (depending on how your page is constructed), and tend to be of limited benefit unless your HTML page is quite large.

Etags

There are a few ways to generate an etag for your dynamic page, the simplest is probably just to create a hash of entire the HTML content before you send it. That way you can be sure it will be different if anything on the page has changed.

If you do this, you need to manually check the If-None-Match header if it's present. If it matches your ETag you send back a 304 instead of the HTML body (saving bandwidth).

I think you said elsewhere you're using PHP, so the code for this would be something like this (untested):

ob_start();

/* All your existing code here */

$content = ob_get_contents();
ob_end_clean();

$etag = md5($content);

header('ETag: '.$etag);

if (isset($_SERVER['If-None-Match']) && $_SERVER['If-None-Match'] == $etag) {
    // no need to send the response body
    header('HTTP/1.1 304 Not Modified');
    exit;
}

echo $content;

(This approach wouldn't be a good choice if you have deliberately randomized elements on your page that change every refresh, or something that changes very frequently like a clock.)

Last modified headers

This one tends to be harder to implement. You can use Last-Modified headers on dynamic pages if you can reliably determine the last modified time for every dynamic element on the page, along with anything else that might change the page's appearance (e.g. user preferences, which you mentioned elsewhere). If so, then whichever is the most recent time is the one you send back as the header. You also need to check for the presence of If-Modified-Since, and skip sending back the body if the timestamp isn't older than the last modified (using similar code to the etag one above).

This approach might be viable if you don't have many dynamic elements on your page. In practice it tends to be more trouble than it's worth.

---

The reason these approaches are not used that often is that both require you to generate the entire page on the server before you can determine whether or not it needs to be sent back to the user. Since it tends to be latency and generation of the page that are the bottlenecks, the benefit here is limited. Actually sending back the HTML to the client (particularly if it is gzipped) doesn't usually take that long unless your HTML page is huge.

As I think I've suggested elsewhere, if you want to improve performance of the HTML page, you'd be better off looking at using something like Varnish, and trying to make the HTML page more cache friendly to facilitate this.