How do I adjust my HTTP caching for optimum page delivery performance?

Question

I read an article about caching at http://vadmyst.blogspot.ca/2005/09/server-side-etag-vs-cache-control-max.html and currently I'm using both cache-control, expires, and Etag type validation on my website.

The article states that I should combine cache-control and Etag for best performance and it mentions earlier that they used 3600 for max-age on their server.

Since my site updates at least once a day (and sometimes several times a day if content is removed) and since many visitors to my site are students of which half are probably drunk, I can't see them being patient enough to wait one hour while the browser serves stale content to the user.

On the flip side, the article also states (and I agree) that if I only use Etag and no cache-control/expires, then requests are always made to the server and depending on how fresh the resulting data is, either an HTTP status 304 is returned or an HTTP status 200 is returned with content. The problem with this idea is that roughly 500 bytes of bandwidth are used per request just to send/receive headers with carriage returns attached. The actual page on my website on average compressed is about 4000 bytes.

Since I'm using apache web server which does not support HTTP 2.0, I feel the only way out is to look for a magic number to use for the cache-control's max-age value.

If I pick a number too high then the impatient students who lack computer knowledge who visit the site may not know to use the refresh function and/or clear cache and then they will tell me my website is not working.

If I pick a number too low, then more requests will be made to the server because the local browser cache will then expire and the only people that will benefit from too low of cache are the super-impatient visitors.

So what would be the best way to deal with this? Should I ditch cache-control/expires and just use Etag and not care about bandwidth usage? or should I use cache-control/Expires? And if I use cache-control/Expires, what should max-age equal to? I want to follow standards and make all visitors happy at the same time.

Answer 1

There isn't a 'one size fits all' solution to HTTP caching. The right solution for you depends on your site and the nature of your traffic. E.g. do most of your requests come from repeat visitors, or do people visit your site once and never return? If they are repeat visitors, how frequently do they visit? Are they likely to be checking regularly for something new (like visitors might be to a news site)?

For the HTML pages at least, it usually ends up being a compromise between always up-to-date pages or ultra cache friendly pages - you need to decide what the right point on that scale is for your site. If the majority of your traffic is impatient students looking for the latest content, personally I'd drop max-age and just let ETags do their thing. 500 bytes doesn't sound like something to worry about unless your site gets extremely high traffic.

For static assets it's a different story, since you can set far in the future expire headers (or high max-age), but change the URLs when the assets themselves change.