31

What headers should I be sending to outright stop all requests to the server after the content has been cached?

We have a very high latency server (Sigh, VMWare) so even sending a HEAD request to the server takes +40ms.

Currently these are the headers being sent/received;

First request

Client sends;

GET http://dugong:8080/Rvi24mYJkxFRGNzq73PPvgWGh1j/IMG_2071.jpg HTTP/1.1
Host: dugong:8080
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:9.0) Gecko/20100101 Firefox/9.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Pragma: no-cache, no-cache, no-cache
Cache-Control: no-cache, no-cache, no-cache

Server responds;

HTTP/1.1 200 OK
Server: nginx/1.0.11
Date: Wed, 01 Feb 2012 14:51:51 GMT
Content-Type: text/plain
Vary: Accept-Encoding
Last-Modified: Tue, 31 Jan 2012 10:45:11 GMT
Content-Length: 14
Expires: Thu, 31 Jan 2013 14:51:51 GMT
Cache-Control: max-age=31536000

So it sends a Cache-Control and Expires header set to 365 days in the future. Unfortunately on the second refresh it requests the object again with an If-Modified-Since header.

Second Request

GET http://dugong:8080/Rvi24mYJkxFRGNzq73PPvgWGh1j/IMG_2071.jpg HTTP/1.1
Host: dugong:8080
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:9.0) Gecko/20100101 Firefox/9.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
If-Modified-Since: Tue, 31 Jan 2012 10:45:11 GMT
Cache-Control: max-age=0

Response;

HTTP/1.1 304 Not Modified
Server: nginx/1.0.11
Date: Wed, 01 Feb 2012 14:58:00 GMT
Vary: Accept-Encoding
Expires: Thu, 31 Jan 2013 14:58:00 GMT
Cache-Control: max-age=31536000

Unfortunately due to silly outdated proxy software we can't use Keep-Alive, or put any other servers/proxys in front of the application. We also can't improve the performance of the server and reduce the network latency. I've been trying to figure out what headers we can send to get rid of the 301 requests. I've tried using ETags but that makes no difference, it still sends a If-modified-since header. I've also tried removing the Last-Modified header but that just causes a standard GET request with no caching (Checked the logs, server still receiving requests).

Clients are a mix of Firefox (mostly), IE 7, 8 and (some) 9, Chrome and Safari but this behavior seems to be appearing in all browsers tested.

TL;DR;

Terrible network, what headers should I send to tell clients to never ever ever send If-modified-since requests to the server to validate their cache, and keep content cached until the Expires header is met?

I'm probably missing something obvious but everything I try seems to yield the same results.

We have an NGINX server sat in front of our application server so I can add/remove any headers as I please. Our proxy doesn't support Keep-Alive and theirs no way to improve the attrocious network performance. Due to terrible software design the web app loads +100 resources on each page load (Yeah, enterprise software sucks) with a latency of ~40-50ms per object.

|improve this question
  • 1
    Hmm, that's odd. Sending the Expires and max age headers should prevent further requests of the image. Edit: by the way, what's the deal with sending a JPG as text/plain? – DisgruntledGoat Feb 1 '12 at 15:46
  • 1
    @DisgruntledGoat Ahh, you've made the assumption a .jpg file is actually an image rather than a text document. Welcome to my world =) (It's actually a text file that contains 'Hello World' for my testing, the software just renames all files sequentially IMG_xxxx.jpg regardless of type. Cool huh?) – Smudge Feb 1 '12 at 15:58
  • what were you using to set the http request headers? – barlop Jul 28 '19 at 12:30
25

You can't really control what headers user agents decide to send to you. If the file in question is in the browser's cache and it decides it need to check for a new version then it will. According to this article, these are the situations browsers will request using If-Modified-Since:

  • The cached entry has no expiration date and the content is being accessed for the first time in a browser session
  • The cached entry has an expiration date but it has expired
  • The user has requested a page update by clicking the Refresh button or pressing F5

So if you are reloading the page to test your caching, it won't work since the browser will re-request the images. Try clicking a link then another link back to the first page. If your users are regularly reloading pages then you may need to rethink your site/app structure to prevent that.

One thing that may help is adding "public" to the cache control header, i.e. Cache-Control: public, max-age=31536000. I also learned recently that an expiry date of more than one year is invalid. Since your expiry date is exactly one year, perhaps lowering that by a few days or weeks would ensure the file stays in browser caches and is not discarded.

|improve this answer
  • Interesting, I'll lower the expires to 60 days and add the public flag, and see what happens. This seemed to be happening on link-clicks rather than F5s (according to Firebug and the server logs) – Smudge Feb 1 '12 at 17:52
  • Technically, the HTTP/1.1 spec only says that "servers SHOULD NOT send Expires dates more than one year in the future" (presumably because that's a ridiculously long time before expiration) and that "approximately one year" in the future is the appropriate expiration time to send for content that is never expected to expire. – Ilmari Karonen Feb 1 '12 at 19:07
  • 1
    After a bit of playing around, I've concluded the 365d expiry is not affecting our clients however I've dropped it down to be safe, it seems the Cache-Control: public,... was the key to this specific situation. – Smudge Feb 2 '12 at 8:42
  • Do you mean the "public" header fixed the unnecessary round-trips ? I tried it, but with no success... – phtrivier Jan 16 '13 at 10:47
  • 2
    In case it's not clear from my answer, reloading the page in your browser will request the files again. Just click links to open pages and the browser uses its cache. – DisgruntledGoat Oct 18 '13 at 12:37
5

What you want is the immutable keyword in your Cache-Control line. Example for php:

header('Cache-Control: public, max-age=80000, immutable');

Source: https://code.facebook.com/posts/557147474482256/this-browser-tweak-saved-60-of-requests-to-facebook/

|improve this answer
3

I had the same issue, and the Requests are definitely hitting the server for it to respond with the 304 status - I'm sending the 304 via some C# and for sure it hits the server..

I only had Cache-Control: private set. No max-age and no Expires It operated as expected; hit the server with the If-Modified-Since where I test the value compared to what I expect and deliver 304 w/empty response body - else 200 & full the response body.

Setting Expires header had the desired results, 200 - (from cache) on client & no HTTP requests hit the server.

But.. I found that setting BOTH max-age= & Expires can cause browsers to not send If-Modified-Since header AND not to cache at all if the values don't match.

Something to be aware of if you have caching issues & used the different headers in combination.

|improve this answer
1

A little off-topic but maybe helpful. Another improvement for your requests of cached content, is to cache in sessionStorage so that you don't need to ask server to validate cache and receive a 304. Look for example google, open the console and write sessionStorage. You will see that they are caching CSS or DOM with the sessionStorage. ofc, you cannot use that in old IE browsers.

|improve this answer
0

Look through your source code and ensure there are no META REFRESH to transition to another page. Use something like sendRedirect instead. On my setup META REFRESH produces 304s in IE, but not Chrome. sendRedirect does not produce this on either browser.

<meta http-equiv="refresh" content="0;URL='nextpage'" />

vs

<% response.sendRedirect("nextpage") %>
|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.