Is it worth it to change my entire user images file structure to take advantage of simple browser caching?

Question

On one of my mobile sites, I simply store my user's profile images as '1.jpg' in their user folder, and incrementally go from there for any extra pics they upload. This means that whenever they change their profile pic, for example, the file name stays the same.

I've been wanting to take advantage of image caching so that the same old pic doesn't get downloaded over and over again whenever a user's profile is viewed and re-viewed, but at the same time, I want my users' browsers to download the new one if it has changed.

From what I've been reading, it seems that the only way to truly do this is to actually use random file names and keep track of all those file names in the DB, so that you can set a non-expiring cache, while recently-changed pics get pulled again since they have a new file name. The beauty of the way I have them structured up until now, however, is that I can skip the database entirely and access the files directly since their location is predictable.

So my question is, is it worth it for me to change the entire file structure of my site, plus add the DB element, for the benefit of eternal caching and automatic re-downloading upon new upload?

This is a huge undertaking, but if it's deemed worthy, I have no problem moving forward with this drastic change. I just want to make sure this is how the "big boys" do it so that I never have to change the file structure ever again.

Thanks.

Answer 1

One commonly used solution is to make your image URLs look something like this:

http://www.example.com/path/to/images/1.jpg?v=123456

Here, /path/to/images/1.jpg is the actual URL path of the image, while ?v=123456 is just a dummy query staring tacked onto the end of the URL. The query string can be anything — a version number, a timestamp, a hash of the image content — as long as you change it whenever the image changes, and keep it the same when it doesn't.

The trick is that the web server, when asked to serve such a URL, will ignore the query string, since the URL in fact points to a static file. But to the user's browser (and to any proxies in between), URLs with different query strings will be completely different, and so any change to the query string forces the browser to reload the file.

Thus, you can configure your web server to send Expires and Cache-Control HTTP headers to allow indefinite caching, safe in the knowledge that you can force a reload by changing the query string. One way to do that, if you're using Apache with mod_expires, is to put an .htaccess file in your image directory with the lines:

ExpiresActive On
ExpiresDefault "access plus 1 year"

This technique is used by many popular websites. For example, if you look at the HTML source of this very page, you'll find that the style sheet for it is loaded from a URL like this:

http://cdn.sstatic.net/stackoverflow/all.css?v=7cd8ea9d6f1e

Here, the ?v=7cd8ea9d6f1e is a dummy query string just like I described above; you can confirm that by changing it and seeing that it indeed still returns the same file.

Answer 2

There's more than one way to cache.

Conditional GET

If you are storing these images on the file system and serving them directly through the web server, you are probably already using conditional get. The web server will automatically use filesystem metadata to set an ETAG header, and will automatically reply with "304 Not Modified" if the browser includes If-Modified-Since or If-Matches headers in its request. (All browsers will.)

In this case the whole image is not served back, so you have bandwidth savings. However, a GET request will still be issued, so you will still have the overhead and latency of a request.

You can decrease the number of requests slightly at the expense of cache freshness by having your webserver set Cache-Control headers with a public,max-age=N value for your images. This says that caches can keep the resource for at most max-age seconds before they have to check if it is updated.

However, HTTP defines only one way to invalidate a cache entry, which may not fit your application's semantics: if you POST or PUT to a url that updates the profile photo, reply with a Location: [url of photo] header and the cache entry for that url will be invalidated.

(This is the mechanism that allows you to cache a webpage with comments, and then have the page forcibly reloaded by the browser after the user posts a new comment. The browser would reply to a POST /comment with 303 See Other and a Location: /page/with/comment. Note that this didn't used to work in Firefox due to a longstanding bug.)

Unless you have a lot of traffic, this approach to caching is fine.

Changing urls

A url is a representation of a resource, so another way to manage caching is not to change cache parameters for the resource, but to make a brand new resource with an "cache forever" directive. This is the approach that the "big boys" favor, because it allows them to generate no extra requests, saving them lots of bandwidth. The downside is it requires much more additional bookkeeping.

There are two general techniques for this.

Query strings

Web servers ignore query strings when serving a file from the file system. Caches, however, do not: /1.jpg?t=12345 and /1.jpg?t=67890 are two completely different, unrelated resources, even though the server thinks they are the same.

So one easy thing you can do is append the filesystem timestamp as a query string whenever you make a reference to a resource in your html, and set a long Expires header. The browser will then cache this resource forever and not do any GETs as long as the query string does not change.

A downside is that it's difficult or impossible to instruct the webserver of the new url for an item if you want to forcibly invalidate a cache. For example, if a browser has a cached HTML page with a /1.jpg?v=1 reference, but happened to clear the entry for /1.jpg?v=1 (maybe it ran out of file or memory space), it will make a new request to /1.jpg?v=1. If in the meantime the image has changed to /1.jpg?v=2, the proper response is either:

1. Serve the old version of the file. You would do this if you wanted all resources to be consistent with one another as they were at a certain point in time. This is what you should do with CSS files, for example, since a new css file with an old html file may not work properly!
2. Redirect to the new version of the file using 301 Moved Permanently. You would do this if you wanted all resources to be as new as possible.

Both of these are difficult to do with a webserver alone, which means you need to invoke a web application even for image requests, which can be both more complicated and more resource-intensive. Webservers are very fast at serving files, so the overhead of a web application may end up swallowing your bandwidth and latency gains.

File names

Instead of adding a query string, you change the filename. This means it's easy to keep multiple versions of files on the file system, but you will probably need to store file metadata and do other database book-keeping to keep track of your resources and their names.

Answer 3

read about http status 304 Not Modified, you should be able to response to a download request with 304, and by that tell the server to use the cached data, insted of resending it to the browser. and read this question https://stackoverflow.com/questions/2978496/make-php-page-return-304-not-modified-if-it-hasnt-been-modified