1

I have a client whose site has an absolutely massive directory of images; some dozens of gigabytes. (No, it's not that kind of site.)

I'd like to come up with a good way for the staging copy of the site to have access to the files in this directory in a safe way. Namely, any new deletions or uploads that using the staging site may trigger should not effect the image directory as accessible by the live site. The easiest approach to this would be to simply create a copy of the directory that the staging site can play in, but the absolutely massive site of this directory makes it impractical. (My current approach is to just copy over small subdirectories of the main directory from time to time, which works if my client and her employees only tests things in small parts of the site.)

I figure I could make this partially work by creating an empty directory to use as the file directory which new files can be uploaded to and configure Nginx to check for files in that directory first and then fall back to the "live" image directory next when requests for files in that directory come in. However, those files in the live directory would not be readable by the PHP site code itself and we wouldn't be able to "delete" existing files.

So I guess what I'm hoping for is a file-system-level trick which would allow me to mirror the contents of a directory while allowing for effective new file creation or existing file deletion from that directory but without actually affecting the original directory as would happen with a symlink. But any other ideas on how to handle this sort of situation would also be greatly appreciated.

2

Generally, it's not a good idea to share content between a live site and its staging installation. A staging copy should be isolated, precisely to avoid messing up anything on the production installation. On top of that, adding content on the staging copy would properly test those execution paths and create a consistent database—but this can be scaled only so far.

If you really want to fill the staging copy with realistic content, a good approach is to copy a subset of the database from production. You'll need to identify how you can slice the database so it's still consistent, i.e. there are no dependencies on items that you won't copy. Then you dump the selected database records and associated files with a script and load them in the staging copy.

I doubt that you can do anything automatic on the lower level just for the images, without potentially causing headaches. Since you need access to the files in the backend code, you'll have to have a layer that checks for the existence of a file before reading—either in your code or on some level below that. You can route all image access in your code via a function that does the check and copies the file if it doesn't yet exist; and then also direct Nginx to serve images via this code. You'll enable this only for the staging copy. It's doable, but sorta dirty in that you'll have to remember to use this layer everywhere image access is needed—though it shouldn't break anything if you try to reach for the file directly.

As for the filesystem, there are FSes that use ‘copy-on-write’, such as ZFS or Btrfs. I.e., possibly you could simply ‘copy’ all the images, and they wouldn't take extra space for the data and wouldn't be overwritten—instead, new files would be created on the staging copy when written. However, I'm not sure that the FSes actually make this functionality available to users and it's not just an internal detail—you'll have to check for yourself (it's likely that ‘copying’ would need to go through some special tools and not regular cp). Also, I doubt it that you'll want to change the FS on your live server just for this feature. Alternatively, since filesystem drivers can do basically anything, you can find some FS driver or a combination of them that presents exactly the virtual copy that you want, on top of a regular FS: OverlayFS seems to be the closest thing, keeping writes in isolated layers. The downside is that you'll need to have the staging copy access the files of the production site, but it's possible that you could put OverlayFS on top of SSHFS or another networked filesystem, e.g. via WebDAV (preferably configured for read-only access).

| improve this answer | |
  • OverlayFS is also used by Docker in images and containers, so if you use Docker it's possible that you could have the whole image-isolating thing as a container, even with SSHFS below it on the server or maybe via Docker's own networking. Not sure, though, that it supports the use-case of ‘put an overlay on top of live files’. – aaa Jan 28 at 15:20
  • It took us a while to get there, but it turns out OverlayFS is very close to what I'm looking for. it's unfortunate that it seems to only work on Linux, but that will suffice at least for this client. I just need to figure out how to force the lower directory to be mounted as readonly. – Garrett Albright Jan 30 at 23:18
  • Okay, after a little experimentation, it's appearing that OverlayFS is smart enough to not write to the specified "lower" directory even if it is writable. I've now got this all set up and working just as I was hoping it would. Thank you. – Garrett Albright Jan 31 at 3:05
0

Selective mirroring of an external site can be achieved with a "reverse proxy". It is pretty easy to set up a reverse proxy that serves local files and proxies them if they don't exist. You can use either nginx or Apache servers to do this.

Apache

Put this in the /assets/.htaccess on the staging server:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ https://livesite.exmaple/assets/$1 [P,L]

The condition checks that the file doesn't exist locally and if not, it reverse proxies it ([P] flag) from the live site.

This requires that mod_rewrite and mod_proxy both be enabled. I also had to add SSLProxyEngine on to my Apache config files to be able to proxy a HTTPS site. If you get a 500 Internal Server Error when you try it, check Apache's error_log and it should helpfully point you in the right direction.

Nginx

I don't have much experience with Nginx myself, but look at these questions that have some examples of what you want:

| improve this answer | |
  • Thank you, but as I mention in my question, there are two major problems to solving this through pure web server configuration. One, the files aren't actually there to be read by the script itself (as opposed to the web server as part of serving a file), and two, there's no way to mark a file as "deleted" in the staging directory if it still exists on the live directory. – Garrett Albright Jan 26 at 22:17
  • You can mark files as deleted with a rewrite rule like RewriteRule my-deleted-file.css - [G] that would go above the other rule but after turning the rewrite engine on. – Stephen Ostermiller Jan 26 at 22:30
  • If it is a script that is consuming these files as opposed to a webserver, you'd have to implement the proxy logic in your script rather than in webserver configuration. – Stephen Ostermiller Jan 26 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.