Updating a web app without any downtime

Question

It's a PHP app. How can I minimize the downtime while updating the entire codebase?

Answer 1

What we generally do, at work is :

• before we update, the document root of the server is :
  • in /www/app-2009-09-01
  • but is accessed via a symbolic link, called /www/application
• we put the whole new code base to /www/app-2009-09-08
• once the whole code base is there :
  • we remove the old symbolic link
  • we create a new symbolic link, still called /www/application, but which points to the new sources : /www/app-2009-09-08
• we reload apache to force the modification to be taken into account.

All this process is done via an automatic script (the only not-automatic thing is us launching it when needed). This means :

• Everything goes fast (especially the switching of symbolic link, which is the important part)
• No risk of doing an error : the script has been well-tested, and working for months/years

Another advantage of this symbolic link precedure is that it is very easy to "rollback" an update if we notice a catastrophic bug only after putting the new version of the sources to production : we just have to switch the symbolic links back.

Of course, this doesn't prevent you from testing the new version on your staging server before putting it to production -- but, who knows... Sometimes, there is a really big bug that no-one has been able to see while testing :-(
For instance, because there is no load-testing done on a regular basis on the staging machine.
(I've seen the "rollback" thing used something like 4 or 5 times in 3 years -- each time, it saved the day -- and the websites ^^ )

---

Here is some kind of a quick example : suppose I have this VirtualHost in my Apache configuration :

<VirtualHost *>
        ServerName example.com
        DocumentRoot /www/application
        <Directory /www/application>
            # Whatever you might need here (this example is copy-pasted from a test server and test application ^^ )
            Options Indexes FollowSymLinks MultiViews +SymLinksIfOwnerMatch
            AllowOverride All
            php_value   error_reporting 6135
            php_value short_open_tag  on
        </Directory>
</VirtualHost>

Pretty "standard"... The only thing is /www/application is not a real directory : it's just a symbolic link to the current version of the sources.
Which means when you have put the sources to the server, but not switched yet, you'll have something like this :

root@shark:/www
# ll
total 8
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-01
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-08
lrwxrwxrwx 1 root root   19 2009-09-08 22:08 application -> /www/app-2009-09-01

Notice that the symlinc points to the "old version"

Now, that the new version has been totally uploaded to the server, let's switch :

root@shark:/www
# rm /www/application
root@shark:/www
# ln -s /www/app-2009-09-08 /www/application

And, now, the /www/application points to the new version of the sources :

root@shark:/www
# ll
total 8
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-01
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-08
lrwxrwxrwx 1 root root   19 2009-09-08 22:09 application -> /www/app-2009-09-08

And we just have to restart Apache :

root@shark:/www
# /etc/init.d/apache2 restart
 * Restarting web server apache2

The three steps "remove the link ; create the new link ; restart apache" should be made quickly ; ie, by an automated script, and not by a human being.

Using this solution :

• you can take as much time as you need to upload the new version of the sources : apache will not use them as long as the symlic has not been changed
• when everything is OK, just switch the symlink : it'll go faster than changing even 1 or 2 files... Which means virtually no downtime :-)

And if using some opcode-cache like APC with the stat option at 0, it can mean even less risk of downtime, I suppose.

Of course, this is the "simple" version -- if you have some uploaded files, for instance, you'll have to use another symlink somewhere, or another VirtualHost or whatever...

Hope this is more clear :-)

Answer 2

Can't you take the existing code and migrate the project into a separate test php file, and use that while you are making your updates? What I mean is, you should have a test server and a production server so that when you have to make an update you don't incur any downtime.

Answer 3

Set up a second server with the updated codebase and switch them as fast as possible. :-)

If not possible, make sure your codebase is divided into dozens of smaller parts. Then the downtime would be limited to just one subpart one at the time. The smaller codeblocks are easier to replace and most will just continue to run without problems. Just try this on a test environment first, though!

Answer 4

First off, I often use and like a method similar to Pascal MARTIN's response.

Another method that I also like is to use my SCM to push new code. The exact process depends on your type of SCM (git vs svn vs ...). If you're using svn, I like to create an "online" or "production" branch that I checkout as the document root on the server. Then, whenever I want to push new code from another branch/tag/trunk, I just commit the new code into the "online" branch and run svn update in the document root. This allow for very easy rollbacks as there is a complete revision log of what's gone up/down to the server and who did it and when. You can also easily run that "online" branch on a test box, allowing you to vet the app you're about to push.

The process is similar for git and other styles of SCM, just modified to be more natural for their style of work flow.

Want to pull/poll instead of pushing updates? Just have a cron job or other, smarter mechanism automatically run svn update.

Extra: You can also use this process to backup files that your application wrote to disk. Just have a cron job or some other mechanism run svn commit. Now the files your application created are backed up in your SCM, revision logged, etc. (ex., if a user updates a file on disk, but wants you to revert it, just push the old revision).

Answer 5

I use a similar approach to Pascal MARTIN's, too. But instead of uploading multiple versions of my app to the production server, I keep the "builds" behind my firewall, each in a separate directory with the build number and date. When I want to upload a new version I use a simple script that includes "rsync -avh --delay-updates". The "delay=updates" flag will upload everything (that's different) to a temporary folder until all the updates are there, and then move everything over at once at the end of the transfer to their proper paths so the app will never be in a half-old-half-new state. It has the same effect as the method above, except I only keep one version of the app on the production site (best to only have only the bare essential files on the production server, IMO).