Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

It's a PHP app. How can I minimize the downtime while updating the entire codebase?

share|improve this question

migrated from stackoverflow.com Nov 18 '11 at 3:32

This question came from our site for professional and enthusiast programmers.

add comment

5 Answers

up vote 42 down vote accepted

What we generally do, at work is :

  • before we update, the document root of the server is :
    • in /www/app-2009-09-01
    • but is accessed via a symbolic link, called /www/application
  • we put the whole new code base to /www/app-2009-09-08
  • once the whole code base is there :
    • we remove the old symbolic link
    • we create a new symbolic link, still called /www/application, but which points to the new sources : /www/app-2009-09-08
  • we reload apache to force the modification to be taken into account.

All this process is done via an automatic script (the only not-automatic thing is us launching it when needed). This means :

  • Everything goes fast (especially the switching of symbolic link, which is the important part)
  • No risk of doing an error : the script has been well-tested, and working for months/years


Another advantage of this symbolic link precedure is that it is very easy to "rollback" an update if we notice a catastrophic bug only after putting the new version of the sources to production : we just have to switch the symbolic links back.

Of course, this doesn't prevent you from testing the new version on your staging server before putting it to production -- but, who knows... Sometimes, there is a really big bug that no-one has been able to see while testing :-(
For instance, because there is no load-testing done on a regular basis on the staging machine.
(I've seen the "rollback" thing used something like 4 or 5 times in 3 years -- each time, it saved the day -- and the websites ^^ )


Here is some kind of a quick example : suppose I have this VirtualHost in my Apache configuration :

<VirtualHost *>
        ServerName example.com
        DocumentRoot /www/application
        <Directory /www/application>
            # Whatever you might need here (this example is copy-pasted from a test server and test application ^^ )
            Options Indexes FollowSymLinks MultiViews +SymLinksIfOwnerMatch
            AllowOverride All
            php_value   error_reporting 6135
            php_value short_open_tag  on
        </Directory>
</VirtualHost>

Pretty "standard"... The only thing is /www/application is not a real directory : it's just a symbolic link to the current version of the sources.
Which means when you have put the sources to the server, but not switched yet, you'll have something like this :

root@shark:/www
# ll
total 8
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-01
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-08
lrwxrwxrwx 1 root root   19 2009-09-08 22:08 application -> /www/app-2009-09-01

Notice that the symlinc points to the "old version"

Now, that the new version has been totally uploaded to the server, let's switch :

root@shark:/www
# rm /www/application
root@shark:/www
# ln -s /www/app-2009-09-08 /www/application

And, now, the /www/application points to the new version of the sources :

root@shark:/www
# ll
total 8
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-01
drwxr-xr-x 2 root root 4096 2009-09-08 22:07 app-2009-09-08
lrwxrwxrwx 1 root root   19 2009-09-08 22:09 application -> /www/app-2009-09-08

And we just have to restart Apache :

root@shark:/www
# /etc/init.d/apache2 restart
 * Restarting web server apache2

The three steps "remove the link ; create the new link ; restart apache" should be made quickly ; ie, by an automated script, and not by a human being.

Using this solution :

  • you can take as much time as you need to upload the new version of the sources : apache will not use them as long as the symlic has not been changed
  • when everything is OK, just switch the symlink : it'll go faster than changing even 1 or 2 files... Which means virtually no downtime :-)

And if using some opcode-cache like APC with the stat option at 0, it can mean even less risk of downtime, I suppose.


Of course, this is the "simple" version -- if you have some uploaded files, for instance, you'll have to use another symlink somewhere, or another VirtualHost or whatever...


Hope this is more clear :-)

share|improve this answer
    
It's a kind of server swapping too. :-) –  Wim ten Brink Sep 8 '09 at 20:00
    
mod_rewrite to manage symbolic links? –  gAMBOOKa Sep 8 '09 at 20:01
    
@gAMBOOKa : no : just a matter of Apache's DocumentRoot (or VirtualHost DocumentRoot), which is /www/application ;; ie, the symbolic link -- no matter where that one points to. –  Pascal MARTIN Sep 8 '09 at 20:02
2  
Terrific answer. One more tip, though: you can make the symbolic link occur without unlinking it. As quoted: "The three steps … should be made quickly ; ie, by an automated script, and not by a human being." The mv command is an atomic operation, so you can create a symlink like 'ln -s /www/app-2011-01-28 /www/application-temp' and then do a 'mv -T /www/application-temp /www/application'. –  Joe Liversedge Jan 29 '11 at 11:24
1  
There are something was not covered by the symlink method. Your way works with Apache+mod_php but it could fail on lighttpd+fastcgi. In a high traffic website, request will be served in the middle of swapping the link that, php code dependency will fail with mixed version. –  Dennis Cheung May 25 '11 at 2:25
show 9 more comments

Can't you take the existing code and migrate the project into a separate test php file, and use that while you are making your updates? What I mean is, you should have a test server and a production server so that when you have to make an update you don't incur any downtime.

share|improve this answer
add comment

Set up a second server with the updated codebase and switch them as fast as possible. :-)

If not possible, make sure your codebase is divided into dozens of smaller parts. Then the downtime would be limited to just one subpart one at the time. The smaller codeblocks are easier to replace and most will just continue to run without problems. Just try this on a test environment first, though!

share|improve this answer
    
Since the app has not be tested with fragmented modules, it might result in unexpected scenarios. –  gAMBOOKa Sep 8 '09 at 20:02
    
Which means that this would be on your ToDo list for after this update. :-) Make it more modular and you can update per module. –  Wim ten Brink Sep 8 '09 at 20:05
1  
That is on the ToDo list, but is a long term goal. We're a young startup, so organization within the dev team will naturally take a while. =D –  gAMBOOKa Sep 8 '09 at 20:07
add comment

First off, I often use and like a method similar to Pascal MARTIN's response.

Another method that I also like is to use my SCM to push new code. The exact process depends on your type of SCM (git vs svn vs ...). If you're using svn, I like to create an "online" or "production" branch that I checkout as the document root on the server. Then, whenever I want to push new code from another branch/tag/trunk, I just commit the new code into the "online" branch and run svn update in the document root. This allow for very easy rollbacks as there is a complete revision log of what's gone up/down to the server and who did it and when. You can also easily run that "online" branch on a test box, allowing you to vet the app you're about to push.

The process is similar for git and other styles of SCM, just modified to be more natural for their style of work flow.

Want to pull/poll instead of pushing updates? Just have a cron job or other, smarter mechanism automatically run svn update.

Extra: You can also use this process to backup files that your application wrote to disk. Just have a cron job or some other mechanism run svn commit. Now the files your application created are backed up in your SCM, revision logged, etc. (ex., if a user updates a file on disk, but wants you to revert it, just push the old revision).

share|improve this answer
add comment

I use a similar approach to Pascal MARTIN's, too. But instead of uploading multiple versions of my app to the production server, I keep the "builds" behind my firewall, each in a separate directory with the build number and date. When I want to upload a new version I use a simple script that includes "rsync -avh --delay-updates". The "delay=updates" flag will upload everything (that's different) to a temporary folder until all the updates are there, and then move everything over at once at the end of the transfer to their proper paths so the app will never be in a half-old-half-new state. It has the same effect as the method above, except I only keep one version of the app on the production site (best to only have only the bare essential files on the production server, IMO).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.