Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Big websites (Amazon, Facebook, Yahoo etc.) don't schedule downtime for upgrades. Usually they are done "live" and rolled progressively through the server farm. They also have big infrastructure and teams to manage this.

Smaller websites usually take the entire site offline to update the database structure and upgrade the code running on the web servers. The downtime can be very minimal but its still an interruption to customers.

How did you make the jump to no-downtime rolling updates? What are the minimum requirements to get this done? What can we do to build applications that make this possible from the start?

share|improve this question
    
After writing my answer, I noticed this could be a duplicate of: webmasters.stackexchange.com/questions/1838/… -- however, there are some good answers here so I'm not voting to close. –  timdev Aug 15 '10 at 20:57
    
This is a good article on how Facebook is dealing with updates in their server farm with no downtime. They ship small updates once a week. Code rolls out in phases to ever larger number of servers. Ops monitors for issues and can roll back. framethink.wordpress.com/2011/01/17/how-facebook-ships-code I'm OK with the answer being "you have to ship small releases more often with a larger team". –  Gareth Farrington Feb 17 '11 at 18:07
add comment

3 Answers 3

up vote 5 down vote accepted

What are the minimum requirements to get this done?

Once you have at least two servers behind a load-balancer, you can sequentially remove a server from the cluster, update it, and add it back to the cluster to complete the update (insofar as the visitor is concerned).

What can we do to build applications that make this possible from the start?

Design your application with load-balancing requirements in mind.

share|improve this answer
1  
That's not so easy, when the database structure changes. When multiple servers access the same database, you can't update the schema, before all servers are also up to date. And you can't update (+start) any server, before the schema is updated... –  Chris Lercher Aug 16 '10 at 0:38
    
Why would you rely upon a single instance of a single database if you're making changes to the database schema? –  danlefree Aug 16 '10 at 5:25
    
The problem doesn't disappear when you have multiple database instances (as long as multiple servers need to access the same data). When you stop the first DB server, and continue creating TXs on the second one, and then restart the first DB, you will need to synchronize the transactions from the second DB, before you can continue using the first DB (otherwise you'd change the order of transactions!). But you can't do that before you have also upgraded the second DB. So you have a downtime. Or do you see an easy solution? –  Chris Lercher Aug 16 '10 at 9:51
    
Let's assume you have control over the legacy code - you'd include triggers to port any important transactions over to your updated schema (which should already include an import of the legacy data). Synchronization is not much of an issue with multi-master replication. en.wikipedia.org/wiki/Multi-master_replication –  danlefree Aug 16 '10 at 17:02
    
@danlefree: This could probably lead to a solution. But multi master replication isn't free - usually you either lose ACID properties, or you introduce 2phase-commit locking overhead (which would again mean downtime, until all servers agree, that they can actually execute the transaction). I assume, there may be advanced methods to overcome (most of) the problems - I'd be really interested in seeing such a solution! –  Chris Lercher Aug 16 '10 at 17:48
show 1 more comment

There's always some reason for scheduled downtime, but it can be minimized.

Depending on your infrastructure, different strategies can minimize downtime. Regular-old-updates ought not require downtime.

On a number of PHP-driven sites I manage, I maintain side-by-side copies of the codebase, let's say version 1.0, 1.1 and 1.2:

/sites/site-1-0-0
/sites/site-1-1-0
/sites/site-1-2-0

And then create symlinks that the web server can use:

/sites/production --> /sites/site-1-1-0
/sites/staging    --> /sites/site-1-2-0

This way, I can stage my code on the production server for last-minute sanity checks, and when I want to go live, I just:

$ rm /sites/production; ln -s /sites/site-1-2-0 /sites/production

The web server uses the symlinks in the DocumentRoot specification, so the cutover is practically instantaneous.

There are, of course, gotchas, here. One needs to ensure that external data is stored somewhere, er, external. You don't want to be writing temp files, or storing user-generated content in the filesystem under the site-x-y-z directories.

Another alternative, if you've got multiple servers is to make the cutover via routing. Some VPS vendors (Linode comes to mind) make it easy to take two virtual machines and swap their IP addresses. So you set up your new version on a new server, do whatever testing is necessary, and then swap IPs to deploy your update. The same issues about keeping non-code assets up-to-date apply, but some careful thinking and planning can make that a non-issue.

With more robust, load-balanced setups, strategies like those suggested in danlefree's answer work as well.

share|improve this answer
add comment

0 downtime is the webserver equivalent of "the design must look exactly the same in every browser." Scheduled downtime is ok, just schedule it and put a static notice up. Unless it actually costs you tons of hits or money, which as a small website it would be definition not. If you do not want anyone to see it do it over the Fourth of July (or Thanksgiving or another opportune moment) unless your website is the #1 Google search result for "Firework burn treatment" or "Francis Scott Key lyrics" you will be just fine.

Doing this from the start usually leads to a much larger risk: over-engineering.

share|improve this answer
    
Scheduled downtime is the norm now. I don't see that going on forever. Expectations go up as customers see other sites not having any downtime. Maybe planning for this from the start can avoid over-engineering? Maybe some architectural change makes this easy? A/B testing is a very similar problem. Running 2 (or more) versions at once. –  Gareth Farrington Aug 15 '10 at 5:56
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.