Moving web servers without downtime

In IOLA, we've been moving some of the sites we're hosting, most notably iola.dk with associated services.

The problem with moving web services is DNS. DNS is a silly protocol - it uses a hierarchy to scale, which perhaps makes sense admin-wise (don't forget that scaling means more than just performance, administration is usually at least as important given the cost of man hours). But a hierarchy is a disaster when it comes to performance since the root nodes quickly end up being swamped - think 100 million clients all querying .com domains simultanously. DNS uses caching extensively to alleviate this problem.

And this is the problem starts. There is as far as I know no way to clear the cache. So with a typical DNS entry with maybe a time to live of 12 hours, you have a pretty long period in which some clients may see the old IP address and some clients are seeing the new.

In some cases, it's not a problem to have both web servers running at the same time. But if a site is modifying the database, it's more difficult as you don't want to have two inconsistent databases.

There are several ways to fix this, including staying up late at night until all your visitors have gone to bed, but I'll show a neat trick which is really easy to pull off and which shouldn't cause any sweat.

Start by installing Varnish, a reverse proxy server, on the old server. If you are on Debian, it's aptitude install varnish and you then get a /etc/default/varnish file. In there are a couple of lines like:

DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -b localhost:8080 \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

The first line means that Varnish is listening on port 6081 and the third line means that it is forwarding everything it sees to localhost port 8080. So change that third line to -b 123.456.789.012:80 or whatever the IP address of your shiny new server, execute /etc/init.d/varnish restart and go test it by accessing your old host on port 6081, e.g. http://www.example.com:6081. It should return the page from your new server - you can verify it by looking in the log on the new server.

What happens here is that Varnish parses the request from your browser, forwards it to the backend (your web server on the new server) and caches and returns the web server response to the browser. It's of course a pretty dumb idea to fetch the content over the web twice, from the new server to the old server and then from the old server to the web browser, but we won't be doing it for long. And Varnish will cache as much as its default configuration allows for.

Then when you think you're ready, change the first line above so that Varnish is listening on port 80 instead:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -b 123.456.789.012:80 \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

Then stop your web server, e.g. with /etc/init.d/apache2 stop and restart varnish with /etc/init.d/varnish restart. This puts Varnish in the place where the old web server was before. So from now on, there's only one web server seing the requests, the one on the new server. So you can switch over DNS from the old IP address to the new address, and when the last DNS entries have expired, you won't be getting hits anymore on Varnish on the old server.

Varnish can process requests based on URL and host so if you're serving multiple hosts from the same server, it's possible to forward only some request to the new server. It's pretty easy to do, you just need to enable a configuration file and put the stuff in there. In Debian there's an example configuration in /etc/varnish/default.vcl. As long as Varnish is not listening on port 80, it's pretty easy to experiment with it.

While you're at it you might want to start with putting Varnish on the new server in front of your ordinary web server. If there's something Varnish can cache, pages that look the same for several clients, it can save some serious load.

Losing sight of the big picture

I just had the most wonderful session with my piano. I haven't been practicing much lately, if at all, so I was mostly playing through old faster pieces to get my fingers up to speed.

Herr Mozart

A funny thing happened. As I'm playing the first movement of a sonata by Mozart, I suddenly realize that I'm struggling with the phrasing, articulating the bold and the faint passages clearly. This piece has lots of f and p markings (forte and piano, strong and quiet). I'm not too happy with the sound either. So I start focusing on getting it more precise, only to discover that the extra precision makes the sound worse.

The attention to detail draws attention away from the big picture, which in this case is the melody itself. Some pieces are simple, the music is in the tones. I think this is what most people relate to when they think of a melody. But this doesn't hold for most of the classical music I've heard and played. The tones are still important, but usually the overall dynamics are what will carry people away. Think of a symphony, an enticing crescendo towards the thundering finale, like the theme from Space Odyssey 2001 (it's from "Also sprach Zarathustra"). If it can't erect the small hairs at the back of your head, you need a treatment with a five-pounder shovel.

So by focusing on the details, which at most can make the music more interesting, although if you haven't played the piano you probably won't notice 90% of it, one can happily leave the most important of the piece to the coincidence.

It didn't work for me. So I gave up the details and started listening for the bigger picture, and voila! Pure joy. For five minutes, I was completely carried away.

The interesting thing is that the exercise turned out to be surprisingly harder than you'd think because all those little fs and ps kept breaking my concentration. I know this happens to professional players too, because I've experienced it at least once with a brilliant player, playing technically well and with lots of dynamics - but with no overall structure to push my emotions making the whole thing plain boring.

There's a long and interesting discourse into the land of interpretation of music here. But let me instead draw a parallel to software engineering.

For both piano playing and software engineering, often all we got is a clearly incomplete specification of how the thing is going to work. But if you (just) follow the spec, the result is guaranteed to be mediocre. And the more detailed the spec is, the more it'll distract you from the overall picture which is usually where the fate, success or failure, is determined.

Now witness Wilhelm Kempff, a legend whose playing to me represents the opposite of the brilliant player mentioned above. He may make a couple of mistakes and he obviously bends the spec into what he sees fit. But I'm carried away.