Spring!

Spring is upon us here in Denmark! The signs are subtle but sure.

My snowdrops are flowering.




Galanthus, snowdrops

The garden looks as if it is almost dead, but upon closer inspection small green leaves are shooting up. And the ground is soft and wet, not hard and frosty.


My mostly dead, but not quite!, garden

It's getting warmer, my trusty winter coat and wide scarf is getting too warm for driving up the hill on bike.


Me and a tame deer in Arden in December

Yesterday a pair of ducks had occupied the pond in the park. This morning two coots were grassing when I passed downhill. And in the evening blackbirds are singing.

Yay!

PS: If you're quick, there's still time for our special offer for new art on paper and canvas on YayArt.

Ideology and pragmatism

Perhaps I failed to mention the most important point in the thing I posted about capitalism the other day. Actually, I've been thinking about analogies between distributed software architecture (big systems) and society (a big system) for long, but I regret having spent most of the post about it because that kind of ego-centered pattern-matching is usually in itself pretty tiresome to read about.

A more interesting point is that of ideology versus pragmatism.

Interestingly, there's a similarity to them. When you believe in an ideology, you'll defend your choices with the end justifying the means. We may leave the poor bastards in the mud or we may kill half a million Iraqis or we may suppress people who think different from us and destroy the historical monuments of our past, but in the end we're defending freedom or equality and that's more important than anything else. It will prevail, and everything will be better than it once were.

When you believe in pragmatism, you'll defend your choices with the end justifying the means. We may spend ten times as much fixing the minds of violent criminals as the traumatized minds of their victims, we may have to see half of our help to poor nations disappearing in corruption, we may have to accept that people with big salaries get the lion's share of a tax cut. But in the end, we can prove that everyone is generally better off than with the alternative.

Religious pragmatism (from Flickr)

Pragmatism is discomforting. A true pragmatist will defend actions that are against the values that person believes in. Of course, ideology is discomforting too, as it requires you to suppress the truth.

This is all fine in theory. Most people will probably agree that we should base political decisions on ration and facts rather than beliefs.

However, in practice nobody knows for sure what the future outcome of a decision is. This is worse in some fields than in others. There's also the aside that without strong personal beliefs, lots of the valuable work being done in this world would probably cease to happen. Ideologically founded people will put lots of energy to an unselfish end in satisfying their drive.



I'm personally ideological when it comes to environmental issues. I would like to think, self-indulgently, that it's because I'm thinking further ahead than people in general but the truth is that I don't know why. As the windmill production in Denmark has grown to an international leadership, this has become more of a pragmatic decision. To my dismay, the right-wing in Denmark then blocked the whole windmill programme for most of the 00s out of what I see as ideological issues (money rests better in people's own pockets).

However, in other areas I find it easier to be pragmatic. Dismissal of known facts is something not to be taken lightly. This is especially true when it comes to issues where the ideological argument is persuit of justice or equality. The world is not just (sometimes children are molested), and we're not born equal, and although there's nothing wrong with building a system that tries to ameliorate these things, it's important to keep focus on reality, not the ideas.

Actually, that leads me software design again, but I think I'll save that for another blog post. :-)

Caching on the web

When you build a web application intended for a large scale, there are two separate performance issues when it comes to generating the HTML the server spits out.

One is server load. How many visitors can the server cope with?

The other thing is response time as seen by each visitor. How long must I wait for the page to load?

When using a relatively slow dynamic web framework (as Python + Django), the issue with server load is mostly related to CPU time. If you read the Django docs, the recommended solution is caching, i.e. instead of generating the HTML on each hit, we generate it once, store it for a while, handing out the stored snippet until it times out. It's a simple idea that works incredibly well.

First question is where we do the caching. To reduce the workload as much as possible, we want it to happen as close to the browser as possible. For a low-cost solution that works for everyone, we need to stick to the same physical location as the web server, however.

With Django on Apache, you've got some relatively fast C code serializing the requests to separate Python processes running the Django code. The process model limits the concurrency level possible because each process gobs a fair amount of memory, and Python is just generally an order of magnitude slower than C. So while putting the cache in the web framework is arguably very convenient, it's also pretty slow. From my experiments, one or two orders of magnitude.

Ideally, we'd just run a simple process written with speed in mind in front of Apache to do the caching. Afterall, the idea is relatively straightforward. Enter Varnish, a web cache written in C.

With Varnish in place, you know that the server only has to generate a page once per timeout. Some of the pages on YayArt take about 1 second to generate because they need to do multiple complex database queries. Let's say we have a beefy server that can do 10 of these in parallel. Then we could support 10 requests/s. With caching in Varnish, we can scale up to whatever it takes Varnish to retrieve a string from its cache and serve it. You can easily reach 6000 requests/s. For comparison, 100 requests/s is 8.6 million hits/day.

However, as it turns out, while simple whole page caching can solve a large part of the server load problem with one blow without giving up the convenience of a dynamic web framework, it's not necessarily the solution to the response time. If the page is in the cache, the response time is close to perfect. But what if it's a slow day with few visitors and it's not?

The general solution is to precompute the answer. Unfortunately, how to do this is more application specific.

You might be wondering how to combine caching with personalized pages. On YayArt I use Javascript to integrate the per-user data into the page through AJAX. So the per-user server load is reduced to processing the actual per-user data instead of a whole page.

Market economy

For Christmas, my parents bought me 3 months of subscription to Information, a small Danish newspaper with a focus on analysis and criticism of the daily politics. The audience is mostly intellectual (and also mostly left-wing).

Information is interesting for several reasons, one of them being that major politicians in Denmark, the people who actually make the rules for the rest of us, regularly post in the newspaper, and thus presumably also read it. For an old internet-addict, that's a whole new experience.

There's been a lot of talk lately about capitalism in the face of the current financial crisis. The thing is that over the past 20 years, almost everyone in the political landscape has been pursuing a the-more-market-economy-and-deregulation-the-better strategy to a some degree. And now it turns out that too few rules for the gamblers in the financial sector have put us into a global recession. So people are, again, beginning to question whether capitalism is such a hot idea after all.

When I went to high school, I always thought it was possible to do better. Because there is such an obvious waste in our market economy. For an example, go to your nearest supermarket and look at the shelves with shampoo and hair products. Or the shelves with soft drinks. Or the shelves with morning cereals for children. And ask yourself, how much value does all these colourful and overly expensive things bring to our society? Or the really classy ads that never mention any factual qualities in their products, but instead try to install in us an irrational idea of their products, because of the packaging so to speak, bringing improvements to our lives.

I ask you, how can drinking water with added sugar and various brown chemicals make you cool? Because a large company has invested enormous efforts in convincing everyone that it is so.

Surely ration can do better than that.

However, at university I spent a couple of years working with distributed systems. The most important lesson I learned is that centralized systems are bad, unless the scale is very small. Decentralized (peer-to-peer) systems are more adaptive to change, more robust and much more efficient - several orders of magnitude as the system scales. Yes, there's waste, redundancy, suboptimal behaviour.

But it's self-organizing. If I want something from a peer, I just ask - I don't need to contact a central authority which then has to decide how to respond to the change. Overall it just works incredibly better.

I think that the same is true of society. With a market economy, decision-making is decentralized, in spite of the tendency for old industries to advance towards monopolies. In the large scale, this decentralization is unbelievably effective compared to a centralized control because of the complexity involved.

It's also a lot more free than a democracy. In a democracy, 55% can decide for the remaining 45% (that's why we haven't built any new windmills in Denmark lately). With a market economy, everyone decides for himself. Within the limits of the market, of course.

However, not all is good. One of the more peculiar problems of market economy is the drive towards monopolies, i.e. a collapse of the market (read a treatise of Karl Marx's works to understand why). Like any game we set up, it needs rules.

Karl Marx

And the market cannot do long-term thinking. As we have just seen, it can happily drive over the cliff edge because of a phenomenon called the tragedy of the commons. It doesn't hurt me a lot that I exploit the system: I get the whole benefit, the downside is shared between everyone.

The tragedy is that even if some market players want to stop, they face the competition from the others. If price is the only factor, a short-term thinking competitor can drive the others off the market. So they are forced to follow, unless a force beyond the market sets down rules that cannot be ignored. The same reasoning goes for unethical behaviour.

So the issue here is how we change the rules of the game to ameliorate the bad things without throwing the basic idea out of the window. For in spite of the waste, it's working better than the alternatives. Ration is bounded, it's not enough to deal satisfactorily with the needs of millions of humans.

Converting MySQL to UTF-8 the easy way

This is just a quick note to myself.

So you're running on the latest version of INSERT NAME OF FANCY WEB FRAMEWORK HERE? Think character set problems are a relic of the past? Not so with MySQL. The default configuration is using Latin-1. When you install MySQL, the first you should do is ensure MySQL is using UTF-8. This problem will go away at some point when distributions change the defaults, but until then.

Meanwhile, if you're like me, you might have created some tables before discovering the problem. It is, after all, difficult to see before you create the tables. So here's a recipe for converting a whole database (idea stolen from Wordpress).

First type in these commands, replacing mydb with the name of your database:

USE information_schema;
SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', column_type, ' CHARACTER SET utf8;') FROM columns WHERE table_schema = 'mydb' and data_type LIKE '%char%';
SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', column_type, ' CHARACTER SET utf8;') FROM columns WHERE table_schema = 'mydb' and data_type LIKE '%text%';
SELECT CONCAT('ALTER TABLE ', table_name, ' CHARACTER SET utf8;') FROM tables WHERE table_schema = 'mydb';

This should output the commands you need to feed into MySQL to do the change. If you start the MySQL shell with -s, it's easier to copy-paste. Then type

USE mydb;
ALTER DATABASE mydb CHARACTER SET utf8;
[... pasted commands ...]

The problem here is that the character set is stored on multiple levels. Both column, table and database level. The second line fixes the character set of the database, and the pasted in commands fixes the columns and tables.

This works for Django. If you have used a framework that allows you to put UTF-8 characters into the Latin-1 columns, you need to do something else. The Wordpress link has the details.

Moving web servers without downtime

In IOLA, we've been moving some of the sites we're hosting, most notably iola.dk with associated services.

The problem with moving web services is DNS. DNS is a silly protocol - it uses a hierarchy to scale, which perhaps makes sense admin-wise (don't forget that scaling means more than just performance, administration is usually at least as important given the cost of man hours). But a hierarchy is a disaster when it comes to performance since the root nodes quickly end up being swamped - think 100 million clients all querying .com domains simultanously. DNS uses caching extensively to alleviate this problem.

And this is the problem starts. There is as far as I know no way to clear the cache. So with a typical DNS entry with maybe a time to live of 12 hours, you have a pretty long period in which some clients may see the old IP address and some clients are seeing the new.

In some cases, it's not a problem to have both web servers running at the same time. But if a site is modifying the database, it's more difficult as you don't want to have two inconsistent databases.

There are several ways to fix this, including staying up late at night until all your visitors have gone to bed, but I'll show a neat trick which is really easy to pull off and which shouldn't cause any sweat.

Start by installing Varnish, a reverse proxy server, on the old server. If you are on Debian, it's aptitude install varnish and you then get a /etc/default/varnish file. In there are a couple of lines like:

DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -b localhost:8080 \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

The first line means that Varnish is listening on port 6081 and the third line means that it is forwarding everything it sees to localhost port 8080. So change that third line to -b 123.456.789.012:80 or whatever the IP address of your shiny new server, execute /etc/init.d/varnish restart and go test it by accessing your old host on port 6081, e.g. http://www.example.com:6081. It should return the page from your new server - you can verify it by looking in the log on the new server.

What happens here is that Varnish parses the request from your browser, forwards it to the backend (your web server on the new server) and caches and returns the web server response to the browser. It's of course a pretty dumb idea to fetch the content over the web twice, from the new server to the old server and then from the old server to the web browser, but we won't be doing it for long. And Varnish will cache as much as its default configuration allows for.

Then when you think you're ready, change the first line above so that Varnish is listening on port 80 instead:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -b 123.456.789.012:80 \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

Then stop your web server, e.g. with /etc/init.d/apache2 stop and restart varnish with /etc/init.d/varnish restart. This puts Varnish in the place where the old web server was before. So from now on, there's only one web server seing the requests, the one on the new server. So you can switch over DNS from the old IP address to the new address, and when the last DNS entries have expired, you won't be getting hits anymore on Varnish on the old server.

Varnish can process requests based on URL and host so if you're serving multiple hosts from the same server, it's possible to forward only some request to the new server. It's pretty easy to do, you just need to enable a configuration file and put the stuff in there. In Debian there's an example configuration in /etc/varnish/default.vcl. As long as Varnish is not listening on port 80, it's pretty easy to experiment with it.

While you're at it you might want to start with putting Varnish on the new server in front of your ordinary web server. If there's something Varnish can cache, pages that look the same for several clients, it can save some serious load.

Losing sight of the big picture

I just had the most wonderful session with my piano. I haven't been practicing much lately, if at all, so I was mostly playing through old faster pieces to get my fingers up to speed.

Herr Mozart

A funny thing happened. As I'm playing the first movement of a sonata by Mozart, I suddenly realize that I'm struggling with the phrasing, articulating the bold and the faint passages clearly. This piece has lots of f and p markings (forte and piano, strong and quiet). I'm not too happy with the sound either. So I start focusing on getting it more precise, only to discover that the extra precision makes the sound worse.

The attention to detail draws attention away from the big picture, which in this case is the melody itself. Some pieces are simple, the music is in the tones. I think this is what most people relate to when they think of a melody. But this doesn't hold for most of the classical music I've heard and played. The tones are still important, but usually the overall dynamics are what will carry people away. Think of a symphony, an enticing crescendo towards the thundering finale, like the theme from Space Odyssey 2001 (it's from "Also sprach Zarathustra"). If it can't erect the small hairs at the back of your head, you need a treatment with a five-pounder shovel.

So by focusing on the details, which at most can make the music more interesting, although if you haven't played the piano you probably won't notice 90% of it, one can happily leave the most important of the piece to the coincidence.

It didn't work for me. So I gave up the details and started listening for the bigger picture, and voila! Pure joy. For five minutes, I was completely carried away.

The interesting thing is that the exercise turned out to be surprisingly harder than you'd think because all those little fs and ps kept breaking my concentration. I know this happens to professional players too, because I've experienced it at least once with a brilliant player, playing technically well and with lots of dynamics - but with no overall structure to push my emotions making the whole thing plain boring.

There's a long and interesting discourse into the land of interpretation of music here. But let me instead draw a parallel to software engineering.

For both piano playing and software engineering, often all we got is a clearly incomplete specification of how the thing is going to work. But if you (just) follow the spec, the result is guaranteed to be mediocre. And the more detailed the spec is, the more it'll distract you from the overall picture which is usually where the fate, success or failure, is determined.

Now witness Wilhelm Kempff, a legend whose playing to me represents the opposite of the brilliant player mentioned above. He may make a couple of mistakes and he obviously bends the spec into what he sees fit. But I'm carried away.