Ruminations

When two becomes three

2011-10-06T01:26:00.001+02:00

The evening before, when Janne and I were going to bed, at around 11, she complained about a pain in the stomach and got up to take a bath. I was mildly suspicious, but the last couple of weeks she'd had some painful practice contractions and trouble with the back. I was exhausted, having slept poorly the previous week, told myself that if true labor contractions had started, she would tell me, and continued to try to sleep poorly.

At around 6:00 in the morning, I finally wake up to a meak yell from the bath room - Ole, I think we need to hurry up. As I later discover, the light of my life has been in labour all night long, not thinking it was a big deal until she can feel the water break and the baby's head coming down. She gets out of the bath room, calls the hospital, they agree to see her at 7:00. As we later learns, she should have made clearer how far she was and they would have sent an ambulance.

Instead, on her request, I then call my parents who live nearby and get an awake and triumphant mother in the other end at the first ring; she had a bet with my sister on the birth being this very day. My father will come to pick up us. At this point, noone but Janne has any idea how far she is. Yet, my father hurries through the dark empty Sunday morning streets of Aalborg and we reach the hospital safely.

There we're greeted by first an assistant and then a midwife whose shift is ending. She decides to have a look at J. anyway and finds out the cervix is fully open, declares her the most tough and calm woman in labor she's seen so far and immediately walks her to the nearest delivery room.

Lots of pressing, sweating, leg-pulling (the midwife recommends a birth on the side so one leg has to be pulled up during contractions by J. and finally me) and about an hour later at 8:45, our son is fully born. Near the end, when the head is just out, there's a funny episode where the three non-labouring women in the room, the mid-wife, assistant and midwife student are rejoicing over the fine baby head and trying to persuade it into not breathing, which isn't possible as the torso is still not out and compressed by J.

But it is over. Janne gets the child, I cut the umbilical cord, we get to see the placenta and Janne gets some stitches while the midwife try to distract her by talking about breastfeeding.

The midwife assistant was kind enough to give us a good pull at the foot so we got all the mucus screamed out of the throat

One happy new mother a couple of hours after the birth (and some breakfast) - if the little guy looks somewhat baked, that's because somebody forgot to tell the new father who was ordered to dress him that he was supposed to turn off the heating lamp in the box after giving the boy outdoor clothes on, oops - this was after the father nearly managed to burn his own hair by sticking it into the heating lamp

Apple, Google, beware - they had a magic tablet, when we got in they wrote our names on it, and by magic, when the birth was over there were happy flags and everything on it, I swear I never saw anyone get near it (the text is "first child, fine boy :), pregnant for 41+0 [weeks/days], congratulations with him")

Afterwards, we were kindly allowed to stay for four days at a patient hotel as a special service for new parents. Four days of getting to know the little guy, the three of us alone, apart from occasional visitors, getting help from midwifes to get breastfeeding going (whoever said the nipple is the only intuitive interface obviously wasn't a mother), and lots of good food. It was fantastic!

One happy, and somewhat tired, new father

The little guy in a transportable cradle we used to get him to the hotel restaurant - the book of hypnosis is used to pacify him temporarily

It's 12 days ago today. We've already learned a lot, all three of us. He's now looking around, taking in the world, folding his little fingers, trying to use his limbs, practicing looking angry, smiling, grinning.

One thing that really surprised me, in addition to the enormous wave of happiness which so far has steered us through sleep deprivation, occasional moments of angry, nerve-wrecking screaming; what really surprised me is how funny such a little guy can be. For instance, at one critical moment in the middle of the night we both hold our breath at a pause in the crying, looking at him, his eye brows pulled together, mouth turned downwards, will he accept our attempt at comforting him? And then he farts loudly, gives us a quick happy grin and falls asleep.

Closeup of the little guy 6 days later, the red marks above the eyes are now fading, they probably come from J. who gave him a good squeeze when the head was half-way out (the midwife ordered a pause)

One of these days we're going to give him his first bath. And I can't wait to see what he will do.

Cells and biology

2011-07-31T03:06:00.002+02:00

I recently borrowed a book from my sister who has a Master's in biology, Zoology by Dorit, Walker and Barnes, about 900 pages, intended audience is college biology students I believe. I've been reading it over the summer, and I must say it's the most interesting book I've read in years. I can't recall the last time I've learned so much in such a short time span.

The book goes through what stuff cells are made of, how respiration and cell division works, how protein compounds are made, how it is thought various solutions to problems evolved, how various features of species stem from physical limits and constraints in the environment. For instance, tiny animals don't have lungs. They don't need them because no part of their internal body is ever far from air or water. The complicated blood and lung system of humans is really an extraneous necessity required by our huge size.

The book then covers the various animal groups one by one. As it turns out, most of the stuff I at least tended to associate with humans and higher animals is really nothing new. Sex was not invented with the monkeys. Also air is a pretty inhospitable environment, as the book puts it, every animal not living in water is constantly living at the risk of desiccation.

One of the things I've learned is that the world of cells is an extremely complicated world in its own. A big animal, like a human, is really a big blob of cooperating cells, each cell being a living being in its own right, although of course dependent on the others. The cells in your foot are dependent on the cells in your mouth to swallow food and the cells in your intestine for taking in nutrients from the soup that passes through. And vice versa, no feet means no catching food, at least in nature.

It's a purpose of life, feeding your cells, and a good one I believe. But each cell is actually a pretty complex thing, with small bacteria-like sub-components called organelles like the mithocondria that take in oxygen and various other stuff and output cell fuel, ATP, not to mention the complex chemical machinery that replicates and executes the DNA code. Cells have a life of their own. For instance some cells in the body feed on the bacteria and other stuff that enters the tissue by engulfing and digesting it; that's an important aspect of the immune system.

Structure of a typical animal cell (from Wikipedia)

Likewise, the interaction between the cells is highly developed, not the least keeping in mind that the chemical cell machinery must at all stages be able to self-repair. It is also grown, unlike a human-made machine in the macro world assembled from ready-made pieces, all animals and all their parts must be made from tiny single cells that collect nutrients and out of them build bigger structures.

And everything is controlled by the laws of physics and long self-replicating double-helix strands of 4-character code that specifies what happens when.

It is simply amazing.

3D in the browser

2011-06-17T17:11:00.001+02:00

We've recently done a project where we needed to do 3D visualizations in the browser. It's not incredibly fancy, mostly showing walls and floors of rooms and some stuff in them.

Before we started coding, we were discussing what platform to run it on. After some going back and forth, we ended up deciding that despite the 3D requirement, going for a web app would probably pay off in the long run because we then don't have the hassle of installed software and perhaps worse, the problem of predicting what particular platform Microsoft will declare obsolete next year.

So in the end, we settled on the web, going for HTML and canvas for the 2D part and WebGL, the new OpenGL standard for the web, for the 3D.

There's only one problem, WebGL was so new that at the time, only Chrome had a release with support. Not long after, Firefox joined the ranks. But still, it's a technology you can't rely on unless you can control the user base and ensure they install the right browser.

Similar problems kept the 2D HTML canvas back until a project like excanvas turned up. Until Internet Explorer support is there, new standards are a no-go for many.

Thus Martin has been working on and has now released JebGL, a WebGL emulation layer for older browsers and most importantly Internet Explorer, back to IE 6.

It relies on the OpenGL support in Java, translating WebGL calls into JOGL calls in a Java applet. We've had a look at other options, such as using the Flash 3D API, but we needed support for some of the fancy stuff in OpenGL so it would be a pretty huge task to try to emulate that in a completely different API (on a lower level, that's what Google is doing with the ANGLE project to try to work-around bad OpenGL drivers).

So as long as Java is in, you can now rely on WebGL (well, barring bugs and missing features in JebGL). And if Java is not there, there's at least the option of asking people to install it, most browsers seem to handle that gracefully these days, rather than requiring them to switch to a completely different browser.

Organizational patterns

2011-05-30T12:40:00.000+02:00

We went to a seminar not too long ago in a local interest group on agile software development. One of the speakers was Jim Coplien. He's a good at it, speaking emphatically on what to do and not to do.

Also he had the interesting point that if you're going to try to change what a software development organization is doing, process improvement programmes turn out to be less effective than one thinks because processes tend to come from the underlying structures. So if you don't change them, people fall back after some time.

For instance, if you have the problem that the developers aren't good enough at coming up with what the users need, then instead of installing a process where they must fill in check lists to try to force them to listen, maybe you need to look at the fact that they aren't communicating directly with the users, but perhaps through a manager who dictates what should happen. If you change the manager's role, maybe the rest will follow automatically.

Anyway, he talked at some length about some organizational patterns he and others have come up with, which was interesting because many of the seem to describe what we're doing at IOLA. He's collected a big bunch in a book he's written (together with Neil B. Harrison), "Organizational Patterns of Agile Software Development".

If you're interested in software development and what works when you structure the work, the team and relationships to stakeholders and users, it's worth a read.

What's interesting in it is that most patterns may be profound in their effect, but are really lightweight in what you actually need to do. For instance, if you have the problem that developers are interrupted too much, you can sacrifice one person and make him a firewall that people have to go through. Simple as that. No big piles of lists and specifications and meeting minutes and memos necessary.

I think the big downside of the book is that it's not very friendly written. It's in the style of a reference in many places with only little background rationale and so many crossreferences that you need to have read most of the book to understand what's going. This probably works well for Jim Coplien when he travels around on his research and consultancy campaign to fix sick organizations, but I think the audience for a gentler style would be much higher.

Still, it has material I haven't seen elsewhere and it's actually based on research on succesful software teams rather than just handwaving.

GNOME 3 is nice

2011-04-08T23:36:00.000+02:00

So GNOME 3 is out and I decided to give it a try with the Fedora live CD (or should I say USB stick). I must say I'm impressed. Overall it's really clean and just works. The new application chooser is downright brilliant compared to the old menu. I've tested some of the previews of GNOME Shell, and the thing has come a long way.

Of course, Alt-tab is still broken, the default (and only) theme is a bit top-heavy, window movement wasn't quite smooth on Intel GMA 950, there was the occasional crash of the new settings stuff, but all in all not bad for a .0 release. And the new direction means that there are now people working on the core user experience, something that appears to have been more or less neglected for a very long time.

Plus much of the new shell stuff is written in Javascript so should be easy to hack on in case the designers don't come to their senses any time soon (there's already an Alt-tab hack).

A take on good, reusable Java design

2011-03-29T22:19:00.000+02:00

This is just too amusing not to pass on.

Spring time!

2011-03-23T22:03:00.001+01:00

It's that time of the year again here in Denmark.

The crocuses started flowering, and I managed to sit outside for quite a while in a short-sleeved shirt without freezing to death. There's no doubt. Spring arrived this weekend.

It's been a busy winter. I...

... managed to fall sick with flue-resembling symptoms and fever three times in 6 weeks, beginning with the Christmas holidays.

... shoveled a great deal of snow away, like last year.

... closed probably around 100 issues in the Flot issue tracker (and had the misfortune not to discover a well-meaning fork until a month after).

... mapped a big bunch of roads in OpenStreetMap, mostly centered around Janne's parents home. Thanks to the effort of volunteers, we've got aerial photos of almost the entire country and will, I think, within this year have a complete free street map of Denmark. I found out mapping is a fine hobby; it's a nice way to relax while gaining the usual open source feeling of contributing to something greater than yourself.

... read 16 books on local history of Vendsyssel and Himmerland, the regions where I live, told by old people that recount their youth around 1900-1930, mostly with farming, quite an eye-opener, really, and a great complement to the things we were taught in history lessons in school. I also read the Arthur trilogy by Bernard Cornwell (it wasn't quite as good as his Saxon book series, although maybe I'm just biased because I prefer the brought-up-as-a-Dane protagonist from those).

... wasted some time with Tremulous 1.2pre after having stopped playing Tremulous for a long time. It is a surprisingly difficult multiplayer FPS. I mostly play the alien side because I like the speed, but since I'm still not very good at it, this means I have to trick other players into coming close enough to be vulnerable.

... and annoyingly failed to get into the habit of regular piano practicing again.

Flot 0.7 released!

2011-03-22T13:11:00.000+01:00

I just released Flot 0.7! Highlights are multi-axis support, area charts and a pie chart plugin, and a big bunch of bug fixes, including an IE 9 fix and reworked replotting support that reuses the canvas elements to put an end to the memleak troubles. The complete list of changes is here. There are some API changes from the axis rewrite.

In related news, I've moved the source code repository to github to make it easier for other people to contribute.

It has been clear to me for some time that more of the things that are currently inside core Flot needs to move to plugins. As time goes, more specific things turn up on people's wishlist, and right now it's just not possible to satisfy them all without bloating the core. Having specific feature sets in a plugin also makes it much easier for people to customize them a bit by hacking the plugin code without having to maintain a delta with the whole of Flot.

Fortunately, some people have expressed interest in helping out, partly due to me taking a hiatus from the mailing list for some time last year, so with a bit of luck some of these things may happen in a not too distant future. Maybe we'll even get a better hyperlinked format for the documentation, something that has never really managed to climb to the top of my list of priorities so far.

Lyrics for GNY Beirblakken

2011-02-25T22:29:00.004+01:00

Janne and I had some trouble finding the lyrics for Beirblakken performed by GNY (listen to it on their Facebook page). Turns out it's from a Norwegian folksong Beiarblakkjen (which perhaps originally means a war horse). There are a couple of versions of it. The full reconstructed text sung by GNY is approximately this:

Der satte tre kjerringar under ei bro
- dei trea gullsko
Dei skapte blakkjen av mannblo',
- så vide fara dei løyndar ord

Dei skapa blakkjen af manne bein,
- dei trea gullsko
og Beiarblakkjen så kalla dei han.
- så vide fara dei løyndar ord

Og då dei hadde blakkjen gjort,
- dei trea gullsko
då sende dei han til kongjens port.
- så vide fara dei løyndar ord

Da kom'e Niklas fride,
- dei trea gullsko
sa: Eg tore blakkjen ride.
- så vide fara dei løyndar ord

Niklas red til han bli' mo.
- dei trea gullsko
Til støvlane hans bli' fulle av blo'.
- så vide fara dei løyndar ord

Dei første steg som blakkjen sprang
- dei trea gullsko
Da kom han sig til helvede frem
- så vide fara dei løyndar ord

Da han kom til himmelriks dør,
- dei trea gullsko
der hadde blakkjen voret før.
- så vide fara dei løyndar ord

Gravde dei ham i vigde jord,
- dei trea gullsko
så inkje ravnane drikke hans blo'.
- så vide fara dei løyndar ord

This Norwegian dictionary was most helpful in deciphering some words - "kjerring" means wife/old woman, "løyndar" hidden, "trea" tread, "fride" fair, "mo" tired/paralyzed, "vigde" consecrated, and "blakkjen" is a horse (of the old Norwegian fjord horse kind).

Bulk inserting Django objects

2010-11-09T20:18:00.004+01:00

I've added another Django utility member to my garden of Python, bulkops.py.

Currently, it consists of two functions, insert_many and update_many. They are intended to solve the problem of inserting a big bunch of objects in Django. If you use the standard save() or create() methods in Django, you get one SQL query per object. With thousands of objects, you'll simply drown in overhead in executing all these queries.

There are some alternatives out there, but unfortunately they are pretty complex and specific to a previous version of Django. Instead this code simple makes a list of tuples with the fields on the model and feeds that to the executemany() function in the DB API. Usage is really simple

for x in seq:
    o = SomeObject()
    o.foo = x
    o.save()

becomes

l = []
for x in seq:
    o = SomeObject()
    o.foo = x
    l.append(o)
insert_many(l)

It's tested to work with Django 1.2 and doesn't really touch the ORM internals so should be safe for at least some future versions.

Sadly, there's no support for associated many-to-many relationships. I actually needed that myself, but I don't think there's an efficient way of getting the primary ids when you do a bulk insert with the Python DB driver API. So one has to use a select query anyway afterwards, and then it was actually more convenient to do it by hand by rearranging the logic in the caller.

Cache busting in Django

2010-11-08T19:42:00.010+01:00

One of the really annoying things in web development is dealing with caching. Browsers incorporate an elaborate caching scheme to speed up common browsing operations, which is a really great idea. But a side-effect is that when you release a change on a site, you risk that some people get the new HTML while their browser is still using an old CSS or Javascript file because their cache hasn't timed out yet.

It has bitten me several times during development too, some browsers have really aggressive caching schemes. Debugging problems caused by a stale CSS/JS file is just not fun.

One way to fix it is to use a framework to run the CSS and Javascript through a combiner/minifier that at the same time outputs a versioned filename, e.g. by hashing the contents. There are even a couple of Django projects going down this route to make it happen automatically.

I haven't found one I liked yet, though. You have to modify views or templates and fit your paths into the system, it doesn't help with changing images, if you use third-party code you have to modify that too, and the minification step makes it pretty hard to debug Javascript problems; even if you isolate it to live, problems happen on live too, sometimes.

Really, if we take a step back from the minify dream, the simple solution here is to append some kind of version number to the file. Then new HTML should have filenames with new version numbers so clients can't used their old cached files because the paths don't match.

I've found a really simple way to automate this transparently for a project by modifying one line in settings.py. The idea is to hook into Django with a middleware class, search the generated HTML for local URLs and replace them with URLs with version numbers. This way, we can catch all CSS and Javascript references as well as images no matter where they occur.

Since common file systems don't provide version numbers on the files they store, we'll use the modification timestamp instead. And instead of renaming files to contain the version number, we'll just append a ?_=timestamp to the URL:

  <img src="/media/images/logo.png">

becomes

  <img src="/media/images/logo.png?_=4321234">

Everything after the ? is ignored when looking up in the file system by the static file web servers I've tested so far. So it just works.

As a bonus, when this is running, you can safely modify the web server serving the static files to set an expiration date far out in the future for files requested with ?_=xxxxx. This allows an aggressive, yet perfectly safe caching scheme to be employed by any upstream cache, including the browser.

The above trick can be done in about 50 lines of Python. You can grab the little self-contained middleware module modtimeurls.py from my brand-new Garden of Python. It has been in production on several sites for over a year. The only problem I've found so far was with a PNG transparency fixer for IE6 that assumed .png files would end with .png rather than .png?_=xxxxxx.

Stoicism

2010-11-02T15:23:00.001+01:00

This is philosophy in practice. One of the most interesting posts I have read this year.

Emacs learning curve

2010-10-11T14:19:00.003+02:00

Think this is old. But still funny.

Piano synthesis

2010-09-20T00:02:00.006+02:00

I've bought a digital piano. Its most important feature is head phones (not included). No more neighbour complaints.

Because of this, I've also been spending some time, actually too much time, on looking into piano sound synthesis recently.

The problem is that commercially available synthesizers built into digital pianos don't sound very good. Actually, for people who don't play the piano, they're probably fine. But that doesn't help me!

So I set out to find out what else is out there. Basically, there are two approaches.

Play back
One approach is that you sit down in front of a real piano with a couple of microphones and record the sound from all of the keys several times.

You can't just record one note and shift its frequency because the character of the tones change dramatically from the low end to the high end.

Furthermore, you can't just record each key once because when you press a key harder, the sound is not just louder but also changes character. For a pianist this change of character is very important. I don't know whether this is true for others, but at least for me my brain quickly adjusts itself to changes in loudness so without a change of character, the result is a flat sound inside my head.

So the end result is that you need a lot of recordings. 88 notes times at least 15 hardness levels and the decay of the low notes can easily last over 30 seconds before they die out. That's a pretty large amount of data to record and process; as I understand it, you need postprocessing to smooth out inconsistencies in the recordings. And still, in my experience, nobody has figured out how simulate a short staccato attack convincingly with a long sample. Not to mention other interesting things you can do with a real piano, once you get it singing.

Mathematical models
Instead of recording a real piano, another approach is synthesizing the sound from some sort of mathematical model, just like using ray tracing a modeled 3D scene to generate realistically looking special effect images in movies.

Now, some models start from the idea of trying to emulate the sound of the piano itself directly, based on observations like the sound being nearly harmonic so if we just add some sine waves at regular intervals, we've almost got a piano sound.

As far as I can tell these all sound really awful.

One of the first pianos. It was before the steel frame was invented. From Wikipedia.

I think the explanation is that generating sound is easy, even easier when you've got a computer to add all the sine waves you like, but getting an interesting sound is really hard. And piano makers have worked on this since at least 1720 using wood and metal, materials with plenty of opportunities for interesting sounds. So a fairly simple model is just not going to cut it, not when it's up against a 400 kg carefully refined wooden steel-string monster.

Another idea is to try to deduce how the various things inside the piano affect each other. If you look closer, the vibration of a string is a periodic push and pull at the ends of the string. If we follow the interactions from a key hitting a hammer that in turn hits the strings, causing them to vibrate, setting the big wooden plate inside the piano, the sound board, into motion, causing the air to vibrate, which is finally perceived as sound in our ears after undergoing reverberations from the room walls; then we can, in theory, reconstruct a perfect sound through the formulas from physics governing the interactions. A similar idea is used to simulate what happens when things collide in games these days, by the way.

Now, in practice, it's of course not so simple.

First, as far as I can tell, nobody has an accurate complete physical model from hammers to strings to soundboard to dampers. So the equations are not there - not yet, although there appears to be a few people working on it.

Second, modeling a whole piano in a naive manner is just not tractable in real-time with commodity computers because you need to simulate the whereabouts of a huge number of points at least 44100 times per second to generate a playable 44.1 kHz sound. Thus it is vital to identify what constitutes the audible important dynamics in the process and spending the time simulating them.

Despite the challenges, these physical models can generate quite convincing sounds. You can buy them, currently either in the form of Pianoteq or the Roland V-Piano. What's nice about them is that you can get them to sing like a real piano because they're actually simulating vibrating strings, not just turning recordings on and off.

Unfortunately, the models aren't perfect yet, and it's certainly audible, generally mostly in the middle registers, the reason being, I think, that at the upper end the sound is actually mostly the thump from the key hitting the bottom of the keybed, and at the lower end it's mostly about generating a really big set of partials so errors here are less noticeable.

Research on physical modeling
When I started looking at synthesizers I was mostly annoyed that neither the company behind Pianoteq or Roland disclose how they do it. Which probably explains why both haven't gotten it quite right yet. So I started googling for research.

It does not take more than a cursory glance to see that synthesizing achoustic instruments of various kinds is an active research topic. As far as I can tell there are two cooperating groups working on piano synthesis these days, at least in the English-speaking part of the world.

One group is at Helsinki University of Technology, some papers published last year, some earlier. Unfortunately, I got the impression that their model is not very convincing yet.

The other group is at Verona University, in collaboration with an Italian company and instrument maker Viscount. It's possible to google up some information about this group, but I think only Balázs Bank who's also been at Helsinki has an up-to-date home page.

I have spent some time working on a test implementation of the Verona model based on the paper describing it, and I'll dedicate the rest of this blog entry to my findings.

Modal piano synthesis
The Verona model is based on modal synthesis where one starts from the equations from physics and from them figure out at which frequencies the strings exhibit standing self-reinforcing waves; which means they last and are audible after several seconds, an eternity in the millisecond world of impacts.

The physical responses at these frequencies or modes or partials are then simulated with a simple second-order model. Second-order means that the simulation on each time step uses the previous two values. So for instance, for one string partial, the displacement at time n is calculated as y[n] = a*y[n-1] + b*y[n-2] plus a contribution from the hammer if it is present. If you're wondering how this is started when n = 1, well, we just assume the string starts at rest so y[n-1] = 0. The calculated displacement is basically the sound.

Hence, once the constants a and b have been chosen, the math is actually pretty simple.

Of course, in reality it is a bit more complex. The model keeps track of the hammer motion, simulates the impact on the string and the resulting forces and motions at the modes which include both transverse modes (up-down or left-right motion) and longitudinal modes (motion back-forth in the length of the string) plus some longitudinal motion stemming from tension changes by the transverse modes. In order to simulate the sound passing through the soundboard (and the room), the motion of the modes are added and convolved with a recorded soundboard impulse response.

The details are in the paper.

Disappointingly, if you implement what's described in the paper and enter some reasonable values of the constants, it doesn't sound like a piano. This is of course with the caveat that I might have made a mistake in my implementation.

However, as far as I can tell, there are at least two important things missing from this model. One is the dampers. All pianos have a felt damper connected to each key, the purpose of which is to silence the strings of the key when it is not pressed down. Pianos produce quite powerful sounds so without some way of silencing notes again, it would be hard to make quick pieces sound good. Surprisingly, there does not appear to be much literature on dampers. The only thing I've found is this paper from last year.

The other thing missing is some feedback loops. From the soundboard back to each string, between the strings, especially between the strings on the same key (most keys on the piano have more than one string), and between the different transverse directions on the same string. These feedback loops cause interesting dynamic effects.

In the paper the soundboard is simply dealt with by setting the decay rates of the modes according to a simple model. This corresponds to the soundboard stealing energy from each mode and not delivering anything back. I did an analysis of the piano samples available from the University of Iowa, the decays of the partials are plotted below together with the simple model fitted to the data.

As you can already see from the plot, this is not a terribly accurate model.

In the following, the spectrograms show at each frequency the level of sound over time, capped at 5 kHz and 7 seconds, I've generated them with sox. First a sampled tone, C3 at fortissimo.

I think the weird vertical bars might be echos from the initial thump. Now, if you use the simple function derived from the decay data, the result synthesized sound looks like this (sound example).

The thing is that the soundboard doesn't really follow the simple model in the way it responds to the strings, e.g. it has modes of it own. As a quick measure, I tried to set the decay to decay rate of the nearest measured mode, then it looks like this (sound example).

I'm not sure what the conclusion is. One attractive feature of the simple model is that only has a couple of parameters so it's easy to change around. I think if one has a more complete model of a soundboard, it might be possible to deduce from that more precisely how it absorbs the sound and set the decay rates accordingly. But still there might be some dynamic effects that aren't taken into account.

A thing one notices in the spectrogram for the sampled tone is the pulsating nature of the partials. They don't just die out exponentially. The pulses come from both the interaction between the different transverse directions on each string and also the interaction between the other strings on the same key.

The paper has a couple of suggestions for dealing with these. Basically it boils down to adding some extra modes that can produce beating. The thing is that if you add a mode close to another, the difference in frequency causes the sound to pulsate because of the interference between the two.

But there's no recipe for how to set the beating.

My conclusion so far is that you can't use a global value for the beats, they need to be specific for each mode. I haven't gotten around to actually measure these values from the samples. As a quick work-around, I'm right now setting a random beat for each partial; this improved the sound considerably. While measuring samples may be one way to fix the issue, I can't help thinking that this is one area where physics could solve the problem much more precise with less hassle if somebody could figure out how.

I haven't yet gotten to the point where this sounds realistic enough to warrant a full synthesizer that can eat MIDI. But I get the impression that Pianoteq is using modal synthesis too, so it's certainly possible. If you are interested in the code I've written so far, feel free to contact me.

If you're interested in how the piano works, a good resource is the Five lectures on the acoustics of the piano.

I'll conclude with a random note. The Verona paper only operates with one string per key, but actually a real piano has more so if you use physical meaningfully parameters and don't take this into account when computing hammer impacts, it's going to be off by quite a lot; for a hammer, there's a difference between hitting one string and three.

Software development and the waterfall model

2010-08-29T19:02:00.009+02:00

As mentioned, I recently read The Design of Design by Fred Brooks.

He covers a lot of ground, albeit not in any rigid fashion. As such, it is not a handbook, but more a collection of thoughts about the design process, most grounded in personal experience. Fred Brooks knows his stuff.

An interesting point he makes is that the waterfall model of software development is completely flawed and should be abandoned.

In short, the principle in the waterfall model is reason - good solid thinking: first we figure out all what's needed in a given system, then we figure out all aspects of how to build it, then we build it.

He has several arguments for why the model is flawed. One of the simple arguments is that it doesn't model how good designers work. Naturally, if you want everyone to follow a certain mode of operation, it would seem like a good idea to copy the mode followed by the good ones.

For me, the biggest flaw of the waterfall model and its companion, the big fixed-price project, is that it requires one to make the decisions up front where the uncertainty is greatest and the consequences of each decision most remote.

This shows up even in the earliest phases, e.g. when discussing whether to include a certain requirement or not. Inclusion may mean that the project will run over budget and schedule and produce a completely impractical never-used feature. However, the budget overrun may not even be visible in the design phase, but only show up late in development, and the futility of the feature may not show up until years after deployment where it is discovered accidentally that nobody uses it.

An important point in the book is that the design process is a learning experience, a process where one learns what's needed and what's possible. And what's elegant and what's good in the particular circumstances. I'd add that the same is true of the embodiment of the design, the code itself. Often people won't understand a system properly until they sit in front of it and have to start doing what they need to do. Deployment is always a learning experience. And similarly, it can be hard as a developer to assess the consequences of a paper design. When the code is written, maybe that fancy idea everybody liked beforehand isn't that hot afterall.

If one doesn't allow for learning, both for the receivers of the system and the people who deliver it, it's really difficult to get the software right.

By this thinking, the rational model becomes a bureaucratic way of delivering what might just turn out to be the wrong system.

Garden design

2010-08-14T22:38:00.018+02:00

I recently read The Design of Design by Fred Brooks, the guy behind the mythical man month and the magical silver bullet debunking. I'll save further commentary on the book for now, but one of things he mentions is that rules sometimes help the designer because it sets some boundaries within which creativity can blossom.

I have a couple of rules for my little patch of a garden.

A passiflora growing outside in the summer

A first simple rule is that there shall be no grass. None at all. While lawns are nice and everything, it takes a lot of labour to keep a lawn, ideally it needs mowing at least once a week. Even then, unless you have the perfect growing spot for grass (plenty of sun and water), chances are it won't be a pretty one. This is compared to the other beds which I weed maybe two or three times a year while otherwise enjoying the flowers.

Also everybody has a lawn. Most don't look pretty. No fun at all.

Another rule is that there shall be as little uncovered ground as possible. Naked fertile soil is a highly unstable labour-requiring condition, nature will have its way. It also looks silly.

Yet another, less simple principle is that the garden must grow organically, chaotically, like nature, as opposed to being a tidy, trimmed show case of culture. As few straight lines as possible. Make no mistake, it's still designed in the sense that I exert control, it's not nature; if I didn't manipulate it heavily, it would turn into a shrubbery in a few years. But I'm attracted to idea of a green wilderness, to seeing things grow vigorously.

So I tend only to pull up the most annoying weed, the species that are hard to get rid of, and I let the decaying plants stay where they die (from a pull-up or otherwise). It's easy and it works well, most of it is gone when the season is over.

I try not to have too many of the same species in the same spot. Nature seldomly has 10 similar specimens right next to each other at exactly the same age. This also offers some protection against various pests.

And most of the plants I've bought are from seeds or small specimens from the local super markets. There's no overall plan to follow, hence I can satisfy sudden impulses without getting into trouble. And I don't mind having to wait to see the results, watching things grow and looking forward to the adult specimen is part of the joy. It's like buying yourself a gift that you cannot in any way unpack until a year or two after.

A side-effect of the wilderness idea is that my little patch has a lot of small visitors. Which is certainly something to be happy about. Sitting a summer day in the sun, with bees buzzing around collecting honey from the flowers one has put in place many months ago, removing weeds here and there, noticing details one has never seen before, discovering new species that have invaded the soil on their own. I can think of nothing more peaceful.

Of course, I'm learning from this approach. This year I'm beginning to think I need one more rule. Plants on the pathways must be cut down no matter how interesting they look, otherwise it gets too cumbersome to get around, and then I seldomly get to see the outermost corners.

On being the the right size

2010-06-11T16:39:00.003+02:00

Long time, no write. And now I'm just going to link to an article about sizes, from 1928.

Javascript and Emacs Lisp

2009-11-12T13:25:00.004+01:00

Warning: if you don't subscribe to the One True Religion, this probably won't interest you.

With my recent affair with Elisp and the development in GNOME where some influential people are dropping C in favour of Javascript for GUI programming (we may wonder how many kilolines of boiler-plate C code they've churned out to reach that conclusion), I couldn't help thinking about replacing Elisp with Javascript in Emacs.

First a bit of introduction. Emacs is an incredibly old and incredibly smart text editor with a smallish C core and most of the intelligent stuff written in a Lisp dialect called Emacs Lisp. Lisp is a very flexible language that intermingles data and code. Making a system built on it extensible is relatively straight-forward. This is probably the main reason behind the success of Emacs, the editor has been extended in literally thousand of directions over the years. If you read my previous blog post, I'm using what nowadays is dubbed aspect-oriented programming to hack a built-in Emacs Lisp module.

It's my impression that even within the Emacs community there are few people who think Emacs Lisp is a great language. It's not Common Lisp, it has some weird idioms and as a small Lisp dialect it still lacks the simplicity of Scheme.

However, weird idioms aside, in my humble opinion the big problem with Emacs Lisp is momentum. When Emacs was started, Lisp was hot. People were thinking that Lisp would be the future. Back in those days, they even had hardware that ran Lisp! The mind boggles. Lisp was an obvious choice for a dynamic extension language.

Today much has changed and few people are using Lisp. The obvious choice for a dynamic extension language today is something else. Basic, Python, Lua, etc. Or Javascript.

A die-hard Lisp fan would think the solution is another Lisp, but the official GNU Scheme embeddable language Guile looks pretty dead to me. While learning Lisp is a worthwhile goal that will definitely teach you some lessons, I think there's good reason why most programmers aren't writing parenthesized lists all day long.

By switching to Javascript, Emacs would once again be ahead of the curve. And the Emacs maintainers would be relieved of having to maintain a language interpreter with its related problems. For instance, and I don't know if things have changed recently, the garbage collector in Emacs used be pretty basic. Probably because when it just sort of works, nobody wants to mess with it anymore, people naturally want to focus on writing the best editor possible.

I don't think the idea would fly, however, for several reasons. There is an enormous body of Lisp code out there for Emacs. And since Lisp is the extension language for Emacs, everybody contributing to it is a Lisp hacker. It's hard to see how this actually pretty large group of people making up the community would accept anything else than a Lisp. And in the past, even proposals to reform Emacs Lisp or switch to other Lisp dialects have all failed.

To add to that, Javascript has a bad reputation, especially among people who haven't done any client-side web development lately.

Thus I think the only way Javascript in Emacs could happen meaningfully is by targeting the interpreter: replace the Emacs Lisp interpreter with a Javascript engine like Spidermonkey or V8 and a bridge that translates the Lisp and makes Lisp symbols available to Javascript and vice versa. So all Lisp code would run unaltered, and everyone can continue writing Lisp as they do today.

The aim would be less code to maintain in Emacs and the benefit of a well-maintained optimizing engine that is getting faster every day, with several independent free ones to choose from.

I think you could maybe sell that idea to the Emacs maintainers. A project like CEDET that replicates the code parsing engines of modern IDEs would definitely benefit from a speedup.

As a side-effect, it would then be possible to extend Emacs with pure Javascript. And a growing horde of web developers will suddenly find it much easier to extend Emacs.

I had a very brief look at the C source of Emacs, and it looks like you would have to rewrite and port maybe 10-20.000 lines of C code. So I think it's doable.

Of course, there are several open questions. How hard is it to do the translation? Can a Javascript engine actually run Lisp-translated-to-Javascript code efficiently? Is it a good idea at all or just a futile exercise in chasing the latest fad of the day? I am not sure. Javascript has been around for a decade and the web is still growing as a platform, so I don't think it qualifies as a fad. And GNOME is currently betting its future on it.

Another option would be to just embed the Javascript engine on top of the current core. As far as I know, that has been tried already with Python, and didn't work out. I don't think people will use it unless it is bundled with Emacs and sitting right in the core. After all, most customizations start out with a little hack building on existing code in Emacs.

Django shell in Emacs

2009-11-11T23:47:00.004+01:00

Yet another recipe. One of the neat things with Python is its REPL, the interactive shell. When working with Django models, you need to setup a couple of things before you can use the Python shell. Django provides a shell command for doing this, but it makes it difficult to use the built-in Emacs Python REPL.

I was talking to Anders about it the other day when it occurred to me that it could be a quite powerful environment.

The main problem with the Python shell is that it lacks easy reload support. So you start it up, import the module you're writing and run a couple of functions to test it. Discover an error, fix it with Emacs, save, and then you're in trouble because you can't reload the module in the shell. You have to quit the shell, restart it and reimport the module. History helps but it's still silly (well, actually it's a blatantly stupid oversight, maybe in the top 3 of things to fix in Python to speed up development time).

But with a shell integrated in Emacs, I could solve this problem, right? So I set out to write the necessary Elisp. Dump the following in your .emacs, and it should magically work:


;; run Django shell when editing Django Python code

(defun get-file-in-upstream-dir (location filename)
  (let* ((dir (file-name-directory location))
         (path (concat dir filename)))
    (if (file-exists-p path)
        path
      (if (not (equal dir "/"))
        (get-file-in-upstream-dir (expand-file-name (concat dir "../")) filename)))))

(defadvice run-python (before possibly-setup-django-project-environment)
  (let* ((settings-py (get-file-in-upstream-dir buffer-file-name "settings.py"))
         (project-dir (file-name-directory settings-py)))
    (if settings-py
        (progn
          (setenv "DJANGO_SETTINGS_MODULE" "settings")
          (setenv "PYTHONPATH" project-dir)))))

When the Python interpreter is started, the hook above looks for a settings.py file in the parent directories. If one is found, it sets up Django and also sets the PYTHONPATH to the project toplevel.

Use the code with C-c C-z to show interpreter window, C-c C-c to evaluate buffer, C-c C-r to evaluate region, etc. (they're in the Python menu at top).

You'll probably want the following snippet too, it disables the warning that a Python process is still active when you quit Emacs:


(add-hook 'inferior-python-mode-hook
          (lambda ()
            (set-process-query-on-exit-flag (get-process "Python") nil)))

I think python-mode should do this by itself, but in the end I didn't have the energy to report a bug and fight with the Emacs maintainers.

So did it fix the import problem in Python? Of course not. The shell inside Emacs still can't reimport a module. Sigh. But at least it's easier to test the current module because you can easily evaluate the whole buffer (which works fine).

Flot 0.6 released!

2009-10-23T15:48:00.004+02:00

I just released Flot 0.6! Highlights are a new plugin system, or at least the beginnings of it, lots of new features (most of them in plugins) and bug fixes, including fixes for IE 8. Check out the complete list of changes. Note that there are some API changes.

There were still some things I would have liked to do before the release, but it has been dragging along for far too long.

Otherwise the future is looking bright. The plugin system is not complete yet, but I think the general direction is OK, and will allow third parties to extend Flot considerably without going through me.

On the technological front, canvas support is gaining traction in the browsers and even support for what used to be advanced stuff like text rendering and export to image is getting more common place. I suspect we will soon have to think hard about whether to continue with the current labels-are-HTML approach or switch to rendering them directly on the canvas.

Completely unrelated to Flot: I haven't blogged for some time. It's entirely due to lack of time, I have been preoccupied, or gone crazy is probably closer to the truth, with another hobby project which is still some effort away from being useful.

Using CSS3 box-shadow with IE

2009-08-14T19:23:00.019+02:00

At YayArt, we show a lot of images. An important part of the presentation is subtle drop shadows.

My original solution to that problem was a Javascript hack that inserts a couple of 1 pixel wide divs at the edge of any element with a shadow class. However, although I have tweaked it several times, it has always been a bit slow. For instance, for the shop front page we're currently rendering more than 60 elements with shadows which takes a couple of seconds with Firefox and Firebug on my old trusty PIII 950 MHz laptop. I'll note that this is actually an order of magnitude faster than the (more general) code I based it on from somewhere else. The really tricky part was decent-speed IE 6 support because that actually required measuring the DOM elements.

However, Firefox 3.5 was released about a month ago, and one of the good news was support for the box-shadow CSS3 property. It's a relatively simple property for adding drop shadows. So given that Safari has it, and has had for some time, and most Firefox users will soon have it, it starts getting interesting. For reference, about half of our visitors are Firefox + Safari (+ Chrome which uses the same rendering engine as Safari).

If you search for box-shadow, some are already using it to add a bit of extra embellishment. However, on YayArt, it's an integrated part of the design. So I had to figure out how to make it work on all browsers.

I ended up with a three-way solution. For recent Firefox/Safari/... I use box-shadow. For non-box-shadow supporting browsers, I fallback to the old hack. And for Internet Explorer, I'm using a new hack based on the proprietary filter stuff in IE.

Here's a quick sketch. I put this in the general stylesheet:

.shadowed {
-moz-box-shadow: 2px 2px 3px #969696;
-webkit-box-shadow: 2px 2px 3px #969696;
}

This makes it work with browsers based on the Gecko and Webkit engines. Some people recommend adding a plain box-shadow line, but I need full control over which browsers are responding, and besides it seems to defeat the purpose of the browsers using the -engine- prefix. Here's a demo:

If your browser is recent enough, you can see a shadow here.

Now for the fallback, I need to discover in Javascript whether the browser is using the CSS:

function supportsBoxShadow() {
var s = document.body.style;
return s.WebkitBoxShadow !== undefined || s.MozBoxShadow !== undefined;
}

The assumption is that the symbols will be defined only if the browser actually draws the shadow. This seems to be about right, the only exception I've found so far is the Webkit port to GTK (there's a bug report open).

Final point is Internet Explorer. As it turns out, there is a DropShadow filter. It's not terrible useful for this purpose, however. There are no soft edges, and it shadows the content rather than the containing box (it simply redraws the content with an offset in another color). So for my purpose I need to set a background color to ensure it's drawing a rectangle and not a text shadow. You could probably hack the lack of blurring with another filter, but in the end I went with a simple three-one-pixel-wide-lines approach that looks like my old hack but is much faster (updated, replaced the filter with a more appropriate one as suggested in the comments):

.shadowed {
background-color: #fff;
zoom: 1;
filter: progid:DXImageTransform.Microsoft.Shadow(color='#969696', Direction=135, Strength=3);
}

This is served for IE only. The zoom: 1 is a hack to ensure that the element gets a layout (to work around the usual IE 6 bug).

So there you have it. Practical drop shadows. I will have to adjust the code a little bit as the other browsers get support, but I can live with that.

Emacs 23...

2009-08-07T20:01:00.008+02:00

... is out and has been so for over a week. Why didn't somebody tell me?

In any case, I upgraded to Emacs 23 on my Debian unstable laptop. This means a new font. I put Emacs.font: Bitstream Vera Sans Mono 9 in my .Xresources file and am enjoying it so far. It's a bit muddy so I wasn't sure at first, but after having stared at it for a while the old font is just too skinny. Note that configuring the font in .emacs is still broken (sigh), in that it causes the window to jerk when you start Emacs.

I've been lobbying for changing the defaults over to make copy-paste between Emacs and other X applications (from this century) work properly. Apparently, (setq x-select-enable-clipboard t) is now better, but still not according to the spec. And it's not default yet.

I noticed some Tramp updates (for accessing remote files) in the NEWS and decided to give it a spin for the first time. Try C-x C-f someserver:myfile.txt and with a little patience, you're editing a remote file locally! Very neat.

Feeling adventurous I also tried IDO, an update of iswitchb that works with file find (so overrides C-x C-f). I'm turning it on with

(ido-mode 1)
(setq ido-enable-flex-matching t)
(setq ido-use-filename-at-point t)

replacing iswitchb and ffap settings. We'll see how it goes.

Finally, Anders mentioned a new Javascript mode, js2-mode, which I'm definitely going to try out.

Using X11 over high-latency ADSL and an AltGr NX solution

2009-07-04T15:15:00.005+02:00

X11 has been bad at high-latency links for a long time. Supposedly, the Xlib replacement XCB would fix some of that, but so far no-one seems to have picked it up. I found some work on GTK+ in 2006 but nothing since then. Sigh.

So there's this other proprietary solution, Nomachine NX with a free companion FreeNX. I decided to give it a try today, and after spending some time contemplating how to get it installed on Debian, it's not in the main repository because the basic architecture is broken (it's including it's own copy of X and SSH and some other stuff), I finally realized that the easiest setup was probably just downloading and installing the .debs on Nomachine's web site.

I was hoping to get ssh somehost then emacs & working, but it's a bit weirder than that. You have to start up a graphical connection thing, in which you can fortunately select a console. This gives you a remote xterm. The rest is simple.

I must say I'm very impressed with the performance. Of course, I'm only using Emacs but it works really well. It's like being connected through a local network.

Of course, once I started coding, the first time I needed a curly brace, which happens soon enough with Javascript, it was bam, back to start. AltGr sends an arrow-key left. After some digging and hair-pulling, it turns out I needed to run setxkbmap [layout] where layout is "dk" or "de" or similar. Except setxkbmap wasn't installed. So I installed it. Then it was missing it's data. I installed that too. Now everything is fine.

If someone would now just port this kind of thing to the default X.org setup, then I would be a happy man. No more console Emacs sessions.

Safe truncation of HTML

2009-05-01T15:46:00.009+02:00

Another recipe, this time for solving the problem of truncating a piece of HTML, i.e. turning "<p>Blah blah blah</p>" into "<p>Blah ...</p>". Google didn't really turn anything useful up, except for a suggestion of using a full-blown HTML parser and then simplifying the result, so I thought I would post the snippet here for Google to pick up.

The code never splits a valid tag or character entity. It should be able to cope with invalid HTML too, but note that it won't sanitize it. So for instance, if there's an unbalanced <a> in the source string, it won't fix it. Character entities are dealt with by counting them as one character.

The basic idea in the snippet is that we just skip through the string unless we encounter an opening tag. If so, we see if we can find the corresponding end tag and save it for later. When we got enough non-HTML characters, a ... is put in and any saved but not yet used end tags are added to the output.

Here's the code in Python (it's easily turned into a Django filter), I aimed for readability rather than ultra-regexp ninja tricks:

import re

tag_end_re = re.compile(r'(\w+)[^>]*>')
entity_end_re = re.compile(r'(\w+;)')

@register.filter
def truncatehtml(string, length, ellipsis='...'):
    """Truncate HTML string, preserving tag structure and character entities."""
    output_length = 0
    i = 0
    pending_close_tags = {}
    
    while output_length < length and i < len(string):
        c = string[i]
        if c == '<':
            # probably some kind of tag
            if i in pending_close_tags:
                # just pop and skip if it's closing tag we already knew about
                i += len(pending_close_tags.pop(i))
            else:
                # else maybe add tag

                i += 1
                match = tag_end_re.match(string[i:])
                if match:
                    tag = match.groups()[0]
                    i += match.end()
  
                    # save the end tag for possible later use if there is one
                    match = re.search(r'(</' + tag + '[^>]*>)', string[i:], re.IGNORECASE)
                    if match:
                        pending_close_tags[i + match.start()] = match.groups()[0]
                else:
                    output_length += 1 # some kind of garbage, but count it in
                    
        elif c == '&':
            # possible character entity, we need to skip it
            i += 1
            match = entity_end_re.match(string[i:])
            if match:
                i += match.end()

            # this is either a weird character or just '&', both count as 1
            output_length += 1
        else:
            # plain old characters
            skip_to = string.find('<', i, i + length)
            if skip_to == -1:
                skip_to = string.find('&', i, i + length)
            if skip_to == -1:
                skip_to = i + length
                
            # clamp
            delta = min(skip_to - i,
                        length - output_length,
                        len(string) - i)

            output_length += delta
            i += delta
                        
    output = [string[:i]]
    if output_length == length:
        output.append(ellipsis)

    for k in sorted(pending_close_tags.keys()):
        output.append(pending_close_tags[k])

    return "".join(output)

Test if uploaded file is JPEG, PNG or TIFF

2009-04-17T17:19:00.008+02:00

I've been looking at some of uploads that went wrong on YayArt lately, and it turns out that people sometimes submit images with the wrong extension, e.g. "someimage.png" when it's really a JPEG. This confuses the image backend we're using to process large images, VIPS, so it reports back an error.

I did a bit of googling, and it seems the easiest way out is to simply check the first few bytes of the file for magic numbers. So here's a bit of Python code for checking for whether the file data belongs to a JPEG, PNG or TIFF image:

def is_jpg(data):
    return data[:2] == '\xff\xd8'

def is_png(data):
    return data[:8] == '\x89PNG\x0d\x0a\x1a\x0a'

def is_tiff(data):
    return data[:4] == 'MM\x00\x2a' or data[:4] == 'II\x2a\x00'

If the file is already on disk, you can grab the first few bytes with

f = open("somefile.jpg", 'r')
data = f.read(11)
if is_jpeg(data):
    ext = ".jpg"
elif is_png(data):
    ext = ".jpg"

Of course this won't test that the whole file is valid. But it's easier to do that afterwards with an image library once the extension is correct.

The magic numbers are documented in the specifications for the formats. You can also find some help for other formats in the source code of the file command on Unix systems.

Update: I'm liking this so much that I ended up putting it in a separate file and making a convenience function for getting an extension like '.jpg'. Grab the Python file here. I also added support for GIF. Here's another easy reference for magic file numbers.

Second update: I've updated the code, there was a bug detecting JPEGs from certain digital cameras that put Exif data in the first segment. Suffice to check the two first bytes of the JPEG, then the problem does not occur.

Good art and bad art

2009-04-08T12:07:00.015+02:00

There's a couple of things I've learned in the process with YayArt that I'd like to share. First, the key question: what is good art?

If you're like me, you'll probably think that this is impossible to say, or at least so difficult to answer that you need to have studied art for many years, be part of the art elite, to answer in a qualified way.

That's right. But in my opinion also wrong.

One approach to definition of good art is the Aristotelian approach. We try to pinpoint the common traits of the subject. This is a slippery path, but we can still do some.

First, art is a result of craftmanship which is performed with the purpose of engaging you in some way, by moving you, touching your feelings. It must be more than just an everyday thing.

If you're part of the art elite, you would add to this that good new art must bring something fresh to the table every time. It must be innovative. The past is important, to the point that new works that in the past would have been considered good art seize to be interesting.

If you're not part of the art elite but like me, you're probably more interested in the looks than prior art. It must be pleasing to the eye, or at least pleasingly unpleasant. The small details that are hard to define, but nevertheless obvious, must be right.

Good or bad art?

There's a conflict between these opposing views, and I believe this conflict leads directly to museums and obscure works like the infamous diamond skull by Damien Hurst, which most people are likely to agree is pretty ugly.

Why do most people hardly ever go to museums for inspiration? Is it because they're not susceptible to art? They lack the gene that enables them to experience art? This seems like a dubious explanation. Witness the commercial success of photography and Hollywood. Pictures work. And good art is, by definition, capable of engaging people.

Good or bad art?

The explanation is probably rather that going to a museum is cumbersome, and when you finally get there you might only see art that is definitely different, but doesn't engage you. It doesn't speak to you. Art that doesn't engage you is bad art, by definition - for you.

With YayArt we're trying to make it less cumbersome to see the art. And we are trying to replace the elitist notion of good art as being innovative art by something else. It's not that having an art elite is bad. It's just that we think there's a large proportion of people who could enjoy and benefit from art if the current art market would serve them. Or a new art market emerged.

The definition of good art we're using is a platonic definition, i.e. it works by examples. For instance, to define a chair to native from the jungle we wouldn't talk about furniture and four legs but instead point at the six chairs around the dining table and say, now do you understand?

At YayArt we show you a piece of art and ask, is it good or bad?

Good or bad art?

This is not abstract, so it is easy for everyone, not just art pros, to answer. What we're implicitly asking are questions like, does it move you? Do you like it? Everybody can answer that.

And when you think about it, this is really what good art is about.

Ketil Bjørnstad

2009-03-26T15:30:00.004+01:00

I finished reading To Music by Ketil Bjørnstad the other day.

It's mixing a realistic presentation of an young aspiring pianist and his chaotic life before his carrier starts with a sneaky, subtle weirdiness, a bit like Twin Peaks; very unnerving but at the same time funny. Of course, once I started on it, it was impossible to put down. For any interested in their general health condition, I recommend staying away. Seriously.

It makes me wonder what I look for in a book when I go to the library. My previous book was by Alistair MacLean, which today requires an humorous attitude towards his anachronistic a-man-is-man world views to read, at least for me.

Beautiful code

2009-03-20T20:25:00.013+01:00

Anders pointed me to a talk by some Ruby guy (Marcel Molina) speaking about beautiful code. I must admit that I was a bit sceptical, but Anders was very convincing.

The verdict?

When it comes to ruminating on software design, I think there's a big nasty trap which he unfortunately walked straight into (as foreseen) with few extenuating circumstances (not expected).

The problem is that the field is hidden in a fog of mysteries, buried in what's governed by intuition and tacit knowledge rather than explainable ration. Good programming, hah, that's an art, nobody can tell you how to do that!

What we need here is to be able to talk about the thing. More ration, less intuition. Trying to explain things as beautiful or not is a step in the wrong direction. It's romantic self-indulgence, like when you look at people younger than you and think, people these days... An operational set of values for evaluating code is an essential thing for an aspiring programmer. How can beauty be operational if you don't even know how to argue about it with a fellow programmer?

Beauty? Digital graphic art from YayArt.

In any case, I think his main point can be summed up to: ensure that the code is as small as possible, as clear as possible and does what it's supposed to do. As I mentioned last time I wrote about software design, I think this can be simplified to make it easy to understand.

Save the interesting but tricky beauty discussions for things like this.

The repair shop

2009-03-14T12:52:00.007+01:00

One of the annoying things about modern home electronics is that they are black boxes. Like a microwaver, cold food in, push buttons, warm food out. Who knows what happened inside?

Most people are probably happy they don't know. But a life without curiosity is a life with less passion. I was fortunate enough to get an excuse to take apart two home appliances recently because they broke. And it wasn't even my fault.

The first successful repair was Janne's younger sister's laptop. The power chord had been loose for some time, at some point the laptop simply stopped working with symptoms of no power.

The anatomy of the repair is not too far from debugging software you don't know (I've previously talked about debugging software you've written yourself). Take the system apart, examine the individual components, collect information, reason. Identify the faulty component and apply the easiest fix you can think of.

Taking apart the laptop

The faulty component, the switchboard that the power chord plugs into (visible at the finger tip)

I had an edge here. I'd heard about this problem before. So instead of giving up beforehand, it got me thinking that if it was a common problem, the remedy would probably be well-known among electro-hobbyists. A bit of googling revealed that some people had success with soldering off the small house the power jack is inserted into, and replacing it. When we examined the house, something was in fact wrong with it, as witnessed by a simple currency test with a multimeter.

So we set out to solder it off. Unfortunately, that didn't work out. The soldering metal wouldn't melt properly. Instead we ended up replacing the whole component, i.e. the small part of the mother board that the ports were sitting on. The web is extremely handy here. Just jot down the spare part number and search for it, or parts of it.

We ordered a spare part which ended up costing less than 1/10 of the price of a new laptop, and put it in.

In reality, this was a bit harder than it sounds, because the first two places that turned up on the web didn't actually have the part when we tried ordering it. Also because taking apart a laptop is a bit complicated because of all the tiny screws and chords and plastics that have to be bent in awkward positions, sometimes more violently than you'd like to think about.

Last week, I had a much nicer experience changing the BIOS battery on my trusty old Pentium III laptop. Ugly looks, but nice internals.

Continuing on this saga, I've also fixed our microwave oven. Microwave ovens are a bit more complicated in the feature set than I hinted above. They also make the food turn around, slowly. But our oven stopped doing that. So I took it apart, hoping that it would be a bad connection.

One non-rotating Samsung microwave oven

The faulty component in the microwaver

However, there was nothing wrong with the connections inside the oven. The funny thing about hardware is that as soon as you take off the shell, it looks complicated and futuristic, but in reality it's just a set of interconnected smaller components. In this case, the sealed turntable motor was broken. Again, with the component number it wasn't hard to find a spare part on the web.

However, this presented me with an interesting real-life dilemma. Is it worth 22 £ to me to be able to see the food inside the oven carouseling past? My first answer was no. I already got to see the oven inside, it's actually pretty simple, and identify the problem. I didn't have to actually fix it.

A couple of weeks later, the oven fried a very useful corn bag we're using to loosen stiffened neck muscles, a useful cure for some kinds of head-aches. The oven had burnt a hole at one particular spot in the non-rotating bag.

So today I installed the spare part. Works like a charm.

This may sound silly, but fixing supposedly unfixable things is really rewarding. You feel powerful and virtuous.

A couple of extra garden pictures:

Enjoying the spring sun at noon

Next day it's snowing

Spring!

2009-03-10T19:21:00.013+01:00

Spring is upon us here in Denmark! The signs are subtle but sure.

My snowdrops are flowering.

Galanthus, snowdrops

The garden looks as if it is almost dead, but upon closer inspection small green leaves are shooting up. And the ground is soft and wet, not hard and frosty.

My mostly dead, but not quite!, garden

It's getting warmer, my trusty winter coat and wide scarf is getting too warm for driving up the hill on bike.

Me and a tame deer in Arden in December

Yesterday a pair of ducks had occupied the pond in the park. This morning two coots were grassing when I passed downhill. And in the evening blackbirds are singing.

Yay!

PS: If you're quick, there's still time for our special offer for new art on paper and canvas on YayArt.

Ideology and pragmatism

2009-03-08T12:58:00.006+01:00

Perhaps I failed to mention the most important point in the thing I posted about capitalism the other day. Actually, I've been thinking about analogies between distributed software architecture (big systems) and society (a big system) for long, but I regret having spent most of the post about it because that kind of ego-centered pattern-matching is usually in itself pretty tiresome to read about.

A more interesting point is that of ideology versus pragmatism.

Interestingly, there's a similarity to them. When you believe in an ideology, you'll defend your choices with the end justifying the means. We may leave the poor bastards in the mud or we may kill half a million Iraqis or we may suppress people who think different from us and destroy the historical monuments of our past, but in the end we're defending freedom or equality and that's more important than anything else. It will prevail, and everything will be better than it once were.

When you believe in pragmatism, you'll defend your choices with the end justifying the means. We may spend ten times as much fixing the minds of violent criminals as the traumatized minds of their victims, we may have to see half of our help to poor nations disappearing in corruption, we may have to accept that people with big salaries get the lion's share of a tax cut. But in the end, we can prove that everyone is generally better off than with the alternative.

Religious pragmatism (from Flickr)

Pragmatism is discomforting. A true pragmatist will defend actions that are against the values that person believes in. Of course, ideology is discomforting too, as it requires you to suppress the truth.

This is all fine in theory. Most people will probably agree that we should base political decisions on ration and facts rather than beliefs.

However, in practice nobody knows for sure what the future outcome of a decision is. This is worse in some fields than in others. There's also the aside that without strong personal beliefs, lots of the valuable work being done in this world would probably cease to happen. Ideologically founded people will put lots of energy to an unselfish end in satisfying their drive.

I'm personally ideological when it comes to environmental issues. I would like to think, self-indulgently, that it's because I'm thinking further ahead than people in general but the truth is that I don't know why. As the windmill production in Denmark has grown to an international leadership, this has become more of a pragmatic decision. To my dismay, the right-wing in Denmark then blocked the whole windmill programme for most of the 00s out of what I see as ideological issues (money rests better in people's own pockets).

However, in other areas I find it easier to be pragmatic. Dismissal of known facts is something not to be taken lightly. This is especially true when it comes to issues where the ideological argument is persuit of justice or equality. The world is not just (sometimes children are molested), and we're not born equal, and although there's nothing wrong with building a system that tries to ameliorate these things, it's important to keep focus on reality, not the ideas.

Actually, that leads me software design again, but I think I'll save that for another blog post. :-)

Caching on the web

2009-01-27T17:49:00.006+01:00

When you build a web application intended for a large scale, there are two separate performance issues when it comes to generating the HTML the server spits out.

One is server load. How many visitors can the server cope with?

The other thing is response time as seen by each visitor. How long must I wait for the page to load?

When using a relatively slow dynamic web framework (as Python + Django), the issue with server load is mostly related to CPU time. If you read the Django docs, the recommended solution is caching, i.e. instead of generating the HTML on each hit, we generate it once, store it for a while, handing out the stored snippet until it times out. It's a simple idea that works incredibly well.

First question is where we do the caching. To reduce the workload as much as possible, we want it to happen as close to the browser as possible. For a low-cost solution that works for everyone, we need to stick to the same physical location as the web server, however.

With Django on Apache, you've got some relatively fast C code serializing the requests to separate Python processes running the Django code. The process model limits the concurrency level possible because each process gobs a fair amount of memory, and Python is just generally an order of magnitude slower than C. So while putting the cache in the web framework is arguably very convenient, it's also pretty slow. From my experiments, one or two orders of magnitude.

Ideally, we'd just run a simple process written with speed in mind in front of Apache to do the caching. Afterall, the idea is relatively straightforward. Enter Varnish, a web cache written in C.

With Varnish in place, you know that the server only has to generate a page once per timeout. Some of the pages on YayArt take about 1 second to generate because they need to do multiple complex database queries. Let's say we have a beefy server that can do 10 of these in parallel. Then we could support 10 requests/s. With caching in Varnish, we can scale up to whatever it takes Varnish to retrieve a string from its cache and serve it. You can easily reach 6000 requests/s. For comparison, 100 requests/s is 8.6 million hits/day.

However, as it turns out, while simple whole page caching can solve a large part of the server load problem with one blow without giving up the convenience of a dynamic web framework, it's not necessarily the solution to the response time. If the page is in the cache, the response time is close to perfect. But what if it's a slow day with few visitors and it's not?

The general solution is to precompute the answer. Unfortunately, how to do this is more application specific.

You might be wondering how to combine caching with personalized pages. On YayArt I use Javascript to integrate the per-user data into the page through AJAX. So the per-user server load is reduced to processing the actual per-user data instead of a whole page.

Market economy

2009-01-25T11:02:00.003+01:00

For Christmas, my parents bought me 3 months of subscription to Information, a small Danish newspaper with a focus on analysis and criticism of the daily politics. The audience is mostly intellectual (and also mostly left-wing).

Information is interesting for several reasons, one of them being that major politicians in Denmark, the people who actually make the rules for the rest of us, regularly post in the newspaper, and thus presumably also read it. For an old internet-addict, that's a whole new experience.

There's been a lot of talk lately about capitalism in the face of the current financial crisis. The thing is that over the past 20 years, almost everyone in the political landscape has been pursuing a the-more-market-economy-and-deregulation-the-better strategy to a some degree. And now it turns out that too few rules for the gamblers in the financial sector have put us into a global recession. So people are, again, beginning to question whether capitalism is such a hot idea after all.

When I went to high school, I always thought it was possible to do better. Because there is such an obvious waste in our market economy. For an example, go to your nearest supermarket and look at the shelves with shampoo and hair products. Or the shelves with soft drinks. Or the shelves with morning cereals for children. And ask yourself, how much value does all these colourful and overly expensive things bring to our society? Or the really classy ads that never mention any factual qualities in their products, but instead try to install in us an irrational idea of their products, because of the packaging so to speak, bringing improvements to our lives.

I ask you, how can drinking water with added sugar and various brown chemicals make you cool? Because a large company has invested enormous efforts in convincing everyone that it is so.

Surely ration can do better than that.

However, at university I spent a couple of years working with distributed systems. The most important lesson I learned is that centralized systems are bad, unless the scale is very small. Decentralized (peer-to-peer) systems are more adaptive to change, more robust and much more efficient - several orders of magnitude as the system scales. Yes, there's waste, redundancy, suboptimal behaviour.

But it's self-organizing. If I want something from a peer, I just ask - I don't need to contact a central authority which then has to decide how to respond to the change. Overall it just works incredibly better.

I think that the same is true of society. With a market economy, decision-making is decentralized, in spite of the tendency for old industries to advance towards monopolies. In the large scale, this decentralization is unbelievably effective compared to a centralized control because of the complexity involved.

It's also a lot more free than a democracy. In a democracy, 55% can decide for the remaining 45% (that's why we haven't built any new windmills in Denmark lately). With a market economy, everyone decides for himself. Within the limits of the market, of course.

However, not all is good. One of the more peculiar problems of market economy is the drive towards monopolies, i.e. a collapse of the market (read a treatise of Karl Marx's works to understand why). Like any game we set up, it needs rules.

Karl Marx

And the market cannot do long-term thinking. As we have just seen, it can happily drive over the cliff edge because of a phenomenon called the tragedy of the commons. It doesn't hurt me a lot that I exploit the system: I get the whole benefit, the downside is shared between everyone.

The tragedy is that even if some market players want to stop, they face the competition from the others. If price is the only factor, a short-term thinking competitor can drive the others off the market. So they are forced to follow, unless a force beyond the market sets down rules that cannot be ignored. The same reasoning goes for unethical behaviour.

So the issue here is how we change the rules of the game to ameliorate the bad things without throwing the basic idea out of the window. For in spite of the waste, it's working better than the alternatives. Ration is bounded, it's not enough to deal satisfactorily with the needs of millions of humans.

Converting MySQL to UTF-8 the easy way

2009-01-23T19:00:00.008+01:00

This is just a quick note to myself.

So you're running on the latest version of INSERT NAME OF FANCY WEB FRAMEWORK HERE? Think character set problems are a relic of the past? Not so with MySQL. The default configuration is using Latin-1. When you install MySQL, the first you should do is ensure MySQL is using UTF-8. This problem will go away at some point when distributions change the defaults, but until then.

Meanwhile, if you're like me, you might have created some tables before discovering the problem. It is, after all, difficult to see before you create the tables. So here's a recipe for converting a whole database (idea stolen from Wordpress).

First type in these commands, replacing mydb with the name of your database:


USE information_schema;  
SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', column_type, ' CHARACTER SET utf8;') FROM columns WHERE table_schema = 'mydb' and data_type LIKE '%char%';  
SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', column_type, ' CHARACTER SET utf8;') FROM columns WHERE table_schema = 'mydb' and data_type LIKE '%text%';
SELECT CONCAT('ALTER TABLE ', table_name, ' CHARACTER SET utf8;') FROM tables WHERE table_schema = 'mydb';

This should output the commands you need to feed into MySQL to do the change. If you start the MySQL shell with -s, it's easier to copy-paste. Then type


USE mydb;
ALTER DATABASE mydb CHARACTER SET utf8;
[... pasted commands ...]

The problem here is that the character set is stored on multiple levels. Both column, table and database level. The second line fixes the character set of the database, and the pasted in commands fixes the columns and tables.

This works for Django. If you have used a framework that allows you to put UTF-8 characters into the Latin-1 columns, you need to do something else. The Wordpress link has the details.

Moving web servers without downtime

2008-10-30T13:28:00.006+01:00

In IOLA, we've been moving some of the sites we're hosting, most notably iola.dk with associated services.

The problem with moving web services is DNS. DNS is a silly protocol - it uses a hierarchy to scale, which perhaps makes sense admin-wise (don't forget that scaling means more than just performance, administration is usually at least as important given the cost of man hours). But a hierarchy is a disaster when it comes to performance since the root nodes quickly end up being swamped - think 100 million clients all querying .com domains simultanously. DNS uses caching extensively to alleviate this problem.

And this is the problem starts. There is as far as I know no way to clear the cache. So with a typical DNS entry with maybe a time to live of 12 hours, you have a pretty long period in which some clients may see the old IP address and some clients are seeing the new.

In some cases, it's not a problem to have both web servers running at the same time. But if a site is modifying the database, it's more difficult as you don't want to have two inconsistent databases.

There are several ways to fix this, including staying up late at night until all your visitors have gone to bed, but I'll show a neat trick which is really easy to pull off and which shouldn't cause any sweat.

Start by installing Varnish, a reverse proxy server, on the old server. If you are on Debian, it's aptitude install varnish and you then get a /etc/default/varnish file. In there are a couple of lines like:

DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -b localhost:8080 \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

The first line means that Varnish is listening on port 6081 and the third line means that it is forwarding everything it sees to localhost port 8080. So change that third line to -b 123.456.789.012:80 or whatever the IP address of your shiny new server, execute /etc/init.d/varnish restart and go test it by accessing your old host on port 6081, e.g. http://www.example.com:6081. It should return the page from your new server - you can verify it by looking in the log on the new server.

What happens here is that Varnish parses the request from your browser, forwards it to the backend (your web server on the new server) and caches and returns the web server response to the browser. It's of course a pretty dumb idea to fetch the content over the web twice, from the new server to the old server and then from the old server to the web browser, but we won't be doing it for long. And Varnish will cache as much as its default configuration allows for.

Then when you think you're ready, change the first line above so that Varnish is listening on port 80 instead:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -b 123.456.789.012:80 \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

Then stop your web server, e.g. with /etc/init.d/apache2 stop and restart varnish with /etc/init.d/varnish restart. This puts Varnish in the place where the old web server was before. So from now on, there's only one web server seing the requests, the one on the new server. So you can switch over DNS from the old IP address to the new address, and when the last DNS entries have expired, you won't be getting hits anymore on Varnish on the old server.

Varnish can process requests based on URL and host so if you're serving multiple hosts from the same server, it's possible to forward only some request to the new server. It's pretty easy to do, you just need to enable a configuration file and put the stuff in there. In Debian there's an example configuration in /etc/varnish/default.vcl. As long as Varnish is not listening on port 80, it's pretty easy to experiment with it.

While you're at it you might want to start with putting Varnish on the new server in front of your ordinary web server. If there's something Varnish can cache, pages that look the same for several clients, it can save some serious load.

Losing sight of the big picture

2008-10-04T17:00:00.008+02:00

I just had the most wonderful session with my piano. I haven't been practicing much lately, if at all, so I was mostly playing through old faster pieces to get my fingers up to speed.

Herr Mozart

A funny thing happened. As I'm playing the first movement of a sonata by Mozart, I suddenly realize that I'm struggling with the phrasing, articulating the bold and the faint passages clearly. This piece has lots of f and p markings (forte and piano, strong and quiet). I'm not too happy with the sound either. So I start focusing on getting it more precise, only to discover that the extra precision makes the sound worse.

The attention to detail draws attention away from the big picture, which in this case is the melody itself. Some pieces are simple, the music is in the tones. I think this is what most people relate to when they think of a melody. But this doesn't hold for most of the classical music I've heard and played. The tones are still important, but usually the overall dynamics are what will carry people away. Think of a symphony, an enticing crescendo towards the thundering finale, like the theme from Space Odyssey 2001 (it's from "Also sprach Zarathustra"). If it can't erect the small hairs at the back of your head, you need a treatment with a five-pounder shovel.

So by focusing on the details, which at most can make the music more interesting, although if you haven't played the piano you probably won't notice 90% of it, one can happily leave the most important of the piece to the coincidence.

It didn't work for me. So I gave up the details and started listening for the bigger picture, and voila! Pure joy. For five minutes, I was completely carried away.

The interesting thing is that the exercise turned out to be surprisingly harder than you'd think because all those little fs and ps kept breaking my concentration. I know this happens to professional players too, because I've experienced it at least once with a brilliant player, playing technically well and with lots of dynamics - but with no overall structure to push my emotions making the whole thing plain boring.

There's a long and interesting discourse into the land of interpretation of music here. But let me instead draw a parallel to software engineering.

For both piano playing and software engineering, often all we got is a clearly incomplete specification of how the thing is going to work. But if you (just) follow the spec, the result is guaranteed to be mediocre. And the more detailed the spec is, the more it'll distract you from the overall picture which is usually where the fate, success or failure, is determined.

Now witness Wilhelm Kempff, a legend whose playing to me represents the opposite of the brilliant player mentioned above. He may make a couple of mistakes and he obviously bends the spec into what he sees fit. But I'm carried away.

Flot 0.5 released!

2008-09-25T12:58:00.003+02:00

New version of Flot is out. See the changes.

The big news is support for mouse tracking and highlighting points. So you can make the plots much more interactive. Also included is support for dual axes and improved background markings, and a long list of bug fixes, the most noticeable probably being that timestamps are now interpreted according to UTC instead of the local time zone of the client.

Consensus decision making

2008-09-20T20:35:00.002+02:00

Is that crazy? Maybe not. Is a good manager the person who makes the decisions or the person who makes sure he doesn't?

The abysmal state of web layout technology - or not?

2008-09-20T18:48:00.003+02:00

This post is about CSS and HTML.

Coming from a desktop application programming background, the web platform is actually pretty flexible and concise. Customized layouts can be done with remarkably simple markup and a bit of separate style information.

The big problem with the web is that HTML and CSS is designed for pages with a single stream of information, like a manual or a scientific article about an experiment in physics. Or a Wikipedia article. To layout another stream of information, like say something as exotic as a navigation bar, you basically have to use a hack which requires you to understand some pretty complicated concepts about HTML box layout or turn the markup into an unmanageable table hell.

I've been integrating a design for an internal web application for a customer the last couple of days. The thing about CSS is that apart from the problem with multiple streams of information, it's usually quite straightforward to work with. As long as you stick to the HTML mindset of how the world looks like.

The basic building block in this design is boxes (fine, HTML layout is about boxes) with shadows (urgh, no support for that) and rounded edges (double urgh, no support for that either) and shiny buttons (no built-in support for shininess). Clean markup out of the window, table soup and images to the rescue. Matters is made worse by unstable rendering by a certain browser which is a disaster for a box where the visual effect depends of getting the edges exactly right.

The most annoying thing about this is perhaps that if just someone had put multiple background image support into CSS, then I could have my clean markup back and get over this in no time. I was thinking about this earlier today, and even got so far as to have a simple draft proposal for fixing this case. I was determined to submit it to whoever is in charge of CSS and push for its inclusion.

But upon investigation, it turns out that a solution has already been proposed and written into the CSS draft specification years ago as border-image. There's a nice overview on css3.info.

Apparently, the situation with respect to adoption has been dead-locked for a long time. The spec can't be made final until at least two different renderings engines have implemented it. And the rendering engines can't implement them for real before the spec is final. So it's going to take some determination to get going with proprietary attributes to begin with. Luckily it appears that 2008 is the year that the push is finally happening.

So there you are. It will change someday. I think this is typical of the web. There's some really good stuff in it, and a whole slew of annoyances. But they're going away, slowly.

Listening to music and last.fm

2008-09-19T11:18:00.005+02:00

Some time ago Anders created a last.fm group for YayArt. Anders showed me how Rhythmbox can integrate with last.fm, reporting in what I'm listening to. So I ended up creating a profile.

At first I find last.fm pretty confusing and leave it at that. A couple of days pass where I play music from my personal collection with Rhythmbox. Then I visit last.fm again. Behold, now it has lots of interesting suggestions. I just click "Play your recommendations" and I get a pretty varied assortment of music. You can control the stream to some degree by either hitting love or ban to what you're listening to.

There are some minor glitches, e.g. I yesterday loved too many film music pieces so it was getting a bit stuck on that kind of music until I started banning some of the pieces, but overall it's working incredibly well.

An interesting side effect is that you can listen to people's music taste. Now if I only knew someone on last.fm whose music taste I would like to listen to... My profile is here in case you want to listen to mine.

The role of culture - and what's an object?

2008-09-16T10:21:00.021+02:00

I had an interesting thought today. Why do you eat of plates? Our dining table is covered by a fine oilcloth, why don't we eat of that?

The plain answer, I think, is that we've been taught when we were young to use plates, to the point that it even seems unhygienic not to do so. Imagine a dinner with meat, potatoes and a batch of sticky sauce on your dining table.

Image by ayalan

If you're looking for deeper reasons for why one eats of plates it's probably easy to find out. Just have a dinner directly on your table. I suspect the reasons have something to do with the difficulty of cleaning up afterwards and sharp knives and scratches in the surface.

Pottery is a pretty old technology. The point is that it's part of our culture, we never think about it, we just do it. But it's one of the million of things that make up the great technological advancements civilisation has made over the past centuries. If we weren't able to absorb them, if we had to think about all of them, we would never have gotten anywhere.

A similar thing goes on in software. With a programming language you can build anything you can think of, provided you have the hardware to run it on. But still it takes time to invent new things. Old things have to be absorbed into our culture before we can move on, otherwise the leap is too great.

Of course behaviours that once have been embedded might stop making sense as the environment changes, unnoticed. When you dig into it the reasons stop being reasonable. I think you'll find that traditions and rituals provide a rich source of examples of this phenomenon - it made sense to those who concocted it, but 50 years later it's really silly when you think about it. Of course, that might not stop you from enjoying it.

Image by Thomas Hawk

In any case, I think it's sometimes interesting to turn things we take for granted upside-down. Today I've been thinking about the corner stone of object-oriented programming. Let's be pragmatic. What's an object?

You might think of attributes and methods. My take on it is that it depends.

In some types of code, it's a very useful modeling utility.

For instance, just think of the string class in your favourite programming language. It encapsulates most of the stuff you'd want to do with a string in a neat little package you can easily instantiate as many of as you like. If you've ever tried the horror that is the built-in string handling in the C standard library (not to pick on it, it was probably fine for its time), you'll really appreciate that. You might miss the occasional method or two, but still, it's very useful and easy to understand.

Library code is full of neat little packages like this. Dynamic arrays, linked lists, HTTP handling, database connections, widgets in a graphical toolkit.

As you go to application code, the answer is in many cases less clear cut. To a certain extent, this is perhaps because the concepts are less understood or perhaps just less well-defined.

In my applications I find that I have objects that are more or less just data (plain old data, PODs as they call them in C++ lingo) and objects that aren't really objects that I instantiate, let do a well-defined job and then throw away, but more of the sort of puppeteers or glue. These are all objects that give me head-aches. I also find some objects that are like the library objects, neat little packages. These are usually a great relief.

The first type of objects, typical application objects, are troublesome because they go against the culture of object-oriented programming. Which says that you shall divide your code into neat little interacting packages with methods and encapsulated attributes, instantiated and used dynamically.

But if we step back, then what's the purpose of objects? As I see it, it's a way of organising code. Think of it as a hierarchy. At the lowest level, you have the individual lines of code. We organise them into functions which let us deal with perhaps 10-100 times the amount of code, still maintaining an overview, like the headlines in a newspaper that groups the thousands of lines of text into articles.

Image by jurvetson

As a program grows, you need another level, one that will give you 10-100 more code without losing the grip of the program. This is plain hierarchy theory. Look at the folders on your computer for a prime example. An object is one such next level, one that is more flexible out of the box than traditional function library modules because it has state.

However, just because you need another level, it doesn't mean that the code will fit well into the object-oriented thinking of a neat little package where the data is encapsulated inside. I personally think it's impossible in many applications; otherwise it's going to take a lot of hard thinking, and I mean really a lot, because I have often wasted hours trying to find a way out of my crippled semi-objects.

It is true for the plain old data structures. In many cases, you just need to group a set of attributes under a name without the functionality making use of the attributes logically or even illogically belonging to the grouping. So just put them in, recognising this as a case where the object thinking is useful but not a 100% fit.

If you're writing getters and setters for this kind of stuff, you're wasting your time serving the culture of object-oriented programming as it was taught to you.

The same goes for glue code. It may organise the code better to put it inside an object. If so, go for it even if you're never going to do typical object stuff with it.

In any case, as I see it the goal here is a well-organised program. A well-organised program is obviously easier to understand, and thus easier to work with, extend, fix bugs in, etc. Think about that and forget culture next time your wrestle with program design and nothing seems right.

Weird web troubles

2008-09-14T17:59:00.009+02:00

No, I'm not dead yet. I have even been doing some of the maintainer tasks for Flot I have neglected the past months.

One of the new things in Flot is support for hovering/clicking with the mouse on the points. I was spending some time yesterday adding highlights so that Flot can show which point you're pointing the mouse at.

I got to the point that I was ready to performance test it, on my trusty old Pentium 3 950 MHz laptop. It felt a bit slow. Surprisingly, the profile from Firebug indicated that the slow function was the jQuery function offset().

I use it to compute where a mouse event is inside the plotting canvas - for some reason computing this offset separately is the only reliable way to know where the mouse is inside the element where the event happened because you only get the global coordinates reliably in the event. You would have thought local coordinates was a natural thing to standardize and every browser support, but no.

Anyway, offset() walks up the DOM tree, computing offsets along the way. And it was taking 25 ms per invocation on average! That's just ridiculous.

The funny thing is that I was meanwhile updating my system, apparently also updating Firefox from 2.x to 3.0.1. So when I boot up today with a couple of ideas for working around the slow offset function, it's now 4 ms on average. That's still far too much for querying a simple DOM tree, but it's not a show stopper anymore.

I think the lesson here is that as soon as you do exotic things in the browser, things that are not really about browsing HTML pages, you get bitten by weird bugs. But the state of affairs is improving.

Pssst. We're having a sale on YayArt. I have it from a reliable source that it's going to end soon.

The way of nature

2008-04-30T16:53:00.007+02:00

I was helping my father put up new boards for the roof of my parent's house today. The old one was, well, old and rotten. It was the last remains covering the gutter on the north side of the house, we've already replaced the boards on the other sides of the house (got quite a thrill out of the top ones on a wobbly aluminium scaffold 8 meters above ground).

Anyways, as we've done about half of the boards and reach the far end of the house, a female blackbird starts examining the wooden box around the gutter carefully at the other end, apparently interesting in using it for a new home. A few moments later the male blackbird, maybe its mate, does the same while we stand there, hammering boards in a few meters away.

Leave things alone a few minutes and life around you starts occupying the good spots. Nature or perhaps rather evolution is wasteful with life, but certainly not with space.

I sometimes wonder how fast a city would disappear if all humans were all suddenly extinct by a new plague-like disease.

YayArt is up!

2008-04-17T17:29:00.002+02:00

So it finally happened, we launched YayArt! It's a community-based approach to new digital art for the people (real prints).

That's all. I'm wasted now. :-)

Flot 0.4 released!

2008-03-07T16:11:00.003+01:00

New version of Flot is out. See the changes.

The date/time support has finalled arrived and there's support for coloring background areas. For instance, you might use this to plot the visitors to the Flot homepage. There's also support for segmented lines. Plus a long list of bug fixes and tweaks.

For the next release, I'll probably be focusing on adding more interactive features. There are some patches floating on the mailing list I need to take a look at.

I also think it would be a good idea with a hand-holding tutorial that for people who are not accustomed to Javascript plotting. I have something in the works, more about that later.

Favourite Linux bug of all times

2008-03-07T11:03:00.009+01:00

Here's the story about my favourite Linux bug: the thrashing hell syndrome.

To reproduce it you need to have swap activated and a run-away process. Traditionally, for the latter Netscape Navigator was great, in these days it might happen with Firefox with Flash or Monodevelop. Or if you're developing something and accidentally make an eternal loop that allocates memory.

What will happen is that the system quickly slows down. You'll have some seconds to react to kill the process. If you're too slow, the graphical interface locks up and you're toast while the machine enters thrashing hell. You can usually get to a terminal and even enter username and password, but it locks up again before the shell is running. Probably because you can't spawn a new process when in thrashing hell.

The condition will last for anywhere between a couple of minutes and more than half an hour. Unless you power off the machine, there's nothing you can do. At all. Other than to stare at a frozen screen while the machine entertains itself.

At some point an out-of-memory killer was introduced in an attempt, I believe, to mitigate this problem. Its job was to kill a process to free up memory in memory-tight situations. Here's a hilarious analogy by Andries Brouwer. I suspect Netscape was the main cause, although I might be wrong, I've never been close to kernel development anyway.

Anyway, that was long time ago.

Fast forward to the introduction of Ubuntu, the really easy-to-use GNU/Linux. I should warn you that I'm going to do some bashing of Ubuntu now, which is not really fair since Ubuntu is pretty cool.

When I first got into thrashing hell on a Ubuntu system, I didn't believe it had happened. I thought the OOM killer should have saved me, but apparently not. Probably it'll only kick in when all of the swap is used. But hey, Ubuntu is for the masses, right? The system should have protected me. That's what I naively thought.

So I filed a Ubuntu bug report on the kernel. I hoped that a couple of the knowledgeable people who are working on Ubuntu would fix this old problem in a snap.

Which didn't happen. The bug wasn't accepted. As far as I can tell because the guy in the other end thought the process limiting support in the kernel is advanced enough to take care of the problem. I wasn't sure whether he was right. But I filed a new bug report with the hope that someone else would see the vision.

That's over two years ago. So far no progress. Which is not what I'd hoped, because it really puts the system in a bad light. If this happened to an average user, that person would instantly conclude that Ubuntu/Linux just crashed.

But it's sort of fine, because it's open source etc. etc. and the people in the other end are free to spend their time as they like.

Then today there was an update. Cool! I thought. Until I found out that it's just a drone reply. Even a particularly annoying one, apparently it's not good enough to report a bug on the latest release since the problem might theoretically be fixed in the next alpha. The bug has already been auto-closed once because if noone's interested then the report must be invalid.

So noone wants to listen. That makes me think this will continue to be my favourite Linux bug for many more years to come.

PS: My favourite recent Windows bug is probably the one where Windows interprets all mouse clicks as right-button clicks.

Quiescence

2008-02-24T23:03:00.005+01:00

No update for a long time. I've been busy, mostly at work but also at home with the garden.

I currently have our small garden full of bulbs. Many already have the first small leaves up, including some daffodils and tulips (in spite of it still being February!). I've buried so many bulbs that I have but the faintest idea of what is where. The last few batches are pretty much placed completely randomly.

I've also invested in seeds from Plant World Seeds. I chose them because they have several bags with different mixtures; our garden is not of a size where it's relevant with 50 flowers of the same kind. Janne also bought some seeds from a Danish merchant, mostly small flowering trees, so the house is now full of pots with moist soil and small white labels.

I'm currently mostly excited about the 12 pots of clematis montana (four seeds in each pot, unfortunately it can take three months before we see any seedlings), and a bag of dactylorhiza maculata (heath spotted orchid) that I haven't figured out how to sow yet. The funny part is that we haven't even sowed half of the seeds. The summer is still a bit too far away for most of them. I think we're going to have a problem by then.

I read somewhere that the original meaning of "paradise" is a garden. Last year it was a little empty. Not surprising given that I dug all of it in April and May. But we'll see this year.

In other news, I've spent some time today and yesterday working on time series support in Flot. I just committed the code to the SVN repository. I need to clean up a couple of things, test it a bit more and write up some documentation before release. Next on the list is reviewing the patches on the bug tracker and on the mailing list.

Flot 0.3 released!

2007-12-14T19:04:00.000+01:00

New version of Flot is out. See the changes. This is mostly a bug fix release because the previous zip/tarball didn't contain the jquery.js file.

I've also started a mailing list which I really recommend using.

I did sneak in a new feature - that's why it is 0.3 instead of 0.2.1. You can now enable people to click on the plot and get the (x,y) coordinates (in the unit of the plot, not pixels) out.

This is a primitive form of interaction. Eventually I want to support automatic data coupling so you can know what has been clicked and in the same vein highlight the stuff underneath the cursor. But at the moment I'm pretty swamped with things to do, and this clicking was enough for our internal application. I suspect it will be useful for lots of other things too, at least until Flot is clever enough to map from points to data.

Flot 0.2 released!

2007-12-07T16:56:00.000+01:00

I've released a new version of Flot. The API is now documented and legends have a background. And you can get the whole package as a zip/tarball. See the changes.

Lots of people are asking for pie charts. And some people have difficulty seeing the charts in IE 6. Which is really unfortunate because I can't reproduce it myself. All the machines with IE 6 I have access to render them fine. It's probably something silly, but it's going to take a bit of debugging to find it.

My plans for the near future mostly deal with getting proper support for time series and some of the other interactive stuff if I can squeeze it in. And get a forum or mailing list up running.

If you have a public application of Flot online, I'd like to hear about it, I intend to compile a list of real-world examples.

Flot 0.1 released!

2007-12-04T12:48:00.000+01:00

We've released the first version of Flot, a new Javascript plot library for jQuery!

It draws on inspiration from Plotr and PlotKit, and several other commercial packages. But also from venerable old gnuplot - there's nothing more boring than hand-picking axis scales so I wanted the default algorithm to be smart.

The project actually got started because we were growing dissatisfied with Plotr which looks nice but is pretty dumb when it comes to auto-detecting stuff and didn't have any interactive features which we needed for a custom web-based business intelligence application.

I find it interesting that I initially thought plotting to be a hard problem - but this project got started a couple of months ago and I've only been hacking on it sporadically. Right now, the library is still missing support for time series, but is otherwise pretty much feature-complete for what we need it to do. I'm looking forward to hearing other people's opinion.

Codename Nemo 0.1 alpha released

2007-11-29T10:57:00.001+01:00

We've finally released an alpha version of our file manager project!

This has been a long way. The idea has been the same throughout the project, to focus on the aspects of documents that people actually remember instead of forcing a hierarchical tree down their throat.

Believe it or not, a lot of novices find it difficult to understand the hierarchical folder structure, and it's not particularly easy to navigate or keep in order for most computer literates either. People have trouble remembering what folder they put things in - and they would rather not put things into folders if they can avoid it.

This has been known for a very long time. We're building on research from the 80s here.

The big question is how you design a GUI that makes it possible to find the files again without a hierarchy. There's been a lot of attempts on this. Recently, people have been using pure text search - but while it can help, it's not a panacea. It's not always easy to remember something precise enough to put into the seach bar.

There's also been a lot of experiments with 3d spatial systems that mimic the physical world, usually with a high bling-bling factor. They look funny, but in my opinion they're never going to work because a) cleaning up in the physical world is really painful and something most people tend to avoid and b) the physical world scales really bad. Sure it's easy to find something on your desk if you only have 10 documents on it. But what if you have 1000?

So what do you do? We started from something that looked completely different from what this alpha release looks like and went through a couple of iterations before we arrived at what we have now.

There are lots of problems in this alpha release, but I personally think the basics are about right now. Is it usable? I think so. You can't select files or delete them, or even rename them because we haven't coded a proper custom widget for it yet (like e.g. in Nautilus). Must of the automatic stuff that's supposed to be smart is really pretty dumb because it hasn't been refined yet.

But you can get an overview of your files, organize them and find them again. And that's what it is all about.

Adding quote marks with Django

2007-11-27T17:24:00.000+01:00

I've posted my first Django snippet on djangosnippets.org. It's a template filter for adding quotes around a piece of text. It's clever enough to take paragraph tags into account.

Debugging

2007-11-23T10:00:00.000+01:00

I've been thinking about debugging lately. In my humble opinion, there are four phases in the art of debugging.

The first phase, which I think of as the debug prevention phase is where you proactively do something to prevent silly bugs from happening.

The thing is that debugging is time-consuming. Not only because you have to find the bugs, but also because it disturbs the flow. Silly bugs are like faulty brakes on a bicycle. It's neither fun or productive to push and push to get up to speed just to find yourself hanging over the handlebars at a sudden halt.

So it's worthwhile to do something to avoid obvious sources of bugs.

Here's an example of an obvious source that I recently encountered (apologies to the source): you have a model class with a certain state which can change on its own while also being changed in the user interface by the user. One way to approach this is to code the model and user interface separately, each with its own state, then add a bit of glue code to synchronize the two.

The (obvious) problem is that the two copies of the states can get out of sync, in which case you're toasted. It's also pretty confusing - when you ask for the real state, the master state so to speak, where do you ask? The model or the UI? So avoid the copy in the UI, even if it's less convenient to have to go ask the model all the time.

Of course, obvious sources of bugs are probably not obvious unless you spend some time reflecting on what you're doing. Here's a simple rule of thumb: If something is so confusing to think about that you can't immediately make head or tails of it, then it's probably worthwhile spending some time refactoring it because otherwise it's likely to become a source of bugs. Sometimes it's just a matter of renaming variables and functions.

The second phase is testing. This is where you find the bugs. The relation to debugging is that you should test and never assume that things are working just because the compiler/interpreter/whatever ate it. Untested code is an obvious source of bugs and it's always better to fix the bugs before you've embarassed yourself by shipping them. Debugging in the open is not fun. It's also much easier to find the bugs if you test early.

The third phase is the trial-and-error phase. This is the initial reaction most people have when something stops working: What, it worked before, let me just try this, then it'll probably work again. Repeat.

What's important in this phase is to leave it early if the first couple of tries don't work out. Trial and error is time-consuming and mentally draining because it puts you through lots of frustrating failures with no clear reward. Don't get stuck.

Instead consider proceeding to phase four which is the observation-thinking cycle. Here the goal is not to fix the problem, but to understand what's going on. Think of yourself as the analyzing surgeon who's on a mission to make the smallest possible cut, not the rampant safari hunter with the over-size elephant gun.

How do you understand your program? By running it, either in a debugger or with print statements sprinkled so you can see why the code is failing. You start at the innermost level, then gradually trace your way through the call stack.

The idea is not to trust anything you see. It's always a bad idea to assume something is working before you have printed the result and verified it in the actual faulty run. For instance, it might be a good idea to really start at the innermost level even though there's nothing exciting going on there. I've spent countless hours trying to understand why something obviously correct wasn't working just to find out that the code I was looking at wasn't actually the code running. Maybe you've forgotten to save the file. Or upload it. Or recompile. Or maybe it's running another, similar function you've forgotten about. Or maybe it suddenly crashed because of something seemingly unrelated. A print statement can tell you the truth.

You observe (real output, not source code) and you think, add more debug code, and observe and think. When you're done, you know what the program, your little creation, is doing. That's a nice, rewarding feeling. You're in control. And it makes the kind of story you can tell your (geeky) friends: I had this really annoying bug, and I traced it through ..., and finally I had it nailed!

Sometimes it's difficult to proceed to phase four because the debugging environment, frankly, is not up to the task. In which case you'd better start working on the debugging environment rather than building up frustration when the bugs begin to manifest themselves. So little is needed - if you can get printf-like output and short compile-run cycles, then you're good to go.

Peopleware

2007-11-21T10:24:00.000+01:00

I finished reading Peopleware: Productive Projects and Teams. It's about the human factor of conducting projects. Which according to the book completely overshadows any other aspects of the project, even though it's often ignored completely.

The way the book is written it's addressing project managers. But don't let that fool you. If you're somehow involved in a software project, you're also involved in the management of it. If you're not, there's something wrong - read the book to find out why.

I think the main message of the book is that you should focus on keeping the people involved in a project happy to make the productive.

Make their work fun, go to great lengths to ensure they can work together well as a team, stay out of decision making that others are in a better position to do in spite of what hierarchy suggests and avoid interrupting everyone all the time.

In doing so, you're likely to have increased their productivity by an order of magnitude more than any Methodology (as they put it) can give you. The reason is that people are not machines. If you expect them to be and make them feel that way, and this is easier to do than you might think if you come from a software background where it's all about the logics, then it shifts from high gears to low gears, the magical stuff that makes a design or a program uncover itself from nothing just by the help of human mind.

There a lot of interesting tidbits in the book too. For example, it's often postulated that you need a deadline to make people work efficiently. So okay, people are not machines, they're much worse, they need a bit of whipping to make them work hard. The book cites a study which found that people are almost twice as productive on projects without a deadline compared to projects where the project manager sets one.

Another example is that people tend to react to change in unpredictable ways. For instance, if you do something different from what you use to do, you might be more productive, not necessarily because the new way is better than the old, but simply because you expect it to be better (I guess this is the Placebo effect) or because it's new and exciting and you're generally more productive when you're excited. The authors call the phenomenon the Hawthorne effect after an experiment with illumination in a factory. Increase the lighting and people are more productive, decrease the lighting and people are also more productive.

Anyway, if you're somehow involved in software projects, or any other kind of mind-worker projects for that matter, I recommend you read this book. You won't regret it. It taught me more about my work than any other book I've ever read.

Torturing prisoners

2007-11-05T10:11:00.000+01:00

I unsubscribed last Friday.

I can't believe some of the stories you hear. What happened to the Geneva Conventions? What happened to treating other people, including your enemies, with respect? What happened to being decent people?

Making foreach useful in the corner cases

2007-10-21T22:34:00.000+02:00

One of the nice things about recent developments in mainstream programming languages is improved support for looping.

In a functional language with proper support for lambda functions, it might not be a problem but if you're programming in something a bit more mainstream, chances are that you're writing loops over data structures all the time. In my experience, it turns out to be a real improvement if the language offers a bit of syntactic sugar and allows you to write something like:

for element in datastructure 
    do something with element

Python's got it, C# and Java have it, last I heard there were plans to introduce it in C++ too.

Until recently I thought this particular piece of syntactic sugar was of the kind which is really useful in 90% of the cases and completely useless in the remaining 10%. For example, what if you want to extract two elements and not just one? Or loop over two datastructures? Or only loop over certain elements?

In fact, there are elegant solutions to all these problems. I've been pondering over them this week. Here's how you do in Python, but all this requires of the language is the concept of iterators and some kind of convenient support for tuples.

Extracting only part of the sequence: Slice the datastructure with a function that skips the parts you don't want.

Python has the [start:end] operator which is as succinct as it gets for extracting an interval. There's also a more generic islice function in the itertools module which works with iterators:

from itertools import islice
mylist = [1, 2, 3, 4, 5]
for x in islice(mylist, 1, 4):
    print x
(... prints 2, 3, 4)

This idea can be extended in many ways, e.g. if you need every second element, it's easy to come up with a function that only iterates over them. If you need to exclude elements based on something related to the elements themselves and not their position in the datastructure, there's always the filter function.

Extracting elements from several datastructures: Construct an iterator that iterates over the sequences in parallel, returning a tuple with one element from each.

Python has the built-in zip function which returns the complete sequences zipped up as a list of tuples. Until Python 3 is out, izip in itertools is closer to what we want:

from itertools import izip
xlist = [1, 2, 3]
ylist = ["a", "b", "c"]
for x, y in izip(xlist, ylist):
    print x, y
(... prints 1 a, 2 b, 3 c)

So it's no problem. We can still maintain our nice, succinct for loop, there's no need to manually iterate over each sequence in turn. Note that zip and izip will automatically stop when one of sequences runs out of data.

Extracting several elements from the same datastructure: Construct an iterator that returns a sliding window of the sequence as a tuple.

You could code this manually, but it's even easier to do once you know zip: just zip the datastructure with itself, with a little offset. In Python, that's zip(seq, seq[1:]), or more generally:

from itertools import izip, islice
def slidingwindow(seq, windowsize):
    t = ()
    for i in range(windowsize):
        t += (islice(seq, i, None),)
    return izip(*t)

That's all you need to get two or more elements out at the same time:

mylist = [1, 2, 4, 8]
for prev, val in slidingwindow(mylist, 2):
    print val - prev
(... prints 1, 2, 4)

No need to manually set prev to the predecessor in each iteration. You could easily add padding or let the window advance in larger steps if needed. The latter is probably easiest to do by combining the function with an appropriate slicing operation.

Which brings me to the final point. An important part of the beauty of these operations is that they can be combined in a multitude of ways.

Note: if you're going to use slidingwindow in production and want something to paste in, here's an even more general version that works with arbitrary iterator input:

from itertools import izip, tee
def slidingwindow(iterable, windowsize):
    t = tee(iterable, windowsize)
    for j in range(1, windowsize):
        for i in range(j, windowsize):
            try:
                t[i].next()
            except StopIteration:
                pass
            
    return izip(*t)

The tee function duplicates an iterator by storing the data returned by it until all its duplicates have passed it too. The code above advances each of the returned iterators by the correct amount. It seems it's a bit faster to use tee than doing the equivalent stuff by hand.

Landmark

2007-10-16T22:45:00.000+02:00

Today I have lived for a quarter of a century. I was born in 1982. According to Wikipedia, it was the year the first computer virus escaped into the wild.

Django-rendertext 0.2 and a new jquery plot plugin

2007-10-05T10:17:00.000+02:00

A quick update. I've released django-rendertext 0.2. The most notable change is support for GIF output, mostly intended for the browser that didn't support PNGs properly until very recently.

I deliberately built rendertext with PNGs first because it's the format of the future. IE7 usage is picking up (see for instance here or here), within a year or two it's probably completely dominant among IE users. Now that I whipped up the GIF support, I'm glad I didn't start with it because it was a real PITA to do with PIL (Python Imaging Library). PIL doesn't have proper documentation of the corner cases that turn out to be the most important and only sketchy support for palettes.

I've also been working on a plotting plugin for jQuery. We've not been entirely satisfied with Plotr which otherwise appeared to be the prettiest and most advanced Javascript-only plotting library out there. It's for Prototype and not for jQuery but most importantly I've found it confusing to hack on. And it's completely missing the interactive aspects.

So I have started my own project. So far I think I've discovered that the hardest part of a library for plotting is not actually drawing the stuff, but layouting and auto-adjusting the axes. In any case, in two days I've gotten pretty far. Here's a screenshot (click to get the full version):

What's missing right now is support for plotting points and columns, options for controlling the plot, legends, support for time and the interactive stuff such as zooming, highlighting graphs, tooltips for points on mouse hover. Also, the auto-adjusting algorithm is still a bit rough but I think I managed to crack it on paper last night so I just need to implement and test it.

In other news, from Werner Vogels' blog here's how Amazon manages storage (warning! scientific paper ahead). Now that's as far as you can get from N=1 thinking. As an aside, I'm fond of that guy's name, Werner Vogels (note for English native speakers: in German the surname is pronounced like [fogels] not [wogels]). It's just like der Vogelfänger bin ich ja I guess, nice rhythm.

Object-oriented database systems

2007-09-21T22:11:00.000+02:00

I had a discussion with Anders today about databases.

I'm not very fond of relational databases, or perhaps more precisely SQL. Having to convert back and forth between a database model and SQL and a programming language with all it entails of proper quoting to prevent SQL injection attacks and joining tables all over the place is really a great source of frustration. And a good waste of time.

On the other hand, I've seen what a record-based file system with no built-in join can do to people. And the built-in indexing stuff that lets you search on different attributes without a linear scan is just neat.

I think the great success of the relational database systems has a few simple explanations. Many business-related applications are really about the data. They spend their time collecting and displaying lots of simple, boring data - unlike an entertainment program or a standard desktop application that spends a lot of time dealing with small amounts of data, maybe less than say 1000 objects.

This data may be larger than what can be kept in memory (the possibility of this happening is enough to make it a problem). Furthermore the data must be kept when the machine is turned off. So you have to deal with secondary storage, a topic largely ignored by most programming languages that otherwise go to great lengths to make dynamic use of primary storage as transparent as possible.

Finally, you need to be able to access all that boring data in a many different ways. Say you have a situation that you naturally think of as a hierarchy, e.g. a company with multiple departments with each department having a number of employees and each employee having far to many tasks to do.

In object-oriented thinking, you would model this as a list of departments, each with some attributes and a list of employees, each again with a list of tasks. Want to show the org chart in the company? Easy, just loop through the departments and print the list of employees one by one. Super efficient. Assign a task to an employee? Easy, just add it to his list. Super efficient. But what if you want to know who's task it is to empty the bin? Then you have to wade through two outer loops plus lots of unrelated stuff, just to get to the actual data you're interested in.

Relational database systems, while solving the secondary storage problem, have an answer to this problem, and it's simple and elegant.

Then never mind that getting data in and out of them is hugely painful, especially hierarchical data, even if you're not trying to force everything leaving the database into a class.

I've recently been working a lot with Django's object-relational mapper, the ORM (which BTW means "worm" in Danish), and it really takes the pain out of the situation since it obviates the need for writing SQL strings in favour of native, albeit somewhat weird, Python code.

However, I still wonder why true object-oriented databases haven't overtaken the world ages ago. One that I hand over my object to and later can get it back from, in one piece with no fuss.

It turns out that there's a pretty large open-source one available, db4o. I think it solves the problem of retrieving objects based on arbitrary attribute values by incorporating a special query system where it's possible to define indexes like for a relational database, but I'll have to do some more reading.

There's a smaller one, DyBASE, for Python (and Ruby and PHP). It could be interesting to compare the performance against Django's ORM + MySQL.

The interesting question is of course whether you need a relational database if you can have the simpler object-oriented one. Interoperability issues aside, some people apparently seem to believe that the relational model is crucial for performance with huge amounts of data.

It would seem to me, though, that the natural partitions which the object-oriented paradigm splits the data into would tend to make things faster since there's less data to search through. For instance, if you're only concerning yourself with data from one department in the company, then it's rather wasteful to work with the maybe ten times larger table containing data for all departments. Perhaps it has something to do with joins and restrictions being difficult to express simultaneously in one query with the object-oriented approach. There are some benchmarks on www.polepos.org although they are not exactly easy to interpret at a glance.

Other people, like this guy on Slashdot, seem to think that the abstract relational model makes for better databases as opposed to application-specific persistent storages.

This makes me wonder whether many, perhaps most, uses of a relational database today wouldn't be better served by a simple object-oriented application-specific data store. I think most run-of-the-mill web sites need easy object persistence far more than terabyte scalability or an enterprise-wide database. Leave the heavy artillery for when it's really required.

Then, on the other hand, if you can get easy object persistence with Django's ORM, maybe this is a moot point.

Django-rendertext 0.1 is out

2007-09-06T09:52:00.000+02:00

We're releasing the first version of rendertext, a Django app for making use of custom fonts on a web page!

It's designed to work with the built-in Django template system by providing a simple template filter you can use to render images dynamically.

The first showcase is Talent Nord. I hope to get a longer list as more people start using it. The design of Talent Nord is by Fata Morgana, house of two intelligent designers.

I think you need a pretty recent Django for the app to work. Here's the code that does the hard work:

import md5, os
import Image, ImageFont, ImageDraw, ImageColor
from django.conf import settings

def render(text, fontalias, size = 12, color = "#000", rotation = 0):
    """Construct image from text.

    Returns a tuple with the image file path, width and height. Takes
    the parameters 'fontalias', 'size' (default 12), 'color' (default
    black) as CSS hex string, and 'rotation' (default 0) which is the
    degrees to rotate the text counter-clockwise. The constructed
    image is stored in a cache under the media root in the
    RENDERTEXT_DIR set in settings.py (default is 'rendertext/').

    The font file to use is determined by mapping the fontalias
    through the RENDERTEXT_FONTMAP setting, a dictionary from
    fontalias strings to font file paths, e.g. {'verdana':
    '/home/django/fonts/verdana.ttf'}."""

    # get settings
    render_dir = "rendertext/"
    if hasattr(settings, "RENDERTEXT_DIR"):
        render_dir = settings.RENDERTEXT_DIR
    
    fontmap = settings.RENDERTEXT_FONTMAP

    fontfile = fontmap[fontalias]
    info = "|".join([text, fontalias, str(size), str(color), str(rotation)])
    name = md5.new(info.encode('utf-8')).hexdigest()
    filepath = render_dir + name + ".png"
    
    dim = (-1, -1)
    if not os.access(settings.MEDIA_ROOT + filepath, os.F_OK):
        # construct the image
        imf = ImageFont.truetype(fontfile, size)
        dim = imf.getsize(text)
        im = Image.new("RGBA", dim, ImageColor.getrgb(color) + (0,))

        draw = ImageDraw.Draw(im)
        weird_font_renderer_fix = " "
        draw.text((0, 0), text + weird_font_renderer_fix, font=imf, fill=color)

        if rotation != 0:
            im = im.rotate(rotation, Image.BICUBIC, True)
            dim = im.size

        if not os.access(settings.MEDIA_ROOT + render_dir, os.F_OK):
            os.makedirs(settings.MEDIA_ROOT + render_dir)
        
        im.save(settings.MEDIA_ROOT + filepath, "PNG")
    else:
        # read width and height
        im = Image.open(settings.MEDIA_ROOT + filepath)
        dim = im.size

    return (settings.MEDIA_URL + filepath, dim[0], dim[1])
    

def render_tag(text, *args, **kwargs):
    """Construct image tag from text."""
    (path, width, height) = render(text, *args, **kwargs)
    if not path:
        return ""
    else:
        dim = ""
        if width > 0 and height > 0:
            dim = 'width="%s" height="%s" ' % (width, height)
            
        return '<img src="%s" alt="%s" %s/>' % (path, text, dim)

Groups and social software

2007-07-27T15:50:00.000+02:00

From a link from Joel on Software is this essay on groups and social software. Lots of interesting points, e.g. that the group needs rules to prevent it from ruining itself, that social and technological issues are inseparable and that a core of members arise which should be nurtured.

No posts for long time, I have been on vacation, away from computers. I've gotten through a good deal of books, all fictional. Not having read much fiction for the last year or so, I find it surprising that a book can still keep me occupied in bed in the morning so that breakfast turns into lunch and I spend most of the day on the couch reading.

I've also been thinking. Mostly about life and what I like to spend my time on. Nothing major came out of it, though. :-)

When you're smiling...

2007-06-22T14:39:00.000+02:00

Greetings! They come in various flavours, depending on how well you know the person you're greeting and the time since the last greeting. I came across a funny one today.

In descending order of warmness, there's the hug, the informal "hi!" or "hey!", the slight nod of the head, and then there's the funny smile. It's not a real smile. The real smile is reserved for people you can't say hi to because they're on the phone or for people you're flirting with.

The funny smile is the kind of smile where you consciously try to move the corners of your mouth a little bit upwards while keeping the lips pressed together. The meaning is something like, "I see and acknowledge you but a real greeting is not warranted".

The funny thing about it is the way it looks. I haven't thought about it before, but someone sent me such a funny smile while I was on my bicycle to work. And you surely appear to be sucking your lips into your mouth, like the archetypal toothless old witch.

Think about that next time you funny-smile someone.

Software design, part I

2007-06-02T15:04:00.000+02:00

What makes a good software design?

I have been thinking about this for some time. The more I think about it, the simpler I come to think it is. Forget all the -tion, -ability, -ance words that are usually used to describe properties of good designs. In my mind a good design is one that is easy to understand.

As developers we think that ease of use/understanding is something that first and foremost applies to our users. We are used to horribly complicated IDEs, command lines that requires memorizing half a dictionary of cryptic commands, debuggers that speak assembler, etc. I argue that as a program grows from 50-100 lines of code to several thousands lines, the only thing that really matters is that you're not lost when you read and navigate it. And not whether you could squeeze a couple of lines out of it by some obscure technique you recently read about in a blog post.

Why? Because you need to understand the code in order to extend it, to fix bugs, to avoid introducing errors, to be able to estimate effects of changes, and in a sort of self-referential way, to still keep the design ease to understand.

In my mind, there are three goals with programming. In order of importance: to get the thing running the way it is intended to as fast as possible, to have as few bugs as possible, and to have as much fun as you can in the process. These are complementary - the faster you proceed, the more fun it usually is, at least if you are attracted to the creative dimension of programming. And the fewer bugs you have, the faster you can proceed. A non-existing defect that you don't have to try to reproduce, debug and then release a fix for is a real time-saver. Likewise, the more fun it is, the better you can concentrate and the faster you will go along.

Easy-to-understand code generally supports all three of these goals. If the code is difficult to understand, it will slow down the process immensely. And it will increase the risk of subtle bugs of the kind where the logic is faulty in corner cases. Even a clever compiler can't help with you out of those. Finally, it is simply more fun to work with code you understand.

How do you make a design easy to understand? I believe at the code level itself it is very much about using clear language and avoiding cryptic, non-descriptive names. Especially over time. If an identifier changes it meaning, even in a subtle way, so that its name is no longer descriptive, it's important to rename even if you have to change code all over the place.

At the design level it's on one hand about keeping things together that belong together and on the other hand about splitting up the code into smaller chunks so that the reader is not overwhelmed. Two opposing forces.

It's not that complicated really. To be able to understand something, it helps that the lines of code are sitting right next to each other. If you have to look them up in different places, you've lost your overview - and how do you know you found all the places unless the issue is easily greppable? But also, the piece of code needs to be small enough to fit in your brain, else you have to break it down yourself through analysis which is time-consuming and takes away the focus on what you really are trying to accomplish.

Keep things together that belong together. If you are designing a database application, don't put the logic that decides how the screen widgets should look like in the code that computes stuff from data received from the database. Likewise, don't put code that performs complicated computations in GUI code which is already busy with presenting an intuitive interface that prevents people from doing unnecessary damage to their data.

When you program, violations of this principle might be easy to see if you look at the dependencies. So the database code had to explicitly import a GUI module? Maybe it has more concerns than the database itself then. But sometimes violations are more difficult to spot. I think a main culprit here is the decision making.

Code is all about making decisions. In fact a computer program is at heart just a long series of decisions - first I'll do this, then I'll do that, then if this happens, I'll do this again. Most of these decisions are not important decisions, however, more like minor tweaks. It's not easy to see this in the code itself. One statement looks as good as the other, even though the first might decide what the user is going to look at and the other just adds a 2 pixel space around the border.

But what you usually want to do is to keep decisions about the same kind of things together. Then it's easy to understand. What fundamental things does the user see on screen? The answer should be along the lines of, take a look at this piece of code and you can see all of it. Not, well, grep for XYZ in these 40 source files. That's why lowly GUI code which is often preoccupied with tweaking spacing and interactive behaviour shouldn't be allowed to do much on its own, but immediately transfer control to somewhere else when anything less than trivial is to be done.

Since the programming language does not directly help with arranging the decision making in a sensible way, you should explicitly take it into account in the design, e.g. by forbidding certain parts of the code to do certain things or by designing modules that are explicitly made responsible for nothing more than high-level decision making. Or by arranging a calling structure that makes it easier to delegate decisions to upper management, e.g. by means of signals/callbacks/hooks/events/delegates.

Sometimes it means more hassle to follow this principle. But no pain, no gain. In most situations the long-term consequences of understandability are much more important than whether it takes an extra minute to code.

I have much more to say about this, but for now I'll conclude with this remark: When you start working on a new aspect of program, one that you haven't worked on before in another project so that you have a design already, it's easy to become irresolute, to get stuck on how to proceed in the best possible way. But if you simply design to keep things together that belong together while splitting the code into chunks that are small enough to be understood, it's probably going to be a healthy design and the details are less important. So just go ahead and see what happens.

How not to do management

2007-05-22T19:52:00.000+02:00

Joel on why simple economics is not a good way to manage other people.

Labeling

2007-05-22T19:06:00.001+02:00

Labeling is the new, Web 2.0-enabled way of organising information...

Human rights

2007-05-20T11:51:00.000+02:00

I received the latest issue of Amnesty's Danish member magazine. A horrible and moving reading as usual. Hit by a cluestick: hey, there's a world around you!

The theme this time was Iraqi refugees. It didn't even go into much detail about the two million people who have left the country (two million people!), but instead explained how the Danish authorities are detaining the 519 who have come to Denmark in 2006. 47 of them have been let through and can restart their life.

The remaining are left in limbo. They have nothing to do, they are not allowed to get a job and do some meaningful work, their food is prepared to them so they can't choose what to eat, clothes and other personal stuff is only available at a local shop that operates by giving each refugee a number of points they are allowed to spend. According to one of them who has been living in the camp through her teenage years it's like living on a train station. Imagine growing up on a train station with nothing to do.

Meanwhile Sweden is letting 80% pass through. And Sweden didn't start the war.

The problem in Denmark appears to be the rule set. Imagine trying to codify compassion. It's a human trait, and an important one as well, maybe the one that makes us so successful compared to other species. But what is compassion anyway? How do you explain it in terms that a bureaucrat can understand and manage?

Apparently, the rule set the Danish authorities apply for accepting refugees requires that there is a personal element of prosecution. If your house has just collapsed from an American bomb and there are fightings in the street, is that personal? Maybe not. Do you have to move? Yes. And if you have the responsibility of taking care of others, if you have children, then you have a damn obligation to get away as fast as possible.

So the system is rotten. There are some hints for fixes in the magazine. The Danish association of doctors wants to examine the people living the in the camps. If they are getting provably ill in a medical sense, then they can declare the camps medically irresponsible.

Fixing the system itself is unfortunately not going to happen soon. That requires going to the politicians in charge - which in turns with the current aliens-generally-smell-rob-our-homes-and-rape-our-children political climate requires that the people starts changing it's collective mind. Appealing to the politicians themselves is not going to work.

So how do you change the mind of the people? I think the standard news stuff like "bomb in Iraq, 40 people killed" and then you see a smashed ruin covered with dust is part of the problem rather than of the solution because it gets people used to the sufferings. They start thinking, 40 persons killed, that's not so bad, it happened yesterday too.

I think we need more documentaries and even fictional movies about how it's like to having your house bombed and then come to Denmark and live at a train station with no freedom and lots of spare time for 5 years.

Oh, I almost forgot. The magazine also had an interview with Hans Blix, the former chief of the UN's weapons inspectors in Iraq. The US federal government wanted the war so badly that they even fabricated evidence (or let evidence be fabricated) about the famous non-existing weapons of mass destruction to convince people... Now two million Iraqi people have fled their homes and country.

Imagine being responsible for that. Of course the issue is more complex than that, but still. Two million people!

Algorithms

2007-05-07T22:34:00.000+02:00

Just today I implemented bubble sort in Javascript to speed up the inner loop of a search algorithm. I think that's the first time I've actually implemented a sorting algorithm for real in spite of having read pages and pages about them. It's just one of the things that all real programming environments solved long before my time.

Of course, right after having implemented the bubble sort thing, I read on Wikipedia that it's actually considered inferior to insertion sort. So I then switched to insertion sort (I'm only sorting short sequences so asymptotic performance is irrelevant). It's my lucky day, two classic algorithms in such a short span of time...

The simple optimisation work did improve the running time by more than 400%, so that's cool.

When I think of algorithms I invariably come to think of the Knuth–Morris–Pratt algorithm, an algorithm for searching for occurrences of a search phrase in a larger text. A very famous string searching algorithm because it seems clever in theory and with toy examples. Your standard text book on algorithms probably does not mention this, but in practice it is not really an improvement over the simplest possible algorithm, because the issue it is trying to optimise, repeated examinations of the same character, almost never occurs when searching in natural language texts.

I found this out as an undergraduate when we were curious how KMP and the Boyer-Moore algorithm would stack up in practice. We discovered that KMP was only a 1% improvement in the number of characters compared, compared to the simple algorithm.

It's probably a great algorithm for searching DNA strings, though.

I added a note to the Wikipedia page. If I should die tomorrow, please consider it my contribution to the field of computer science.

Update

2007-05-07T21:02:00.000+02:00

Long time, no post. I finished reading Naked Economics, it didn't take me more than a week. I still highly recommend it. It did have lots of interesting points on globalisation.

One is the notion of human capital. From a macro-economic point of view, one of the best thing a country can do is to invest in its people - education, health-care and the like. The return of investment as measured in cold cash is worth it.

One example of this is the fact that countries that are rich in natural resources don't fare particularly well in the long run - because they haven't had to invest in human capital. Another example is a research facility of well-educated engineers and scientists that has to close down. A local disaster in the short run, but the engineers and scientists are highly motivated individuals that soon find something else to do. If the facility is not simply bought by a large corporation that has no other interest in the facility than its human capital.

Another point concerns one of the dark sides of capitalism. Creative destruction - those who cannot compete cannot survive. Instead of fighting this built-in ground rule, the author suggests that the transition period be made less severe by retraining the involved workers and support from the authorities. Focusing on the future instead of the past.

From the point of view of the Far East sweatshops, the author notes that the people working in these shops would not be there if it weren't for the fact that they are better than the alternatives. I think that's simplifying things (*ahem*, did anyone mention the word "exploitation"?), but nevertheless an interesting point to ponder.

In other news, I upgraded to the new Ubuntu version at work. More incremental improvements, as always.

Naked Economics

2007-04-13T10:34:00.000+02:00

I started on Naked Economics by Charles Wheelan which is about how basic economic stuff shape our society and make the wheels go around. We had some basic economic theory in high school in the history classes, so I didn't hold my hopes too high but I've already learned some profound insights after having read the first chapter. It's a great book.

One of the first thing he writes about is the market and pricing system as a decentralised system for distributing goods. Think about it.

Having worked a couple of years on decentralised large-scale software systems for the internet, it makes a lot of sense. Centralised software systems have many scaling issues. They easily break down or are very inefficient, or both. But it's not exactly easy to construct a large-scale decentralised software system that is sustainable in the long run either. And software, as complex as it is, is nowhere near the complexity of a human society.

The book also points out flaws of capitalism and how government tries (or should try) to amend those, e.g. through taxes on goods that harm the environment and thus incurs hidden costs to all of us. I'm looking forward to some well-founded thoughts on globalisation.

So far the worst thing about the book has been the foreword by some other guy (I've never heard of him, but he's probably famous). The book would actually have been better off without it.

Battle scene revisited

2007-04-11T20:28:00.000+02:00

I had a look at the battle scene of yesterday. I can see why the youngster was cycling in the middle of the bicycle lane, with the downhill corner and a thorny rose plant leaning over the lane.

But he made two mistakes. He didn't compensate for the fact that he had a sidecar mounted on his bike, leaving too little space for me to get past him unless I drove over the edge of the lane. And with such a dangerous path he failed to pay enough attention to whether someone was coming round the corner. I hope for him that he didn't bang his head so hard in the lane that he suffered a concussion. My little finger is surely still hurting.

In other news I watched Al Gore speak in An Inconvenient Truth today at NOVI. Disturbing non-news. I am still determined to buy a windmill if I ever get rich. A local upstart called ClimAid handed out adverts at the event, they appear to be selling CO2 reductions.

The duel

2007-04-10T20:18:00.000+02:00

What do a young computer scientist and a medieval knight have in common?

They both have to duel to get to work.

I dueled today on my way to work at lunch time. A malefactor rounded the corner downhill just before the tunnel I was parting through. I initially thought we would be able to pass each other without dispute, but no, he went for it with his sidecar. A big clang, my steed was forced to its knees and we were both down in the mud. My opponent hit the trodden road behind me. I landed on my hands, gauntlets taking most of the shock but little finger crushed, though not broken, then got on my feet to witness his fate.

He did not came back for another round, instead he simply glared at me, still seated on his charger on the ground, blood trickling from his face. A fair lady ran to his help; yet another tender champion ready for the hospital. Pity his ill fate. I turned my attention to my steed, its neck turned around, mud in the ear, but otherwise virtually unharmed. Checking up on my opponent again, I eventually left the scene of battle.

No young upstart shall bring a seasoned warrior down for good. A duel was fought, blood was spilled, but in the end we both survived, without permanent injury I believe.

But I don't think I'll be able to dig the rest of the garden the next couple of days.

Camera batteries

2007-03-25T23:59:00.001+02:00

I've been looking for new batteries for Janne's digital camera. We're getting tired of buying expensive Duracells. The cheap batteries go dead almost immediately.

After having read the Wikipedia articles on batteries and discovering that lithium-ion AA-compatible batteries do not appear to exist (update: I later found some on batteribyen.dk, but unfortunately not rechargeable), I was going to buy Sanyo 2700 mAh rechargeable batteries from batteribyen.dk. The shop owner recommended Sanyo Eneloop, however. They are supposed to be able to keep their energy for longer. But they are only rated for 2000 mAh...

But then I found this page (which in turn references this test, I also found another big test). Apparently some cameras are very picky with the voltage they get from the battery, turning off at the point most batteries deliver the bulk of their energy at.

But not the Eneloops. So I'm going to be buying a set of those with a charger.

The small things

2007-03-23T21:33:00.000+01:00

Still settling in at our new home. It's funny how small things can make a big difference.

Like the lighting. We've bought and put up a couple of lamps. Transforming the apartment from a grave or a scene in a detective story in the evenings. Actually I have already formed a habit not to press the light switches since they didn't work. Will have to unlearn that.

Or the stove and cooker hood. The stove is quick to warm up and easy to clean. And the hood doesn't make much noise while actually sucking pretty hard. The one at the old place sucked a lot too, but not in the sense you want it to.

Or the fridge, kindly borrowed from Janne's sister, which has one large instead of two small vegetable drawers, meaning that it can store cucumbers, probably the second most common vegetable after tomatoes in Denmark.

Or the wall-mounted things that I haven't had the time to put up before now. Changed the setting from construction site to something closer to a home.

Or the fact that the two phalaenopsis orchids which have bloomed the last few months are still carrying their flowers while the two others decided to bloom almost simultaneously. So we now have four blooming phalaenopsis in a row at the counter between the kitchen and the living room. Pictures soon, I promise.

Still on the todo list is digging the garden. Yesterday I put felt feet under the piano, in attempt to reduce floor vibrations, hopefully transmitting less sound to the neighbours. Tilting the piano was like lifting an elephant.

Efficient work force

2007-03-16T12:06:00.000+01:00

Janne mentioned yesterday that the head of a recycling company was eager to hire former farmers. Over the last couple of decades a lot of farmers in Denmark have had to stop because small farms cannot produce enough surplus to give a decent income. They used to large or medium farms, now they're simply too small. So many farmers have had to find a job.

A similar wave hit the country when industrialisation started kicking in and women in large numbers began joining the work force. I remember reading about egg-packers in a factory in literature classes in high school. The point is that those women were also valued because they were used to work for themselves and thus act responsibly and conscientious.

I think there's a parallel here to open source developers who have run and nurtured their own projects. It shows that they care.

Whether there really is a difference, I don't know. I haven't met that many non-open source developers and in any case I'd think that discipline is much more important in non-creative factory work.

Finally moved

2007-03-12T12:44:00.000+01:00

So we moved the remaining things Saturday and are now officially living at the new place. We were pretty bombed Sunday but managed to buy some flower bulbs and a small raspberry bush to put in the garden. I'll post pictures soon. It's nice with the extra space, although the bed got stuck in the staircase a couple of times - we barely managed to squeeze it through.

We still need to do some cleaning and painting in the old place.

Also the internet connection arrived today (5Mbit/256 kbit ADSL2) right on schedule. So far it's working fine.

AJAX

2007-03-08T10:25:00.000+01:00

I started looking into AJAX. We want to use AJAX for doing searches for client.

Turns out it is much simpler than I thought, at least if you have worked with asynchronous programming, callbacks and closures before. First thing to note is that it's a good idea to use a library to do most of the Javascript stuff because of browser incompatibilities. I digged up jQuery which seems sensible. I suggest starting with this tutorial from the documentation page.

There's really not much more in it than a few lines of simple Javascript code (because of the library) on the client side. You register a callback and can inject things into the page when you get data. On the server-side is a standard PHP/Django/ASP/whatever script that parses GET or POST parameters and returns an XML string. It's even simpler than normal form processing because you just return the data you need on the client side instead of a whole page.

There are also a couple of links to tutorials on the Wikipedia page on AJAX to understand what jQuery is doing under the hood.

Moving

2007-03-08T10:12:00.000+01:00

We started moving our stuff last weekend to Vilsundvej. Yesterday the piano was moved. I'm relieved it's done, but it was a sad moment. The acustics in the empty living room with its hard walls and wooden floor was just perfect for playing Bach.

This weekend we'll move the things we use daily, completing the journey - we still live in the old apartment. I took some pictures, will post them later.

Then it's just a question of getting the old apartment cleaned up and repainted.

Fantasy/science fiction books

2007-02-28T09:55:00.000+01:00

I just finished reading the Fionavar Tapestry by Guy Gavriel Kay. Borrowed the three books in one cover at the local library, in a Danish translation. I enjoyed the read, although the first book started slow. It shows that it was his first. The writing in the later books is much, much better.

Other great books I would recommend are the Hyperion series by Dan Simmons, and in similar vein the Riverworld series by Philip José Farmer. And the Foundation series by Isaac Asimov. And how could I not mention Ender's Game by Orson Scott Card.

I recently read the Earthsea series by Ursula K. Le Guin series. A bit boring at times, but otherwise good. They feel different than most fantasy/science fiction books I've read. Perhaps the gender of the author shows.

I have also read the series by David Eddings, they are not exactly great works of art. But they do their job, exciting.

I recently reread Midworld by Alan Dean Foster (it's called Grøn = Green in Danish, I didn't even know it was part of a series). I remembered it as a favourite, but was disappointed when I read it again. Too naive. I think I've had too much scientific training. It's like the Dragonlance books I used to read and really enjoy when I was young, at about the age of 15 they were simply unbearable.

How to dress

2007-02-25T23:19:00.000+01:00

Some tips for dressing, if you're in the trade.

I already knew most of them, of course. Janne told me.

Reminds of thingsmygirlfriendandihavearguedabout.com, probably the longest domain name ever registered.

The secret of success in business

2007-02-22T10:18:00.000+01:00

This illustration is just hilarious.

While I'm at it, I stumbled upon a webshop with orchids. It's short, pretty and to the point. Great example of how to construct a webshop.

And here's how to choose a colour scheme.

Success!

2007-02-19T21:03:00.000+01:00

The local newspaper I've mentioned before called us 8 o'clock this morning. We won an iPod Nano, amongst 34 correct solutions!

This was the fourth week we participated, with the number of correct solutions so far being 17, 27, 46 (that week it was particularly easy) and 34. Thus the probability of winning one iPod has been 1/17 + 16/17 * 1/27 + 16/17 * 26/27 * 1/46 + 16/17 * 26/27 * 45/46 * 1/34 = 14%. I guess we were lucky.

There's one problem, though. All my ripped music is encoded in Vorbis. I definitely don't want to store my data in a proprietary format that Apple is controlling. I'll have to take a look at Rockbox, although it seems the 2nd generation iPod Nano is not supported. Sigh.

Frustrated users

2007-02-06T14:32:00.000+01:00

Spent some time with a link I got from the blog of Federico Mena-Quintero. Apparently, people get frustrated often with the programs they use most (word processing comes out top) and waste on average about 40% of their time.

Some of the suggestions in the paper are simply to rephrase unclear wording and work on the error messages.

Another paper suggests an adaptable file browser. Their concept is pretty simple - replace seldomly used folders with an ellipsis. Easily generalised.

REST, web speak, architectures

2007-02-04T17:50:00.000+01:00

Why are new technologies on the web so slang infested? I found a reference to REST which I thought from earlier impressions was an architectural style for building web applications. Apparently, it is more complicated than that. Researching the matter is difficult, the usual ACRONYM gabble-gabble, future gobble-gobble.

So I looked it up on Wikipedia and found this little gem, out of the blue:

Systems that follow Fielding's REST principles are often referred to as RESTful; REST's most zealous advocates call themselves RESTafarian.

Rereading Wikipedia's summary the ideas in REST begin to make sense: build systems with URLs and send-command/return-contents protocols that are stateless and cacheable, instead of the usual remote procedure call/method invocation paradigm.

An interesting aspect of XML-RPC and SOAP is that you take a flexible format like XML that can easily support forwards and backwards compatibility and force a strict RPC interpretation down on top of it (if I understand things correctly, admittedly I'm no expert on web services). It may be easier than parsing the XML itself, but it is a poor man's version of a protocol. And why use XML when you're not really exploiting it anyway? Certainly not of performance reasons...

I have a feeling the requirement of a stateless protocol is a problem in many circumstances, though, having spent some time working on distributed file systems. Sometimes it's just better if the server can send a notification back to a client instead of the client having to poll. I believe NFS has left its much touted stateless approach with version 4.

Harpsichord

2007-02-03T18:09:00.001+01:00

My mother has bought the scores for the chromatic fantasia and fuga by J. S. Bach (BWV 903) for me, apparently mostly because she heard it in the radio, liked it and then the scores suddenly were there in the book shop.

So I have been trying to find a free recording of the piece on the web. I don't have time to study it right now, but I need to know where to put it in the ever-growing stack of pieces I'm planning to work on in the future. No real luck yet, although I managed to find a very nice sample on harpsichord by George Malcolm (a famous harpsichord player). Still looking for a piano recording, although I suspect I will have to visit the library or spend some money on it.

I ended up on Magnatune. Their collection of classical music has grown quite a lot. Apparently about 30% of their sales are for classical music, no wonder, they have some real gems in there. I'm currently listening to the French Suites after having enjoyed Rameau for a moment. There's nothing like harpsichord solos to bring forth the baroque mood. Tiring after while, unfortunately, I suspect because of the lack of dynamics. That's one point a Yamaha keyboard has going for it. You can play harpsichord with graduated touches.

Feed reader

2007-02-02T22:39:00.000+01:00

I finally succumbed and started using an RSS feed reader. I'm trying Google Reader right now, we'll see how it goes.

It's quite fun to add feeds. It's like creating your own newspaper.

Gambling

2007-01-27T15:34:00.001+01:00

One of the local newspapers has a puzzle with the prize of a 2 GB iPod Nano. You send an SMS with the solution to a number, pay 5 DKK (app. $1, a little less than 1€) and take part in the draw if your solution is correct.

I recently noticed that they are kind enough to print the number of correct solutions sent to them the previous week. It turns out it is very few. This week it was 28. That probably says something about the quality of the newspaper...

Anyway, what's interesting here is that you pay 5 kroner to get about 1/28 chance of winning a prize worth 1600 kroner. Assuming a probability of 1/28 every week, the outcome is binomially distributed. The expected number of wins is thus np, the number of draws times the probability. Consequently, after 28 weeks you can expect to have won the prize once, paying 140 kroner to get an iPod!

So we've decided to start sending in the solutions from now on.

The thing about gambling is that the variance is usually high. For the binomial distribution the variance is np(1-p). Since the probability is 1/28, the extra factor compared to the expected value is 1-1/28, or almost 1. In other words, the variance is almost the size of the expected value. We take the square root to get the standard deviation, but this does not change the value much.

For the normal distribution, which the binomial distribution approximates, one should not be surprised to get a value within 2 standard deviations of the mean (this is about 95% of the probability mass). Here are two plot that shows the expected value (red line) with one and two standard deviations added as a function of the number of weeks. The first one is zoomed in on 0-28 weeks.

The good news is that after 28 weeks, I wouldn't be surprised to have won two or even three iPods. The bad news is that I might as well not have won one at all. As you can see on the second plot, it takes more than 150 weeks to make the expected value minus two standard deviations greater than one.

I am crossing my fingers.

Open source community building

2007-01-22T09:43:00.000+01:00

A guy describes his experiences with building an open source project. The most interesting part is the check list to the right.

A model of eating

2007-01-21T21:38:00.000+01:00

I have known for some time now that Janne and I don't eat a lot of meat. It's not that we are vegetarians, most of our meals include meat in some form or another, it's just that the amounts are small.

On the other hand, we eat a lot of vegetables. Since moving in with Janne I'm beginning to think that a standard Danish dinner consisting of some kind of meat with boiled potatoes and a sauce is extremely boring. It needs to be augmented with vegetables.

So today I was wondering whether there is a correlation between these two facts, less meat and more garniture. I think I've found a model that can explain it.

When I eat, I usually alternate between putting the various stuff on the plate in my mouth, e.g. with potatoes and a beef it is one mouthful of potato with sauce, one mouthful of beef, one mouthful of potato, etc. Let's pretend it takes 100 mouthfuls to get full, ignoring differences in the saturation effect of different types of food. Then roughly 1/2 of these, 50 mouthfuls, will be of meat.

If instead the plate contains of two kinds of vegetable garniture plus the potatoes (Janne and I almost always cook them in the oven and forgo the sauce) and the beef, then I'll usually alternate in a round-robin fashion between the vegetables, the potatoes and the meat. So only roughly 1/4 of the mouthfuls will be of meat, i.e. 25 mouthfuls.

If the model is correct, and I'll think about validating it, then I eat about twice as much meat at a traditional dinner at my parents' compared to a typical dinner cooked by Janne or me, without really noticing it.

So now I'm just left wondering exactly how many mouthfuls it takes for me to get full. I think I'll try counting them.

Starving

2007-01-16T16:45:00.000+01:00

Some guy tried living on $30 for a month. Incidentally, he appears to have found an effective means of losing weight.

Talk by Claus Elgaard

2007-01-15T09:56:00.000+01:00

A couple of months back I attended a talk by Claus Elgaard with my iola comrades. Coming from the world of sports and athletes, he talked about motivation, preparation, excitation and hybris in the extreme. Fascinating. I found my notes while cleaning up the office in preparation for our move.

Unfortunately, almost noone showed up so financially it was a disaster. A picture of life, perhaps.

Small games

2007-01-13T17:30:00.000+01:00

I recently discovered Pang Zero, a small game in which you try to avoid getting hit by balls and at the same time shooting them away. I have been playing it a couple of times lately. It reminds me of another game, AckAck!, that I played back when I was using Allegro. Unfortunately, I can't find it anymore.

Could be interesting to try speed programming small games like these with something like Pygame. I've always spent too much time mulling over the details with my game projects. Perhaps that's why I only got Monster Masher finished enough to actually release. I have the source code for a Spacewar game, a Scorched Earth-like game (we called it Gnome Tanks) and a clone of Helter Skelter stored somewhere, none of them complete enough to be really playable.

Still, I'm secretly proud of Gnome Tanks. The AI can aim perfectly in all terrains, even when it is windy. Wind transforms the equations into something that cannot be solved analytically so I had to research and implement a numerical solver. When I wrote the code, it was the only Scorched Earth-like game that could do this. I wonder if others have figured out how to do it by now.

Usability testing

2007-01-10T23:11:00.001+01:00

I just discovered that betterdesktop.org has gotten quite some content now. Cool!

I think they could do with a usability test of the site itself, though. It's a bit confusing.

About burn-out

2006-12-30T23:05:00.000+01:00

Found an article on burn-out. Interesting, albeit with more talk than thought. Quote from the article: "... happiness equals reality divided by expectations."

I found the link via the blog of some guy working at Google. The blog is an entertaining read. Check out the famous people he's seen.

Update: I subsequently read that Danes are generally more content than people in southern Europe because they on average expect less of their life.

Age of plants

2006-12-27T12:52:00.000+01:00

I found a fascinating article in the Danish Wikipedia about the biology of trees. Written in a intriguing language, like this quote about defense mechanisms (my translation):

Within seconds the tree knows that it has been hurt. As soon as the bark is breached, the tree needs to defend itself. If fungus and bacteria spores find an uncovered piece of wood - and they will - they will launch an attack on the wound and into the wood.

I wonder whether it'll survive the editing process in the long run.

I was looking for information about how old plants can grow. Indoor perennial plants and flowers are commonly multiplied by division. But that doesn't reset their age. If they have one. I still don't know.

Imagery

2006-12-26T16:38:00.000+01:00

I was showing my sister how to do some basic things with The Gimp. Just yesterday I was experimenting with the gradient tool in The Gimp to create some graphics for my website with footage from photos we've taken. I'm a complete beginner when it comes to photography, but I managed to find a decent photo.

Reminds me about thefairest.info. And thecutest.info. Excellent stuff!

Christmas

2006-12-24T10:56:00.000+01:00

That time of the year again.

I heard a small passage from Bach's Weinachtsoratorium on P2 (Danish national radio channel). Reminded me of Messiah by Händel. One of the verses, song by the alto, is about "the lamb of God":

He was despisèd and rejected of men: a man of sorrows, and acquainted with grief.

A quote from the bible I believe. I'm not a Believer, but that quote is ingenious. Take that, happy Christmas! :-) You have to hear it in Händel's interpretation to really appreciate it. But that's easy, just scroll down on the Wikipedia page and there should be a link.

Charity

2006-12-23T18:33:00.000+01:00

I'm in the midst of cleaning up my home page in preparation for moving it from the university domain to our company server. So I stumbled upon the link I put in to Amnesty International.

I have an ambivalent attitude towards charity organisations in general. I think we ("we" as in people like me who can listen to Wilhelm Kempff and Alfred Brendel all day long) have a responsibility to help people, even if they are half way round the globe. The mere fact that we know about them and easily have the resources to help is enough to institute the responsibility.

It's like if you encounter someone lying on the highway with a broken leg in the middle of the night. As soon as you see that person, you have a responsibility to help. It's a criminal offense not to do so.

That said I'm not particularly fond of private organisations spending money and resources on collecting monetary donations. They do so by promoting guilt, which is a horrible thing to do. It also seems wasteful. The tax system is already implemented and works. And I think it favours short-sighted fire extinguishing instead of long-term pulls. People donate a little to nurture their conscience when a major disaster has happened, and then half a year later go vote for a party that cuts the foreign support in half.

Reminds me about the tragedy of the commons, an interesting essay which among other things touches upon why a completely free market is not always a good idea and why taxes are sometimes preferable to the initiative of individuals, given the nature of human kind.

But back to Amnesty. I'm a member of Amnesty (and paying to be so) because I believe Amnesty is actually working on a long-term project, securing the rights and the freedom of people. Basic human rights cannot be taken for granted, not even today.

Magnatune, Händel and playing Beethoven

2006-12-23T14:20:00.000+01:00

I'm listening to music on Magnatune, an online music store that lets you listen to the music (all of it) before you buy and offers non-CRAP formats.

They've added opera. I'm not a big fan of opera, but it turns out to be composed by Händel. Who I'm also not a big fan of, but supposedly he was good at the opera stuff. So I'm giving it a try. Frankly it sounds great!

On a related note, I'm practising a sonata by Beethoven for the time being. A small and delightful one, opus 78 according to the scores. It's not the first one. I've previously played and enjoyed the Moonlight and the Waldstein sonatas.

The third and probably less known movement of the Moonlight sonata is an aggressive contrast to the tranquility of the first movement (the one with the moonlight) and the joyous mood of the second movement. Fun albeit not easy to play. Plus it's a bit loud; a problem with three neighbours in close vicinity.

The Waldstein sonata is very long. I don't think I would have the stamina to play it in its entirety in front of an audience. It's great, though. I only started rehearsing it because I fell in love with a recording of the piece by Wilhelm Kempff (from Deutche Grammophon if I recall corretly). Mmmm. The sound quality is obviously not great, but the interpretation and the excellence of control...

What I like about Beethoven's piano music and symphonies is the tremendous passion.

Chopin's music has the same quality in his preludes and nocturnes, but it's mostly about melancholy. Less anger, less joy. I think my next piece to study will be either the last or second to last sonata by Beethoven. I have great recordings of these by Alfred Brendel. I feel fortunate to be able to listen to such an artist, by a mere poke of my finger, as often as I like. Great times we live in.

Fiction editing

2006-12-22T17:46:00.000+01:00

Sometimes interesting stuff shows up at Slashdot. A post about editing of fiction novels (in the context of celebrity writers).

CSS fun

2006-12-13T22:06:00.000+01:00

I'm going to redo my home page. Still thinking about how to do it. Meanwhile I found a nice little page about CSS positioning. And here's a longer tutorial on floating stuff with CSS.