Bulk inserting Django objects

I've added another Django utility member to my garden of Python, bulkops.py.

Currently, it consists of two functions, insert_many and update_many. They are intended to solve the problem of inserting a big bunch of objects in Django. If you use the standard save() or create() methods in Django, you get one SQL query per object. With thousands of objects, you'll simply drown in overhead in executing all these queries.

There are some alternatives out there, but unfortunately they are pretty complex and specific to a previous version of Django. Instead this code simple makes a list of tuples with the fields on the model and feeds that to the executemany() function in the DB API. Usage is really simple

for x in seq:
    o = SomeObject()
    o.foo = x
    o.save()

becomes

l = []
for x in seq:
    o = SomeObject()
    o.foo = x
    l.append(o)
insert_many(l)

It's tested to work with Django 1.2 and doesn't really touch the ORM internals so should be safe for at least some future versions.

Sadly, there's no support for associated many-to-many relationships. I actually needed that myself, but I don't think there's an efficient way of getting the primary ids when you do a bulk insert with the Python DB driver API. So one has to use a select query anyway afterwards, and then it was actually more convenient to do it by hand by rearranging the logic in the caller.

Cache busting in Django

One of the really annoying things in web development is dealing with caching. Browsers incorporate an elaborate caching scheme to speed up common browsing operations, which is a really great idea. But a side-effect is that when you release a change on a site, you risk that some people get the new HTML while their browser is still using an old CSS or Javascript file because their cache hasn't timed out yet.

It has bitten me several times during development too, some browsers have really aggressive caching schemes. Debugging problems caused by a stale CSS/JS file is just not fun.

One way to fix it is to use a framework to run the CSS and Javascript through a combiner/minifier that at the same time outputs a versioned filename, e.g. by hashing the contents. There are even a couple of Django projects going down this route to make it happen automatically.

I haven't found one I liked yet, though. You have to modify views or templates and fit your paths into the system, it doesn't help with changing images, if you use third-party code you have to modify that too, and the minification step makes it pretty hard to debug Javascript problems; even if you isolate it to live, problems happen on live too, sometimes.

Really, if we take a step back from the minify dream, the simple solution here is to append some kind of version number to the file. Then new HTML should have filenames with new version numbers so clients can't used their old cached files because the paths don't match.

I've found a really simple way to automate this transparently for a project by modifying one line in settings.py. The idea is to hook into Django with a middleware class, search the generated HTML for local URLs and replace them with URLs with version numbers. This way, we can catch all CSS and Javascript references as well as images no matter where they occur.

Since common file systems don't provide version numbers on the files they store, we'll use the modification timestamp instead. And instead of renaming files to contain the version number, we'll just append a ?_=timestamp to the URL:

  <img src="/media/images/logo.png">

becomes

  <img src="/media/images/logo.png?_=4321234">

Everything after the ? is ignored when looking up in the file system by the static file web servers I've tested so far. So it just works.

As a bonus, when this is running, you can safely modify the web server serving the static files to set an expiration date far out in the future for files requested with ?_=xxxxx. This allows an aggressive, yet perfectly safe caching scheme to be employed by any upstream cache, including the browser.

The above trick can be done in about 50 lines of Python. You can grab the little self-contained middleware module modtimeurls.py from my brand-new Garden of Python. It has been in production on several sites for over a year. The only problem I've found so far was with a PNG transparency fixer for IE6 that assumed .png files would end with .png rather than .png?_=xxxxxx.

Stoicism

This is philosophy in practice. One of the most interesting posts I have read this year.

Emacs learning curve

Think this is old. But still funny.

Piano synthesis

I've bought a digital piano. Its most important feature is head phones (not included). No more neighbour complaints.

Because of this, I've also been spending some time, actually too much time, on looking into piano sound synthesis recently.

The problem is that commercially available synthesizers built into digital pianos don't sound very good. Actually, for people who don't play the piano, they're probably fine. But that doesn't help me!

So I set out to find out what else is out there. Basically, there are two approaches.

Play back
One approach is that you sit down in front of a real piano with a couple of microphones and record the sound from all of the keys several times.

You can't just record one note and shift its frequency because the character of the tones change dramatically from the low end to the high end.

Furthermore, you can't just record each key once because when you press a key harder, the sound is not just louder but also changes character. For a pianist this change of character is very important. I don't know whether this is true for others, but at least for me my brain quickly adjusts itself to changes in loudness so without a change of character, the result is a flat sound inside my head.

So the end result is that you need a lot of recordings. 88 notes times at least 15 hardness levels and the decay of the low notes can easily last over 30 seconds before they die out. That's a pretty large amount of data to record and process; as I understand it, you need postprocessing to smooth out inconsistencies in the recordings. And still, in my experience, nobody has figured out how simulate a short staccato attack convincingly with a long sample. Not to mention other interesting things you can do with a real piano, once you get it singing.

Mathematical models
Instead of recording a real piano, another approach is synthesizing the sound from some sort of mathematical model, just like ray tracing a modeled 3D scene to generate realistically looking special-effect images in movies.

Now, some models start from the idea of trying to emulate the sound of the piano itself directly, based on observations like the sound being nearly harmonic so if we just add some sine waves at regular intervals, we've almost got a piano sound.

As far as I can tell these all sound really awful.


One of the first pianos. It was before the steel frame was invented. From Wikipedia.

I think the explanation is that generating sound is easy, even easier when you've got a computer to add all the sine waves you like, but getting an interesting sound is really hard. And piano makers have worked on this since at least 1720 using wood and metal, materials with plenty of opportunities for interesting sounds. So a fairly simple model is just not going to cut it, not when it's up against a 400 kg carefully refined wooden steel-string monster.

Another idea is to try to deduce how the various things inside the piano affect each other. If you look closer, the vibration of a string is a periodic push and pull at the ends of the string. If we follow the interactions from a key hitting a hammer that in turn hits the strings, causing them to vibrate, setting the big wooden plate inside the piano, the sound board, into motion, causing the air to vibrate, which is finally perceived as sound in our ears after undergoing reverberations from the room walls; then we can, in theory, reconstruct a perfect sound through the formulas from physics governing the interactions. A similar idea is used to simulate what happens when things collide in games these days, by the way.

Now, in practice, it's of course not so simple.

First, as far as I can tell, nobody has an accurate complete physical model from hammers to strings to soundboard to dampers. So the equations are not there - not yet, although there appears to be a few people working on it.

Second, modeling a whole piano in a naive manner is just not tractable in real-time with commodity computers because you need to simulate the whereabouts of a huge number of points at least 44100 times per second to generate a playable 44.1 kHz sound. Thus it is vital to identify what constitutes the audible important dynamics in the process and spend the time simulating them.

Despite the challenges, these physical models can generate quite convincing sounds. You can buy them, currently either in the form of Pianoteq or the Roland V-Piano. What's nice about them is that you can get them to sing like a real piano because they're actually simulating vibrating strings, not just turning recordings on and off.

Unfortunately, the models aren't perfect yet, and it's certainly audible, generally mostly in the middle registers, the reason being, I think, that at the upper end the sound is actually mostly the thump from the key hitting the bottom of the keybed, and at the lower end it's mostly about generating a really big set of partials so errors here are less noticeable.

Research on physical modeling
When I started looking at synthesizers I was mostly annoyed that neither the company behind Pianoteq or Roland disclose how they do it. Which probably explains why both haven't gotten it quite right yet. So I started googling for research.

It does not take more than a cursory glance to see that synthesizing acoustic instruments of various kinds is an active research topic. As far as I can tell there are two cooperating groups working on piano synthesis these days, at least in the English-speaking part of the world.

One group is at Helsinki University of Technology, some papers published last year, some earlier. Unfortunately, I got the impression that their model is not very convincing yet.

The other group is at Verona University, in collaboration with an Italian company and instrument maker Viscount. It's possible to google up some information about this group, but I think only Balázs Bank who's also been at Helsinki has an up-to-date home page.

I have spent some time working on a test implementation of the Verona model based on the paper describing it, and I'll dedicate the rest of this blog entry to my findings.

Modal piano synthesis
The Verona model is based on modal synthesis where one starts from the equations from physics and from them figure out at which frequencies the strings exhibit standing self-reinforcing waves; which means they last and are audible after several seconds, an eternity in the millisecond world of impacts.

The physical responses at these frequencies or modes or partials are then simulated with a simple second-order model. Second-order means that the simulation on each time step uses the previous two values. So for instance, for one string partial, the displacement at time n is calculated as y[n] = a*y[n-1] + b*y[n-2] plus a contribution from the hammer if it is present. If you're wondering how this is started when n = 1, well, we just assume the string starts at rest so y[n-1] = 0. The calculated displacement is basically the sound.

Hence, once the constants a and b have been chosen, the math is actually pretty simple.

Of course, in reality it is a bit more complex. The model keeps track of the hammer motion, simulates the impact on the string and the resulting forces and motions at the modes which include both transverse modes (up-down or left-right motion) and longitudinal modes (motion back-forth in the length of the string) plus some longitudinal motion stemming from tension changes by the transverse modes. In order to simulate the sound passing through the soundboard (and the room), the motion of the modes are added and convolved with a recorded soundboard impulse response.

The details are in the paper.

Disappointingly, if you implement what's described in the paper and enter some reasonable values of the constants, it doesn't sound like a piano. This is of course with the caveat that I might have made a mistake in my implementation.

However, as far as I can tell, there are at least two important things missing from this model. One is the dampers. All pianos have a felt damper connected to each key, the purpose of which is to silence the strings of the key when it is not pressed down. Pianos produce quite powerful sounds so without some way of silencing notes again, it would be hard to make quick pieces sound good. Surprisingly, there does not appear to be much literature on dampers. The only thing I've found is this paper from last year.

The other thing missing is some feedback loops. From the soundboard back to each string, between the strings, especially between the strings on the same key (most keys on the piano have more than one string), and between the different transverse directions on the same string. These feedback loops cause interesting dynamic effects.

In the paper the soundboard is simply dealt with by setting the decay rates of the modes according to a simple model. This corresponds to the soundboard stealing energy from each mode and not delivering anything back. I did an analysis of the piano samples available from the University of Iowa, the decays of the partials are plotted below together with the simple model fitted to the data.



As you can already see from the plot, this is not a terribly accurate model.

In the following, the spectrograms show at each frequency the level of sound over time, capped at 5 kHz and 7 seconds, I've generated them with sox. First a sampled tone, C3 at fortissimo.



I think the weird vertical bars might be echos from the initial thump. Now, if you use the simple function derived from the decay data, the result synthesized sound looks like this (sound example).



The thing is that the soundboard doesn't really follow the simple model in the way it responds to the strings, e.g. it has modes of it own. As a quick measure, I tried to set the decay to decay rate of the nearest measured mode, then it looks like this (sound example).



I'm not sure what the conclusion is. One attractive feature of the simple model is that only has a couple of parameters so it's easy to change around. I think if one has a more complete model of a soundboard, it might be possible to deduce from that more precisely how it absorbs the sound and set the decay rates accordingly. But still there might be some dynamic effects that aren't taken into account.

A thing one notices in the spectrogram for the sampled tone is the pulsating nature of the partials. They don't just die out exponentially. The pulses come from both the interaction between the different transverse directions on each string and also the interaction between the other strings on the same key.

The paper has a couple of suggestions for dealing with these. Basically it boils down to adding some extra modes that can produce beating. The thing is that if you add a mode close to another, the difference in frequency causes the sound to pulsate because of the interference between the two.

But there's no recipe for how to set the beating.

My conclusion so far is that you can't use a global value for the beats, they need to be specific for each mode. I haven't gotten around to actually measure these values from the samples. As a quick work-around, I'm right now setting a random beat for each partial; this improved the sound considerably. While measuring samples may be one way to fix the issue, I can't help thinking that this is one area where physics could solve the problem much more precise with less hassle if somebody could figure out how.

I haven't yet gotten to the point where this sounds realistic enough to warrant a full synthesizer that can eat MIDI. But I get the impression that Pianoteq is using modal synthesis too, so it's certainly possible. If you are interested in the code I've written so far, feel free to contact me.

If you're interested in how the piano works, a good resource is the Five lectures on the acoustics of the piano.

I'll conclude with a random note. The Verona paper only operates with one string per key, but actually a real piano has more so if you use physical meaningfully parameters and don't take this into account when computing hammer impacts, it's going to be off by quite a lot; for a hammer, there's a difference between hitting one string and three.

Software development and the waterfall model

As mentioned, I recently read The Design of Design by Fred Brooks.

He covers a lot of ground, albeit not in any rigid fashion. As such, it is not a handbook, but more a collection of thoughts about the design process, most grounded in personal experience. Fred Brooks knows his stuff.

An interesting point he makes is that the waterfall model of software development is completely flawed and should be abandoned.

In short, the principle in the waterfall model is reason - good solid thinking: first we figure out all what's needed in a given system, then we figure out all aspects of how to build it, then we build it.

He has several arguments for why the model is flawed. One of the simple arguments is that it doesn't model how good designers work. Naturally, if you want everyone to follow a certain mode of operation, it would seem like a good idea to copy the mode followed by the good ones.

For me, the biggest flaw of the waterfall model and its companion, the big fixed-price project, is that it requires one to make the decisions up front where the uncertainty is greatest and the consequences of each decision most remote.

This shows up even in the earliest phases, e.g. when discussing whether to include a certain requirement or not. Inclusion may mean that the project will run over budget and schedule and produce a completely impractical never-used feature. However, the budget overrun may not even be visible in the design phase, but only show up late in development, and the futility of the feature may not show up until years after deployment where it is discovered accidentally that nobody uses it.

An important point in the book is that the design process is a learning experience, a process where one learns what's needed and what's possible. And what's elegant and what's good in the particular circumstances. I'd add that the same is true of the embodiment of the design, the code itself. Often people won't understand a system properly until they sit in front of it and have to start doing what they need to do. Deployment is always a learning experience. And similarly, it can be hard as a developer to assess the consequences of a paper design. When the code is written, maybe that fancy idea everybody liked beforehand isn't that hot afterall.

If one doesn't allow for learning, both for the receivers of the system and the people who deliver it, it's really difficult to get the software right.

By this thinking, the rational model becomes a bureaucratic way of delivering what might just turn out to be the wrong system.

Garden design

I recently read The Design of Design by Fred Brooks, the guy behind the mythical man month and the magical silver bullet debunking. I'll save further commentary on the book for now, but one of things he mentions is that rules sometimes help the designer because it sets some boundaries within which creativity can blossom.

I have a couple of rules for my little patch of a garden.


A passiflora growing outside in the summer

A first simple rule is that there shall be no grass. None at all. While lawns are nice and everything, it takes a lot of labour to keep a lawn, ideally it needs mowing at least once a week. Even then, unless you have the perfect growing spot for grass (plenty of sun and water), chances are it won't be a pretty one. This is compared to the other beds which I weed maybe two or three times a year while otherwise enjoying the flowers.

Also everybody has a lawn. Most don't look pretty. No fun at all.



Another rule is that there shall be as little uncovered ground as possible. Naked fertile soil is a highly unstable labour-requiring condition, nature will have its way. It also looks silly.



Yet another, less simple principle is that the garden must grow organically, chaotically, like nature, as opposed to being a tidy, trimmed show case of culture. As few straight lines as possible. Make no mistake, it's still designed in the sense that I exert control, it's not nature; if I didn't manipulate it heavily, it would turn into a shrubbery in a few years. But I'm attracted to idea of a green wilderness, to seeing things grow vigorously.

So I tend only to pull up the most annoying weed, the species that are hard to get rid of, and I let the decaying plants stay where they die (from a pull-up or otherwise). It's easy and it works well, most of it is gone when the season is over.

I try not to have too many of the same species in the same spot. Nature seldomly has 10 similar specimens right next to each other at exactly the same age. This also offers some protection against various pests.

And most of the plants I've bought are from seeds or small specimens from the local super markets. There's no overall plan to follow, hence I can satisfy sudden impulses without getting into trouble. And I don't mind having to wait to see the results, watching things grow and looking forward to the adult specimen is part of the joy. It's like buying yourself a gift that you cannot in any way unpack until a year or two after.

A side-effect of the wilderness idea is that my little patch has a lot of small visitors. Which is certainly something to be happy about. Sitting a summer day in the sun, with bees buzzing around collecting honey from the flowers one has put in place many months ago, removing weeds here and there, noticing details one has never seen before, discovering new species that have invaded the soil on their own. I can think of nothing more peaceful.




Of course, I'm learning from this approach. This year I'm beginning to think I need one more rule. Plants on the pathways must be cut down no matter how interesting they look, otherwise it gets too cumbersome to get around, and then I seldomly get to see the outermost corners.