A take on good, reusable Java design

This is just too amusing not to pass on.

Spring time!

It's that time of the year again here in Denmark.

The crocuses started flowering, and I managed to sit outside for quite a while in a short-sleeved shirt without freezing to death. There's no doubt. Spring arrived this weekend.

It's been a busy winter. I...

... managed to fall sick with flue-resembling symptoms and fever three times in 6 weeks, beginning with the Christmas holidays.

... shoveled a great deal of snow away, like last year.

... closed probably around 100 issues in the Flot issue tracker (and had the misfortune not to discover a well-meaning fork until a month after).

... mapped a big bunch of roads in OpenStreetMap, mostly centered around Janne's parents home. Thanks to the effort of volunteers, we've got aerial photos of almost the entire country and will, I think, within this year have a complete free street map of Denmark. I found out mapping is a fine hobby; it's a nice way to relax while gaining the usual open source feeling of contributing to something greater than yourself.

... read 16 books on local history of Vendsyssel and Himmerland, the regions where I live, told by old people that recount their youth around 1900-1930, mostly with farming, quite an eye-opener, really, and a great complement to the things we were taught in history lessons in school. I also read the Arthur trilogy by Bernard Cornwell (it wasn't quite as good as his Saxon book series, although maybe I'm just biased because I prefer the brought-up-as-a-Dane protagonist from those).

... wasted some time with Tremulous 1.2pre after having stopped playing Tremulous for a long time. It is a surprisingly difficult multiplayer FPS. I mostly play the alien side because I like the speed, but since I'm still not very good at it, this means I have to trick other players into coming close enough to be vulnerable.

... and annoyingly failed to get into the habit of regular piano practicing again.

Flot 0.7 released!

I just released Flot 0.7! Highlights are multi-axis support, area charts and a pie chart plugin, and a big bunch of bug fixes, including an IE 9 fix and reworked replotting support that reuses the canvas elements to put an end to the memleak troubles. The complete list of changes is here. There are some API changes from the axis rewrite.

In related news, I've moved the source code repository to github to make it easier for other people to contribute.

It has been clear to me for some time that more of the things that are currently inside core Flot needs to move to plugins. As time goes, more specific things turn up on people's wishlist, and right now it's just not possible to satisfy them all without bloating the core. Having specific feature sets in a plugin also makes it much easier for people to customize them a bit by hacking the plugin code without having to maintain a delta with the whole of Flot.

Fortunately, some people have expressed interest in helping out, partly due to me taking a hiatus from the mailing list for some time last year, so with a bit of luck some of these things may happen in a not too distant future. Maybe we'll even get a better hyperlinked format for the documentation, something that has never really managed to climb to the top of my list of priorities so far.

Lyrics for GNY Beirblakken

Janne and I had some trouble finding the lyrics for Beirblakken performed by GNY (listen to it on their Facebook page). Turns out it's from a Norwegian folksong Beiarblakkjen (which perhaps originally means a war horse). There are a couple of versions of it. The full reconstructed text sung by GNY is approximately this:

Der satte tre kjerringar under ei bro
- dei trea gullsko
Dei skapte blakkjen av mannblo',
- så vide fara dei løyndar ord

Dei skapa blakkjen af manne bein,
- dei trea gullsko
og Beiarblakkjen så kalla dei han.
- så vide fara dei løyndar ord

Og då dei hadde blakkjen gjort,
- dei trea gullsko
då sende dei han til kongjens port.
- så vide fara dei løyndar ord

Da kom'e Niklas fride,
- dei trea gullsko
sa: Eg tore blakkjen ride.
- så vide fara dei løyndar ord

Niklas red til han bli' mo.
- dei trea gullsko
Til støvlane hans bli' fulle av blo'.
- så vide fara dei løyndar ord

Dei første steg som blakkjen sprang
- dei trea gullsko
Da kom han sig til helvede frem
- så vide fara dei løyndar ord

Da han kom til himmelriks dør,
- dei trea gullsko
der hadde blakkjen voret før.
- så vide fara dei løyndar ord

Gravde dei ham i vigde jord,
- dei trea gullsko
så inkje ravnane drikke hans blo'.
- så vide fara dei løyndar ord

This Norwegian dictionary was most helpful in deciphering some words - "kjerring" means wife/old woman, "løyndar" hidden, "trea" tread, "fride" fair, "mo" tired/paralyzed, "vigde" consecrated, and "blakkjen" is a horse (of the old Norwegian fjord horse kind).

Bulk inserting Django objects

I've added another Django utility member to my garden of Python, bulkops.py.

Currently, it consists of two functions, insert_many and update_many. They are intended to solve the problem of inserting a big bunch of objects in Django. If you use the standard save() or create() methods in Django, you get one SQL query per object. With thousands of objects, you'll simply drown in overhead in executing all these queries.

There are some alternatives out there, but unfortunately they are pretty complex and specific to a previous version of Django. Instead this code simple makes a list of tuples with the fields on the model and feeds that to the executemany() function in the DB API. Usage is really simple

for x in seq:
    o = SomeObject()
    o.foo = x
    o.save()

becomes

l = []
for x in seq:
    o = SomeObject()
    o.foo = x
    l.append(o)
insert_many(l)

It's tested to work with Django 1.2 and doesn't really touch the ORM internals so should be safe for at least some future versions.

Sadly, there's no support for associated many-to-many relationships. I actually needed that myself, but I don't think there's an efficient way of getting the primary ids when you do a bulk insert with the Python DB driver API. So one has to use a select query anyway afterwards, and then it was actually more convenient to do it by hand by rearranging the logic in the caller.

Cache busting in Django

One of the really annoying things in web development is dealing with caching. Browsers incorporate an elaborate caching scheme to speed up common browsing operations, which is a really great idea. But a side-effect is that when you release a change on a site, you risk that some people get the new HTML while their browser is still using an old CSS or Javascript file because their cache hasn't timed out yet.

It has bitten me several times during development too, some browsers have really aggressive caching schemes. Debugging problems caused by a stale CSS/JS file is just not fun.

One way to fix it is to use a framework to run the CSS and Javascript through a combiner/minifier that at the same time outputs a versioned filename, e.g. by hashing the contents. There are even a couple of Django projects going down this route to make it happen automatically.

I haven't found one I liked yet, though. You have to modify views or templates and fit your paths into the system, it doesn't help with changing images, if you use third-party code you have to modify that too, and the minification step makes it pretty hard to debug Javascript problems; even if you isolate it to live, problems happen on live too, sometimes.

Really, if we take a step back from the minify dream, the simple solution here is to append some kind of version number to the file. Then new HTML should have filenames with new version numbers so clients can't used their old cached files because the paths don't match.

I've found a really simple way to automate this transparently for a project by modifying one line in settings.py. The idea is to hook into Django with a middleware class, search the generated HTML for local URLs and replace them with URLs with version numbers. This way, we can catch all CSS and Javascript references as well as images no matter where they occur.

Since common file systems don't provide version numbers on the files they store, we'll use the modification timestamp instead. And instead of renaming files to contain the version number, we'll just append a ?_=timestamp to the URL:

  <img src="/media/images/logo.png">

becomes

  <img src="/media/images/logo.png?_=4321234">

Everything after the ? is ignored when looking up in the file system by the static file web servers I've tested so far. So it just works.

As a bonus, when this is running, you can safely modify the web server serving the static files to set an expiration date far out in the future for files requested with ?_=xxxxx. This allows an aggressive, yet perfectly safe caching scheme to be employed by any upstream cache, including the browser.

The above trick can be done in about 50 lines of Python. You can grab the little self-contained middleware module modtimeurls.py from my brand-new Garden of Python. It has been in production on several sites for over a year. The only problem I've found so far was with a PNG transparency fixer for IE6 that assumed .png files would end with .png rather than .png?_=xxxxxx.

Stoicism

This is philosophy in practice. One of the most interesting posts I have read this year.