Landmark

Today I have lived for a quarter of a century. I was born in 1982. According to Wikipedia, it was the year the first computer virus escaped into the wild.

Django-rendertext 0.2 and a new jquery plot plugin

A quick update. I've released django-rendertext 0.2. The most notable change is support for GIF output, mostly intended for the browser that didn't support PNGs properly until very recently.

I deliberately built rendertext with PNGs first because it's the format of the future. IE7 usage is picking up (see for instance here or here), within a year or two it's probably completely dominant among IE users. Now that I whipped up the GIF support, I'm glad I didn't start with it because it was a real PITA to do with PIL (Python Imaging Library). PIL doesn't have proper documentation of the corner cases that turn out to be the most important and only sketchy support for palettes.

I've also been working on a plotting plugin for jQuery. We've not been entirely satisfied with Plotr which otherwise appeared to be the prettiest and most advanced Javascript-only plotting library out there. It's for Prototype and not for jQuery but most importantly I've found it confusing to hack on. And it's completely missing the interactive aspects.

So I have started my own project. So far I think I've discovered that the hardest part of a library for plotting is not actually drawing the stuff, but layouting and auto-adjusting the axes. In any case, in two days I've gotten pretty far. Here's a screenshot (click to get the full version):



What's missing right now is support for plotting points and columns, options for controlling the plot, legends, support for time and the interactive stuff such as zooming, highlighting graphs, tooltips for points on mouse hover. Also, the auto-adjusting algorithm is still a bit rough but I think I managed to crack it on paper last night so I just need to implement and test it.

In other news, from Werner Vogels' blog here's how Amazon manages storage (warning! scientific paper ahead). Now that's as far as you can get from N=1 thinking. As an aside, I'm fond of that guy's name, Werner Vogels (note for English native speakers: in German the surname is pronounced like [fogels] not [wogels]). It's just like der Vogelfänger bin ich ja I guess, nice rhythm.

Object-oriented database systems

I had a discussion with Anders today about databases.

I'm not very fond of relational databases, or perhaps more precisely SQL. Having to convert back and forth between a database model and SQL and a programming language with all it entails of proper quoting to prevent SQL injection attacks and joining tables all over the place is really a great source of frustration. And a good waste of time.

On the other hand, I've seen what a record-based file system with no built-in join can do to people. And the built-in indexing stuff that lets you search on different attributes without a linear scan is just neat.

I think the great success of the relational database systems has a few simple explanations. Many business-related applications are really about the data. They spend their time collecting and displaying lots of simple, boring data - unlike an entertainment program or a standard desktop application that spends a lot of time dealing with small amounts of data, maybe less than say 1000 objects.

This data may be larger than what can be kept in memory (the possibility of this happening is enough to make it a problem). Furthermore the data must be kept when the machine is turned off. So you have to deal with secondary storage, a topic largely ignored by most programming languages that otherwise go to great lengths to make dynamic use of primary storage as transparent as possible.

Finally, you need to be able to access all that boring data in a many different ways. Say you have a situation that you naturally think of as a hierarchy, e.g. a company with multiple departments with each department having a number of employees and each employee having far to many tasks to do.

In object-oriented thinking, you would model this as a list of departments, each with some attributes and a list of employees, each again with a list of tasks. Want to show the org chart in the company? Easy, just loop through the departments and print the list of employees one by one. Super efficient. Assign a task to an employee? Easy, just add it to his list. Super efficient. But what if you want to know who's task it is to empty the bin? Then you have to wade through two outer loops plus lots of unrelated stuff, just to get to the actual data you're interested in.

Relational database systems, while solving the secondary storage problem, have an answer to this problem, and it's simple and elegant.

Then never mind that getting data in and out of them is hugely painful, especially hierarchical data, even if you're not trying to force everything leaving the database into a class.

I've recently been working a lot with Django's object-relational mapper, the ORM (which BTW means "worm" in Danish), and it really takes the pain out of the situation since it obviates the need for writing SQL strings in favour of native, albeit somewhat weird, Python code.

However, I still wonder why true object-oriented databases haven't overtaken the world ages ago. One that I hand over my object to and later can get it back from, in one piece with no fuss.

It turns out that there's a pretty large open-source one available, db4o. I think it solves the problem of retrieving objects based on arbitrary attribute values by incorporating a special query system where it's possible to define indexes like for a relational database, but I'll have to do some more reading.

There's a smaller one, DyBASE, for Python (and Ruby and PHP). It could be interesting to compare the performance against Django's ORM + MySQL.

The interesting question is of course whether you need a relational database if you can have the simpler object-oriented one. Interoperability issues aside, some people apparently seem to believe that the relational model is crucial for performance with huge amounts of data.

It would seem to me, though, that the natural partitions which the object-oriented paradigm splits the data into would tend to make things faster since there's less data to search through. For instance, if you're only concerning yourself with data from one department in the company, then it's rather wasteful to work with the maybe ten times larger table containing data for all departments. Perhaps it has something to do with joins and restrictions being difficult to express simultaneously in one query with the object-oriented approach. There are some benchmarks on www.polepos.org although they are not exactly easy to interpret at a glance.

Other people, like this guy on Slashdot, seem to think that the abstract relational model makes for better databases as opposed to application-specific persistent storages.

This makes me wonder whether many, perhaps most, uses of a relational database today wouldn't be better served by a simple object-oriented application-specific data store. I think most run-of-the-mill web sites need easy object persistence far more than terabyte scalability or an enterprise-wide database. Leave the heavy artillery for when it's really required.

Then, on the other hand, if you can get easy object persistence with Django's ORM, maybe this is a moot point.

Django-rendertext 0.1 is out

We're releasing the first version of rendertext, a Django app for making use of custom fonts on a web page!

It's designed to work with the built-in Django template system by providing a simple template filter you can use to render images dynamically.

The first showcase is Talent Nord. I hope to get a longer list as more people start using it. The design of Talent Nord is by Fata Morgana, house of two intelligent designers.

I think you need a pretty recent Django for the app to work. Here's the code that does the hard work:

import md5, os
import Image, ImageFont, ImageDraw, ImageColor
from django.conf import settings

def render(text, fontalias, size = 12, color = "#000", rotation = 0):
    """Construct image from text.

    Returns a tuple with the image file path, width and height. Takes
    the parameters 'fontalias', 'size' (default 12), 'color' (default
    black) as CSS hex string, and 'rotation' (default 0) which is the
    degrees to rotate the text counter-clockwise. The constructed
    image is stored in a cache under the media root in the
    RENDERTEXT_DIR set in settings.py (default is 'rendertext/').

    The font file to use is determined by mapping the fontalias
    through the RENDERTEXT_FONTMAP setting, a dictionary from
    fontalias strings to font file paths, e.g. {'verdana':
    '/home/django/fonts/verdana.ttf'}."""

    # get settings
    render_dir = "rendertext/"
    if hasattr(settings, "RENDERTEXT_DIR"):
        render_dir = settings.RENDERTEXT_DIR
    
    fontmap = settings.RENDERTEXT_FONTMAP

    fontfile = fontmap[fontalias]
    info = "|".join([text, fontalias, str(size), str(color), str(rotation)])
    name = md5.new(info.encode('utf-8')).hexdigest()
    filepath = render_dir + name + ".png"
    
    dim = (-1, -1)
    if not os.access(settings.MEDIA_ROOT + filepath, os.F_OK):
        # construct the image
        imf = ImageFont.truetype(fontfile, size)
        dim = imf.getsize(text)
        im = Image.new("RGBA", dim, ImageColor.getrgb(color) + (0,))

        draw = ImageDraw.Draw(im)
        weird_font_renderer_fix = " "
        draw.text((0, 0), text + weird_font_renderer_fix, font=imf, fill=color)

        if rotation != 0:
            im = im.rotate(rotation, Image.BICUBIC, True)
            dim = im.size

        if not os.access(settings.MEDIA_ROOT + render_dir, os.F_OK):
            os.makedirs(settings.MEDIA_ROOT + render_dir)
        
        im.save(settings.MEDIA_ROOT + filepath, "PNG")
    else:
        # read width and height
        im = Image.open(settings.MEDIA_ROOT + filepath)
        dim = im.size

    return (settings.MEDIA_URL + filepath, dim[0], dim[1])
    

def render_tag(text, *args, **kwargs):
    """Construct image tag from text."""
    (path, width, height) = render(text, *args, **kwargs)
    if not path:
        return ""
    else:
        dim = ""
        if width > 0 and height > 0:
            dim = 'width="%s" height="%s" ' % (width, height)
            
        return '<img src="%s" alt="%s" %s/>' % (path, text, dim)

Groups and social software

From a link from Joel on Software is this essay on groups and social software. Lots of interesting points, e.g. that the group needs rules to prevent it from ruining itself, that social and technological issues are inseparable and that a core of members arise which should be nurtured.

No posts for long time, I have been on vacation, away from computers. I've gotten through a good deal of books, all fictional. Not having read much fiction for the last year or so, I find it surprising that a book can still keep me occupied in bed in the morning so that breakfast turns into lunch and I spend most of the day on the couch reading.

I've also been thinking. Mostly about life and what I like to spend my time on. Nothing major came out of it, though. :-)

When you're smiling...

Greetings! They come in various flavours, depending on how well you know the person you're greeting and the time since the last greeting. I came across a funny one today.

In descending order of warmness, there's the hug, the informal "hi!" or "hey!", the slight nod of the head, and then there's the funny smile. It's not a real smile. The real smile is reserved for people you can't say hi to because they're on the phone or for people you're flirting with.

The funny smile is the kind of smile where you consciously try to move the corners of your mouth a little bit upwards while keeping the lips pressed together. The meaning is something like, "I see and acknowledge you but a real greeting is not warranted".

The funny thing about it is the way it looks. I haven't thought about it before, but someone sent me such a funny smile while I was on my bicycle to work. And you surely appear to be sucking your lips into your mouth, like the archetypal toothless old witch.

Think about that next time you funny-smile someone.

Software design, part I

What makes a good software design?

I have been thinking about this for some time. The more I think about it, the simpler I come to think it is. Forget all the -tion, -ability, -ance words that are usually used to describe properties of good designs. In my mind a good design is one that is easy to understand.

As developers we think that ease of use/understanding is something that first and foremost applies to our users. We are used to horribly complicated IDEs, command lines that requires memorizing half a dictionary of cryptic commands, debuggers that speak assembler, etc. I argue that as a program grows from 50-100 lines of code to several thousands lines, the only thing that really matters is that you're not lost when you read and navigate it. And not whether you could squeeze a couple of lines out of it by some obscure technique you recently read about in a blog post.

Why? Because you need to understand the code in order to extend it, to fix bugs, to avoid introducing errors, to be able to estimate effects of changes, and in a sort of self-referential way, to still keep the design ease to understand.

In my mind, there are three goals with programming. In order of importance: to get the thing running the way it is intended to as fast as possible, to have as few bugs as possible, and to have as much fun as you can in the process. These are complementary - the faster you proceed, the more fun it usually is, at least if you are attracted to the creative dimension of programming. And the fewer bugs you have, the faster you can proceed. A non-existing defect that you don't have to try to reproduce, debug and then release a fix for is a real time-saver. Likewise, the more fun it is, the better you can concentrate and the faster you will go along.

Easy-to-understand code generally supports all three of these goals. If the code is difficult to understand, it will slow down the process immensely. And it will increase the risk of subtle bugs of the kind where the logic is faulty in corner cases. Even a clever compiler can't help with you out of those. Finally, it is simply more fun to work with code you understand.

How do you make a design easy to understand? I believe at the code level itself it is very much about using clear language and avoiding cryptic, non-descriptive names. Especially over time. If an identifier changes it meaning, even in a subtle way, so that its name is no longer descriptive, it's important to rename even if you have to change code all over the place.

At the design level it's on one hand about keeping things together that belong together and on the other hand about splitting up the code into smaller chunks so that the reader is not overwhelmed. Two opposing forces.

It's not that complicated really. To be able to understand something, it helps that the lines of code are sitting right next to each other. If you have to look them up in different places, you've lost your overview - and how do you know you found all the places unless the issue is easily greppable? But also, the piece of code needs to be small enough to fit in your brain, else you have to break it down yourself through analysis which is time-consuming and takes away the focus on what you really are trying to accomplish.

Keep things together that belong together. If you are designing a database application, don't put the logic that decides how the screen widgets should look like in the code that computes stuff from data received from the database. Likewise, don't put code that performs complicated computations in GUI code which is already busy with presenting an intuitive interface that prevents people from doing unnecessary damage to their data.

When you program, violations of this principle might be easy to see if you look at the dependencies. So the database code had to explicitly import a GUI module? Maybe it has more concerns than the database itself then. But sometimes violations are more difficult to spot. I think a main culprit here is the decision making.

Code is all about making decisions. In fact a computer program is at heart just a long series of decisions - first I'll do this, then I'll do that, then if this happens, I'll do this again. Most of these decisions are not important decisions, however, more like minor tweaks. It's not easy to see this in the code itself. One statement looks as good as the other, even though the first might decide what the user is going to look at and the other just adds a 2 pixel space around the border.

But what you usually want to do is to keep decisions about the same kind of things together. Then it's easy to understand. What fundamental things does the user see on screen? The answer should be along the lines of, take a look at this piece of code and you can see all of it. Not, well, grep for XYZ in these 40 source files. That's why lowly GUI code which is often preoccupied with tweaking spacing and interactive behaviour shouldn't be allowed to do much on its own, but immediately transfer control to somewhere else when anything less than trivial is to be done.

Since the programming language does not directly help with arranging the decision making in a sensible way, you should explicitly take it into account in the design, e.g. by forbidding certain parts of the code to do certain things or by designing modules that are explicitly made responsible for nothing more than high-level decision making. Or by arranging a calling structure that makes it easier to delegate decisions to upper management, e.g. by means of signals/callbacks/hooks/events/delegates.

Sometimes it means more hassle to follow this principle. But no pain, no gain. In most situations the long-term consequences of understandability are much more important than whether it takes an extra minute to code.

I have much more to say about this, but for now I'll conclude with this remark: When you start working on a new aspect of program, one that you haven't worked on before in another project so that you have a design already, it's easy to become irresolute, to get stuck on how to proceed in the best possible way. But if you simply design to keep things together that belong together while splitting the code into chunks that are small enough to be understood, it's probably going to be a healthy design and the details are less important. So just go ahead and see what happens.