Some ways to get probabilities wrong

An interesting set of paradoxes in probability.

Some of them are somewhat mind-boggling.

It's too bad school doesn't generally teaches us enough to get the intuition right.

Lys' vuggevise/lullaby

Despite not having blogged much for the past years, or rather, not writing down many blog posts, because I do in fact have had a lot of blogs composed inside my head, I sometimes wish I had a Danish blog for stuff that only makes sense in Danish or for Danes.

This is one of them. For reference, here's someone singing an original version, with gestures (apparently the text is by Oskar Schlichtkrull and music by Finn Høfding). Obviously, when trying to get your baby daughter to sleep, you don't do any gestures.

Lys' vuggevise v. 3
Gentle, deep voice

Jeg har en flyvemaskine,
den har vinger på.
Du kan tro at de er fine,
for de er malet blå.

Og når jeg flyver om natten,
så tager jeg lygter på.
Der er ingen der kan ta' dem,
for hvem kan himlen nå?

Min mor, hun sidder og strikker,
og hun har lovet mig:
Den dag der ik' er flere masker,
flyver hun og jeg.

Min far, han ligger og læser.
Men han har sagt til mig:
En dag hvor vinden rigtig blæser,
flyver han og jeg.

Min kat den sidder og spinder,
for jeg har lovet den:
En dag hvor solen rigtig skinner,
flyver vi afsted.

Jeg flyver hen over havet,
jeg flyver over land.
Propellen har jeg selv lavet,
den kør' så hurtigt som den kan.

Og mens min motor den brummer,
så kigger jeg op og ned,
for at se om der er andre folk der flyver,
så'en en dag hvor solen bare bli'r ved...

Here's a rough translation:

I have a flying machine,
it has wings on.
You can believe they're so fine,
because they're colored orange.

And when I fly at night,
I put the lights on.
There is noone who can take them,
because who can reach the sky?

My mum is sitting and knitting,
and she has said to me:
The day she's out of stitches,
she and I will fly.

My dad is on the couch and reading.
But he has promised me:
A day where the wind is really blowing,
he and I will fly.

My cat is in the window purring,
because I've promised it.
A day the sun is really shining,
we will fly away.

I'm flying over the ocean,
I'm flying over land.
The propeller is one of my invention,
turning as fast as it can.

And while my engine is humming,
I'm looking up and down,
to see if other people are out flying,
such a day where the sun is high...

Disappointment...

That's the feeling you get a late night when you hit M-/ in Emacs (hippie-expand), and Emacs doesn't use the web page you are looking at in Firefox as a source to auto-complete the word you've halfway written.

I think I need to retrain myself to use the browser built into Emacs.

Some people use an IDE that semantically analyses their code and pops up a suggestion box at every opportunity with an exhaustive list of valid choices.

I program with structural templates/duck types in Python and Emacs lets me complete all sorts of stuff, in comments, from other comments, or from the documentation I'm editing. It'll sometimes suggest some weird stuff, including whole line completions. It doesn't care. I don't care. I guess we're hippies.

Hippie van by ma noelle

It won't complete stuff we haven't shared between each other recently, though. I suppose the Lisp engine would rather get into it than wade through a lot of boring code analysis.

Intuition and design

So I was writing about intution, and somehow got sidetracked into how not to do ranking. So like a review of a game where the reviewer would judge the game on 4-5 different aspects, game play, sound, story, graphics etc. and then compute a final score based on fixed weights. How you'd fit a smash hit like Minecraft into that, I don't know.

What I really wanted to talk about was design.

Software development is full of design decisions. How do we present the information to the user? How do we let them act on it? How much to show, how much to hide?

How do we structure the code internally? What data do we store, how do we model it, how much do we capture? What other software do we build on?

What interface do we provide to other software developers?

It's a sea of decisions. Although it can be helpful for analyzing some outcomes, you can never hope to get through this sea with logic alone. You can often make reasonable deductions that something is simultaneously a good and a bad idea.

This makes it hard to discuss and evaluate these decisions. Often, we do not have the words, and even if we had, the actual components involved may be interrelated in such a complex manner that you'd never get anything out of studying them directly. You'd have to condense this complexity into something simpler, a process which may take much longer than the design process itself. And in simplified form, you can't be sure deductions are still valid.

So we're left with the intuitions we train for ourselves.

In other words, a prime concern of a software developer should be training this intuition effectively, i.e. input good training data with valid, confirmed outcomes - this works in this situation, this doesn't.

The main road to wisdom in programming, as in other crafts, I believe, is learning by doing. So experimenting or playing around if you wish, and seeing what happens. Not just in a strictly scientific sense where the experiment is artificial and controlled. It may be as simple as recognizing a common problem, working out a possible solution and implementing it and watching it unfold.

Sometimes you can take a shortcut by learning from others. There are various ways to go about this. Reading literature, or source code, or asking questions, or getting them to comment on your work. It's mostly about basic curiosity. Typical developer chat when someone is talking about something new they've done circle around how they made it work and what problems they encountered.

But the annoying thing about learning from others is, again, that it can be hard to talk about these things. Design decisions change when the details change. Sometimes minutia can decide whether one model is better than the other. How do you communicate this effectively? The person you are talking doesn't understand your problem completely. You don't understand the situation the other person was in either. So how do you know that the advice is correct?

I think this is here analytical skills are important. You always have to be sceptical, train your intuition into recognizing discrepancies and evaluate what context causes something to be judged to be good or bad.

For instance, if someone you know to have developed a sound intuition about a certain area find an idea of yours to be cause of alarm, you'd be a fool to ignore it. But unless you possess the same intuition about the problem, it can be hard to deduce exactly what causes the alarm and whether it's a problem in the context.

Frustratingly, it can also be hard to explain the root cause on the spot. Often, there is no explanation in the intuition. And examples can be hard to think up. I've sometimes been asked for an opinion, found cause for alarm but haven't been able to figure out why until several years later. I know that because those alarms haunt me for years, making me feel like a grumpy old man seeing ghosts everywhere.

One example I have is that of software you depend upon, as I touched upon in my previous entry. Over the years, I've come to the conclusion that some dependencies serve you exceedingly well, and others serve you hell.

Probably the hardest part of design is designing for the future. We never know what happens. Still, careful design mitigates the risks.

The failure of logic

Some time ago, I read a book about experiments on brains, You Are Not So Smart. As it turns out, many of the ideas humans commonly have about their own thinking capabilities are pretty far off the mark because our brains sort of works with illusions.

For instance, human eyes only have a small field in the center where the resolution is good, yet when we open our eyes, a wide field of vision appears. Because the images our brains see is something it has composed itself.

One of the recurring themes was logic versus intuition.

The point was that people mostly think with intuition.

Intuition is powerful. It lets you reason from the cumulative experience you've had. It works effortlessly and often only takes moments to do its job.

Logic is extremely limited. Like a simple computer processor, you can only work with a limited set of concepts at the time and make simple deductions.

People seem to think they make rational, logical decisions. But when you actually test them in experiments where intuition and logic disagree and logic has the upper hand, then most people don't wait and think through the deductions. So they arrive at the wrong conclusions and aren't actually as smart as they think they are.

But it has occurred to me that in some instances, you can see the opposite problem. People trying to use logic to solve problems that require more than simple deductions.

For instance, how do you use logic to decide what you're going to do in life? What education, where to work.

In the software world, an eternal question is what software to bet your future on. It affects end-users as well as developers. In the office next to IOLA, there once was a company that was betting on a particular technology offered by Microsoft. It was even embedded in their name. From the outside, they appeared prosperous. Then one day Microsoft decided to shut down their framework.

How do you decide which programming language framework will serve you well for many years?

I think you can use logic to elucidate some points. Likewise, you can try things out to gain more knowledge. But overall it is a huge trade-off with many factors.

Some people realize that and try to split the hard decision up. So you might look at it from different angles and make up a matrix comparing all the choices.

Helpful as it may be in understanding the problem, it can lead down a path where people spend their time arguing about unimportant matrix details. And someone may get the brilliant idea that all that's needed is a scheme for assigning scores to the various parts and then combining them with weights into a final rank.

This is, I think, the ultimative failure of the faith in logic - thinking that simple-minded strategies will beat the raw power of a trained human brain. How do you assign the weights so that the end result is meaningful? Nobody knows.

If you need a complex ranking, feed the details to a trained intuition, one that has seen what works and what doesn't, and let it decide.

When four becomes five

We're now a family of five. Since last week.

Now, there's a history of quick births. Last time we arrived at the hospital in an ambulance.

This time Janne woke me up at around 2:40 in the night. At that point, there was plenty of time. So we called the grand parents.

By the time they got here, maybe half an hour later, Janne wasn't sure there was still plenty of time, so we hurried of, arrived at hospital safely thanks to my father, and half an hour later, it was all over, Janne had given birth to a little girl.

They wanted to keep Janne around for while because she'd lost a bit more blood than they liked, but changed their mind later the same day after having tested her hemoglobin level. By five o'clock in the afternoon, we were all together again.

So for the past week and a half, we've been taken care of a little baby, once again. Rewarding in some ways, demanding in other ways. She's got a nice loud scream for when she's not happy.

Stimergy

I've always been attracted to the idea of self-organization by informal means, and there was an article not long ago talking about stimergy in a software development context.

The idea is that everyone somehow knows what to do without talking to each other, but just by doing the work and observing the signs left behind, implicit and explicit.

For instance, how do we add a new page to the app? By copying an existing. How do we know something is wrong? Well, either someone already removed the problematic part so we don't see it anymore, or someone added a FIXME note with the intention of doing it later. Or perhaps it just sticks out so visibly that nobody is in doubt it's bad.

Arrows

I think of it as a room full of arrows lying on the floor.

If you are bad at software development, the arrows will be pointing in random directions, and confusion ensues at every point in the process. You can never sprint, only stumble.

The goal is to make sure everything is tidied up so the arrows point in the same direction, the direction we want to go in. We need to constantly check and revise the code to align the arrows, not only align them to each other but also align them with the evershifting goals of the system. The goal of training newcomers is to teach them to read the arrows and make new arrows point in the right direction.

For instance, if a function is named after how it was conceived, not after how it is currently being used, it's pointing in the wrong direction. When you later try to reason with it, your mind will tend to go down the wrong paths and you'll reach the wrong conclusions, leading to bugs, or at least waste time having to retrace your steps.

You can add a comment to the function, but it's far more powerful to fix the name so it's pointing in the right direction, even if it's more work up front. Arrows matter.

I think this is a powerful metaphor.

Simple processes

I also think it's a powerful method.

Everything is done by engaging with the main activity, programming. It's through this activity that we learn and exchange the acquired knowledge.

Anyone can contribute to the direction. The value of the contribution is correlated with the confusion the contribution leaves in the greater system. It can be negative, but it can also be a stroke of genius, the subtle touch that makes the arrows of the system appear in greater clarity and alignment.

It is dynamic. If a bad decision is made, it's precisely as fixed as the program. The effort it takes to fix the program is the effort it takes to change the decision, no more and no less. Once something enters the program, it takes a life of its own, removed from the egos of programmers involved. The ego might point in one direction, but the code has its own signs.

Yet it's really simple. It works for ants.

Ant by Katja Schulz

Ants

Of course we're not ants. We need to and should communicate. We can pre-align our internal arrows to solve a complex problem much more efficiently than a bunch of ants that are mostly relying on evolved instincts.

But once the program starts, the communication becomes fragmented and loses some of its relevance compared to the work we actually do. You want to know what I did last week? Look at my commit log.