Tuesday, June 7, 2016

Intuition and design

So I was writing about intution, and somehow got sidetracked into how not to do ranking. So like a review of a game where the reviewer would judge the game on 4-5 different aspects, game play, sound, story, graphics etc. and then compute a final score based on fixed weights. How you'd fit a smash hit like Minecraft into that, I don't know.

What I really wanted to talk about was design.

Software development is full of design decisions. How do we present the information to the user? How do we let them act on it? How much to show, how much to hide?

How do we structure the code internally? What data do we store, how do we model it, how much do we capture? What other software do we build on?

What interface do we provide to other software developers?

It's a sea of decisions. Although it can be helpful for analyzing some outcomes, you can never hope to get through this sea with logic alone. You can often make reasonable deductions that something is simultaneously a good and a bad idea.

This makes it hard to discuss and evaluate these decisions. Often, we do not have the words, and even if we had, the actual components involved may be interrelated in such a complex manner that you'd never get anything out of studying them directly. You'd have to condense this complexity into something simpler, a process which may take much longer than the design process itself. And in simplified form, you can't be sure deductions are still valid.

So we're left with the intuitions we train for ourselves.

In other words, a prime concern of a software developer should be training this intuition effectively, i.e. input good training data with valid, confirmed outcomes - this works in this situation, this doesn't.

The main road to wisdom in programming, as in other crafts, I believe, is learning by doing. So experimenting or playing around if you wish, and seeing what happens. Not just in a strictly scientific sense where the experiment is artificial and controlled. It may be as simple as recognizing a common problem, working out a possible solution and implementing it and watching it unfold.

Sometimes you can take a shortcut by learning from others. There are various ways to go about this. Reading literature, or source code, or asking questions, or getting them to comment on your work. It's mostly about basic curiosity. Typical developer chat when someone is talking about something new they've done circle around how they made it work and what problems they encountered.

But the annoying thing about learning from others is, again, that it can be hard to talk about these things. Design decisions change when the details change. Sometimes minutia can decide whether one model is better than the other. How do you communicate this effectively? The person you are talking doesn't understand your problem completely. You don't understand the situation the other person was in either. So how do you know that the advice is correct?

I think this is here analytical skills are important. You always have to be sceptical, train your intuition into recognizing discrepancies and evaluate what context causes something to be judged to be good or bad.

For instance, if someone you know to have developed a sound intuition about a certain area find an idea of yours to be cause of alarm, you'd be a fool to ignore it. But unless you possess the same intuition about the problem, it can be hard to deduce exactly what causes the alarm and whether it's a problem in the context.

Frustratingly, it can also be hard to explain the root cause on the spot. Often, there is no explanation in the intuition. And examples can be hard to think up. I've sometimes been asked for an opinion, found cause for alarm but haven't been able to figure out why until several years later. I know that because those alarms haunt me for years, making me feel like a grumpy old man seeing ghosts everywhere.

One example I have is that of software you depend upon, as I touched upon in my previous entry. Over the years, I've come to the conclusion that some dependencies serve you exceedingly well, and others serve you hell.

Probably the hardest part of design is designing for the future. We never know what happens. Still, careful design mitigates the risks.

Monday, May 30, 2016

The failure of logic

Some time ago, I read a book about experiments on brains, You Are Not So Smart. As it turns out, many of the ideas humans commonly have about their own thinking capabilities are pretty far off the mark because our brains sort of works with illusions.

For instance, human eyes only have a small field in the center where the resolution is good, yet when we open our eyes, a wide field of vision appears. Because the images our brains see is something it has composed itself.

One of the recurring themes was logic versus intuition.

The point was that people mostly think with intuition.

Intuition is powerful. It lets you reason from the cumulative experience you've had. It works effortlessly and often only takes moments to do its job.

Logic is extremely limited. Like a simple computer processor, you can only work with a limited set of concepts at the time and make simple deductions.

People seem to think they make rational, logical decisions. But when you actually test them in experiments where intuition and logic disagree and logic has the upper hand, then most people don't wait and think through the deductions. So they arrive at the wrong conclusions and aren't actually as smart as they think they are.

But it has occurred to me that in some instances, you can see the opposite problem. People trying to use logic to solve problems that require more than simple deductions.

For instance, how do you use logic to decide what you're going to do in life? What education, where to work.

In the software world, an eternal question is what software to bet your future on. It affects end-users as well as developers. In the office next to IOLA, there once was a company that was betting on a particular technology offered by Microsoft. It was even embedded in their name. From the outside, they appeared prosperous. Then one day Microsoft decided to shut down their framework.

How do you decide which programming language framework will serve you well for many years?

I think you can use logic to elucidate some points. Likewise, you can try things out to gain more knowledge. But overall it is a huge trade-off with many factors.

Some people realize that and try to split the hard decision up. So you might look at it from different angles and make up a matrix comparing all the choices.

Helpful as it may be in understanding the problem, it can lead down a path where people spend their time arguing about unimportant matrix details. And someone may get the brilliant idea that all that's needed is a scheme for assigning scores to the various parts and then combining them with weights into a final rank.

This is, I think, the ultimative failure of the faith in logic - thinking that simple-minded strategies will beat the raw power of a trained human brain. How do you assign the weights so that the end result is meaningful? Nobody knows.

If you need a complex ranking, feed the details to a trained intuition, one that has seen what works and what doesn't, and let it decide.

Friday, February 12, 2016

When four becomes five

We're now a family of five. Since last week.

Now, there's a history of quick births. Last time we arrived at the hospital in an ambulance.

This time Janne woke me up at around 2:40 in the night. At that point, there was plenty of time. So we called the grand parents.

By the time they got here, maybe half an hour later, Janne wasn't sure there was still plenty of time, so we hurried of, arrived at hospital safely thanks to my father, and half an hour later, it was all over, Janne had given birth to a little girl.

They wanted to keep Janne around for while because she'd lost a bit more blood than they liked, but changed their mind later the same day after having tested her hemoglobin level. By five o'clock in the afternoon, we were all together again.

So for the past week and a half, we've been taken care of a little baby, once again. Rewarding in some ways, demanding in other ways. She's got a nice loud scream for when she's not happy.

Friday, January 15, 2016

Stimergy

I've always been attracted to the idea of self-organization by informal means, and there was an article not long ago talking about stimergy in a software development context.

The idea is that everyone somehow knows what to do without talking to each other, but just by doing the work and observing the signs left behind, implicit and explicit.

For instance, how do we add a new page to the app? By copying an existing. How do we know something is wrong? Well, either someone already removed the problematic part so we don't see it anymore, or someone added a FIXME note with the intention of doing it later. Or perhaps it just sticks out so visibly that nobody is in doubt it's bad.

Arrows


I think of it as a room full of arrows lying on the floor.

If you are bad at software development, the arrows will be pointing in random directions, and confusion ensues at every point in the process. You can never sprint, only stumble.

The goal is to make sure everything is tidied up so the arrows point in the same direction, the direction we want to go in. We need to constantly check and revise the code to align the arrows, not only align them to each other but also align them with the evershifting goals of the system. The goal of training newcomers is to teach them to read the arrows and make new arrows point in the right direction.

For instance, if a function is named after how it was conceived, not after how it is currently being used, it's pointing in the wrong direction. When you later try to reason with it, your mind will tend to go down the wrong paths and you'll reach the wrong conclusions, leading to bugs, or at least waste time having to retrace your steps.

You can add a comment to the function, but it's far more powerful to fix the name so it's pointing in the right direction, even if it's more work up front. Arrows matter.

I think this is a powerful metaphor.

Simple processes


I also think it's a powerful method.

Everything is done by engaging with the main activity, programming. It's through this activity that we learn and exchange the acquired knowledge.

Anyone can contribute to the direction. The value of the contribution is correlated with the confusion the contribution leaves in the greater system. It can be negative, but it can also be a stroke of genius, the subtle touch that makes the arrows of the system appear in greater clarity and alignment.

It is dynamic. If a bad decision is made, it's precisely as fixed as the program. The effort it takes to fix the program is the effort it takes to change the decision, no more and no less. Once something enters the program, it takes a life of its own, removed from the egos of programmers involved. The ego might point in one direction, but the code has its own signs.

Yet it's really simple. It works for ants.

Ant by Katja Schulz

Ants


Of course we're not ants. We need to and should communicate. We can pre-align our internal arrows to solve a complex problem much more efficiently than a bunch of ants that are mostly relying on evolved instincts.

But once the program starts, the communication becomes fragmented and loses some of its relevance compared to the work we actually do. You want to know what I did last week? Look at my commit log.

Thursday, January 14, 2016

Wednesday, January 13, 2016

Emacs: ido is currently the replacement for iswitchb, not icomplete-mode

So Emacs recently, well, recently in Emacs time, obsoleted iswitchb, the package I've been using to switch buffers.

The suggestion is to try icomplete-mode instead. But icomplete-mode isn't really tuned for buffer switching in the same manner as iswitchb. After some digging, it looks like there was a plan to gradually integrate the features of iswitchb/ido into icomplete-mode, but that hasn't happened yet, and yet iswitchb was obsoleted.

Fortunately, it looks like ido these days allows one to only turn it on for buffers. This will do the trick as a replacement for (iswitchb-mode 1):

(ido-mode 'buffers)

Once that's done, you can even enable icomplete-mode safely without having it meddle with buffer switching, so:

(ido-mode 'buffers)
(icomplete-mode 1)

Wednesday, January 6, 2016

Last-minute bugs

You've just completed a new big feature.

There's been ups and downs - you started thinking how hard can it be, then discovered the inherent ambiguities, the dissonances with the existing code base, got stuck, pacing the floor of the office, chatting to coworkers, slowly building, discarding and rebuilding refined models, getting stuck and unstuck again and again, then gradually transcending into a coding frenzy, with the code bending towards your goals and your will, constantly in the zone, writing tests, wrapping things up, thoroughly testing everything manually, finally letting some trusted users try it out.

And now you're ready for release. Release!

Except just now someone remarked how something is a little bit odd. Perhaps it's even yourself, the overworked brain having mentally let go of the whole thing and suddenly remembering a little forgotten detail.

Ah. But I just need to fix this. It is simple. The world is my feet. And the test works. I can do this.

Except you didn't bother running the full test suite. Or going through the full test procedure. Or full release procedure. Or checking with the users.

This is the time of the last-minute bug where the little innocent fix turns out to violate an assumption somewhere else.

My piano teacher Karen-Marie once explained to me that finales can be the most dangerous parts for a musician. When you stop thinking ahead, instead thinking about what you've accomplished, letting concentration slip.