Tuesday, 15 May 2018

The Role of Abstractions in Software Engineering

An abstract representation of a concrete apple
This is the text of a lightning talk, a five minute presentation, given at PyCon 2018.

This is an abstract talk. There isn’t time to give examples but I hope that the application to the day to day challenges of the practise of software engineering is clear. The only theory worth a damn is the theory of the practise. This is a talk about the role of abstractions in software engineering.

Programming is all about the use of abstractions. We often say that the fundamental language spoken by the machine is ones and zeros. Binary. This isn’t true. Ones and zeroes are an abstract representation of the fundamental operation of computers. It’s a way of representing what central processors do in a way that can be understood by people.

The actual language spoken by computers is the electromagnetic dance across wires and etched silicon, choreographed by the beating of a quartz crystal at the heart of the machine.
Ones and zeroes are a representation of that dance, understandable by humans in order for us to reason about the behaviour of the system.

That’s a very low level abstraction. Very close to the actual operation of computers, but very hard to work with. The next step up is assembly language where we use mnemonics, symbolic instructions like JMP for jump, to represent these patterns of ones and zeroes. We can also use human recognisable labels for memory locations instead of numbers and allow the assembler to calculate offsets for us. Much easier.

Next we have languages like C and then right at the very top we have Python where each construct, a print statement for example, may correspond to as many as millions of the lowest level operations.
Computer programming is communication in two directions. Programming provides a language the computer understands, and is able to execute deterministically, whilst also communicating with humans so they can conceptualise the behaviour of the system. A programming language is a set of conceptual tools to facilitate that communication in both directions.

The art and craft of software engineering is taking the conceptual tools that programming languages provide and using them to solve real world problems. This is the difference between science and engineering. Science is the theory, engineering is the application.

In order to be able to do this we have to have an understanding of the problem domain. We conceptualise it. We think about it. Software is easy to understand and maintain when the abstractions you build map well to the problem domain. If the way you think about the problem is close to the way you think about your software then you have to do less mental translation between the problem and your code.

Joel Spolsky talks about the law of leaky abstractions. Any abstraction that maps to lower level operations in the system will leak. At some point something will go wrong and you will only be able to fix it by understanding the lower level operations too.

I’ve heard it said, and it rings true, that a good programmer can hold about ten thousand lines of code in their head. So if your system is less than ten thousand lines of code, even if it’s terrible code, you don’t need to build higher level building blocks to hold it in your head.

An all too common situation is that a system becomes too complex to reason about, so an engineer decides to create abstractions to simplify how they think. So they create black boxes, abstractions, in which to place the complexity. These type of abstractions conceal complexity. So now you don’t have to look at the mess you just made.

You can reason about your system with your abstractions, but in order to understand the actual behaviour (at a lower level) you need to go digging in all that dirt.

Instead of concealing complexity a good abstraction will explain and point you to the lower level operations. Good abstractions simplify and reveal complexity rather than concealing it.
We can also use this kind of reasoning to think about product and system design. What user experience are you providing, what’s the user story? Your users also think about the problem domain using conceptual tools. The closer the abstractions your software presents to your user map to the way they already think about the problem the easier your software will be to use.

And here we come full circle. If the way you build your software maps well to the problem domain then it will be easy to reason about and maintain. If the abstractions you present to the user map well to the problem domain then it will be easier for your users to think within your system and it will be more intuitive to use.

So abstractions matter. They’re the raw stuff of our world.

You can watch the presentation, from the Saturday afternoon lightning talks, four minutes into this video.

Friday, 4 May 2018

Prunes and Funerals

Does your life tell a story? Tell me your life story.

I was enjoying a sweet treat of prunes and dried apricots with Delia last night and she remarked that prunes reminded her of funerals. She still likes them though.

Delia grew up in a medium sized town in Romania called Roman, in the North East of Romania in the poorest region of the country called Moldova and bordering the country of Moldova where they also speak Romanian. 

Until Delia's mother fell into a bleak depression lasting several years both of Delia's parents worked. So Delia was a latch-key  kid even from her primary school years. Delia's father worked in the local chocolate factory under the communists. Money was tight for everyone so much of the economy ran on a barter scheme. A visit to the doctor or hospital was free, except if you actually wanted anything doing (like your sheets changing or you wanted feeding during a hospital stay) bribes were required. This minor corruption became so endemic that it's still prevalent. We gave our first ever bribe on a train ride from the nearest city back to Roman. Working at the chocolate factory provided a regular stream of bags of flour and cocoa and pats of butter that not only provided enjoyment for family and friends but were a useful source of bribes. That was until Delia's father lost an eye in a chocolate factory accident. No the eye didn't end up in someone's chocolate, and he was pensioned off on a reasonable pension. But that's another story.

Delia's mum was so determined that Delia would have an education that Delia grew up without toys but books instead. She was the first in her family to go to university and completed a degree in Romanian and English Literature at the prestigious university of Iasi. She lived in student accommodation and her parents would regularly visit to provide food and a small amount of money to live off. 

As a child during  the three month summer holiday Delia and her young aunts and cousins would run wild in the countryside staying with her grandmother. They would roam the woods and fields, staying out late until Bunica called them in to sleep top to tail all in the one enormous bed with grandmother in a two room cottage. Apparently Delia's father slept in the same bed as his sister until she left to get married. A different culture, a different way of life.

Delia's grandmother lived the traditional way. Subsistence farming, outdoor toilet, getting up at dawn. She dried plums for the winter and stored the prunes in her loft. They would be brought out as a special treat and served on a stick instead of sweets at events like funerals.

So prunes remind Delia of funerals. And running free in the countryside as a child, afraid of nothing and no-one except her mother. And she still loves them.

"Compassion is the deepest pleasure. Tune the pleasure principle into this and your own animal drive will compel you into goodness."

Monday, 12 February 2018

A Very Short Love Letter to Agile

We rejoice in the ambiguity, for in it lies redemption
I love the word rigour. It conveys either, or both, strict discipline or something that was really hard work.

I've found the rigorous application of theoretical principles a really useful way of learning those principles. Learning what they really mean, and what those principles are good at achieving and what they're not good at achieving.

I've been rigorous in my discipline in meditating. I've meditated for an hour a day, generally six days a week, for a number of years now.

My trade is as a software engineer, a computer programmer. I taught myself to program by becoming really passionate about it. What you love you learn. I learned the art and craft of engineering in my first professional job, at a small startup in London called Resolver Systems.

There, for the four years I worked there, we rigorously applied the principles of Extreme Programming, a strict variant and really the progenitor of the "agile" movement. The goal is engineering processes that make the development process agile and fluid, able to change direction quickly and able to check that it is continuously delivering value in the work being done, whilst also creating the software in ways that ensure as much as is possible you are creating a quality and useful product.

This includes full Test Driven Design (often now called "test first" in a great misunderstanding of the value of TDD), with a full test coverage from unit to functional (full stack). We had about three to four times more test code than production code. We built a beautiful thing.

It also included full pair programming. So for four years we worked together and thought together and learned together. The product failed, unfortunately, an idea ahead of its time. With Python finally being added to Excel as a scripting language it's possible that the idea of applying proper engineering principles to the creation of complex spreadsheets may have its day after all.

"You have to see the abstraction and see through the abstraction. A lot of how we understand modern life is through the use of abstractions. Money being the classic example of course."

Friday, 9 February 2018

IAO I Adore Thee: The Names of God

Something of an experiment, a bit of spoken word. Here I read a favourite of my poems, mercifully short, and explain what it means to me. It's on one of the names of God.

IAO I adore thee, magickal thou art.

"Seeing this world as evil and doomed is the substantial part of the gnostic heresy that Paul warned against. It leads to horrors."

My Dragon is Growing

Working out what the work is is the work.
My dragon is growing. The consciousness of an animal is not fundamentally different from that of a human, we are after all merely a different kind of animal. So it's fun to observe and try to understand the lizard brain.

Sapphira's body and brain are different, with much more behaviour "hard wired" and instinctual rather than learned, but nonetheless her conception of the world and her surroundings exist in her imagination. By watching how she sees her world and interacts with it, by becoming part of her world and interacting with her, I can start to feel how she sees her world. She is a hunter. She kills and eats her prey without compassion or mercy. It is her nature. Nature itself is red in tooth and claw, and just as we do she partakes of that nature.

As I am part of her world, she is also part of mine. I live in her imagination and she in mine. Our worlds intersect. And in my imagination she breathes fire. The part of her that is alive in me is very fierce indeed. This I see in her and into her. Sapphira the wise.

My other hunter, Rosie the cat, walks the night. Rosie and Sapphira have an interesting relationship. Sometimes they spend hours just looking at each other. It seems to me that they get on fine, but still, Rosie would kill Sapphira if she could. It is merely her nature. So I try to show Sapphira the place in Rosie's mind where she is afraid, my little fraidy cat. I teach them both not to fear. I don't think they want to tangle with each other, not really. So instead they sit and watch, and dream of each other.

"Beauty is a mind virus. You have to choose whether to let it change you or to kill it. You can't know real beauty and not be changed."

Thursday, 8 February 2018

Understanding Holy Texts: A Curse, A Dead Lion and War in the Bible

Psychological force and spiritual power are not different things.

A Dead Lion

The topic of how to read holy texts, what kind of truth they contain and how to find it, is a difficult one. This is most true of the Bible of course, there is no book more misunderstood nor so misused.

I love that the Bible itself teaches you its nature. The two different creation myths of genesis one and genesis two, written at very different times and in different styles, contradict each other on the order of creation. If you try to read them as historical truth rather than spiritual poetry then they can't both be true. Similarly the two accounts of the death of Judas in the New Testament contradict each other. They can't both be literally true. So the Bible itself teaches that literalism is not where its truth is to be found.

This is the sort of truth I find in the bible. Recounted here without bothering to check any of the details.

I just remembered one of my favourite bible verses. It so applies to the wreckage of my church right now: "Out of the strong comes forth sweetness"

It's from the old testament, from the story of Samson I believe. He's wandering out in the wilderness, as you do. That's as much of that bit as I remember.

And he finds this dead lion. No backstory here as far as I know. Just a dead lion. Inside this corpse (nice story) is a beehive. No idea if that's plausible but that's the story.

So Samson breaks open the hive and eats the honey. And that's what he says "out of the strong comes forth sweetness".

Most bible stories are like that. They don't really make a whole lot of sense. But sometimes they make a powerful point. An image painted in words, long ago, that resonates into our world now.

My old church thought it was so strong, impregnable. And now it's so broken.

But out of it are these beautiful people, some of them so hurt. But many of them have genuinely lived for years trying to love and serve other people. And those of us who went through it together. We've been through so much together, we've seen so much together. We have such a strong faith and such a strong love for one another. And we're all so hurt.

It's so funny. Such a weird unexplainable set of circumstances. So hard to describe and talk about. So bad in so many ways!

And really, some of it so exhilarating as well.

A Curse

The Bible is full of interesting stories, most weird, some terrible and a few really beautiful.

Given my apocalyptic bent I've always been fond of Ezekiel's curse against the city of Tyre. There's nothing quite like a good curse to warm your old cockles. And you never know when a time-tested powerful curse will come in handy.

Tyre was in fact attacked by King Nebuchadnezzar, although historically it's doubtful that the curse was fulfilled exactly as Ezekiel gave it.
Therefore this is what the Sovereign Lord says: I am against you, Tyre, and I will bring many nations against you, like the sea casting up its waves. 
They will destroy the walls of Tyre and pull down her towers; I will scrape away her rubble and make her a bare rock. Out in the sea she will become a place to spread fishnets, for I have spoken, declares the Sovereign Lord. She will become plunder for the nations, and her settlements on the mainland will be ravaged by the sword. Then they will know that I am the Lord.
They will plunder your wealth and loot your merchandise; they will break down your walls and demolish your fine houses and throw your stones, timber and rubble into the sea. I will put an end to your noisy songs, and the music of your harps will be heard no more. I will make you a bare rock, and you will become a place to spread fishnets.

If you dig this kind of thing you should check out Lamentations. It's just awful.

"Your own horror take you, your own nightmare devour you" is still the best curse I know. And as the recipient does it to themselves there are no ethical issues with it.


The hardest parts of the Bible to draw useful meaning from are the parts that describe the angry, vengeful, tribal war God who seemingly condones genocide and every atrocity to free the children of promise.

I love the work of Steve Chalke in this area, particularly his pamphlet "Restoring Confidence in the Bible" which has done just that for me and countless others in the Progressive Christian movement (Christianity without the awful bits).

However, there is an alternative and darker reading that may still have value. The climax of the Bible is the Revelation, which amongst other weirdness describes the final battle between good and evil and the ultimate victory of good. This theme of us joining a war that is already won is a theme woven all through the Bible. The way of war that fights having already won is The Way of the Risen Lord. Much of the early parts of the Bible are a manual of war.

Jehoshaphat orders the worship leaders to be placed at the front of the army, and the enemy is defeated before they arrive.
After consulting the people, Jehoshaphat appointed men to sing to the Lord and to praise him for the splendor of his holiness as they went out at the head of the army, saying:
“Give thanks to the Lord,
for his love endures forever.”
As they began to sing and praise, the Lord set ambushes against the men of Ammon and Moab and Mount Seir who were invading Judah, and they were defeated. The Ammonites and Moabites rose up against the men from Mount Seir to destroy and annihilate them.
Elijah, prophet of God, slaughtering the prophets of the evil religion of Baal.
Answer me, O Lord, answer me, that this people may know that you, O Lord, are God, and that you have turned their hearts back.” Then the fire of the Lord fell and consumed the burnt offering and the wood and the stones and the dust, and licked up the water that was in the trench. And when all the people saw it, they fell on their faces and said, “The Lord, he is God; the Lord, he is God.” And Elijah said to them, “Seize the prophets of Baal; let not one of them escape.” And they seized them. And Elijah brought them down to the brook Kishon and slaughtered them there.
Isaiah, prophet of God, calls on the Angel of Death and tens of thousands of Assyrians are killed.
That night the angel of the Lord went out and put to death a hundred and eighty-five thousand in the Assyrian camp. When the people got up the next morning—there were all the dead bodies! So Sennacherib king of Assyria broke camp and withdrew. He returned to Nineveh and stayed there.
Deborah, prophet of God, leads the Israelites to victory.
Now Deborah, a prophet, the wife of Lappidoth, was leading Israel at that time. She held court under the Palm of Deborah between Ramah and Bethel in the hill country of Ephraim, and the Israelites went up to her to have their disputes decided. She sent for Barak son of Abinoam from Kedesh in Naphtali and said to him, “The Lord, the God of Israel, commands you: ‘Go, take with you ten thousand men of Naphtali and Zebulun and lead them up to Mount Tabor. I will lead Sisera, the commander of Jabin’s army, with his chariots and his troops to the Kishon River and give him into your hands.’ ”
Then Deborah said to Barak, “Go! This is the day the Lord has given Sisera into your hands. Has not the Lord gone ahead of you?” So Barak went down Mount Tabor, with ten thousand men following him. At Barak’s advance, the Lord routed Sisera and all his chariots and army by the sword, and Sisera got down from his chariot and fled on foot.
Barak pursued the chariots and army as far as Harosheth Haggoyim, and all Sisera’s troops fell by the sword; not a man was left.
Elisha, prophet of God, commands the armies of heaven.
When the servant of the man of God got up and went out early the next morning, an army with horses and chariots had surrounded the city. “Oh no, my lord! What shall we do?” the servant asked.
“Don’t be afraid,” the prophet answered. “Those who are with us are more than those who are with them.”
And Elisha prayed, “Open his eyes, Lord, so that he may see.” Then the Lord opened the servant’s eyes, and he looked and saw the hills full of horses and chariots of fire all around Elisha.
And many more of course. The interesting point I draw from this final story of Elisha is to notice that an army of horses and chariots were the state of the art in military technology at the time. I wonder how the armies of heaven appear now.

"The reality of perfect love is the best possible idea. And therefore it must win, simply because it is better in every possible way than any other idea that might stand against it. If it were even possible to only imagine it for an instant, then it would exist. And then it would have to win. It would have been foretold."

Monday, 5 February 2018

Authority and Today's Pain

If you make decisions you can feel proud of then you're able to respect yourself.


I like having authority figures in my life. Not the kind of authority where people decide who is in charge and that they have the right to tell you what to do. That's delegated authority and it generally becomes awful because it's too easy for self interest to take over. I recognise that authority when its wise to do so, but I don't like it.

The authority I like is to find people in my life who know more than me on a topic, and who demonstrate in practise that they do. They're not hard to find, I doubt there's a single topic where I know more than anyone else in the world. There's always someone, somewhere, who knows more. I will then consider them an authority on the topic and be more likely to believe what they say, including within the area of ethics and morality. It's very easy to hurt people and it's very nice to have people in your life, or in a situation, where you feel you can trust their advice. It can help you avoid hurting people unnecessarily and hurting people is really bad if you can avoid it.

I still make my own decisions, I still act on advice or follow rules because I've considered it and think it's good advice. Not taking personal responsibility for your decisions and owning them, letting other people tell you what to do, can mean you make bad decisions and hurt people without realising it (we all do it anyway to be fair) and its still your fault even if you let someone else make the decision for you. Be a human, not a part of a machine that no-one is really driving.

Part of the fun of any social situation is working out what rules of behaviour people seem to be following, what's the sense of group identity at work? You don't always have to follow it, sometimes it's bad. But where you do follow it it's easier for people to interact with you because the code of social conduct provides a framework for people to know each other safely. The rules of conduct, in a good community or setting, will help you to avoid hurting people. This is why consent is such an important topic in many modern communities, because so many people have been hurt because we haven't cared enough about consent.

Today's Pain

It's really interesting how a pain caused by external life circumstances brings back similar pain from the past. Like it touches the same part of your soul.

If you haven't dealt with the pain of the past and aren't willing to face it then you won't be able to face the pain of today. You won't be able to cope with the same pain in other people either. If you recognise your pain in another person empathy will cause you to feel that pain. If you're subconsciously blocking yourself from feeling that pain because you're not yet ready to deal with it then you will have to shut out people with the same pain, or at least that part of them. You won't be able to really see it and you can't help them even if you know they're hurting. The ugly part is when we rationalise shutting out the other person in a way that doesn't require us to admit to ourselves that we can't cope. We often do that by finding a way to blame, to judge, the other person.

If you are willing to face yourself then the pain of today, as inconvenient as it is, can be used as a powerful tool to uncover the pain of the past. Use your pain wisely. It is a sharp tool.

If you don't deal with today's pain honourably then life, the normal operation of your unresolved subconscious, will just engineer circumstances one way or another so you can have another try. Best just to give in as quickly as you're able.

"Given what we know about DNA, we're all just a jumbled up mix of everyone else anyway."