Monday, 23 October 2017

The Craft of Software Engineering: Article Preview

The art of the science is the science of the art

A short preview of an article on software engineering that I've started. I hope I make the time to complete the article. The title is "The Craft of Software Engineering: Abstractions and Processes".
Note: there are two follow-on articles from this one:  


Programming is about abstractions. We sometimes say that the language that computers speak is ones and zeros, binary. In fact representing the most fundamental level of computer operations in this way is an abstraction to enable humans to understand. The language computers speak is the electromagnetic dance across silicone and wires precisely choreographed by the vibrations of quartz crystals at the heart of the machine. Sequences of ones and zeros are an abstract representation of that dance. But ones and zeros, each stanza representing data consumed by the machine or an instruction for it to perform, are a very low level abstraction and hard for humans to understand.

So the art and craft of programming is creating abstractions that make it easier for humans to understand a large system that is ultimately executed as whirling electrons in etched silicone of unimaginable complexity (themselves far too complex to understand without higher level abstractions for the building blocks of their functionality). Abstractions provide a way of conceptualising elements of software as "black boxes" without having to know all the details of their implementation to understand their role and behaviour. This is why computer programming is an engineering field. The substance of software engineering is found in design and structure, the way parts of a system are related and how they interact. The creation of useful abstractions that make it possible to reason at a high level about the behaviour of a whole system consisting of many tens of thousands of lines of code and many different constituent parts.

At its heart an abstraction is a metaphor, a way to simplify complex ideas by applying a label to them. You can use abstractions to provide the structure of your application, making it possible to understand the structure without having to know the details, and you can also use abstractions to model functionality of your application and model the problem domain it is working in. In this way abstractions are best used to simplify and reveal complexity. A typical application includes a lot of inherent complexity and it is impossible to understand it all in terms of lines of code. A good programmer can hold about ten thousand lines of code in their head irrespective of structure, beyond that you need structure and abstractions in order to be able to think about the behaviour of the system as a whole. Used in this ways abstractions simplify and reveal complexity.

Another way to use abstractions is to hide complexity. You take a bunch of implementation complexity that feels like it is necessary complexity and slap a label on it so you no longer need to think about it. So your abstraction conceals and obscures complexity instead of revealing it. The great advantage of this style of development is that you no longer have to think very much about the mess you're making. The disadvantage is that you can no longer see the mess you're making and it's correspondingly harder to find patterns in it that might emerge to make better abstractions as you refactor your model to better match the new areas of the problem domain your application is moving into tackling. I really like the definition of a good abstraction as something that simplifies and reveals complexity. The requirement for this to be true is that your abstraction must accurately convey to the readers of your code the intent of the abstraction and provide a guide into understand the details of the implementation so that it may reveal the inherent complexity rather than obscure it.


What is currently seen as the normal model for the cycle of software development and release is the reifing of the male sexual experience.  All that effort and angst culminating in a glorious self-celebratory climax that leaves everyone involved exhausted, a bit used up and slightly ashamed without really knowing why. 
What we need is something a bit more rhythmic, something a bit more nurturing and sustainable, something a bit more female. 
I would note that, unlike the average male sexual encounter, the software development cycle typically takes longer than expected...
Agile as a movement exploded and immediately became commercialised and caricatured. Just because people talk a lot of nonsense about it doesn't mean that there isn't a gem of an extraordinarily good idea there. The essence of agile is engineering business processes to permit you to change quickly.A side goal and effect of agile is to have processes be lightweight and continuous enough to be background and absorb minimal cognitive effort. When regular tasks are routine enough they become less effort. This means that humans operate most efficiently with regular tasks freeing their conscious mind for creative effort, like understanding a problem and applying a logical framework (a programming language) to creatively solving it. The trouble with this is that being aware of, and therefore able to understand, processes you're deliberately allowing to become subconscious (effortless) is tricky. Seeing and understanding the processes you actually follow (and not just what you think you follow) is vital to be able to change. It's not impossible though, you just have to watch and think about what you do both individually and collectively.

Agile processes understand they're being performed by humans and work with the ways humans work efficiently. For example minimise context switching. Many companies looked at the fad of agile and concluded that most of what was being said was nonsense, which is a reasonable conclusion. Many companies however did adopt elements of agile that make clear sense, like sprints and kanban boards.

Let's look at a typical company using a waterfall development approach. They have a six monthly release cycle, which for many of their customers is quite enough thank you and possibly too much. Enterprise expectations around the cost of upgrading software has grown up alongside the industry standard development practises which say it takes about six months or so to release a new version. If we can change the perception of upgrade costs, by making them robust and reliable, that could gradually change.

A company like this may well have taken kanban and sprints and started using them as tools, whilst maintaining waterfall as the core development model. For a company like this if an important customer comes with a feature requirement then according to their standard development model the soonest they might be able to deliver that feature is a year. If planning is already done for this cycle then it will have to go into the following release, which is another six months after the current release is done. In practise it's not realistic to have to delay key features up to a year, so planning for a full release cycle is more often than not a work of part fiction. A few months in a chunk of it will be abandoned, cut or replaced with something else. Another few months in the same will happen. So that heavy planning and specification period includes a lot of wasted effort.

Fortunately adding sprints to a waterfall process enables a key step towards agile development. Prioritising and specifying and estimating per sprint rather than cycle. This makes it really easy to change direction quickly. Your next prioritisation session will be a maximum of, say, three weeks away instead of six months away. You still do all the planning, specification and estimating, but you amortize it across the project. You keep a current list of all your top priorities, but you get the opportunity to revise and update that as you go. If you are able to find development workflows that let you keep the mainline up to date and releasable, then you can ship a new version with some working new features at almost any point. Ideally at the click of a single button.

Product Design

In software engineering the abstractions you use will be easiest to understand and maintain if they map closely to the problem domain. This way by understanding the problem domain you will be able to understand the structure of the code, and understanding the code will bring you into a better understanding of the problem domain.

The same kind of thinking can be applied to product design. The abstractions you present to the user, which will often map to the abstractions you use for your product implementation as its core model, provide a framework for them to think about the problem. So the closer the abstractions you present to the user map to the problem domain the more "intuitive" they will find your product. Within this there are useful questions we can ask ourselves. How many of our core concepts, our model, does a user need to already understand in able to accomplish a simple task? The answer to this question will determine how easy it is to learn to use your software, what path into learning do you provide and can users start with a very simple model and gradually expand it to more fully understand the capabilities of your software. Unfortunately it is often the case that in order to use a software product at all you already need to understand how it all works. This is a high barrier to entry, we present the test to the user before the lessons...

As you introduce new concepts and abstractions into your software, how well do they fit in to the existing abstractions or is it entirely a new set of concepts your user will be required to understand? Do you overload terms to have different meanings in different places ("status" is a common one), so users have to juggle multiple means for the same term depending on what they're doing? Is your internal model consistent or are there lots of special cases to learn? These are the sort of questions the answer to which will dramatically affect the user experience and the ability of users to get the best from your software. This is why standard agile practises encourage a customer or customer representative (someone who understands the problem domain and how the software is actually used) be involved with prioritisation and specification (but not estimating which is exclusively a developer concern).

A follow on to this preview, applying the engineering mindset to business processes, can be found at: Abstractions: Business Processes.

(My conclusion will be, with an exploration of the human psychology that makes it true, that elements of the agile process and principles from continuous deployment match much more closely to the natural way that humans work effectively. How visible and understood process become habitual, and therefore less effort, if they're applied continuously and rhythmically (with momentum). I'll also look at the process(es) of engineering and how the development of abstractions also applies to product functionality as well the development of software.)

Friday, 20 October 2017

Love, you are more precious than silver

"I will let my heart break. I will try and let my heart be broken."

Love you are more precious than silver,
love you are more costly than gold.
Love you are more beautiful than diamonds,
and nothing I desire compares with thee.

A slight modification of a song by Lynn DeShazo from 1982.

"The rushing of the wolves. What if the wildness, the savagery, isn't dark just chaotic. What if it can be good."

Friday, 13 October 2017

My Son

You probably have to die, but no-one says you have to go quietly
I watch my son grow
Learn to think.
How speech comes from thought
And thought is growing.

That he loves to communicate.
To really be known and understood.
How precious that is.
And how words grow inside him.

And how his soul burns.
How he flames with life.
So rough. So loving.

My friend.

Benjamin is rough, too rough. I played with him last night, whilst waiting for him to fall asleep. Every time I stopped playing with him he'd wait for his opportunity and then poke me in the eye to get my attention again. Little git. I tried to be cross but it was too funny.

One of the most fun aspects of teaching him is teaching him to be gentle without squashing the life and roughness out of him. I teach him it's ok to be rough with me, but he must control it. He must learn to control himself. He's not allowed to be rough with Delia at all, and he knows it (he doesn't always stick to it, but he knows it). He's allowed to be rough with Irina, but only with the limits she sets. That one's harder but he's getting there, Irina can be quite fierce when she wants to which helps.

Rosie, our cat, has her own way of teaching Benjamin to be gentle...

"The more compassionate you are with yourself the more compassionate you're able to be with others. And vice-versa. Being compassionate means understanding without blaming or judging. It doesn't mean excusing or denying problems."

Truth and Beauty and the Here and Now

Life is hard, but love will help. Love is on our side.
If your understanding of poetry is that it is the expression of the beautiful, then by any reasonable measure love is poetry. Therefore it is only possible to express any deep understanding of love in a form that is pure art and reveals pure beauty. Therefore the best philosophy of love must be poetry, and this to me is the essence of the language of spirituality. Truth expressed in beauty, revealing love.

 This is how we're able to say that truth is beauty. Not that mere appearance of beauty ensures truth, but that anything that claims to be a deep truth about love, if it is not beautiful it simply cannot be true.
 Not that all of reality is beautiful. To claim that would clearly be false. It's just that I think love is better and I believe that because it is better, love wins.

Truth can be painful, whilst still being beautiful. Truth is able to remain beautiful through the pain, because it carries hope. We can have faith in the hope, and that itself is beautiful. We trace the rainbow through the rain.

I've travelled the world, a bit, and everywhere I go I meet good people.

You have an incredibly rich and complex understanding of the world, deeper and broader than at any point in history before. Merely by living, and functioning (mostly), in modern society there is a huge amount you just know and understand (and take for granted) that was just not understood by those who came before us and upon whose shoulders we stand.

Whatever you may think of yourself, that means you're clever. You know a lot of stuff and you use that knowledge all the time.

But the truth is, we all have our gifts and our qualities. The essence of what it is to be you or to be me. We all can have a place, a part to play. Things to do and ways to be useful, and to enjoy doing it. It just takes a bit of working out between us.

What a thing it is when you're able to really see people and to really trust people, whilst keeping your eyes open.

And yes, maybe some people are dumb. But only relatively speaking, not in absolute terms. And it makes the challenge of raising children such a task. There's so much for them to learn and understand (and explore and create and enjoy).

We have to care for those we love, but that still leaves us (sometimes) with the capacity to care for others too. It seems like wishful thinking, but it actually happens in small communities all the time, where people really care for each other. It's the nature of what it is to be a family.

Negative behaviour and negative world views inevitably come from the way people see the world, how they learned to see the world from their childhood and life experiences. Many, probably most, want to be good but the world is scary, complex and confusing with so many contradictory voices and hosts of difficulties.  A negative worldview (including how you see other people including people groups) can seem like the truth, like the right way to think.

The challenge, and it is a challenge, is to demonstrate that a positive world view can work and that caring is actually just a better way to live.

However far apart we may feel, and honestly everyone feels like an outsider and that's scary, the reality is that we do actually live together. We share this place, like it or not.

If we accept that because love is beautiful any expression of the nature of love must also be beautiful and that this is the essence of spirituality, this tells us something of the nature of the truth any purported spiritual work or text contains. The truth of anything claiming to be spiritual, claiming to reveal something of the nature of love, is to be found in its beauty. So we read spiritual works not for the mundane and literal truth they may (or may not) contain, but in order to understand the beauty within them and how that beauty reflects and casts light on reality and the substance and ways of love.

"That which art love, I will trust it completely. That which art the enemy I will fight. Even though I am scared I will fight."