The Craft of Software Engineering: Article Preview
 |
| The art of the science is the science of the art |
A short preview of an article on software engineering that I've started. I hope I make the time to complete the article. The title is "The Craft of Software Engineering: Abstractions and Processes".
Note: there are two follow-on articles from this one:
Abstractions
Programming is about abstractions. We sometimes say that the language that computers speak is ones and zeros, binary. In fact representing the most fundamental level of computer operations in this way is an abstraction to enable humans to understand. The language computers speak is the electromagnetic dance across silicone and wires precisely choreographed by the vibrations of quartz crystals at the heart of the machine. Sequences of ones and zeros are an abstract representation of that dance. But ones and zeros, each stanza representing data consumed by the machine or an instruction for it to perform, are a very low level abstraction and hard for humans to understand.So the art and craft of programming is creating abstractions that make it easier for humans to understand a large system that is ultimately executed as whirling electrons in etched silicone of unimaginable complexity (themselves far too complex to understand without higher level abstractions for the building blocks of their functionality). Abstractions provide a way of conceptualising elements of software as "black boxes" without having to know all the details of their implementation to understand their role and behaviour. This is why computer programming is an engineering field. The substance of software engineering is found in design and structure, the way parts of a system are related and how they interact. The creation of useful abstractions that make it possible to reason at a high level about the behaviour of a whole system consisting of many tens of thousands of lines of code and many different constituent parts.
At its heart an abstraction is a metaphor, a way to simplify complex ideas by applying a label to them. You can use abstractions to provide the structure of your application, making it possible to understand the structure without having to know the details, and you can also use abstractions to model functionality of your application and model the problem domain it is working in. In this way abstractions are best used to simplify and reveal complexity. A typical application includes a lot of inherent complexity and it is impossible to understand it all in terms of lines of code. A good programmer can hold about ten thousand lines of code in their head irrespective of structure, beyond that you need structure and abstractions in order to be able to think about the behaviour of the system as a whole. Used in this ways abstractions simplify and reveal complexity.
Another way to use abstractions is to hide complexity. You take a bunch of implementation complexity that feels like it is necessary complexity and slap a label on it so you no longer need to think about it. So your abstraction conceals and obscures complexity instead of revealing it. The great advantage of this style of development is that you no longer have to think very much about the mess you're making. The disadvantage is that you can no longer see the mess you're making and it's correspondingly harder to find patterns in it that might emerge to make better abstractions as you refactor your model to better match the new areas of the problem domain your application is moving into tackling. I really like the definition of a good abstraction as something that simplifies and reveals complexity. The requirement for this to be true is that your abstraction must accurately convey to the readers of your code the intent of the abstraction and provide a guide into understand the details of the implementation so that it may reveal the inherent complexity rather than obscure it.
Processes
What is currently seen as the normal model for the cycle of software development and release is the reifing of the male sexual experience. All that effort and angst culminating in a glorious self-celebratory climax that leaves everyone involved exhausted, a bit used up and slightly ashamed without really knowing why.
What we need is something a bit more rhythmic, something a bit more nurturing and sustainable, something a bit more female.
I would note that, unlike the average male sexual encounter, the software development cycle typically takes longer than expected...Agile as a movement exploded and immediately became commercialised and caricatured. Just because people talk a lot of nonsense about it doesn't mean that there isn't a gem of an extraordinarily good idea there. The essence of agile is engineering business processes to permit you to change quickly.A side goal and effect of agile is to have processes be lightweight and continuous enough to be background and absorb minimal cognitive effort. When regular tasks are routine enough they become less effort. This means that humans operate most efficiently with regular tasks freeing their conscious mind for creative effort, like understanding a problem and applying a logical framework (a programming language) to creatively solving it. The trouble with this is that being aware of, and therefore able to understand, processes you're deliberately allowing to become subconscious (effortless) is tricky. Seeing and understanding the processes you actually follow (and not just what you think you follow) is vital to be able to change. It's not impossible though, you just have to watch and think about what you do both individually and collectively.
Agile processes understand they're being performed by humans and work with the ways humans work efficiently. For example minimise context switching. Many companies looked at the fad of agile and concluded that most of what was being said was nonsense, which is a reasonable conclusion. Many companies however did adopt elements of agile that make clear sense, like sprints and kanban boards.
Let's look at a typical company using a waterfall development approach. They have a six monthly release cycle, which for many of their customers is quite enough thank you and possibly too much. Enterprise expectations around the cost of upgrading software has grown up alongside the industry standard development practises which say it takes about six months or so to release a new version. If we can change the perception of upgrade costs, by making them robust and reliable, that could gradually change.
A company like this may well have taken kanban and sprints and started using them as tools, whilst maintaining waterfall as the core development model. For a company like this if an important customer comes with a feature requirement then according to their standard development model the soonest they might be able to deliver that feature is a year. If planning is already done for this cycle then it will have to go into the following release, which is another six months after the current release is done. In practise it's not realistic to have to delay key features up to a year, so planning for a full release cycle is more often than not a work of part fiction. A few months in a chunk of it will be abandoned, cut or replaced with something else. Another few months in the same will happen. So that heavy planning and specification period includes a lot of wasted effort.
Fortunately adding sprints to a waterfall process enables a key step towards agile development. Prioritising and specifying and estimating per sprint rather than cycle. This makes it really easy to change direction quickly. Your next prioritisation session will be a maximum of, say, three weeks away instead of six months away. You still do all the planning, specification and estimating, but you amortize it across the project. You keep a current list of all your top priorities, but you get the opportunity to revise and update that as you go. If you are able to find development workflows that let you keep the mainline up to date and releasable, then you can ship a new version with some working new features at almost any point. Ideally at the click of a single button.
Product Design
In software engineering the abstractions you use will be easiest to understand and maintain if they map closely to the problem domain. This way by understanding the problem domain you will be able to understand the structure of the code, and understanding the code will bring you into a better understanding of the problem domain.The same kind of thinking can be applied to product design. The abstractions you present to the user, which will often map to the abstractions you use for your product implementation as its core model, provide a framework for them to think about the problem. So the closer the abstractions you present to the user map to the problem domain the more "intuitive" they will find your product. Within this there are useful questions we can ask ourselves. How many of our core concepts, our model, does a user need to already understand in able to accomplish a simple task? The answer to this question will determine how easy it is to learn to use your software, what path into learning do you provide and can users start with a very simple model and gradually expand it to more fully understand the capabilities of your software. Unfortunately it is often the case that in order to use a software product at all you already need to understand how it all works. This is a high barrier to entry, we present the test to the user before the lessons...
As you introduce new concepts and abstractions into your software, how well do they fit in to the existing abstractions or is it entirely a new set of concepts your user will be required to understand? Do you overload terms to have different meanings in different places ("status" is a common one), so users have to juggle multiple means for the same term depending on what they're doing? Is your internal model consistent or are there lots of special cases to learn? These are the sort of questions the answer to which will dramatically affect the user experience and the ability of users to get the best from your software. This is why standard agile practises encourage a customer or customer representative (someone who understands the problem domain and how the software is actually used) be involved with prioritisation and specification (but not estimating which is exclusively a developer concern).
A follow on to this preview, applying the engineering mindset to business processes, can be found at: Abstractions: Business Processes.
(My conclusion will be, with an exploration of the human psychology that makes it true, that elements of the agile process and principles from continuous deployment match much more closely to the natural way that humans work effectively. How visible and understood process become habitual, and therefore less effort, if they're applied continuously and rhythmically (with momentum). I'll also look at the process(es) of engineering and how the development of abstractions also applies to product functionality as well the development of software.)
