Software Engineering Abstractions: Design and Testing
Note: this is the third article in a series. The two previous are:
Back to the topic of abstractions and software engineering and thinking about the design of software and engineering systems.
The art and science of Software Engineering is the developing of good abstractions that map well to the problem domain. A good way to create these is to let the abstractions, their shape and form, emerge from gradually solving aspects of the problem you are working on. If instead you design abstractions first, typically in a design phase prior to implementation, then you impose those abstractions onto the way you solve the problem. The way you look at the problem is the way you implement your solution. So when you look at the system, when you look at the problem you see your abstractions. If you design your abstractions at a high level up front then you really need to get them right. You need to properly understand the problem, and the details and shape of implementing that, before you start doing it.
But in practise, in the way software is written, you're going to be doing it bit by bit anyway.
This is what I learned from and loved about Test Driven Development. I worked at a fully agile shop for four years and we religiously applied TDD and we included as an agile practise regular assessments of our structure and abstractions, how we modelled the problem domain. We kept checking if our current structure and model still mapped well to the problem domain and new parts of the problem domain we were expanding into. We included the cost of refactoring within the cost of other work we did.
The Test Driven approach, especially the test first aspect of it, encourages this kind of thinking. This thing that I am building right now, the way I'm changing the code right now, how should it look and what would be the best design. You think about each piece of your code in this way and write tests for this behaviour before you write the code. For experimental stuff where you don't know how to solve the problem without writing some code you "spike", you build one to solve the problem and then you throw it away and build a properly designed one based on your new understanding. We did this religiously. I'm not trying to convert you, I'll try and draw a lesson about design from it.
Note that this process is very agile. Changing your design is built into the process, and happens all the time, so it's not something too onerous. We have a shared understanding of the code, with different specialities and roles within the development team. We have a shared understanding of the design and we evolve that together. (In full agile pair programming is a great tool for building and working on that shared understanding. We did that for four years together.)
Your test system then becomes a great benefit and a great cost in this work. Your good tests, that test behaviour rather than implementation, and especially your functional and integration tests (depending on how you define these terms) are able to tell you when a refactoring is done - because starting the refactor causes them to fail and when behaviour is restored they pass again. Your bad tests that are too tightly coupled to the implementation (plus tests for specific units that have behavioural changes as part of the refactor) will fail and be expensive to fix.
For many developers I know who learned testing and learned design processes through a test first approach, they eventually found that full TDD is too expensive to be practical in many places. Throwing away a spike for example is expensive. However if you learned the lessons around design thinking from TDD and you're clever and expressive in the tests that you do write (more black box tests, fewer white box tests) you can still use the design techniques you learned. Doing TDD in a disciplined way for a good period of time is still a great way to learn those lessons and burn them in.
Thinking about design, the shape and structure and components of software is still thinking with and working on abstractions.
Much of life is actually about thinking in abstractions. In basic approach there can be a big difference between allowing your abstractions to be formed through understanding, and trying to have an abstract understanding of abstractions (theory) and applying that. (The reality being that we all do both most of the time in different proportions depending on how much we've had a chance to learn and from where.) Let me give a practical example. From the sports world we have a concept of "a team". We also all have an idea of what it means to be "a team" in this context. Our understanding of what it means to be a team is the bundle of understanding that we label "sports team". What they actually are is a bunch of individuals with particular skills, playing together for money, and whole bunch of associated activity around that. So the concept is a way of modelling human behaviour. Given our understanding of the best way that teams work we are able to assess if any player is a good member of the team and knowing all the players we can assess the qualities of the team.
The concept of the team is a useful abstraction for modelling and understanding behaviour and being able to predict and understand aggregate behaviour of a group of individuals. The concept of the team is a higher level abstraction for reasoning about more detailed behaviour of a whole group of individuals. It's not in itself "real", it's an abstract understanding of behaviour. An abstraction for thinking.
If the manager is a good manager they will understand that the important qualities of a team come from a shared sense of purpose, caring about the purpose and caring about each other. Humans like to do these things and it brings out the best in them. So a good manager will try to create an environment where that becomes possible.
A bad manager (and I'm only considering the extremes, it's a spectrum that everyone is on somewhere) will have a conception of how a team ought to behave. If that conception doesn't really map very well to how humans are actually able to behave then they will see "bad team members" who aren't fulfilling their role. In fact, if people don't fulfil their role then it's possible that the team processes and procedures don't really permit people to work at their best. A big part of that is down to the team members too. That culture where people can thrive requires people to be willing to really work together, and that requires being known. People can only be known if they feel and know that who they are will be accepted. That's a cultural issue and it takes work.
So the lesson here is that abstractions (and processes and procedures are abstractions and descriptions of the operation of abstractions) must map well to the best possible understanding of the actual situation. It's rare to be able to do that in the abstract without actually being in the middle of the situation, as it were. So gradually and progressively as our understanding grows, considering our abstractions and being willing and able to change them is a good approach.
The progressive approach also works well for API design and feature design. My guiding principle is that simple things should be simple and complex things should be possible. This makes for APIs and features that are easy to use, but using the basic features exposes you to the concepts and abstractions you need to understand in order to be able to achieve more complex stuff. Using the API and product in basic ways is your tutorial on how to use it.
The nice thing is that you can retrofit this approach, or expand on what you have of this approach already, into an existing complex product or framework. Develop higher level abstractions, easier ways in, that map well to your existing concepts and tools. Allow the user to gradually expand their use of the product and in the process learn more of the key concepts they need in order to use the more powerful features. Then it becomes a fun product to use, instead of being confusing because of all the power and complexity, the product gradually reveals power and complexity whilst providing ways to understand and manage the complexity.
