Why does Process Matter?

This post is part of a series on Creating Effective Teams.

Our industry has shifted from a process-driven focus, with Waterfall and then Agile methodologies, to a people one, where we have been focusing on engineers driving the output of teams independently. While this shift has made many things better, it changed a few fundamental aspects for the worse.

I believe this is a situation we can improve. But why does the process matter?

The process as a chore

When talking about process, the thought seems to go straight to something rigid and inflexible. Engineers that are old (or unlucky) enough will have experienced truly waterfall projects where there are practices that are hard to comprehend. And when confronting an apparent inefficiency, the answer would be “it’s the way it’s done here.” It is the process we follow.

While the waterfall example is obvious, the same happens with more modern approaches. The principles in the Agile Manifesto created the practices in Scrum, Kanban, etc. And then, the methods got expanded, and we forgot the principles in most cases. It started the same mechanical behavior described in the paragraph above. I mean, you don’t have to go far. A small but ironic example is that we are all still having “Stand up” meetings with everyone sitting down in a video call.

Blindly following a rigid system is an issue. You eventually end up performing routines where nobody understands their reasoning. John Seddon summarizes it well.

I have learned that as soon as you introduce controls on human behavior, you lose the game, particularly when those controls are at odds with the work — John Seddon

With that in mind, it’s easy to understand the process aversion we have in the technology industry, where things change pretty rapidly. When the context is often in flux, adhering to specific practices can inhibit innovation. And making hard-to-find, highly skilled employees adhere to inefficient rules can only inhibit motivation and retention as well.

There is always a system

The challenge with anti-process thinking is that there is always a system.

Therefore, not actively defining will only mean that an organic system will emerge, determined mainly by the people and personalities that exist in the team. In my experience, taking that approach in software teams will lead to suboptimal results.

The tendency I’ve seen is for individuals to focus on their strengths, working on areas they are comfortable with and that will get them points on the board. Unfortunately, it will eventually lead to knowledge silos, hero engineers, and unclear participation rules that inhibit inclusion and learning.

Examples of it going sideways are widespread. In one team I’ve managed, the planning system that emerged was so technical that the product manager would come to me after meetings to ask me what had happened. He had no clear view of how the project he was delivering was going. In another situation, work definition was left to individuals, and people that were new to the team couldn’t define their work well due to lack of context. That meant they would also not deliver well individually and were considered to have lower performance.

Keep that in mind. If you manage a team and don’t care about the system, it only means you are letting other people define it, intentionally or not. And it will have consequences not only on the team’s output but also on every individual’s career.

The art of software delivery

Another common concern is that software development is a creative process. A significant (and necessary) movement around software craftsmanship highlights how much of the discipline is based on an engineer’s experience and inspiration.

While that is true, we have to be aware that even truly creative artists need a routine. Chuck Close reminds us that “Inspiration is for amateurs. The rest of us just show up and get to work”.

As in a sports team or any other collaborative effort, the goal is to create a system that elevates the team and individuals. It should allow engineers to focus on their strengths and abilities while also enabling collaboration, learning, and high productivity for the group.

In this scenario, focusing on the individual is the wrong approach. As stated by Deming, most of the productivity improvements are in how the system works.

The fact is that the system that people work in and the interaction with people may account for 90 or 95 percent of performance. — W. Edwards Deming

Finding the right balance

Having a well-defined system will mean that individuals can focus on their strengths, and everyone understands how to participate. It also means you can improve it continuously since you have a solid ground to stand on. It’s impossible to iterate if all the team does is change practices in a not even defined process.

Other industries have done it effectively. Lean Production Systems are the primary example of an area where the process is strictly enforced but also owned by their participants while constantly evolving.

While software engineering is not a production line, the same benefits can be seen in it. Having done this for multiple teams over the years, I can say that clear working rules for software teams will allow engineers to focus more on engineering and less on management, will create more equitable rules of participation, and ultimately increase productivity.

While each team and organization needs to define the optimal systems for them, I will share some of the ideas and practices that I have applied over the years in the following posts of this series.