Forgetting Management
How the software industry is leaving essential knowledge behind
This post is part of a series on Creating Effective Teams.
I believe we have made delivering software more complicated than it should be. What used to be a process-driven industry has recently moved to a people-focused one. While this shift has made many things better, it changed a few fundamental aspects for the worse.
We need to understand why.
Focusing on process
In the early 2000s, with the Agile Manifesto, the technology industry moved to lightweight management techniques, valuing software as a distinct engineering category.
It was highly effective. Smaller feedback cycles and more on-demand planning radically changed how teams could deliver.
Yet, these teams were mainly process-focused. If you look at the most successful Agile methodologies, like Scrum or Kanban, you will see there is not much they say about the technology itself. And there is a lot they say about process management.
Team composition at the time would also represent the same thinking. Project managers (PMs) led teams. While they were technically aware, they usually weren’t software engineers. But they had extensive insight into systems thinking and traditional project management. And line management for engineers was the role of managers who were not embedded in any team.
Moving to an engineering-first mentality
However, the industry evolved. We saw the rise of tech-first companies, where technology drives business. Team leadership also needed to adapt, since the traditional project manager was not enough. They were managing highly-skilled engineers without understanding engineering themselves.
More tech-first companies also meant that engineering talent became scarce. This change forced companies to shift their focus to people. To hire and keep the best talent, you need to make engineers fulfilled. It means giving them autonomy and focusing on their career growth.
In this context, the Engineering Manager (EM) role is created together with the dual-track career ladder for engineers. The new ladder allowed engineers to progress their careers while continuing to be technical. And it also offered them a clear career path into management, becoming EMs, then Directors, VPs, or CTOs.
This shift was positive, with skilled engineers managing engineering teams. It created another challenge, though. Engineering managers don’t have project management experience, and teams miss this vital skill.
The NYT article about Google’s Project Oxygen portrays this culture well. The article summarizes management thinking at the time as
Leave people alone. Let the engineers do their stuff. If they become stuck, they’ll ask their bosses, whose deep technical expertise propelled them into management in the first place.
Why does it matter?
Managing a software team is more than leaving people alone and solving problems reactively. It’s also more than just managing the performance and career of individuals. It requires more skills than only understanding technology and people management.
In practice, though, many Engineering Managers go into the role without much experience beyond working as an engineer. Their only expertise is as an individual contributor, and they learn management by making mistakes.
This situation impacts teams negatively very often. While this is not an exhaustive list, here are a few issues I’ve seen personally:
- Teams managed as a collective of individual contributors who don’t collaborate effectively. Personal efficiency is high, but it usually doesn’t result in effective delivery.
- Teams where the lack of process makes it hard for engineers to learn from others on the job. Their primary educational process becomes comments in pull requests.
- Teams where engineers have their performance questioned (and careers affected) because teams are so dysfunctional that individuals don’t have a clear direction on the work they need to do.
- Teams where there is no decision structure, creating an environment where senior engineers are disappointed because they can’t lead, and junior engineers are frustrated because they have to make decisions beyond their experience.
Adding to all the above, anecdotal evidence and perceptions became some of the primary tools used in engineering management. Simple questions about the effectiveness of a software team rely primarily on observation.
In summary, we could do better.
What is better?
Engineering management means understanding how the team works — creating a system that will lead to the group’s best performance. It requires knowledge of the engineering activity, the people, and how all the pieces interact to deliver the best result.
And individual contributors themselves cannot be expected to improve their process. It’s tough to understand and improve the system when you are within it.
“A system cannot understand itself without help from outside the system” — W. Edwards Deming
Engineering management needs engineers and needs systems thinkers with deep knowledge of the system of work itself. They need to be as passionate about how the work works as much as they are about technology choices.
Other industries have done it already. Toyota/Lean Production Systems use the idea of a manager who is deeply skilled in the work and on the system. In The Toyota Way to Lean Leadership, the author talks about four levels that need to be achieved by a successful leader:
- Commit to Self Development
- Coach and Develop Others
- Support Daily Kaizen (system improvement)
- Create Vision and Align Goals
In technology, the first item above is a requirement. Managers are engineers that demonstrate the capacity to improve themselves. The second is often taught, as companies upskill managers in how to manage people. However, the third and fourth levels are not addressed and left for leaders to learn by themselves.
Companies should better support this transition. The journey towards engineering management needs to train engineers in processes, systems thinking, and leadership. We need to move from thinking about the 10x engineer to the 10x team, finding systems of work that allow engineers to deliver their best performance while working as an effective team.
It is not simple. There are often challenges in creating harmony: how to think about systems in software? How to balance individuals and team needs? How to use processes without constraining individuals? How to use the right amount of metrics to be inspired but not constrained?
These are all ideas we will be exploring in the next few posts.
