Growth as a Software Engineer

Key skills to go beyond senior

For many software engineers, career progression can feel frustratingly linear, with a low ceiling, outside of a few large corporations, which might not be places everyone wants to work. From junior to ‘mid’ to senior can sometimes be achieved in a few years, and those looking to further their prospects and their pay packet from that point can struggle to avoid the management track.

Photo by Josh Felise

It goes without saying that what makes a good engineer a good engineer very rarely translates into making them a good manager. Different companies structure themselves very differently too which doesn’t help - team leads, tech leads, technical product managers, software architects, staff engineers there are a multitude of names and roles given to the most senior engineers within an organisation, and no clear consensus on what the best structure is - often they can emerge reactively from trying to retain key people.

This can make it tough for those in a senior role looking to progress to the next step. Are you a staff engineer or a software architect? You may find two roles with those two names that present very similar requirements, however two roles for staff engineer - even within the same company - may have very different requirements!

It seems it’s best to ignore roles, names, the specifics of what individual companies may want their most senior engineers to achieve, and instead focus on developing the skills that will get you there, regardless of what the roles might be.

In no particular order I’ve listed a few key areas of development I think are neccessary—in some combination—to get you through the threshold of senior engineer and into roles above:

1. Holistic Engineering

   This is the skill of understanding the context in which you are working as broadly as possible. For a system, product, module you are working on, you should be able to trace its value all the way back to fundamentals. For a given piece of work, however big or small: Who does it benefit (probably multiple personas, internal and external)? How does it benefit them? Where are the lines of communication to those people? Do they know what you’re doing?
   
   Understanding this at all times—and if it’s not understood making efforts to make sure it is so—pays dividends. It aids everything else you do as an engineer, especially in a product organisation.

   
   A key side-effect of this skill is that, in the right organisation, it can engender trust and autonomy. When the trust isn’t there, the same effect can be approximated with stakeholder meetings, where all of this is worked out ‘on paper’. This process is expensive, time-consuming and often frustrating, but it’s a habit many organisations fall into.
   
   The right engineers, with the right team around them, empowered in the right ways can—sometimes—cut through this.

2. Decision Making

   Opinions are something that develop most noticeably with engineering experience. People just starting out are likely to follow blindly, or adopt the opinions of those around them without question. As you become more experienced, some opinions will burnish, others might change. You may come up with some of your own ideas. A large-enough collection of engineers will always disagree about some things, big and small. Choice of language, choice of architecture, team structure, meeting schedule, there’s an endless list of sources of friction.

   
   Decision making sounds trivial at first, but it’s a key leadership skill. Can you deal with making a decision that you know will make at least some people unhappy, and own it and the disgruntlement that comes with it? What about a decision that makes most or all people unhappy but you know is for the greater good?¹

   
   For example you may find yourself in a situation where a key production system is written in, say, VBA. It’s difficult to maintain, breaks frequently, and it’s very hard to hire new people into the project. But the people maintaining it have put years into understanding its many quirks and are experts in it. To break it up and re-write it might be the best decision for the company, but might meet resistance from those for whom it has become their speciality.

3. Wrangling Data

   Being able to be fluent in data is useful in a multitude of situations within and without engineerning itself. You probably have some capability here already, but it’s worth getting really good at it. Finding and fixing the nastiest of bugs can often require a data-oriented approach, and quickly and reliably being able to extract complex metrics from your observability platform or log data is the kind of skill that will make you very useful to a lot of people.
   
   For example, are you reasonably fluent in a data-wrangling tool, for example Jupyter Notebooks? Can you build something that takes in 100MB of csv log data and produces a graph of something using string matching? Can you take a spreadsheet from finance and validate it against the transaction log in your database? Can you make a detailed case for a new approach using a fully worked cost estimate?

4. Understand the whole stack at some level

   Similar in concept to holistic engineering, it’s important to understand a bit about the whole stack of software within which you’re working (and at least a rudimentry understanding of the hardware too!). In an online application there’s probably at the very least a front end or client, a back end, some data stores, some caches, potentially split across many (micro?) services and using quite a few pieces of technology and third-party services.

   
   Other systems may involve more or fewer layers of complexity, but there are always layers. If you are writing a native application you will be dealing with the operating system. If you are working as a data scientist you will be using tools like pandas or R as well as plenty of other libraries. It is important to understand not just how these tools work at some level but also why they work, why they’ve been chosen, why they’re popular (or unpopular), and what the alternatives are.

5. Write less code, communicate more

   In a commercial setting, there’s a cap on how useful an excellent programmer can be if all they do is write code, and if all you really want to do is write code, it’s probably best not to pursue advancements in your career.
   
   This is because advancement will always be related to impact, and by working essentially alone there is a limit to how much impact you can deliver. Working alone is also a liability for your employer, particularly if you’re working on critical or complex systems - what happens when you’re not around? This is also a short path to burnout.

   
   If you do want to pursue that advancement though, a fundamental skill you will need to develop is leadership. Line management is one type of leadership, and a trap many organisations fall into is providing this as the only path to its engineers. However technical leadership—which involves just as much communication—is just as important to an engineering org of any substantial size.

   
   Working with others - pairing, mentoring, guiding, etc., as well as participating in meetings about larger decisions will become a more and more useful application of your knowledge and skills.
   
   Building good relationships with others and helping them understand engineering constraints and capabilities is a very good use of your time too. Managers, leadership, and whatever structure exists in your organisation outside engineering will benefit from your insights, especially if there isn’t a clear communication path already. Your data wrangling skills and the ability to answer interesting questions or present novel information will help a lot here!

I hope at least some of this was obvious to you, and I hope at least some of these skills are things you’re aware of and developing, even if I didn’t articulate them in the way you might be used to - I’m learning too!

1. “The Greater Good!”