There are serious limitations to applying metrics to knowledge work like software engineering. For instance, productivity is extremely difficult to estimate and quantify given that every piece of work is unique and each piece of work can be highly variable in terms of size, value and complexity. Attempts to measure productivity using effort estimates like story points are flawed (e.g. very easy to game). The connection between any given piece of work and the bottom line for the business is usually very hard to evaluate. A software development team differs markedly from a factory in this regard.
What gets measured gets managed
—not Peter Drucker
In the business world there is a drive to measure everything. Measuring things allows us to understand them and alter them through experimentation, essentially treating running a business like conducting a science experiment. A lot of very successful enterprises embrace this data-driven approach.
One thing I've seen leaders try to measure is productivity within engineering departments. The thinking is that it would be great if we could understand the productivity of individuals and teams so that we can attempt to maximize it and realize an optimal return on our developer costs.
Wikipedia defines productivity as "efficiency of production" and defines production as "a process of combining various material inputs and immaterial inputs (plans, know-how) in order to make something for consumption (output)". So in software engineering terms productivity is the efficiency of the production of software.
Here's a thought experiment to illustrate an interesting aspect of this:
- Let A and B be two engineers on an engineering team.
- Let A work very hard over the course of a week and deliver several large features that noone uses and have no impact on revenue.
- Let B produce a single small feature that is highly valued by customers and leads to a 20% jump in revenue growth
By our definition of productivity A is more productive than B even though A has produced no actual value.
What this boils down to is that productivity is a measure of outputs, not of outcomes or benefits. Productivity is far more important in unskilled manufacturing work than it is in highly-skilled knowledge work. Productivity is still important in knoweldge work because without it nothing gets done, it's just not the most important thing that management should be optimizing for.
Ultimately we shouldn't care how many features an autonomous, self-organized, multi-disciplinary product team delivers, what really matters is the bottom line and how far the team and the individuals in that team are advancing towards achieving the organization's mission.
If you're still interested in measuring productivity it's worth taking a look at what options are available and whether any of them are suitable.
First we should consider what the output of a software engineer actually is? It isn't lines of code in the same way that lines of text isn't the output of an author or a journalist. What is actually being produced is software and we can roughly divide software up into features. This seems to be the best candidate for our software engineering output, but features are abstract and not easy to measure - you cannot just count the number delivered as they all require different amounts of effort. Asking engineers to estimate the effort involved is an option, but not a great one.
There are all sorts of interesting metrics you can collect on your teams that will give you insight into things like team health, or if you're interested in things from a DevOps perspective Accelerate suggests measuring deployment frequency, lead time, change fail rate and mean time to recovery. When it comes to productivity though I am not convinced there are any good objective measurements and certainly no objective measurements that can paint a full picture.
Not everything that can be counted counts, and not everything that counts can be counted
—William Bruce Cameron, Informal Sociology: A Casual Introduction to Sociological Thinking
Most things in life cannot be objectively quantified and so we are forced to rely on the subjective evaluations of people we regard as qualified to give these evaluations (e.g. film critics or sommeliers). Obviously subjective evaluations lack objectivity and are prone to all sorts of biases and errors, but they are much more holistic and nuanced than simple quantified metrics.
In the case of evaluating software engineering teams and individuals I don't believe there is any better option than to talk to someone who you trust is capable of making and honestly communicating that judgement. I also don't believe that productivity is the main criteria that we should be evaluating, rather we should be asking how well teams and individuals contribute to achieving the organization's mission and how well they multiply the contributions of those around them.
A high-performing organization is organized around shared objectives and glued together with trust.