This week’s book is not on complexity per se but looks at the role of metrics which can be detrimental when managing complexity. As many of us have just completed our OKRs for Q2, now is a good time to reflect on the metrics we have set. The book The Tyranny of Metrics by Jerry Z Muller was a book that helped me make sense of why metrics-driven approaches were often failing to live up to my expectations.
The premise of the book is that:
There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive.
Many organizations fall into the trap of what is considered metrics fixation, which Muller defines as:
- the belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics);
- the belief that making such metrics public (transparent) assures that institutions are actually carrying out their purposes (accountability);
- the belief that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay-for-performance) or reputational (rankings).
This isn’t to say metrics can’t be helpful. They certainly can be when they are the right metrics, used correctly, and for the right reasons. At the same time, it can be useful to be very aware of Marilyn Strathern’s variation on Goodhart’s Law, which states, “When a measure becomes a target, it ceases to be a good measure.” That is, any metric that is used as a target can and will be gamed.
Flaws in setting metrics
Several flaws are common in setting metrics.
Measuring the most easily measurable…measuring the simple when the desired outcome is complex.
A recent interviewee answered a question about metrics by sharing a story in an e-commerce experience where they hit their metrics “reducing the return rate of items ordered.” But later noticed that the number of orders had also dropped dramatically. So while they achieved their goal, they hurt the business.
Measuring inputs rather than outcomes.
One classic example of this is the cobra effect. The British government was concerned about the number of cobras in India, so they offered a bounty for any dead cobra. This approach failed as people started breeding cobras to kill and collect the bounty. As a result, the government ended the program, and those breeding cobras released the cobras. The problem was even worse than before.
Degrading information quality through standardization.
Standardization can be beneficial, as it gives a way to compare things. It can seem useful to measure every team member by the same metrics. If they are workers on an assembly line all producing the same widgets, that may be fair, but in most knowledge work, that type of standardization can lead to a loss of important context.
Gaming Metrics
There are also a few different ways that people game metrics:
Gaming through creaming. creaming. This takes place when practitioners find simpler targets or prefer clients with less challenging circumstances.
Improving numbers by lowering standards….airlines improve their on-time performance by increasing the scheduled flying time of their flights.
I once heard of a team that set a goal to hit a number of promotions in a quarter. Their intent was good; they wanted to encourage growth, but putting the promotions as the target led to lower standards to ensure that leaders hit their promotion target.
Improving numbers through omission or distortion of data…Police forces can “reduce” crime rates by booking felonies as misdemeanors.
Cheating
A recent example of this was when employees at Wells Fargo got in trouble for creating fake accounts to hit their account goals.
Pay for performance
Attaching pay-for-performance to just what is measurable can be appropriate:
There are indeed circumstances when pay for measured performance fulfills that promise: when the work to be done is repetitive, uncreative, and involves the production or sale of standardized commodities or services; when there is little possibility of exercising choice over what one does; when there is little intrinsic satisfaction in it; when performance is based almost exclusively on individual effort, rather than that of a team; and when aiding, encouraging, and mentoring others is not an important part of the job.
However, most of the time, that is not the case:
People do want to be rewarded for their performance, both in terms of recognition and remuneration. But there is a difference between promotions (and raises) based on a range of qualities, and direct remuneration based on measured quantities of output.
It is important not to default to metrics as the key to pay-for-performance. For example, a support team could be measured by the number of tickets they pick up. The most senior members of the team may spend more time helping more junior team members with tickets and picking up the most complex cases. If you rewarded them solely on the number of tickets they closed you would disincentivize those team members from doing the team-oriented activities you expect.
Similarly, a focus on pay-for-performance can negatively impact intrinsic motivation:
when mission-oriented organizations try to use extrinsic rewards, as in promises of pay-for-performance, the result may actually be counterproductive. The use of extrinsic rewards for activities of high intrinsic interest leads people to focus on the rewards and not on the intrinsic interest of the task, or on the larger mission of which it is a part. The result is a “crowding out” of intrinsic motivation: having been taught to think of their work tasks primarily as a means toward monetary goals, they lose interest in doing the work for the sake of the larger mission of the institution.
Performance metrics as a measure of accountability help to allocate blame when things go badly, but do little to encourage success, especially when success requires imagination, innovation, and risk.
Negative Consequences
Too much of a focus on metrics can also have negative consequences for the organization.
Goal displacement through diversion of effort to what gets measured…workers who are rewarded for the accomplishment of measurable tasks reduce the effort devoted to other tasks.
Not everything that matters can be measured. It is nearly impossible to put a helpful metric on being a good manager, for example, but if we focus on what is measurable, attention will slip from these non-measurable important activities.
Promoting short-termism. Measured performance encourages what Robert K. Merton called “the imperious immediacy of interests … where the actor’s paramount concern with the foreseen immediate consequences excludes consideration of further or other consequences.”
It is a lot easier to measure short-term consequences than long-term. A relentless focus on metrics can lead to focusing on what we can directly control, which has short-term value. As a result, we won’t focus on long-term wins. This is even more dangerous at Senior levels where leaders have numerical MBOs.
A few other issues to look out for include the following:
Costs in employee time…Those within the organization end up spending more and more time compiling data, writing reports, and attending meetings at which the data and reports are coordinated.
Diminishing utility. Sometimes, newly introduced performance metrics will have immediate benefits in discovering poorly performing outliers…The problem is that the metrics continue to get collected from everyone. And soon the marginal costs of assembling and analyzing the metrics exceed the marginal benefits.
Rewarding luck. Measuring outcomes when the people involved have little control over the results is tantamount to rewarding luck.
And finally, there can be risks to the business as too much of a focus on what is measurable can lead to “Discouraging risk-taking,” “Discouraging innovation,” and “Discouraging cooperation and common purpose.”
When to use metrics and how
With all of this in mind, Muller concludes with a checklist of what to watch for when choosing what to measure.
I found a few of these to be the most useful:
What kind of information are you thinking of measuring? The more the object to be measured resembles inanimate matter, the more likely it is to be measureable…Measurement becomes much less reliable the more its object is human activity, since the objects—people—are self-conscious, and are capable of reacting to the process of being measured.
How useful is the information? To put it another way, ask yourself, is what you are measuring a proxy for what you really want to know?
How useful are more metrics? Remember that measured performance, when useful, is more effective in identifying outliers, especially poor performers or true misconduct. It is likely to be less useful in distinguishing between those in the middle or near the top of the ladder of performance.
To what purposes will the measurement be put, or to put it another way, to whom will the information be made transparent? Here a key distinction is between data to be used for purposes of internal monitoring of performance by the practitioners themselves versus data to be used by external parties for reward and punishment.
Remember that even the best measures are subject to corruption or goal diversion.
As I looked at the OKRs for my team this quarter I realized that we fell into some of the traps this book calls out. We will look to improve next quarter.
How do your OKRs look?