The Tyranny of Metrics

The Tyranny of Metrics by Jerry Z. Muller, Princeton University Press, 240 pages, Hardcover, February 2018, ISBN 9780691174952

Is it possible that our fixation on metrics—on gathering, compiling, and evaluating them, on meeting the goals spelled out and measuring performance according to them—is leading us to not actually accomplish the real goals of our work, or meet the purpose of the organizations we work in? According to Jerry Muller’s new book, The Tyranny of Metrics, it is not only possible, it is nearly a pandemic. It occurs when, in an increasingly data-driven world, we’re forced to spend an inordinate amount of time replying to the demands for that data that justifies the existence of our jobs instead of doing the job itself. It also occurs when what we’re measuring, or the way in which we measure it, is untethered, even antithetical, to our real world goals.

It happens when it results in the kind of short-term thinking in the business and financial world that led to the financial crisis of 2008. It happens when political leaders and law enforcement officers chase favorable crime stats by racking up low level arrests at the expense of devoting time and energy to higher-level, longer-term cases against more serious criminals that drive street-level crime, or when police departments massage statistics by reporting crimes as less serious than they really were, such as when the LAPD misclassified nearly 1,200 violent crimes as minor offenses. It happens when school leaders attempt to get their standardized test scores up by teaching to those tests, which not only doesn’t guarantee a better school, it undermines the predictive value of those very tests as a measure of the general knowledge they’re instilling in students and detracts from “goals that are less measurable but no less important, such as instilling good behavior, inspiring a curiosity about the world, and fostering creative thought.” It happens as a result of doctors' and surgeons' unwillingness to take on the hardest cases in order to keep their success rates high, literally costing the lives of those higher risk patients who are denied care.

It all points to a fundamental flaw of being fixated on metrics, rather than focused on serving people:

There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive.

Muller, a historian, explains how he came to the topic—as most of us have—because of more and more requests for data in his own professional life. It all began, for him and his colleagues, with the Commission of the Future of Higher Education convened by the Bush administration, which trickled down through the Middle States Commission on Higher Education to the administration of his university and onto him as the chair of his department.

Soon, I found my time increasingly devoted to answering queries for more and more statistical information about the activities of the department, which diverted my time from tasks such as research, teaching, and mentoring faculty. There were new scales for evaluating the achievements of our graduating seniors—scales that added no useful insights to our previous measuring instrument, namely grades.

Since then, the university has had to hire “ever more data specialists,” and has even added a vice-president for assessment to oversee the “data arms race” that resulted. All of this did more to suck away time and enthusiasm from the job of teaching than it did to improve the quality of education they were offering. Yet, in subsequent years, the demand for data, and the desire to tie resources and funding to such performance metrics in education has only strengthened. It is actually embedded into the law in legislation like No Child Left Behind and Race to the Top. The subject of Jerry Muller's scholarship, in the history of public policy and history of capitalism—"especially the ways in which intellectuals have thought about the social, moral, and political prerequisites and ramifications of business"—left him well placed to dig into the phenomenon he calls metric fixation.

It turns out that this idea can be traced back to the source of so many of our problems—Rob Lowe. Sorry… that is, to a Liberal member of the English parliament, Robert Lowe, who in the 1860s suggested that funding for schools should be tied to test results in the Three R’s (reading, writing, and arithmetic) in those schools. Lowe, it turns out, also brought forth the bill that created Joint Stock and Limited Liability Companies, so he is also known as “the father of modern company law.” One prominent critic at the time, Mathew Arnold, argued that, rather than improving performance in the poorest performing schools (which also happened to be filled with the poorest and most disadvantaged children), this policy would have the effect of removing funding from the schools and pupils that need it most. Over 150 years later, as similar policies have taken hold in America, this is exactly what seems to be happening. Instead of an effort to Save Chicago’s Public Schools in the most underserved neighborhoods, they are being depleted of resources and shut down.

This kind of management by measurement really took hold in America under the influence of Frederick Winslow Taylor in the early part of last century, and Robert McNamara in the post war years. If you’ve studied management even rudimentarily, you know of Taylor and the widespread effect of his ideas on “scientific management,” or Taylorism. But, if you know of McNamara, you probably know him as the architect of the Vietnam War (which, of course, went swimmingly). What you might not know is McNamara, who began as “an accountant who at 24 became the youngest professor at the Harvard Business School,” came out of that same school of data-driven thought that Taylor pioneered. As Muller writes:

The decades in which McNamara rose from business school professor, to Ford Motor Executive, to Secretary of Defense, and finally to president of the World Bank also saw a transformation of American business schools. In an earlier era, business schools had focused on preparing their students for jobs in particular industries and enterprises. From the 1950s onward, the business school ideal became the general manager, equipped with a set of skills that were independent of particular industries.

 

The core of managerial expertise was now defined as a distinct set of skills and techniques, focused on a mastery of quantitative methodologies. Decisions based on numbers were viewed as scientific, since numbers were thought to imply objectivity and accuracy. Management theorists and gurus who dispensed this new wisdom ascended to the office once ascribed by Shelley to poets as “the unacknowledged legislators of mankind.”

Of course, such a narrow focus on bean counting has a way of dehumanizing the objectives of organizations—literally so in the case of the U.S. Armed Forces in the Vietnam War, during which “McNamara championed the metric of ‘body counts’ as a purportedly reliable index of American progress in winning the war.” Meanwhile, the morale of the soldiers on the ground, who often lost their lives searching for corpses to document the numbers, plummeted.

Yet, the cold analysis of inherently incomplete information that comes from metric fixation not only became the accepted standard for how businesses should be run, it seeped into every other corner of the professional world, as well. Muller breaks down not only why the now almost universally accepted standard of running nonprofits, schools, and government like a business is a terrible idea, but why the idea of running businesses in such a manner is suspect. At the same time pay-for-performance schemes were beginning to enter nonprofit and public service, all the psychological research being done on the value of extrinsic rewards versus intrinsic rewards in the workplace clearly favored the latter, and demonstrated that a focus on the extrinsic may “dampen (or even destroy),” in the words MIT organizational economics professor Robert Gibbons, “intrinsic motivation and social relations.” When it comes to business as a whole:

Profit surely matters. But so, in the long run, does reputation, market share, customer satisfaction, and employee morale, which makes it possible to adapt and to find solutions to new problems that will inevitably arise in the marketplace. In an economic world characterized by unpredictable change, there is a need for ongoing innovation, small and large, that is not readily reducible to a single performance target.

In fact, many of the financial performance targets that are set are often antithetical to innovation. It is more the work of calculating machines than managers. Muller documents the political reasons, deep-seated in the ideologies and history of the right and the left, that metric fixation has become so entrenched in government—along with the arguments against it by critics on both sides of the aisle.

There are, of course, also technological reasons that metric fixation has become so rooted in professional life. “The spreadsheet is a tool,” journalist Steven Levy all the way back in 1984, “but it is also a worldview—reality by the numbers.” Muller quotes him further:

You can’t really duplicate a business inside a computer, just aspects of a business. And since numbers are the strength of spreadsheets, the aspects that get emphasized are the ones easily embodied in numbers. Intangible factors aren’t so easily quantified.

Muller echoes contemporary critics by comparing “the measurable performance targets for schools, hospitals, police forces, and corporations” to Soviet central planning and output targets. 

And just as Soviet managers responded by producing shoddy goods that met the numerical targets set by their overlords, so do schools, police forces, and businesses find ways of fulfilling quotas with shoddy goods of their own: by graduating pupils with minimal skills, or downgrading grand theft to misdemeanor-level petty larceny, or opening dummy accounts for bank clients.

After turning an historian’s eye to the causes of the professional world’s metric fixation, he devotes a chapter to seven separate fields (Colleges and Universities, Schools, Medicine, Policing, The Military, Business and Finance, and Philanthropy and Foreign Aid) to explore some of the ways in which that fixation is currently harming the very outcomes they purport to be pursuing. And it’s not because metrics are inherently bad. In fact, he prefaces most chapters explaining the inherent promise of metrics in each field, documenting instances when they have been used to bring about real improvements. He also concludes the book with a chapter on “When and How to Use Metrics,” providing a checklist for successful performance measurement. So he is decidedly on the side of using metrics, so long as they’re properly aligned to real world goals and performance, rather than undermining them. His argument is how important it is to use metrics to inform judgment rather than replace it. As more business literature is reminding us to account for our unconscious bias, Muller reminds us that it is as important to guard against the predictable, characteristic distortions of metrics.

He sums some of those up in the chapter devoted to Business and Finance by referencing the work of two contemporary scholars:

Dan Cable and Freek Vermeulen of the London Business School recall many of the problems we have explored: the depressive effect of performance pay on creativity; the propensity to cook the books; the inevitable imperfections of measurement instruments; the difficulty of defining long-term performance; and the tendency for extrinsic motivation to crowd out intrinsic motivation. They’ve concluded that that it might be more advantageous to abolish pay-for-performance for top managers, and replace it with a higher fixed salary. They even suggest, rather heretically, that you might not want people motivated by primarily by extrinsic motivation at the head of your company: yet the more compensation is variable and linked to measured performance, the more likely that that will be precisely the sort of people you will get.

Freek Vermeulen, you may remember, is the author of Breaking Bad Habits: Defy Industry Norms and Reinvigorate Your Business, which we reviewed here last year. The “heretical” idea of breaking the pay-for-performance model of top executives is also explored at length in Steven Clifford’s CEO Machine, and echoes a larger question:

A question that ought to be asked is to what extent the culture of metrics—with its cost in employee time, morale, and initiative, and its promotion of short-termism—has itself contributed to economic stagnation?

After reading this book, I think you’ll most likely be convinced the answer is significantly. When you tie financial incentives or pay to performance measures and metrics, it is even more likely that people will game the system. And yet, The “principal-agent theory” that has come to dominate the corporate world since the 1970s, which favors the interest of shareholders and maximizing profitability above all else, has also led to a greater alignment of monetary rewards and performance metrics.

Because belief in its efficacy seems to outlast evidence that it frequently doesn’t work, metric fixation has elements of a cult. Studies that demonstrate its lack of effectiveness are either ignored, or met with the assertion that what is needed is more data and better measurement. Metric fixation, which aspires to imitate science, too often resembles faith.

I hope The Tyranny of Metrics is a book that changes the conversation, that can break through our fixation on metrics, and turn our focus back to the important work we all must do to improve the real world, and people’s lives in it. The argument is so well-constructed, so well tied together from one end of the book to the other, and so unbent by any particular ideology, that I wish I could just quote the book in its entirety for my review—like a map of the world that is life-sized. Barring that, I hope what I’ve written here inspires you to pick it up for yourself.