Home > Uncategorized > If you redefine “performance,” magical things happen

If you redefine “performance,” magical things happen

June 1, 2015

Crossposted on nakedcapitalism.com.

There is a study out, entitled The Best and The Rest: Revisiting the Norm of Normality of Individual Performance, written by two business school professors, that has been bothering me recently. I’ll explain why soon, but first a thought experiment.


Imagine a group of people competing for something. They’re all driven, talented people, who have put serious resources into getting good at this particular thing. They’ve also all had help of some form, and encouragement from their community to compete in this arena. At the very least they have to have deep confidence in their own abilities to even compete in this particular area.

At the end of the competition, that particular one, these people are ranked according to how they’ve done. By luck, by skill, depending on previous practice, resources, or direct support from external advisors, some of them have achieved impressively high rankings, while others, in spite of their hard work and efforts, are falling behind.

Next, there’s a community feedback element. This group of people are not done – they’re hoping to become famous worldwide, or at least in this arena, for being highly ranked, maybe even the best. And the community has direct influence on what happens next, in future rounds of competition. So, individuals can vote for certain people to win, or directly give them more time or resources to do so, or even help them in their next round.

In subsequent rounds, the ranking gets more defined and the community becomes increasingly certain that the winners deserve to be there and that they are truly fabulous at this particular skill, even though the original native differences in talent are not enormous. Luck, resources, and self-confidence were all important indicators of success in that first round, some just as important as native skill.

This continues for years. At retirement, the highly ranked individuals have produced a massive amount compared to the ones that did poorly in the early rounds. In fact, the distribution is highly skewed, and seems to serve as proof that the original ranking was warranted.


I didn’t specify what field the above story took place in, so let me suggest a few that might work. First, there’s the music industry. Lots of would-be rock stars vie to be the next Taylor Swift. Heck, even Taylor Swift vied, once upon a time, to become herself. Of course, it helped that she was able to persuade her wealthy parents to move to Nashville when she was 14 to pursue her career. And – not to say she isn’t talented, because she most definitely is – we all know that once you have a hit, your career is much more likely to go well after that, with contracts, money, support, and great musicians flocking to you.

Or, it could be academics. If you stand out as an undergrad, especially at the right college, you get into a good grad school, and if you have enough confidence, determination, and the good luck to get a nice thesis problem, you might have a thesis that stands out, which leads to NSF grants, reduced teaching loads, opportunities to speak at conferences, semesters off of teaching to pursue research, and a host of co-authors who are increasingly willing to do the work to write up joint results. Again, none of this happens without determination, drive, and talent, but it definitely happens more and faster with the help of a supportive community. It’s all about the feedback loop of success.

Or, here’s another arena: sales. If you are known as a successful salesman, if you have a slightly better reputation than the next salesperson, then you’ll get the dibs on the jobs in a typical organization. That means you can be choosy, and take the easy pickings, and pass over the harder jobs. Over time your likability and personal network grows, and you become the go-to person in the organization for success, partly because of your hard work ethic, but partly because of the way success breeds success.

Or how about basketball? All professional basketball players are amazingly good at what they do. How much better does one have to be to get more playing time? Which leads, of course, to more points, more double doubles, or what have you.


Now to the paper. It talks about the distribution of performance, and notes that in arenas above, performance, which they equate with output of songs for musicians, or papers for academics, or sales figures for salesmen, are distributed more as a power law probability distribution than as a bell curve. Of course, that is true, and I think we know why, from above. It even has a name: the Matthew Effect, which is even referred to in the paper, on page 112.

The primary goal of the paper is to make the case that “performance” is not normally distributed. It is distributed with a much fatter tail. They suggest using the Pareto distribution:


Before I go on, let me mention that their examples are restricted to researchers, entertainers, politicians, and amateur and professional athletes. They never mention secretaries, computer programmers, marketers, cashiers, or data analysts. In fact most of the people who work at regular jobs are completely excluded from this study.

So it’s really more accurate to say that the primary goal of the paper is to redefine the word “performance”. They switch from one definition to the other without explanation, so their studies on pro athletes somehow magically refer to average workers.

That brings us to the second goal of this paper. Namely, the conclusion that we should use this “performance isn’t normally distributed” rule to focus even more on elite actors.

Here’s one version of the elitism argument (page 108):

Leadership theories that avoid how best to manage elite workers will likely fail to influence the total productivity of the followers in a meaningful way. Thus, greater attention should be paid to the tremendous impact of the few vital individuals. Despite their small numbers, slight percentage increases in the output of top performers far outweigh moderate increases of the many. New theory is needed to address the identification and motivation of elite performers.

What’s particularly irksome is this kind of logic (page 112):

For selection, this means that there are real and important differences between the best candidate and the second best candidate. Superstars make or break an organization, and the ability to identify these elite performers will become even more of a necessity as the nature of work changes in the 21st century (Cascio & Aguinis, 2008b)

If you think back to our original thought experiment, there is actually very little difference between good candidates at the beginning. Second, this “we absolutely need to keep our talent” mentality is exactly the argument we see time and time again excusing pay raises for CEO’s. And now there’s a “mathematical” reason for it.

That brings us to the third and final goal of the paper, the “CEO pay is not exorbitant” argument, (page 112):

Likewise, compensation systems such as pay for performance and CEO compensation are an especially divisive issue, with many claiming that disproportionate pay is an indicator of unfair practices (Walsh, 2008). Such differences are seen as unfair because if performance is normally distributed then pay should be normally distributed as well.

Let me rephrase: since “performance” isn’t normally distributed, there’s no way pay should be either, when we define it for everyone. So let’s just go ahead and overpay CEO’s.

It might be a good moment to remind people that even in academics, the top performers don’t make 100 times what the lower performers get. Compare that to McDonalds, where the burger flippers would have to work 1 million hours to get one year of CEO pay.

In pop music and pro sports, there is a crazy pay differential, but that’s not something to be proud of or something we want to replicate.

Categories: Uncategorized
  1. June 1, 2015 at 7:40 am

    Yeah, to defend things as they are at this point in time, people often have to disregard the dynamics, or the cumulative effects of iterative processes that you describe. Andrew Gelman likes to quote Flynn on this: e.g. http://www.washingtonpost.com/blogs/monkey-cage/wp/2014/07/24/meritocracy-wont-happen-the-problems-with-the-ocracy/

    It occurred to me reading your post that people have a tendency to do something similar with standardized tests like SATs or IQ tests: taking them as “measurements” of some static property of people instead of considering the dynamics of the measurement process where instrument and subject are interacting in complex ways.


  2. Aaron Lercher
    June 1, 2015 at 11:25 am

    Never mind having a model of the phenomena one is studying, or being clear what is being studied or what about it is valuable, when something related to it is easy to count. (Such as money, citations, test scores, election results, sports scores, etc.)
    If one isn’t bothered by pesky stuff like what one is studying, how it works, what one is trying to learn about it, then it’s much easier to get papers written (or to make policy, if victims of the policy are politically weak).
    If one identifies a pattern that applies to everything, then everyone will want to talk about your analysis.


  3. Ed Seedhouse
    June 1, 2015 at 1:11 pm

    The competition you talk about at the top fits absolutely perfectly with the world of competitive tournament chess. The ranking system used is the Elo system, in which performance is presumed to vary normally about one’s “real” strength, or close to that. The renumeration can be huge (well, not Golf or Baseball huge, but millions a year) but only at the extreme top end.

    Being in the top 1% gets you a “master” title but virtually no money. Being a standard distribution better than 1% gets you an international title (generally “Grand Master”) and may allow you to earn a decent living playing the game or teaching it. The big money starts when you reach the top ten in the world which is a whole standard deviation higher than the merely internationaly titled players.

    Only the top two or three in the earn really good money, with the World Champion title being good enough to earn you a few million a year these days.


  4. Guest2
    June 1, 2015 at 4:17 pm

    This is the law of small numbers that Randall Collins has made famous using the analysis of social networks. See the first few chapters of his The Sociology of Philosophies. http://socresonline.org.uk/4/3/collins.html “Among the main elements of the ‘theory of intellectual change’ of the subtitle of the book is the ‘law of small numbers’ which operates in a ‘limited attention space’. This law stipulates that there can never be just one preeminent philosophical position which dominates the attention space – all positions need to define themselves in relation to rivals. On the other hand, there can rarely be more than six positions which are serious contenders, and this is because only a few positions can remain visible as they are carried over generations by means of making alliances. A further element of the theory are thus the intellectual networks, horizontal alliances between schools of thought and vertical ones between teachers and pupils which stretch over generations. The waxing and waning popularity of factions within these networks is only partly explained, according to Collins, by the material conditions of their organizational bases which have traditionally been invoked in the sociology of knowledge. Instead, they are mainly determined by the internal structural conditions of the field: ideas only get to the centre of the attention space by lining themselves up alongside dominant positions and pushing them further – or by opposing dominant positions in opportune conditions.”


    • Guest2
      June 1, 2015 at 4:35 pm

      Robert Merton, also a sociologist, identified the Matthew Effect, which is compatible with the law of small numbers because it described the benefits of having ties to “stars” in whatever field you care to name. Remuneration is a variable that depends on the particular network involved.

      What this is getting into is wealth stratification, and why Proudhon was so totally wrong about the benefits of the Industrial Revolution leading to leisure for everyone. The distribution of wealth is also controlled by networks, and their cultural elements.


  5. A. Wells
    June 1, 2015 at 4:21 pm

    … written by two business school professors. The whole puzzle of the wrongness is answered right there. I didn’t read the paper, I don’t have access, but from what is here about the talent vs pay distributions, can be easily answered.
    Q1: what happens to the organization if nobody is willing to work for less than [some cutoff, say] 10% of CEO’s pay.
    Q2: What if they quit? I assume the obvious answer – the organization collapses.
    Q3: if they raise the pay scale so they don’t have to quit? I assume that the organization again collapses as it can’t be profitable.
    Q3: If they adjust the pay scale so that the cutoff is satisfied and the company remains profitable. Obviously, this is possible short term. Long term obstacle is that the talent will leave the one company. Assuming, that the outcome is otherwise desirable, this could be easily dealt with income tax policy.
    My conclusion is that a social norm for pay per performance could be easily implemented. Is it desirable? There we are in religious territory.


  6. June 2, 2015 at 3:25 am

    Even looking at the data on their terms, while they do seem to care about power laws specifically, not just skewed distributions, they didn’t provide any comparison to, say, a lognormal. If the performance data fits a lognormal distribution, they couldn’t really make strong claims about how performance works against an alternative explanation of just needing to log transform their performance measure.

    Since Aaron Clauset & Cosma Shalizi have shown most claimed ‘power law’ examples fit a lognormal at least as well as a power law, and have provided free software to do the relevant tests, this isn’t just being picky.


  7. Funkygroovycool
    June 3, 2015 at 7:15 am

    Somehow i’m confused by all this. I agree with you that there is something fishy about the definition of performance. Actually, the article doesn’t seem to be measuring “performance” as much as it it measuring “success”, which is only a potential manifestation of performance. The feedback loop you describe also refers to success.

    So, success is not normally distributed. That’s not big news, really. It’s essentially a cultural choice going back to ancient Greece. We love role models and inspiring stories. Isn’t that largely the promise of the American dream, the “happy few” as Shakespeare called them?

    An interesting question – to me at least – is whether we want to, as a community, encourage and support those happy few, i.e. keep feeding the success loop. It would surely take quite some effort to break the loop, but to a large extent, the masses are choosing this. OK, maybe not in the case of the CEO’s pay, but that’s why we also need regulators.

    Another interesting question is how performance is linked to success. It’s probably hard to have any intrinsic measure of performance, and I expect the answer to be something along the lines of “performance (in the sense of raw talent) leads to success when combined with hard work ethics, self promotion capabilities, and a favorable environment (the feedback loop)”. You can probably make this more refine or even quantifiable, but am not sure there’s anything new here either.

    I wonder how this would work for artists: it would be really hard to have an objective definition of performance that is not linked to success. How many “artistic geniuses” have never been “understood” (read successful)? Actually, there’s a highly praised book written on this topic called “Blockbuster”, by… a harvard b-school teacher 😉 I should probably read it.


    • Aaron Lercher
      June 3, 2015 at 2:57 pm

      Very good point about “performance” versus “success.”
      I think this is yet another example of the distinction between process and outcome, which are easy to mix up.
      One might perform well in many different ways, and yet never have success in getting a good outcome, let alone in an outcome that happens to be measured.
      Or one might have a lucky success in a measured outcome, without performing well much of the time one engages in an activity.
      But in a world of continuous measurement of everything humans do, it is easy to imagine that every form of good performance has a corresponding successful outcome, and that this success is measured.


  8. Guest2
    June 4, 2015 at 11:06 am

    Continuous measurement is itself an outcome of networks, and locally determined (i.e., never continuous everywhere). Social network topologies make it impossible to imagine good performances and successes in any way other than being local affairs, with local metrics.


  9. Otter
    June 6, 2015 at 6:53 am

    Consider the sixties when (to grossly simplify) every other young man and quite a few young women had guitars; and there were coffeehouses or park benches sprinkled throughout the city. There was an explosion of creativity, providing modest, often merely supplemental, income to many.

    Record corporations, and stadium corporations, and megaconcert corporations, gathered the “best” musicians to themselves. Along with the millions of fans. Along with the millions of fans modest budgets. Rents were monopolized by a few musicians and corporations, and creativity was starved out.

    A few “high performers” were rewarded. A few parasites were rewarded. Most regular people were completely excluded.

    Hockey players. Dancers. Story tellers. …


  1. No trackbacks yet.
Comments are closed.
%d bloggers like this: