How long before you find out if you’re really good, or you’re really lucky?

Cole Trickle, Days of Thunder (1990)

Is Elon Musk a genius or an idiot?

Michael Hiltzik, Los Angeles Times, 7 November 2022

These two quotations, one from a movie character played by Tom Cruise, the other from a newspaper business columnist, are a good introduction to my topic for this post—a paper by Keuschnigg, van de Rijt and Bol, published in the European Sociological Review in January 2023.

What concerns Keuschnigg and colleagues is whether people who command high wages and social prestige are necessarily smarter than the mere mortals around them. Are they actually really good at their jobs, or are they just really lucky, in other words.

Both quotations come with a false dichotomy, though—Cole Trickle might be quite good and quite lucky, for example; and Elon Musk might be less bright than some imagine, without being an actual idiot. What Keuschnigg et al. have done is come up with some evidence that helps us think about the interaction between intelligence and good fortune in the real world.

They build on some theoretical work by Denrell and Liu in a paper opaquely entitled “Top performers are not the most impressive when extreme performance indicates unreliability” published in Proceedings of the National Academy of Sciences in 2012. What they were doing makes more sense when you read their abstract:

The relationship between performance and ability is a central concern in the social sciences: Are the most successful much more able than others, and are failures unskilled? Prior research has shown that noise and self-reinforcing dynamics make performance unpredictable and lead to a weak association between ability and performance. Here we show that the same mechanisms that generate unpredictability imply that extreme performances can be relatively uninformative about ability. As a result, the highest performers may not have the highest expected ability and should not be imitated or praised. We show that whether higher performance indicates higher ability depends on whether extreme performance could be achieved by skill or requires luck.

To explore this relationship between performance, skill and luck, Denrell and Liu came up with a couple of simple mathematical models, one focussing on “self-reinforcing dynamics” and one on “noise”. In each model, a bunch of abstract agents are assigned a particular level of “skill” which, in the absence of extraneous influences would result in a corresponding level of “success” or “performance”. These quantities in quotes are all represented by abstract numerical values. “Skill” is not uniformly distributed—any given agent is more likely to have near-average skill than particularly high or particularly low skill. Other things being equal, that would make outstanding or abysmal performance correspondingly rare.

First, Denrell and Liu explore the effect on this toy model of “self-reinforcing dynamics”. What if an initial success makes an agent more likely to succeed again in a subsequent trial? And likewise for failure. This sort of thing happens in real life when, for instance, consumers perceive a particular product from a particular company as being desirable, or when luck in landing a particular job means that an individual gets on to a “career ladder” denied to others. This is sometimes called the Matthew Effect, from the Biblical verse Matthew 13:12:

For whosoever hath, to him shall be given, and he shall have more abundance: but whosoever hath not, from him shall be taken away even that he hath.

So Denrell and Liu take their mathematical agents and put them through repeated trials which will resulting in success or failure according to the agent’s skill, but with a bias added according to the agent’s successes or failures in previous trails. Without the Matthew Effect, the relationship between skill and number of successes is linear, and success is therefore a good proxy for skill. But if we wind up the weighting associated with previous successes and failures, things start to look a bit different. Extreme success is associated with a lower level of skill (on average) than more moderate success; and likewise, in reverse, for extreme failures.

So what’s going on? Well, extremely skilled agents are still scoring a lot of successes, but the Matthew Effect can’t improve their already excellent performance by much. And these extremely skilled agents are rare (as they are in real life). So they find themselves competing with moderately skilled agents who have benefited from the Matthew Effect to reach higher scores than they would otherwise achieve. And since moderately skilled agents are much more common than extremely skilled agents (as is the case in real life), they come to dominate at the high-success end of the scale. When we look at very successful agents, then, we are more likely to find one that’s moderately skilled but lucky, rather than one that is extremely skilled. (And you can of course reverse the whole argument at the low-success end of the scale.)

The authors go through the process again with “noise” (which is just a random variation superimposed on the performance of agents) and find similar results. When noise is high, we find that agents with excellent skills still have good performance, but are again swamped by moderately skilled agents who’ve simply had a lot of a luck. And these extremely skilled agents are particularly penalized by noise, because the noise has minimal scope to improve their performance, and so will preferentially degrade the average performance of highly skilled agents. The result is again an erosion of the correlation between performance and skill—the greater the noise, the less we can judge skill by looking at performance.

(In graph B, noise is small, and performance is a good indicator of skill; in graph A, with higher noise, it becomes difficult to pick out skilled agents on the basis of performance alone.)

As the authors summarize:

Noise and [the Matthew Effect] not only introduce unpredictability but also change how much one can learn from extreme performances and whether higher performance indicates higher skill. In particular, we show that when noise and [the Matthew Effect] can strongly influence performance, extreme performances can be relatively uninformative about skill. As a result, higher performance may not indicate higher skill. The highest performers may not be the most skilled and the lowest performers may not be the least skilled. The implication is that one should not imitate the highest performers nor dismiss the worst performers. More generally, we show that whether higher performance indicates higher skill depends on whether extreme performance could be achieved by skill or requires luck.

So that’s all well and good—an entertaining little model that looks like it might have some connection to the real world. But is it actually relevant? Are skilled agents in the real world significantly affected by noise and the Matthew Effect, to the extent that the connection between performance and skill becomes unreliable?

Enter Keuschnigg et al.:

Are the best-paying jobs with the highest prestige done by individuals of great intelligence? Past studies find job success to increase with cognitive ability, but do not examine how, conversely, ability varies with job success. Stratification theories suggest that social background and cumulative advantage dominate cognitive ability as determinants of high occupational success.

And later:

Elite jobs are of special interest, for two reasons. First, income distributions have strong right skew. In all Western countries, top income shares have been steadily rising since the 1980s, with the 1 per cent highest earners receiving 9 per cent of national income in Sweden and even 20 per cent in the United States—excluding capital gains. This extremity of top incomes as well as their public salience render it crucial that they be earned by very capable individuals. Second, those with the most prestigious jobs wield the greatest economic and political power, and the intelligence of their decisions is consequential.

And:

Our argument draws on the role of two key non-meritorious determinants of occupational success: Family resources
and luck. The class- and network-advantages of those with elite family backgrounds are assumed instrumental for gaining access to the most privileged and best-paying jobs. Second, rich-get-richer processes are assumed to allow inequalities in job success to grow between those who got a lucky break early in the career and those who did not.

Don’t you just love the phrase “non-meritorious determinants of occupational success”?

There then follows a review of the literature concerning these non-meritorious determinants, confirming that, yup, if you come from a rich and/or influential family, the evidence overwhelmingly suggests that you gain advantages in education and job opportunities.

So it seems that Denrell and Liu’s mathematical model of luck and the Matthew Effect might be applicable in the real world to the relationship between one’s cognitive ability and one’s job. Perhaps extremely well-rewarded and socially prestigious jobs are occupied by rare individuals with exceptional cognitive abilities, plus a whole bunch of moderately able but lucky people.

But how could we find out? Well, we could go to Sweden, where mandatory military service for men persisted until 2010, and all those men underwent a standardized test of cognitive ability. And Sweden being a country with proper joined-up data about its citizens, those cognitive tests can be tied to later average earnings during an 11-year window centred on the age of forty, as well as to a registered job description. The job description can in turn be tied to a standard measure of a job’s social prestige, called the International Socio-Economic Index scale. Who knew such a thing existed? Not I. But you can find the original paper describing the scale here (download the pdf using the button at top right).

So the researchers ended up with three data points for each of 49,000 Swedish men: a cognitive test score (from something similar to the Armed Forces Qualification Test used by the US military), an average wage in later life (inflation adjusted), and the ISEI rating of their registered occupation.

Plotting earnings against cognitive score produced this:

It looks remarkably similar to the output of Denrell and Liu’s models, particularly if we break down the earnings data into centiles (right), rather than using the raw numbers (left). We can see that cognitive ability plateaus above earnings of about 600,000 Swedish kronor (about €60,000), and in fact takes a slight downturn at the extreme. Likewise, those with the lowest earnings turn out not to have the lowest cognitive abilities, on average. But in the broad middle range of earnings, financial reward is responsive to cognitive ability.

The results for job prestige show a lot of scatter in cognitive ability at the top and bottom ends of the scale, producing something of a blurry plateau for prestige less than 30 or over 70.

Checking the tables in the ISEI paper I link to above, this suggests there’s not much to cognitively pick and choose between life scientists, medical doctors, dentists, mathematicians, lawyers, judges, teachers in higher education, heads of government … and so on. Likewise, a bunch of varied low-prestige jobs show no corresponding variation in cognitive ability.

There are, of course, limitations to this study—it deals only with men (mandatory military service for women was introduced only in 2010, the year in which military conscription ended up being mothballed, to be reintroduced in a more limited manner in 2017); it deals only with Sweden, where extreme earnings are rarer than in some other countries; and it addresses only cognitive ability assessed by a particular test, while neglecting other potentially important variables like motivation, social skills and creativity. The authors acknowledge these limitations and (as is customary) tell us that more research is needed. But their conclusion seems important:

Recent years have seen much academic and public discussion of rising inequality. In debates about interventions against large wage discrepancies, a common defence of top earners is the superior merit inferred from their job-market success using human capital arguments. However, along an important dimension of merit—cognitive ability—we find no evidence that those with top jobs that pay extraordinary wages are more deserving than those who earn only half those wages. The main takeaway of our analysis is thus the identification, both theoretically and empirically, of two regimes of stratification in the labour market. The bulk of citizens earn normal salaries that are clearly responsive to individual cognitive capabilities. Above a threshold level of wage, cognitive-ability levels are above average but play no role in differentiating wages.