More Proof that the Dunning-Kruger Effect Is Fake

Only an ignorant person would use the Dunning-Kruger Effect
to shame you, says a mathematician.

We reported on 25 Sept 2021 about a psychologist, Andrew Danvers, who debunked the Dunning-Kruger Effect as bad science. The Dunning-Kruger Effect is frequently touted as a meme in social media to shame people. The effect can be stated various ways:

• The least competent people are the most proud of their knowledge or ability.
• Confidence in a subject is inversely proportional to knowledge about it.
• If you are really, really stupid, then it’s impossible for you to know you are really, really stupid.

A common graph used to illustrate the Dunning-Kruger Effect is based on bad science.

You get the idea; “dumb people are clueless but proud of their ignorance.” That’s the psychological “law” David Dunning and Justin Kruger supposedly established in the 1990s.

Danvers showed that it was bad science, and an illustration of the “replication crisis” in psychology. Now, a mathematician from Bowdoin College in Brunswick, Maine, shows that it is bad mathematics, too. Dunning and Kruger were the clueless ones.

Debunking the Dunning-Kruger effect – the least skilled people know how much they don’t know, but everyone thinks they are better than average  (The Conversation, 8 May 2023). Eric C. Gaze, a mathematics professor at Bowdoin, confirms one part of the story: like the folks in Lake Wobegon where “all the children are above average,” most people rate themselves better than average, which is mathematically absurd. So yes, pride is a common human foible. In general, “overestimation is pervasive across many skills – including logic tests,” he says. “But it is mathematically impossible for most people to be better than average at a certain task.”

Most importantly, the key claim of the Dunning-Kruger Effect—the notion that people are clueless about their ignorance—came from a flawed use of statistics. The mathematician gives three reasons for the error:

• Those with the worst test scores rated themselves more highly in D-K’s experiment because they were farthest from a perfect score; this is a logical truism, not a finding.
• Since everyone assumes they are better than average, it is not surprising that the least skilled also did.
• Those with the lowest scores did not overestimate their ability on the specific test.

Part One

To establish his debunking, Gaze showed that he got the same results when he used Dunning and Kruger’s own procedure on an arbitrary set of 1,154 imaginary test takers assigned random scores.

Then, just as Dunning and Kruger did, we divided these fake people into quarters based on their test scores. Because the self-assessment rankings were also randomly assigned a score from 1 to 100, each quarter will revert to the mean of 50. By definition, the bottom quarter will outperform only 12.5% of participants on average, but from the random assignment of self-assessment scores they will consider themselves better than 50% of test-takers. This gives an overestimation of 37.5 percentage points without any humans involved.

“Reversion to the mean” guarantees that those farthest from 50 will automatically show up as more significant than those closest to the average, no matter what population (human or imaginary) is used in the analysis. This supports what Danvers also said about the fake science; “the effects they supposedly measured did not differ from randomness,” we reported about his work.

Part Two

Then, Gaze debunked the claim that “that the least skilled can adequately judge their own skill.” This required some new empirical work.

My colleague Ed Nuhfer and his team gave students a 25-question scientific literacy test. After answering each question, the students would rate their own performance on each question as either “nailed it,” “not sure” or “no idea.”

Working with Nuhfer, we found that unskilled students are pretty good at estimating their own competence. In this study of unskilled students who scored in the bottom quarter, only 16.5% significantly overestimated their abilities. And, it turns out, 3.9% significantly underestimated their score. That means nearly 80% of unskilled students were fairly good at estimating their real ability – a far cry from the idea put forth by Dunning and Kruger that the unskilled consistently overestimate their skills.

Conclusions

The upshot is that this widely popular meme is unsupportable scientifically and mathematically.

The original paper by Dunning and Kruger starts with the quote: “It is one of the essential features of incompetence that the person so inflicted is incapable of knowing that they are incompetent.” This idea has spread far and wide through both scientific literature and pop culture alike. But according to the work of my colleagues and me, the reality is that very few people are truly unskilled and unaware.

What is the take-away lesson from these debunkings of fake science? Ironically, it’s those who repeat the Dunning-Kruger effect to shame people who are the least competent to judge its truth. “To claim otherwise suggests, incorrectly, that much of the population is hopelessly ignorant.”

Over the years, we have seen this Dunning-Kruger meme thrown at us on Twitter by atheists who claim “You just don’t understand evolution” or “You don’t understand science.” Those same people often did not read the article they criticize, displaying their own ignorance.

Incompetence and arrogance are not the sole domain of ignorant people. As we said in 2021, “In our experience, the opposite of Dunning-Kruger is often observed: the best experts in the world on Darwinian evolution are often the worst dogmatic blowhards.”

Have you had anyone repeat the Dunning-Kruger Effect, or use it on you? Tell us about your experience in the comments.