To Dunning-Kruger and Beyond

When you go through graduate school, you get to know a lot of doctors. There are the professors, of course, but also your fellow students. From your first year in the lab, you start seeing coworkers make the magical transition. I think magical best describes it because it's not anything you can see or measure. In fact, it's not much of a transition at all. You work for years on learning your field and pushing the boundaries of knowledge. Then you write down what you've learned, and go to a high stress meeting. After that, they mail you a piece of paper that your mother insists on framing.

There are a lot of lessons that come out of graduate school. Here's one.

The more somebody insists that you call them "doctor," the less they probably deserve the title.

There is a cognitive bias known as the Dunning-Kruger effect, in which people who don't know that much about a topic think they know a lot more. It's one of things that seems obvious in retrospect. When you don't know that much, you don't know what you don't know. As you learn more, you become more aware of where you need work. That's why experts might hem and haw about whether they can do a certain project in a given amount of time, while the novice will say "I'll bang that out in no time!" A perfect example is how long it took me to "just make my own chart" below.

In the studies, they brought in students and asked them to rate their own abilities on things like humor, grammar, and logic. Then they tested them. Those at the bottom of the curve for ability pretty consistently thought they were near the top. Giving the poorer students tutorials increased their actually ability, but decreased their self reports of their ability.

It's a neat set of studies. But they don't go far enough.

I've started to notice an after-effect. Imagine if instead of taking the students more or less at random and then testing them, they announced a school-wide contest in some subject or skill set with words in the poster like "only the best need enter." Maybe there could be verbiage to indicate that this will take place in public, or be filmed and posted online. All the world will see your success, or your utter failure in perpetuity.

Consider what the Dunning-Kruger effect says about who's going to even bother to enter such a contest. The best students are going to stay away, fearing permanent embarrassment. If you wanted to find the most accomplished students, you've probably guaranteed that it's not going to happen.

"We only accept 2% of our applicants" might mean <1% are qualified.

"We only accept 2% of our applicants" might mean <1% are qualified.

I'm pretty sure I've seen this effect in action. I've learned to grow suspicious of degrees from BIG NAME schools that are known for being extremely exclusive. They're only exclusive of the people that apply, and their reputation is probably filtering out the best.

I've worked beside folks who were among the most accomplished and knowledgable at their subjects, but who said "I could never get a PhD." I've also worked alongside alums from BIG NAME engineering schools who I eventually learned (the hard way) to never leave alone with sensitive instrumentation.

As we enter the new presidential election season, you'll probably be thinking about this phenomenon a lot.