Rendered at 11:02:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
brlewis 2 days ago [-]
The key claim: "Random data actually mimics the effect really well."
This makes some sense. If people are asked to guess a number between 1 and 6 and then roll a die, the people who roll low are more likely to overestimate and the people who roll high are more likely to underestimate. But the key is precisely how well random data mimics the effect.
Jensson 2 days ago [-]
No, the argument is that the best person cannot overestimate his own rank, and the worst person cannot underestimate it. The better you are the less room there is for you to overestimate your skill, second place can at most be off by one etc.
This effect would disappear almost completely if they instead of estimating their rank they estimated their score, since then unless the test is so easy the best scores perfectly there will be a lot of room for everyone to overestimate and underestimate themselves.
But as is when the top 10% all estimate themselves to be in the top 10%, you will say they are underestimate themselves since on average the top 10% are in the top 5%. At the same time if the bottom 10% say they are in the bottom 10%, you will say they overestimate themselves since actually on average they are bottom 5%. But both these groups were making the same mistake, and its impossible for that not to happen unless everyone is perfect.
brlewis 1 days ago [-]
>This effect would disappear almost completely if they instead of estimating their rank they estimated their score
In the article, in the section "The effect is in the noise", just before the graphs, it says they estimated their score. Where did you get that they estimated their rank?
5555watch 2 days ago [-]
They're simulating randomness incorrectly: relationship between true and perceived will average 0.5, not 0; and bias will average 50%, not 0%. That's why their "random data" is sloped.
Add negative relationship and negative bias, and the random data will act as intended - hovering randomly around 50%.
speerer 1 days ago [-]
Is this correct? I was fairly certain that the correlation between random data points approached 0.
Is it also plausible that the relationship between true and percieved is wholly independent (as this presumes)?
They're sampling slope and bias for the line from U[0,1] and U[0,100], the expected values of which will be 0.5 and 50. That's why they get what they get.
They say that randomness is any positively sloped line with any positive bias. So they assume dependence, just a very high variance of it. It's incorrect as they miss the negative half of the parameters - it would then correctly yield zeroes, for the presumed independence.
21 hours ago [-]
algoth1 2 days ago [-]
They, themselves, show the Dunning-Kruger effect by overestimating how much they understand randomness
oulipo 2 days ago [-]
Indeed. And the obvious reason is that when you simulate the "self-assessment" using a Gaussian noise around the "actual intelligence", and clamp it to [0, 100] so that it doesn't go "out of bound" (eg "negative intelligence" is not allowed), you will necessarily skew the low scores upward and the high scores downwards.
But it's not because "some statistical model exhibit a bias that's similar to the result" that this implies "therefore the result is a statistical error"... that's a backward reasonning
MBCook 2 days ago [-]
Even if it isn’t true, it’s got the feeling of truthiness (1).
I don’t expect it to ever go out of the public consciousness. Like other things that were never real like Stockholm Syndrome I suspect it’s just stuck in the zeitgeist now.
Power poses, imposter syndrome, Stanford prison experiment, marshmallow test...
It seems like the overlap between "real psychological effect" and "subtle enough that it requires research to discover" is vanishingly small. I guess that's not really surprising.
The reason people find the DK effect persuasive is that everyone has run into this many times in their own lives. Like with an idiot boss who thinks he knows your job better than you do, or an article by someone that gets things completely wrong about a topic that you yourself know very well, through personal experience or extensive study. And we have all been dismayed about the impact such people can have, like through their decisions or influencing each other.
If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
pibaker 1 days ago [-]
We want theories to be backed by more rigorous data precisely because "having run into this many times in their own lives" is not reliable. People's own perception is incredibly biased. We can see a bias at play in your very own closing sentence — "your study is bad if it doesn’t not confirm my existing belief" is plainly not science.
belviewreview 12 hours ago [-]
I was explaining why people believe it. I think they are often correct in their belief. So let me ask you, have you experienced this yourself, like for a subject that you have a good knowledge of, observed someone who claims they do but is wildly off track?
dolni 19 hours ago [-]
It's not science, but also a lot of stuff that purports to be science is not. "Science" is subject to many of the same forces that politics is.
Using your personal experiences and the personal experiences of people you trust as an anchor is a good thing. It keeps you from falling for propaganda. Note that this DOES NOT mean studies conflicting with your personal experience are all wrong.
The tricky part here is understanding the limit of what your own experience can tell you.
jeffnash 1 days ago [-]
Related: The Gell-Mann Amnesia Effect. In Michael Crichton's own words:
"Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know."
specialist 22 hours ago [-]
Michael Crichton also was an outspoken skeptic of climate crisis.
Crichton's Law: unwittingly serving as the best example of one's own snarky law.
klustregrif 20 hours ago [-]
> If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
It’s the opposite problem. The research is bad because it “demonstrates the DK effect” even for cases where it cannot possibly be true. Like “people who tend to roll lower dice rolls overestimate their own ability to roll high dice values and people who tend to roll higher values tend to underestimate their own ability”
Which is just obviously incorrect and the true result should have been “there’s no correlation between people’s estimate of their ability to roll dice and their actual outcomes”
If you want to demonstrate an actual effect like DK, you have to set up an experiment such the if it truly doesn’t exist then the experiment tells you so. A great way of checking this is to throw random data at it in which case it should come out saying “I saw no correlation” but that’s no the case with their setup.
rootusrootus 1 days ago [-]
Maybe the problem is that DK isn't about individuals, but about topics? I.e. perhaps everyone exhibits the phenomenon about topics they are most ignorant about.
(and I am ignorant about DK, so...)
PeterHolzwarth 1 days ago [-]
No, I think it's about people - about us. About you, me, all of us.
qsera 1 days ago [-]
What is amusing to me is when I think about what happens when the so called "experts" have this effect? Because "experts" are people too...
"about all of us when it comes to things we are not very competent at", according to the article.
icepush 1 days ago [-]
We are the answer.
boxed 1 days ago [-]
That was always what the DK effect was about.
fasterik 1 days ago [-]
>If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
I don't have an opinion about Dunning-Kruger either way, but I wanted to point out that this is a pre-replication crisis mindset. Social psychology spent decades producing catchy, intuitive results that turned out not to replicate. We've learned over and over that we can't trust it when a result feels right, we need to rely on proven statistical methods.
qsera 1 days ago [-]
>If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
Why do the research if you are so convinced from your own personal anecdotes? Quite a weird thing to say...
namlem 1 days ago [-]
Confirmation bias. You don't notice when people back down from their bad takes because that's normal and expected. Or possibly just selection bias if you are in situations where people are more likely to be arrogant.
usef- 1 days ago [-]
Your anecdote is not describing dunning-kruger, though, it's describing "some people exist in society with too much confidence in their ability" which can be true on its own, and is not exclusive to beginners.
nickysielicki 1 days ago [-]
> If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
What? This is anecdotal and subjective. If the research doesn’t back up those anecdotes, then the research doesn’t back up those anecdotes. You want the research to back up your lived experience because it’s validating. That doesn’t mean that’s the actual reality.
datakan 2 days ago [-]
Replication crisis. More than half of all psychology studies are not reproducible.
I'm at the point honestly, where I don't even consider psychology to be a science anymore.
vehemenz 2 days ago [-]
I wouldn’t assume the cause of the replication crisis is 100% due to a lack of rigor in psychology and the other relevant fields. These fields deal with concepts and phenomena that are often abstract and difficult to measure. In some sense, it’s just harder.
Besides, deciding what is and isn’t science is a question for the philosophy of science, not science itself.
glial 2 days ago [-]
Yes, that's why psychology is the hardest science.
Jensson 2 days ago [-]
Hard in "hard science" doesn't mean hard as in difficulty, it means hard as in not soft. Soft sciences are difficult to explore since they aren't rigid, they move around as you prod at them etc, you can't get a good grasp of its shape since they are so soft.
Physics on the other hand is hard as in unyielding. It is easy to figure out boundaries of physics and map out what is and isn't true, and the few cases were we made a mistake everyone can agree a mistake was made and that formulas needs to be updated since physics is so extremely hard that even a tiny error will get noticed.
Hope that clears it up, physics isn't hard as in difficult, its hard as in rigid. And psychology is soft, not easy.
So, your statement doesn't make sense at all in this discussion, they just said psychology has soft traits, and then you say "ok, so its hard since its soft!". No, soft is difficult, not hard.
So for example, physics is like describing the shape of a metal spoon, and psychology is like describing the shape of a pillow. You can see how describing the shape of the pillow is massively more difficult, because its not fixed, so you have to come up with a language to describe all the ways it can deform and how that would work.
glitchc 20 hours ago [-]
"Hard" in hard science means rigorous methodology and experimentally verifiable predictions. Psychology achieves neither, ergo it is a soft science, or more precisely, opinion masquerading as fact.
zajio1am 2 days ago [-]
The replication crisis is not limited to psychology, that was just the first area where it was noted. I saw some meta-study comparing multiple social sciences and according to it replication failure in sociology and educational science was even worse than in psychology.
anon48293 2 days ago [-]
Unfortunately it also is present in medicine, chemistry and increasingly physics.
majorchord 2 days ago [-]
> More than half of all psychology studies are not reproducible.
Do you have a source for this?
datakan 2 days ago [-]
There are tons going back to around 2015. Search Replication Crisis and Psychology. Only about 36-39% of studies can be replicated.
rawgabbit 1 days ago [-]
They did “replicate” the original Dunning Kruger. They replicated it using artificial data and then claimed their artificially generated data invalidated the hypothesis.
danielmarkbruce 2 days ago [-]
If you ever took psych 101, it was immediately obvious it's mostly horse shit. Made up nonsense theory, with studies that have few participants and are done by people who don't know math.
spidersouris 2 days ago [-]
It is both amazing and saddening to me how you can have such an arrogant view of a whole field with just taking a 101 course.
Jensson 2 days ago [-]
Psych 101 says stuff like people being more willing to accept a date with a person after a scary experience since they mistake their quick beating heart for love, rather than humans sees a scary experience as a stronger signal of bonding than a casual experience.
There is so much simple tests that they draw extreme conclusions from in the very first lectures you hear, and when they teach those things as true when its highly debatable its hard to not throw the entire field in the trash.
Of course there is probably some good psychology work done, but you shouldn't have so many shoddy examples in the first course then since that gives people a bad taste for the entire field.
danielmarkbruce 2 days ago [-]
Take physics 101 (or something, anything from a hard science) and understand how they know what they claim to know, and compare it to psych. It's night and day.
One doesn't need to continue learning a topic to know it's bs when it's bs right from the start.
lern_too_spel 2 days ago [-]
A 101 course should not teach frontiers of research. It should have fundamental established theories upon which the rest of the field is based. My experience with introductory psychology textbooks is that they spend a lot of time discussing ideas discarded decades ago, and then they spend some time discussing ideas that were discarded after the textbook was written. There is typically one poorly explained chapter on neuroscience that the authors summarized from another textbook.
anon48293 2 days ago [-]
I did 2 psychology degrees and completely agree with his assessment.
Sorry, most psychological research is sketchy at best, if not blatantly fraudulent
teamonkey 2 days ago [-]
In a thread talking about the Dunning-Kruger effect, no less
SubmarineClub 2 days ago [-]
It’s pretty handy as a signal. The moment someone starts banging on about Jungian this and Freud that, I know immediately to discount anything they say, ever.
PaulKeeble 1 days ago [-]
The overwhelming majority doesn't survive 20 years, when someone tries to replicate it the results disappears. Some of it takes longer than that but it almost all falls eventually. Its just not a rigorous field and alas that doesn't stop the newspapers printing sensational headlines about it claiming effects that are a bit of a stretch given how they actually studied the participants.
hoherd 2 days ago [-]
But the answer to bad science is good science.
danielmarkbruce 2 days ago [-]
I don't know that it's possible in psych. The economics of doing larger scale studies make it extremely difficult.
datakan 2 days ago [-]
A lot of them contradict each other too which is what got me going down the rabbit hole of researching it.
grebc 2 days ago [-]
It’s mostly just window dressing on blaming your parents.
_0ffh 1 days ago [-]
Damn, I remember I worked through the whole damn thing some years ago when I read this article for the first time, and it didn't hold up to it's claim. I was actually contemplating to do a write-up, but it wasn't really important to me so I didn't. Seeing the same article re-appear on HN for the second time now I really wish that I did. Unfortunately I've forgotten the details by now.
audreyfei 1 days ago [-]
If you ever do, post it! I'm sure it would be an interesting read
It's an interesting read. Curiously, it doesn't really debunk anything.
The fact that given X and Y random and independent, that Y-X is correlated with X doesn't disprove the Dunning Kruger. It in fact proves that Y = 1 X is a poor predictor, and the true model is Y = 0 X. In other words, perceived ability (of the human) cannot predict the actual test scores. Which is exactly what DK claims, but to a very extreme effect.
Note that, if there is actual signal (plus noise), e.g., if Y = X + eps; so the actual score is exactly the perceived score plus some added variation, the (Y-X)~X will be uncorrelated. In such case, there will be no DK effect, because the users are good at predicting their actual test scores, plus some constant variation.
henrycoler 8 hours ago [-]
Are you having credit issues?? I will advise you all to get to thechoosenhacklord He's an expert in increasing and repairing of credit scores and negative collections. He helped me increase my credit score from 645 to 815 and i am so grateful to him because I've been scammed twice before getting to know him and i can prove to you that he's a legit good hacker because he did came through and he helped me with the little i have. He's services is affordable, fast and reliable. contact him at thechoosenhacklord@outlook Don't get ripped anymore by the fake hackers. Get to him so you can get your problems solved.
5555watch 2 days ago [-]
Very hard to understand the meat behind all the fluff of the article, especially as the simulation code is not available, and as the presented simulated and original graphs are effectively the same (I don't see a disagreement).
It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.
And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.
card_zero 2 days ago [-]
Huh? The point is that the two graphs come out looking the same, making the original no more meaningful than random.
Now it's much more clear. The simulated data tries generating the true relationship between actual and perceived scores from 0.0 to 1.0, and bias in self-reporting from 0% to 100%.
So the output graph should be the average of all these data generating processes, yielding perceived relationship around 0.5 and bias around 50%, with some high variation.
If you have the access, run their Shiny code with these values, and you will see the published plot.
I'd argue that this demonstration is much weaker than "making original no more meaningful than random". It's more that the "simulated 50% bias and 0.5 true correlation looks similar to what DK published", which is also far fetched given the data generation they did.
Note: true random (what they were going for) would cover negative relationships, yielding the random true relationship around 0; and if they wouldn't correct the sign of Bias, it would also average at around 0; yielding a realistic "random" with the slope hovering about 50% for any percentile.
rawgabbit 1 days ago [-]
Line 272 of SimDK.R is where they produced the quartiles for top and bottom performers. But their code for generating both is exactly the same.
Except "two graphs look the same" in no way means "therefore the results are equivalent"
The article is baseless and fluffy
jszymborski 2 days ago [-]
It's annoying they didn't plot the new and old "perceived" curves on the same figure, but if you pay attention to the y-axis, there is a very big difference.
In the old plot, the bottom quartile has about a 50 percentage point margin between actual and perceived performance while the new one is 30 percentage points, which is a 50% difference between the old and new curve. The second quartile has 3x more margin in the old version relative to the new one.
5555watch 2 days ago [-]
So what does that change? If there's no error bars on the graphs you can't discuss significant differences easily. And if they do differ, plotting differences will yield the U shape curve.
jszymborski 1 days ago [-]
If the error bars are so large that the old and new analyses aren't statistically different, then the margins are too uncertain to make any claim in the original work... which is sorta the claim that the new work is making.
unwise-exe 1 days ago [-]
The chart they show from the original paper is approximately just:
perceived_ability = 0.5 + (actual_ability / 4)
(With the caveat that "ability" is a percentile rather than some absolute measure.)
That's not the negative correlation that it's often presented as in discussions. And it could as easily relate to mis-measuring everyone else's skill as to mis-measuring your own skill.
program_whiz 23 hours ago [-]
Interesting, maybe the fact that it is replicated by random noise is actually confirming the theory though. The article claims "completely random guessing replicates the DK graph". But that is compatible with "the ability to assess one's skill is not correlated with measured skill". If random guesses reproduce DK, it implies that people might be randomly guessing at their level of skill.
As a result people with little knowledge over-estimate their ability because a random dice roll will tend to be higher than their low ability (and the converse for people of high ability).
If you look at the DK graph, its close to random, maybe a very slight positive slope -- but ultimately the DK theory would be even stronger if it was truly random, because it means people have NO ability to assess their skill (0 correlation). I think the claim in DK is slightly weaker, that people at least have some ability, just not as much as we like to think.
klustregrif 20 hours ago [-]
So your claim is that if there is no correlation between A and B then it proves that there is in fact negative correlation between A and B because there is no correlation?
And so the Dunning Kruger effect becomes true of all things that are uncorrelated?
program_whiz 17 hours ago [-]
And it would only be true of things that are uncorrelated with an ascending sequence. As an example, I hand 4 people envelopes filled with a random amount of cash. Each looks at the cash and I ask what rank they think they are (1 to 4, 1 being least cash received). Since its a random guess, it will look like "people with the most cash underestimate their rank" and people with the least cash "overestimate their rank". The truth is, they have no information, they are just guessing. The rank and guess are uncorrelated, but it shows the DK effect (but its not because "people with more money tend to estimate their rank lower", its due to randomness). You'd probably see something like this:
And of course it would vary based on cash. If you hand someone $1M maybe they guess correctly they are the highest, but if everyone has $50-$60 dollars, everyone will probably guess 2 or 3. If you're in a class with a bunch of people in your major taking a random test, how well you think you did relative to others is a similar question (or how many questions you got right, if you don't already know).
program_whiz 17 hours ago [-]
not really, my claim is that if random data reproduces the trend (e.g. experts undepredict, layment overpredict skill), then it might actually make the argument stronger than the traditional explanation. The traditional explanation is that there is some secret bias where idiots think they are smart and geniuses think they are dumb. A stronger explanation supported by the random data example is "people can't assess their skill at all". The expert / laymen tails appear only because they are at the high/low end of the distribution and there isn't any higher/lower to go, so the errors go towards the middle. So there isn't a bias its just that the ability to understand skill is a skill itself, so the random errors look like a bias.
hector_vasquez 1 days ago [-]
Man sets out to write article on DK Effect, comes across one skeptical study, confidently writes article refuting DK Effect.
It writes itself.
pryelluw 1 days ago [-]
The author could have a promising career at The Onion.
sa46 1 days ago [-]
Weird take. The author corresponded with Dunning and reproduced the classic graph using random data.
Aurornis 2 days ago [-]
The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:
If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”
If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.
The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.
So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.
throwawaythekey 1 days ago [-]
I think people overlook the natural experience of Dunning-Kruger we all have.
You think you are good at chess because you win against your neighbour, then eventually a class mate absolutely embarrasses you multiple times and you realize there is a whole world you don't know, and before now you didn't know that world existed. Then you eventually beat your classmate before getting whomped by the beginner in the chess club and the whole cycle happens again. After this happens a few times, you should begin to find each time less surprising than the last.
To me this is the essence of DK and not really something to be proven/disproven by maths. You don't know what you don't know, and it takes some learning before you are wise enough to realize there are unknown unknowns.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
I think this is trying too hard on the researcher's part... if the data were reinterpreted such that small estimation errors are considered correct then the effect would reappear.
Fnoord 1 days ago [-]
That one time I was able to beat a 'good' chess player at school :-) I still remember that one. Usually I lost, but... I also came to the chess club rather to play Magic: the Gathering, so I didn't mind.
With that in mind, it has been a while (more than twenty five years) but what you said sums up the one time I went to my first Magic: The Gathering tournament, by having to build a deck with a limited amount of packs. You don't have the advantage of owning any power cards there (unless if you are lucky). What happened? Well, you could see who was doing well based on where they sat. And, I got absolutely wrecked! But, I also had a very bad (useless) rare [Shauku, Endbringer]. Which I shouldn't even have put in my deck, but I still did, cause.. it was a rare! [1] Still, there's much more luck to be had with decent commons and uncommons. In later tournaments (different types), I did better, and my fav experience (apart from some of my cards being stolen on a later tournament at that location) was the one where you'd buy a bunch of packs, open a pack up, and pass the packs on. Some people would pick only meta cards, some people would only pick expensive ones, others would always pick rare ones. If you don't intend to win, that could be a decent strategy. But I went for the strategy where I would pick the best cards for my deck, and that worked out well.
Paracompact 1 days ago [-]
When "natural experience" gets tossed around, I tend to immediately think: What about the _opposite_ natural experience?
In this case, it would be something like: Most people who are earnestly interested in a subject are so inclined to look upward (at the world elite) or to their side (at their own competition) that they tend to forget how much of humanity is increasingly left beneath them. We perceive everything we have already learned as "the easy stuff," the stuff we will never learn as "black magic," and the stuff we are currently learning as "getting interesting." But these labels are always relative.
kevin_nisbet 2 days ago [-]
> The strict academic definition hasn’t followed the colloquial usage for a long time. Maybe ever:
This is my understanding as well, and as I recall the results of the research were also more nuanced than most people seem to indicate. So not only were the novices with a wider range of self-estimates, the average result of those perceived scores was still below the experts. So it was never that low performers thought they were experts (although this may show up in the raw results with some portion of the population tests), just that on average they perceived their performance to be better than it actually was. And the high competence group while overestimating their results, still thought they'd do better than the low performers.
Tactical45 2 days ago [-]
You've hit the nail on the head, it's a conditional defenition, rather than an absolute one.
andy99 2 days ago [-]
It’s obviously real, at least as used in conversation, whether it meets some rigorous definition I’m sure there’s an out, but we’ve all known these people. With vibe coding they’re everywhere. Is this going to be a modern “begging the question” where everyone knows what you mean but someone pipes up that actually the technical meaning is different?
bonzini 2 days ago [-]
The problem is that the original formulation was "most people" are unaware of being unskilled, but by now the name is used to mean "some group of people" is unaware of being unskilled. In that sense the article confirms that “a small number are", 5-6%, and therefore the effect exists in the colloquial sense but not in the scientific sense.
Mentioning the Dunning-Krueger effect incorrectly is wonderfully meta...
lokar 2 days ago [-]
My experience with new CS grads was that most of them greatly overestimated what they knew, or alternatively, underestimated how much they did not know.
Jensson 2 days ago [-]
My experience with every person was that most of them overestimate what they know regardless of experience level. You just notice that more in new grads since its easier to tell when people are wrong about simple things than when they are wrong about more difficult things.
Software engineers tend to repeat the mantra "you cannot make accurate time estimates". That is true regardless of experience level, and everyone seems to be off by about the same amount. So there we have evidence that people overestimate their skills at every level, and its not that different.
teekert 1 days ago [-]
It’s my experience that I should over-estimate what I know otherwise I miss out on assignments that take me about 1-2 focused days of studying to get to sufficient level.
Ie I was once perfect for a project except for point 7 out of 10 which was experience with Keycloak (if you’re higher via an HR dept it’s even worse, they just tick boxes, who cares if you are smart and have broad knowledge).
So I’m now proudly Dunning-Krugering around, and use LLMs for super-charged learning. If I underestimated something I take the cost/time myself. So far it hasn’t happened to a significant degree.
shapefrog 2 days ago [-]
gell mann effect leads me to pay closer attention and then its game over
2 days ago [-]
al_borland 2 days ago [-]
This then boils down to people being generally bad at estimating their own level of ability.
xboxnolifes 2 days ago [-]
Which is exactly what the Dunner-Kruger effect is. Knowledgeable people statistically underestimating their knowledge and non-knowledgeable people overestimating their knowledge.
burpingtree 1 days ago [-]
It that’s the whole point that it’s just a mirage and is the same with random data. The people on the low end can’t underestimate their results as much and the people on the upper end can’t overestimate their results as much. That will come out of any correlation that is not perfectly correlated, which is why the article talks about it being replicable with random data. An interesting graph that would demonstrate a novel effect would be something nonlinear.
Dylan16807 1 days ago [-]
There are specific distributions that can be explained by random noise, and other distributions that cannot be explained by random noise.
But even if it is entirely caused by normal distribution variance, that doesn't make it a mirage, it gives you a reason.
3form 2 days ago [-]
I've seen at least 3 different ideas on this topic:
- people not skilled in a thing are bad at estimating their skills
- people are generally bad at estimating their skills
- people skilled in a particular areas often feel they are intellectually fit in other areas
Out of these three I feel like there's some truth in it, at least anecdotally.
CM30 2 days ago [-]
RationalWiki calls the third one 'Ultracrepidarianism', and says it's rather common with engineers in general:
But I'd say it's a subtype of the Dunning-Krueger effect rather than part of the general definition.
Personally I've always thought of Dunning-Krueger as an extension of the first definition here. Newbies/amateurs overestimating their skills and experts underestimating them, though I feel like the second definition is probably more true of human behaviour.
LiquidSky 1 days ago [-]
>RationalWiki calls the third one 'Ultracrepidarianism'
LOL because RationalWiki was written by engineers. I've often seen it referred to as "Engineer's Disease" for how it's commonly encountered.
efavdb 2 days ago [-]
for the last one, that's the whole story of Socrates being the wisest man because unlike others he knew the bounds of his knowledge:
"I seem, then, in just this little thing to be wiser than this man at any rate, that what I do not know I do not think I know either."
For those interested, it’s not just in Apology this comes up. In Theaetetus he does a pretty good job disproving that it is even possible to know what knowledge is.
bee_rider 2 days ago [-]
I think the third one is a separate thing. I don’t know of a name for it, but I’d call it the engineer’s (or physicist’s) blindspot or something like that.
I can’t remember of DK suggests some sort of effect where the expert has undue self-doubt, though…
dofm 2 days ago [-]
> I can’t remember of DK suggests some sort of effect where the expert has undue self-doubt, though…
There is a brief bit where they attribute that observation to other causes:
In making these predictions, we felt that we could account for an
anomaly that appeared in all three previous studies: Despite the
fact that top-quartile participants were far more calibrated than
were their less skilled counterparts, they tended to underestimate
their performance relative to their peers. We felt that this miscalibration had a different source then the miscalibration evidenced by
bottom-quartile participants. That is, top-quartile participants did
not underestimate themselves because they were wrong about their
own performances, but rather because they were wrong about the
performances of their peers. In essence, we believe they fell prey
to the false-consensus effect (Ross, Greene, & House, 1977). In the
absence of data to the contrary, they mistakenly assumed that their
peers would tend provide the same (correct) answers as they
themselves—an impression that could be immediately corrected
by showing them the performances of their peers. By examining
the extent to which competent individuals revised their ability
estimates after grading the tests of their less competent peers, we
could put this false-consensus interpretation to a test.
My experience has been that this isn't limited to engineers. I'm sure most people here have had people from other departments tell them how they need to do their jobs in one way or another.
For example, there are always at least a few people who you have to consult regularly because their tickets are always written how they perceive a problem should be solved rather than explaining the underlying problem to be solved. Then there are the sales and marketing types who like to dictate how the UX of the front-end should be because they like to retell anecdotes to prospective clients about how they solved a problem. Then there's the guy who hands you a spreadsheet he's maintained for years and tells you he wants you to put something exactly like it in the flagship product as a new feature.
CM30 2 days ago [-]
I'd say celebrities and creatives in general are often big offenders here too. Lots of actors/athletes/musicians acting like being good at what they do means they're also qualified to talk about science and engineering. And a lot of reviewers, documentary creators and specialist writers that decide to wade into discussions about topics they have no knowledge of/cover something they're new to, only to look ridiculously out of touch as a result.
See the Nostalgia Critic review of the Wall. Or the Alex Meyers review of Obsession.
magarnicle 1 days ago [-]
Or all the celebrities who write children's' books.
LiquidSky 1 days ago [-]
>My experience has been that this isn't limited to engineers.
To some extent, but the most severe form does seem fairly unique to engineers, specifically programmers. That is, I've met many arrogant lawyers and doctors, but never a lawyer or doctor who assumes they could program a computer due to their legal or medical expertise. But I've seen many a programmer assume that because they are proficient in programming they can confidently and authoritatively expound on all other areas of human endeavor (those being obviously inferior and trivial in the face of their own).
antonvs 2 days ago [-]
Medical doctors and physicists also regularly suffer from this. It’s more of a disease of expertise in general, it seems.
PowerElectronix 2 days ago [-]
Well, if I'm systems engineer and everything that exists can be modelled as a system...
There's also the idea on the other side of the coin - whereby experts on a topic underestimate how much they know about it compared to the average person.
Imo it would be more accurate to say that topic experts overestimate how much the average person knows about it.
falcor84 1 days ago [-]
Sorry, I'm confused. My formulation "experts on a topic underestimate how much they know about it compared to the average person" and your "topic experts overestimate how much the average person knows about it" seem semantically identical to me, just from a different side of the relation. What nuance did I miss?
refulgentis 2 days ago [-]
What is “it”, of those 3? The last one?
qingcharles 2 days ago [-]
The four levels of competence:
[ Unconscious Incompetence ] --> "You don't know what you don't know"
↓
[ Conscious Incompetence ] --> "You know what you don't know"
↓
[ Conscious Competence ] --> "You know, but you have to think about it"
↓
[ Unconscious Competence ] --> "You know it so well, it's second nature"
bigfishrunning 2 days ago [-]
I think it's beautiful irony that many people have the dunning-kruger effect specifically concerning the dunning-kruger effect itself
PunchyHamster 2 days ago [-]
It can be true tho. You might be genuinely skilled in one domain and realize the extent of your skill, while being over-confident in one you have little experience.
I also noticed people really successful at one thing tend to underestimate other domains, either thinking they are easy or that they can have meaningful input while being essentially novice (see any time Musk opens his mouth for example)
hn_throwaway_99 1 days ago [-]
With all due respect, being cocksure enough to declare that the Dunning-Kruger effect is definitely a thing, even when more careful analysis shows that the study conclusions were in error due to faulty analysis in the original study, kinda sounds like the perfect example of the Dunning-Kruger effect...
riddley 1 days ago [-]
The one that doesn't exist?
hn_throwaway_99 1 days ago [-]
Yes, that one.
shore88 2 days ago [-]
[dead]
danbruc 2 days ago [-]
Is any of the raw datasets of such an experiment available? I would like to see a scatter plot of self-assesed score vs actual score instead of the data aggregated into four bins.
robomc 2 days ago [-]
My favourite aspect of DK, which isn't changed by this argument really, is that people who reference it very often believe it said that low skill people are MORE confident in their abilities than high skill people, when the authors never claimed that. Which is kind of like doing a DK yourself when trying to deploy the DK findings.
oytis 2 days ago [-]
Hm... I vaguely remember a different article debunking the Dunning-Kruger. Basically the conclusion was that the data from the experiment shows that people's estimations of their results are all over the place, with people scoring high being actually slightly (but only slightly) more confident that they did well.
Maybe the core reasons are buried in this, but the amount of empty statements makes it hard to find
2 days ago [-]
bee_rider 2 days ago [-]
The plot in the blog post seems to be very symmetrical around 50% (to the point where there must be some identity going on). The plot from the paper seems to cross over around 75%. So the 3rd quartile still has some explaining to do, right?
jp57 2 days ago [-]
There are so many strange things about the original Dunning-Kruger plot. Why use quartiles for one axis and percentile for the other? Why use higher precision for the subject's estimate, which is by definition imprecise, and lower precision for the true score, which is known precisely?
I think the only conclusion you can draw from that plot is everyone thinks they'll be in the third quartile.
tonmoy 2 days ago [-]
Just because two graphs looks similar doesn’t mean they are “same”. Shouldn’t we subtract the random bias from the real world data and see if there is any slope remaining (or something similar)?
While the pop-culture notion of the Dunning-Kruger Effect is "idiots don't know they're idiots," the actual results of the paper were (essentially) that F students thought they were D students, whereas the A students thought they were B students. The argument here seems to be that the original effect is explained as essentially a kind of reversion of the mean argument (people assume themselves to be more average than they are), but I don't entirely buy that--especially since the simulation results they present don't really look like the original Dunning-Kruger results, since the crossover point is in the wrong place, and that's actually kind of significant in the original analysis...
DarkNova6 2 days ago [-]
Indeed. The true irony behind the Dunning Kruger Effect is that the pop culture understanding of it is essentially describing itself. Showing overconfidence in areas that you have little clue about.
dilawar 1 days ago [-]
The popular version of the Dunning-Kruger effect says that stupid people overestimate their capabilities. This is true, but the effect is more general than that.
It says if your expertise level is different from the average, then you judge the "average competency" differently. So if you have above-average expertise. Folks who had above-average education tended to overestimate what average education looks like (and vice versa).
I notice it when I use LLM. Things that I know well, I can spit LLM slopiness. On other things, LLM output reads like 'expert'.
jpfromlondon 1 days ago [-]
What percentage of the public think they can take a bear in a fight.
2 days ago [-]
2 days ago [-]
austin-cheney 2 days ago [-]
Regardless of whether Dunning-Kruger is real the solution is the same. DK concerns poor performing people who cannot accurately address their performance relative to a group. Forget DK. The bigger problem is missing objectivity, which is a very real concern. So, just measure for objectivity.
Can they measure things or do they just guess? Are they willing to seek evidence? Even if evidence is immediately available will they use it? Everybody has bias, but is their bias primarily self-oriented?
The consequences for poor objectivity are profound and measurable, but then its an invisible failure for people that struggle with this in the first place. In many industries poor objectivity can result in termination, law suits, criminal penalties, physical harm, and more. Software just seems to pretend this is vapor.
0xWTF 1 days ago [-]
I first ran across the Dunning-Kruger effect in my own data: a survey of my classmates which asked them 34 questions about their study habits, and a question in the middle asked them what quintile of the class they believed they stood in.
Even in this completely self-reported sample, the data strongly pointed to a Lake Wobegon bias in self-assessment - everyone self-assessed as above average, despite having more than half the class reporting. Statistically impossible, and not at all subtle.
I think about that study a lot. Particularly when I think I understand a something.
Fnoord 1 days ago [-]
One problem here is that you know about the outcome. If the outcome is going to be public, this creates a different type of competing pressure, where classmates are invited to compete with each other, while the one they should be competing with, is themselves.
It is like you have a teacher saying: 'my previous year's group, they were like so amazing [exhibit 1, 2, 3, ...]' and you know your group is never going to reach those limits -> #very_demotivational since in many ways, the way people are isn't going to be malleable. So a teacher is going to have to work with the material they got, and take that as a starting point. So instead, the teacher needs to have the intelligence acting like an adult, and refraining from expressing said enthusiasm.
I believe, if the results are discussed 1[-3] on 1 ie. in private chat, there is more room for openness, honesty, humbleness, and -yes- humility. On a macro level, this also exists on schools. Like, when the parents come to school to discuss their child's performance, it is a private setting without the other children and other parents, but with teacher, child, and parents. 1[-3] is still a private setting, but there are still things the child would only discuss with their parents, or only with their teacher. You could also achieve this in small groups on school, but as soon as you got a too big group there's this huge competing factor going out. Which is going to trample individuals. The shy kid isn't going to have their say, the ADHD kid gets annoying, ... and it can be tackled, with smaller classes.
gmuslera 2 days ago [-]
The article may not take into account the possible effect of knowing about the Dunning-Kruger effect (or cultural sayings that goes in a similar direction) may bias measurements. Before it was widely enough known it was not a factor.
Also, negative knowledge comes in two flavours, what you know that you don't know and what you don't know that you don't know. There it may be ground for that effect, but also changes in culture may affect that, specially with exposure to internet/global culture and attitudes, that may make you more aware of what you don't know, and stories of success/fail for taking the wrong approach.
luciana1u 2 days ago [-]
The autocorrelation argument explains the graph but not the people. Anyone who's managed a team has watched this play out in real time.
Finster 2 days ago [-]
I think it's mostly misapplied. The best example of Dunning-Kruger is an intelligent, competent, Ph.D. in physics thinking 9/11 was faked because "jet fuel can't melt steel", not realizing that steel loses significant tensile strength as it heats up without necessarily melting, which I think most engineers would be aware of. His great knowledge in one area blinds him to his woeful lack of knowledge in another.
2 days ago [-]
analog8374 1 days ago [-]
Knowing and not-knowing look a lot alike, subjectively speaking. Because both deliver a state of few-unresolved-riddles.
rimiform 2 days ago [-]
I think what people need to realize is that the Dunning-Kruger effect is mostly "not real" because, on average, everyone (regardless of competence) overestimates themselves. Saying that incompetent people overestimate themselves doesn't prove Dunning-Kruger is real, because it doesn't negate the fact that competent people also do this.
rossdavidh 2 days ago [-]
I mean, we all remember the cases where it was true, but do you really think most people think they are good at computer programming? Or speaking Russian? Or playing the harp? Or gardening? In the vast majority of cases, people who are not skilled at something, know that they are not. There are, sure, a few people who are overconfident, but the D-K effect as generally used in conversation was always pretty obviously untrue.
ars 2 days ago [-]
This article does not make its case. He shows a graph of "random" data, and then just kind of keeps going. But that random data is the meat of the whole thing.
Cut out 60% of the useless text, and focus on explaining why random data should look like that.
mobeets 2 days ago [-]
Agreed. Also it seems like, if the actual test scores and perceived test scores were both sampled iid, the “perceived” line should just be flat, with everyone being at 50th percentile. The fact that the displayed graph deviates from 50 in a way that more resembles the empirical data makes it seem like a cherry-picked random sample
root-parent 2 days ago [-]
So this is a case of the Dunning-Kruger Effect?
salynchnew 2 days ago [-]
I was going to say: is this a joke?
But really, the article seems to be going out of the way to make the author's particular point... but reads to me that the original paper is often understood... it simply shows that "specialists who are very knowledgeable about a subject are more likely to accurately identify gaps in their own knoweldge, when compared to any population less knoweldgeable on the same subject."
For example, I am apparently the most knowledgeable birder in my family. I've taken graduate-level ornithology courses, identify a fair number of N. American birds by their calls, etc. However, I recognize that I know nothing about birds compared to anyone who actually works in the field with them... I don't know enough to even estimate what I don't know.
timoth3y 2 days ago [-]
I've always found it somewhat ironic that the people who are least familiar with the actual research on the Dunning-Kruger effect tend to be the most confident in discussing it.
It's a sort of recursive Dunning-Kruger effect.
Alien1Being 2 days ago [-]
The AI-Kruger effect is here today...
...when very mediocre vibe coders think they are brilliant developers....
jesse_dot_id 2 days ago [-]
Having worked in tech my entire life, no amount of research will convince me that the Dunning-Kruger effect is not real. You might as well tell me that this isn't air that I'm breathing.
gruntled-worker 1 days ago [-]
Claude, please write a paper proving Dunning-Kruger is false. I'm not an expert, but you'll have no difficulty finding sources as it's clearly 100% false.
attila-lendvai 1 days ago [-]
publishing auto-correlation is not a 'data artefact', but a mistake.
another modelling problem with it: ceiling effect
https://news.ycombinator.com/item?id=38416412
How much can your top performers overestimate their performance?
The opposite problem happens for the worst performers.
The DK effect says roughly, "low performers tend to overestimate their abilities." Yet when researchers analyzed the data, they found that high and low performers overestimate and underestimate with the same frequency. [0] It's just that high performers are more accurate than low performers (note how this statement differs from the DK effect). Since you can completely explain the "X graph" by the random noise combined with the ceiling effect, and since beginners' self evaluations are noisier than experts', you don't even need regression to the mean to explain why you get the "X graph."
0. Nuhfer, Edward, Steven Fleisher, Christopher Cogan, Karl Wirth, and Eric Gaze. "How Random Noise and a Graphical Convention Subverted Behavioral Scientists' Explanations of Self-Assessment Data: Numeracy Underlies Better Alternatives." Numeracy 10, Iss. 1 (2017): Article 4. DOI: http://dx.doi.org/10.5038/ 1936-4660.10.1.4
1 days ago [-]
thehamkercat 1 days ago [-]
I've only read a little about the Dunning-Kruger effect, but I think I understand it pretty well
lowbloodsugar 2 days ago [-]
Ok. Let’s read the papers cited:
> Our results further confirm that experts are more proficient in self-assessing their abilities than novices.
jknoepfler 2 days ago [-]
I'm 100% confident it's real but I have no expertise on the subject. Checkmate, Atheists.
m000 2 days ago [-]
Isn't trying to discount published research with a glorified blog post the Dunning-Kruger Effect in action?
ErroneousBosh 2 days ago [-]
Do you think there's some element in published research that makes it automagically correct?
Papers saying that lead in petrol was totally safe were "published research", as were the papers saying that replacing tetraethyl lead with benzine made it safer.
Both of those turned out to be pretty majorly wrong, but they were "published research".
burnte 2 days ago [-]
I completely agree.
burnte 2 days ago [-]
It is absolutely real.
shuwix 23 hours ago [-]
Anyone who is desperatelly trying to downplay Dunning-Kruger effect is a perfect example of Dunning-Kruger effect.
qsera 1 days ago [-]
> The Dunning-Kruger effect
Other wise known as "Trust the experts" effect. I understand why this has 100+ votes.
jordand 2 days ago [-]
Excessive willful/unwillful ignorance is the root cause of someone exhibiting the Dunning-Kruger Effect. We've all at some point worked or lived with someone with real illusions/delusions about their abilities, and the root of it is ignorance. There's little we can do in our workplaces to mitigate these people. Word of advice from my experience: Never co-found a vc-backed software startup with someone that's done genuine innovation....and been completely ignorant and oblivious about everything else.
kittikitti 2 days ago [-]
Regardless of whether it's real or not, there's a significant public perception that it doesn't matter. There's a real "fake it till you make it" sentiment coupled with anti-intellectualism that will make people overestimate their ability.
On that note, I don't think the "random simulation" is described well enough. If I randomly assign a self-assessment and an actual score on a test, all of the quartiles will be the same because they should be uniformly distributed. I read through the papers mentioned and in these simulations, they hard-coded the correlation, "As in the Kruger and Dunning (1999) comparison, these random variables were correlated r=0.19"[1] so of course the graph will look similar.
On the other hand, the original Kruger and Dunning comparison could have been explained through differences in expected test scoring. It looks like people who thought they would get a D (60 percentile) or an F (50 percentile) objectively scored within the ranges of a 10 to 40 percent. While this is an overestimation, perhaps further studies can instead bucket the test scores according to how we expect them on a grading scale instead.
My takeaway is that the conclusions and discussions from the Dunning-Kruger effect study are valid. At the same time, the methodology and statistical significance is different from how popular science presents it. Also, there might be better ways to measure this phenomenon, and I would be interested in understanding how those who believe college is worth it and those who don't compare on objective testing.
Maybe this is reveals more about me than anyone else but the whole usage of dunning-kruger is just another arrow in the quiver for media to talk down to a group that they dismiss because they have different priorities.
I find references to the effect in pop culture are almost always used in an insulting, smug manner.
ranger_danger 2 days ago [-]
> The Dunning-Kruger Effect Is Probably Not Real
Self-deception by any other name is still self-deception.
The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self-estimates.
But overall I think real intelligence by definition requires empathy and humility.
One has to realize that we can't know the things we don't know, which includes the fact that we can't always trust our own beliefs and opinions because we might be relying on faulty or incomplete information, or we might be suffering from a mental health problem, whether we are aware of it or not.
"As a rule, strong feelings about issues do not emerge from deep understanding." -Sloman and Fernbach
DrewADesign 2 days ago [-]
I think the Dunning-Kreuger effect is more pernicious in smart people. I think Engineer’s Disease is basically DK by another name.
gaigalas 2 days ago [-]
A lot of Dunning-Kruger specialists here, apparently.
oulipo 2 days ago [-]
This article seems really dumb (no Dunning-Kruger joke intended).
The two lines on the graph are basically linear (for the "actual performance" the quasi-linearity is obvious by the design, for the "estimated performance" it still means that even though dumber people over-estimate their performance, all group still think they do best, when they actually do best, in a relative linear way)
And when they do their simple model (we assume they just generated "real performance" from a gaussian, then added some gaussian noise for the "performance" and another gaussian noise for the "self-assessment") they still (obviously) got two linear graphs that crossed each other.
And then they conclude that this means there is no effect, because "the graphs are eerily similar" (whatever that means)
But obviously the simple model is going to make two lines cross (in particular if you use a min(100, max(0, actual_performance + noise)) since at each extreme, then min and max will tend to skew the line). To put it simply: someone really stupid will STILL not pretend that he's "negatively stupid".
The argument "I can make a simple model without using actual humans which shows some kind of bias that vaguely ressembles the result of a paper" doesn't mean that the actual paper is wrong...
josefritzishere 2 days ago [-]
But what if that is the Dunning-Kruger effect?
1 days ago [-]
IshKebab 2 days ago [-]
Damn if only this article actually explained why you see this effect from random data. Unfortunately it doesn't seem like they understand the maths enough to know. Does anyone fancy reading those papers and giving us a TL;DR?
JackFr 2 days ago [-]
Yeah - I kept looking for the meat of the argument. The graph of random data looks a little correlated and I’d love to know why.
That being said I loved the mercury/Glasgow explanation. Anecdotally I see that all the time.
Jensson 2 days ago [-]
The graph of random data looks like that since its capped. The first place person cannot overestimate his position, and the bottom place person cannot underestimate his position. So any randomness at all will replicate the effect unless there is much more overconfidence in the high achievers than low achievers.
rawgabbit 2 days ago [-]
They constructed a straw man and then bitched about the straw man. In effect, they proved Dunning Kruger.
pessimizer 2 days ago [-]
I feel like the only important point in the article would be to explain how the random data was generated, yet it was relegated to the single sentence: "There was no bias in the coding that would lead these fictitious students to guess they had done really well when their actual score was very low."
Because on the surface, it doesn't make any sense for two sets of "random" numbers between 0-100 selected in pairs to deviate from each other based on whether the first number in the pair was low or not. You would not expect the first number chosen in a pair to influence the second number. Whether the first number was between 0-25 or 76-100, you would expect the second number to be about 50.
So this is obviously some sort of structured randomness that may be entirely justifiable, but the only way to find that out would be to read the two articles that this article purports to summarize for the layman. Instead there's over 1300 words of slop before this sentence, then nearly 700 words of slop after this sentence. Turns out we don't need AI for this. Speaking of random, I don't think that 2000 words is random.
-----
edit:
maybe the point of the papers is that low scorers can't underestimate their abilities - as in they literally don't have enough room? If so, that just means that the Dunning-Kruger affect is unavoidable. But the fact is that people are not choosing numbers at random, they are choosing them based on their expectations. People who got zero questions right and expected 100% are as likely as anyone else from a random number generator, and non-existent from actual people.
edit2:
OK, I've worked it out. I was making the mistake of thinking that they were evaluating absolute performance rather than relative performance. So each of the first numbers in the pair is unique. But that still leaves the fact that the random draw still predictably sits at 50% where the Dunning-Kruger data is around 65% based on the graph. Seems like norming that with the random data would give you better information.
edit3:
> In Dr. Nuhfer’s own papers [...] his team [...] showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
How is "narrower range" not an indication of more accurate self-evaluation? With that, and since people on the higher end of the scale have less room to overestimate their standing, and people on the lower end of the scale have less room to underestimate their standing, wouldn't you expect "Dunning-Kruger"? People on the low end of the scale would have wild swings that would be gated at zero, and people on the high end of the scale would have small swings that would be gated at 100. That would lead to small underestimates at the top, and large overestimates at the bottom. More accurate self-evaluation at the top of the scale is exactly what Dunning-Kruger is about, and the direction of the mistakes is predictable if this is true.
final, tldr:
Honestly, the entire debate is garbled. People are not being asked about their performance on a test, they're being asked about their standing within a sampling of people chosen by the experimenter, something which they have no reason to know anything about other than on the experimenter's word.
I think how people interpret Dunning-Kruger, and the only interesting thing about it, is that people who have more knowledge of a subject are more accurate in their assessment of how much they know about that subject. This seems likely (but not evidently) to be true, due to the range of (relative) self-assessment error being narrower in the top quartile as compared to the bottom quartile. This is what people found intuitive and compelling.
If it is true, the top quartile would tend to small underestimation (because of the narrower range and that they can't choose numbers higher than 100) and the bottom quartile would tend to larger overestimation (because of the wider range and that they can't choose numbers lower than 1.) That the average direction of over- and underestimation is forced by the nature of the evaluation doesn't make the effect any less true.
CurbStomper 2 days ago [-]
[dead]
Ozzie-D 1 days ago [-]
[flagged]
cyanregiment 1 days ago [-]
It makes a lot more sense if you think of it as a mere “effect” and not a “syndrome” the way it’s used.
Simply: Anytime anyone overestimates their ability, they will perform worse than if they had approached it with humility (on average).
Rather than speaking it like people “have Dunning Kruger”
Just a fact (heuristic) of behavior that anyone can fall subject to
throwawaythekey 1 days ago [-]
> Simply: Anytime anyone overestimates their ability, they will perform worse than if they had approached it with humility (on average).
Do you have a basis for that?
It seems like in life some tasks reward overconfidence, e.g. starting a business, and some tasks reward underconfidence e.g. retirement savings.
I'm not sure there is a strong bias one way or the other.
cyanregiment 1 days ago [-]
It's estimating one's own ability at performing a task - not "confidence" in a personality sense.
Someone can estimate their own ability at something at 0 and because of their own confidence and determination, they - to your point - may outperform a less confident person who also estimates their ability at 0.
The point of the study was that those who estimate their ability higher generally performed worse (regardless of confidence, impulsiveness, which we more broadly attribute to general "confidence" in personalities).
blini-kot 1 days ago [-]
a great article, one can somewhat expect this given the history of research in major psychological phenomena: Stockholm syndrome, prison experiment, etc etc
as for the arrogant people - of course they do exist, but it does not really warrant an "effect" of its own name, especially that most of us know in one way or the other why estimations fail and how it happens too
This makes some sense. If people are asked to guess a number between 1 and 6 and then roll a die, the people who roll low are more likely to overestimate and the people who roll high are more likely to underestimate. But the key is precisely how well random data mimics the effect.
This effect would disappear almost completely if they instead of estimating their rank they estimated their score, since then unless the test is so easy the best scores perfectly there will be a lot of room for everyone to overestimate and underestimate themselves.
But as is when the top 10% all estimate themselves to be in the top 10%, you will say they are underestimate themselves since on average the top 10% are in the top 5%. At the same time if the bottom 10% say they are in the bottom 10%, you will say they overestimate themselves since actually on average they are bottom 5%. But both these groups were making the same mistake, and its impossible for that not to happen unless everyone is perfect.
In the article, in the section "The effect is in the noise", just before the graphs, it says they estimated their score. Where did you get that they estimated their rank?
Add negative relationship and negative bias, and the random data will act as intended - hovering randomly around 50%.
Is it also plausible that the relationship between true and percieved is wholly independent (as this presumes)?
They're sampling slope and bias for the line from U[0,1] and U[0,100], the expected values of which will be 0.5 and 50. That's why they get what they get.
They say that randomness is any positively sloped line with any positive bias. So they assume dependence, just a very high variance of it. It's incorrect as they miss the negative half of the parameters - it would then correctly yield zeroes, for the presumed independence.
But it's not because "some statistical model exhibit a bias that's similar to the result" that this implies "therefore the result is a statistical error"... that's a backward reasonning
I don’t expect it to ever go out of the public consciousness. Like other things that were never real like Stockholm Syndrome I suspect it’s just stuck in the zeitgeist now.
1. https://en.wikipedia.org/wiki/Truthiness
It seems like the overlap between "real psychological effect" and "subtle enough that it requires research to discover" is vanishingly small. I guess that's not really surprising.
https://www.smbc-comics.com/comic/dongworld
If the research can't demonstrate the DK effect, then the research is bad, like poorly designed.
Using your personal experiences and the personal experiences of people you trust as an anchor is a good thing. It keeps you from falling for propaganda. Note that this DOES NOT mean studies conflicting with your personal experience are all wrong.
The tricky part here is understanding the limit of what your own experience can tell you.
"Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know."
Crichton's Law: unwittingly serving as the best example of one's own snarky law.
It’s the opposite problem. The research is bad because it “demonstrates the DK effect” even for cases where it cannot possibly be true. Like “people who tend to roll lower dice rolls overestimate their own ability to roll high dice values and people who tend to roll higher values tend to underestimate their own ability”
Which is just obviously incorrect and the true result should have been “there’s no correlation between people’s estimate of their ability to roll dice and their actual outcomes”
If you want to demonstrate an actual effect like DK, you have to set up an experiment such the if it truly doesn’t exist then the experiment tells you so. A great way of checking this is to throw random data at it in which case it should come out saying “I saw no correlation” but that’s no the case with their setup.
(and I am ignorant about DK, so...)
I don't have an opinion about Dunning-Kruger either way, but I wanted to point out that this is a pre-replication crisis mindset. Social psychology spent decades producing catchy, intuitive results that turned out not to replicate. We've learned over and over that we can't trust it when a result feels right, we need to rely on proven statistical methods.
Why do the research if you are so convinced from your own personal anecdotes? Quite a weird thing to say...
What? This is anecdotal and subjective. If the research doesn’t back up those anecdotes, then the research doesn’t back up those anecdotes. You want the research to back up your lived experience because it’s validating. That doesn’t mean that’s the actual reality.
I'm at the point honestly, where I don't even consider psychology to be a science anymore.
Besides, deciding what is and isn’t science is a question for the philosophy of science, not science itself.
Physics on the other hand is hard as in unyielding. It is easy to figure out boundaries of physics and map out what is and isn't true, and the few cases were we made a mistake everyone can agree a mistake was made and that formulas needs to be updated since physics is so extremely hard that even a tiny error will get noticed.
Hope that clears it up, physics isn't hard as in difficult, its hard as in rigid. And psychology is soft, not easy.
So, your statement doesn't make sense at all in this discussion, they just said psychology has soft traits, and then you say "ok, so its hard since its soft!". No, soft is difficult, not hard.
So for example, physics is like describing the shape of a metal spoon, and psychology is like describing the shape of a pillow. You can see how describing the shape of the pillow is massively more difficult, because its not fixed, so you have to come up with a language to describe all the ways it can deform and how that would work.
Do you have a source for this?
There is so much simple tests that they draw extreme conclusions from in the very first lectures you hear, and when they teach those things as true when its highly debatable its hard to not throw the entire field in the trash.
Of course there is probably some good psychology work done, but you shouldn't have so many shoddy examples in the first course then since that gives people a bad taste for the entire field.
One doesn't need to continue learning a topic to know it's bs when it's bs right from the start.
Sorry, most psychological research is sketchy at best, if not blatantly fraudulent
The fact that given X and Y random and independent, that Y-X is correlated with X doesn't disprove the Dunning Kruger. It in fact proves that Y = 1 X is a poor predictor, and the true model is Y = 0 X. In other words, perceived ability (of the human) cannot predict the actual test scores. Which is exactly what DK claims, but to a very extreme effect.
Note that, if there is actual signal (plus noise), e.g., if Y = X + eps; so the actual score is exactly the perceived score plus some added variation, the (Y-X)~X will be uncorrelated. In such case, there will be no DK effect, because the users are good at predicting their actual test scores, plus some constant variation.
It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.
And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.
Source code is here: https://github.com/pem725/Dunning-Kruger (found here: https://pem725.github.io)
Now it's much more clear. The simulated data tries generating the true relationship between actual and perceived scores from 0.0 to 1.0, and bias in self-reporting from 0% to 100%.
So the output graph should be the average of all these data generating processes, yielding perceived relationship around 0.5 and bias around 50%, with some high variation.
If you have the access, run their Shiny code with these values, and you will see the published plot.
I'd argue that this demonstration is much weaker than "making original no more meaningful than random". It's more that the "simulated 50% bias and 0.5 true correlation looks similar to what DK published", which is also far fetched given the data generation they did.
Note: true random (what they were going for) would cover negative relationships, yielding the random true relationship around 0; and if they wouldn't correct the sign of Bias, it would also average at around 0; yielding a realistic "random" with the slope hovering about 50% for any percentile.
Code>
# Lowest performers t.low.t <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$statistic # first quartile t-test (paired) t.low.p <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$p.value # first quartile t-test (paired)
# Highest performers t.hi.t <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$statistic # first quartile t-test (paired) t.hi.p <- t.test(Pair(Ability,Perception)~1, data=subset(df.w,Quants==1))$p.value # first quartile t-test (paired)
The article is baseless and fluffy
In the old plot, the bottom quartile has about a 50 percentage point margin between actual and perceived performance while the new one is 30 percentage points, which is a 50% difference between the old and new curve. The second quartile has 3x more margin in the old version relative to the new one.
That's not the negative correlation that it's often presented as in discussions. And it could as easily relate to mis-measuring everyone else's skill as to mis-measuring your own skill.
As a result people with little knowledge over-estimate their ability because a random dice roll will tend to be higher than their low ability (and the converse for people of high ability).
If you look at the DK graph, its close to random, maybe a very slight positive slope -- but ultimately the DK theory would be even stronger if it was truly random, because it means people have NO ability to assess their skill (0 correlation). I think the claim in DK is slightly weaker, that people at least have some ability, just not as much as we like to think.
And so the Dunning Kruger effect becomes true of all things that are uncorrelated?
Cash Quartile: 1 2 3 4 Average guess: 2.2 2.4 2.3 2.5
And of course it would vary based on cash. If you hand someone $1M maybe they guess correctly they are the highest, but if everyone has $50-$60 dollars, everyone will probably guess 2 or 3. If you're in a class with a bunch of people in your major taking a random test, how well you think you did relative to others is a similar question (or how many questions you got right, if you don't already know).
It writes itself.
If a specific novice is over-confident and out of their depth, we say “Dunning-Kruger”
If a specific is under-confident and performing better than their self-estimate, that’s not commonly considered Dunning Kruger, in the colloquial use. It’s called imposter syndrome, or not labeled at all.
The researchers aren’t really disagreeing with that. They found that novices had a wider range of self-estimates of their performance than experienced people. So in the novice group you were more likely to find someone who was grossly over-confident in their abilities, but you also found people who underestimated themselves.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
Which doesn’t precisely contradict the idea that among novices you can find people who overestimate their skills. Which is how it’s commonly used.
So I can believe it’s a statistical wash when averaging across all subjects. But I never considered the common use of Dunning-Kruger to be applied to averaged groups of people. It was always brought out for those outliers on the long tail of the novice grout who thought didn’t even know what they didn’t know.
You think you are good at chess because you win against your neighbour, then eventually a class mate absolutely embarrasses you multiple times and you realize there is a whole world you don't know, and before now you didn't know that world existed. Then you eventually beat your classmate before getting whomped by the beginner in the chess club and the whole cycle happens again. After this happens a few times, you should begin to find each time less surprising than the last.
To me this is the essence of DK and not really something to be proven/disproven by maths. You don't know what you don't know, and it takes some learning before you are wise enough to realize there are unknown unknowns.
> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
I think this is trying too hard on the researcher's part... if the data were reinterpreted such that small estimation errors are considered correct then the effect would reappear.
With that in mind, it has been a while (more than twenty five years) but what you said sums up the one time I went to my first Magic: The Gathering tournament, by having to build a deck with a limited amount of packs. You don't have the advantage of owning any power cards there (unless if you are lucky). What happened? Well, you could see who was doing well based on where they sat. And, I got absolutely wrecked! But, I also had a very bad (useless) rare [Shauku, Endbringer]. Which I shouldn't even have put in my deck, but I still did, cause.. it was a rare! [1] Still, there's much more luck to be had with decent commons and uncommons. In later tournaments (different types), I did better, and my fav experience (apart from some of my cards being stolen on a later tournament at that location) was the one where you'd buy a bunch of packs, open a pack up, and pass the packs on. Some people would pick only meta cards, some people would only pick expensive ones, others would always pick rare ones. If you don't intend to win, that could be a decent strategy. But I went for the strategy where I would pick the best cards for my deck, and that worked out well.
In this case, it would be something like: Most people who are earnestly interested in a subject are so inclined to look upward (at the world elite) or to their side (at their own competition) that they tend to forget how much of humanity is increasingly left beneath them. We perceive everything we have already learned as "the easy stuff," the stuff we will never learn as "black magic," and the stuff we are currently learning as "getting interesting." But these labels are always relative.
This is my understanding as well, and as I recall the results of the research were also more nuanced than most people seem to indicate. So not only were the novices with a wider range of self-estimates, the average result of those perceived scores was still below the experts. So it was never that low performers thought they were experts (although this may show up in the raw results with some portion of the population tests), just that on average they perceived their performance to be better than it actually was. And the high competence group while overestimating their results, still thought they'd do better than the low performers.
Mentioning the Dunning-Krueger effect incorrectly is wonderfully meta...
Software engineers tend to repeat the mantra "you cannot make accurate time estimates". That is true regardless of experience level, and everyone seems to be off by about the same amount. So there we have evidence that people overestimate their skills at every level, and its not that different.
Ie I was once perfect for a project except for point 7 out of 10 which was experience with Keycloak (if you’re higher via an HR dept it’s even worse, they just tick boxes, who cares if you are smart and have broad knowledge).
So I’m now proudly Dunning-Krugering around, and use LLMs for super-charged learning. If I underestimated something I take the cost/time myself. So far it hasn’t happened to a significant degree.
But even if it is entirely caused by normal distribution variance, that doesn't make it a mirage, it gives you a reason.
- people not skilled in a thing are bad at estimating their skills
- people are generally bad at estimating their skills
- people skilled in a particular areas often feel they are intellectually fit in other areas
Out of these three I feel like there's some truth in it, at least anecdotally.
https://rationalwiki.org/wiki/Ultracrepidarianism
(the Wikipedia article has that as a subheading on the Ne supra crepidam page instead: https://en.wikipedia.org/wiki/Ne_supra_crepidam#Ultracrepida...)
But I'd say it's a subtype of the Dunning-Krueger effect rather than part of the general definition.
Personally I've always thought of Dunning-Krueger as an extension of the first definition here. Newbies/amateurs overestimating their skills and experts underestimating them, though I feel like the second definition is probably more true of human behaviour.
LOL because RationalWiki was written by engineers. I've often seen it referred to as "Engineer's Disease" for how it's commonly encountered.
"I seem, then, in just this little thing to be wiser than this man at any rate, that what I do not know I do not think I know either."
https://en.wikipedia.org/wiki/I_know_that_I_know_nothing
I can’t remember of DK suggests some sort of effect where the expert has undue self-doubt, though…
There is a brief bit where they attribute that observation to other causes:
In making these predictions, we felt that we could account for an anomaly that appeared in all three previous studies: Despite the fact that top-quartile participants were far more calibrated than were their less skilled counterparts, they tended to underestimate their performance relative to their peers. We felt that this miscalibration had a different source then the miscalibration evidenced by bottom-quartile participants. That is, top-quartile participants did not underestimate themselves because they were wrong about their own performances, but rather because they were wrong about the performances of their peers. In essence, we believe they fell prey to the false-consensus effect (Ross, Greene, & House, 1977). In the absence of data to the contrary, they mistakenly assumed that their peers would tend provide the same (correct) answers as they themselves—an impression that could be immediately corrected by showing them the performances of their peers. By examining the extent to which competent individuals revised their ability estimates after grading the tests of their less competent peers, we could put this false-consensus interpretation to a test.
Link if you need it:
https://www.researchgate.net/publication/12688660_Unskilled_...
This I think is a separate phenomenon, maybe Nobel Disease but there might be a more general term, for example that includes celebrities.
https://en.wiktionary.org/wiki/engineer%27s_disease
For example, there are always at least a few people who you have to consult regularly because their tickets are always written how they perceive a problem should be solved rather than explaining the underlying problem to be solved. Then there are the sales and marketing types who like to dictate how the UX of the front-end should be because they like to retell anecdotes to prospective clients about how they solved a problem. Then there's the guy who hands you a spreadsheet he's maintained for years and tells you he wants you to put something exactly like it in the flagship product as a new feature.
See the Nostalgia Critic review of the Wall. Or the Alex Meyers review of Obsession.
To some extent, but the most severe form does seem fairly unique to engineers, specifically programmers. That is, I've met many arrogant lawyers and doctors, but never a lawyer or doctor who assumes they could program a computer due to their legal or medical expertise. But I've seen many a programmer assume that because they are proficient in programming they can confidently and authoritatively expound on all other areas of human endeavor (those being obviously inferior and trivial in the face of their own).
And of course there's a relevant xkcd: https://xkcd.com/2501/
I also noticed people really successful at one thing tend to underestimate other domains, either thinking they are easy or that they can have meaningful input while being essentially novice (see any time Musk opens his mouth for example)
UPD: probably this one https://economicsfromthetopdown.com/2022/04/08/the-dunning-k...
The article in the post is older though
I think the only conclusion you can draw from that plot is everyone thinks they'll be in the third quartile.
It says if your expertise level is different from the average, then you judge the "average competency" differently. So if you have above-average expertise. Folks who had above-average education tended to overestimate what average education looks like (and vice versa).
I notice it when I use LLM. Things that I know well, I can spit LLM slopiness. On other things, LLM output reads like 'expert'.
Can they measure things or do they just guess? Are they willing to seek evidence? Even if evidence is immediately available will they use it? Everybody has bias, but is their bias primarily self-oriented?
The consequences for poor objectivity are profound and measurable, but then its an invisible failure for people that struggle with this in the first place. In many industries poor objectivity can result in termination, law suits, criminal penalties, physical harm, and more. Software just seems to pretend this is vapor.
Even in this completely self-reported sample, the data strongly pointed to a Lake Wobegon bias in self-assessment - everyone self-assessed as above average, despite having more than half the class reporting. Statistically impossible, and not at all subtle.
I think about that study a lot. Particularly when I think I understand a something.
It is like you have a teacher saying: 'my previous year's group, they were like so amazing [exhibit 1, 2, 3, ...]' and you know your group is never going to reach those limits -> #very_demotivational since in many ways, the way people are isn't going to be malleable. So a teacher is going to have to work with the material they got, and take that as a starting point. So instead, the teacher needs to have the intelligence acting like an adult, and refraining from expressing said enthusiasm.
I believe, if the results are discussed 1[-3] on 1 ie. in private chat, there is more room for openness, honesty, humbleness, and -yes- humility. On a macro level, this also exists on schools. Like, when the parents come to school to discuss their child's performance, it is a private setting without the other children and other parents, but with teacher, child, and parents. 1[-3] is still a private setting, but there are still things the child would only discuss with their parents, or only with their teacher. You could also achieve this in small groups on school, but as soon as you got a too big group there's this huge competing factor going out. Which is going to trample individuals. The shy kid isn't going to have their say, the ADHD kid gets annoying, ... and it can be tackled, with smaller classes.
Also, negative knowledge comes in two flavours, what you know that you don't know and what you don't know that you don't know. There it may be ground for that effect, but also changes in culture may affect that, specially with exposure to internet/global culture and attitudes, that may make you more aware of what you don't know, and stories of success/fail for taking the wrong approach.
Cut out 60% of the useless text, and focus on explaining why random data should look like that.
But really, the article seems to be going out of the way to make the author's particular point... but reads to me that the original paper is often understood... it simply shows that "specialists who are very knowledgeable about a subject are more likely to accurately identify gaps in their own knoweldge, when compared to any population less knoweldgeable on the same subject."
For example, I am apparently the most knowledgeable birder in my family. I've taken graduate-level ornithology courses, identify a fair number of N. American birds by their calls, etc. However, I recognize that I know nothing about birds compared to anyone who actually works in the field with them... I don't know enough to even estimate what I don't know.
It's a sort of recursive Dunning-Kruger effect.
...when very mediocre vibe coders think they are brilliant developers....
and a rather ironic one at that.
The #Dunning-Kruger Effect is #Autocorrelation https://economicsfromthetopdown.com/2022/04/08/the-dunning-k...
A Statistical Explanation of the Dunning–Kruger Effect https://www.frontiersin.org/journals/psychology/articles/10....
The DK effect says roughly, "low performers tend to overestimate their abilities." Yet when researchers analyzed the data, they found that high and low performers overestimate and underestimate with the same frequency. [0] It's just that high performers are more accurate than low performers (note how this statement differs from the DK effect). Since you can completely explain the "X graph" by the random noise combined with the ceiling effect, and since beginners' self evaluations are noisier than experts', you don't even need regression to the mean to explain why you get the "X graph."
0. Nuhfer, Edward, Steven Fleisher, Christopher Cogan, Karl Wirth, and Eric Gaze. "How Random Noise and a Graphical Convention Subverted Behavioral Scientists' Explanations of Self-Assessment Data: Numeracy Underlies Better Alternatives." Numeracy 10, Iss. 1 (2017): Article 4. DOI: http://dx.doi.org/10.5038/ 1936-4660.10.1.4
> Our results further confirm that experts are more proficient in self-assessing their abilities than novices.
Papers saying that lead in petrol was totally safe were "published research", as were the papers saying that replacing tetraethyl lead with benzine made it safer.
Both of those turned out to be pretty majorly wrong, but they were "published research".
Other wise known as "Trust the experts" effect. I understand why this has 100+ votes.
On that note, I don't think the "random simulation" is described well enough. If I randomly assign a self-assessment and an actual score on a test, all of the quartiles will be the same because they should be uniformly distributed. I read through the papers mentioned and in these simulations, they hard-coded the correlation, "As in the Kruger and Dunning (1999) comparison, these random variables were correlated r=0.19"[1] so of course the graph will look similar.
On the other hand, the original Kruger and Dunning comparison could have been explained through differences in expected test scoring. It looks like people who thought they would get a D (60 percentile) or an F (50 percentile) objectively scored within the ranges of a 10 to 40 percent. While this is an overestimation, perhaps further studies can instead bucket the test scores according to how we expect them on a grading scale instead.
My takeaway is that the conclusions and discussions from the Dunning-Kruger effect study are valid. At the same time, the methodology and statistical significance is different from how popular science presents it. Also, there might be better ways to measure this phenomenon, and I would be interested in understanding how those who believe college is worth it and those who don't compare on objective testing.
[1] https://www.sciencedirect.com/science/article/abs/pii/S01918...
I find references to the effect in pop culture are almost always used in an insulting, smug manner.
Self-deception by any other name is still self-deception.
The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self-estimates.
But overall I think real intelligence by definition requires empathy and humility.
One has to realize that we can't know the things we don't know, which includes the fact that we can't always trust our own beliefs and opinions because we might be relying on faulty or incomplete information, or we might be suffering from a mental health problem, whether we are aware of it or not.
"As a rule, strong feelings about issues do not emerge from deep understanding." -Sloman and Fernbach
The two lines on the graph are basically linear (for the "actual performance" the quasi-linearity is obvious by the design, for the "estimated performance" it still means that even though dumber people over-estimate their performance, all group still think they do best, when they actually do best, in a relative linear way)
And when they do their simple model (we assume they just generated "real performance" from a gaussian, then added some gaussian noise for the "performance" and another gaussian noise for the "self-assessment") they still (obviously) got two linear graphs that crossed each other.
And then they conclude that this means there is no effect, because "the graphs are eerily similar" (whatever that means)
But obviously the simple model is going to make two lines cross (in particular if you use a min(100, max(0, actual_performance + noise)) since at each extreme, then min and max will tend to skew the line). To put it simply: someone really stupid will STILL not pretend that he's "negatively stupid".
The argument "I can make a simple model without using actual humans which shows some kind of bias that vaguely ressembles the result of a paper" doesn't mean that the actual paper is wrong...
That being said I loved the mercury/Glasgow explanation. Anecdotally I see that all the time.
Because on the surface, it doesn't make any sense for two sets of "random" numbers between 0-100 selected in pairs to deviate from each other based on whether the first number in the pair was low or not. You would not expect the first number chosen in a pair to influence the second number. Whether the first number was between 0-25 or 76-100, you would expect the second number to be about 50.
So this is obviously some sort of structured randomness that may be entirely justifiable, but the only way to find that out would be to read the two articles that this article purports to summarize for the layman. Instead there's over 1300 words of slop before this sentence, then nearly 700 words of slop after this sentence. Turns out we don't need AI for this. Speaking of random, I don't think that 2000 words is random.
-----
edit:
maybe the point of the papers is that low scorers can't underestimate their abilities - as in they literally don't have enough room? If so, that just means that the Dunning-Kruger affect is unavoidable. But the fact is that people are not choosing numbers at random, they are choosing them based on their expectations. People who got zero questions right and expected 100% are as likely as anyone else from a random number generator, and non-existent from actual people.
edit2:
OK, I've worked it out. I was making the mistake of thinking that they were evaluating absolute performance rather than relative performance. So each of the first numbers in the pair is unique. But that still leaves the fact that the random draw still predictably sits at 50% where the Dunning-Kruger data is around 65% based on the graph. Seems like norming that with the random data would give you better information.
edit3:
> In Dr. Nuhfer’s own papers [...] his team [...] showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.
How is "narrower range" not an indication of more accurate self-evaluation? With that, and since people on the higher end of the scale have less room to overestimate their standing, and people on the lower end of the scale have less room to underestimate their standing, wouldn't you expect "Dunning-Kruger"? People on the low end of the scale would have wild swings that would be gated at zero, and people on the high end of the scale would have small swings that would be gated at 100. That would lead to small underestimates at the top, and large overestimates at the bottom. More accurate self-evaluation at the top of the scale is exactly what Dunning-Kruger is about, and the direction of the mistakes is predictable if this is true.
final, tldr:
Honestly, the entire debate is garbled. People are not being asked about their performance on a test, they're being asked about their standing within a sampling of people chosen by the experimenter, something which they have no reason to know anything about other than on the experimenter's word.
I think how people interpret Dunning-Kruger, and the only interesting thing about it, is that people who have more knowledge of a subject are more accurate in their assessment of how much they know about that subject. This seems likely (but not evidently) to be true, due to the range of (relative) self-assessment error being narrower in the top quartile as compared to the bottom quartile. This is what people found intuitive and compelling.
If it is true, the top quartile would tend to small underestimation (because of the narrower range and that they can't choose numbers higher than 100) and the bottom quartile would tend to larger overestimation (because of the wider range and that they can't choose numbers lower than 1.) That the average direction of over- and underestimation is forced by the nature of the evaluation doesn't make the effect any less true.
Simply: Anytime anyone overestimates their ability, they will perform worse than if they had approached it with humility (on average).
Rather than speaking it like people “have Dunning Kruger”
Just a fact (heuristic) of behavior that anyone can fall subject to
Do you have a basis for that?
It seems like in life some tasks reward overconfidence, e.g. starting a business, and some tasks reward underconfidence e.g. retirement savings.
I'm not sure there is a strong bias one way or the other.
Someone can estimate their own ability at something at 0 and because of their own confidence and determination, they - to your point - may outperform a less confident person who also estimates their ability at 0.
The point of the study was that those who estimate their ability higher generally performed worse (regardless of confidence, impulsiveness, which we more broadly attribute to general "confidence" in personalities).
as for the arrogant people - of course they do exist, but it does not really warrant an "effect" of its own name, especially that most of us know in one way or the other why estimations fail and how it happens too