Wednesday, July 22, 2015

Overconfidence Goeth Before a Fall

What would we do without psychologists?

Nobel prize winner Daniel Kahneman is arguably the most influential psychologist in the world today. So people grant credence to his views… justifiably so.

In an interview with the Guardian Kahneman explains that if he could rid the human species of any bias, any flaw in the human decision-making process, he would abolish overconfidence.

The Guardian explains:

Not even he believes that the various flaws that bedevil decision-making can be successfully corrected. The most damaging of these is overconfidence: the kind of optimism that leads governments to believe that wars are quickly winnable and capital projects will come in on budget despite statistics predicting exactly the opposite. It is the bias he says he would most like to eliminate if he had a magic wand. But it “is built so deeply into the structure of the mind that you couldn’t change it without changing many other things”.

Naturally, one asks oneself where one might have heard this before. And the answer comes quickly to mind from the book of Proverbs:

Pride goeth before destruction, and an haughty spirit before a fall.

Better it is to be of an humble spirit with the lowly, than to divide the spoil with the proud.

What Kahneman and other psychologists call overconfidence bears a striking resemblance to the deadliest of the seven deadly sins: the sin of what is usually called pride. It might even be considered to be too much of a good thing.

Overconfidence has many faces and many facets. As a rule it refers to states of mind where one’s beliefs about one’s abilities and achievements far surpass one’s actual abilities and achievements.

One suspects that those who are hawking self-esteem, to the point where every child receives a trophy, regardless of ability or achievement, are building overconfidence.

One might note, among the curiosities of human language, that the word overconfidence does not have a correlate: underconfidence. Yet, some people are excessively humble, excessively lacking in confidence, to the point where they are unwilling or unable to take any risk.

Psychologists demonstrate the prevalence of overconfidence by asking students how well they have done on tests. Said students seem invariably to overestimate their scores, by a significant factor. Of course, one should ask whether the same students, had they been suffering from less self-esteem would have done better or worse. A student who lacks sufficient confidence might test poorly... because of a pessimistic attitude. But, an overconfident student might more easily breeze through the exam and not put in enough work. A student who is less confident might take more time and might revise his answers more carefully.

Excessive self-esteem seems to be a way of avoiding a larger expenditure of effort. It is antithetical to the work ethic.

The same applies when a professional asserts expertise. 

In Kahneman’s words:

Overconfident professionals sincerely believe they have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion.

Surely, there is truth to the assertion. I am confident that we place far too much credence in the views of so-called experts. At times, we expect that experts can accurately predict the future. Since no one really knows the future, in many cases, when it comes to predicting the GDP or the stock market, they are wrong. And one knows that experts in childhood development often give advice that is driven more by ideology than by what is best for this or that child.

One might count oneself as an expert because of training or experience or degrees. But, do some experts believe so strongly in their expertise that they refuse to accept any outcome that makes them look less expert?

To take an example, how many experts in climate science believe that they know, to a certainty, what the climate will be a century from now?

On the other hand, young professionals often overestimate their expertise. This might be normal when they do not have enough experience. But also, what sometimes looks like arrogance can be a motivating factor or a sales pitch. If you cannot persuade yourself and others that you are better than you are, you will often not get the chance to improve.

How are these young people going to gain experience and expertise if they do not allow themselves to undertake tasks that are slightly beyond their expertise?

Some of Kahenman’s examples involve predicting the future. Regardless of what is and is not built into the structure of the human mind, one’s ability to undertake a difficult task, depends at the least on confidence, and at times on overconfidence. A government may need to fight a war and might know that the public will never support the undertaking if it does not believe that the war can be won, and quickly. If the government's representatives are to communicate effectively, perhaps they need to convince themselves that the war can be won quickly and easily.

As for the capital projects that may or may not come in on budget, the need for the project and the need to motivate people to do the job might cause them to offer too optimistic an appraisal of its cost.

But if a boss exhibits too much confidence, even to the point of seeming to be detached from reality, he will lose the confidence of his team.

And then, if a coach at halftime exhorts his team to go out and win the game because he knows that they can do it, does a loss cause them to think that the coach was lying to them? 
Within the language game that is coaching, it is understood that certain ideas are intended to motivate and to produce an outcome, not to predict it.

I suspect that there is no such thing as confidence that is perfectly attuned to an unknown outcome. Overconfidence might be a psychological strategy to engage one in an activity that might be more necessary than desirable. 

Consider some other examples from the research, summarized by Wikipedia:

If plaintiffs and defendants were prone to believe that they were more deserving, fair, and righteous than their legal opponents, that could help account for the persistence of inefficient enduring legal disputes.[26] If corporations and unions were prone to believe that they were stronger and more justified than the other side, that could contribute to their willingness to endure labor strikes.[27] If nations were prone to believe that their militaries were stronger than were those of other nations, that could explain their willingness to go to war.[28]

Some of these seem to be a reach. When deciding whether or not to sue or to settle a lawsuit, lawyers and their clients often calculate risk and reward. I am not confident that overconfidence is as prevalent as one might think.

On the other hand, some criminal defendants are so overconfident in their cases that they refuse plea deals because they believe that they will be acquitted at trial. The name of Martha Stewart comes to mind.

And of course, there is the case of Jarndyce vs. Jarndyce, from Dickens’ Bleak House. That case seemed to have a mind and momentum of its own... it went on and on and on.

As for the strength or weakness of one’s armies, it is clear that the Estonian army, such as it is, is considerably weaker than the Russian army. But, beyond the obvious gross disparities, how do you know whether your army is weaker or stronger until you have tested it in battle. And also, sometimes stronger armies have had great difficulty fighting against counterinsurgencies led by forces that are considerably weaker? There is a lot more to choosing to fight wars than an estimation of the strength of one's military.

We decided to fight in Iraq in 2003. We decided not to fight in Syria or Iraq after 2011. Which was the better choice? Which produced the better outcome?

And then, there’s the question of how one presents oneself to the public at large. If a politician, Wikipedia suggests, projects confidence, or even overconfidence, people are more likely to vote for him. In this case, it sometimes appears that the more the better:

For instance, those most likely to have the courage to start a new business are those who most overplace their abilities relative to those of other potential entrants. And if voters find confident leaders more credible, then contenders for leadership learn that they should express more confidence than their opponents in order to win election.

Of course, Barack Obama projected and continues to project a confidence in his ability to do his job. By all indications, he cannot. Yet, many people in the country are happy to affirm his overconfidence and protect him from the harsh judgment of reality.

And, Donald Trump, who seems lately to have cornered the market in overconfidence has managed to persuade a large number of people that he can just step into the oval office and do the job of president. You might say it’s a giant bluff and you might say—as I would—that it’s filled the Republican presidential nominating process with some rather juvenile showboating, but for now Trump seems to have attracted a considerable support.

Sometimes one feels that the Obama presidency has produced so much mind numbness that it believes overconfidence is true confidence. Trump and his supporters do not seem to understand that success in business, such as it is, does not mean success in all endeavors, great and small. It's like saying that because you have mastered the art of driving a car you will naturally know how to pilot a plane.

Watch out below.


Ignatius Acton Chesterton OCD said...

"On the other hand, some criminal defendants are so overconfident in their cases that they refuse plea deals because they believe that they will be acquitted at trial."

Indeed, and the ones who are convicted express this even more overconfidence. The highest scores for self-esteem in the United States are from... people in our prisons. That was from research by Roy Baumeister at Florida State University. Shockingly, Baumeister advocates for emphasis on self-control rather than self esteem. Madness!

The Proverbs verse was a wonderful touch, but our modern society needs opaque, intelligent-sounding jargon and verbose explanations to believe in anything. We only trust people who are really, really smart. Only fools read the Bible.

Ares Olympus said...

I heard of Daniel Kahneman from his talk (and book) thinking, fast and slow. Daniel Kahneman on Thinking, Fast and Slow THINKING, FAST AND SLOW BY DANIEL KAHNEMAN | ANIMATED BOOK REVIEW,_Fast_and_Slow
The book's central thesis is a dichotomy between two modes of thought: "System 1" is fast, instinctive and emotional; "System 2" is slower, more deliberative, and more logical. The book delineates cognitive biases associated with each type of thinking, starting with Kahneman's own research on loss aversion. From framing choices to people's tendency to substitute an easy-to-answer question for one that is harder, the book highlights several decades of academic research to suggest that people place too much confidence in human judgment.

I learned I preferred system 2 from High school when I went to a Science Olympiad, first solving problem as a computer programmer, where I had time to plan and think, while even after I solved the problems, I kept checking for limitations of my solutions, and I was the only person who stayed the full time limit. Then I was an unplanned substitute for the quiz bowl team and never answered a single question because someone else would always press the buzzer first and it surprised me to hear afterwards from team members they'd press it even knew if they knew the answer. They just assumed adding 5 seconds between pressing the buzzer and having to speak, they could come up with the answer. For me that was a completely offensive attitude, but curious since they were willing to look foolish to guess on something when they didn't know.

For me I'm 100% in love with "system 1" thinking, whether it is gut-thinking or intuition, but ONLY if there are no irreversible consequences from listening to it, if there's time to evaluate it and reverse course if its wrong. So ideally I'd use System-1 to avoid danger, and retreat to a safe distance, and then use System-2 to analyze a situation more carefully, looking for affirmations and contradictions for each possible interpretation I can find.

I'm probably also happy for others, experts or not, to be over-confident (whether from their system 1 or system 2) because that positions sort of "forces action" that my observer role can't generate. So if someone "cries wolf" I'm willing to consider that a legitimate point of view that may need fast and slow responses, fast response to test and face immediate danger, and slow response to evaluate the danger in proportion to other dangers, like the fable of the lions where the old lions roar from one direction to send prey into the path of the younger ones. So known dangers may be more manageable than unknown ones.

Probably the biggest "confidence games" is the stock market and financial world now. It would seem "conservatives" should not trust markets, and would instead invest their savings as close to home as possible, but strangely "free markets" are a virtue to Conservatives, pretending such a thing existed, pretending buyers and sellers are equals, ignoring (liberal) Trump's boasts of "screwing people" with his expert negotiation skills like the best used car dealer. Somehow we're all "over-confident" we can play, we can invest retirement savings for 30 years with the sharks and expect to get our fair share in the end, just for trusting them.

Distrusting markets (or distrusting government) both in their own ways can be a "out of the frying pan, into the fire" proposition. And in either case, we have to trust to play the game. That's where FDR's "The only thing we have to fear is fear itself" comes in perhaps? But we still have to decide which fear to face and which to flee from. We need both System I and II here.

Larry Sheldon said...

I'm thinking "In all things, moderation." I don't know who said that first. it was not me.

At closing out 77 summers I have lived most of my life where occasionally time and circumstance required and allowed the development of the perfect answer, but the ordinary case called for a good-enough (preferably tunable) answer now is required, a perfect answer in the future has no value, real or potential.

I wonder if the first line of that Proverb is the most frequently misquoted bit of the Bible.

Anonymous said...

In summer camp, when asked, "Can you swim?," most children eagerly answer "Yes." The few exceptions are those who are naturally more timid and fearful of water. The timid generally are less distracted during lessons yet require more patience with their fear to get them to practice effective skills that later boost their confidence. The eager children are more easily distracted and want to play with each other rather than practice swim skills. If an adult and some older children demonstrate skills the eager children begin to practice too, like a community of apes, and over the course of a season the pool becomes like an extended classroom, as the older children become swimmers the younger ones skills improve via practice and imitation without thinking so much about the swim lessons anymore. Too little fear would get us killed in many situations but confidence in self and others is necessary to take action and learn, so I think nature favors over-confidence in most children. Deep open water is absolutely deadly for non-swimmers so the timid children have a more accurate self-perception, and yet, for learning swimming more rapidly I observe that the "I can swim" attitude is more effective.

Aging doctors, lawyers, and other technical experts probably experience a decline in skills after mid-life which is hard to admit to oneself. There is definitely a biological sweet spot in most professional sports where rookies become skilled veterans and aging veterans can't compete as effectively anymore with younger athletes. The correlation between confidence (subjectively experienced) and skill objectively measured by observing learning and competition is pretty hard to map.

Dennis said...


As one matures one of the first things they learn, or should learn, is that one cannot make things happen. One has to start thinking about creating the conditions for them. So much of being able to keep performing at a high level in one's chosen profession is in how the mind accepts adversity and challenges. A good deal of people's greatest works happens in their later years of life when experience is combined with talent, skill, humility and an acceptance of life's vagaries.
At a certain point one recognizes, hopefully, that it is not the amount of "practice," but the quality of that practice that counts. Life is far simpler and easier than we believed when we were young.
It took me a long time to get my wife to recognize that going outside and doing the flowers, landscaping and other tasks of gardening until she was so tired she could not move for days would be easier if she worked until she started to feel a little bit worked and then stop. This way she could come back the next day a do more and get far more accomplished in a given time period. This applies to almost any endeavor we want to achieve. It does help to have a system in which to classify the importance of the work as well.
As an analyst if I started to feel like I was getting too involved I would get up and walk around, go eat or do something else. Invariably when I sat back down the answer or idea would just happen. I believe one's subconscious never stops working to find an answer to what is requested of it. This may sound strange, but I could then visualize it, manipulate it, et al in the space just in front of me. For a while this may have bothered some people, but they began to leave me alone staring up in space for I got results.
As long as one keeps an active involved mind age is almost never a problem. The mind lives on constant challenge.

priss rules said...

Kahneman is over-confident in his conclusion about over-confidence.

priss rules said...

"One suspects that those who are hawking self-esteem, to the point where every child receives a trophy, regardless of ability or achievement, are building overconfidence."


But in time, apathy among the truly talented and cynicism among the untalented.

priss rules said...

'Over-righteousness' is a problem among the 'progressives'.

Ignatius Acton Chesterton OCD said...

Regarding the trophy kids, there are two things we tend to overlook: (a) value; and (b) child resilience/adaptability.

Trophies are used to distinguish merit. If everyone gets a trophy, they might as well get a paperclip, because there's no distinction for performance. Dangerously foolish people create the idea that everyone should get a trophy. Just because your kid wants one, or you want your kid to have one, is not the issue here. Human beings are insatiable in their wants, needs and desires. When we get everything we want, we miss the truth about value, and that is devastating to success in life. Trophies for achievement allow for signaling: this kid excels, this kid does not. There's a gift there. The idea that a child must excel at, say, baseball, is an absurdity of imagination and indicative of disordered attachment on the part of the child or the adult. Children are children. It's the adult's job to be an adult. This seems to be an increasingly difficult idea to fathom. Who do you think came up with the idea there should be a trophy for every kid? An 8-year-old?

One damaging thing that I think psychology has emphasized is this idea of scarring memories. You know, that moment when your Uncle Harry the budding schizophrenic freaked out at you at the dinner table and called you all kinds of terrible things? Yes, those kinds of unfortunate things that happen to all of us. It's the time when we learn that the world isn't a safe place and we'd better watch out. Every human being since the beginning of time has experienced these incidents. It is a fairly recent development that parents put their kids in "bubbles" hoping that there kid will not have such a scarring experience. This is understandable... no parent wants their child to experience pain. Yet we all do. We all have those memories. Hopefully we get the support to work through them. "Honey, your Uncle Harry is a sick man, and didn't mean those things he said to you." It's still a penetrating memory, but good parents work through things. But increasingly, parents seem to be preventing children from having any negative experiences and we are getting more and more sensitive and anxious because we're not teaching kids to work things out. We're teaching them not to take care of affairs, and that others will be there to take care of us. I'm not suggesting we should reinstitute some kind of Spartan childhood, but this prophylactic childhood fantasy is quickly becoming a nightmare as we re-fit the college experience to protect young adults (notice the noun) from "triggers." Good grief.

Ignatius Acton Chesterton OCD said...

Back to the trophy thing... What if your kid is no good at baseball? Maybe he'll be good at something else. Find it. But the other important lesson is failure. The kid who excels all the way through never has to deal with failure... if things come easy to him, he doesn't learn effectively. All human beings have limits. If young people don't see those limits and make different choices, they get into this one-track thing like you see in communities with "travel" sports leagues. How is it healthy for your kid to do one sport all the time? Is that about you or in service of him? It's never made sense to me. Yes, if you're good at something, have at it, by all means. But if that's all you're doing, I'm not sure you ever really grow and experiment and learn where you experience joy. It's a process of trial-and-error. Young people have so many choices now, and it seems like a disservice to keep them locked into one in hopes of mastery and glory. Your kid isn't going pro.

And this is where child resilience comes in. You watch kids and see their desire to learn, grow, be in motion, ask questions, adapt, etc. and it's just a miracle. If your kid doesn't get a trophy, support her in exploring and finding something she does enjoy. It's the joy and the experience that we're looking for. Sure, we've educated an entire generation to "follow your bliss," which is poor advice in terms of career utility. But there's also wisdom in saying that not performing in baseball and not getting a trophy is telling you something. There are all kinds of other things one can investigate, learn and do. Maybe your kid goes into the forensics tournament and is phenomenal and now has a passion and becomes a successful forensic scientist. That's not a bad outcome, considering that your kid was never going to be Derek Jeter. Turns out the non-consolation prize was pretty valuable in the end. If your kid is willing to listen, and the parent has some perspective. Yet many don't. Such is life.

I'm sure the Obama Administration will come up with a preposterously wasteful social program to give your kid the feeling that he really will be successful at something he's not. That way, their inability can become a "disability," and you can some trial lawyer to file a social security disability insurance claim for him at a young age so he'll never have to work again.

Anonymous said...

This article describes how optimism may contribute to financial bubbles: