Dan Klein on Confirmation Bias and Jumping to Conclusions

by Don Boudreaux on November 9, 2011

in Science, Seen and Unseen

What a superb essay by my colleague Dan Klein in The Atlantic.

Be Sociable, Share!



41 comments    Share Share    Print    Email


Rick Hull November 9, 2011 at 5:21 pm

Bob Murphy takes issue with the form of some of the questions and especially, given the poor form, with Klein’s conclusions regarding “correctness”: http://consultingbyrpm.com/blog/2011/11/dan-klein-and-iuc-may-i-have-another-sir.html

Daniel Kuehn November 9, 2011 at 9:17 pm

Ya – for anyone that just browses Austrian economics out of a vague sense that “that’s the economics that libertarians believe in”, Bob’s detailed treatment of IUC issues and ordinal preferences is very good to read.

He’ll make up stuff about what neoclassicals and mainstream economists think, but you can just ignore him for that part :-D

Rick Hull November 10, 2011 at 1:20 am


EG November 9, 2011 at 5:34 pm

Well, at the very least the method of analyzing the answers should have been more rigorous (its pretty easy to analyze those statistics differently). And combined with better design of the questions, you may not have even had to worry about whether one answer is “correct” or “incorrect”.

Klein’s criticism of the survey were “obvious” to a lot of people at the time as well.

PS: And saying “economics” doesn’t tell us the level of knowledge one would need to answer them, besides if they have a bias or not. There’s easy economics questions, and there’s hard economics questions

PPS: How come he gets to publish results like that in a scientific journal (no offense meant to his work), while my stuff gets rejected for far less things? Not fair!!

vikingvista November 9, 2011 at 6:29 pm

What field is your stuff in?

EG November 9, 2011 at 7:56 pm

Management. Not that I have a lot of stuff published, just a couple of articles so far (hopefully more when I get a PhD). But one similar article I had; also a surveys, was rejected on far more flimsy grounds than Klein’s criticism of his work.

Klein does a good job at pointing out the shortcomings of the previous survey, primarily the questions not being designed for that use, and the statistical analysis method (whats the point of using a likert scale, if you’re only going to give right or wrong values?)

But I don’t see how this was fixed in the second article. Its exactly the same. The first article may have been lacking not in that it didn’t “provoke” libertarians and conservatives, but in the design and analyses.

Invisible Backhand November 9, 2011 at 5:35 pm

Shouldn’t a college professor have known better? Perhaps. But adjusting for bias and groupthink is not so easy…

A great way to adjust for bias and groupthink is to follow the money.

Invisible Backhand November 9, 2011 at 6:27 pm

For example, my own bias and groupthink can be confirmed by noting that I am a paid shill, troll, and informant for Anonymous and MoveOn.org.

Invisible Backhand November 9, 2011 at 8:27 pm

Party on Wayne! Party on Don!



BZ November 10, 2011 at 6:03 pm

Hmm.. Not sure I believe this. Anonymous isn’t necessarily left wing, and I doubt MoveOn.org has the money to throw away on paid trolls.

A more likely theory is this: some people are just ornery, and enjoy verbal conflict. Full disclosure: I tried to join a local atheist organization for this reason once. :)

Daniel Kuehn November 9, 2011 at 9:05 pm

Well, one thing I’m wondering is what the editorial process is like at EJW. I disagree with you (and have disagreed with you many times before) that financial support biases GMU economists. However, Klein edits his own publication and any other editors he uses are likely to think a lot like him. A more diverse editorial staff might have caught the first article before it was even published.

Invisible Backhand November 9, 2011 at 9:27 pm

I’m more saying that if the study of snails recommended low tax rates, deregulation, polluting the environment, climate change denial and union bashing, this place would be called CafeEscargot and be very well funded by the Institute for Helix Aspera studies.

Invisible Backhand November 9, 2011 at 11:03 pm

Furthermore, I am saying that my own bias and groupthink can be confirmed by noting that I am a paid shill, troll, and informant for Anonymous and MoveOn.org.

L. F. File November 10, 2011 at 9:22 am

Well at least he got a good 17 month run on the WSJ propaganda piece. Somehow I doubt that the citations of the original paper and op-ed all over the web will not be overshadowed by his recent study showing it was nonsense.

Klein understands as much psychology as he needs to to earn his paycheck.


EG November 10, 2011 at 12:22 pm

Well…thats an issue in just about every publication out there in the social sciences. If you know the editors, and if you’re known yourself, you’re likely to get a pass on things other people wouldn’t.

You know that happens.

vikingvista November 10, 2011 at 3:38 pm

Even in the hard sciences. But since there are countless science journals, you can probably get published eventually *somewhere*, unless the work is obviously complete junk.

Andrew_M_Garland November 10, 2011 at 3:01 pm

To . F. File,

Both studies were good (not nonsense). Klein’s conclusions were bad in the first study. He thought Libertarians/Conservatives would show less confirmation biased than Liberals. It seems that confirmation bias makes both groups wrong about various, different economic insights.

L. F. File November 10, 2011 at 5:48 pm

Garbage in — Garbage out! Neither study is much good.

The admitted experimenter bias in the first study can’t be fixed by more experimenter bias in the second!

I have read the op-ed and the Atlantic article and not the studies themselves but he it was too late to introduce any controls in the first study and I see nothing about controls in the second.

None of the results are good for anything more than coffee table conversation.


Fred November 9, 2011 at 6:36 pm

It’s all in how you word the questions. Make them misleading enough and you can get an ignoramus to give you the answer you want.

g-dub November 10, 2011 at 1:32 am

“Polls” suck. They can be answered at almost no cost.

kyle8 November 9, 2011 at 7:06 pm

(when two people complete a voluntary transaction, they both necessarily come away better off (agree);)

In What way is that an untrue statement.? I understand that a person could make a transaction they later regret, but if it is voluntary then that person is better off in their view point at the time of the transaction or they would not make it.

kyle8 November 9, 2011 at 7:12 pm

Also the question on drug availability is not, perhaps the right question. Although anyone who thinks it through should realize that indeed drug laws will restrict the access, but that is itself the problem with the laws.

Restrictions will necessarily create an underground demand, and will result in binge use. Just as with prohibition, overall alcohol use went down but binge drinking went up because you would not waste your opportunity on casual consumption.

Sam November 10, 2011 at 6:40 am

Perhaps, the key is the use of the word, “necessarily”, which I would take to mean that it is logically impossible for them not to both be better off after the the completion of the transaction.

This may just be a tautology, depending on how you define “voluntary”. If so, then it must be true. But I can imagine a number of situations where someone walks away from an un-coerced transaction worse off.

g-dub November 10, 2011 at 6:33 pm

But I can imagine a number of situations where someone walks away from an un-coerced transaction worse off.

Sure. Information is asymmetrical and tastes/values can change on a dime. Asymmetry does not necessarily mean fraud. A way people learn is by making mistakes, including poor buying decisions. None of this violates the voluntary nature of the trade, as you point out.

PF November 10, 2011 at 2:15 pm

It’s only true if there is no distinction to be made between being better-off and thinking you’re better-off. Surely there is such a distinction to be made?

Craig November 9, 2011 at 7:38 pm

This is particularly unconvincing. The second set of questions is quite different from the first. The whole exercise strikes me as an exercise in “proving ourselves wrong”. I suppose I am guilty of confirmation bias squared.

Daniel Kuehn November 9, 2011 at 9:03 pm

I was shocked EJW even began any analysis with that data or even published the first article – it had a lot of problems with it, which I was invited to summarize here along with three other critics in the same issue: http://econjwatch.org/articles/kuehn-response-to-bk

It’s worth reading all four of the critiques, because they cover a lot more problems with the first article than Dan and Zeljka even address in their own follow up.

The first article got a ton of press – a WSJ op-ed and it was widely circulated in the blogosphere. I’m not sure an article in the Atlantic is going to make up for that jump to conclusions, but it’s very nice to see Dan Klein put this out there rather than keeping the new results exclusively at EJW.

L. F. File November 10, 2011 at 10:26 am

Yeah and it only took him 17 months! Plenty of time for the propaganda value of the original piece to accumulate.


jorod November 9, 2011 at 10:11 pm

In my college course on Public Opinion, we looked at surveys as a way to determine peoples attitudes and beliefs rather than their knowledge on a particular subject. Perhaps the survey is really measuring attitudes not knowledge.

Bastiat Smith November 9, 2011 at 11:39 pm

“…when two people complete a voluntary transaction, they both necessarily come away better off (agree);…”

Excuse my candid question: How is ‘agree’ the wrong answer?

robert_o November 10, 2011 at 2:50 am

The question didn’t specify ex-ante or ex-post. Given that the question could have meant ex-post, then “agree” is not the answer: it’s possible for one or both people to later on regret having made the transaction.

I would have chosen a different wording, but Klein is technically correct (the best kind of correct).

Dom November 10, 2011 at 9:16 am

I have to agree with Bastiat here. I was struck by the same point. I wonder if that’s a typeo, and “disagree” was supposed to be the wrong answer. After all, my guess is that most leftists would disagree, and most libertarians would agree, and the confirmation bias was supposed to be towards the left in this question.

PF November 10, 2011 at 2:19 pm

But what is it that happens when you come to regret a transaction? Surely it is simply realising that (regardless of what you thought previously) you are not *in fact* better-off for having made the transaction. When I make a voluntary transaction I am not hoping to think I am better-off but to actually *be* better-off.

That is not yet to go so far as to say that voluntary exchange is not always just or whatever, but surely you can admit the distinction without giving up support of voluntary transactions?

g-dub November 10, 2011 at 6:42 pm

There was wording in the second set that was poor. For example, “a dollar means more to a poor person than it does to a rich person (disagree)”

That is very poorly worded, and I would strike any results on that one. Presumably it means something like “a single dollar is a much higher fraction of the holdings of poor persons than rich persons, and thus a higher fraction of holdings of poor persons goes to essential survival needs.” But who knows what such subjective junk means?

Also presumably, “dollar holdings” are very important to rich people — that’s why they try so hard to get and hold them. After all, that is the “greed” motive we are reminded about so often by progressives (regressives). I guess if the poor thought dollars were so important, they’d see to it they had more of them.

L. F. File November 10, 2011 at 10:25 am

Interesting Don, Here are your comments from June 2010 on the original paper:

“My GMU Economics colleage Dan Klein has this excellent and revealing op-ed in today’s Wall Street Journal. In it, he reports the results of a survey that he, along with Zogby researcher Zeljka Buturovic, conducted to test American adults’ grasp of basic economic ideas. They find that people who self-indentify as “libertarian,” “very conservative,” or “conservative” understand basic economic principles far better than do people who self-identify as “liberal” or as “progressive.”

I think your confirmation bias against liberals and progressives was certainly on display there. I think we can take your encomiums here as just confirmation bias in favor of a colleague/co-conspirator.


Julien Couvreur November 10, 2011 at 12:10 pm

“a dollar means more to a poor person than it does to a rich person”

Intuitively that seems to be true, but that cannot be established without some form of interpersonal comparison of value/utility.

SMV November 10, 2011 at 1:09 pm

I thought that myself. I understand that my next $ will have less utility than my last, but it seems highly possible that I would value my next $ less than some people with greater wealth.

robert_o November 10, 2011 at 3:51 pm

Robert P Murphy makes the same point on his blog (and his book).


faithkills November 10, 2011 at 12:49 pm

His questions to challenge his own position were much closer to trick questions. EG “when two people complete a voluntary transaction, they both necessarily come away better off (agree); when two people complete a voluntary transaction” The way that is not true is to apply some objective measure and that in itself has to be a flawed assessment of a subjective process.

An honest way to phrase that question would be “every external observer will always agree that both parties come away better from any voluntary exchange”.

The original set of questions were not deceptive in this way. So I judge he is guilty of confirmation bias TWICE. In the second case he wanted to show that we are all equally dumb to regain his MS economics cred.

And his data conformed to that goal.

Kendall November 10, 2011 at 1:29 pm

Pro sports teams do voluntary transactions all the time which may leave one party worse off than before. They may be worse off immediately but don’t find out until later.

g-dub November 10, 2011 at 7:22 pm

“Rank and file” “progressives” aren’t just ignorant of economics, they deny its very existence. Economics is about scarcity. They don’t care about scarcity, they just know that there is a problem and that it must be fixed at any cost. There is always a rich person with too much money, who didn’t deserve it, who thus can have their assets seized to pay for “what must be done.”

Previous post:

Next post: