A few more empirical thoughts

by Russ Roberts on October 29, 2007

in Data

Colleague Robin Hanson gives me a hard time over at Overcoming Bias (reacting to this earlier post):

If Russ relies little on data to draw his conclusions, then on what
does he rely?  Perhaps he relies on theoretical arguments.  But can’t
we say the same thing about theory, that we mainly just search for
theory arguments to support preconceived conclusions?  If so, what is
left, if we rely on neither data nor theory? 

Try saying this out loud: "Neither the data nor theory I’ve come across
much explain why I believe this conclusion, relative to my random whim,
inherited personality, and early culture and indoctrination, and I have
no good reasons to think these are much correlated with truth."  That
does not seem a conclusion worth retaining.  If this is really your
situation, you should move to a nearly intermediate position of
uncertainty.   Either you should believe that truth-correlated data or
theory has substantially influenced your belief, or you should retain
only a very weak belief.

In this week’s EconTalk podcast I added this postscript to try and clarify what I was saying:

In last week’s podcast I spoke with Ian Ayres about the power and limitations of data analysis. Ayres emphasized the power and I kept mentioning the limitations, especially in the postcript I added after the interview. I want to clarify a few issues. My basic point was that when it comes to high-powered sophisticated statistical techniques, our biases as researchers and as consumers of that research often triumph over truth. The truth is elusive in complex systems with many things changing at once. It’s hard to isolate the independent effect of one particular variable. When scholars can run hundreds of multivariate regressions at very low cost, it easy to convince yourself that the results that confirm your prior beliefs are the “right “ results. The ones that failed must be the “bad ones.”

When I was in college at U of NC, I took a wonderful course from Richard Smyth where I learned about the American philosopher Charles Peirce and the philosophy of pragmatism. Peirce and the Pragmatists, which include William James and others, believed that the rationalism of Descartes had a dangerous element of hubris. The worship of rationality could lead to deluding oneself to the reliability of one’s thoughts. Prof. Smyth but it this way—your grandmother is right. She believes in certain things. When you ask her to justify her beliefs she shrugs and says she can’t. Some things you do because that’s just the way they’ve always been done. You feel superior to your grandmother because you only do things that are rational. If you can’t justify something via reason, you simply reject it. But your grandmother (and Hayek), were on to something. Norms of behavior that survive, survive because they’re effective even when no one understands why.

Prof. Smyth discussed the Cartesian belief that you should examine every one of your beliefs. If it passes the test of reason, keep it. Otherwise, throw it out. Seems reasonable. But the pragmatists argued that that was akin to examining the planks of your boat while you were at sea. Tearing them out because they look imperfect is the road to ruin. It’s particularly true when you’re less than objective in deciding whether to reject or accept a belief.

Smyth quoted Benjamin Franklin: When fortresses and virgins get to talking, the end is in sight. That is—when you’re besieged, once you start negotiating, it’s easy to talk yourself into giving in and finding a way to justify it as the right thing to do.

All of which is to say that we shouldn’t pretend to be scientific when we’re only doing something that has the veneer of science. That’s much more dangerous than saying, I don’t know or we can’t answer that question.

I certainly didn’t mean to imply that Ian Ayres or John Lott (whose work came up in the conversation with Ayres) are biased researchers. I also did not mean to imply that data and evidence are irrelevant in how we form our beliefs about what is true. Or that our biases never get overturned. The example of Friedman and Schwartz’s Monetary History of the United States is the gold standard. Facts can be decisive. Statistical analysis can persuade. But I am struck by how few controversial viewpoints in the field of economics have been accepted based on sophisticated statistical analysis. By sophisticated analysis, I mean for example, the use of instrumental variables with a data set that has limited information about the myriad of factors that affect the variable we care about.

Where does that leave us? Economists should do empirical work, empirical work that is insulated as much as possible from confirmation bias, empirical work that isn’t subject to the malfeasance of running thousands of regressions until the data screams Uncle. And empirical work where it’s reasonable to assume that all the relevant variables have been controlled for. And let’s not pretend we’re doing science when we’re not.


Add a Comment    Share Share    Print    Email

Previous post:

Next post: