A few more empirical thoughts

by Russ Roberts on October 29, 2007

in Data

Colleague Robin Hanson gives me a hard time over at Overcoming Bias (reacting to this earlier post):

If Russ relies little on data to draw his conclusions, then on what
does he rely?  Perhaps he relies on theoretical arguments.  But can’t
we say the same thing about theory, that we mainly just search for
theory arguments to support preconceived conclusions?  If so, what is
left, if we rely on neither data nor theory? 

Try saying this out loud: "Neither the data nor theory I’ve come across
much explain why I believe this conclusion, relative to my random whim,
inherited personality, and early culture and indoctrination, and I have
no good reasons to think these are much correlated with truth."  That
does not seem a conclusion worth retaining.  If this is really your
situation, you should move to a nearly intermediate position of
uncertainty.   Either you should believe that truth-correlated data or
theory has substantially influenced your belief, or you should retain
only a very weak belief.

In this week’s EconTalk podcast I added this postscript to try and clarify what I was saying:

In last week’s podcast I spoke with Ian Ayres about the power and limitations of data analysis. Ayres emphasized the power and I kept mentioning the limitations, especially in the postcript I added after the interview. I want to clarify a few issues. My basic point was that when it comes to high-powered sophisticated statistical techniques, our biases as researchers and as consumers of that research often triumph over truth. The truth is elusive in complex systems with many things changing at once. It’s hard to isolate the independent effect of one particular variable. When scholars can run hundreds of multivariate regressions at very low cost, it easy to convince yourself that the results that confirm your prior beliefs are the “right “ results. The ones that failed must be the “bad ones.”

When I was in college at U of NC, I took a wonderful course from Richard Smyth where I learned about the American philosopher Charles Peirce and the philosophy of pragmatism. Peirce and the Pragmatists, which include William James and others, believed that the rationalism of Descartes had a dangerous element of hubris. The worship of rationality could lead to deluding oneself to the reliability of one’s thoughts. Prof. Smyth but it this way—your grandmother is right. She believes in certain things. When you ask her to justify her beliefs she shrugs and says she can’t. Some things you do because that’s just the way they’ve always been done. You feel superior to your grandmother because you only do things that are rational. If you can’t justify something via reason, you simply reject it. But your grandmother (and Hayek), were on to something. Norms of behavior that survive, survive because they’re effective even when no one understands why.

Prof. Smyth discussed the Cartesian belief that you should examine every one of your beliefs. If it passes the test of reason, keep it. Otherwise, throw it out. Seems reasonable. But the pragmatists argued that that was akin to examining the planks of your boat while you were at sea. Tearing them out because they look imperfect is the road to ruin. It’s particularly true when you’re less than objective in deciding whether to reject or accept a belief.

Smyth quoted Benjamin Franklin: When fortresses and virgins get to talking, the end is in sight. That is—when you’re besieged, once you start negotiating, it’s easy to talk yourself into giving in and finding a way to justify it as the right thing to do.

All of which is to say that we shouldn’t pretend to be scientific when we’re only doing something that has the veneer of science. That’s much more dangerous than saying, I don’t know or we can’t answer that question.

I certainly didn’t mean to imply that Ian Ayres or John Lott (whose work came up in the conversation with Ayres) are biased researchers. I also did not mean to imply that data and evidence are irrelevant in how we form our beliefs about what is true. Or that our biases never get overturned. The example of Friedman and Schwartz’s Monetary History of the United States is the gold standard. Facts can be decisive. Statistical analysis can persuade. But I am struck by how few controversial viewpoints in the field of economics have been accepted based on sophisticated statistical analysis. By sophisticated analysis, I mean for example, the use of instrumental variables with a data set that has limited information about the myriad of factors that affect the variable we care about.

Where does that leave us? Economists should do empirical work, empirical work that is insulated as much as possible from confirmation bias, empirical work that isn’t subject to the malfeasance of running thousands of regressions until the data screams Uncle. And empirical work where it’s reasonable to assume that all the relevant variables have been controlled for. And let’s not pretend we’re doing science when we’re not.

Be Sociable, Share!



Add a Comment    Share Share    Print    Email


Bruce G Charlton October 30, 2007 at 2:11 am

Science gets its power not from the puritanical mental hygeine of its individual participants, but from the competitive and selective process of science.

As DL Hull describes in Science as a Process, individual scientists are of many personality types – some are extremely partisan in favour of their own work.

This is lucky for science, because if it had to depend on the mental fastidiousness of individual scientists it would not get far or go fast.

In other words, science is just like economics in the sense that individual CEOs don't need to be objective about the social utility of their organization – the good effects come from the competitive and selective processes of the market.

saifedean October 30, 2007 at 2:20 am


This is a very interesting discussion, and I largely agree with most of what you say. However, allow me to disagree with you on your pragmatist position, and will demonstrate why with a hypothetical argument:

Farm subsidies have been part of American economic policy for decades now, and America has never run out of food. Data, theory and maths can be used to demonstrate all sorts of different impacts of removing farm subsidies. But since they have been "working" so far and we have not run out of food; it is the pragmatist position to stick to what has been working so far. Therefore, I advocate continuing, and increasing, farm subsidies. After all, if this institution has persisted for decades, then there is a reason for it. At different points in time, the same could've been said about price controls, import quotas and tariffs and all protectionist economic policies.

If one were to take your pragmatist approach seriously, one could argue to maintain belief in anything, on the ground that it has survived for a while. There is no end to this, one can use it to justify belief in Santa Claus, the Tooth Fairy, God, the flatness of the earth, and the Loch Ness monster. We would in fact still be believing in all the thousands of Gods that humanity has created.

What I find most problematic with this position is that its advocates obviously are not calling for believing in everything that has been around for a while, but they invoke it only when it suits them, and only when they can not marshal enough evidence to support their position.

The question really is, when do you decide it is not wise to question something and take it on blind faith out of pragmatism, and when do you decide to investigate it empirically. My personal view is to start by investigating everything and only believing what the evidence supports. My problem with the pragmatist position is that it does this only when it finds the evidence to its liking.

And yes, if indeed I was on a working boat I would not take it apart in the middle of the sea to investigate whether the planks are working; but believing in the efficacy of gun ownership, grandma's version of God, or farm subsidies are completely different stories, and the analogy with the boat is not very relevant.

john Lott October 30, 2007 at 2:34 am

1) While you still haven't acknowledged it, this concern about confirmation bias in right-to-carry work is part of the reason that I have used Ed Leamer's work on specification sensitivity. This is also why I provide my data whenever asked to anyone who asks, even those who don't return the favor (http://johnrlott.tripod.com/postsbyday/may8.html). Sharing data is the ultimate way of ferreting out problems, though that does not mean that critics are always that honest.
2) It is also one reason that I worked with David Mustard, who had quite different views (http://www.terry.uga.edu/~dmustard/culture-data.pdf) than I suspect most would guess that he had when we started our work. In my research, I have frequently tried to work with people whom one might believe would disagree with me. It is also more fun to work with them on the papers.
3) Russell also seems unaware that there are no refereed academic papers that claim that right-to-carry laws are associated with more accidental deaths, suicides, or violent crime. My question is this: before this research, how many academics would have believed that at least some refereed research would show that right to carry laws did not increase accidents, suicides, or violent crime rates? I think that most would believe that at least some would find these results. For those interested, here (http://johnrlott.tripod.com/postsbyday/RTCResearch.html) are some papers that show that RTC laws produce benefits with regard to violent crime.
4) Ayres' claims about empirical work in his book are very old claims. I would have been interested if Russell had tried asking Ayres some probing questions such as: why haven't any car companies started putting LoJacks on all their cars as a way of internalizing these externalities? If Porche put this on all of its cars and it worked, theives would stay away from Porche. It would capture all the benefits from the devices it put on its cars. If there really was a $5,000 per device benefit, why aren't car companies tripping over each other to install the device? How many years have these companies had to learn about the wonderful benefits? What about the claims that insurance companies make about the small benefits from the device in that the cars that they get back are already stripped?
5) Russell in his interview was completely unprepared to challenge Ayres on false claims that he has made regarding the right-to-carry research. Just take Ayres' claim about errors in the data affecting my results when he was actually referring to a paper published by two other researchers.

John Lott October 30, 2007 at 4:21 am

Take your last paragraph. Anyone can make such broad claims. Be specific. What is it that you think should have been done regarding right-to-carry laws? What should have been controlled for that wasn't? What combination of control variables should have analyzed that wasn't?

How about the same thing for LoJack? Though, I think that is a much easier task.

Lee Kelly October 30, 2007 at 7:51 am


Contrary to what Robin Hanson suggests, I do not think that we need to rely upon anything, neither experiment, data or theory. If we are interested in the discovering the truth, then I think talk of relying on this or that is mistaken, rather we should not rely on anything.

There is no telling where we might err, and our reliance upon anything, presuming we do rely upon something, has no bearing on the truth. I think, rather, that it would be an impediment to rational investigation, to introduce 'reliance' as a standard to select among rival propositions.

The degree of reliance we attach to data or theory seems little more than a subjective relation to that data or theory i.e. a feeling of dependence. That such feelings will vary from one to another would seem to render it an odd preoccupation for those interested in the truth.


Lee Kelly October 30, 2007 at 8:12 am

"Neither the data nor theory I've come across much explain why I believe this conclusion, relative to my random whim, inherited personality, and early culture and indoctrination, and I have no good reasons to think these are much correlated with truth." – Robin Hanson

I would be interested to find out what good reasons Robin Hanson supposes that he has, and whether he has any good reasons to suppose that they are good reasons rather than bad reasons. I suggest, that Robin Hanson, much like everybody else, has no good reasons to believe anything.

In fact, I do not think that "good reasons" have anything to do with reasoning: a position for which I have no good reason to take. Instead, it is my choice to value reason, truth, and rational discussion as an alternative to violence and coercion. I take reasoning seriously, even if I have no good reason to do so.

Those who need a "good reason" to make a choice or decision have, either explicitly or implicitly, submitted to an authority. The authority is one which is is relied upon to provision good reasons to take a particular position or belief.

The traditional problems of philosophy arise when an authority is not strong enough to permit a common sense belief, such as when Hume discovered that sense experience could not sanction the belief that the sun will rise tomorrow.

Thankfully, there is no need to follow Hume's example.


Lee Kelly October 30, 2007 at 8:42 am

Either you should believe that truth-correlated data or theory has substantially influenced your belief, or you should retain only a very weak belief.

I am quite unsure what "truth-correlated data or theory" is. In order to correlate data or theory with the truth, we would need to determine what the truth is. However, if we had already determined what the truth is, presupposing we had such a method, then why would we care about the data?

I am also puzzled by the concern with what Russ Roberts believes, and even the strength of that belief. I might believe some things very strongly, others very weakly, or some not at all, but whatever the strength of my belief, it has nothing to do with the truth of the proposition which is believed. In fact, I have difficulty beleiving in some things even though I think they are true, such as quantum or cosmological theory.

I find it doubly puzzling to couch this criticism of Dr. Roberts as a matter of ethics. There is no claim that any theory or conjecture is false, but rather it is suggested that particular beliefs are wrong to hold.

This concern with the strength of belief has little to do with the search for truth. It matters not one jot how strong or weakly I believe in a proposition, or at all. The truth is objective, and the consequences of acting on a proposition are the same irrespective of the extent which they are believed.

I would rather act on the truth than a mistaken belief.


AO October 30, 2007 at 10:47 am


You said in your interview with Bob Frank that economics is more of an art than a science. Do you feel that in general economists proffer hypothesis that aren't testable and falsifiable? Popper argues in the Evolution of Knowledge that in order for the knowledge to continue towards the truth, hypothesis which are bold, testable, and important must be offered and then aggressively criticized. Individual unbiasedness isn't as important as the unbiasdness of science as a method, and the assurance of economics as a science.

Tom S. October 30, 2007 at 12:40 pm


I'm perplexed by your use of pragmatism/pragmaticism here. Warranted Assertability grew from the insufficient power of induction. In this piece, pragmatism is presented like folk wisdom.

Henri Hein October 30, 2007 at 2:42 pm

Like Prof. Roberts, I'm also sympathetic to pragmatism, because it helps us to understand the limits of reason.

However, to the extent pragmatism says that just because one human mind cannot model reality, we cannot know truth at all, I think it's wrong. I haven't seen any scholarly work linking the two, but it seems to me that post-modernism takes its root in pragmatism.

If a ruler shows my desk to be three feet wide, I see little reason to disbelieve it. The star example is illustrative. If one astronomer measures the position of one star, from one location, using one instrument, the result is likely to be off. If thirty different astronomers, in thiry different locations and using thirty different instruments, measure the position of the same star, and you discard the outliers, you will be able to determine the exact position of the star.

Truth, in a more general sense, must be determined using similar methods.

Henri Hein October 30, 2007 at 2:43 pm

OT: Have I been blacklisted? The Cafe is refusing to remember my personal info… :(

Russ Roberts October 30, 2007 at 4:04 pm


You have not been blacklisted. It doesn't even remember the personal info of the proprietor! Don't know why. If anyone has any insight into the cause of this problem, please let me know.

AO October 30, 2007 at 9:01 pm


If economics is an art and not a science, does it have any advantage in truth approximation over, say, sociology?

Also, you pointed out in the podcast that you're not accusing Lott, Levitt, or Ayers of performing the sort of biased art of economics you warn about. So can you point to examples of specific research which has those negative qualities and has also been accepted as "good" research? If the kind of bad economics your describing is rejected by consensus, then isn't it not a criticism of the field, but of specific economists? If that's the case, then why denigrate economics as an art instead of just some economists as artists?

Lee Kelly October 31, 2007 at 8:56 am

I think half the problem with this is that economists become engaged in the argument from efficiency, and confuse a economic question with the policy decision.


I think that the science of economics will be polluted as long as people continue to argue policy decisions on these terms.

TGGP October 31, 2007 at 8:51 pm

Robin Hanson has a reply to your reply here.

David Zetland November 7, 2007 at 9:50 pm

Russ — I cannot find the attribution to "when fortresses and virgins….". Neither Franklin nor anyone else appears to have said that. It's a good quotation, so do we attribute it to Smyth?

Previous post:

Next post: