Fancy empirical work

by Russ Roberts on June 14, 2009

in Data

In the last part of this post, I issued a challenge:

I continue to ask the question: name an empirical study that uses
sophisticated statistical techniques that was so well done, it ended a
controversy and created a consensus—a consensus where former opponents
of one viewpoint had to concede they were wrong because of the quality
of the empirical work.

I then discussed why The Monetary History of the United States while an extraordinary piece of empirical work, didn't count. It wasn't fancy regression analysis, but rather a careful marshalling of facts. Not the same thing. I concluded:

I am open to other suggestions. I'd like one example, please. One
example, from either micro or macro where people had to give up their
prior beliefs about how the world works because of some regression
analysis, ideally usually instrumental variables as that is the
technique most used to clarify causation.

One example. There should be dozens. Or hundreds. But I'll take one.

Both Tyler Cowen and Bryan Caplan responded to the challenge. And there were a lot of interesting comments. Here is Tyler's list:

1. The interest-elasticity of investment is lower than people once thought.

2. We have a decent sense of the J Curve and why a devaluation or depreciation doesn't improve the trade balance for some while.

3. Dynamic revenue scoring tells us over what time horizon a tax cut is partially (or fully) self-financing.

4. Most resale price maintenance is not for goods and services involving significant ancillary services.

5. More policing can significantly lower the crime rate (that one does use instrumental variables).

6. The term structure of interest rates is whacky.

Bryan's answer is the entire field of behavioral economics.

Lauren Landsburg, in a very thoughtful email, suggests Time on the Cross, Fogel and Engerman's study of slavery.

Most or all of these observations miss the point, or at least the point I was trying to make.

Empirical work is very important.

Facts matter.

A careful study of the facts can have tremendous influence.

Sophisticated regression analysis can narrow our guesses as to magnitudes. But I don't think we need fancy regression to conclude that people aren't always rational. Or that police can reduce crime. Or to look at the nature of resale price maintenance. On the stuff where people have priors and bias—such as the dynamic impact of taxes on revenue—I don't think the empirical evidence is very convincing of the skeptic.

I understand that science moves slowly and that people at the margin are who eventually count.

But I really don't think the empirical record of sophisticated empirical work is very impressive. In fact, I think I could make a case that sophisticated empirical work is most productive for publishing papers and less productive at establishing truth or useful findings that are reliable.

Be Sociable, Share!



19 comments    Share Share    Print    Email


James D. Miller June 14, 2009 at 3:55 pm

Do you have examples of where high level economic theory convinced many economists to change their views on an important real world issue?

Alex June 14, 2009 at 3:58 pm

Empirical work does not have to conclusively resolve an important open question to be valuable. This is asking too much and is at odds with how theory and empirical work feedback upon one another in economics and other sciences as well.

Careful empirical work should shed light on where theory fails. New theory should explain the gaps in existing theory exposed by empirical investigation. The marginal benefit of solving the next most important empirical question ought to equal the marginal benefit of solving the next most puzzling theoretical result.

A better measuring stick for how productive empirical work has been over the course of the last 30 years would be: What theoretical questions would not have existed had empirical papers X,Y and Z never been published? For instance, Mehra and Prescott (1985) surely didn't conclusively resolve any controversies, but it opened up a vast field of new questions about macroeconomics and asset pricing. This interpretation is inline with Bryan's point: We are now able to consider and resolve a wide variety of new questions because of empirical advances.

This feedback loop between theory and empirical work is standard in most other scientific fields as well. We only see deviations when the research production function is fundamentally different. For example, in high energy physics, we might well see a major confirmation or rejection of current theory as a result of the LHC. However, in this situation it costs billions to create the new data. It is natural here that we observe major problems resolved. The scientific benefit to running these tests ought to be worth billions!

K Ackermann June 14, 2009 at 5:44 pm

In order to sway a person's belief, they have to be hit with some new set of truths that are ideally self-evident, or at least provable by some respected majority.

Regression is not ideal for that beacuse the events already happened so there was some chance that people were already alert to the possiblility of what the test show.

If it leads to some new truthful insight that was never postulated before, then it was probably coincidence the bolt from the blue happened while crunch data.

The problem with doing regression in economics is that it is all dynamic. Economics generally does not take place outside human activity. We affect the system that affects us. That is a dynamic system, and free will almost guarantees there will be no totally stable periods in the system.

Don posted a link the other day to a pile of economic theory and data links. When I get the time, I want to do a Fourier analysis on suitable datasets just to see what shows any periodicity.

It's a nightmare. I can tell you right now if I was to look for correlations with taxes, I probably would find nothing anybody wants to hear, because the data reeks of inconsistency. The data shows that sometimes taxes hurt, and sometimes they appear to help. It means nothing.

If anyone knows of fine-grained economic data out there, please let me know.

P.S. This is not a put down to the people who offered up the cases above. That was some good application to a tough question, and only you know how much you had to stretch to make the fit.

vikingvista June 14, 2009 at 6:33 pm

"I don't think we need fancy regression to conclude that people aren't always rational. Or that police can reduce crime. Or to look at the nature of resale price maintenance."

Powerful point.

Lee Kelly June 14, 2009 at 6:59 pm

I laughed when I read Bryan Caplan's response.

I mean, something is very seriously, and quite horribly, wrong with economics, if it took a fancy statistical study to convince everyone that humans aren't completely "rational".

mandeville June 14, 2009 at 8:22 pm

The answer is zero because statistics regarding human action can only confirm what is already understood about human behavior or human nature.

Statistical analysis can only verify what happened in the past. History never repeats itself. For example, you can study past price levels for various products, but they don't have anything to do with what the future prices of those products will be.

If you extend the analysis to human behavior, you don't need statistics to prove what is already known about human nature. For example, we know of the disutility of labor, or the law of diminishing marginal returns, etc. These are general laws of human action that we can discover through reflection of ourselves. We don't need to study our past actions to discover these things about ourselves. We know that man adapts his behavior to facts around him that are in a constant state of flux. His choice of action is rarely the same from one day to the next, let alone from year to year. Even if in a controlled study one can make a man's environment the exact same now as it was a year ago, he wouldn't act the same. This is because over time he has changed physically and mentally causing the needs of his ego to be different.

One doesn't need a statistical study to prove the disutility of labor, but it might be helpful in order to convince a fool. If one thinks that just because 75% marginal tax rates caused people to work 40% less in the past, it should be the same in the future, one is mistaken. It is known that people will work less, and that is it.

By the way, every human act is rational to the actor at the moment of action. That is a law of human nature too. What may look irrational to an observer is not to the actor. The actor may reflect back and think he made poor choices, but those conclusions are based on a different set of valuations than what went through his mind and senses at the moment he acted.

vikingvista June 14, 2009 at 9:30 pm

"Statistical analysis can only verify what happened in the past. History never repeats itself"

As a general statement, this is obviously false. Otherwise, time would make everything utterly incomprehensible–literally.

Complex historical events never repeat themselves EXACTLY. It is the similarities that people use to make experimental conclusions. It is the differences that people use to deny those conclusions.

Other types of persuasive reasoning is always needed. Particularly persuasive are relations for which no exception has been observed–the laws of nature. Even more persuasive are statements for which no exception can be imagined–the so-called categorical or apodictic truths.

However, all statements deserve skeptical scrutiny, and connecting complex results to broadly-accepted basic truths can be illusory.

"The actor may reflect back and think he made poor choices"

This just supports the statement that humans do not always act rationally. Maybe that was your point.

Lee Kelly June 14, 2009 at 10:22 pm

Statistical analysis can only verify what happened in the past. History never repeats itself. For example, you can study past price levels for various products, but they don't have anything to do with what the future prices of those products will be. – mandeville[/blockquote]

Don't confuse inferential valdity with truth. While there is no logical implication, that does not mean that there is no relation.

Abstractions from previous price changes may very well be very informative about future price changes, but only if such abstractions are true, a fact inderivable from the previous prices changes themselves.

Lee Kelly June 14, 2009 at 10:22 pm


I made a mistake with the tags.

Daniel Kuehn June 15, 2009 at 6:32 am

Those are great examples they raise, but I think the point still is what one commenter – i believe K Ackermann – posted on your last thread: that you set too high a bar for empirical work, and that's not "the point" of empirical work in any science.

Lobby Lloyd June 15, 2009 at 7:41 am



You're claim that you don't "need" fancy empiricism to prove more police cause less crime is at odds with the fact that it was a contentious point that theory and simple empiricism could not settle. Are you willing to say that you find the results of his paper "incredibly un-credible"? If so, I think the you should provide your colleague Alex with specific criticisms. It's not much of a criticism to say that some people won't ever be convinced by fancy empiricism; that's a criticism of them if you can't also criticize the work itself. If you won't make this claim about Alex's paper then I think you need to move that goalpost a little further still and concede your criticism is weaker then you irst contended.

Bob D June 15, 2009 at 12:26 pm

How about the numerous studies that show that Charter Schools and Vouchers are a better way of increasing the educational achievement of inner city children. I would direct anyone interested to check out The Friedman Foundation for Educational Choice. The only people who refuse to accept these facts are the various interest groups who are adversely affected by the ramifications that must follow.

MnM June 15, 2009 at 12:40 pm

But I really don't think the empirical record of sophisticated empirical work is very impressive.

Is the empirical record of the empirical record of empirical work impressive? ;o)

Sorry, I just couldn't help myself.

I still contend that the problem is that many economists are too impressed with their cute statistics. We need to understand them better, not shun their use.

Daniel Kuehn June 15, 2009 at 12:53 pm

MnM -
Re: "I still contend that the problem is that many economists are too impressed with their cute statistics. We need to understand them better, not shun their use."

I think you hit the nail on the head. I was presenting a paper at a conference a couple weeks ago – I suppose there were about twenty of us presenting. I along with maybe two or three other people were the only ones without an IV paper. I felt a little self-conscious at first – it was already tough enough since I'm such a young researcher.

Then I realized something – out of all the presenters who did have IV models, nobody in there believed the instrument except for the presenters themselves. Perhaps there were a couple exceptions, but it was pretty much across the board. It made me feel better – perhaps my approach was more simplified, but it was no less convincing in people's mind than anything else that was presented that weekend.

MnM June 15, 2009 at 1:31 pm

I can't say I'm surprised, Daniel. I've done similar things in less formal settings (I'm not a professional researcher).

I remember a conversation in my econometrics class that always stuck with me. The professor said something like, "all models are wrong; and they're wrong by definition".

The class was a little stunned; not what we expect from an econometrics teacher. But, regardless, he was right. Models and studies approximate reality and help us understand it. To my knowledge, they were never meant to act as some kind of graphical mirror reflecting some absolute image. They aren't true in any objective sense of the word.

Randal Samstag June 15, 2009 at 4:52 pm

Just a quick reaction on empirical falsification. I think a great example of that is Nicholas Kaldor's refutation of the views of Mrs. Thatcher's Financial Secretary of the Treasury (Mr. Lawson) to the effect that "the PBSR and the growth of the money supply and interest rates are very closely related." In Kaldor's Evidence to the Treasury and Civil Service Committee of July 1980 (published in the book, The Sourge of Monetarism, 1982) Kaldor presents regressions that he maintains "show conclusively that the role of the unfunded PBSR was quite insignificant (to the growth of the money supply); it explains only 5 percent of the change in LM3 in the last fourteen years." (p 92, see Table X on p. 93)

In the same book Kaldor presents more old-fashioned refutations of Friedman's recapitulation of the quantity theory of money which was part of the "consensus" that you mentioned earlier. There he shows that the principal claims of monetarism (the invariance of the velocity of circulation and the exogenous, rather than enogenous, nature of the money supply) are not true for a credit-money economy. As a student of Hayak and translator of several of his economics papers and books, I wonder why Kaldor's name is not much mentioned here?

Juan Carlos June 16, 2009 at 12:48 am

how about Daron Acemoglu's, Simon Johnson's and James Robinson's series of articles testing the hypothesis that institutions are te cause of long run growth?? they use instrumental variables in a variety of historical examples and prove that is in fact the truth.
i think the consensus in the profession is that they (along with others, for example Engerman and Sokoloff) beat the 'geography is the cause' people

{ 2 trackbacks }

Previous post:

Next post: