My response to Robin's latest

by Russ Roberts on November 1, 2007

in Data

Robin Hanson continues to give me a hard time. First he quotes me:

scholars can run hundreds of multivariate regressions at very low cost,
it easy to convince yourself that the results that confirm your prior
beliefs are the "right" results.  The Pragmatists … believed that the
rationalism of Descartes had a dangerous element of hubris. … Your
grandmother is right. She believes in certain things. When you ask her
to justify her beliefs she shrugs and says she can’t. … Norms of
behavior that survive, survive because they’re effective even when no
one understands why.


The Cartesian belief that you should examine every one of your
beliefs … the pragmatists argued that that was akin to examining the
planks of your boat while you were at sea. … All of which is to say
that we shouldn’t pretend to be scientific when we’re only doing
something that has the veneer of science. That’s much more dangerous
than saying, I don’t know or we can’t answer that question. … I also
did not mean to imply that data and evidence are irrelevant in how we
form our beliefs about what is true. Or that our biases never get

Then Robin responds, combining my observations on pragmatism with my argument that we are unlikely to be able to use econometric analysis to settle the question of whether concealed handguns reduce crime:

I’m still confused.  Does Russ think his life will "sink" if he further
considers his reasons for his believing in concealed handguns, even
though it was he who raised this issue?  Does he think only scientists
should want reasons for their beliefs?  Does he believe concealed
handguns deter crime because his grandma had hidden guns and lived a
long life?  Or is it because grandma’s society had them and was

Yes, if one prosperous society had concealed handguns that is at least
weak evidence for the reasonableness of that behavior.  But since other
prosperous societies forbid concealed handguns, I don’t see how you can
have more than a weak belief on this basis, unless you rely on one of
those complex data analyzes Russ distrusts.  Russ doesn’t seem to think
there is simple clear data on concealed handguns, and he hasn’t taken
up my suggestion to claim that he is persuaded by theory, either
complex and subtle or simple and clear.   And yet Russ still seems to
retain a strong belief.  This seems literally unreasonable to me.

Robin is taking my observation about pragmatism and applying it to handguns. I didn’t mean to. I brought up pragmatism in order to highlight the general dangers of excessive faith in reason. Assuming that econometric analysis always trumps an anecdote is an example of the potential dangers of econometric analysis. Yes, relying on anecdotes is lousy science. But lousy econometrics is lousy science, too. What my podcast with Ayres made me realize is that lousy econometrics may be the norm rather than the exception.

Such cynicism can come cheap. It also seems to leave us with anecdotes. Well, there’s also common sense, intuition and general lessons gleaned from experience and empirical work that is less prone to manipulation.

So again, my question to my better-read colleagues in the profession–give me an example of a statistical analysis that relies on instrumental variables, for example, that is done well enough, that is so iron-clad, that it can reverse the prior beliefs of a skeptic.

Consider this excerpt from the summary of a paper by Dube, Eidlin and Lester that finds that Wal-Mart lowers wages when it comes to town:

A potential problem with studying store openings to
estimate the impact on wages is that  Wal-Mart does not choose randomly
where to expand.  If Wal-Mart’s expansion into  local markets were
random throughout the United States across the ten-year period of
study, then we could simply look at what happened to wages in counties
after Wal-Mart’s entry as compared to before.  But Wal-Mart may have
taken into account several factors  for expanding into certain markets
and not others, including the cost of labor in those  areas at that
time.  Economists call this problem “selection bias.” In other words,
Wal-Mart’s own criteria for expansion into certain markets may
interfere with our ability to  test for a causal relationship between
Wal-Mart entry and a change in local wages.   

In this paper, we devise a novel way to resolve this problem.  We take
the fact that WalMart store openings spread out over time starting from
Arkansas and moved outward to  the coasts, much like a ripple from a
drop of water.  In other words, the farther a county was from
Arkansas—ground zero for Wal-Mart—the later it experienced the
Wal-Mart  growth spurt.  This was an actual pattern of expansion, one
that made sense for the  company as it focused on utilizing its
distribution networks most effectively and lowering  overall costs of
expansion.  By following this ripple of store openings across the
country  and over time, we are able to test whether retail wages follow
a similar ripple pattern.   Looking at store openings based on both how
far the county is from Wal-Mart’s “ground  zero” and the year in
question, our estimates are not subject to the selection bias that is
often a problem for similar studies. 

Results and Implications 

We find strong evidence that in urban and suburban counties (counties
that are part of a  Metropolitan Statistical Area), a Wal-Mart store
opening led to a 0.5% to 0.8% reduction  in average earnings per
workers in the general merchandising sector.  This finding is
consistent with Wal-Mart jobs paying about 10% lower wages than the
jobs they  displaced.  A Wal-Mart store also reduced average earnings
per grocery worker in that  county by 0.8% to 0.9%.  Taking both wage
and possible employment effects into  account, we found that a single
Wal-Mart store reduced the total earnings of general  merchandise and
grocery workers in that county by about 1.3%. In rural counties, the
pattern was different.  A Wal-Mart store opening there was associated
with an increase in the average earnings per general merchandise worker
and a  decrease in the average earnings per grocery worker.  However,
combining wage and  employment effects, the impact on the total
take-home pay of the affected retail  workforce was a wash.

Are you convinced by this evidence? I’m not. Does it lead you to conclude that large cities and suburbs should keep out Wal-Mart in order to protect workers?

At a conference, I asked
one of the authors (Dube) why wages in urban areas fell more than in
rural areas. Wouldn’t you expect an urban area with lots of jobs to be
less affected by the entry of Wal-Mart than in a rural area? Even if you believe that Wal-Mart has some monopoly power, wouldn’t it be smaller in larger cities than in rural areas? His
response: only if you have a neoclassical view of the world. We got
interrupted before I could ask him what his view of the world was. But
I suspect his view of the world is agnostic–he lets the data speak.
The truth is there. I say no. The illusion of truth is there. Using
distance from Bentonville is clever but does it really work? How would you know?
Does it convince you that Los Angeles should keep out Wal-Mart? (I don’t know what the authors actually believe on this latter question. I do know that some people have interpreted their results as evidence for keeping Wal-Mart out of an area.)

Here’s how my biased neoclassical view works. One, I’m not convinced
that Wal-Mart reduces wages because I’m not convinced that distance
from Bentonville is a good instrument. What other instruments did you
try first? (Here is Emek Basker’s paper that critiques a paper that uses a similar methodology.) But even if you could convince me that it is a good
instrument, my next argument is that Wal-Mart might lower wages, but
that’s because it draws low-wage workers into the labor force and the
lower overall wages you observe are a composition effect. Those lower wage are
a blessing not a curse. You won’t convince me that increasing the
demand for labor is going to lower the wages of workers who already are
employed. Their wages can still go up when Wal-Mart arrives. And even if wages are lower in places where Wal-Mart arrives, I’m still going to argue that causation runs the other way, that Wal-Mart’s entry is a response to lower wages, that you didn’t hold everything else constant, your instrument is flawed. You’re going to find it very hard to convince me that an increase in demand holding everything else constant lowers wages. I’m going to presume you really didn’t/couldn’t hold everything else constant.

So whose view is more scientific? I’m not so sure. I’m open to attack. Give me a hard time. I’m not interested in debating the merits of Dube, Eidlin, and Lester—if you think their paper is not a good example of instrumental variables, give me a
better one. Give me one that’s so well done it will cause me to doubt
my neoclassical world view or other views I hold on policy or human behavior. Is it possible? Is there an empirical study
on the other side that would convince Dube, Eidlin and Lester that
Wal-Mart is good for workers? Show me an example where sophisticated statistical analysis has won over the skeptics or trumped common sense because it was so well done.

For what it’s worth, I am less of a hard-core neoclassical economist
than when I left Chicago. Why? I suspect that various kinds of evidence
have persuaded me to take a richer viewpoint. So again, evidence and
facts matter.   

Finally, a commenter at Overcoming Bias, Constant, does a much better job than I
did explaining the differences between the Pragmatist and the Cartesian:

I will try to approximate the pragmatist position as it contrasts
with the Cartesian position. The Cartesian approach is to discard every
idea unless it can be verified. The point is to believe only things
which are certainly true, and to avoid believing falsehoods. It is a
maximally skeptical approach. In contrast, the pragmatist approach is
to keep every idea until it is falsified (or found wanting in some
other particular way). So the Cartesian gets rid of all the planks of
his boat that he can’t demonstrate to his satisfaction are necessary
for the boat’s operation, while the pragmatist keeps all the planks of
his boat unless he can demonstrate to his satisfaction that a
particular plank is unnecessary.

Applying this to the current case, the pragmatist approach says that
if you have a belief about guns and the evidence does not contradict
the belief, then keep the belief. The Cartesian approach says that if
you have a belief about guns and the evidence does not demonstrate this
belief, then discard the belief.


Add a Comment    Share Share    Print    Email

Previous post:

Next post: