My response to Robin's latest

by Russ Roberts on November 1, 2007

in Data

Robin Hanson continues to give me a hard time. First he quotes me:

scholars can run hundreds of multivariate regressions at very low cost,
it easy to convince yourself that the results that confirm your prior
beliefs are the "right" results.  The Pragmatists … believed that the
rationalism of Descartes had a dangerous element of hubris. … Your
grandmother is right. She believes in certain things. When you ask her
to justify her beliefs she shrugs and says she can’t. … Norms of
behavior that survive, survive because they’re effective even when no
one understands why.


The Cartesian belief that you should examine every one of your
beliefs … the pragmatists argued that that was akin to examining the
planks of your boat while you were at sea. … All of which is to say
that we shouldn’t pretend to be scientific when we’re only doing
something that has the veneer of science. That’s much more dangerous
than saying, I don’t know or we can’t answer that question. … I also
did not mean to imply that data and evidence are irrelevant in how we
form our beliefs about what is true. Or that our biases never get

Then Robin responds, combining my observations on pragmatism with my argument that we are unlikely to be able to use econometric analysis to settle the question of whether concealed handguns reduce crime:

I’m still confused.  Does Russ think his life will "sink" if he further
considers his reasons for his believing in concealed handguns, even
though it was he who raised this issue?  Does he think only scientists
should want reasons for their beliefs?  Does he believe concealed
handguns deter crime because his grandma had hidden guns and lived a
long life?  Or is it because grandma’s society had them and was

Yes, if one prosperous society had concealed handguns that is at least
weak evidence for the reasonableness of that behavior.  But since other
prosperous societies forbid concealed handguns, I don’t see how you can
have more than a weak belief on this basis, unless you rely on one of
those complex data analyzes Russ distrusts.  Russ doesn’t seem to think
there is simple clear data on concealed handguns, and he hasn’t taken
up my suggestion to claim that he is persuaded by theory, either
complex and subtle or simple and clear.   And yet Russ still seems to
retain a strong belief.  This seems literally unreasonable to me.

Robin is taking my observation about pragmatism and applying it to handguns. I didn’t mean to. I brought up pragmatism in order to highlight the general dangers of excessive faith in reason. Assuming that econometric analysis always trumps an anecdote is an example of the potential dangers of econometric analysis. Yes, relying on anecdotes is lousy science. But lousy econometrics is lousy science, too. What my podcast with Ayres made me realize is that lousy econometrics may be the norm rather than the exception.

Such cynicism can come cheap. It also seems to leave us with anecdotes. Well, there’s also common sense, intuition and general lessons gleaned from experience and empirical work that is less prone to manipulation.

So again, my question to my better-read colleagues in the profession–give me an example of a statistical analysis that relies on instrumental variables, for example, that is done well enough, that is so iron-clad, that it can reverse the prior beliefs of a skeptic.

Consider this excerpt from the summary of a paper by Dube, Eidlin and Lester that finds that Wal-Mart lowers wages when it comes to town:

A potential problem with studying store openings to
estimate the impact on wages is that  Wal-Mart does not choose randomly
where to expand.  If Wal-Mart’s expansion into  local markets were
random throughout the United States across the ten-year period of
study, then we could simply look at what happened to wages in counties
after Wal-Mart’s entry as compared to before.  But Wal-Mart may have
taken into account several factors  for expanding into certain markets
and not others, including the cost of labor in those  areas at that
time.  Economists call this problem “selection bias.” In other words,
Wal-Mart’s own criteria for expansion into certain markets may
interfere with our ability to  test for a causal relationship between
Wal-Mart entry and a change in local wages.   

In this paper, we devise a novel way to resolve this problem.  We take
the fact that WalMart store openings spread out over time starting from
Arkansas and moved outward to  the coasts, much like a ripple from a
drop of water.  In other words, the farther a county was from
Arkansas—ground zero for Wal-Mart—the later it experienced the
Wal-Mart  growth spurt.  This was an actual pattern of expansion, one
that made sense for the  company as it focused on utilizing its
distribution networks most effectively and lowering  overall costs of
expansion.  By following this ripple of store openings across the
country  and over time, we are able to test whether retail wages follow
a similar ripple pattern.   Looking at store openings based on both how
far the county is from Wal-Mart’s “ground  zero” and the year in
question, our estimates are not subject to the selection bias that is
often a problem for similar studies. 

Results and Implications 

We find strong evidence that in urban and suburban counties (counties
that are part of a  Metropolitan Statistical Area), a Wal-Mart store
opening led to a 0.5% to 0.8% reduction  in average earnings per
workers in the general merchandising sector.  This finding is
consistent with Wal-Mart jobs paying about 10% lower wages than the
jobs they  displaced.  A Wal-Mart store also reduced average earnings
per grocery worker in that  county by 0.8% to 0.9%.  Taking both wage
and possible employment effects into  account, we found that a single
Wal-Mart store reduced the total earnings of general  merchandise and
grocery workers in that county by about 1.3%. In rural counties, the
pattern was different.  A Wal-Mart store opening there was associated
with an increase in the average earnings per general merchandise worker
and a  decrease in the average earnings per grocery worker.  However,
combining wage and  employment effects, the impact on the total
take-home pay of the affected retail  workforce was a wash.

Are you convinced by this evidence? I’m not. Does it lead you to conclude that large cities and suburbs should keep out Wal-Mart in order to protect workers?

At a conference, I asked
one of the authors (Dube) why wages in urban areas fell more than in
rural areas. Wouldn’t you expect an urban area with lots of jobs to be
less affected by the entry of Wal-Mart than in a rural area? Even if you believe that Wal-Mart has some monopoly power, wouldn’t it be smaller in larger cities than in rural areas? His
response: only if you have a neoclassical view of the world. We got
interrupted before I could ask him what his view of the world was. But
I suspect his view of the world is agnostic–he lets the data speak.
The truth is there. I say no. The illusion of truth is there. Using
distance from Bentonville is clever but does it really work? How would you know?
Does it convince you that Los Angeles should keep out Wal-Mart? (I don’t know what the authors actually believe on this latter question. I do know that some people have interpreted their results as evidence for keeping Wal-Mart out of an area.)

Here’s how my biased neoclassical view works. One, I’m not convinced
that Wal-Mart reduces wages because I’m not convinced that distance
from Bentonville is a good instrument. What other instruments did you
try first? (Here is Emek Basker’s paper that critiques a paper that uses a similar methodology.) But even if you could convince me that it is a good
instrument, my next argument is that Wal-Mart might lower wages, but
that’s because it draws low-wage workers into the labor force and the
lower overall wages you observe are a composition effect. Those lower wage are
a blessing not a curse. You won’t convince me that increasing the
demand for labor is going to lower the wages of workers who already are
employed. Their wages can still go up when Wal-Mart arrives. And even if wages are lower in places where Wal-Mart arrives, I’m still going to argue that causation runs the other way, that Wal-Mart’s entry is a response to lower wages, that you didn’t hold everything else constant, your instrument is flawed. You’re going to find it very hard to convince me that an increase in demand holding everything else constant lowers wages. I’m going to presume you really didn’t/couldn’t hold everything else constant.

So whose view is more scientific? I’m not so sure. I’m open to attack. Give me a hard time. I’m not interested in debating the merits of Dube, Eidlin, and Lester—if you think their paper is not a good example of instrumental variables, give me a
better one. Give me one that’s so well done it will cause me to doubt
my neoclassical world view or other views I hold on policy or human behavior. Is it possible? Is there an empirical study
on the other side that would convince Dube, Eidlin and Lester that
Wal-Mart is good for workers? Show me an example where sophisticated statistical analysis has won over the skeptics or trumped common sense because it was so well done.

For what it’s worth, I am less of a hard-core neoclassical economist
than when I left Chicago. Why? I suspect that various kinds of evidence
have persuaded me to take a richer viewpoint. So again, evidence and
facts matter.   

Finally, a commenter at Overcoming Bias, Constant, does a much better job than I
did explaining the differences between the Pragmatist and the Cartesian:

I will try to approximate the pragmatist position as it contrasts
with the Cartesian position. The Cartesian approach is to discard every
idea unless it can be verified. The point is to believe only things
which are certainly true, and to avoid believing falsehoods. It is a
maximally skeptical approach. In contrast, the pragmatist approach is
to keep every idea until it is falsified (or found wanting in some
other particular way). So the Cartesian gets rid of all the planks of
his boat that he can’t demonstrate to his satisfaction are necessary
for the boat’s operation, while the pragmatist keeps all the planks of
his boat unless he can demonstrate to his satisfaction that a
particular plank is unnecessary.

Applying this to the current case, the pragmatist approach says that
if you have a belief about guns and the evidence does not contradict
the belief, then keep the belief. The Cartesian approach says that if
you have a belief about guns and the evidence does not demonstrate this
belief, then discard the belief.

Be Sociable, Share!



Add a Comment    Share Share    Print    Email


Lee Kelly November 1, 2007 at 6:06 am

The problem with pragmatism is that we can always pose a question: useful for what?. It might be very pragmatic for a politician to believe and perpetuate an idea which is false, because that is the pragmatic thing to do from his position, relative to his goals and interests.

I think 'pragmatism' is a misleading term to use, especially since what you describe seems close to Popper's critical rationalism, which earned that name 'critical rationalism' because it was self-critical. That would seem to reflect your sentiments regarding faith in rationality.

In this view, we conduct rational investigation by conjecture and refutation. In other words, we do not draw, induce or justify our ideas from anything, neither pure reason or sense observation. Instead, we first conjecture or guess, and then try and refute our guesses by severe criticism and tests, a survival of the fittest, so to speak.

This is also how to solve Hume's problem of induction. I wrote this a little while back to a friend:

"Regarding the problem of induction, when analysed in the context of Popper's critical rationalism, where rationality concerns critical preference and not justification, the problem does not arise. In other words, critical rationalists do not presuppose any principle, such as that of induction, or the uniformity of nature. Instead, critical rationalists propose uniformity in nature through theories, and do not presuppose it to justify theories.

In other words, critical rationalists do not presuppose, or commit to, any principle of induction, or uniformity of nature. It may be true, that for scientific investigation to bear fruit, nature must be uniform, but there is no need to presuppose it, since it is entirely possible that there is no uniformity. The purpose of scientific investigation is the search for laws that express regularities, though science cannot promise that there is anything to find.

That there may be no scientific laws to find is a possibility, but not one that a critical rationalist is eager to accept, and thus makes a methodological decision: to search tirelessly for laws and regularities wherever they may be, to conjecture and experiment. Ironically, we arrive at the view that there is uniformity in nature, by trying our best to find disorder, to bring every theory which proposes order, to the severest tests and criticism.

The problem of induction originally arose because it was supposed that synthetic statements must be justified by sense experience i.e. derivable from sense-data, or singular statements. That the principle of induction, or the uniformity of nature, are obviously synthetic statements, yet also irreducible to sense-data, or singular statements, caused the problem. It seemed that we could have no rational justification for any statement about the future.

It is because critical rationalism employs sense experience as a means of criticism, rather than justification, that the problem of induction does not arise. In other words, the critical rationalist does not try to derive theories from sense experience, but tries to contradict theories with sense experience. The method is reversed, and the problem of induction dissolved."

There are many problems which arise. For example, what to do when faced with two rival but as yet unfalsified theories?

I think this problem can easily be solved by going back to our goals as scientists or rationalists i.e. discovering the truth. This is another excerpt from something I wrote to address this problem:

Testability is a way of forming a critical preference by favouring those theories which are more highly testable i.e. more falsifiable. Of course, that doesn't mean it is rational to prefer a more testable theory even if it has been falsified, our goal is to discover true theories, not highly testable theories. It is only when selecting between rival unfalsified theories that testability steps up as important.

The explanation is simple. The more highly testable a theory is, the more potential events it prohibits and thus the easier it is to falsify. Moreover, the more potential events a theory prohibits then the more informative, and better guide to future action it is. Such a rule is it not meant to imply that untestable, or less testable, theories are false; only that we should avoid, where possible, adopting theories which are difficult to test, since if such a theory is mistaken we should find it very difficult to refute it.

This final sentence is meant to capture the idea that people who hold to unfalsifiable theories, often come to resemble a dogmatist who is immune to rational debate, even if the person does not intend to take a dogmatic attitude.

In regard to economics, take the argument between Ayres and Lott, specifically Lott's claim that LoJack must be ineffective because insurance companies ignore it. In response Ayres simply says, "so what? insurance companies make mistakes, and they have made a mistake with LoJack!".

Now the question is: does Ayres' explanation of Lott's criticism increase or decrease the testability of Ayres' theory?. The answer, it seems to me, is that it decreases testability. In other words, there is now less which can contradict Ayres' original theory.

I think this is bad methodology. This is not to say that Ayres is wrong and Lott is right, but rather that we should try and expose our theories to criticism, not protect them from criticism by deflecting ralcitrant facts with ad hoc explanations.

Anyway, I should be working!


Lee Kelly November 1, 2007 at 6:29 am

Incidently. In the view sketched out above, the most useful way to characterise 'scientific', I think, is as a property of theories, where a theory is scientific if and only if it is possible to criticise that theory by experiment or observation.

Whether or not we render our ideas, beliefs, theories, etc. open to scientific criticism, is a choice we make. It is always possible to deflect or explain away facts we do not like, but it will not help us develop our knowledge or test our theories.

Jeffrey Rae November 1, 2007 at 7:05 am

I agree with your line of argument.

I have no doubt that Cartesian rationality does run a strong risk of hubris. And that hubris has been responsible for most of the really nasty things that human beings have done to each otherin the past 200 years: the Terror of the French Revolution, the Final Solution of the 3rd Reich, the Red Terror of Stalin, and the Cultural Revolution of Mao Zedong.

I have also have no doubt that empirical quantification cannot settle most of the questions that are asked of it. Human behaviour is too complex and the data that records it too imperfect for statistical techniques to isolate the various components. Even if they do there is a one in twenty (or hundred) chance that any statistically positive result is due to happenstance. I not that most of the results of most epidemiological studies cannot be replicated.

Accordingly I do not accept the results obtained by Dube et al at face value, any more than I accepted the Card and Kreuger results on the positive employment effect of raising the minimum wage. Both sets of results could be true but more evidence is needed, to my mind, to establish that there are good reasons to suspend the strong presumption in downward sloping demand curves in these two cases.

A further reason for doubt is that Cartesian rationality can establish the existence of some things, eg the existence of love in a (non-physical) relationship. The fact that the people affected believe it exists to the point where they act on the basis of their belief to their material detriment, without statistically significant evidence is, however, enough for me.

I think that it is called Pragmitism.

Jeffrey Rae November 1, 2007 at 7:09 am

In the 2nd last para. of my post I meant to say that Cartesion rationality 'cannot' establish existence of some things.

My apologies.

Jason November 1, 2007 at 8:44 am

I would like to talk more about the Wal-Mart example. I get the feeling that Russ would be more convinced of the methodology of the wage study if the results "seemed right". Or, if the results didn't challenge his beliefs, he wouldn't spend much time thinking about the study. I'm not attacking Russ; isn't this how most of us operate?

Moving on to another point…

Russ writes: Are you convinced by this evidence? I'm not. Does it lead you to conclude that large cities and suburbs should keep out Wal-Mart in order to protect workers?

I think that the jump to the second question is premature. Although, many people probably will use this study to say that WalMart is "bad". I say the second question is premature because we have only looked at part of the issue. We still need to look at the effects of lower prices for goods. If it is true that a worker has a drop in wages, is that more than offset by increased purchasing power? What is the total effect on worker quality of life?

If Wal-mart wants to build a store in your community, and you block that from happening you might be making workers worse off, even if their wages drop. And we haven't even looked at the benefit of lower prices for all non-merchandising sector workers in the community.

Bret November 1, 2007 at 10:21 am

Ah, faith based economics! I have to completely agree with Russell Roberts that applying the veneer of science to economic issues is not much better than applying intuition and common sense. Indeed, I think that a mixture of theory, empirical analysis, intuition, and common sense is about the best one can do.

Brad November 1, 2007 at 12:20 pm

Russ, here's an example of where Robin's approach fails badly. Consider what we should be telling teenagers about smoking today. Realistically, by the time an 18 year old today who smokes moderately develops a smoking related lung problem, we'll be able to print them a new lung. Past performance doesn't guarantee future results especially when there are paradigm changers on the horizon. Of course, there are lots of other reasons to steer kids away from the habit, but early death by lung cancer or emphysema probably won't be supported by future data for those young enough.

Lee Kelly November 1, 2007 at 12:30 pm

What is "faith based economics"? Even if you have no good reason to believe what you do (and I would argue that nobody does), it does not follow that the belief is "faith based". It needn't be "based" upon anything, which is precisely the nature of a conjecture, proposal or guess. Furthermore, a baseless idea is harmless, the important thing is that the idea is held open to criticism.

I think the confusion arises from two alternative definitions of faith, (1) an idea held without good reason, and (2) an idea held beyond criticism or revision. I think it is definition (2) which we need to worry about, since I have never met anyone who could avoid (1).


jmklein November 1, 2007 at 12:49 pm

What a steaming load of sophistry….

john Lott November 1, 2007 at 5:04 pm

A response to Roberts' claims can be found here. Since Roberts is unwilling to respond to my comments, I have posted a long discussion that responds to his statement: "So again, my question to my better-read colleagues in the profession–give me an example of a statistical analysis that relies on instrumental variables, for example, that is done well enough, that is so iron-clad, that it can reverse the prior beliefs of a skeptic." I give you a specific example. In fact, I give a second broad answer also. Will you respond this time Russell?

Russ Roberts November 1, 2007 at 11:02 pm


My point isn't that all empirical work is poorly done. It's that many economists are unconvinced by empirical work even when it's seemingly well done. The fact that you've done interesting and provocative empirical work is orthogonal to my point–that many people maintain their biases in the face of particular kinds of evidence–evidence that is based on sophisticated empirical analysis. Most of the time, that evidence can be dismissed by pointing out that this or that variable was left out, the instruments weren't really the right ones, causation runs the other way. And often those complaints are justified because of the biases of the researcher.

My original point was that these two biases interact to reduce the chance of a consensus being reached on controversial questions.

G November 2, 2007 at 12:12 am

I know this is off-topic, but I'm confused on the Wal-Mart issue. Isn't the lowering of wages in retail merchandise a good thing? Wal-Mart employs fewer and lower-skilled workers than do other competing stores, freeing up people with better skill-sets to work in other industries. I can't speak for all or very many of their locations, but I know the local Wal-Marts I've been to most certainly employ lower-skilled workers than other similar stores (such as Publix).

It seems like Wal-Mart has figured out a way to provide superior services using less-expensive labor. Isn't this how progress occurs?

James Hanley November 2, 2007 at 1:10 am

I'm a little less philosophical than some posters here, but I do think about methodology and epistemology a fair bit.

I'm inclined to agree with Roberts, but take a slightly different tack. I agree that econometric models are powerful tools, but everyone agrees that they're only as good as the assumptions and data put into them, so the real debate, I think, is whether the data and assumptions in these examples–guns, WalMart–are good enough or not. And that's not an easy question to answer.

One (but by no means the only) way is, in fact, to ask if the results "feel" right. That is, do they match up to what else we know? Is there a logical explanation for the econometric finding? Roberts critiique of the logic of WalMart lowering wages is right on: increased labor demand should not lower wages; average wages could go down if WalMart brings more workers into retail at low wages, even if no other retail employees are affected; etc.

The problem of logic doesn't mean the econometric model is wrong–in fact the model's results mean we should carefully review and analyze our logic–but it does give us reason to doubt, and to keep looking for more satisfactory answers, ones that are supported both by empirical data and logical causal explanation. There's never been a good empirical explanation that didn't have good causal logic.

Regarding concealed guns, I find the problem to be that there's a good logical argument on both sides, to support strong empirical claims on both sides. Roberts is right–and I think I'm agreeing with his reasoning–that in this case there's no way to come to concensus. As a political scientist, interested in public policy, I find that dismaying–ultimately we don't have much guidance.

Lee Kelly November 2, 2007 at 10:03 am

I agree that econometric models are powerful tools, but everyone agrees that they're only as good as the assumptions and data put into them.

That's not quite right. It is logically possible for an argument to have a true conclusion even when the premises are false. In regard to practical action it is only relevent that we act on true statements; whether or not those statements were derived from true premises is irrelevent.

Friedrich Hayek was a hamster
All hamsters are economists
Therefore, Friedrich Hayek was an economist


Daniel Earwicker November 3, 2007 at 6:56 am

With regard to Lee Kelly's comments on induction, I have a question. The famous testable theory "All crows are black" is inductively confirmed by each black crow we encounter, but would be deductively refuted by a single pink crow. So it appears that fallible induction is linked with confirmation of theories, and reliable deduction is linked with refutation of theories.

But suppose instead we choose to investigate the theory "Some crows are not black", which is clearly a testable statement about the world just as much as the other theory was. Each black crow merely provides weak, inductive refutation of the theory, but a single pink crow would deductively confirm it for all time. So isn't it the case that the supposed link between fallibility and confirmation on one hand, and between reliability and refutation on the other, is not generally applicable to all testable theories, but is actually dependent on us constructing theories in a certain way?

Followers of Popper have in the past responded to this by saying that "Some crows are not black" isn't a scientific theory!

What about "Our sun isn't the only star with a planetary system"? That's an interesting question about the universe that can be tested. A single confirming instance is all that is needed to prove it true for all time. And how could it ever be refuted for all time?

Do we have to insist that all stars have planets before we can call our theory scientific? Of course not. It's clearly a testable statement about the world, and it's just the kind of statement that real scientists investigate all the time. Of course, no team would bother investigating that particular theory now – it's already been confirmed, and can't be refuted except by calling into question the reliability of the evidence (though such challenges would obviously damage permanent refutations of the other kind of theory as well!)

(Another problem that occurs in real life very often, albeit with different theories, is that if someone says "All crows are black", and you present them with a pink crow, they simply respond, "That's not a crow – look, it's not even black. Stop wasting my time.")

Daniel Earwicker November 3, 2007 at 7:04 am

I've arranged this paragraph dreadfully badly:

"Do we have to insist that all stars have planets before we can call our theory scientific? Of course not. It's clearly a testable statement about the world…"

After the "Of course not", the other comments refer to the theory "Our sun isn't the only star with planets", not the theory "All stars have planets".

Previous post:

Next post: