Here’s a very interesting article in the WSJ (HT: Steve Saletta) on the use of what is called instrumental variables in econometrics to argue that television watching causes autism. It seems like a goofy idea and it may be. The more interesting issue for me is the use of sophisticated statistical techniques to try and observe cause and effect. Too much of the time, for my taste, instrumental variables are used to awe and amaze rather than to ferret out the truth. And too many economists have become enamored of technique for technique’s sake.
The paper discussed in the Journal is a perfect example and the reporter, Mark Whitehouse does a superb job explaining the econometrics for a non-expert:
Prof. Waldman, a recognized expert in the field of
applied microeconomics, doesn’t pretend to be an authority on autism.
He became engrossed in the subject in the fall of 2003, when his
2-year-old son, David, was identified as having an autism-spectrum
disorder. Hoping to eliminate any potential triggers, Prof. Waldman
supplemented the recommended therapy with a sharp reduction in
television watching. His son had started watching more TV in the summer
before the diagnosis, after a baby sister was born.Prof. Waldman says his son improved within six months
and today has fully recovered — a surprising result, given that autism
is typically a lifetime affliction. "When I saw the rapid progress,
which was certainly not what anyone had been predicting, I became very
curious as to whether television watching might have played a role in
the onset of the disorder," he says. He tried to get medical
researchers interested in the idea, to no avail.In late 2004, he decided to look into the subject
himself, ultimately putting together a research team with Cornell
health economist Sean Nicholson and Nodir Adilov, a professor of
economics at Indiana University-Purdue University in Fort Wayne.In principle, the best way to figure out whether
television triggers autism would be to do what medical researchers do:
randomly select a group of susceptible babies at birth to refrain from
television, then compare their autism rate to a similar control group
that watched normal amounts of TV. If the abstaining group proved less
likely to develop autism, that would point to TV as a culprit.Economists usually have neither the money nor the
access to children needed to perform that kind of experiment. More
broadly, randomized trials seldom lend themselves to studying economic
questions, particularly the more traditional ones. It would be unfair
to randomly subject some people to a higher tax rate just to see how it
affects their spending.Instead, economists look for instruments — natural
forces or government policies that do the random selection for them.
First developed in the 1920s, the technique helps them separate cause
and effect. Establishing whether A causes B can be difficult, because
often it could go either way. If television watching were shown to be
unusually prevalent among autistic children, it could mean either that
television makes them autistic or that something about being autistic
makes them more interested in TV.The ideal instrument is a variable that is correlated
with A but has no direct effect of its own on B. It should also have no
connection to other factors that might cause B. If data in a study
nonetheless show that the instrumental variable is linked to B, it
suggests that A must be contributing to B.
After giving a few examples of previous studies that used instrumental variables, the article gets back to Waldman and autism:
By putting together weather
data and government time-use studies, they found that children tended
to spend more time in front of the television when it rained or snowed.
Precipitation became the group’s instrumental variable, because it
randomly selected some children to watch more TV than others.The researchers looked at detailed precipitation and
autism data from Washington, Oregon and California — states where rain
and snowfall tend to vary a lot. They found that children who grew up
during periods of unusually high precipitation proved more likely to be
diagnosed with autism. A second instrument for TV-watching, the
percentage of households that subscribe to cable, produced a similar
result. Prof. Waldman’s group concluded that TV-watching could be a
cause of autism.Criticism quickly arose, illustrating some of the
perils of the economists’ approach. For one, instruments are often too
blunt. As Prof. Waldman concedes, precipitation could be linked to a
lot of factors other than TV-watching — such as household mold — that
could be imagined to trigger autism. At best, his data reflect the
effect of television on those children who changed their habits because
of rain or snow, not on those who did it for other reasons such as a
desire to watch educational shows.
On the surface, the whole thing sounds vaguely plausible. Watching TV is a brain activity. It’s imaginable that watching TV affects your brain and could, at a crucial developmental stage, cause the brain to do various things, both good and bad.
But then you start to think about it a little more. They didn’t show a relationship between TV and autism. They showed a relationship between rainfall and autism. Then they made the leap that more rainfall means more TV. But more rainfall means a thousand other things, too. The "theory" that encouraged the study was based on ONE observation, that Waldman’s kid watched less TV and got better. But surely there were thousands of things that were different in the Waldman household over those six months.
Then there is the problem of defining autism. There are different degrees of autism. Were they all lumped together? How precise was the relationship between rainfall and autism? What was the confidence interval? Did the definition of autism change over the period? The diagnosis of autism is much more common today than in the past, either because of environmental factors or more plausibly, a higher awareness of the phenomenon by both doctors and patients and a widening of the definition of what is considered autistic. Maybe an increase in rainfall just happened to coincide with an expansion of the definition. Surely the increase in cable TV subscriptions is correlated with that expansion as well. And is there really so much variation in snowfall and rainfall in Oregon, California and Washington states? Why were those states chosen? If anything, people in Portland and Seattle are used to lots of rain and snowfall and probably don’t respond by watching TV a lot when it’s raining. Did the authors control for these possible effects?
When I hear these kind of speculative findings (and they are very common in the profession these days) I always want to ask the researchers how confident they really are in their findings and have they really discovered anything we didn’t already know.
In the case of the autism study, I’d ask the following: does the econometric firepower brought to bear on the problem add anything more to our knowledge than the anecdotal story that Michael Waldman’s son got better after watching less TV? I’m not convinced. But here’s what I really want to say to the authors of the study.
You found a correlation between rainfall and autism that might indicate a correlation between TV and autism. Let’s take a thousand kids with autism who currently watch a lot of TV. Let’s do a real experiment where we remove their access to television. How much are you willing to bet that there will be a statistically significant reduction in autism? A thousand dollars? Ten thousand dollars? Ten dollars? Put your money where your econometric mouth is. You found something. Do you really believe it?
