What’s wrong with research in the social sciences? Only two things. How it’s done and how it’s interpreted.
A few weeks ago, some economists, Raj Chetty (Harvard), John Friedman (Harvard), Nathaniel Hilger (Harvard), Emmanuel Saez (UC-Berkeley and winner of the John Bates Clark Award), Diane Schanzenbach (Northwestern) and Danny Yagan (Harvard) gave a talk on their research project. They were trying to find out if your kindergarten experience has much of a lasting effect. Previous research said no. If you had a particularly good teacher, you might have higher test scores for a while in the next few grades. But eventually, any effects tended to die down.
What Chetty et al did is look beyond test scores, which after all, are not intrinsically important. They decided to look at income. Did your kindergarten experience affect your income when you were older. They found that it did and the effects were dramatic:
The graph shows wage earnings in 2007 plotted against the kindergarten test scores of the same kids about 22 years earlier when they were in kindergarten. Whoa. Pretty impressive. It looks like the percentile of someone’s kindergarten score, you can predict their income 22 years later with a lot of precision.
That turns out not to be true at all. The graph is misleading. It shows 19 data points. In fact, there are over 5000 people in the sample. So presumably what they’ve done is taken the different percentiles, 5th, 10th, etc and taken the average wage earnings for people in that percentile. So while on average, the higher your percentile in kindergarten, the higher your income, the fit of the line isn’t anything like the picture. In fact, the score explains about 5% of the variation. The actual data and relationship is much noisier and probably looks more like this:
This picture (taken from a piece on the mortgage crisis by Mian and Sufi) is an example of where there is a positive statistical relationship but the fit isn’t very good–that is, there are a lot of other factors other than the one we’re focusing on and knowing the level of that factor doesn’t help you predict very accurately.
When you find a statistical relationship between kindergarten scores and earnings 22 years later, there are many possible explanations. But the two most obvious are that some people have a wonderful educational experience in kindergarten that lasts the rest of their lives or that there is another underlying factor (genetics, parental influence) that accounts for both the score and the higher earnings.
If it is the wonderful educational experience the cause could be having a great teacher or it could be having great peers or it could be having great parents who augment the experience. What the authors do is exploit the fact that in their data, teachers were assigned randomly (at least in theory) as part of the STAR experiment. So that should rule out problems where smart ambitious parents make sure their kids get good teachers. That might cause you to falsely assume good teacher make for good scores when it’s really the parents’ ability and actions that cause both.
The economists find that the number of years of experience that your kindergarten teacher had affects your earnings 22 years later:
What do we make of this graph? Not so sure. It says that a teacher with no experience is not nearly effective, on average, as a teacher with exactly ten years of experience, but then burn-out sets in for a while. Teachers with 11 years of experience or (the horror) 12 years can hurt you relative to a rookie, but then somehow, the teachers get better again and once they have more than 15 years experience, they outperform a rookie. On average, more experienced teachers make a positive difference. How valuable is having a good teacher. Here’s what David Leonhardt wrote in the New York Times about the research:
All else equal, they were making about an extra $100 a year at age 27 for every percentile they had moved up the test-score distribution over the course of kindergarten. A student who went from average to the 60th percentile — a typical jump for a 5-year-old with a good teacher — could expect to make about $1,000 more a year at age 27 than a student who remained at the average. Over time, the effect seems to grow, too.
Those sentences presumably come from this picture:
This purports to show that having an experienced teacher (or presumably a good one) can be worth about $1000 per year per pupil. The phrase from the New York Times article—”the effect seems to grow”—appears to be based on the fact that the gap between the blue line and the red line gets wider as the students get older. It’s easy to see those two lines diverging even farther. Of course, they could converge and overlap as they do in earlier years. There’s no data on anything past 22 years out of kindergarten in the study. Leonhardt’s bottom-line:
Mr. Chetty and his colleagues — one of whom, Emmanuel Saez, recently won the prize for the top research economist under the age of 40 — estimate that a standout kindergarten teacher is worth about $320,000 a year. That’s the present value of the additional money that a full class of students can expect to earn over their careers. This estimate doesn’t take into account social gains, like better health and less crime.
I have a few problems with this conclusion.
First, this work (as Leonhardt concedes early in the article) isn’t peer-reviewed. It’s not even a paper. It’s from a workshop with some powerpoint slides. So it’s pretty preliminary.
Second, I’ve attended kindergarten. My kids have too. Having a good teacher is important, sure. But is it really plausible that the effects of a good teacher rather than a mediocre one show up 22 years later when the people in the sample are 27? And they don’t show up when 19 years later, in 2004, when the people in the sample are 24? That’s how powerful the effect is. IT LAYS DORMANT FOR 19 YEARS! This doesn’t pass the sniff test for me.
Chetty and his co-authors are not naive. They understand there can be other causal factors. They do lots of other statistical stuff to make sure their conclusion is robust. But this is the way social science is done, especially economics. You find a data set that has a cool feature (random assignment of teachers) and then you squeeze the data set as hard as you can to find something interesting that doesn’t go away when you look for other explanations within that data set. Your conclusion (a good teacher is worth at least an extra $320,000 per year) is simply what the numbers say. You can’t argue with the numbers. When you’re 27 in this data set, after controlling for everything else you can, someone who had a good teacher in kindergarten makes an extra $1000 per year. The fact that there’s no effect when you’re 24 is a sign of how long it takes for the effect to kick in.
So I’m a little skeptical. But maybe it’s right. But either way, now that the New York Times has said that a good kindergarten teacher might be worth AT LEAST $320,000 (don’t forget lower crime and better health!), then it becomes sort of a fact. That lets Nancy Folbre, an economist at the University of Massachusetts write an opinion piece in the Times saying:
A really good kindergarten teacher is worth $320,000 annually, according to one recent estimate, well publicized in this newspaper. That would reflect the present value of the additional money that students in a really good kindergarten class can expect to earn over their careers.
But average pay for kindergarten teachers is only about $50,380. And even at that level, kindergarten teachers, many employed by public school systems, fare relatively well compared with those in similar jobs.
The title of the article is “Why Girly Jobs Don’t Pay Well.” (HT: Political Math) Most kindergarten teachers (97% according to Folbre) are women and they congregate in these low-paying jobs. They’re worth $320,000 but they only get $50,380.
A mere $50, 380 for a nine month a year job that requires little work outside of a pretty short work day? You can only argue that’s unjust by invoking that $320,000.
