There is not much debate that the New Orleans school reforms improved student outcomes. The evidence on that point is strong. The question is how? What exactly was it about the reforms that led to these improvements? After all, “the reforms” are really a multi-faceted package--school choice, charter schools, accountability, and more.
The part of the reform that gets the least attention is the amount of money invested in New Orleans. In the immediate aftermath of Hurricane Katrina, money poured in even as the students flowed out. Over time, as the new system settled in, ERA-New Orleans researcher Christian Buerger finds that per pupil spending in New Orleans increased by $1,000 more than in similar Louisiana districts. Also, FEMA provided $1.8 billion in funding for school buildings. (To put that in perspective, this amounts to roughly $20 million per building.) At our June conference, some speakers argued that the overall effects on student outcomes may have been driven by this spending.
Even those incredibly skeptical about the value of school spending would probably admit that an investment of this size should have at least some impact on student outcomes. But how much?
This is no easy question to answer, in part because the effects of spending probably depend on how exactly the money is used. Some argue that traditional public schools utilize funds effectively, but that there are simply too few resources in total. With more money, schools can reduce class sizes, increase early childhood education, raise average teachers salaries, or, more generally, do whatever they think is best to help students. I’ll call this the resource levels argument.
Others argue that spending matters much more when external incentives are attached. Again, in New Orleans, we had a whole package of reforms and intensive accountability was a key piece. Ten of the city’s schools have been shut down for performance reasons and many others have closed because accountability sanctions loomed. Moreover, part of the reforms involved ending the union contract and tenure protections. This allowed CMO and school leaders to pass the accountability they felt on to individual teachers. In short, the external incentives were strong and this may have allowed the money to have a larger impact than it would have under a traditional school system.
Let’s first consider prior evidence about resource levels. Kirabo Jackson, Rucker Johnson, and Claudia Persico recently released one of the most rigorous studies ever conducted on the topic. Their results are reported in Education Next (coincidentally the same issue of the journal that included three articles by my colleagues and me about New Orleans, on student achievement gains, school choice, and the diversity of schooling options). Importantly, the effects of increased expenditures found by Jackson and colleagues occurred in the absence of intensive test-based or market-based accountability and included spending changes across the nation.
They find that a 10 percent increase in spending leads to a 10 percentage point increase in the probability of graduation for low-income students. We can use this result to roughly estimate the possible effects of the $1,000 per student spending increase in New Orleans, which represents a roughly 10 percent increase. Interestingly, the Jackson/Johnson/Persico estimate of the spending effect on graduation lines up almost exactly with what Louisiana Department of Education reports suggest about the increase in New Orleans graduation rates. The city’s graduation rate increased by about 10 percentage points relative to the state.
Since their study relates to spending in low-accountability settings, this would seem to support the resource level argument, but there is still more to it. In our analyses so far, we see no evidence that the increase in funding was used to reduce class sizes or increase pre-kindergarten. If this version of the argument is to have credibility, we should be able to see some specific resource level increase, especially one that has some research support behind it.
In fact, the only area at this point where we do see an increase in expenditure is transportation, which appears to account for almost half the total resource increase. It seems highly unlikely that transportation was the target in the spending increases studied by Jackson and Johnson, which makes their study less relevant to New Orleans. Also, since the transportation costs are driven by the city’s expansive school choice system, to argue that any increase in student outcomes from transportation costs would be a tacit acknowledgement that choice played a positive role.
We should also look to the one area where we know the vast majority of school funding goes: educator salaries. We are currently working on a New Orleans salary study, but let’s consider other evidence again. In another study using data from across the country, Susanna Loeb and Marianne Page find that a 10 percent increase in salaries reduces dropout rates by 3-4 percentage points. While this might seem smaller than the above 10 percentage point figure, we are being forced here to compare graduation rates with drop out rates, which differ in important ways. Without getting into the details, let me just say if all the non-transportation increase in expenditure went to salaries, then this could get us perhaps half way to the 10-percentage point increase in the New Orleans graduation rate.
The spending on buildings might eventually matter, but only 16 projects had been completed as of 2013 and even fewer were completed as of 2012, the most recent year for which we have studied reform effects on student outcomes. The cause has to precede the effect and this potential one doesn’t.
Therefore, at best, resource levels alone might explain half the apparent increases in graduation rates. Even that seems optimistic, however. First, even the authors of these studies recognize how difficult it is to estimate the effects of spending and salary increases; and I have ignored other studies that find no effects or smaller effects of spending and salaries.
Finally, to believe that it was all about spending levels, we have to assume that all the incentives created by the eliminating tenure, shedding the union contract, opening up school management to alternative providers (CMOs), and dramatically expanding choice all had no influence at all. This seems implausible, especially given recent evidence from the city about charter school effects for example (see studies by a team from MIT and Duke and CREDO). More likely, it was the entire package of reforms--the interplay between spending and the other policy changes--that drove the increase in student outcomes.
Is it all about the money? Very likely not, though there is too little evidence to answer this question confidently.