NOLA Argument #13: Context Matters
It doesn't take much to convince people that New Orleans is different from other cities. Convention-goers and tourists visiting the French Quarter see it from one angle. For most people who live in New Orleans, the situation is flipped--we rarely go to the French Quarter--yet the city still seems quite different from other places we've lived. As the actor and part-time New Orleans resident, Matthew McConaughey, put it so eloquently, New Orleans is a "big, beautiful mess."
But does any of that matter for the school reforms? Are the city's contours so different that we should expect the effects of education programs and policies to be different here from everywhere else? This is a question policymakers have to ask themselves as they try to make sense of all the evidence that comes across their desks for any given policy.
It all comes down to who, what, and how. To be more precise, we have to ask the following questions when deciding whether the results of one study are likely to arise in another context (what researchers call "external validity" or "generalizability").
1) What students were in the study? For example, what are the demographics and academic performance of the students involved?
2) Who implemented the program and how? Did the educators running the program have an unusual degree of skill or interest in making it work? Was it implemented "at scale" by educators under normal schooling conditions?
3) What is the program being compared with? To determine how well one new program works, we have to compare it with something else, but if the other program was especially ineffective then the new program might look better than it really is.
What are the answers for the New Orleans school reforms? First, both pre- and post-Katrina, the New Orleans public school population has been almost entirely black and low-income. Also, despite apparently large increases in test scores (and probably other outcomes) due to the reforms, New Orleans students are still low performing by many metrics. So, we might think the same results are more likely to emerge in other cities with similar student situations like Detroit and Memphis.
The educators involved in implementation of the New Orleans school reforms were also unusual. There was an out-pouring of national support for New Orleans after Katrina that drew people in. Also, once it became clear that the school system would be truly unprecedented, ambitious reformers from around the country wanted to come here. New Orleans has become a national center for school reform in ways that are unlikely in other cities and states. That said, the system is arguably "at scale," having been carried out for 10 years in an entire city.
Finally, on the last question, the New Orleans school reforms are typically compared with what came before it, which, by just about all accounts, was deeply problematic. The pre-Katrina results were dismal (see this report I co-authored with Andre Perry and Christian Buerger and this report from the Cowen Institute). In cities that have already implemented effective policy changes, the results of a New Orleans-style system might be different.
What's the bottom line? Context matters. There are a lot of valid reasons to question whether similar policies would generate similar effects outside of New Orleans. And even that sets aside the political feasibility of adopting a system like this. There is little question that the reforms would not have happened in New Orleans without Hurricane Katrina. As I pointed out previously, the school reforms and Katrina are inseparable.
To be clear, any discussion of context is speculative. We cannot say what effects we would see elsewhere without actually trying it, and studying it, in other places. But we do have good reasons to expect that the effects would be different, and probably smaller on average.
Neerav Kingsland, the former Executive Director of New Schools for New Orleans and current national leader in the school reform movement, argues much more optimistically about the prospects of replicating the New Orleans reforms elsewhere. Others argue against replication, but mainly because they do not think it was successful in New Orleans. That's an entirely different kind of argument that I'll take up next week.