Addressing Criticism of New Orleans Progress
My aim with this blog, and really with all the work I do, is to create a vigorous and research-informed debate about the many challenges we face in trying to create an educational system that serves all children well. I believe in the battle of ideas and that we make more progress when critics and supporters of various policies make the strongest cases possible.
Last summer, as we started to release our findings about New Orleans, I started to see misinterpretations on both sides of the debate, which led me to start a series of posts describing and assessing the most common arguments made about New Orleans. Now is a good time to get back to that because some of the more misleading arguments have reared their heads again.
The most recent example is a report and a related op-ed in Education Week by the Southern Education Foundation (SEF). I have great respect for SEF. Founded back in the 1930s as a merger of several foundations, its real history goes back to the mid-1800s. It is not only one of the oldest, but also one of the most important organizations in the country pursuing educational equity. Their report was driven partly by a proposal in Georgia for state takeover of public schools, along the lines of the Louisiana Recovery School District. (There are many big differences between the Georgia and Louisiana approaches but that's a topic for another day.) The SEF report and op-ed also relied heavily on a report by the Stanford Center on Opportunity Policy in Education (SCOPE) and therefore many of my concerns apply to that report as well. But I'll leave a full discussion of the SCOPE report for another day.
Let me just focus on the basic summary of the New Orleans situation in the SEF op-ed, which the SEF authors co-authored with another group, Research for Action: The authors write:
"A decade later, New Orleans still reports some of the nation's lowest achievement scores and graduation rates . . [and] a host of negative consequences, with a majority of families reporting long commutes to school, overcrowding, a bewildering gantlet of enrollment procedures, high rates of pushout, and difficulty finding schools able to serve students with special needs . . . The research also revealed that New Orleans' charter takeover has resulted in schools' increasing stratification by race and class."
The first sentence is a great example of Argument #8 on my earlier list. This classic "silver bullet" argument holds that any program that doesn't completely solve a problem is a failure. This logic should raise a lot of red flags because the only reason to make a silver bullet argument is that the policy being argued against was actually successful in reducing a key problem.
So it is in New Orleans. While the system was clearly a mess in the early years after Katrina as the city was rebuilding and a new school system took hold, the results in New Orleans have been generally very good ten years later. The reforms increased achievement by 0.2-0.4 standard deviations (8-15 percentile points)--and without increasing segregation, student mobility, or expulsions. Yes, outcomes are still low compared with national standards, but they have improved dramatically.
On Twitter, the authors responded to me on this point by saying that the Louisiana "data has been called into question repeatedly" (Argument #1 on my list). That's true and we are trying to look into it further, including the recent allegations of cheating. In the meantime, the problem is that they are relying on the same data we are in our studies--you can't legitimately argue the data are invalid then say that those same data show the program has failed. Also, there does not seem to be any support for the argument that these outcomes are "some of the nation's lowest," especially given that state tests can't be compared across states and NAEP tests, the only comparable alternative, tell us nothing about New Orleans.
Some parts of the SEF op-ed are a bit stronger. It's true that New Orleans students travel far and that many families report being confused by a system that is indeed hard to understand. Those are important issues, though the travel costs and the enrollment complexity are outgrowths of the city's extensive school choice process, which may benefit students in other ways. We find, for example, that the combination of choice and other policies has yielded a variety of options for families and may allow students to attend schools that better match their needs. Though this last point is more speculative, it doesn't seem reasonable to mention the negative consequences of choice without mentioning the potentially positive ones.
Continuing on, they make a valid point when they say that the system did not serve special education and ELL students well. Even the staunchest defenders would acknowledge this was true in the early years just after Katrina. After a lawsuit by the Southern Poverty Law Center, steps were taken to address that. There isn't much evidence either way as to whether these changes have worked, but I will give this criticism the benefit of the doubt.
Push-out was an issue and may still be, although the rules associated with the centralized enrollment system--a system the authors criticize--actually make push-out harder. New Orleans has also adopted a centralized expulsion system that keeps schools from pushing out students by simply keeping them out of school. Here, too, there is insufficient evidence to judge where the system is now (and, for that reason, we are pursuing a study on the subject).
The "overcrowding" argument is a new one. I think what they actually mean here is that some of the schools in high-demand are not accessible because too many students want to attend them and/or some schools have selective admission policies that prevent enrollment for most students. That is a real problem (see one recent interesting take by Andre Perry), although that problem is generally worse in the sort of traditional public schools that the authors seem to support; school districts regularly exclude students with attendance zones and selective admission magnet schools. Whether or not it's worse in traditional districts, I've never heard anyone argue that real overcrowding--that is, large class sizes and overflowing school buildings--is happening in New Orleans. The data simply don't support it.
The quote concludes with: "The research also revealed that New Orleans' charter takeover has resulted in schools' increasing stratification by race and class" (Argument #4 on my list). This is almost exactly half true. On the one hand, more advantaged students in the city saw larger achievement gains than disadvantaged students. On the other hand, both minority and low-income students seem to have benefited considerably--so much so that New Orleans, a predominantly minority, low-income district that was the second lowest-performing district is now near the statewide average on tests. If you think both forms of equity are important, as I do, then the argument is half true.
On the whole then, the op-ed and report are misleading. Of the seven points, one at least used to be accurate (special education), two are narrowly accurate (travel times and confusing choice processes) but do not recognize important potential side benefits of choice, one has too little evidence to judge (push-out) and another refers to a problem other than intended (lack of real access to quality schools rather than overcrowding) that also happens to be at least as bad under traditional school districts. Finally, the two most sweeping criticisms (poor outcomes and increased stratification) are quite misleading in the first case and half right in the second case.
This misinterpretation of the evidence on New Orleans is unfortunate because SEF is an esteemed organization and there are real problems with the way in which the New Orleans reforms that deserve their attention: the weakening of links between schools and neighborhoods (Argument #7 on my list), the lack of community involvement and local democratic governance, the heavy focus on standardized tests, the potential lack of cultural relevance (Argument #2 on my list), and the fact that the state takeover approach is unlikely to generate the same positive effects in Georgia or other cities and states (Argument #13 on my list). These are not the only limitations, but they do highlight some alternative criticisms that appear stronger than the ones in the SEF report.
These real problems are getting neglected in the debate despite fitting well with other parts of the SEF report, which recommends "restorative practices and a student-centered learning environment" and "deep parent-community-school ties." Warren Simmons, who serves on the SEF board made eloquent comments on some of these points at our June conference on the New Orleans experience.
Another reasonable argument against the New Orleans's reforms is that there is much we do not yet know about the effects of the reforms--there are some reasons to question the test scores as indicators of students' long-term life chances. For these reasons, it would be reasonable to argue that it's too early to judge. But these stronger arguments have given way to weaker ones.
Even with the many positive results in New Orleans on many key measures, it's important to have critics pressing further for improvement. We don't learn much from the people who tell us how great we are. We learn from our critics, especially when their arguments are on the mark.
Douglas N. Harris is Professor of Economics, the Schleider Foundation Chair in Public Education, and Founder and Director of the Education Research Alliance for New Orleans at Tulane University. You can follow him on Twitter.