The Betsy DeVos Agenda: What the Evidence Says
My recent New York Times op-ed and a longer version of the piece that appeared in Education Next seem to have sparked a raucous debate over DeVos and her ideas. I'm glad for that much. We need a rich debate on these important issues, and that's why I'm taking the unusual step of a third round of comment.
Before diving into responses on the specific points made by Jay Greene, the Wall Street Journal, The Economist, and others, there are some important points about the argument itself that are getting missed. First, as far as I can tell, not even the critics of my original piece think the track record of DeVos's ideas is very good. No one is saying, "Wow, what a great success story. Let's take this national." That's the kind of support I would hope to hear for a nominee to the highest education post in the land. Instead, we are debating whether her track record is really a "disaster" or merely not good.
Second, the response to my piece by the head of the Michigan Association of Charter Schools, Dan Quisenberry, suggests a misunderstanding of my argument. To be clear, I'm arguing that DeVos is advocating for ideas that have a poor track record. The Quisenberry piece, in contrast, focuses mostly on how recent change in Michigan law will make things better. He doesn't address whether DeVos supported any of the recent accountability provisions (though she clearly opposed a new commission that had been proposed by a bipartisan group of local leaders). Based on the press releases and the model legislation from her organizations, it would appear that she opposed all of it and was pushed into a compromise. At the very least, I can find no evidence of support for any specific form of accountability beyond simply putting information about schools on web sites.
Being clear on the argument--that the ideas for which she advocates don't work and may do measureable harm--is also important because some of her defenders have argued that what she advocates for is not relevant to her candidacy. They suggest that she might advocate for something else as Secretary of Education. I suppose that's possible. She might change her mind or focus on something else. But given that her laser-focused advocacy for free market choice is really her only qualification for this position and that this position corresponds to President-elect Trump's main stated position on education, this argument seems extremely weak. One might respond by saying that Trump won and that it's his right to choose people who support his position. That's true, but considering that the Constitution gives the Senate the power of confirming Cabinet members, how can anyone argue that DeVos's positions are somehow unimportant to the case for her nomination?
Another theme in the replies to what I've written is that I lack evidence on the effectiveness of the specific kinds of oversight I have in mind. That's not true, as I show below, but the larger point here is that the lack of charter and voucher oversight itself makes it difficult to gather any evidence. Without having someone minding the store (preferably, in my view, some sort of elected person or board), it is very difficult to collect the data necessary to document potential problems, let alone prevent them. One reason we keep coming back to New Orleans is that the oversight of the system generates the data necessary to evaluate it.
Finally, even to the degree some might think there are remaining questions about the evidence on specific points I made, I have tried to emphasize that it is the collection of evidence that matters most. So, if you are able to read all that I've written in the New York Times, Education Next, and this current piece (if so, I appreciate your attention and patience), I simply ask you to consider this question: does the weight of the evidence give you any confidence that Devos's ideas will produce measureable gains in America's schools? Does the evidence lead you to think that she is the right candidate to become the nation's foremost education leader?
These are the questions. I appreciate the back and forth on them. In the end, however, the critiques of my original arguments, and the additional evidence that others have pointed to, seem to have only strengthened what I said before. Based on what we know today, free market education policies--vouchers, virtual schools, and unaccountable charters--are unlikely to produce measureable gains in our schools, and there is a good chance that these policies will make matters worse.
Responses to Specific Points
Let's start with the Wall Street Journal (WSJ). The Journal is famous for its free market advocacy. As an economist myself and general supporter of free markets, I sometimes agree with the WSJ. But they often miss the point when it comes to education policy. In this case, the WSJ tries to change the argument and lower the bar. They emphasize that DeVos has worked hard to try and turn around low-performing schools in Detroit. I agree, but that's not the point. The actual point, and reason for concern, is that she is trying to help by choosing a path that is very unlikely to work.
The WSJ, as well as Jay Green (see below), try to argue that Detroit's system has strong accountability solely on the basis of the number of charter closures. That's a start, but what we really want to know is which schools are being closed and how. I've already written about how New Orleans is one of few cities that have systematically closed schools with low value-added.
In Detroit, it's very unlikely we will have any evidence about the quality of closed schools before the nomination hearings, but perhaps we will eventually. What we do know is that some schools have avoided closure by simply switching authorizers and that this may happen less often with the change in law (despite DeVos's apparent resistance to that law). Someone will probably come up with the school performance scores for the closed schools, but this won't tell us much because those scores are not very accurate, as they focus on test levels rather than student growth.
The other issue with charter closure is whether systems are in place to ensure that students have a smooth transition to a better school. Given the difficulty they face in switching schools, students in closed schools should be given plenty of notice of school closures and given priority to the next school of their choice. This is almost impossible in a free market system.
Next, I respond to Jay Greene's latest blog post, point by point.
JG: Let's consider his [Harris's] four arguments. First he says, "given the lack of oversight in Detroit and evidence from other cities that some charter schools cherry-pick their preferred students, these results may make Detroit's charter schools look better than they are." Got that? He has no evidence that Detroit charters are cherry-picking students at all, let alone that they are doing so at a higher rate than in New Orleans, but he nevertheless posits that "if it's happening, then the charter effects on achievement [in Detroit] are inflated."
DH: My statement clearly acknowledges the lack of good evidence on this point, and again, it is partly the lack of oversight that is creating the lack of evidence.
Because of the limited evidence on cherry-picking, I also added the point that Jay omitted in response: that the lack of oversight creates incentives that encourage cherry-picking. That's not an assumption, and I doubt anyone would question it. I don't think we should design policies that encourage cherry-picking, especially given evidence from other places where it sometimes occurs.
JG: Second, given Harris' assumed concerns about cherry-picking in Detroit charter schools and the inability of the CREDO study to account for that, he examines evidence from the urban NAEP test and finds that the city of Detroit has experienced below average growth in those scores in recent years. Using NAEP results from the entire city to draw conclusions about Detroit's charter schools requires a host of assumptions. He'd have to assume that test results from all schools, charter and traditional, somehow speak to the effectiveness of charter schools. He'd have to assume that demographic and other non-school factors in Detroit do not affect the comparison of test growth in Detroit relative to other cities.
DH: Actually, these assumptions aren't the problem. To the first point, part of the theory of charter schools is that they are supposed to generate competition that "lifts all boats," and that, along with any charter improvement, would be reflected in citywide growth. To the second point, I did carry out demographically adjusted estimates of the type Jay is suggesting, making generous assumptions toward the charters--and it still looks bad. But that whole analysis is built on too many other assumptions, so I focused instead on the stronger evidence.
JG: A number of education analysts have coined a term -- misNAEPery -- to capture how unreasonable it is to make the assumptions required to use NAEP to compare policies across jurisdictions. Oddly, some of those analysts who like to accuse others of misNAEPery, like Morgan Polikoff and Matt Barnum, have somehow failed to denounce Harris' use of misNAEPery in both the NYT op-ed and in the Ed Next reply to critics. Both have even "retweeted" Harris' new post, so we can assume they've read it.
DH: If I had written a piece that relied heavily on the NAEP, then he would be right to call it a misuse of the NAEP data. I will say that I think the NAEP results are more relevant in this case than they usually are because we are talking about a citywide reform and because we are looking at changes in NAEP scores over time in that city. Regardless, and not to belabor the point, NAEP is only one of many pieces of evidence.
JG: And just to anticipate concerns about my own consistency on the use of NAEP, I think comparisons using NAEP that control for observed demographics are about as convincing as the CREDO results, which also rely on comparisons controlling for some observed characteristics.
DH: See above. The demographically adjusted NAEP results make Detroit look terrible. Even though that supports my case, I still don't think it's as convincing as the rest.
JG: At the very least, it is useful to offer a disclaimer that neither NAEP nor CREDO provides convincing causal evidence, even if confession does not assure absolution. Harris does not offer any disclaimer and instead uses his NAEP comparisons to bolster his assumption that Detroit charters may be cherry-picking.
DH: Actually, I did provide a disclaimer. I said, "While each piece of evidence I've discussed has some limitations, collectively, they present a strong case against DeVos's ideas." I think it would have been excessive to repeat that over and over.
JG: Third, Harris builds on the observation that Detroit city has very low NAEP test scores to assume that "the extraordinarily low standing of the city as a whole, to the degree it is caused by low performance of traditional public schools, should make it easier to improve student outcomes when trying something new." It is also quite plausible -- perhaps more plausible -- that a city with extraordinarily low test scores also has severe social and economic problems that are outside of the control of schools, which would make it harder for charters to improve outcomes.
DH: Here, I think Jay just accidentally misread what I wrote. The words he is quoting explicitly say, "to the degree it [the poor test performance] is caused by low performance of traditional public schools." That statement is in there explicitly to address the likely possibility that the severe social and economic conditions are partly driving the low scores. At the same time, it's hard to imagine the low test levels aren't partly driven by the schools.
JG: Lastly, Harris argues that even if Detroit charters have produced gains, the gains are smaller than those in New Orleans, so we should prefer the regulatory approach used in New Orleans to the one in Detroit. But this assumes that any greater gains produced by New Orleans charter schools are caused by the regulatory approach in that city. In fact, we have no idea whether New Orleans' regulations helped, hurt, or had no effect on how large the gains in that city were.
DH: Actually, that's not true at all. I would never claim causality on such flimsy evidence. In fact, we are learning a lot about how the specific types of oversight contributed to the New Orleans results. I gave one example in the prior piece, citing evidence that perhaps the most important element of accountability that the system provides is taking over low-performing schools. When the decisions are actually based on performance, we have found closures and takeovers have large positive effects and explain roughly one-third of the total New Orleans reform effect.
Here is another example: Al Roth won the Nobel Prize in Economics mostly for work that he has since used to create the centralized enrollment system being used in New Orleans and other cities. I think Nobel-prize winning research should also count as part of the evidence base. Oversight of enrollment is one of the most important things we can do to make choice work well. Detroit lacks such oversight substantially because DeVos advocates against this type of government involvement.
JG: Other than these four-assumption-dependent arguments, Harris offers very little to defend the strong claims he made in the NYT that Detroit charter schools have been a "failure" and a "disaster." He does cite some national "charter-friendly" organizations, like CRPE and NACSA, as being critical of Detroit charters. But that is an argument from authority -- not evidence.
DH: This final point is mostly a compilation of the prior points that I think I have shown are not persuasive, but notice here that Jay is not actually challenging the criticisms lodged by these charter supporters against the Detroit model and DeVos's ideas. Instead, he challenges only how we categorize them--as evidence or as an appeal to authority. It is indeed an appeal to authority, but I see that as strength. Jay has been among the most eloquent advocates getting beyond test scores. I think it's worrisome that three different charter-supporting organizations that have on-the-ground knowledge about how the Detroit system works--not just the test scores--describe the system in such unflattering terms. If a group of chefs who have been to a restaurant tell me that the food there is terrible, then I'm going to seriously consider that. It's not the only thing I would consider, but it's clearly relevant.
To this, Jay might respond by saying, "See, you are cloaking your argument in evidence when it is in fact based on your ideology." No, let's be clear, I've cited far more evidence than anyone else in this debate. By the end of this post, I'll be up well over 50 studies cited directly and indirectly, enough to clearly support my argument that this is a triumph of ideology over evidence. Yes, I have an ideology and principles, too--you can't interpret evidence without them. For example, Jay and I share a value for choice and, I believe, for equal opportunity as well. Where we seem to differ is in Jay's extreme aversion to government, even in cases where market theory tells us the government is needed and, most critically here, where there is strong evidence that the government can take certain specific steps that will provide families with better options and do more to facilitate equal opportunity.
JG: He also falsely suggests that Detroit has failed to close failing charter schools. In fact, 30% of Detroit's charter schools have been shuttered. And another national charter-friendly organization, NAPCS, gives Michigan higher marks on closure than Louisiana, noting that Michigan has closed more charter schools than Louisiana, 47 to 26.
DH: It looks like we have two different studies by two different charter school organizations showing different results. The one I cited (the only one I was aware of that the time) shows exactly what I said, and the one Jay cites shows what he said. This could be because the data I'm reporting are more recent. It's hard to tell. Either way, it's not just the number of charters that matters, but which ones are selected and how students are shifted to other schools (see above response to the Wall Street Journal).
JG: Harris also cites two voucher studies with negative results -- one of which is an RCT and the other not -- as proof that lower regulation approaches are less effective. Leaving aside the distinct possibility that the negative RCT result for Louisiana was actually a function of the over-regulation of that program, it's important to note that Harris fails to cite the entire literature of rigorous studies on the effects of private school choice programs, which overwhelmingly shows positive outcomes.
DH: Again, Jay seems to be misreading my statement. As I indicated in the original op-ed, I focused on Louisiana and Ohio because they are most reflective of the type of programs that Devos's group has proposed and what a federal policy is likely to look like. In my response, I also pointed out that the best evidence indicates that "the market-based approach to school reform seems to work better in urban areas," which is based partly on "the entire literature" that Jay refers to. The study Jay points us to reinforces my point: vouchers do sometimes show positive effects on students and don't usually show negative effects. The ones that show positive effects are in urban areas, just as I said.
I think the best and most recent review of the voucher evidence is by Jay's colleagues at the University of Arkansas. This review shows that, among U.S.-based studies, there are no statistically significant effects of using vouchers, in any subject. (The overall study of the review suggests positive effects, but this conclusion only holds when we include studies in other countries that are not useful here. Also, the review omits the recent Ohio study that also found negative effects.) Even if we ignored statistical significance, the effect sizes, averaged across studies are almost exactly zero.
JG: I agree with Doug that ideology, which could more kindly be described as "principles" or "values," has an important role to play in policymaking. I just disagree with him that the evidence clearly shows the superiority of a high-regulation approach to school choice. I think it's more accurate to say that the evidence is unclear on this matter, which means that we may -- appropriately -- need to rely more on our values, principles, and broader ideology when deciding how to proceed.
DH: As you can see, most of these particular critiques of my interpretation are problematic.
Since I think (hope) this will be the last round of responses, let me just summarize where we are:
1. Basic economic theory suggests that certain kinds of markets cannot function efficiently or equitably on their own. In these situations, government oversight, coordination, and accountability can help address the problems. Education is a market where the case for government involvement is very high. This is apparently why a recent editorial from The Economist indicated that DeVos's "schemes should be done properly or not at all."
2. Even the highest-accountability version of vouchers and tuition tax credits has less accountability than the typical charter program. The fact that the achievement effects are positive on average for charter schools, but not for vouchers, is the first sign that government involvement is beneficial.
3. Of the cities that have pursued choice-based systems, the one with the most intense government accountability, New Orleans, has generated large positive system-wide effects that are almost certainly larger than any city that has pursued school choice.
4. The results for virtual schools, which also have very limited oversight, have been strongly negative. This point has not been questioned.
5. In addition to the strong pattern when we compare low- and high-accountability choice systems (see points 2-4 above), there is direct evidence that specific types of oversight and accountability are beneficial (see above on centralized enrollment and performance-based takeovers).
6. The value of oversight and accountability is reinforced by what we know about the place where DeVos has been most influential: Detroit. Whether Detroit charters produce higher test scores than traditional public schools is debatable. More obvious is that the results in New Orleans are certainly much larger than Detroit (even the CREDO results, which greatly understate the differences between the two cities, show more positive effects in New Orleans). Also, several local and national charter supporters knowledgeable about Detroit have real concerns about the lack of oversight and accountability there.
7. Even the more promising choice-based policies, which include government oversight, coordination, and accountability, only seem to work in urban areas. In fact, it is hard to come up with any examples at all where charters, vouchers, or virtual schools having positive effects outside of urban centers. This further reinforces the conclusion that large-scale, choice-based programs of the sort that DeVos has advocated will fail.
This leads us back to the real questions. Does the weight of the evidence give you any confidence that Devos's ideas will produce measureable gains in America's schools? Does the evidence lead you to think that she is the right candidate to become the nation's foremost education leader? I think the answers are clearly, no.
Despite this small mountain of evidence, DeVos has shown no support for oversight and accountability and clear signs of actively opposing it. Her view reflects an extreme aversion to government involvement, even in the case of education, which has all the hallmarks of a market that needs external oversight, coordination, and accountability. Based on the evidence, this free market ideology is standing in the way of good results.
Yes, this really is a triumph of ideology over evidence.