The “still not dead” voucher issue came up again recently when Jay Mathews of WaPo pointed out an updated study from the Foundation for Educational Choice, which claims that vouchers have positive effects on both participating students and non-participating students who remain in public schools.
The report uses the word “science” with great force and frequency, and asserts that random-assignment experiments (several of which show favorable evidence for voucher programs) are the “gold standard” of educational research. To a point, I agree that experiments provide helpful clarity, but they don’t answer complex policy questions for us.
Random-assignment experiments are only the gold standard research methodology for the narrow questions they are designed to answer. For example, a study of whether students who apply for vouchers do better in public schools or voucher-funded private schools can be answered conclusively with a random-assignment experiment:
Unfortunately, it is usually not possible to conduct random-assignment research on education policy. However, school vouchers have been one of the rare exceptions. When there are more applicants for a voucher program than there are slots available, a random lottery is often used to determine who may participate. This creates a naturally occurring random-assignment research design. Students who win the lottery and are offered vouchers can be compared to students who lost the lottery and were not offered vouchers. Win-Win Solution, p. 9
However, broader questions such as “How do vouchers affect public schools and students who do not apply for them?” are not capable of being resolved by an experiment. Random-assignment experiments, like lasers, are powerful tools for focused purposes, but a laser is no substitute for a light bulb. Policy questions require more illumination than a laser alone can provide.
The report attempts to address the impact of vouchers on non-participating students, asserting that
the claim that vouchers "cream" or attract the best students from public schools has no empirical evidence to support it. The best available analyses of this question have found voucher applicants to be very similar to the population of students eligible for vouchers in terms of demographics and educational background.
The argument that vouchers don’t “cream” better-performing students from public schools is both ridiculous and unmeasured. Of course students who apply to voucher programs are different in significant ways from students who do not apply. This might not be evident when extraordinarily broad demographic categories such as ethnicity and FRL status are the sole basis for comparison, but in districts where virtually all public schools students qualify for free or reduced-price lunch, asserting that they are therefore equivalent is absurd. Test scores add something to the equation, but not enough.
The studies the report cites as evidence that vouchers don’t harm—and actually spur improvement in—public schools tend to fall victim to a common flaw: They ignore the statistical tendency of outliers to regress to the mean. In other words, they ignore the fact that the very worst schools tend to be not quite so bad in subsequent years. At the individual level, low-performing students tend to make larger test score gains than high-performing students.
Even ignoring the regression effect, though, low-performing schools are the target of countless interventions; vouchers are far from the only influence on their performance, and it seems much more likely that vouchers are merely hitching a ride on other reforms.
This matters because existing voucher programs disproportionately affect low-performing schools. Finding that public schools affected by voucher programs tend to improve slightly (and in many cases, the gains are tiny) should not surprise us, and it should certainly not convince us that vouchers are the panacea we’ve been waiting for.
While hopefully the voucher issue will continue to remain politically dead in most of the country, it’s at least interesting to consider the way research is used in the debate.