Most studies of charter schools use unsophisticated methods and are flawed in ways that prevent researchers from accurately gauging those institutions' impact on student achievement, a new review concludes.
And while researchers have options for collecting more accurate information about charter school performance, they also face obstacles along the way—some of them related to the unwillingness among states and schools to provide crucial data, the analysis finds.
A meta-analysis of charter school studies revealed that about 75 percent of them do not meet rigorous research standards because they don't account for the differences in academic background and academic histories of students attending charters, when comparing them with those attending traditional public schools, according to the review, published in the renowned journal Science. Those studies typically fail to "disentangle school quality from the preexisting achievement level," or student self-selection of schools, the article says.
The article was written by Julian R. Betts, a professor of economics at the University of California, San Diego, and Richard C. Atkinson, a former president of the University of California system who also once served as director of the National Science Foundation.
High-quality research on charters is nonetheless beginning to emerge, they say. Much of it is coming from charters that have so many applicants that they must use lotteries for admission. Because getting in or not getting in is based on chance, the students who fail to secure a spot represent a sound "control" group necessary for a study, Betts and Atkinson say.
The relatively small number of lottery-based studies of charter schools have generally shown that they either outperform or perform at the same level as traditional public schools, according to the authors. But those studies cover only a small fraction—about 2 percent—of charter schools nationally.
Of course, many education advocates and scholars have long been frustrated by the paucity of rigorous research across the spectrum of school policy—not just with charters. (See my colleague Debbie Viadero's exceptional reporting on this.) The shortage of research is sometimes attributed to the difficulty of arranging high-quality, sustained studies of student populations in school settings.
While the authors say that more randomized controlled trials should be done on lottery-based charter systems, they also recognize limitations in that approach. Most of the nation's charters, for one, do not have more applicants than space, meaning that such studies might not reveal much about the overall effectiveness of charters. In addition, charters that are so popular they need lotteries for admission may be unusually good schools, the authors say, skewing the overall picture.
State policies also stand in the way of good charter research, say Betts and Atkinson.
State laws typically don't require charter schools to make public information about their lottery systems, a policy the authors call "shortsighted."
"Lottery data should not be viewed as the property of the charter school," they argue. "It is incumbent upon authorizers to gather and scrutinize these data, not least to verify that the lotteries are being done in a fair manner."
States could do this and still protect individual students' identities, they say. And the authors believe states can take steps to ensure that researchers are allowed to study and evaluate the lottery process, so that they can understand the process charters follow in admitting students from their wait lists—and whether that process is random or shows favoritism of one kind or another.