This month, Rick is out catching up on various and sundry projects that piled up during the rollout of Letters to a Young Education Reformer. In his stead, we’ve got a terrific slate of guest bloggers. Up this week is Collin Hitt, assistant professor of medical education at Southern Illinois University School of Medicine.
There is fascinating charter-school research on the horizon. On Monday and Wednesday I waxed on about CTE and early-college studies that I think are overlooked classics. I love it when I come across work that neither myself nor my colleagues have heard of. And I have another past time: scouting out studies that are in the works. I’ve been known to dog research teams that are working on projects I’m following for details on release day or even a snapshot of their findings. A few of my favorite artists are in the studio as we speak.
A University of Michigan evaluation of that state’s charter schools is underway. It’s headed by Susan Dynarski and Brian Jacob and—like so much of their work—will be using a random-assignment research design. I wonder why, during all the debate over Betsy DeVos and the impacts of Detroit charter schools, nobody mentioned that a gold-standard study was coming along that would help settle this argument. The sophists will tell you that research doesn’t matter to politics. They are wrong, and I think this is one of the most politically important pieces of research in the works right now.
Up on Cripple Creek, Pat Wolf and Gary Ritter are heading an evaluation of Arkansas charter schools. The working title (really) is, “You Can’t Always Get What You Want.” To date, most of the rigorous research on charter schools comes from the East coast or from no-excuses networks. The Arkansas and Michigan evaluations literally cover new and important ground.
For an overview of the gold-standard charter research, if I may, see a meta-analysis that my colleagues and I published earlier this year in the Journal of School Choice. We compare the impacts on No Excuses schools to other charters. We’ll update the analysis in time, once there are more studies to add. That’ll likely include MDRC’s forthcoming evaluation of Success Academy, and Sean Corcoran and Sarah Cordes’s work on Democracy Prep.
One of the remarkable things about No Excuses schools is how consistently they increase reading and especially math scores. We find grand (i.e. weighted average) effects of 0.17SD in ELA and 0.25SD in math. Very few programs have ever had such large effects on standardized tests across so many settings. And yet I often wonder, so what?
I’ve spent the last week highlighting studies of programs that produce long-run benefits largely without improving test scores. This should invite criticism of tests as our primary metric of success. Tests have always had their critics, and so has test-based accountability. I have not always been one of those people.
The achievement gap in math and reading scores was my primary motivator to enter a career in education policy—and I feel it’s important to know that schools can have a huge impact on test scores. There was a time when people denied that schools could do anything big on reading and math skills. No Excuses schools have proved them wrong.
But do No Excuses schools produce long run benefits, of a magnitude similar to their test-score impacts? It’s too soon to know. The evidence is scant. But the early returns aren’t great.
We could quibble about the handful of studies to speak to this issue. But instead, let’s spend the remainder of my time as your host on RHSU talking about a study that should be a role model for future charter-school research. It appeared three years ago in the journal...Pediatrics.
A team of physicians and social scientists conducted a random-assignment evaluation of three inner-city California charter high schools. They collected the usual outcomes, test scores, grade retention, and graduation rates. But they also did a massive amount of original data collection, including on drug use and other risky behaviors:
We also examined 11 very risky behaviors: the most common were sex without contraception (13.4%), inconsistent condom use (11.6%), binge drinking (8.3%), alcohol or drug use with sex (7.1%), marijuana use at school (6.4%), and drug use other than marijuana (6.3%). Prevalence was ,5% for each remaining very risky behaviors (alcohol use at school, current pregnancy, multiple sex partners, carrying a weapon to school, and gang membership). The proportion engaging in (at least one) of these 11 very risky behaviors was 41.9% in the control group and 36.3% in the intervention group, with an adjusted odds ratio (OR) of 0.73 (P = .048). The intervention effect on engaging in a very risky behavior was more pronounced among those who had attended a low-performing middle school (OR = 0.67, P = .032) than among those who had attended a high-performing middle school (OR = 0.85, P = .40).
Yes, the schools also improve test scores, but that doesn’t appear to be driving the impacts on risky behavior:
Only retention and math and English CST scores were significant mediators (P , .001 and P = .06) and reduced the intervention effect on engagement in very risky behaviors after being individually added into the regression model. These 2 mediators explained 58% and 16.2% of the intervention effect, respectively
Retention explains far more of the impacts on risky behavior. But even then, almost half of the impact is unexplained. Whatever it is these schools are doing to impact kids’ lives is not captured by our interim measures.
The study has its weaknesses, which the authors are extremely clear about. Not all students participated in the health survey. But word is that the research team is doing a follow-up study that will address those issues.
Now that’s education research that I really can’t wait to see. And if it’s anything like its predecessor, it will be appearing in a journal of...medicine.
—Collin Hitt