An item I wrote last week about a Northwestern University study that found a link between the No Child Left Behind law and rises in students’ math scores generated a fair amount of comments, most of them disbelieving of the results. No one, though, seemed to question the researchers’ use of Catholic schools as a comparison group, which I thought was pretty novel.
Perhaps they might have, though.
Researchers Manyee Wong, Thomas D. Cook, and Peter Steiner chose Catholic schools because they were not subject to the same sanctions facing public schools under the federal law. But other research suggests that this may not be the most convincing control group for gauging NCLB’s impact.
That’s because enrollment in Catholic schools dropped significantly right after NCLB was enacted—and right about the time the Boston Globe began running stories uncovering the Boston archdiocese’s role in child sex-abuse scandals in that city. You’ll recall that this news was followed up over the next few years with stories of similar scandals across the country. Read the appendix in this 2009 working paper by Thomas Dee and Brian Jacob for details on that enrollment trend.
The researchers calculate that the student population in Catholic schools fell 10 percent between 2002 and 2004 and then fell an additional 7 percent between 2004 and 2006. While the enrollment drop did not change the racial and ethnic composition of the Catholic-school population, it did coincide with a rise in the average level of educational attainment of Catholic school parents, according to Dee and Jacob.
They write:
One possible explanation for this pattern is that financial pressure on dioceses which were compelled to respond to civil litigation may have led to tuition increases that led more poorly educated parents to withdraw their children from Catholic schools. Overall, these data provide at best suggestive evidence for non-random attrition from Catholic schools.
It’s hard to know how the Catholic-school enrollment declines would have affected school quality. On the one hand, the researchers surmise, the drop might have resulted in smaller classes, which might be a good thing for students. On the other, the scandal could have eroded levels of social trust in Catholic-school communities. Either way, the numbers suggest that something different was going on in Catholic schools during those years.
For that reason, the pair of researchers use a different comparison in their own study of NCLB’s impact. They instead compared improvement patterns on NAEP tests in states that already had “NCLB-like” accountability systems in place prior to the law with those in states that made dramatic changes after the law was put in place. Even so, they reached essentially the same conclusion as the Northwestern researchers did: They find that NCLB spurred test-score improvements in math but not in reading. Read all about it in this EdWeek story from November.