Evaluating the Evaluators of Public Schools
Taxpayers today are exposed to a stream of studies about public education that purport to be based on scientific research. Yet too often the reports issued by institutions with impressive sounding names turn out to be little more than junk science. At least that's the conclusion of a new book, Think Tank Research Quality: Lessons for Policymakers, the Media, and the Public (Information Age Publishing, 2010). It argues that a large number of studies are policy briefs, rather than original research.
This view has far-reaching implications in the debate over how to improve educational quality for all students. Taxpayers have the right to know if students are being well educated. But how many taxpayers possess the expertise to evaluate what they read online or in print? Policy briefs are often written by think tanks with specific agendas that are not always transparent. What they write seems plausible, but is it the truth? It's important to remember that statistics were originally known as political arithmetic. But because those who collected data were trying to improve the state of the state, they were called statists, and their numbers came to be called statistics.
The editors of Think Tank Research Quality, Kevin G. Weiner, Patricia H. Hinchey, Alex Molnar and Don Weitzman, perform a great service by alerting readers to the pitfalls surrounding such controversial issues as vouchers, charter schools, preschool, and teacher quality. Their approach is balanced and informative at a time when debates too often generate more heat than light. For example, in a chapter titled "Markets Versus Monopolies in Education," Clive Belfield cautions that when it comes to parental choice, not all studies "include the controls they should." As a result, unsophisticated readers fail to appreciate the role that selectivity bias plays in outcomes. Similarly, in a chapter titled "How Much Are Public School Teachers Paid?," Sean P. Corcoran and Lawrence Mishel emphasize that the Bureau of Labor Statistics, which collects and publishes this data, explicitly advises users not to compare the hourly pay of teachers to other professions. Nevertheless, the warning is ignored, leading taxpayers to conclude that teachers are overpaid.
Think Tank Research Quality needs to be read in conjunction with Reading Educational Research: How To Avoid Getting Statistically Snookered (Heinemann, 2006) by the late Gerald Bracey. His 32 principles of data interpretation serve as a compass through the thicket of studies being published and then uncritically reported by the media. As Jay Mathews wrote in the foreword, "(Bracey) acknowledges several times that no combatant in the bitter education policy wars has an unquestionable grasp on the truth." Yet taxpayers, whose support for public schools is crucial, tend to accept what they read from think tanks as gospel.
Principle No. 13 serves as a case in point. It states: "Do not confuse statistical significance and practical significance." How many times have reports been headlined in the media as proof of one thing or another simply because the researchers have included an impressive array of numbers? Even if the data are statistically significant, they do not necessarily have any value for the classroom. But when teachers attempt to make that point, they are labeled obstructionists who favor the status quo.
Principle No. 7 warns: "Beware of simple explanations for complex phenomena." The persistent academic achievement gap between different racial groups is an example. Several studies have cited "high-flying schools" as evidence that poverty is no excuse for underachievement. If these schools can be successful, then why can't all schools? But Bracey never said that schools don't matter. What he said is that the quality of schools has little influence on the difference in average achievement between advantaged and disadvantaged students. This is a vital distinction that is overlooked.
In light of the importance given to public school accountability, I try to approach each study with a healthy skepticism. I don't make many friends that way, but too much is at stake to take a less vigilant stance.