Wired Magazine recently published a fascinating article on how lottery scratch-off games can be beaten and used for money-laundering. How did officials find out? Through the use of forensic statistics, which found unusual patterns of lottery wins among suspected mobsters.
Earlier this week, USA Today published a major investigative piece on erasure rates on DC standardized tests, and found through similar forensic statistics that something was amiss while test scores were rising during Michelle Rhee’s tenure as superintendent. In short, many schools were found to have alarmingly (and statistically improbable) high numbers of answers changed from wrong to right on student bubble sheets. In other words, it’s very likely that schools cheated, and the district seems to have swept the issue under the rug.
The story is a damning indictment of Michelle Rhee’s hard-charging, results-or-else leadership style:
The pressure on principals was unrelenting, says Aona Jefferson, a former D.C. principal who is now president of the Council of School Officers, representing principals and other administrators. Every year, Jefferson says, Rhee met with each principal and asked what kind of test score gains he would post in the coming school year. Jefferson says principals told her that Rhee expected them to increase scores by 10 percentile points or more every year. "What do you do when your chancellor asks, 'How many points can you guarantee this year?' " Jefferson says. "How is a principal supposed to do that?"
While Rhee is surely to be credited with improving many district operations, I don’t know of any serious scholars of education who advocate this kind of goal-setting, which seems perfectly designed to foster improper behavior. If the emphasis is only on outcomes, and not at all on substantive improvements in the conditions and processes for teaching and learning, people are likely to cut corners to hit the target.
What results-or-else improvement lacks is a theory of action. Having a goal, at least for a complex outcome like student achievement, does not actually help you bring about improvement. This is different from how more granular goals work. For example, if I need to write 500 words or walk a mile, I can just apply myself until I hit the target.
But if my goal is complex, like building a rocket ship, no amount of goal-setting and chest-beating is going to get me there; I’m going to have to take a series of capacity-building steps toward the goal. If you want to ensure that I reach the goal, you’d be much better off focusing on my progress toward those proximal milestones.
This should be obvious: Schools improve when we improve the conditions that lead to better outcomes, not when we yes-we-can ourselves into a frenzy over standardized test scores. As Donald Campbell predicted decades ago, when a social metric is overemphasized as a target, it distorts the very process it’s intended to measure.
The greater the rewards and sanctions associated with specific numeric targets, the more likely people are to cheat. In DC schools, every indication is that the problem was widespread, especially among schools “improving” rapidly:
Among the 96 schools that were then flagged for wrong-to-right erasures were eight of the 10 campuses where Rhee handed out so-called TEAM awards "to recognize, reward and retain high-performing educators and support staff," as the district's website says. (USAT)
I don’t know if Rhee paid any attention to what was actually happening in schools, but it’s clear that the emphasis on test scores was out of proportion to other types of monitoring for improvement.
Interestingly enough, the testing company, McGraw-Hill, indicated that they are aware of the dangers of Campbell’s Law when they warned against a different manifestation of it, saying that wrong-to-right answer changes should not be equated with cheating, lest students become fearful of checking their work.
If only DC school officials were so concerned about the impact of their efforts.