Madame Secretary Demands Triage, Randy Reback Delivers
"We need triage," Madame Secretary explained last week. This morning, Randy Reback delivered it to my inbox via the Journal of Public Economics' new issue, which includes his paper, "Teaching to the Rating: School Accountability and the Distribution of Student Achievement." Reback analyzed data from Texas, the birthplace of NCLB-style accountability, and here's what he found:
* Schools respond to math performance incentives both by targeting math resources towards specific students and by making broad changes which also help very low achieving students. These responses tend to sacrifice the targeted students’ reading performance and to sacrifice relatively high achieving students’ performance in both math and reading.
* Schools respond to reading performance incentives by targeting resources towards the reading performance of particular students, sacrificing these students’ math performance and sacrificing all other students’ performance in reading.
* Finally, schools devote fewer resources towards students in the terminal grades during years when short-run incentives are low than during years when incentives are high.
Reback concluded:
Whether the finding of non-trivial distributional effects is a positive or negative outcome of this public policy is entirely subjective. If one of the primary goals is to create a sort of educational triage, in which students below minimum grade-level skills are pushed up, then the No Child Left Behind type of accountability system appears to be fairly effective. Furthermore, the results say nothing about the overall impact of this system on performance: it may be a rising tide that lifts all boats (and lifting some more than others), or it may be a falling tide sinking all boats (and sinking some less than others).
The important lesson here is that schools respond to the specific instructional incentives created by the accountability system. Schools' responses include targeting specific students, targeting specific subjects, and making broad changes which affect all students. An accountability system should only create disproportionate incentives concerning student achievement gains if the intention is to help some students more than others and to boost performance in some subjects by more than others. Otherwise, the optimal accountability system requires a more evenhanded approach.
* Schools respond to math performance incentives both by targeting math resources towards specific students and by making broad changes which also help very low achieving students. These responses tend to sacrifice the targeted students’ reading performance and to sacrifice relatively high achieving students’ performance in both math and reading.
* Schools respond to reading performance incentives by targeting resources towards the reading performance of particular students, sacrificing these students’ math performance and sacrificing all other students’ performance in reading.
* Finally, schools devote fewer resources towards students in the terminal grades during years when short-run incentives are low than during years when incentives are high.
Reback concluded:
Whether the finding of non-trivial distributional effects is a positive or negative outcome of this public policy is entirely subjective. If one of the primary goals is to create a sort of educational triage, in which students below minimum grade-level skills are pushed up, then the No Child Left Behind type of accountability system appears to be fairly effective. Furthermore, the results say nothing about the overall impact of this system on performance: it may be a rising tide that lifts all boats (and lifting some more than others), or it may be a falling tide sinking all boats (and sinking some less than others).
The important lesson here is that schools respond to the specific instructional incentives created by the accountability system. Schools' responses include targeting specific students, targeting specific subjects, and making broad changes which affect all students. An accountability system should only create disproportionate incentives concerning student achievement gains if the intention is to help some students more than others and to boost performance in some subjects by more than others. Otherwise, the optimal accountability system requires a more evenhanded approach.


Comments
Interesting analysis. I see something similar with respect to elementary schools and NCLB from the Center on Education Policy, a pretty balanced policy center in DC.
Think it was based on a survey in some 40-odd school districts, but can't swear to that. It found that NCLB's incentives drive additional time to reading and math at the cost of time for science, social studies, gym, phys. ed., recess and lunch.
In an age where policymakers claim to be looking for more emphasis on science and citizenship, along with greater attention to physical fitness and the challenge of obesity, a policy that drives up instructional time for reading and math at the cost of reduced time science, civics, and phys. ed. seems to have unanticipated consequences. NCLB may be satisfying from a short term accountability point of view, but counter-productive in the long terms in light of national interests.
Posted by: wonkery123 | March 23, 2008 5:25 AM
Just a matter of days ago, Sec. Spellings announced announced that her department was rady to use a more "nuanced" approach to accountability for those schools most in need. The latest post on
morethanascore.org is worth checking out for more about wrongheaded this administration has been in the implementation of the deeply flawed law.
Posted by: kathyf | March 24, 2008 6:04 PM