Data-Driven Decision Making Box Scores: Incentivists: 10, Instructionists: 1
What's new is the formalization of this process. Student learning is now quantified in test scores, stored in data warehouses, and made available for teachers to analyze. What's also new is the creation of two very different camps of data-driven reformers, which Sol Stern recently referred to as instructionists and incentivists.
Instructionists like the authors of the book DataWise see data as a useful tool for identifying problems of teaching practice, investigating them, and addressing them. Their focus is on improving student learning, not just test scores, and they are very clear about this distinction. They worry about the dangers of test score inflation and gaming the system, and advise schools to take steps to ensure that their improvements are not simply the result of shortcut practices that do not improve student learning.
Instructionists define data broadly - student work, student attitudes, and more are all relevant. In this view, data are not a replacement for expertise, and data don't make decisions. Rather, data are a useful tool for educators to harness to improve instruction. Instructionists stress that D3M requires a collaborative learning process, and are concerned about approaches that use data to blame individuals rather than support educators' professional growth. For example, Boudett, City, and Murnane wrote in Data Wise:
Agreeing on norms like "no blame" is an essential first step in creating an atmosphere that supports productive data discussions. It is important to emphasize from the beginning that data will not be used to punish teachers, but to help them figure out how to teach their students more effectively.
Incentivists' view of D3M is different. Perhaps best captured in adversarial approaches like CompStat (represented in The Wire as CityStat), D3M is a way to hold people's feet to the fire. This approach is agnostic about the "how" of fixing the numbers, and thus rewards better statistics with little attention to how these numbers were produced. As such, incentivists see little need to study diverse kinds of data - in fact, non-quantified data are dismissed as anecdotal. In this view, data, not educators, are the experts.
Unfortunately, D3M is being (has been?) hijacked by incentivists. The focus is on the numbers, not the process of arriving there, which invites all kinds of mischief. Most of the D3M that I've observed has involved schools figuring out how to cut corners in order to make test score ends meet. And to those who are ready to crucify the teachers for doing so, let me reiterate that it is unrealistic to expect schools to ignore pressure from city, state, and federal muck-a-mucks to rapidly improve test scores, not learning. As long as we only reward end game numbers and ignore the process through which schools get there, I predict that the instructionist approach to D3M will have a hard time getting off the ground. This is regrettable, as the ongoing, systematic analysis of multiple forms of data is an immensely promising strategy for improving teaching and learning.
Tomorrow I'll provide examples (good, bad, and ugly) of how teachers are currently using data. On Friday, I'll investigate the corporations that have entered the K-12 marketplace to provide "data solutions" for school districts.
Comments
I remember a time back in my idealistic youth when we though that the problem was "they just don't know." Example: Johnson uncovering horrendous conditions in Appalachia during the War on Poverty. We thought that the rest of the country just didn't know. Didn't know about how coal companies had strip-mined the land. How families were robbed of the wealth beneath them on their own land by companies who owned the oil and mineral rights.
We thought that it was enough to show the country and the world about the realities of segregation in America. It wasn't enough. It wasn't enough because somebody always benefits from an unjust system. And they don't let go of their ill-gotten gains. It was a rude awakening to us idealists of the 60s and 70s to find out how much we don't want things like equality and justice.
In the present instance, we thought it would be enough to shine a spotlight on the educational injustice that is measured in achievement gaps. We thought that the data would motivate reform for improvement. The instructionists got it. The incentivists didn't. We really believed that education was about democracy--and that we believed in democracy. We didn't suspect the lengths to which we were willing to go to avoid looking at the truth.
Posted by: Margo/Mom | January 23, 2008 11:57 AM
The Eduwonkette, Sol Stern, and Helen Ladd all deserve praise for articulationg issues in a constructive way. The term "incentivists" is not too perjorative. Some have highjacked education for their own political purposes or in order to remain pure in their own ideology. Some incentivists were inspired by promising research and their problem is that they are too far ahead of the curve. We just don't have the knowledge yet to devise a cheap and effective system of data-driven accountability.
Ladd performed a service by phrasing the issues equally well. She acknowledges the costs of her approach but concludes that there is a "good chance" they would work.
"Instructionists," "incentivists," and all other type of "ists," will always be with us. We don't have the luxury of driving any one group from the profession.
Frankly, though, Stern illustrates one reason why I just want to delay NCLB reauthorization until a new president. (If a Republican wins, we're screwed anyway.) We instructionists are on the up swing, and in another 18 months, I'm confident, the research will be even more clear. According to the logic of incentivism, the disappointing results of NCLB should provide incentives for the devlopment of new forms of incentivist approachs. And who knows? Dialogue like in the Wonkette could prompt some hybrid approaches.
DT
Posted by: John Thompson | January 23, 2008 5:06 PM