« Carnival of Education Pre-Party | Main | Exceptional Ed Week Commentary on Testing and Accountability »

Data-Driven Decision Making Box Scores: Incentivists: 10, Instructionists: 1

In many ways, data-driven decision making (D3M) in education is an old idea packaged as a new one. As far back as anyone can remember, teachers have given their students regular quizzes, projects, and tests. When students performed poorly, "data-driven" teachers retaught the material or tried to figure out what went wrong. Without the benefit of spreadsheets or data displays, teachers have attempted to tailor their instruction to different groups of students. To be sure, there have been assumptions, blindspots, and kids overlooked, but the fundamental idea of teaching, assessing, figuring out what works for whom, and re-teaching is as old skool as Tupac.

What's new is the formalization of this process. Student learning is now quantified in test scores, stored in data warehouses, and made available for teachers to analyze. What's also new is the creation of two very different camps of data-driven reformers, which Sol Stern recently referred to as instructionists and incentivists.

Instructionists like the authors of the book DataWise see data as a useful tool for identifying problems of teaching practice, investigating them, and addressing them. Their focus is on improving student learning, not just test scores, and they are very clear about this distinction. They worry about the dangers of test score inflation and gaming the system, and advise schools to take steps to ensure that their improvements are not simply the result of shortcut practices that do not improve student learning.

Instructionists define data broadly - student work, student attitudes, and more are all relevant. In this view, data are not a replacement for expertise, and data don't make decisions. Rather, data are a useful tool for educators to harness to improve instruction. Instructionists stress that D3M requires a collaborative learning process, and are concerned about approaches that use data to blame individuals rather than support educators' professional growth. For example, Boudett, City, and Murnane wrote in Data Wise:

Agreeing on norms like "no blame" is an essential first step in creating an atmosphere that supports productive data discussions. It is important to emphasize from the beginning that data will not be used to punish teachers, but to help them figure out how to teach their students more effectively.

Incentivists' view of D3M is different. Perhaps best captured in adversarial approaches like CompStat (represented in The Wire as CityStat), D3M is a way to hold people's feet to the fire. This approach is agnostic about the "how" of fixing the numbers, and thus rewards better statistics with little attention to how these numbers were produced. As such, incentivists see little need to study diverse kinds of data - in fact, non-quantified data are dismissed as anecdotal. In this view, data, not educators, are the experts.

Unfortunately, D3M is being (has been?) hijacked by incentivists. The focus is on the numbers, not the process of arriving there, which invites all kinds of mischief. Most of the D3M that I've observed has involved schools figuring out how to cut corners in order to make test score ends meet. And to those who are ready to crucify the teachers for doing so, let me reiterate that it is unrealistic to expect schools to ignore pressure from city, state, and federal muck-a-mucks to rapidly improve test scores, not learning. As long as we only reward end game numbers and ignore the process through which schools get there, I predict that the instructionist approach to D3M will have a hard time getting off the ground. This is regrettable, as the ongoing, systematic analysis of multiple forms of data is an immensely promising strategy for improving teaching and learning.

Tomorrow I'll provide examples (good, bad, and ugly) of how teachers are currently using data. On Friday, I'll investigate the corporations that have entered the K-12 marketplace to provide "data solutions" for school districts.

I remember a time back in my idealistic youth when we though that the problem was "they just don't know." Example: Johnson uncovering horrendous conditions in Appalachia during the War on Poverty. We thought that the rest of the country just didn't know. Didn't know about how coal companies had strip-mined the land. How families were robbed of the wealth beneath them on their own land by companies who owned the oil and mineral rights.

We thought that it was enough to show the country and the world about the realities of segregation in America. It wasn't enough. It wasn't enough because somebody always benefits from an unjust system. And they don't let go of their ill-gotten gains. It was a rude awakening to us idealists of the 60s and 70s to find out how much we don't want things like equality and justice.

In the present instance, we thought it would be enough to shine a spotlight on the educational injustice that is measured in achievement gaps. We thought that the data would motivate reform for improvement. The instructionists got it. The incentivists didn't. We really believed that education was about democracy--and that we believed in democracy. We didn't suspect the lengths to which we were willing to go to avoid looking at the truth.

The Eduwonkette, Sol Stern, and Helen Ladd all deserve praise for articulationg issues in a constructive way. The term "incentivists" is not too perjorative. Some have highjacked education for their own political purposes or in order to remain pure in their own ideology. Some incentivists were inspired by promising research and their problem is that they are too far ahead of the curve. We just don't have the knowledge yet to devise a cheap and effective system of data-driven accountability.

Ladd performed a service by phrasing the issues equally well. She acknowledges the costs of her approach but concludes that there is a "good chance" they would work.

"Instructionists," "incentivists," and all other type of "ists," will always be with us. We don't have the luxury of driving any one group from the profession.

Frankly, though, Stern illustrates one reason why I just want to delay NCLB reauthorization until a new president. (If a Republican wins, we're screwed anyway.) We instructionists are on the up swing, and in another 18 months, I'm confident, the research will be even more clear. According to the logic of incentivism, the disappointing results of NCLB should provide incentives for the devlopment of new forms of incentivist approachs. And who knows? Dialogue like in the Wonkette could prompt some hybrid approaches.


Comments are now closed for this post.


Recent Comments

  • John Thompson: The Eduwonkette, Sol Stern, and Helen Ladd all deserve praise read more
  • Margo/Mom: I remember a time back in my idealistic youth when read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School