« Guest Blogger Scott McLeod on Data-Driven Decision Making | Main | Do Quality Reviews Lead to Increased Student Achievement? »

Quality Reviews and the Fetishization of Data: A Fantasy

skoolboy returns for part II on Quality Reviews. You can find his first post here.

The year is 1975. Coach John Wooden of UCLA has just won his 10th NCAA men’s basketball championship in 12 years, a record that will likely never be matched in collegiate sports. Cambridge Associates sends Clive Wingtip to conduct a Quality Review of the UCLA program. Over the course of a day and a half, Wingtip talks with Coach Wooden, his assistant coaches, the players, and other staff, and observes the team practices. He also observes a collaborative activity: a meeting between Coach Wooden and his assistant coaches. The program is evaluated on five quality statements, each scored as either underdeveloped; underdeveloped with proficient features; proficient; well developed; or outstanding. Here’s a summary of the report he filed:

Quality statement 1: “The coach and staff consistently gather and generate data and use it to understand what each player knows and is able to do, and to monitor the player’s progress over time.”

There is evidence that Coach Wooden studies each individual, and his strengths and weaknesses, very carefully. But he does not rely on statistics collected during practice sessions or games to inform his judgment, and his observations are subjective, not objective. He does not measure performance and progress based on comparisons to similar schools. Overall Score: Underdeveloped

Quality statement 2: “The coach and staff consistently use data to understand each player’s next learning steps and to set suitably high goals for accelerating each student’s learning.”

The Coach and staff convey consistently high expectations to the players, and set specific goals for the team and for individuals. But Coach Wooden regularly played only 7 of the team’s 12 in games, and these same 7 practiced as a unit, suggesting that the reserves were not as important as the regulars. Coach Wooden occasionally displays the soft bigotry of low expectations. He states, “There is nothing wrong with the other fellow being better than you are, as long as you did everything you possible could to prepare yourself for the competition. That is all you have control over. It may be that the other fellow’s level of competency is simply higher than yours. That doesn’t make you a loser.” Overall Score: Underdeveloped with Proficient Features

Quality statement 3: “The program aligns its work, strategic decisions and resources, and effectively engages players, around its goals and plans for accelerating players’ learning.”

Although there was evidence of individualized instruction, the Coach and staff did not use objective team and individual data to plan for and provide this instruction. Team members trusted and respected Coach Wooden. Overall Score: Proficient

Quality statement 4: “The program has structures for monitoring and evaluating each player’s progress throughout the year and for flexibly adapting plans and practices to meet its goals for accelerating learning.”

Coach Wooden relies heavily on repetition during the season. One player said, “He never talks about strategy, statistics, or plays but rather about people and character.” The program did not rely on objective measures to assess progress towards goals, such as the final score of games or written tests, and did not have a playbook. Coach Wooden frequently adjusted the plans for each practice, and made notes after each practice about adjustments; but these notes were based on subjective judgments, not hard data on performance. Overall Score: Underdeveloped with Proficient Features

What’s wrong with this picture? How does the man voted Coach of the Century by ESPN receive a quality review rating of “underdeveloped”? It’s all about the quality criteria, which privilege using data to make decisions about how to help student/athletes to learn and develop over simply making good decisions about teaching and learning. I refer to this as the “fetishization of data.” In the Quality Review game, quantitative performance data have become an end in themselves, rather than a means to an end. One of the most cutting insults that one social scientist can hurl at another is to label another’s research as a “data dump.” In some school districts which have embraced external quality reviews, compiling notebooks full of undigested data has become a substitute for thoughtful analysis of a reasonably small number of important themes and problems.

What’s the solution? I’d start by broadening the definition of data beyond quantitative performance measures. There’s no doubt in my mind that Coach John Wooden relied heavily on data to inform the design of his practices, and his approach to cultivating the talents of his players and his team. But those data took the form of the systematic observations and judgments of an expert practitioner. I’d also seek to evaluate schools on the basis of the quality of the teaching within them, not whether the educators in a school arrived at their teaching practices via the analysis of quantitative performance data.

“Don’t mistake activity for achievement…If you spend too much time learning the tricks of the trade, you may not learn the trade.” (John Wooden)

Absolutely fanatastic explanation! I wish I had thought of it.

Today's HARD data accountability true believers are especially guilty of violating Coach Wooden's prime dictum, "Be quick but don't hurry."

"What’s the solution? I’d start by broadening the definition of data beyond quantitative performance measures."

Or, maybe too obvious, noticing that best decisions take into account more than just data...

"Good nutrition depends on eating lots of fruits!" (what about vegetables and whole grains?) "And from now on, vegetables and whole grains are fruits!"

Right? We can pretend that everything is data. Or we can say that data should help make decisions, but there's more good stuff out there than just data.

Comments are now closed for this post.


Recent Comments

  • Jonathan: "What’s the solution? I’d start by broadening the definition of read more
  • John Thompson: Absolutely fanatastic explanation! I wish I had thought of it. read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School