« EduJello Wrestling, Round 1! Gladwell vs. Gladwell | Main | The Full Page Ad That Won't Be in the Washington Post Tomorrow »

Don't Think about Elephants


"Don’t think about elephants," skoolboy’s father used to joke, long before George Lakoff’s manifesto with a similar name. The joke, of course, is that by trying not to think about elephants, all that you can think about is elephants. The harder I tried not to think about elephants, the more I thought about them.

The New York City Department of Education has its own variation. This month, the DOE is sending Teacher Data Reports, which purport to estimate the effect of individual teachers in grades 4-8 on students’ test scores, to school principals, who will then distribute the reports to their teachers after the principals have been trained. "The Teacher Data Reports are not to be used for evaluation purposes," wrote Chancellor Joel Klein and UFT President Randi Weingarten in an October letter to teachers. "That is, they won’t be used in tenure determinations or the annual rating process. Administrators will be specifically directed accordingly." Similarly, the Frequently Asked Questions section of the DOE’s Teacher Data Tool Kit website poses the question "How can you be sure that principals won’t use the Teacher Data Reports to evaluate teachers?" The response: "Principals have been and will continue to be explicitly instructed not to use Teacher Data Reports to evaluate their teachers. The DOE has standard processes in schools for teachers to raise issues or concerns."

And yet. From the Frequently Asked Questions on the DOE’s Teacher Data Toolkit website: "By isolating individual teachers’ contributions to student progress, the Teacher Data Reports provide valuable information to school leaders and teachers about where to focus instructional improvement efforts. …Teacher Data Reports provide information about how individual teachers’ efforts influence student learning … A sophisticated multivariate regression analysis based on NYC data from 1999-2008 determined how much to weigh each factor [to calculate students’ predicted gains] … A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability. Teachers’ value-added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative."

In other words: The Teacher Data Reports rely on sophisticated statistical techniques that are valid, reliable and approved by experts, and they isolate an individual teacher’s contributions to student learning. But, you principals who are under tremendous pressure to increase test scores or face losing your jobs, don’t you dare think about using these Teacher Data Reports to evaluate teachers.

Don’t think about elephants.


At best this is a tremendous waste of resources. Technical experts, statiticians, peer review...$$$$$$!! All to see which subjects/objectives an individual teacher's students scored poorly on-- which can be seen with regular test results?! But more likely this is an insidious attempt to carry out the Klein/Rhee/edu-biz agenda to use teachers as scapegoats in their "reform".

If you want real teaching quality, then this is the way to go. Intimidate them to no end, and they'll be scared straight. People wonder why the little tenets of NYC teaching like tenure and pension are so sacred to us. If not for things like that, this "tracking" sheet would spell bad news for teachers, especially those who have more challenging populations. "Elephants" is right: there's a huge elephant in the room and we're expecting principals to ignore them and teachers to move them out of the room.

Unfortunately, this protocol reminds me of a line Fake Steve Jobs once used to describe management at Apple: "Hold people to an impossibly high standard, but here’s the twist — don’t tell them what that standard is. And fire them if they fall short. You know what that does to people? It makes them crazy... Crazy people are more creative. And more productive."

I definitely agree with your argument. Given that the reports "rely on sophisticated statistical techniques that are valid, reliable and approved by experts," then they SHOULD be used to evaluate teacher performance and it's counterproductive to attempt to prevent principals from using this invaluable source of information. Right on.

Hey SB,

I should clarify that it's the NYC DOE's assertion that the Teacher Data Reports "rely on sophisticated statistical techniques that are valid, reliable and approved by experts," not skoolboy's. What I'm highlighting is the incongruity of the DOE making such an assertion while prohibiting the use of the reports for evaluating teachers. Currently, New York State law prohibits the use of student performance data in making teacher tenure decisions.

I think prohibiting the use of student performance data in making tenure decisions is ridiculous; but I am naturally skeptical about claims that a complex statistical model can isolate a teacher's contribution to student learning, and fear that the veneer of science -- "sophisticated statistical techniques that are valid, reliable and approved by experts" -- could elevate the influence of Teacher Data Reports and other value-added assessment systems beyond what the data can actually tell us.

I came here to say what you said in your last comment. We could add that it's very tempting to compare teachers with a reductionist metric that leaves you with a number when other measures seem more amorphous. That's one reason that college admissions officers are always in danger of leaning too heavily on GPA and SAT scores.

So, here's the question:

It seems like I've every expert in the field I've read says that evaluations like this one are deeply flawed. And yet the NYC DOE is claiming that a panel of experts have signed off on them. So, who are these experts that have done so?

Is there less consensus in the research community than I think, or are the people that Klein cites not actually experts? Or is there some other explanation?

According to the NYC website link you provided, this test data covers certain teachers in grades 5 - 8 on two subjects. What about all the other teachers of all the other subjects and grade levels?

Like Corey, I'd like to know specifically who are the "experts" that have signed off on these results and what methodology did they use to verify these findings?

Comments are now closed for this post.


Recent Comments

  • Renee Moore: According to the NYC website link you provided, this test read more
  • Corey: So, here's the question: It seems like I've every expert read more
  • Dan Willingham: Skoolboy I came here to say what you said in read more
  • skoolboy: Hey SB, I should clarify that it's the NYC DOE's read more
  • SB: I definitely agree with your argument. Given that the reports read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School