« And the Winner is.... | Main | Watch Out, Elizabeth Green, Erin Einhorn, & Jenny Medina!!! »

Who Slipped a Mickey in John Merrow's Kool-Aid?

It wasn't me, that's for sure. John Merrow shed crocodile tears this morning over public education's "upside-down universe where student outcomes are not allowed to be connected to teaching." He gives a special shout out to New York, though conveniently fails to mention that our tests are administered smack in the middle of the year. We'll give him a pass for forgetfulness - but watch your drink next time, J.

You can't swing a fish anymore without hitting a glitch in value-added models - try some of the papers from the recent Wisconsin Value-Added Conference on the complexities of measurement error (value-added models must contend with measurement error in both last year and this year's scores), interval scaling (few tests are scaled so that 1 unit of growth at the bottom of the scale means the same thing as 1 unit of growth at the top of the scale), and non-random assignment (see Jesse Rothstein's new paper on just how large these biases can be). Or you can refer to these earlier posts:

* skoolboy on: The Status of the Status Quo in Education Policy
* More Signs of the Apocalypse! (More on NY's Teacher Tenure Law)
* After NY's Teacher Tenure Law, Blogosphere Plays Union Pinata
* My Value-Added Bucket List
* Do Value-Added Models Add Value? A New Paper Says Not Yet
* The Oops Factor in Measuring Teacher Effectiveness
* Ignoring the Great Sorting Machine
* No Teacher is an Island
* What Does It Mean for a Teacher to Be Good?

Alexander Russo, also commenting on Merrow, makes the mistake of equating teachers' evaluation of students with tests and quizzes with the evaluation of teachers by students' test scores. It's just a bad comparison. Teachers give tests, assignments, reports, homeworks, etc in order to evaluate students and to see what they've learned. These measures are part of an extended interactive process through which a teacher hopes to move students forward. The purpose is not simply to label a student as "good" or "bad" based on one assessment. But when we evaluate teachers based on students' scores, the teacher is being evaluated on a more narrow set of skills than are students, and high-stakes are attached to a single test. So the intent of the process is different; few value-added plans are designed to help teachers improve, but focus instead on assigning rewards and sanctions.

The measurement issues are also different. In an elementary school year, a teacher probably collects 900 data points on student performance (let's say 5 a day); with teacher value-added, we end up with 20-25 data points a year. Teacher value-added is, in short, a low precision enterprise. Readers, what do you think of Alexander's comparison?

Happy weekend, everyone!

It's not clear to me that teachers are absolutely opposed to all assessments. And, even if they were, I'm not sure that the fact they assess students all day would make them bad people.

I'm saddened that John Merrow got sloppy in thinking through the difference between "student performance" and "student performance data."

He wrote accurately that "Now, administrators are required to 'drill down' ... to pinpoint who seems to be an effective teacher." But he didn't even speculate about what percentage of administrators have met that requirement.

Merrow said that "I have met with superintendants, principals, and department chairs armed with this data" ... who identified teachers' strengths and weaknesses, presumably in a fair manner.

How many superintendents, principals, and department chairs has he met who haven't had that data or ability or fairness?

Merrow correctly wrote that "Forward-looking union officials would push for creative uses of student performance data – such as using it to help teachers in areas where the data reveals they are not reaching their students."

But no union official can move forward without evidence that data would be used in a fair and accurate manner.

I don't know the specific personalities, and perhaps Merrow is not as convinced that the proposals coming out of NYC are as fundamentally untrustworthy. I suspect Merrow is making a political point with this article and I suspect he'll try to keep his ideas even-handed, keeping the process moving forward. After all, if he was my administrator, I'd have no qualms being evaluated by whatever data he chose.

Merrow is fundamentally on our side, and I expect he'll push the other side to show some integrity. Until then we should remember Merrow's description of NCLB and his own experiences seeing schools that have reverted to "drill, drill drill"

Given the NCLB experience, I would think that Merrow and others could be convinced that the first step is to kill dishonest accountability schemes. Then we need to move together to create real data-driven accountability.

i think alexander is making a powerful, nuanced point :-)

despite differences of intent and measurement, there is a fundamental, underlying similarity between teachers evaluating students and teachers being evaluated that should not be ignored -- if only to help teachers understand why it is so frustrating to some non-educators to hear teachers do what seems like resisting the common-sense notions of evaluation.

plus which, let's not idealize what's in teachers' gradebooks and why it's there. not all testing done by teachers is all that formative and constructive. many teachers sort and label kids with no real intent or ability to help students improve, just like they don't want to be sorted and labeled by others.


What would you cite as evidence for your claim that "many teachers sort and label kids with no real intent or ability to help students improve"?

I was just about to write what Skoolboy wrote. I find that remark stereotypical and offensive.

It's not really constructive to say that many people from one ethnic group are criminals. It's not much of an improvement to slur members of a profession.

Alexander, I'll help you with this. You can use the two "teachers" whose rooms are on either side of my classroom, a number of others who teach elsewhere in the building, or any countless others around the city.

It's funny that the defense of teachers - the begging for proof - always comes from those who spend every moment making sure that nobody can collect that proof. We can't use data until they're both perfect data and perfectly-and-fairly-used, according to the fans of the status quo, because that would be unfair. Are we to believe, then, that the 900 assessments that elementary school teachers perform each year are all 100% fair and providing 100% solid, useful data? If not, why would we subject our kids to anything we wouldn't subject ourselves to? The answer, I'm afraid, is that the people who argue against testing are either solely or mostly concerned with how whatever system we employ benefits the adults, rather than their students.

thanks, everyone --

for more discussion on this, check out the responses to my original post about teachers and testing:


The Russo post exemplifies how far out of touch our most vocal education policy commentators are -- does he really think that student progress and outcomes play no role in the current system of evaluation for teachers? Does he really think that noone had ever thought of looking at classroom outcomes before Fordham, Ed Sector ad nauseum starting blabbing 24/7 about it?


There is no question that many or most gradebooks are full (or are supposed to be full) of primitive assessments. It is equally true that students have property rights, and the imperfections in the grading systems would face dubious futures in court battles. And it is equally true that these primitive situations are common throughout education. Is any of this news?

The question is whether we want to blame others or create a better system. Why is it that administrators in high poverty neighborhood schools are completely unable to provide backing for teachers who try to raise academic and/or behavioral standards? Administrators have to assume that if a student's family tries to assert their rights, the school won't be able to defend its actions. (which explains why some in management want to push ahead with such dubious evalution systems. If they have to "prove" with a degree of confidence that a teacher is ineffective, they just have one more task on an impossible "To Do List.")

So you want address this situation by taking away the property rights of teachers? Would you gamble your career on how you will be evaluated by a bogus system? We need to recruit and retain more talent, not drive it away.

Being a 55 year old and a classroom veteran, the lack of crediblity that pervades the system does not necessarily effect me. If I need grades to motivate, except in occasional situations, I conclude that I have failed. I don't need disciplinary backing to do my job within the current situation. (But I will honestly admit that I can't get my students to the point where they will pass graduation exams unless the system changes.) The problem is young teachers and teachers who are attempting to teach their students to read for comprehension and to achieve real numeracy. Our best can get our students to progress greatly from the 5th and 6th grade skills that they bring with them to high school. Some of the best rely on grades.

But we can't get our kids close to the level they need to be(to have a healthy and hopeful future) unless we reform a full range of dysfunctional systems within our educational systems. One of which is creating a fair and reliable method of efficently removing ineffective teachers.

Anyone can find numerous proiblems within our system, so assessing how bad things are is no great feat. The question is how do we make things better. I simply can't understand how you can make things better for students, especially those who have been treated so unfairly by family, economics, and history, by just attacking teachers.

By the way, how much time have you spent in schools in spring? Have you experienced to full court press which teachers endure as administrators pressure us to ignore our academic and attendance standards? How would you respond to the nonstop guilt-trip we get to ignore behavior that were never be tolerated in any other institution?

If I want to play a constructive role in our district, I can't get all judgemental on administrators who are struggling with these dilemmas. I wish outsiders would bring a similar understanding to teachers.

It would be much too embarrassing to not correct one thing I said. Being a high school social studies teacher, I try to help teaching literacy and numeracy, in addition to appropriate behavior and my subjects' Standards. Real world, though, I teach literacy and numeracy compensation skills.

And real world, I'd welcome any support in making the system better so I don't have to always use the preface "real world."


I agree with just about everything you wrote about the classroom. I, too, consider it a failure if I have to hold grades over the head of my students to get them to perform, and I too teach literacy and numeracy even in the content subjects, though at a much lower grade level than you. As for whether I've been in a school in the spring time: yes, I have, for more than 10 springs, as a matter of fact.

I disagree, though, that we can't be judgmental toward our administrators. I hate it when our administration insists that we promote kids who have skipped school, started fights, failed classes, etc., as long as they've passed the tests. I consider them either to be committing a crime, or being complicit in its commission. I do sit in judgment of them - it's not OK to hurt kids in this way.

At the same time, though, if we want to hold them accountable, we have to give them the tools of accountability by which they can manage their teachers.


I sure don't want to monopolize the discussion, but I have to apologize about the phrasing I used. I should have remembered that you are a teacher. Can I blame my mother for making me rush on Mother's day?

How about this? We need to make judgements about the way central offices have distorted policies, but we shouldn't get judgemental?

And since outsiders love to big-foot it into schools from the top down, lets pass this mandate. Before issuing the final draft of any accountability proposal, its authors have to reread Catch 22.

I agree that we teachers need to be held accountable for our work and that administrators need better tools for determining teacher accountability. My colleagues over at Teacher Leader Network and I have been advocating for some time now that an effective teacher accountability system requires, in part, more rigorous and comprehensive assessment of student learning than what most schools or district now use (walk-by, oops, walk-through evaluations or standardized test scores alone).

Does anyone know the status of the Performance Standards Consortium in NY? They were working with a model of comprehensive, multiple measures of student performance that got them exempted from the state's Regen exams. I'm in another state, but could it be a model of the types of data about student learning upon which accurate and fair teacher evaluations could be constructed?

Comments are now closed for this post.


Recent Comments

  • Renee Moore: I agree that we teachers need to be held accountable read more
  • john thompson: Socrates, I sure don't want to monopolize the discussion, but read more
  • Socrates: John, I agree with just about everything you wrote about read more
  • john thompson: It would be much too embarrassing to not correct one read more
  • john thompson: Socrates, There is no question that many or most gradebooks read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School