« Still a Bobo in Paradise | Main | Welcome Richard Whitmire's New Blog on Gender and Education »

Everyone's Favorite Sound Bite About Highly Effective Teachers Put to the Test

"By our estimates from Texas schools, having an above average teacher for five years running can completely close the average gap between low-income students and others."
-Steve Rivkin, Rick Hanushek, and John Kain (2005)

"Having a top-quartile teacher rather than a bottom-quartile teacher four years in a row would be enough to close the black-white test score gap."
-Robert Gordon, Tom Kane, and Doug Staiger (2006)

"There are big differences in the amounts and kinds of learning that different teachers help produce....these effects are cumulative."
- Kati Haycock, Education Trust

It's everyone's favorite sound bite: good teachers alone can close racial and socioeconomic achievement gaps. But if the entire teacher effect doesn't persist from year-to-year - that is, a student only retains some fraction of the learning advantage they get from having a highly effective teacher - these claims simply don't hold up.

In a new paper, "The Persistence of Teacher-Induced Learning Gains", Brian Jacob, Lars Lefgren, and David Sims estimate how much of the teacher effect fades out over time. It turns out that kids lose more of these short-term test score gains that we (or at least I) thought:

"Our estimates suggest that only about one-fifth of the test score gain from a high value-added teacher remains after a single year. Given our standard errors, we can rule out one-year persistence rates above one-third. After two years, about one-eighth of the original gain persists."

Yes, you read that correctly. Even if you rely on the upper bound estimates of teacher effect persistance from this study, only a third of that gain sticks around. If you take their point estimate, only 20% of this gain persists. If gains fade out at this rate, we may be overstating the ability of highly effective teachers to contribute to students' long-term academic skills, says Jacob:

"Our results indicate that contemporary teacher value-added measures may overstate the ability of teachers, even exceptional ones, to influence the ultimate level of student knowledge since they conflate variation in short-term and long-term knowledge. Given that a school’s objective is to increase the latter, the importance of teacher value-added measures as currently estimated may be substantially less than the teacher value-added literature indicates."

Jacob and colleagues conclude that we should revisit the "5 great teachers can erase gaps" claim that is so common in education policy discourse:

"Previous researchers have referenced a counterfactual world in which a series of high value-added effects for a hypothetical student with a string of good teachers may be simply added together. Given this scenario, researchers and policymakers have advocated the widespread use of such value-added measures in a variety of education policies including teacher compensation and teacher/school accountability. Our results suggest some caution should be taken in focusing on such measures of teacher effectiveness. If value-added test score gains do not persist over time, adding up consecutive gains does not correctly account for the benefits of higher value-added teachers. Of course, the same caution should be attached to any educational intervention. Hence, the broader implication from this work is that researchers and policymakers should make greater effort to track the long-run impact of education policies and programs."

If you can't access the paper, I've linked to Brian Jacob's contact info above, or shoot me an email (eduwonkette (at) gmail (dot) com).

Update: * To clarify, this paper does not find that teachers "don't matter." If every teacher moved students forward 2 grade levels - an effect twice as large as the gains we expect teachers to produce - we would find no "teacher effects" on test scores, i.e. having one teacher versus another wouldn't matter because all teachers would be equally effective in increasing test scores. But teachers would still make enormous contributions to students' learning in this scenario - they still "matter." Jacob and colleagues' point is simply that the difference between having a below and above-average teacher may be inflated in the current literature because we've been focusing on short-term versus long-term gains.

* Chad Aldeman at The Quick and the Ed misunderstands the implications of the paper: "These findings in no way challenge previous studies indicating teacher effects accumulate over time." This study does find that teacher effects accumulate - a student does, after all, hold on to 20% of the teacher-induced learning gains - but they do not accumulate in the additive way that those quoted at the top of this post have suggested. The above quotes assume that students carry forward the full gain, and that as a result, we can close the achievement gap by giving students five 84th percentile teachers in a row. If teacher-induced gains decay at the rate documented in this paper, this sound bite does not hold up.

In mathematics many of us believe the inverse, that is, one bad teacher before high school can leave a student in poor shape for a long time, in a way that good or reasonable teachers cannot easily undo.

Bad here, though, doesn't mean some percentile rank, it means uncomfortable with content.

I know what will really advance the profession--getting the argument out there that good teachers don't matter!

I'll try my hand at reading the math, but now you mention it, your analysis makes sense. I'm constantly having students come back and thanking me, but I've never had an alumni come back and thank me for raising test scores. We discuss what they learned in our class, what they remember, and how it was relevant. Not surprisingly, my former students retain a way of looking at the world, of looking at school work, an interest and motivation etc. Its always fun to hear about the specifics they retained and why, but I don't doubt that standardizing testing improvements would fade.

I also know that some of the things we learned in class are discussed afterwards on the street corner as homeboys are hanging out. I wish they had left the gang, but of course I'm incredibly proud of that compliment.

But getting back to the teacher quality argument, under what scenario would you find inner city neighborhood schools where students would get four or five warm bodies in a row? Has anyone ever heard of a high poverty neighborhood school where you could get four competent teachers in a row? In my experience, year in and year out, regular students find that about 1/2 of their classes are functional. In many cases, those classes have a good teacher, but given the circumstances, they aren't close to effective.

But I've got news for the left and the right wing true believers. If they could coerce districts into coercing effective teachers to transfer from effective schools, we would still end up the same. Those effective teachers would bail out immediately, unless we did something about chronically chaotic schools. Its the dispect shown to teachers, and the inability of the system to let teachers teach, that has driven so many great young teachers out of high poverty school.

Its such a shame, because if you can do it and maintain an emotional equilibrium, teaching in high poverty schools is the greatest job in the world. Show some respect for teachers and we could provide four or five effective teachers in a row. Until we can address systemic realities, though, the best we can do it just get rid of the most ineffective teachers even if we can't find more that a Long Term Sub to fill in.

John #1 - If the post was unclear, I apologize - but the finding of this paper is not that good teachers don't matter. Even if students don't take forward the entire gain provided by a highly effective teacher, they are still better off having an above rather than a below average teacher. It's simply that the size of these effects on test scores may be smaller than we previously thought - and getting a better read on these effect sizes is important when we're making claims about closing gaps. And as John T. points out above, teachers "matter" in lots of other ways as well.

John T. - You make an important point - that the contexts in which teachers teach enable and constrain good teachers. Plugging in exceptional individuals will never be enough - something that a lot of the teacher quality debate misses.

Jonathan - I like the way you flip this question around.

I've always been surprised that people are so quick to label teachers as "good" or "bad" without considering their teaching environment. As John T. pointed out, a good teacher in a bad teaching situation may not create a stellar outcome. Also, teachers who excel in one type of school are not necessarily going to excel in all types of schools.

For instance, I found that in inner city schools, a highly academic teacher, who would be perfectly suited for teaching AP Physics in the 'burbs, may not have as much success with the kids as a teacher with no graduate degree but with great classroom management skills. A PhD isn't going to do much good if the kids won't sit still for the lesson!

As eduwonkette noted, serious socio-economic problems bedevil students in low income schools -- problems that even the "best" teachers aren't going to be able to fix by themselves.

It's tough to say precisely what a good teacher is, particularly when your sole measure is test scores. A social studies teacher of my acquaintance tells me in his previous school, about 30% of his students passed the Regents exam, and that he had the highest passing rate in the building. In our school, his passing rate is closer to 90%.

He swears his methodology hasn't changed at all.

NYC: Good point. Your info supports my theory that it's difficult to label an educator as "good" or "bad" independent of his or her school setting.

Comments are now closed for this post.


Recent Comments

  • Attorney DC: NYC: Good point. Your info supports my theory that it's read more
  • NYC Educator: It's tough to say precisely what a good teacher is, read more
  • Attorney DC: I've always been surprised that people are so quick to read more
  • eduwonkette: John #1 - If the post was unclear, I apologize read more
  • john thompson: I'll try my hand at reading the math, but now read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School