Here’s my take on the New York tenure law discussion going on around the blogs:
1) The backdoor process was unsavory, and now threatens to displace an important discussion about the limits of value-added measures in New York. Sherman Dorn offers some fertile thoughts on the process issue. Also worth noting that last week’s outragists were hardly outraged about the secrecy surrounding NYC’s teacher experiment.
2) Critics would do well to separate the likely effects of this law from their unhappiness with the process. Consider Robert Gordon’s post, which interprets the law’s effects as follows:
This means that in deciding whether to give a teacher a presumptive right to teach for 30+ years, a principal may not consider evidence of whether the teacher is helping students learn. The principal can consider whether the teacher maintains neat bulletin boards, whether the teacher attends meetings on how to pay for pencils, and whether the teacher is sufficiently deferential in the hallway. But the principal may not consider, based on achievement data, whether children are learning.
Do classroom observations provide no “evidence of whether the teacher is helping students learn?” Value-added measures, after all, are simply a proxy for student learning, and observations also provide proxy data on student learning. Gordon assumes that principals cannot identify teachers with especially low value-added in the absence of test score data. But if value-added measures mean anything, very low performers should be getting poor subjective evaluations too. It turns out that principals are actually pretty good at identifying teachers with low value-added based on subjective evaluations (see this post). If a teacher is a consistent low performer, the three admissable forms of evidence in tenure decisions - 1) observations, 2) peer review, and 3) an evaluation of how teachers use data to inform instruction - already provide lots of information about how teachers affect student learning.
3) To my knowledge, no one has provided a viable technical solution to the middle of the year testing issue. Given existing problems with value-added and the added complication of midyear testing dates, it would be wildly irresponsible to put these measures into place in NY without further study.
If you want new reasons (not related to testing dates) to sweat about the fallibility of value-added, check out this paper, which was presented last weekend at AEFA by Tim Sass (in collaboration with RAND’s J.R. Lockwood and Dan McCaffrey). They looked at the year-to-year stability of value-added estimates in Florida, and found that it’s often the case that teachers who are in the bottom 20% of value-added estimates in one year are not in the bottom 20% the next year. In Broward County, only 41.4% of teachers who were in the bottom 20% in one year were in the bottom 20% the next year, too. In Orange County, only 31.7% of the teachers who were in the bottom 20% in one year were also there the next year!
Update: Robert Gordon cherrypicks a finding from the Jacob and Lefgren paper to make his point. Perhaps if he’d read beyond the abstract and looked at the magnitude of the value-added advantage over principal ratings in predicting future student achievement (a whopping .036 SD in reading and .074 SD in math), he would realize that all is not lost. And again, this minuscule value-added advantage is coming from the middle of the distribution, not the top and bottom - and the bottom is the relevant issue in tenure decisions. From the same paper:
While value-added measures of teacher effectiveness generally do a better job at predicting future student achievement than principal ratings, the two measures do about equally well in identifying the best and worst teachers. With regard to parent satisfaction, we find that a principal’s overall rating of a teacher is a substantially better predictor of future parent requests for that teacher than either the teacher’s experience, education and current compensation or the teacher’s value-added achievement measure.
Moreover, what kind of predictive advantage can we expect inaccurate/noisy value-added estimates to have over principals’ evaluations?