Don't Think about Elephants
"Don’t think about elephants," skoolboy’s father used to joke, long before George Lakoff’s manifesto with a similar name. The joke, of course, is that by trying not to think about elephants, all that you can think about is elephants. The harder I tried not to think about elephants, the more I thought about them.
The New York City Department of Education has its own variation. This month, the DOE is sending Teacher Data Reports, which purport to estimate the effect of individual teachers in grades 4-8 on students’ test scores, to school principals, who will then distribute the reports to their teachers after the principals have been trained. "The Teacher Data Reports are not to be used for evaluation purposes," wrote Chancellor Joel Klein and UFT President Randi Weingarten in an October letter to teachers. "That is, they won’t be used in tenure determinations or the annual rating process. Administrators will be specifically directed accordingly." Similarly, the Frequently Asked Questions section of the DOE’s Teacher Data Tool Kit website poses the question "How can you be sure that principals won’t use the Teacher Data Reports to evaluate teachers?" The response: "Principals have been and will continue to be explicitly instructed not to use Teacher Data Reports to evaluate their teachers. The DOE has standard processes in schools for teachers to raise issues or concerns."
And yet. From the Frequently Asked Questions on the DOE’s Teacher Data Toolkit website: "By isolating individual teachers’ contributions to student progress, the Teacher Data Reports provide valuable information to school leaders and teachers about where to focus instructional improvement efforts. …Teacher Data Reports provide information about how individual teachers’ efforts influence student learning … A sophisticated multivariate regression analysis based on NYC data from 1999-2008 determined how much to weigh each factor [to calculate students’ predicted gains] … A panel of technical experts has approved the DOE’s value-added methodology. The DOE’s model has met recognized standards for demonstrating validity and reliability. Teachers’ value-added scores from the model are positively correlated with both School Progress Report scores and principals’ perceptions of teachers’ effectiveness, as measured by a research study conducted during the pilot of this initiative."
In other words: The Teacher Data Reports rely on sophisticated statistical techniques that are valid, reliable and approved by experts, and they isolate an individual teacher’s contributions to student learning. But, you principals who are under tremendous pressure to increase test scores or face losing your jobs, don’t you dare think about using these Teacher Data Reports to evaluate teachers.
Don’t think about elephants.
Comments
At best this is a tremendous waste of resources. Technical experts, statiticians, peer review...$$$$$$!! All to see which subjects/objectives an individual teacher's students scored poorly on-- which can be seen with regular test results?! But more likely this is an insidious attempt to carry out the Klein/Rhee/edu-biz agenda to use teachers as scapegoats in their "reform".
Posted by: jenbie44 | December 15, 2008 10:19 AM
If you want real teaching quality, then this is the way to go. Intimidate them to no end, and they'll be scared straight. People wonder why the little tenets of NYC teaching like tenure and pension are so sacred to us. If not for things like that, this "tracking" sheet would spell bad news for teachers, especially those who have more challenging populations. "Elephants" is right: there's a huge elephant in the room and we're expecting principals to ignore them and teachers to move them out of the room.
Posted by: Jose | December 15, 2008 10:39 AM
Unfortunately, this protocol reminds me of a line Fake Steve Jobs once used to describe management at Apple: "Hold people to an impossibly high standard, but here’s the twist — don’t tell them what that standard is. And fire them if they fall short. You know what that does to people? It makes them crazy... Crazy people are more creative. And more productive."
Posted by: Ms. Miller | December 15, 2008 10:56 AM
I definitely agree with your argument. Given that the reports "rely on sophisticated statistical techniques that are valid, reliable and approved by experts," then they SHOULD be used to evaluate teacher performance and it's counterproductive to attempt to prevent principals from using this invaluable source of information. Right on.
Posted by: SB | December 15, 2008 11:14 AM
Hey SB,
I should clarify that it's the NYC DOE's assertion that the Teacher Data Reports "rely on sophisticated statistical techniques that are valid, reliable and approved by experts," not skoolboy's. What I'm highlighting is the incongruity of the DOE making such an assertion while prohibiting the use of the reports for evaluating teachers. Currently, New York State law prohibits the use of student performance data in making teacher tenure decisions.
I think prohibiting the use of student performance data in making tenure decisions is ridiculous; but I am naturally skeptical about claims that a complex statistical model can isolate a teacher's contribution to student learning, and fear that the veneer of science -- "sophisticated statistical techniques that are valid, reliable and approved by experts" -- could elevate the influence of Teacher Data Reports and other value-added assessment systems beyond what the data can actually tell us.
Posted by: skoolboy | December 15, 2008 11:35 AM
Skoolboy
I came here to say what you said in your last comment. We could add that it's very tempting to compare teachers with a reductionist metric that leaves you with a number when other measures seem more amorphous. That's one reason that college admissions officers are always in danger of leaning too heavily on GPA and SAT scores.
Posted by: Dan Willingham | December 15, 2008 2:35 PM
So, here's the question:
It seems like I've every expert in the field I've read says that evaluations like this one are deeply flawed. And yet the NYC DOE is claiming that a panel of experts have signed off on them. So, who are these experts that have done so?
Is there less consensus in the research community than I think, or are the people that Klein cites not actually experts? Or is there some other explanation?
Posted by: Corey | December 15, 2008 3:41 PM
According to the NYC website link you provided, this test data covers certain teachers in grades 5 - 8 on two subjects. What about all the other teachers of all the other subjects and grade levels?
Like Corey, I'd like to know specifically who are the "experts" that have signed off on these results and what methodology did they use to verify these findings?
Posted by: Renee Moore | December 15, 2008 7:47 PM