Value-Added is Not a Magical Black Box
We're now in day 5 of the Chicago teachers' strike. I found these quotes from the CTU attorney, Robert Bloch, interesting:
CTU Attorney Robert Bloch said Thursday the CTU finally felt on Wednesday that it was getting through to CPS negotiators on their concerns about the new teacher evaluation system.
"The system they are using to evaluate people is based on an extremely complicated, esoteric formula to measure student growth -- so complicated I think everybody on the CPS team will admit they don't understand it," Bloch said. "Experts developed it but not educators."
Gosh, and golly. Educators didn't develop indoor plumbing, air conditioning, and the internet, too--does that mean schools shouldn't use them either? More productively, I think an analogy to school buildings is useful here: When you're designing a new school building, you want to seriously take into account teachers' perspective on how the space should be laid out and used, what features need to be in classrooms, and so forth. But when it comes to, say, ensuring the correct structural support for the roof, you don't want teachers making those decisions. You want an architect or an engineer. You want an expert. Similarly, in designing the overall shape of teacher evaluation systems, teacher input (from all perspectives, not just the union) should play a critical role. But when you're designing the infrastructure to support that system, particularly the value-added measures, you want an expert. Unless, of course, you want the roof to fall in.
In addition, the complicated algorithms used to determine student growth -- called "value-added" -- are being debated nationwide.
"The problem is, how do you hold teachers accountable for improvement when so many things that are used to evaluate them are outside their control or very complicated?'' Bloch said.
"The science behind the student growth aspects of testing is untested and uncertain, and you're going to risk a teacher's career based on some guy in a back room writing algorithms or students who are not tested in the subject you're teaching?
"There's a lot of unknowns. People's careers should not be decided by factors people don't really understand."
Let's unpack this a bit: Bloch's arguing that value-added measures of student learning shouldn't be used for teacher accountability because they're too mathematically complicated, and specifically because teachers shouldn't be held accountable using something they can't understand. Well, let's stipulate that the math beyond the measures is complicated. It would certainly be possible to hold teachers accountable for student learning using very simple and transparent measures--like the percentage of a teacher's students achieving proficiency. But I'm guessing most teachers wouldn't like that, and would rightly argue that such measures are unfair because they fail to take into account the variety of other factors--demographics, prior performance, etc.--that affect student learning. The complexity of value-added models is exactly the result of trying to take these factors into account. As with any complex issue, you have to make trade offs, in this case, between transparency or simplicity and fairness or accuracy. Unless of course, one doesn't think teachers should be held responsible for student learning at all......
That said, I do think reformers who advocate new teacher evaluation systems ought to take more seriously the fact that the new value-added and growth measures are difficult to understand and often not entirely transparent. Too often proponents of new teacher evaluations tend to talk as if value-added or growth measures are a sort of magical black box that, if you put the right data in, will spit out the truth about teachers' effectiveness in the classroom. But it's more complicated than that. As Mr. Weasley said, "Never trust anything that can think for itself if you can't see where it keeps its brain." We shouldn't be asking teachers to place their trust in a magical black box without being transparent about how this all works and the real limitations of the measures.