Teacher observations under District of Columbia’s IMPACT teacher-evaluation system have been performed with relative consistency, but they only correlate modestly to value-added measures, according to a report released this morning by the Aspen Institute, a Washington-based policy group.
In addition, with about 82 percent of the teaching force in the middle two categories of effectiveness under the system, the district should redouble efforts to use the system and the information it generates to help teachers improve, the report says.
(The group is releasing two other papers on teacher evaluation today, so make sure to check out all of them.)
If you’re coming to IMPACT for the first time, read these two pieces for some background information on how the system works, because we’re digging right into this report today. The basics: Each teacher gets five observations, three by an administrator and two by a content-expert “master teacher"; those ratings are coupled with a student-growth component.
For one, few systems have been subject to this level of scrutiny. In D.C., it is partly because of former schools Chancellor Michelle Rhee’s goals for linking IMPACT both to dismissals and to a bonus-pay system. High stakes indeed.
The system has received praise from some teachers who view it as more relevant (if tougher) than the prior system, but it’s also been attacked by critics for “deprofessionalizing” or taking professional judgment out of teaching and observations. IMPACT is also one of the first operational systems to include value-added data as one criterion in the evaluation, certainly a very contentious topic in our field these days.
This report is a veritable treasure trove of data about IMPACT. For instance, D.C. principals tended to rate teachers high on engagement with students, and low on asking open-ended questions to gauge students’ critical thinking. And their ratings were, in general, very close to those given by the “master educators” with little variation, suggesting that training and calibration is pretty good overall. In fact, the report notes, most of the scoring is pretty high, with scores at about 3 on the system’s 1-to-4 scale for each standard. The report also says that fewer than 1 percent of teachers had widely divergent scores, despite allegations of subjectivity.
Second, the report finds that the classroom-based observations and the individual value-added measures are correlated, but not all that strongly. Ideally you’d want improvements in teaching practices, as measured by the observations, to correlate to improvements in student outcomes, so that achievement goes up as teachers are more skilled. The report found a correlation of 0.34 between the two.
On the other hand, could these findings perhaps mean that the two measures—observations and value-added—are picking up different performance information? See the latter half of this blog item for some discussion of this tricky issue of correlations.
These are important data: Policymakers and practitioners alike have said that evaluation systems have to be carefully calibrated if they’re to be used for high-stakes purposes.
The report also profiles how D.C. officials put together IMPACT, as well as its somewhat bumpy rollout. Among other things, teachers and even some principals protested the fact that there was no pilot for the system and that improvements were being made even as the system was used for accountability. And as I’ve noted before, many teachers didn’t trust the “master educators,” who were selected only by the central office.
Another point worth noting: D.C. officials have already made some changes to the observation protocol, such as reducing the number of standards and loosening some of the language to be less specific (i.e., observers no longer have to check to make sure “four out of four” students know the objective of the lesson being taught).
Some additional interesting findings: In interviews, master teachers said they found that the system has helped to standardize performance expectations across the district, giving teachers and principals a “common language” for discussing the craft of teaching.
The interviews found areas of concern, too. Some principals were concerned about the rigor of the teaching standards. They worried that the system didn’t focus enough on content-specific strategies or about specific teaching skills. (EdWeek‘s recent professional-development special report dug a bit into these issues, so make sure to check that out.) In interviews with teachers, some liked IMPACT’s targets, some felt they needed more examples of performance at different skill levels, and some felt it was biased, lockstep, or unfair.
The report is particularly interesting coming in the wake of a Washington Post story last week that got a lot of deserved attention. In that story, the reporter profiled an interaction between one classroom teacher and one master teacher.
The story concludes with the master teacher offering some help with planning that the classroom teacher seems to eschew.
It’s an interesting exchange, given that, as this report notes, the Washington Teachers’ Union was careful to limit interaction between the master teachers, who were seen as administrators, and the classroom-based teaching coaches provided to schools for professional development.
Given the lack of trust between the union and the district, that separation is perhaps not surprising. But it seems to also have had the consequences of making the professional development provided by coaches less relevant to IMPACT’s goals and targets.
Another finding from the report: Teachers felt it was harder to get high scores on IMPACT in troubled schools. The report has no data on this topic, but elsewhere it’s been reported that a higher proportion of “highly effective” teachers are located in the city’s wealthier—and more heavily white—northwest quadrant than in other areas.
This was also an important subtext of the WaPo story: The classroom teacher profiled had a class with kids who were several grade levels behind, so his lessons plans didn’t always stay on schedule.
The president of the local teachers’ union, Nathan Saunders, has said repeatedly that the system needs to do a better job of accounting for poverty and school context.
D.C. Mayor Vincent Gray has also suggested that changes to the IMPACT system might be in order. His transition team is reportedly taking a close look at IMPACT, too. Odds are that we have not heard the last of the changes to this system.