Race to the Top District Contest: Analyzing the Scores
Judging by the scores of the district-level Race to the Top contest, the race was very, very close.
A mere one point on a 210-point scale separated the 16th winning application from the two that tied for 17th place, which were not funded. What's more, a second round of more-intensive judging cost some applicants valuable points, while others went from a losing score to a winning one.
So this means that even the tiniest judgment calls by the outside peer reviewers during the scoring process mattered a lot.
So it's fitting to review just how the scoring process worked.
Before the finalists were announced in late November, a team of three peer reviewers read and scored each application. Then, the teams talked over the applications on the phone, focusing on large scoring variances. They had a chance to revise their scores, after which the 61 top-scoring applications were announced by the U.S. Department of Education as finalists.
Then, the judging entered phase two. The teams that reviewed the 61 finalists came to Washington, D.C., for more intensive training by department staff members, who helped highlight the areas where the judges disagreed most. (Collectively, for example, the biggest variances were seen in the judging of performance measures that the applicants had proposed, and the sustainability of their proposals.)
Panel monitors made up of department career staff members helped facilitate three- to five-hour, in-person conversations. Separate "competition support teams" from the department (also career staffers) provided the judges with more in-depth expertise in areas such as policy and budgeting. These support teams also reviewed the scores and judges' comments to make sure they all made sense.
Finally, the scores were ranked and publicized, and the winners announced Dec. 11. There was enough money to fund 16 of them.
The department, which initially had not posted the scores from the first phase of judging, did so this week (coincidentally, or not, after I had asked).
And those initial scores reveal who fared the worst, and the best, after the more-intensive second round of judging. The Springdale School District in Arkansas tumbled from second place initially to 25th, and out of the winners' circle. The Kentucky Valley Educational Cooperative fell from sixth to 21st, and the Green Dot charters in California went from ninth to 22nd.
Faring the best between the two rounds was the New Haven Unified district in California, which catapulted from 30th to second place to win $30 million. And the Green River Regional Educational Cooperative in Kentucky went from 24th to 15th, edging onto the winners' list and taking home $40 million.
Given the sometimes big shifts in scores between the first and second rounds of judging, some have wondered whether an applicant that was not a finalist could have ended up winning had that application been subjected to the more intensive judging in the second round.
But department staff members are confident that they got the best applicants in the finalist round.
"What we did was purposely pick many more finalists to account for that type of fluctuation," said Ann Whalen, the department's director of implementation and support.
A couple of districts also may have suffered the "Lousiana" fate, when a judge—rightly or wrongly—deviates from the other two judges with a particularly low, outlying score. (Louisiana loudly complained after it lost the Race to the Top state competition in 2010 that a single judge with an unusually low score tanked its application.)
In the original Race to the Top, there were five judges. In this year's district-level contest, there were three—further elevating the power of a single peer reviewer.
For the Mapleton district in Colorado, which finished in 17th place, 26 points separated its highest score and lowest score (and remember, it lost out by a single point). For Manatee County in Florida, which finished in 22nd place, the point swing was 35 points.
The biggest beneficiary of a point swing was likely the Green River cooperative, which won $40 million and placed 15th. The highest-scoring judge awarded its application 207 points, the lowest-scoring judge gave it 188 points, a 19-point spread.
It's worth noting, however, that among the five highest-scoring applications, the judges awarded consistently high scores.