Oklahoma Department Defends Integrity of A-F School Accountability in New Study
In a study conducted by one of its researchers, the Oklahoma education department defended its school accountability system that uses A-F grades, after a report released earlier this year by the state's two major public universities sharply criticized the system on a variety of fronts.
Back in October, researchers at the University of Oklahoma and Oklahoma State University reported that, according to an analysis of state test scores from 15,000 students at 63 urban schools in the state, as few as three correct answers could ultimately separate A schools from F schools. That means that one group of students at an A school answered roughly three (or 3.67, to be more precise) more questions correctly on average than another group of students at an F school. The report also stated that scores from some D and F schools managed to top the scores from A and B schools, and that low-income students actually had higher achievement levels at the lower-ranked schools.
The report added to the controversy in Oklahoma recently over A-F grades for schools, so it may not be entirely surprising that the state department decided to conduct its own study about the issues the university researchers tackled. The department's report, released late last month, was written by Megan Clifford, who is a Strategic Data Project fellow at the Oklahoma department. (CLARIFICATION: An article in the Tulsa World originally said Clifford was "on loan" from Harvard University to the department, but as a fellow with the Strategic Data Project, which is housed at Harvard's Center for Education Policy Research, Clifford is actually an employee of the Oklahoma department.) Clifford's own report states that the three main conclusions from the October study on A-F grades are wrong or misleading at best.
First of all, Clifford says that her report relies on data from the entire state instead of the much smaller sample size the university researchers use, making her subsequent conclusions "more reliable." But regarding perhaps the most controversial point, how many correct test questions separate top and bottom schools, Clifford says the report from the two universities arrives at its conclusion only "after controlling for race, poverty, and prior achievement." Without controlling for those factors, she said, the average gap was roughly 14 correct questions between A and F schools.
Clifford says the Oklahoma-Oklahoma State researchers' decision to control for nonschool factors should be made clear, even though, she notes, "it may be appropriate to control for such factors." In an interview, however, she said that for the purposes of the school report cards, the grades shouldn't be controlled to reflect those factors.
"We didn't want to do that because we wanted to have the same standards for all students," Clifford told me.
But Clifford also says that even if you control for factors like race and income, it's more important to think about a different gap between A and F schools: months of learning. According to Clifford's research, the test scores show that, on average, 15 months of learning separate the A schools from the F schools in the state, a conclusion that Clifford told me shows there's a much bigger educational gulf between the state's best and worst schools than what the Oklahoma-Oklahoma State researchers led the public to believe.
What about the performance of low-income students by school ranking? Contrary to the university researchers' findings, Clifford says, the higher a school's grade, the more likely students receiving federally subsidized free and reduced-price meals were to achieve higher scores on the Oklahoma Core Curriculum Test, while the same held true for minority students (I've replicated Clifford's chart showing this below).
In the F schools, by the way, the average raw score of students receiving federally subsidized meals was slightly higher than scores from students not receiving those meals. I asked Clifford how she interpreted that finding, and she said she wasn't sure what it meant.
And regarding the university researchers' argument that many schools with low overall grades had higher scores on some tests than their counterparts with higher grades, Clifford in her report uses the analogy of a student who receives both an overall GPA and individual subject grades: "While, on average, A or B schools outperform C or D schools, that doesn't mean that they outperform them in every subject."
It's not clear if Clifford's report will do much to quiet down the strong feelings that have developed about A-F accountability in the state. But the fact that the Oklahoma department felt the need to issue a report in response to another body of research seems to show that the issue is considered important by people on both, or all, sides of the question of school accountability in the state.