As few as three correct responses on Oklahoma state tests can separate those schools receiving an A grade from those receiving an F under the state's accountability system, according to a paper released by the University of Oklahoma and Oklahoma State University this month that says the system has many flaws, among them that it hides the poor performance of racial minorities and low-income students.
"Oklahoma School Grades: Hiding 'Poor' Achievement" examines raw scores on the state's reading and math tests from 15,000 students in 63 urban schools in the state. The study's authors at the Oklahoma Center for Education Policy (University of Oklahoma) and the Center for Educational Research and Evaluation (Oklahoma State University) say they chose these schools because it is crucial in those districts to differentiate high-performing from low-performing schools, and because of their relatively high concentration of low-income and minority students. In addition, the study examined science test scores from about 4,900 students.
What did the study find? According to its authors, they uncovered three glaring problems with the state's A-F school grading system. First, and perhaps most controversially, the joint study found that on average, only about five (4.86 to be exact) correct responses on the state reading test separated A school from D schools, the biggest gap the researchers found between top-ranked schools and their lower-ranked counterparts. The gap between A schools and F schools on the reading test was 3.67 correct responses. (That means one group of students, on average, answered more questions correctly than another group of students in a different school.) When averaging scores on the math and science as well as reading tests, about three to six correct responses separated the best schools from the worst according to the A-F system.
"These are small effects on which to base significant decisions," the authors say. "Many of the achievement differences between letter grades were likely due to chance; even when they reached statistical significance they were of questionable practical utility, generating little confidence in grade distinctions."
Second, the study says that on the math tests, some scores from D and F schools topped those from B and C schools, and that none of the seven highest-scoring schools on the math exam were in fact A schools.
And third, the study says that when examining students receiving federally subsidized meals, they actually scored higher in D and F schools on average than they did in A and B schools, meaning that these supposedly top schools are "masking the especially low performance of poor and minority children."
This year, state lawmakers made changes to the A-F system. These included no longer weighting "proficient" and "advanced" scores differently, and no longer making whole-school performance part of a school's main grade calculation. What do the authors say about these changes? In short, they say the changes don't solve the problem of relying on proficiency on tests, and failing to account for student mobility across districts.
"The fix is quite simple," the paper concludes. "A school's performance should be reported on multiple dimensions—a profile that includes scale scores for subject areas as well as other relevant school conditions (e.g. program coherence, social climate, and faculty and administrative stability)."
I contacted the Oklahoma Department of Education for a response to the study, and received a statement from Superintendent of Public Instruction Janet Barresi: "This analysis appears built on the wrong and dangerous presumption that minority and poverty- stricken children cannot learn. I reject that notion. Equally disturbing is that the researchers' conclusion belies the importance and effectiveness of teachers." (I might be getting an additional response from the Oklahoma department, and I'll update this post if I do.) UPDATE: Oklahoma Secretary of Education and Workforce Robert Sommers also had a response to the report which focuses on what he calls the report's incorrect diminuation of teachers when it comes to student success: "Professional educators working on the front line should be afforded the opportunity to adjust to their approaches, yet be held accountable for the results. Those who succeed well should be recognized; those who don't should be helped in their search for better options."
This year hasn't been full of sweetness and light for A-F accountability systems—former Florida K-12 boss Tony Bennett resigned in August because of revelations from the Associated Press about alterations made to Indiana's A-F system in 2012 when he was state superintendent there. Indiana is currently considering how to change that system in the wake of the story. Questions will no doubt continue to surround these systems as they have gained increasing popularity among states. How they change over the coming years is a crucial question for state education policy.