A new report estimates that the two federally funded assessment consortia won’t have to increase the prices of their tests by more than a few dollars per student even if some states drop out.
This finding, in a report by the Brookings Institution, attempts to quantify one potential kind of damage that opponents to common assessments could inflict on PARCC and the Smarter Balanced Assessment Consortium: driving up the cost of the tests by getting states to withdraw. The report will be a focal point for a discussion at Brookings Wednesday about assessment cost and quality.
As we’ve reported, four states that were consortium members have dropped out in recent months: Utah, Alabama, Georgia and Oklahoma. Others have announced they’re not using the tests, or are still weighing their options, even as they maintain consortium membership. That raises the question: If many states drop out, will the remaining states have to pay more to cover fixed, shared costs such as item development?
Brookings scholar Matthew M. Chingos finds little reason to panic the troops. He writes that the departure of a few states in each consortium will have only “a minimal impact” on the cost of the tests.
“For either PARCC or SBAC to face any real cost-based threat from states dropping out, the political opponents of the Common Core would have to be successful in all of the states where they have been most active and in several additional states,” the report says.
If the 25-state Smarter Balanced group loses six states where there’s been particularly heated debates about the common core—Kansas, Missouri, Michigan, Pennsylvania, South Carolina and Wisconsin—its projected cost of $22.50 for summative tests would rise by about $2.50 per student, according to Chingos’ calculations. The study finds that SBAC could lose half of its members and still keep its test cost under $30 per student.
PARCC currently has 19 members—18 states and the District of Columbia. If one of its big kahunas, Florida, drops out, PARCC’s current projected price of $29.50 for summative tests would rise only 63 cents, Chingos reports. If the consortium dwindles to 15 members, the cost would rise only to $32.08, he calculates.
States would indeed have to share a larger portion of fixed costs, such as for item development, as the number of states in a consortium dwindles. But Chingos finds that a consortium’s membership would have to shrink a good deal for the price to rise significantly.
Artificial Intelligence
The Brookings study also explores another important driver of test cost: how many of its constructed-response and performance items will be scored by hand, and how many by artificial intelligence. PARCC’s price estimate assumes no cost-savings from artificial intelligence, since there are still questions about its accuracy. If AI can be used, some cost savings can be realized down the line.
Until recently, Smarter Balanced planned to have states use artificial intelligence to score half of its constructed-response items and performance tasks. Chingos calculated that if Smarter Balanced used AI scoring for only 25 percent of those items, scoring cost would rise by $1.45 per student. If it abandoned AI, the cost would rise another $1.45 per student.
Chingos learned as he wrote the report, however, that Smarter Balanced has scaled back the role artificial intelligence will play in scoring its tests, at least in the first operational year for the assessments (2015). Less AI scoring of those constructed-response and performance task items, however, means more hand-scoring, which is costlier. So Smarter Balanced has decided to substitute 8 to 10 percent of its constructed-response items—which would require AI or hand-scoring—for items that can be machine-scored.
“We have determined that AI has not yet evolved to a place where it can score the types of writing needed to assess the common core,” SBAC spokeswoman Jacqueline King told me in an email. “Luckily, our cost estimates were conservative, so we have been able to switch to hand scoring while maintaining our cost estimates.”
‘Penny-Wise and Pound-Foolish’
In exploring the cost landscape for common-core tests, Chingos also touches briefly on non-consortium options a few states are using. He reports that Kentucky and New York are paying $34 to $37 per student for their own transitional common-core tests. And Alabama, the first state to sign up for ACT’s new suite of assessments, Aspire, got an introductory rate of $11.70 per student for a test that’s projected to cost closer to $20 when it comes online in 2014, the Brookings study said.
For all the talk about test cost, though, states would take care not to be “penny wise and pound foolish,” shortchanging test quality for test price, Chingos writes. He outlines four aspects of test quality that states should take seriously, including “the kinds of tasks that we want students to learn in school,” such as writing; covering the full range and depth of the standards they assess; providing timely feedback; and being proven to predict success in college and careers.