Thorny Questions Arise About Use of Field-Test Results
California's face-off with the U.S. Department of Education this week illustrates a tricky question that's arising as states figure out what tests to administer to students in the spring of 2014: Can you use the results of field tests? And if so, how?
As you might recall from our story about the California situation, the Golden State is proposing to drop most of its longtime STAR testing system and use only the field tests being designed by the Smarter Balanced Assessment Consortium. Cognizant that a field test is, by definition, still in development, state officials said they would not use those results for accountability. That raised a ruckus in some quarters, where strong accountability proponents decried the lack of feedback for students, parents, and teachers on how schools are doing.
In my reporting for the California story, I heard a tidbit that was intriguing: While California wasn't planning to use results from the field tests, there was another state in the Smarter Balanced consortium that was. So I called the consortium's executive director, Joe Willhoft, to find out what was up with that.
He confirmed that one state in the group is indeed going to get results from the field tests administered to its students. He declined to name the state, and he didn't have details on how that state planned to use the test results, other than for its own analysis.
Willhoft, who oversaw assessment for Washington state before taking the helm at Smarter Balanced, said he has his own reservations about the wisdom of using field-test results in various contexts. (This was a concern echoed by assessment expert Derek Briggs in our story about California.) But the central issue the consortium had to face, Willhoft said, was that the data are not owned by the consortium's managers.
"Frankly, giving states field-test results does present some challenges with regard to the utility of that data," he said. "But the bottom line is, who does the data belong to? When we thought about it, we concluded that fundamentally, the data belongs to the states themselves.
"We are voicing significant caution, but this [one] state feels they have an urgent need for these data, and in the final analysis, it's their data. We have to be agnostic about how states use it. We can establish policies about what we consider best use of the data. But we don't govern the states; they govern us."
The state that has requested the data will have to pay for it, Willhoft said, because supplying such data was not part of the contract with the vendor for the field tests. (That contract covers designing and administering the field tests and analyzing results for the purposes of refining the test itself.)
When Smarter Balanced asked the vendor for an estimate of what it would cost to provide scores for all the field-test items given in that state, it proved to be "a pretty hefty cost," Willhoft said. It's cheaper to provide students' answers, unscored, but providing raw scores gets costlier. Even more expensive: providing the scores on a scale.
That requires the vendor to create a scale for the results, because it's too early in test development to be able to establish an official scale for the Smarter Balanced assessment, Willhoft said. That can't be done consortium-wide until the group obtains feedback from the field tests in all member states, so it understands whether the items measured what they are intended to measure.
The state that is moving ahead to obtain results will get them on a scale created only for its students and the items they took, he said.
"The scored file will be on a scale, but it will not be the Smarter Balanced scale," he said.
What data can be provided, however, is a very different matter from what it can be validly used for. We don't yet know how this one unnamed state will use the field-test data. But I'm guessing that other states will confront similar questions as they move forward.