Non-Cognitive Skills: Is It Enough for Measurements to Be Valid and Reliable?
Reams have been written lately about incorporating measures of students' non-cognitive and social-emotional skills into high-stakes accountability systems.
The key questions: How can states broaden their definition of school success in a way that brings meaningful changes for students? And how can they avoid creating unintended consequences?
The nation's new education law, the Every Student Succeeds Act, requires states to use at least one indicator—like measures of student engagement or access to advanced coursework—alongside more traditional academic measures in tracking schools' success. And some have suggested using measures of students' social-emotional and non-cognitive skills, like social awareness and growth mindset, as that "extra indicator."
But, for many researchers, the concerns swirling around that idea are more significant than the enthusiasm. Those ideas have surfaced again recently in coverage of the California CORE districts, which incorporated measurements of social-emotional skills into their federal accountability system. As I wrote in December:
"Researchers ... have warned that those measures should not be used for school accountability because they are subject to biases and flawed responses. For example, in some schools in other states, students who've been taught about issues like self-control rate themselves lower than their peers because they have a greater awareness of what those concepts mean.
Currently, 'perfectly unbiased, unfakeable, and error-free measures are an ideal, not a reality,' researchers Angela Duckworth and David Yeager said in a May essay published in Educational Researcher that detailed an array of flaws with current measures."
But what if researchers can prove that those measures are valid and reliable? Would that be enough to quell concerns? Maybe not.
As Harvard Graduate School of Education Professor Martin West writes in an essay for the Brookings Institution, even well-tested measures that avoid problems like reference bias present many unknowns when used for high stakes purposes. One of those unknowns: What will schools that score low on non-cognitive indicators do to move the needle? Will it be effective? Will it create unintended consequences?
West evaluated the CORE district's social-emotional measures and found that they did correlate with school success in many areas. (You can learn more about how CORE developed its social-emotional measures in this blog post.) But there are still unknowns, West writes:
"In sum, our preliminary analysis of the data from CORE's field test provides a broadly encouraging view of the potential for self-reports of social-emotional skills as an input into its system for evaluating school performance. That said, the view it provides is also quite limited. It says nothing about how self-report measures of social-emotional skills would perform in a high-stakes setting—or even with the very modest weight that will be attached to them this year within CORE. Nor can we say anything about how CORE's focus on social-emotional learning will alter teacher practice and, ultimately, student achievement. The results presented above are best thought of as a baseline for future analysis of these issues—and many more.
The potential for broader state accountability systems under the new federal law creates a learning opportunity, West writes:
"One reason researchers don't have much to say about these questions currently is that the No Child Left Behind Act effectively required all fifty states to adopt a common approach to the design of school accountability systems. Fifteen years later, we know a lot about the strengths of this approach and even more about its weaknesses—but next to nothing about those of potential alternatives. The recently enacted Every Student Succeeds Act provides both opportunity and incentive for experimentation. What is important is that we learn from what happens next. We need to let evidence speak."
- CORE Districts to Share Social-Emotional Measures to Inform Accountability Plans
- Moving Beyond Just Academics in Assessing Effectiveness
- ESSA Law Broadens Definition of School Success