How Can Teachers Learn Deeply? By Scoring Student Assessments
This post is by Vinci Daro and Ruth Chung Wei who work at the Stanford Center for Assessment, Learning, and Equity (SCALE) at Stanford University.
"This is absolutely critical to improve our instruction."
- California teacher responding to a survey about a professional development training
Teachers have been known to groan about useless workshops and trainings, so it is unusual to hear such high praise about a training session. Can you guess what kind of training this teacher was talking about?
This teacher's perspective is representative of many who participated in Building Educator Assessment Literacy trainings during the Spring of 2015 across three states--California, New Hampshire, and Oregon. The BEAL project is a partnership between SCALE and WestEd (and supported by the William & Flora Hewlett Foundation, the Bechtel Foundation, and the Stuart Foundation). During the two-day BEALtrainings, teachers engage in analyzing and scoring student responses to performance tasks from the Smarter Balanced Assessment Field Test, and reflect on the implications of the tasks, the student work, and the scoring experience for their own instructional practice. While the first round of these professional development events completed last month, BEAL project leaders continue to receive requests for additional training sessions so more teachers can benefit.
Why would teachers have such strong responses to a training focused on scoring answers to test items on a large-scale, standardized assessment?
In this period of transition to new standards, assessment systems that provide reliable evidence of what is working and what is not in schools are still being built, and it will take time to pilot, field test, refine, and validate those assessments. Having now moved through the first implementation year of both the Smarter Balanced and PARCC (Partnership for the Assessment of Readiness for College and Careers) assessments, we have a better understanding of the design of the new assessments, including the role of performance tasks. The purpose of these performance tasks (as explicated on this blog here and here) is to elicit evidence of competencies that matter, but cannot be assessed adequately with multiple-choice and short-answer assessment formats. The challenge and the opportunity--presented by hand-scoring student responses to performance tasks within these assessments--raise a number of important questions. Among them:
What role do we imagine teachers playing within large-scale, standardized assessment systems?
What role do teachers want to play?
We've learned a few things about teachers' perspectives on the question of teachers' role in large-scale assessment systems through our work on the BEAL project. On the post-training survey, participants were asked:
How do you think your experience hand-scoring items on the Smarter Balanced Performance Tasks during this training will (or might) impact your instruction?
Benefits of hand-scoring for teachers and students, in teachers' words
The most common themes in responses to this question spoke to three benefits--for teachers and students--of hand-scoring experience.
- Hand-scoring clarifies the purpose and expectations of the assessment
"This is invaluable to seeing how the rubric criteria translates into a student response, the many different acceptable ways students can respond, and see areas where instruction could be strengthened such as in developing explanations."
"Being aware of how items are scored gives me a better idea of the kinds of tasks students will be asked to do and the level of complexity. This will help me to select appropriately rigorous enough tasks. My teaching focus will be primarily on the thinking process and use of information to solve problems."
- Hand-scoring deepens understanding of student competencies
"... looking at student work will reveal the gaps and guide the shifts that need to be made in the classroom. Hand scoring a writing task is like opening a student's brain and getting a more intimate perspective on the thinking and learning. There is much to be learned from these comprehensive summative performance tasks."
- Hand-scoring supports more focused instructional planning, and more purposeful learning experiences for students
"...teachers could begin to analyze their instruction as it pertains to offering students multiple opportunities to reason, explain their reasoning, and thinking about how assumptions and answers to one part of a question can and does impact other portions. Also, the idea that one needs to consider "what is reasonable" when answering a question and be able to logically defend that decision."
"I will be more intentional about classroom discourse and assure my students are doing real problems that push their mathematics to the deeper thinking level."
Benefits of teacher involvement for large-scale assessment systems
Through our experiences with the BEAL project as well as other assessment-related projects, we have found that teacher involvement in designing, implementing, and scoring student work has benefits for the assessment systems themselves.
- Credibility depends on transparency
Teachers can't be asked to support an assessment system that they don't fully understand; to a large extent, the credibility of an assessment system depends on transparency. Giving teachers meaningful, purposeful experiences participating in the efforts of an assessment system is one way to make a system transparent. Scoring student work on performance tasks gives teachers direct access to the criteria for success. Moreover, scoring also provides a window into the item development process, rubric development process, and the reporting system.
- We need greater coherence between state assessments and instructional contexts
As new standards are adopted, state assessment systems must undergo revision and change. Teachers need reliable ways of tracking these changes, and how they relate to what their own students are accustomed to. Teacher involvement is necessary if assessment is to be linked in meaningful, timely, and locally particular ways to curriculum and instruction. Involving teachers as designers, implementers, and scorers will deepen--and keep current--their understandings of the assessment system.
- Engaging educators in hand-scoring can reduce the cost of including performance assessments in large-scale assessment programs
Although most of the hand-scoring for both the PARCC and Smarter Balanced Consortium members is currently completed by external contractors (e.g., professional raters employed by ETS, Pearson, etc.), there is no reason why teachers should be excluded as raters, provided that they are not scoring their own students' responses. In New York State, teachers accepted their scoring responsibilities related to the Regents Examinations as part of their teacher contracts for decades. Formally training teachers to meet calibration standards within a blind, distributed scoring system would reduce some of the burden and cost of scoring assessments, while supporting teachers' professional learning.
Do teachers want to be involved?
Yes. On the BEAL survey, more than 95 percent of respondents agreed or strongly agreed with the statement: It is valuable for teachers to have experience hand-scoring large-scale summative assessment responses (N = 475). More than 94 percent agreed or strongly agreed with the statement: When teachers play a role as scorers for large-scale summative assessments, it is valuable for an educational assessment system (N = 474).
Let's treat teachers as capable professionals by entrusting them with high-stakes student work. Let's draw on teachers' expertise in interpreting student work--who spends more time interpreting student work than teachers, anyway?--while simultaneously building this expertise in teachers.
Increasing numbers of states, such as those participating in the Council of Chief State School Officers' Innovation Lab Network, are exploring the feasibility of incorporating performance assessments as a key component of balanced systems of assessment. If these states are going to invest in building clear, coherent, and engaging performance tasks that invite students to demonstrate -in meaningful ways--what they know and can do in the context of large-scale assessments, then let's give teachers meaningful opportunities to learn from student performance on these tasks. And if we're going to invest in that, let's give the system meaningful opportunities to learn from teachers.
We know that without the involvement of teachers, any attempt at education reform will fail, because it is the teachers who must execute the policies through direct interactions with students in their classrooms. If the ultimate goal of administering large-scale assessments is to generate data that can be used to improve what teachers do with students, then teachers need to be involved.