Exclusion Rates in NAEP: A Pot Still Simmering
The reaction to yesterday's NAEP reading scores tilted more toward disappointment than outrage. I guess I was sorely mistaken when I thought that the scores would provide easy ammo for education-reform advocates of various stripes.
One thing I found pretty interesting in yesterday's release was something the National Assessment Governing Board didn't highlight in its report: the exclusion and accommodation rates for students with disabilities or limited English fluency.
If you flip to the back of the report, past the pretty multicolored pages of results to the pages of black-and-white columns of figures, you will see the proportions of children each state excludes from the NAEP or provides with testing accommodations (Tables A-7 for 4th grade rates and A-8 for 8th grade, pages 50 and 51 of the report). To say that these rates vary a bit from state to state would be, um, an understatement.
The interplay between exclusion and assessment is intriguing. Some states include more kids, but also provide more accommodations (Louisiana, for instance, excluded only 9 percent of its ELL or special-ed 4th graders, but 72 percent of the students in that group who took the test did so with accommodations.). Other states exclude whopping portions of kids from the test. Maryland did so with 57 percent of its 4th grade ELLs and spec-ed students.
The "assessed without accommodations" column provides a quick view of what states are doing. It essentially tells you what portion of the ELL and spec-ed kids took the NAEP the same way as their English-fluent, regular-ed peers. And the numbers vary wildly here. UPDATE: An Alert Reader cautions here that I should be clear on something: The column in this table, like the similar one in Table A-8, shows the portion of ELLs and spec-ed students kids assessed without accommodations, not the portion of all students identified, excluded or accommodated. If you want that, it's in tables A-4 and A-5. Thank you, alert reader!!)
In 8th grade, only 5 percent of New York's ELL and spec-ed students took the NAEP without accommodations. Seven percent did so in Florida, and 8 percent did so in Delaware. By contrast, 64 percent of those student groups in 8th grade took the test without accommodations in Alabama, and 70 percent did so in California. In 4th grade, New York (6 percent) again comes up on the low end of the spectrum with the portion of ELL and spec-ed students who took the test without accommodations. New Jersey (10 percent) and Ohio (11 percent) help carry the bottom end, compared with California, which administered the NAEP without accommodations to 77 percent of its spec-ed and ELL 4th graders, and Alabama, which did so with 61 percent.
Kentucky, which I point out in my story as the only state that saw gains on reading at both the 4th and 8th grade levels, tested only 30 percent of its 4th grade ELLs and spec-ed students and 13 percent of those students in 8th grade without accommodations. It outright excluded nearly half of that group in 4th grade and more than half at the 8th grade level.
National averages among states for assessing ELL and special-ed students without accommodations: 31 percent at the 8th grade level and 40 percent at the 4th grade.
So what effect does all this have on the NAEP scores? West Virginia education commissioner Steven Paine, a NAGB member who helped present the NAEP findings yesterday, said the varying exclusion rates were an ongoing concern for him. "Inevitably," he said, "they do not seem fair and they raise questions and concerns." I decided to venture into that a bit more, and asked if the NAGB had any definitive answers about the extent of the effect that exclusion rates have on the scores.
Stuart Kerachsky, the deputy commissioner for NCES, which oversees the NAEP, said the agency has been studying the issue and trying to "calibrate" the effect it has on the scores. He didn't provide numbers, but said he would "characterize the effect as small." The problem of NAEP exclusion gets complicated, too, he said, by states' own, varying standards for identifying and characterizing their spec-ed and ELL populations.
This tension has been going on for quite some time. Back in 1992, we were writing about the problem. And as recently as last week, my colleague Steve Sawchuk wrote about a new policy the NAGB adopted to try to reduce exclusion rates.
There has been much in between those times as well. For a few examples, see EdWeek's coverage of draft proposals of policies and a hearing about it last year, a move by NAGB in 2007 to be more transparent about it by featuring the exclusion rates more prominently in reports, and a report from 2005 in which the GAO revised its 5 percent estimate of exclusion rates to—oops!—40 percent. EdWeek did an overview of these simmering issues back in 2003, as well.