Consider this puzzle: in 2007, the average scale score on the New York State ELA Test was 661. In 2008, it is also 661. Yet the overall level of proficiency has increased by 3 percentage points, from 68% to 71%. How is this possible?
When we measure student achievement solely based on the proportion of students who have jumped over a bar, we can end up with pretty misleading picture of student performance.
Take a look at grades 3, 5, and 8 in the graph below, which shows the change in ELA average scales scores and passing rates for New York state. In each case, the average scale score increased by 2 points, or about .05 standard deviations. But the increases in the percentage of students who were proficient varied widely across those grades. In 3rd grade, there was an increase of 3 percentage points. In 5th grade, there was a much larger increase - 9.5 percentage points. And in 8th grade, though the average scale score increased, the percentage of students who were proficient actually decreased .9 percentage points.
Should we conclude that our 5th graders are much better off than they’ve been in the past, and 8th graders are falling behind? Definitely not - 5th grade just happened to hit the sweet spot of the distribution - but that’s what you’d get if you relied only on passing rates.
In short, know what you’re buying when you’re looking at passing rates. They can increase substantially by moving a small number of kids up a few points - just enough to clear the cut score. In some of the grade levels above, there are good reasons to suspect that these small moves may partially explain large jumps in proficiency on the New York State ELA test.