When Measuring Achievement Gaps, Beware the Proficiency Trap
There are at least two ways of thinking about the relationship between achievement and kids' life chances. The first is to consider, in absolute terms, the set of skills that students have. The second views achievement as relative. Most coveted opportunities - jobs, college admission, a good grade in a college course, or positive evaluations in the workplace - are not divvied up based on students crossing an arbitrary line of proficiency or competence. We don't give everyone a job who's passed a basic reading test, nor do we admit everyone to UC-Berkeley who's received more than a 700 on the verbal SAT. Every student in a college course at NYU can't get an A, and faculty measure students' performance against others to assign grades. In short, all of these decisions are made by comparing the performance of those in a pool, and choosing those who come out near the top.
The proficiency view, to my mind, is certainly important to consider when we are thinking about building stocks of human capital. But if we are concerned about inequality and social stratification - ensuring that, on average, every demographic and socioeconomic group is equally prepared to compete in higher education and the workplace - relative achievement measured on a continuous scale is what matters, not proficiency rates.
Which brings us to how we currently measure "achievement gaps" between social groups, and why this method is tragically flawed. For example, if you look at the NYC press release from this year's test scores, you'll see that gaps are defined as the difference between the percentage of students that are proficient in each group. If the gap in proficiency between black and white students was 29 percentage points last year in 4th grade ELA, and now is 26 percentage points, we hear that the gap has narrowed by 3 percentage points. But it's possible that the gap in the achievement that matters - the continuous measure of achievement - has actually grown.
Let me give a brief example to illustrate. If we use the proficiency logic, the achievement gap that separates the Bronx and the affluent suburbs of Westchester is closing. And indeed on Monday, Mayor Bloomberg crowed that NYC is catching up to the suburbs. If we take a look at 7th grade math, we see that there was a 30 percentage point Westchester/Bronx gap in proficiency in 2007 (73% versus 43%), but this year, there is only a 25 percentage point gap (83% versus 58%). If we use a proficiency measure, the achievement gap has closed by 5 percentage points.
Not so fast. The achievement gap, if we measure the differences in the average student scores in Westchester and the Bronx, has actually increased in 7th grade math. The scale score gap was 28 points last year. Put differently, the average Bronx 7th grader scored at the 23rd percentile of the Westchester distribution in 2007. This year, the gap was 30 points. Now the average Bronx 7th grader has dropped to the 21st percentile of the Westchester distribution, even though the achievement gap, as measured by proficiency, is closing.
Take-home point: when you hear about achievement gaps closing based on proficiency scores, beware of what you're being sold.