Trouble in Gotham
"Over the past six years, we’ve done everything possible to narrow the achievement gap – and we have. In some cases, we’ve reduced it by half."New York City Mayor Michael Bloomberg and Chancellor Joel Klein seem intent on taking the "New York City miracle" national. However, a close review of racial and ethnic achievement gaps in New York City over their tenure suggests that Bloomberg and Klein would do well to get their own house in order first.
-Mayor Michael Bloomberg, testifying before the House Committee on Education and Labor, July 17, 2008
"Our African-American and Latino students have gained on their white and Asian peers....What does this show? Achievement for high-needs students is not a dream. It’s happening."
-NYC Chancellor Joel Klein, testifying in the same House Committee hearing
Analyzing data from the NAEP Trial Urban District Assessment from 2003-2007, I will show over the course of this week that in every grade and subject, achievement gaps separating New York City's white and Asian students and their African-American and Hispanic counterparts are unchanged. What's more, there's some suggestive evidence that Asian-black and Asian-Hispanic gaps are actually growing.
You ask, "But Bloomberg and Klein say the achievement gap is closing. Are they lying?" It's not lying as much as it's finessing with the intention of providing a rosier picture of progress than is warranted. But the hard facts - for example, that in 2007, the average black 8th grader in New York City performed at the 20th percentile of the white distribution in reading and in math, and there has been no statistically significant change in the size of that gap since 2003 - makes Bloomberg's victory march, as he likes to say, unconscionable.
Why? First, any respectable psychometrician would advise against measuring achievement gaps by using proficiency rates. I have explained why comparing group differences in proficiency rates provides a misleading measure of between-group inequality here: When Measuring Achievement Gaps, Beware the Proficiency Gap and Scale Score Magic! Why We Shouldn't Rely on Proficiency Rates to Measure Academic Achievement. The central problem is that the black-white achievement gap can be increasing even as the difference between the proficiency of black and white students is closing. (If you are asking yourself, "What the heck are scale scores?", see skoolboy's excellent testing primer.)
Second, the Department of Education has refused to release average scale scores for racial and ethnic groups on the New York state tests. As a result, it's tough to tell what's really going on on the state test. It is possible that gaps in average scale scores are unchanged on the NAEP, but closing on the state test. If this is the case, one possible explanation is that students from historically disadvantaged groups are increasingly drilled in skills that pump up their scores on the state test, but do not generalize to other measures of achievement. (See Why We Should Care About Test Score Inflation.)
Everyone's favorite "but they're different tests" argument has some validity when we are looking at overall levels of achievement. We should not expect state and NAEP tests to track each other perfectly. But that argument is highly suspect in explaining why the size of gaps varies significantly between tests. It's also possible that gaps as measured by the group differences in average scale scores on the state test are unchanged, but small gains by black and Hispanic "bubble" students across the proficiency cut score have led to large increases in proficiency.
To adjudicate between those two explanations, I requested scale score data from Truth Squad captain David Cantor almost a week ago, and was told I would receive these data by last Thursday. I guess the data got lost in the Internets. Unfortunately, denying data access appears to be a growing Department of Education strategy - in this case, the DOE failed to release data, and in other cases, they have released data only in PDF formats that no one can analyze. There is something deeply troubling about an administration that bows down at the altar of "data-driven decision making" but refuses the public access to data that rightfully should be available in a spreadsheet on their website.
Luckily for us, data from the NAEP Trial Urban District Assessment are publicly available, so the Department of Education doesn't get the last word on the size of achievement gaps in New York City. Nor do you have to take my word for it. The National Center for Educational Statistics analyzed the NAEP TUDA scale scores, and found that in every subject and grade level, racial and ethnic achievement gaps have not closed in NYC between 2003-2007. You can find all of these data, as well as NCES's own conclusions about changes in NYC achievement gaps, here. (Click on Advanced --> select a grade level and subject --> under "Jurisdiction," select "Urban District" and "New York City" --> under "Variables," select "Gaps and Changes in Gaps" for "all assessments.")
I hope you will join me this week in trying to make sense of the persistence of achievement gaps in New York City schools and the Department of Education's unflinching willingness to cast facts aside and tell the public otherwise.