"Over the past six years, we’ve done everything possible to narrow the achievement gap – and we have. In some cases, we’ve reduced it by half."New York City Mayor Michael Bloomberg and Chancellor Joel Klein seem intent on taking the "New York City miracle" national. However, a close review of racial and ethnic achievement gaps in New York City over their tenure suggests that Bloomberg and Klein would do well to get their own house in order first.
-Mayor Michael Bloomberg, testifying before the House Committee on Education and Labor, July 17, 2008
"Our African-American and Latino students have gained on their white and Asian peers....What does this show? Achievement for high-needs students is not a dream. It’s happening."
-NYC Chancellor Joel Klein, testifying in the same House Committee hearing
Analyzing data from the NAEP Trial Urban District Assessment from 2003-2007, I will show over the course of this week that in every grade and subject, achievement gaps separating New York City's white and Asian students and their African-American and Hispanic counterparts are unchanged. What's more, there's some suggestive evidence that Asian-black and Asian-Hispanic gaps are actually growing.
You ask, "But Bloomberg and Klein say the achievement gap is closing. Are they lying?" It's not lying as much as it's finessing with the intention of providing a rosier picture of progress than is warranted. But the hard facts - for example, that in 2007, the average black 8th grader in New York City performed at the 20th percentile of the white distribution in reading and in math, and there has been no statistically significant change in the size of that gap since 2003 - makes Bloomberg's victory march, as he likes to say, unconscionable.
Why? First, any respectable psychometrician would advise against measuring achievement gaps by using proficiency rates. I have explained why comparing group differences in proficiency rates provides a misleading measure of between-group inequality here: When Measuring Achievement Gaps, Beware the Proficiency Gap and Scale Score Magic! Why We Shouldn't Rely on Proficiency Rates to Measure Academic Achievement. The central problem is that the black-white achievement gap can be increasing even as the difference between the proficiency of black and white students is closing. (If you are asking yourself, "What the heck are scale scores?", see skoolboy's excellent testing primer.)
Second, the Department of Education has refused to release average scale scores for racial and ethnic groups on the New York state tests. As a result, it's tough to tell what's really going on on the state test. It is possible that gaps in average scale scores are unchanged on the NAEP, but closing on the state test. If this is the case, one possible explanation is that students from historically disadvantaged groups are increasingly drilled in skills that pump up their scores on the state test, but do not generalize to other measures of achievement. (See Why We Should Care About Test Score Inflation.)
Everyone's favorite "but they're different tests" argument has some validity when we are looking at overall levels of achievement. We should not expect state and NAEP tests to track each other perfectly. But that argument is highly suspect in explaining why the size of gaps varies significantly between tests. It's also possible that gaps as measured by the group differences in average scale scores on the state test are unchanged, but small gains by black and Hispanic "bubble" students across the proficiency cut score have led to large increases in proficiency.
To adjudicate between those two explanations, I requested scale score data from Truth Squad captain David Cantor almost a week ago, and was told I would receive these data by last Thursday. I guess the data got lost in the Internets. Unfortunately, denying data access appears to be a growing Department of Education strategy - in this case, the DOE failed to release data, and in other cases, they have released data only in PDF formats that no one can analyze. There is something deeply troubling about an administration that bows down at the altar of "data-driven decision making" but refuses the public access to data that rightfully should be available in a spreadsheet on their website.
Luckily for us, data from the NAEP Trial Urban District Assessment are publicly available, so the Department of Education doesn't get the last word on the size of achievement gaps in New York City. Nor do you have to take my word for it. The National Center for Educational Statistics analyzed the NAEP TUDA scale scores, and found that in every subject and grade level, racial and ethnic achievement gaps have not closed in NYC between 2003-2007. You can find all of these data, as well as NCES's own conclusions about changes in NYC achievement gaps, here. (Click on Advanced --> select a grade level and subject --> under "Jurisdiction," select "Urban District" and "New York City" --> under "Variables," select "Gaps and Changes in Gaps" for "all assessments.")
I hope you will join me this week in trying to make sense of the persistence of achievement gaps in New York City schools and the Department of Education's unflinching willingness to cast facts aside and tell the public otherwise.
I've always thought it was odd to measure student gains based on proficiency categories, rather than on average scores.
As a former teacher in California, my colleagues and I were explicitly instructed to focus on pushing the "bubble students" (as you call them) over the edge into the next category. As such, I agree with your assessment of the NYC situation: An increase in the number of students who attain "proficiency" status on an exam does not necessarily indicate a corresponding increase in actual, average test scores across a population.
Thanks for pointing this issue out to the education blogosphere.
I suppose, though, when you modify your arguments by reducing them to "some cases," they're so utterly nebulous that you can claim pretty much anything. I'm reminded of the Tweed claims of having reduced class sizes by .2 students per class, or whatever it was they said.
That didn't stop credulous professional media commentators from stating without reservation that class sizes were down in NY. Clearly none of them were seeing what I see every day.
By the way, I love that picture. And I think the "New York Miracle" is a particularly apt comparison.
Don't you think it's interesting that teachers are subjected to absolute transparency - to the point that it is almost offensive at times - yet, those at the very top twist the truth in such ways? Those of us on the ground are urged to "use the data" to drive our instruction...and while I think that can be a very good thing for teachers, I am beginning to wonder if I misinterpreted "using" the data. I thought it meant to honestly look at the numbers, etc to help guide future efforts, not to find loop holes and ways to pull the wool over people's eyes. My bad.
Attorney DC, NYC Educator, and Mimi - I am so glad that you three always weigh in with insight into how this is panning out in real classrooms and schools - we just can't get that perspective from the 30,000 feet above stuff that I present here.
Attorney DC - Agreed that this bubble issue is no joke - and why people continue to brush it off as "anecdotal evidence," I don't know. There are now at least two major studies that identify this effect with large datasets.
NYC Educator - Thanks for the props on the pic. I think Bloomberg did worse here than his usual "truth stretching" as there is no subject or grade where the gap has closed by half - if we are to believe NCES, it has not closed at all.
Mimi - The irony of asking teachers to convene "Data Inquiry Teams" while massaging and finessing their own data should not be lost on any of us. If you're for data-driven decision making, as DOE claims to be, you'd better be willing to put all your data on the table for everyone to review. That they have pulled back on public access to data - most recently, by only posting the Learning Environment Survey for 2008 in PDF files for each schools, though they did post an Excel for 2007 - is very distressing. This makes it impossible to analyze the data without entering all of the data by hand or spending oodles of time scanning each PDF in to make them machine readable - which is obviously something no one has time to do.
I cam across a research paper. Here is what it says, in part:
I am afraid educational research itself is the problem!
And my favorite soundbite from the meeting came from Rhee, where she, in one breath, says it will be years after she is gone that her effects will be seen. In the next breath she touts her amazing (they are amazing. they amaze me, like magic amazes me.) successes from being at the helm for a whole year!
National standards with national tests (like the existing NAEP) would go a long way towards eliminating/reducing all the state testing shenanigans.
As a political aside, you could equate Bloomberg's fudging NY state test results to potentially gain a leg up on McCain's vice presidential selection the same way George W fudged Texas state tests to put himself in position for the presidential nomination in 1999.
For true magic, follow Paul Vallas around the nation as he pulls the wool over everyone's eyes from Chicago to Philly to New Orleans yet somehow nothing sticks.