« Cool People You Should Know: Marta Tienda | Main | In New York City, A Long Wait Ahead to Close the Math Achievement Gap »

Trouble in Gotham


"Over the past six years, we’ve done everything possible to narrow the achievement gap – and we have. In some cases, we’ve reduced it by half."
-Mayor Michael Bloomberg, testifying before the House Committee on Education and Labor, July 17, 2008

"Our African-American and Latino students have gained on their white and Asian peers....What does this show? Achievement for high-needs students is not a dream. It’s happening."
-NYC Chancellor Joel Klein, testifying in the same House Committee hearing
New York City Mayor Michael Bloomberg and Chancellor Joel Klein seem intent on taking the "New York City miracle" national. However, a close review of racial and ethnic achievement gaps in New York City over their tenure suggests that Bloomberg and Klein would do well to get their own house in order first.

Analyzing data from the NAEP Trial Urban District Assessment from 2003-2007, I will show over the course of this week that in every grade and subject, achievement gaps separating New York City's white and Asian students and their African-American and Hispanic counterparts are unchanged. What's more, there's some suggestive evidence that Asian-black and Asian-Hispanic gaps are actually growing.

You ask, "But Bloomberg and Klein say the achievement gap is closing. Are they lying?" It's not lying as much as it's finessing with the intention of providing a rosier picture of progress than is warranted. But the hard facts - for example, that in 2007, the average black 8th grader in New York City performed at the 20th percentile of the white distribution in reading and in math, and there has been no statistically significant change in the size of that gap since 2003 - makes Bloomberg's victory march, as he likes to say, unconscionable.

Why? First, any respectable psychometrician would advise against measuring achievement gaps by using proficiency rates. I have explained why comparing group differences in proficiency rates provides a misleading measure of between-group inequality here: When Measuring Achievement Gaps, Beware the Proficiency Gap and Scale Score Magic! Why We Shouldn't Rely on Proficiency Rates to Measure Academic Achievement. The central problem is that the black-white achievement gap can be increasing even as the difference between the proficiency of black and white students is closing. (If you are asking yourself, "What the heck are scale scores?", see skoolboy's excellent testing primer.)

Second, the Department of Education has refused to release average scale scores for racial and ethnic groups on the New York state tests. As a result, it's tough to tell what's really going on on the state test. It is possible that gaps in average scale scores are unchanged on the NAEP, but closing on the state test. If this is the case, one possible explanation is that students from historically disadvantaged groups are increasingly drilled in skills that pump up their scores on the state test, but do not generalize to other measures of achievement. (See Why We Should Care About Test Score Inflation.)

Everyone's favorite "but they're different tests" argument has some validity when we are looking at overall levels of achievement. We should not expect state and NAEP tests to track each other perfectly. But that argument is highly suspect in explaining why the size of gaps varies significantly between tests. It's also possible that gaps as measured by the group differences in average scale scores on the state test are unchanged, but small gains by black and Hispanic "bubble" students across the proficiency cut score have led to large increases in proficiency.

To adjudicate between those two explanations, I requested scale score data from Truth Squad captain David Cantor almost a week ago, and was told I would receive these data by last Thursday. I guess the data got lost in the Internets. Unfortunately, denying data access appears to be a growing Department of Education strategy - in this case, the DOE failed to release data, and in other cases, they have released data only in PDF formats that no one can analyze. There is something deeply troubling about an administration that bows down at the altar of "data-driven decision making" but refuses the public access to data that rightfully should be available in a spreadsheet on their website.

Luckily for us, data from the NAEP Trial Urban District Assessment are publicly available, so the Department of Education doesn't get the last word on the size of achievement gaps in New York City. Nor do you have to take my word for it. The National Center for Educational Statistics analyzed the NAEP TUDA scale scores, and found that in every subject and grade level, racial and ethnic achievement gaps have not closed in NYC between 2003-2007. You can find all of these data, as well as NCES's own conclusions about changes in NYC achievement gaps, here. (Click on Advanced --> select a grade level and subject --> under "Jurisdiction," select "Urban District" and "New York City" --> under "Variables," select "Gaps and Changes in Gaps" for "all assessments.")

I hope you will join me this week in trying to make sense of the persistence of achievement gaps in New York City schools and the Department of Education's unflinching willingness to cast facts aside and tell the public otherwise.

I've always thought it was odd to measure student gains based on proficiency categories, rather than on average scores.

As a former teacher in California, my colleagues and I were explicitly instructed to focus on pushing the "bubble students" (as you call them) over the edge into the next category. As such, I agree with your assessment of the NYC situation: An increase in the number of students who attain "proficiency" status on an exam does not necessarily indicate a corresponding increase in actual, average test scores across a population.

Thanks for pointing this issue out to the education blogosphere.

I suppose, though, when you modify your arguments by reducing them to "some cases," they're so utterly nebulous that you can claim pretty much anything. I'm reminded of the Tweed claims of having reduced class sizes by .2 students per class, or whatever it was they said.

That didn't stop credulous professional media commentators from stating without reservation that class sizes were down in NY. Clearly none of them were seeing what I see every day.

By the way, I love that picture. And I think the "New York Miracle" is a particularly apt comparison.

Don't you think it's interesting that teachers are subjected to absolute transparency - to the point that it is almost offensive at times - yet, those at the very top twist the truth in such ways? Those of us on the ground are urged to "use the data" to drive our instruction...and while I think that can be a very good thing for teachers, I am beginning to wonder if I misinterpreted "using" the data. I thought it meant to honestly look at the numbers, etc to help guide future efforts, not to find loop holes and ways to pull the wool over people's eyes. My bad.

Attorney DC, NYC Educator, and Mimi - I am so glad that you three always weigh in with insight into how this is panning out in real classrooms and schools - we just can't get that perspective from the 30,000 feet above stuff that I present here.

Attorney DC - Agreed that this bubble issue is no joke - and why people continue to brush it off as "anecdotal evidence," I don't know. There are now at least two major studies that identify this effect with large datasets.

NYC Educator - Thanks for the props on the pic. I think Bloomberg did worse here than his usual "truth stretching" as there is no subject or grade where the gap has closed by half - if we are to believe NCES, it has not closed at all.

Mimi - The irony of asking teachers to convene "Data Inquiry Teams" while massaging and finessing their own data should not be lost on any of us. If you're for data-driven decision making, as DOE claims to be, you'd better be willing to put all your data on the table for everyone to review. That they have pulled back on public access to data - most recently, by only posting the Learning Environment Survey for 2008 in PDF files for each schools, though they did post an Excel for 2007 - is very distressing. This makes it impossible to analyze the data without entering all of the data by hand or spending oodles of time scanning each PDF in to make them machine readable - which is obviously something no one has time to do.

I cam across a research paper. Here is what it says, in part:

A basic research goal for educational research is to determine causation. The advantage of using SBR is that having a control group allows a better explanation of the variance in the data. Even with a control group, however, educational research must account for the intentionality of both students and teachers. This is not trivial [emphasis mine]. Intentionality refers to the ability of students and/or teachers to behave in a way that is not norm-regulated (Howe, 2005, p. 311). To determine the effectiveness of an intervention, researchers must assume that control and test group teachers deliver the same content in exactly the same way to the same mix of children. Research design options can minimize errors for these assumptions, but not without considerable efforts and collaboration (National Research Council, 2005, p. 30). Student populations vary widely in their ethnicity, socio-economic status, and family backgrounds. These factors can be controlled for. The student motivation, the intentionality of students and teachers and the variety of administrative philosophies make it impossible to control for all variability [emphasis mine]. Many of these constructs cannot be measured or scaled as a data point; however, each can strongly influence educational achievement. The variance inherent within each of the constructs does not lend itself well to SBR, even with randomization and a control group.

I am afraid educational research itself is the problem!

And my favorite soundbite from the meeting came from Rhee, where she, in one breath, says it will be years after she is gone that her effects will be seen. In the next breath she touts her amazing (they are amazing. they amaze me, like magic amazes me.) successes from being at the helm for a whole year!

National standards with national tests (like the existing NAEP) would go a long way towards eliminating/reducing all the state testing shenanigans.

As a political aside, you could equate Bloomberg's fudging NY state test results to potentially gain a leg up on McCain's vice presidential selection the same way George W fudged Texas state tests to put himself in position for the presidential nomination in 1999.

For true magic, follow Paul Vallas around the nation as he pulls the wool over everyone's eyes from Chicago to Philly to New Orleans yet somehow nothing sticks.

Comments are now closed for this post.


Recent Comments

  • Norm: For true magic, follow Paul Vallas around the nation as read more
  • Paul Hoss: National standards with national tests (like the existing NAEP) would read more
  • tft: I cam across a research paper. Here is what it read more
  • eduwonkette: Attorney DC, NYC Educator, and Mimi - I am so read more
  • Mimi: Don't you think it's interesting that teachers are subjected to read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School