« Historical amnesia | Main | Miracle superintendents and nonsense »

Distorting data


Dear Deb,

Now that our mayor, Michael Bloomberg, is readying himself as a potential candidate for the Presidency, it is clear that education will be one of his signature issues. Sadly, he knows no more about education today than he did when he became mayor in 2001, based on his latest plan to pay poor kids to get higher test scores. That strategy seems to me to be an abject admission of cluelessness: When you don't know anything about teaching or curriculum, then just pay for results.

I understand your frustration about the historical amnesia that you encountered. It seems to be the policy of our New York City Department of Education to wipe out all historical memory and at that they have been quite successful. Apparently, this administration wants the world to believe that whatever it is doing is historic, unprecedented, and of course dazzlingly successful. Part of their strategy is to launch one initiative after another, to persuade the public that they are on the move, when in fact they are merely lurching from one confused plan to another.

Last Friday, the Department released dazzling statistics about the graduation rates of its new small schools. The Department has created about 200 of them, of which 47 have a graduating class this year. The rate for these 47 schools is 73 percent, compared to a citywide rate of.... This is where it gets complicated. The city says it has a graduation rate of 60 percent. The state says the city's graduation rate is 50%. Education Week's latest Diplomas Count report says the city's graduation rate is 45 percent.

So the small schools graduation rate is impressive, right? Wrong. Dee Alpert, who publishes SpecialEducationMuckraker.com is a relentless devourer of data from the city and state education departments, and she sent out a bulletin describing her inability to decipher the real graduation rates at the "new small schools." She says:

1. NYCDOE gives no list of the 'new, small schools' included in its calculations for the spinrelease, so...
2. They give percentages, not raw numbers, for their graduation rates: you can't even try to work backwards to see what was included.
3. Most importantly, they gave no numbers for 'still enrolled,' nor for 'discharged.' ...NYCDOE is notorious for mis-reporting dropouts as having enrolled elsewhere, i.e., discharged from a NYCDOE school's rolls.
4. The numbers for graduates who earned local v. Regents diplomas is also critical, and missing. In prior years, local hs diplomas predominated. According to the NYS Court of Appeals' CFE [Campaign for Fiscal Equity] decision, a local diploma resulting from passing RCT [low-level competency examinations] put a kid-depending on the subject—at between the 6th and 9th grade level. This isn't exactly college prep.

Dee Alpert concludes: "It's all obviously part of the Bloomberg presidential campaign spin." And she strongly recommends "an editorial moratorium on reporting NYCDOE spin numbers unless the complete data set accompanies the press release."

I hope not to befuddle our national readership with too much news and talk about New York City, but there is an important point here. Alpert's analysis reveals how easily education data are distorted for political purposes. We have seen this done before, but seldom so blatantly or so cleverly. It is likely the case that every school superintendent wants to release test scores and graduation rates that show what a success he or she has been. But what we have seen over the past five years is a determined political campaign—not just Bloomberg for President, although that may yet happen—but rather a political campaign to "prove" that mayoral control without any checks or balances is the absolute best way to manage a school system. The Department is incapable of impartial research. Its press releases are filled with the kind of P.R. spin that we have come to associate with politicians running for office, not with research departments where someone has his or her professional reputation on the line.

Whenever there is a new release of test score data from the state, I invariably get calls from reporters, asking what I think of the latest numbers. I have learned over the past few years never to answer their questions until I have had a chance to see the complete data set with my own eyes. I know that the Department has massaged the data and sought out every glimmer of good news while creating a narrative that distracts the reporters' attention from anything unfavorable.

Forgive me, Deborah, but all this media manipulation persuades me that we need national tests (with no stakes), so that states and cities and districts can't play games with the numbers. Failing that, I think that every state should have an independent agency to administer tests and report their results, and that these agencies should be run by professional psychometricians who can neither take credit nor blame when the scores rise or fall. In New York State, when the math scores went down last year, the State Education Department said that the test was harder; when the math scores went up this year, the State Education Department was congratulating itself and the Regents (our state board) for its wise policies. How about an agency that dispenses the facts without fear or favor or self-praise?



I would just point out that graduation rates are inherently muddy. It is not the state of nature that this is a straightforward statistic that is only obfuscated through deliberate sabotage. Its definition is open to dispute, and it is difficult technically to track.

The question Tom is whether they are using the same definition as the one used in 2002?

I'm assuming that Bloomberg is claiming that the June 2002 graduating class at MLK Jt was only roughly 40% the size of its average incoming freshman class. Plus or minus a few percent. I feel comfortable saying this because MLK is a huge school and because Blooberg is claiming increases that are astounding in size--from 40% to 90%. (With similar data for a dozen other schools.) So we could be off a bit in either direction and it would still be astounding. An odd glitch here or there can't account for this massive shift (a few deaths, a few more students moving to Puerto Rico vs those moving back, etc). He's making a staggering claim, and we can't pass it off because statistics are not such straightforward things. A Math 101 student stops and takes note when he gets results on this order of magnitude. I'm assuming Klein/Bloomberg did too.

Even if MLK Jr stopped taking special ed or ELL kids, I'd still find it unlikely. But, in that case making a comparison would be pointless.

We should either stop usig data to say anything or we have to hold people responsible for the deliberate use of the data they rest their work on.
How do you explain it, Tom?


Amazing one-year jumps in graduation rates and reading scores are becoming common in school districts under Mayoral control. Here in Chicago, 40-50% increases test scores are quite common and always make headlines. At one of our small schools (Deb has visited) our 8th-grade reading scores increased 48% last year. They usually do in election years. This, even though we were teaching basically the same kids, with the same teachers, teaching the same curriculum in the same way. This year, the scores dropped about 8% with the same conditions operating. What conclusions can you draw? In small schools especially, test scores and graduation rates can swing wildly from year to year. Most of the test makers even warn in writing that such swings should NOT be used for high stakes evaluations. At the end of the day, 12 years after the Mayor Daley took over and installed his regime of politically-based and scientifically-proven, high-stakes school evaluation, we still find that black students are at the bottom of the so-called "achievement gap." New York is also near the bottom, according to last week's Washington Post story on AP testing. Neither Bloomberg nor Daley have cause for celebration.

In the education world, no news is bad news. Below, a link to a recent NYS Ed. Dept. "report" uploaded to its web site with no press release at all. NYS public high school grads' college-going rates have plummeted, (1990 - 74.1% to 2006 - 65.0%), while non-public high schools' grads college-going rates have increased significantly. This is a disaster and warrants real investigation to ferret out the causes.

Frankly, unless the "new, small high schools" really are producing a whopping percentage of graduates with Regents diplomas, I doubt this drastic decline will slow. Local diploma recipients just don't have the math and science background to get into college and place out above the "must take remedial" level. Nor to get into really good vocational programs at the community college level. And this with SUNY and CUNY no longer using SAT/ACT scores for admissions cutoffs!

And lest we be beguiled by NYSED and NYCDOE graduate numbers spin ... the quote below is directly from a recent NYSED advisory to district administrators:

SUBJECT: Unique Identifier Audit System Update

The e-mails sent by the Department to districts on June 20, 2007, indicating that discrepancies were found in the Unique Identifier Audit System (UIAS) were sent prematurely. The Department will be rerunning the audit and sending new e-mails at a later date. Until then, the following steps should be taken:

· If you received a memorandum regarding “Disappearing Student Audit Results,” no action is required at this time, as many of these errors will be resolved when records for Grades 9–12 students are entered into the Student Information Repository System (SIRS).

· If you received a memorandum regarding “Simultaneous Enrollment Audit Results,” review your student records to ensure that enrollment records for each listed student have accurate beginning and ending dates. ...

"If you received a memorandum regarding “False Dropout Audit Results,” you may change the enrollment ending reason for listed students to “transferred with documentation.” If necessary, the ending date should also be changed. These changes will ensure that these students are not counted as dropouts from your school." June 22, 2007 memo to school district administrators at: http://www.emsc.nysed.gov/irts/SIRS/documentation/UIAS-AuditUpdate6-22-07.htm

A previous year, NYSED announced that its system showed about 20% of all "graduating" students had been reported as actively attending 2 districts at the same time, i.e., dopplegangers galore. NYSED's response was to "assume" clerical errors and disregard the dopplegangers entirely.

We rather desperately need an independent agency in NYS, and nationally, too, to collect and verify all student-related numbers reported. Test scores; graduation/dropout rates; the whole shebang.

Unfortunately, there will always be fiscal penalties for states and districts which report fictitious data because much federal funding is done on a student-per capita basis, and exposes of graduation/dropout misreporting, or downright fraud, often call into question whether entities have received more per capita funding than they were entitled to. The more money attributable to a student, the more likely the numbers are off. Special ed. is a genuine case in point.

Diane - While I relentlessly devour data, fact is, these days, real data availability is getting so poor that ... I'm losing weight.

Dee Alpert


Massachusetts has recently adopted a program which tracks kids from the time the entire class enters ninth grade, through their four years in high school, until graduation. This tabulates kids who pass the state MCAS tests to earn their competency determination and counts them as graduates. Results are still preliminary but a deplorable drop out rate (approximately 20% statewide)continues to predominate, especially among poor and minority youngsters. One of the interesting discoveries by the DOE is transient students, those who move two, three or more times per year. They prove to be very difficult to track with the respective cohort group they started with in ninth grade. Another questionable cohort is homeless youngsters. Talk about impossible to estimate or to track over four years.

Yes, we do need national tests, accompanied by national standards and a common definition of "proficient." Only then will students coast to coast have equal access to the same rich body of knowledge we would want for our own children.

I have the dubious distinction of being the only historian of dropping out in the U.S., after David Angus's death. Diane is right about the susceptibility of graduation statistics to manipulation or at least to friendly misreadings. NYC is not alone in this -- Texas's "dropout rate" scandal is another example, and Florida's claim to have true longitudinal graduation rates is belied by their Enron-like exclusion from the denominator of students who drop out to enter GED programs and then inclusion of GED graduates in the graduation rate.

Tom is right that no one agrees right now on a standard definition of graduation rates, but that doesn't excuse overblown claims in NYC or elsewhere.

(For an ethnographic look at dropping out in NYC in the 1980s, read Michelle Fine's work. I have some material on 1960s NYC dropout programs in Creating the Dropout, and I recommend Rob Warren's work for analyzing different ways of calculating graduation rates.)

The media certainly needs to do its homework, but I wonder how much homework they actually can accomplish, given Dee Alpert's comments. Seems no one can make sense of the DOE's numbers because they don't seem to provide the entire story. Which is a story in and of itself, if you ask me. Ed reporters might do well to devote some coverage to the "fuzzy math."

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments