We’ve reached a point in public education where the informed taxpayer and the generalist elected official can’t rely on data and analysis ostensibly provided to shed light on important policy problems. Every study has to be matched up against the interests of the author or proponent, subjected to a methodological review, and finally understood in the context of technical choices that inevitably tend to bias results one way or another. Opponents of whatever prescription is presented with the report attack the data and analysis with a ferocity that suggests the other side is utterly unscrupulous.
Wading through this mess is tedious. And when all is said and done, the difference between the two sides often doesn’t pass the “so what?” test.
This describes the wars over privatization by Education Management Organizations, charter schools, vouchers, reading, math - and now high school dropouts.The November 18 issue of the Des Moines Register includes an opinion piece by that city’s superintendent of schools, Nancy Sebring. The subject, John Hopkins researcher Robert Balfanz’s recent study identifying 1700 high schools in the United States as “dropout factories”. His criteria - schools where 60 percent or fewer of high school freshman graduate three years later. His method, divide the number of graduates by the number of freshmen three years earlier. This approach rates three Des Moines high schools (East, North, and Lincoln) as averaging a bit more than 50% graduation over the three school years ending in the spring of 2006.
Attacking the Method
Sebring sets the context of her remarks by questioning the analyst’s grasp of the real world, puting Balfanz in that group of academics who are far removed from the day-to-day issues facing public education. She spends most the first 593 words of her 753 word article questioning the report’s methodology and explaining her preferred alternative.
Sebring does not point out that many of her criticisms were addressed in Balfanz’s method. Consider her first example:
Sebring:
For example, according to Johns Hopkins, students who move to a different school district or transfer to a different high school within a district are dropouts.
Balfanz:
There are local reasons why a school’s promoting power (his term for true graduation rates) may be more or less than the state reported graduation rate. The main reason promoting power would be lower than a school’s reported graduation rate is if there is substantial net out migration. In other, words if between the start of the freshman year and the start of the senior year considerably more students transfer out of the school than transfer in. However, for earlier years, we checked this against county level census data and showed that there were only a few percent of schools where net out migration was more than 10%. Moreover, using a three year average, guards against one-time events i.e. a major employer leaving town or a military base closing. In any given class of students, this could cause a large decline in enrollments between the freshman and senior year. However, in the following year, the reduction of students should impact freshmen enrollments as well and promoting power would then measure what percent of these students made it to 12th grade.
After the one-sided attack on Balfanz’s method, she reports that according to figures kept by the state of Iowa, Des Moines’ overall graduation rate was 84 percent. Note that the superintendent does not report for each of the three schools. Is one closer to Balfanz’s numbers?
Sebring does not explain that the sorry state of the graduation statistical data on which she relies is widely recognized and actually prompted Balfanz’s original research paper, Locating the Dropout Crisis.
Balfanz (without citations):
Recent reports reveal, however, that there is much confusion among policymakers and the lay public about the scale and scope of the dropout problem. Researchers from major research institutes that span the political spectrum have shown that federal dropout statistics underestimate the number of students who dropout of high school. Others have shown that state and school level reporting of graduation rates under No Child Left Behind is subject to significant error. One reason for this is
that the most widely used method to calculate graduation rates for NCLB, the graduation rate formula developed by the National Center for Education Statistics (NCES), is ultimately dependent on high schools accurately self reporting how many students dropout. Recent investigations into dropout reporting in New York City and Houston indicate how difficult this is to do.
As a result, there is no ready understanding of how many high schools have high dropout and low graduation rates, where they are concentrated, or the extent to which they dominate the educational opportunities provided to different groups of students. It is not known, for example, the extent to which all states and large cities have significant numbers of high schools that produce large numbers of dropouts, or if the problem is concentrated in a sub-set of states and cities.
Sebring devotes the last 94 words to an explanation that Iowa is now putting in place an information system that “will enable all school districts in Iowa to more accurately track progress and to know exactly which students complete high school and which do not.”
The unspoken ending here is “but in the meantime, I’ve been content to remain clueless about the real extent of my dropout problem, and I’m content to wait until the state education agency gets its act together.” (Without explaining that the state collects its data from the folks at the source - the district, and her central office.)
So What?
When I get to the end of the story, having done a bit of fact checking on the internet of the materials Sebring attacks, I come to the following conclusions:
• Sebring’s high schools graduated between something greater than 50 percent and something less than 85 percent of its students.
• Balfanz has given us a pretty decent idea of the floor. I don’t think it could be much lower. It might be a bit higher, but my guess based on experience is that the individual errors of over and underestimation will tend to wash out.
• We have two reasons to believe Sebring’s ceiling is too high. First, the general critique of state and local numbers noted by Balfanz. Second, while Sebring offered general reasons why Balfanz’s estimate may be too low, she had nothing specific to lower his estimate of her schools. She doesn’t know what the real number is, doesn’t think it’s important, or doesn’t want to tell us. None of these choices leads me to give her estimate high credibility.
Let’s say I was a decision maker - a legislator or the governor, or a citizen of Des Moines, Iowa trying to make sense of the problem - recognizing the need to do something now in the face of less than perfect evidence. How would I look at this fact pattern?
To get some sense of the importance of the problem identified by Balfanz, let’s say he’s off by half, and assume Sebring’s graduation rates are perhaps 75 percent across the three high schools.
So what if some 25 percent of students aren’t graduating?
Let’s say that in 2007, on average a public school student costs on the order of $9000 per year to educate. If we use graduation as a benchmark for the completion of a public school district’s work, and a student drops out after their freshman year - each dropout represents about $900,000 of costs. We all know there’s a vast difference in earning power and other life chances between getting a high school diploma and not. The student’s not a waste, but the district’s effort sure isn’t success. Nine years of a twelve-year education doesn’t yield 3/4 the payoff.
To get a quick sense of the waste, consider two facts. In a 2002 report the U.S. Census Bureau estimated the average annual earnings of a working individual without a high school diploma at $23,400 vs. $30,400 (in 1999 dollars) for a high school graduate. In 2006, the Bureau of Labor statistics reported that 76 percent of high school graduates who did not go on to college were employed, compared to 51 percent of dropouts. And that’s before we get into the costs dropouts impose on society in terms of their higher levels and frequency of involvement with the justice, health and social welfare systems.
I can’t think of any product or service with that kind of a unit price - we’re talking yachts, Italian sports cars, limos, private aircraft - that wastes 25 percent of its production costs. And even if it’s more like 15 percent, I don’t know any area where that’s an acceptable loss rate - at nearly a million dollars a clip.
Indeed, if Sebring were the CEO of a publicly-traded firm and came to her board with the same response and attitude to the Balfanz report, she’d be looking for a new job. It’s not ok here because it’s the taxpayers money, and it’s certainly not ok because it’s children rather than manufactured goods.
Instead of engaging in petty arguments, telling the good readers of the Des Moines Register why the Balfanz report might be flawed and that she is waiting on better data, Superintendent Sebring should be explaining what she is doing to identify the right number today and the individual kids, and what she is doing to turn things around.
If edbizbuzz readers want reasons for why I support NCLB’s unrelenting focus on each kid rather than the average kid, consider this story. I can take the time to identify a CYA exercise that obscures an underlying truth, rather than shedding light on it for the public. Most readers and elected officials can’t. Simple - indeed simplistic - as they are, NCLB’s formula’s do it for them. The Sebring piece suggests why that less than ideal approach is required. It’s pretty much the only way we have to get public school leaders to internalize the true costs of a poor education.
