« The Regulated Lives of Teachers | Main | Democracy Rests on Disagreements »

No Foolproof Measures of Success


Dear Diane,

Did you read The New York Times Magazine piece called "Do We Really Know What Makes Us Healthy?" by Gary Taubes? Or the follow-up on Oct. 9 in the Times Science Section by John Tierney?

Medicine (and nutrition) have all the odds in their favor vs. education when it comes to being "scientific". There's a lot less disagreement about what constitutes good health, for one thing. Politics—in the best and worst senses—is less intimately tied to medicine. It's easier to have placebos and random samples. And it's easier to track patients for long enough to assess side effects.

John Tierney takes up the "consensus" claims about low-fat diets, and the nutrition end of Taubes' work. He describes how easy it is for an interesting hypothesis (high fat, bad for heart), with the right "political" support to become a consensus, and thus the continued existence of fads in medicine and nutrition. Ditto for schooling, Diane. And—more another time—the "consensus" around the current "science" of reading is a case in point. At least in medicine it doesn't get worked into federal law!

Author Taubes concludes that "we end up having to fall back on the following guidelines when it comes to scientific research about medicine". (What follows is my summary) (1) Look for all other possible explanations for the data. (2) Assume the first reported association is incorrect or meaningless—be skeptical when it first hits the news. (3) If the correlation appears in many studies and populations but is small (in the range of 10s of percents), continue to doubt it. (4) If the correlation involves some aspect of human behavior then question its validity. (5) "The best advice is to keep in mind the law of unintended consequences." There's too much that either can't be measured or in which the measurement itself is subjective—even if it can be coded. The principal investigator of a famous large-scale Nurse's Health Study concluded: "I'm back to the place where I doubt everything."

I say all this, Diane, because that's where I end up, too—and I actually did not begin there—about almost all school-related data. Because above and beyond all the reasons Taubes gives above, there is inevitably more bias—meaning individual values—involved in education research, and more political pressure on schools to comply. Teachers and principals, as you noted, don't have the freedom professors do. In part it's also because I have never met two kids who responded the same way to anything—even if on a coded response sheet it might look as though they do.

Yet one cannot fall back on nihilism; so one reaches some conclusions—makes one's best guess (judgment) and leaves the door open for those with other conclusions! It's still a one-on-one kind of diagnosis. One even encourages, as in medicine, continued research of less likely minority views.

It's not all that much different than what we have to do in any field which is bigger than our own anecdotal evidence. Like Iraq, or human-induced global warming, etc. We simply do not and cannot wait for certainty. (So, as laymen, we go along with the experts we most trust.)

But it's also the reason I keep the door open to the idea of being wrong even as I act as vigorously and persuasively as I can on the assumption I'm right! That's why I like kids to hear knowledgeable and expert adults disagreeing. My friend Brenda tells me they have a rule at home: we never argue about something that can be settled by looking it up (Googling these days). I think that's a good rule, although that assumes we can know for sure when it's "that kind" of argument, and besides there are some advantages in kicking a fact around for a while before looking it up.

Democracy rests on disagreements—if there weren't any we wouldn't need it. If there were mostly wrong and right answers to life's dilemmas, we'd just be able to choose our rulers by a standardized test.

But there are no such tests, nor probably no such foolproof graduation or attendance data, or measures of what we all would even agree is "success". But that doesn't make trying to figure out how best to assess this or that less useful. We just need a much healthier sense of tentativeness about our assessments—an open mind.

As with the baby-sitter I described in my last letter, this is in part why I'm for as FEW, not as many, "no-no's" as we can get away with. Let's not "make a law against it" unless pressed to the wall. But let's collect all kinds of data and expose all those interested to a good debate about the data, leading in turn to further data, and more debate. We don't always need to end it with a vote—or a rule.

I recently attended a Mission Hill Board meeting (what an amazing event it is). On the agenda was a discussion of what kind of data we wanted to collect about the kids who graduate from our school. What kind we collect will, after all, affect the kind of answers available to us. A former teacher who is writing his doctoral dissertation (in Wisconsin) has agreed to sort through the last 13 years of data. That could be the assigned service task for every doctoral student!



Debbie, I also loved "Do We really Know What Makes Us Healthy" Gary Taubes, NYT Magazine, Set 16, 07, but I had a different take. From my perspective as an inner city high school teacher, I saw it as an explanation of why magnet schools, low poverty schools, and high poverty neighborhood schools aren't comparable.

I’d argue that the Gates’ and others who seek to replicate best practices have ignored the common sense reality that elementary schools are different from high schools, low poverty schools are different from high poverty schools, and high poverty magnet schools are fundamentally different from high poverty neighborhood schools. (Moreover, Fordham research has reinforced that common sense conclusion, showing that sustaining best practices and turning around dysfunctional schools are two completely different things.)

In education, we call it "creaming," as the more privileged schools "cream" off the more motivated, less traumatized students. This leaves inner city comprehensive schools with a greater concentration of troubled kids.

Taubes explains the problems with observational epidemiology using the H.R.T. or Harvard Nurses Study which said that estrogen is effective in reducing heart attacks. This error, which damaged tens of thousands of women, came from a methodology that was far more sophisticated than the methodology of education researchers who copied it. Even so, they overlooked several biases that should have been obvious:
1. "The bias of healthy users,"
2. "The bias of compliance"
3. "The bias of ‘prescribers’" and
4. Not looking at data over time and neglecting the obvious cycles of life.

Even in retrospect, its hard to understand how Harvard researchers could have argued that by studying only one profession, they were minimizing the effects of socio-economic factors. It never occurred to them that some nurses are single mothers and others marry doctors? Use of estrogen was associated with reduced heart attacks, but it was it was also associated with lower homicide risks. They didn’t wonder why highly compliant nurses who receiving placebos had a reduction of heart attacks by a factor of 2 when compared to noncompliant users of placebos? And they didn’t take near enough time to "think like a doctor" when he or she is prescribing.

As it turned out, both the H.R.T. and subsequent studies was accurate on the key issue. When younger nurses were prescribed estrogen, it was beneficial and risks were small. When older nurses continued to take estrogen, however, heart attacks and cancer were increased. (Metaphorically speaking, this is consistent with the argument [also made recently in the Times Magazine] that data-driven accountability is like chemotherapy. If kept up for too long, it becomes poison.)

Applying this logic to education, research regarding best practices in the Gates’ "Small Schools," many charters like KIPP, pilot schools, and other magnets, is irrelevant for whole system reform unless they factor out the effects of "creaming." Moreover, they couldn’t compensate for the last two biases without making much more of an effort to understand the teachers’ perspective i.e. the "prescriber effect" as they deal with students going through their teenage years. As has been explained in EDDRA, elementary teachers, especially in affluent schools, might be able to make a game out of standardized testing. By high school, however, teens can’t be tricked into thinking that high stakes test prep is good for them. (Of course, we would question whether the testing is appropriate for elementary school, but my question is why pro-NCLB researchers have overlooked the differences between little kids and teenagers.)

And if they were following accepted social science methodology, those disclaimers would be placed in a prominent position in their reports.

This reminds me of one of the best nuggets in the 17th Bracey Report. The ACT study which indicated "to know NCLB is to love it," presented a key methodological step in PR materials not in the report! Call me old-fashioned, but I still think that social science methodology must be presented in a transparent manner.

Thanks, John. These are all very good points. They make it important not only to be transparant but to distinguish between "suggestive" data and "definitive" data. The former provokes further research and questions, the latter fools us.

You've noted some factors to be taken into account that I hadn't thought of, and provoked me to think further. e.g. the different ways young vs older students approach the testing task. Again, thanks.


p.s. Just rereading Diane's previous column and realizing how dificult it has been for teachers to conribute to this debate given the nearly two century old tradition of disrespect for teachers. It's not so long ago that women teacher, as an example, were often not allowed to marry or have kids. Teachers joining the debate was probably a clear and loud no-no. How can we get more of them into it??

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments