« This Is Not Good Education | Main | Scare Tactics in NYC »

Data, 'Crises', and the Way We Interpret Them


Dear Diane,

You're right! There is data and then there are various ways to interpret it. The NAEP data is data. But given that the average reader—such as me—doesn't know a lot about how the scores were derived, making sense of it requires interpretation. For example: (1) what kind of questions are on the tests? (2) how great are the differences in right/wrong answers that separate the benchmarks? and above all (3) how valid are the categories proficient, advanced? Some of this requires statistical sophistication, some just more information than is readily available. Most of the time most of us are satisfied if we feel confidence in those that made these decisions to start with. We leave it there.

There's no question that NAEP has helped us to recognize the absurdity of all the other scores the states are using. That's your main point, and I agree with it.

I feel the same way about the international comparisons. Gerald Bracey almost yearly does a report demonstrating how many ways you can interpret the same data, particularly re. math. I just read a new critique from Business Week by Vivek Wadhwa in which he makes the claim that the U.S. looks very good in science and literacy, and pretty good in math! He goes through it pretty thoroughly and has the credentials to do so. But I grabbed onto it, I know, because I was looking for data to support my claim that the "crisis" around education is being drummed up around the wrong issues. That's what I meant about the human tendency to rely on experts one trusts, or who say what one wants to hear.

There may be a crisis, but how we define it will determine our approach to overcoming it. Controlling "the story" is what money helps us do—lots of think tanks, media exposure makes a story seem more reasonable. I think the crisis facing us is one about trust, mutual respect, human relationships on one hand, and around understanding the complexity of the democratic idea, and the role of uncertainty on the other. It requires being exposed to conflicting views and data and trying to sort it out. Both crises, for example, require a better understanding of statistics vs. algebra (choices need to be made). A lot more understanding about the nature of evidence in various fields, and greater respect for our right to make sense of the world is not the likely outcome of current reforms. Too many schools have traditionally robbed us of our "common sense" and more of the same may further weaken democracy. There are always trade-offs.

I spent 48 hours in Switzerland. Seeing schools and even "data" from different cultural norms is always startling. They are proud of their schools—but of course admit that immigrant (foreign?) children are doing badly. So they are trying to do something about it. Surprises? The kindergarten I visited—in one of their best multi-cultural schools—was colorful and lovely and pristine! No clay, paint, sand, water and minimal blocks—although also not "academic". Why? Because at the end of each day the rooms must be left pristine so the cleaning staff can get through the whole building in three hours. Odd way to decide priorities? In general, I found the school delightful and thoughtfully organized. I met two boys from Seattle in the immigrant integration class! Much of the education language is familiar—including our reform language. At the moment they pride themselves on not having bought into the testing mania. There are, they say, no national tests. BUT… at the age of around 14 kids do take exams that determine whether they go to university or secondary technical schools! A grand SAT? Their struggle over what it means to be Swiss reminded me of how differently we define "being American"—and yet the similarities are striking as well. Many of our current debates about outsiders echo those they are having and result in similar schooling issues. Everybody I spoke to—I also attended two university education courses—pushed my thinking. It was an exciting visit. And Zurich is beautiful. (The ceremony at which I received an award was also thrilling for me.)

Diane, in the end, do you think it would be possible to have the kind of national curriculum that would only be "suggestive", without NAEP data becoming an even fiercer and tighter test-monitored curriculum? Why not many suggested ones? The litmus test to me is whether we're willing to do any tests on a sampled periodic basis. Note the "comment" from the anonymous Federal bureaucrat who would like the results sent home to every parent—"Uncle Sam's report on your child".

Because I believe that the crisis we're in cannot be measured by multiple-choice or short-answer tests I opt for measures that reduce the risk of focusing on tests and one-size-fits-all solutions. One picks one's risks. Where you and I disagree tends to involve our greater or lesser fears of centralization of power?



Deborah/Diane, I came across this post when searching for responses to my BusinessWeek article that would provide me further insight into the issue. Why is it that academics, political and business leaders all say the same things about the U.S. falling behind in education and how this is a threat to our long term competitiveness, when the data seem to show no such thing?

The research I wrote about was completed by Hal Salzman and Lindsay Lowell --

My own research (http://www.globalizationresearch.com) has been in consistent with these findings. Most of the immigrants I know agree that the U.S. is ahead by far in almost every aspect of its education system.

Can anyone refute the Salzman/Lowell study? If so, I would love to read about this. If we can't refute this, then I think we need to get our leaders to rethink their national priorities.

My fear is that the U.S. is fixing the wrong problems and risks losing its global competitive edge.

Vivek Wadhwa


Your last post is a wonderful "two fer." First you hit the issue on the head, "the crisis that is facing us is one about trust, mutual respect, and human relationships."

Then you identified the best "litmus test" I've heard, whether we are "willing to do any test on a sampled periodic basis."

Both of your comments trace back to the fundamanental "voice of American democracy" that is not too certain that it is right and respects the views of others.


Deborah - I'm not convinced there is a "crisis" in "trust, mutual respect, and human relationships," at least one that cannot be understood without further elaboration of the problem. These terms are buzz words in education
that have meaning for those with long teaching or enlightened administrative experience. For the rest of the world, they sound somewhat fuzzy in comparison with the manufactured crisis in test scores, basic skills, and content knowledge. If we are to combat the manufactured crisis in terms its believers appreciate, the ideas behind trust, mutual respect, and human relationships must be made rigorous and, at least in appearance, rational and quantifiable. Perhaps a good place to begin is to understand how statistics can help resolve this real crisis.

I guess I was focusing in my thoughts about how easy it is to con people with statistics--given their virtually nonexistent statistical understanding. To really understand statistics would take a revolution in math education! Tests are based on statistics--measurement error is a statistical concept, "the odds" involved in determined validity and reliability not to mention the concept of grade level are based on statistics (or was), even how items are chosen is a statistical concept. Ditto for comparing past and future data, and much more.

But that misses your point, I think. At Mission Hill--Gr Kgtn-8th grade--we developed an assessment of kids as readers on a reading scale (one to five) and created a graph depicting where each grade was on such a scale. The school was small enough to do it child-by-child (no names). It was based on annual tape recordings of children's reading followed by an interview which could be reassessed by any doubting Tom. But it would probably still not be considered "objective" or precise enough by many! And our commitment was that all kids would by 4th grade be at 4 or 5; we did fail on occasion.

But even the best school reviews by external reviewers while falling back on numbers, is more valuable as a thoughtful summary of the judgments of respected expert/colleagues.

Maybe it's as Utopian as to imagine us coming up with a convincing numerical scale for everyone--versus a particular population of families--as convincing everyone that what we have now looks like "the real hard data" and is, in fact, the softest indirect data around!

One way to "check" up on folks might be to include some form of standard tests--as in fact Mission Hill portfolios always did, and occasional standardized tests, and occasional sampling of standardized items. The latter of these to be used to judge schools, not kids.

But then we'd need a public that understood sampling better???? But in fact, democracy IS a form of accountability by numbers, although it also rests on judgments. We know it's not always (often?) to my liking, but it's in defense of the democratic idea that I believe schools are primarily all about.

Thanks for raising this, Tom. I'll try to transfer this to the website at some point!


p.s. When all the best words get jargonized I don't know what we do???


You are completely correct about the subjective nature of education and trying to develop assessments that are completely objective is difficult at best. Your disdain at the “test prep” mentality is well supported. Too many teachers decide that the solution to poor test scores is to give the kids more and more assessments so that they can learn how to take tests instead of focusing on teaching the material/knowledge/understanding that children will need to be successful in our world. Because we as a society can not agree about what a quality education should encompass, we end up with what we have now; a poor education that focuses too strongly on “test taking strategies.”

It is a tragedy that these rather thin reading and math tests drive our schools instead of a quality education. But this is not the fault of the tests but a fault of our school system (and us) that allows this to happen.

So can we develop quality tests? -Absolutely yes. Do we have them now? –Absolutely not.

What we need is a consensus about what constitutes a quality education, first. And then and only then, develop quality assessments to see if our children are learning what we think they should.

Do you think it possible for our country to articulate what a quality education should encompass?

Erin Johnson

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments