« A new federal role | Main | Should Data Matter? »

Campbell's Law and testing


Dear Diane,

I've also been pondering how we could resolve some of our disagreements about national testing—without necessarily resolving our differences on curriculum! Thanks for trying to word it succinctly (less than 500 words!). My response is twice as long, alas. But I wanted to approach it in a way that helped me think through why your solution (a national curriculum with a low-stakes test) doesn't work for me.

We're both distressed—to put it mildly—about the misleading misinformation we're fed about one or another school system's successes and failures. The following data, which I ran across recently, is an example of such mystification.

**Only 33 percent of Swedish 4th graders would meet NAEP's "proficiency or better" standards. (NAEP is the low-stakes National Assessment of Educational Proficiency test designed many years ago to collect data for the federal government, and increasingly being proposed for more high-stakes purposes.)

**Only half of Singapore's 8th graders would be labeled "proficient or better" if measured by NAEP.

These are two puzzlers since Swedish 4th graders and Singapore 8th graders are # 1 internationally. Here's another:

**While American 4th grade kids rank 3rd among 26 nations in science, only 29% measured "proficient or better" on NAEP.

These puzzling statistics look tame compared to the even wilder disparities we get when we compare kids in the nation's 50 states in math or literacy scores. What counts as advanced in some states barely makes basic in another. Even if these kinds of tests were semi-accurate at measuring reading or science achievement (which I have more doubts about than you do) they clearly aren't measuring the same thing. I can see why uniformity might have appeal.

Lee Shulman, in an essay written just a week ago for Carnegie Corp., echoes our concern. He evokes Campbell's Law: "The more any quantitative social indicator is used for social decisionmaking, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." No mention of "rewards and punishments" you'll note.

Something else worried Campbell. And Shulman.

You suggest that we could invent a measurement tool that wouldn't be subject to corruption or ideology. As long as it required agreement about what all kids are expected to learn, then I think the answer is "not likely" on either score. Maybe it could were it restricted to "basic" reading, writing and arithmetic standards. But if we're trying to better understand how our practices impact on the exercise of judgment—I do not think a national test based on a standardized national curriculum will do.

In my idle driving hours I imagine NY State announcing that, given the health and obesity crisis, a panel of experts recommend that we simplify phys ed, requiring high school graduates (unless physically handicapped) to learn to run a mile in 4 minutes (Advanced), 3 (Proficient), and so on. No high stakes attached. If every kid is tested I assume parents, schools, physical ed teachers, and above all kids would want to know how they did, even as they grumbled. Some would privately declare it impossible, but would wait for some politically more naive coach to mention it (like the little boy in The Emperor's Clothes). Others would view news stories about coaches who met their targets with skepticism. (One was a citywide sports magnet.) Schools might ask for more practice time by dropping football and basketball, which weren't being tested. Others would propose a similar test in all sports. Some would find reasons to exclude more kids based on new definitions of what constituted a physical handicap, some kids would try breaking their own legs, and some would just laugh. Some schools or coaches would get new stopwatches designed to meet the new standards. Those who noted that it seemed to have no effect on obesity—in fact it led to more snacking—would be ignored. Missing? Any lively and robust discussion at each school site on what role the school could play in defining and producing healthier children.

Furthermore, I'm not sure if many of the current testing maniacs would see merit in it unless the results were made public and comparisons available—in ways that willy-nilly create their own sanctions. I suspect that those that favor assessment primarily as a way of controlling—not for the purposes of gaining truthful information upon which to better discuss schooling practices—would create rank orders, offer rewards privately, etc. Some track coaches in the scenario above might applaud it because it provided a larger pool of potential runners to compete in the Olympics. Some might like it because only when we're in a crisis mode do we allow ourselves to do the "unthinkable"—like dismantling a long tradition of public education on the grounds that it is beyond saving.

Sampled Federal studies—ala the old NAEP, investments in research that matched local concerns, a new 8-year type study of the long-term impact of various different schools—these we could agree on, Diane.

It's amusing, sort of, that Harvard's Eleanor Duckworth and I were invited to China to discuss promising educational reforms by officials who think that the Chinese exam-based tradition is stifling the kind of creativity and ingenuity that they believe has made America technologically and scientifically so outstanding. They are looking for ways to produce well-educated youth whose ambitions are focused on more than getting the right answers on exams.* They might also discover that creativity and ingenuity if spread too widely have repercussions for democracy as well. Happily you and I agree about that.


*See Daryl Smith and Gwen Garrison (Teachers College Record) "The Impending Loss of Talent: An Exploratory Study Challenging Assumptions About Testing and Merit".


Just a tiny correction in Deborah Meier's letter. The piece by Lee Shulman that she mentions was not written for the Carnegie Corporation, but was instead the introduction to a Carnegie Foundation commentary distributed as Carnegie Perspectives by us. Here is the link.

Do the poor want to be poor
Do you think the poor of DC want to live like that
Living in apt. buildings pushed beside each other with the windows open, looking at other poor people standing in they’re windows, watching they’re sons and daughters dealing drugs, but trying they’re best to look the other way, because some of that money will come into the house, for food and clothes, I observed this in the SE tenements of DC, youngsters standing on the corners, kids I knew when they were babies, now I felt scared to glance they’re way, for fear they might hurt me, do you think they enjoy feeling that this life is all they will ever have, running from the police, from themselves, scared to stand up walk away from the hell that weakness has rain down on them, do they ever stop questioning themselves about why they are there in that dark existence, see I know, I use to be one of those kids standing on the corner, but it was different back then, we didn’t love hurting others, to the point we'd laugh as they fall to they’re death, today youngsters have no since of belonging, for some reason they see it as control, again to a point I was that same way, seeing things that weren’t there, my mind was totally not directing me, it hurts to say, but I to a point was just like those I saw sitting around in hopelessness, I hated what I saw,it was almost a mirror image Of my yesterdays, struggling out there in a world that seems to close, they’re eyes when ever things they don’t understand, I asked you do you think they like being poor, as someone raised in those same street of DC abused, raped, always feeling that no one cared but with the same breath I knew if I got my butt up and tried to find a different way things would change, and they did once I listened to those whispers that had been hounding me in my ear, they did but after years of struggling trying stay alive, now I know why I wanted to stay alive to do what I do today, try to help others I write poetry to share with the weak to let them know there is another way, I ask again do you think DCs under privilege like being poor..
My sister said that as we drove slowly pass all of this madness, children raising children, fathers looking the other way as that child he fathered passed by, laughing at the young girls as if they are the stupid ones for having the kids, we were there to pick up 2 people to do work for my sister in her rental house, as they got into the car it was obvious the woman was on some kind of drugs, she just had that look, as we drove off, the car was silent everyone wanting to get this over with, my sister for her reasons, them for there's, finally the silence was broken as the woman asked if we'd stop so she could get some candy, the man yelled you been eating lots of candy hope you are not pregnant, she just looked down, we stopped she got out bought some candy, got back into the car, you could tell by her look she didn't know if she was but was scared she might be, as soon as we got to the house I went where she went as my brain ran wild with what to say to her, I knew I had to say it quiet and I did, I asked her are you pregnant, she said I think so, so the next question to ask is do you have other children, she said yes 8 kids in all, she laughed but not out loud, it was a nervous laugh, and she was at that moment embarrassement , and asked me if I had kids, I said #1 child and told her I didn't know how people have 8 and 9 kids, I told her my mom had 11 -3 died and she struggled all the way to her death at a early age,
she seemed surprised that my sister and me only had 3 kids between us, I told her having kids today is not easy, life is to hard, and dangerous, I felt she was a prime candidate to hear my story of addiction and abuse so I told her about me, by the time I was finished she confessed her addiction, and said she wanted to go into rehab. I told her she was the only one that could do it for her no one else, at lease she spoke of rehab. I knew it would be a hard road for her , I just listened to her as she spoke to me smoking a cigarette knowing 1st. she should find out if she was pregnant, finally she told me her children, all of them had been taken, another reaction of shame, this is all I found as I drove the streets of Washington, I wanted to help them, I felt for them, but I know by experience there is nothing I can do it is up to them to help them selves, there was a threat to life and limb in DC that made me feel unsure of me, I didn't like that, because I use to be a part of that presence, the addiction, the selling of drugs, I never abused others but myself I abused badly I left there with the same Question I asked my self as I drove thru my home town( Do the Poor of DC want to be poor)

Thanks Gay,
The Carnegie Foundation's Pespective is a wonderful little magazine, and indeed I had it right there by me when I mis-cited it. Lee Shulman's work over the years at the Carnegie Foundation has added so much to my life.

And Patricia--your words speak for themselves, with a resounding "no":, they do not.


Here's a suggestion from former US Secretary of Education Rod Paige for instantly eliminating the achievement gap. Stop the testing in US public schools. That's right boys and girls, if we stop the testing, the achievement gap will magically disappear. No, it won’t! If we pretend for a long enough period of time there's no problem or we ignore a known problem (public school administrators are notorious for both because they see their main function as PR, keeping everyone "happy"), then maybe, just maybe, it will go away. No, it won't! The folks from Fair Test in Cambridge refuse to accept the idea that testing has identified significant differences in achievement of cohorts of students. The achievement gap has turned out to be the civil rights issue of the 21st century...because it's been uncovered. The first step in solving a problem is acknowledging a problem exists - and it does. Our poor/minority students are performing below their White and Asian peers in schools from coast to coast. We now know this because we’ve tested for it. Because we now have this information we can finally start addressing it.

Deb, I really don't consider myself a "testing maniac" but I do believe in employing tests to figure out who has learned something and who has not. From the tests I can apply the appropriate remediation(s) where needed. It's the most objective, quantitative mechanism I know of for determining whether learning has occurred. So called authentic assessments like portfolios are too easily compromised.

FYI: The Massachusetts Department of Education does not announce the MCAS results. They make the results available on their web-site. Radio, television, the Globe and the Herald are the ones responsible for publicizing and comparing the results.

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments