« IQ Testing in Historical Perspective | Main | Right Answers, Wrong Answers »

It All Depends On...


Dear Diane,

The notion of good medical care as "test prepping" is delightfully bizarre, but maybe no less so than traditional forms of test-prepping? Perhaps Richard Rothstein is correct that it would have a greater impact on test score gaps.

Traditional psychometrics is filled with problems built into the excellent history you’ve offered us, Diane. It presumes that scores fit a curve, and that there is a norm, usually labeled 100 or expressed in percentiles with 50 percent below, 50 percent above grade or age “level”, with a particular cluster toward the center. All very precise.

How come half the kids are still below grade level after years of school improvement, I was once asked? Half were below because that's how we defined "grade level", I explained. If more kids started doing better—via test prep or actual skill—they'd renorm the tests. Scores were just a "promise" about where one stood in the rank ordering of all students who might take this test at this particular point in their school career.

Recent attempts to change the scoring method so that it reflects benchmarks of skill rather than percentiles is a positive step, even if you and I disagree regarding current definitions of basic, proficient, etc. The big plus (or weakness) is that they now rest on human judgment, not mathematical formulas.

Incidentally, in response to some of the critics who worried about my two math examples: they weren't intended to "prove" anything except that we know precious little about why "right" answers appeal to some people more often than to other people. I included that duo precisely because they puzzle me, and why I appreciated readers' efforts to explain them, and Jay Rosner’s replies.

What I’ve discovered is that both the kids who get the answers right and the ones who get them wrong can often give persuasive reasons for their selection. Sometimes—and over the years I’ve collected a host of these—the right answer is, upon closer examination, even clearly wrong. But what they are never wrong about is who gets which ones right! Are you following this?

When asked on a 3rd grade reading test, for example, how the children felt when the trucks came and cut down the trees to widen the highway by their house, only the middle-class kids said "sad"—which was the "right" answer. Many of the others said "excited". I know that's a particularly stupid question, but the division was startling.

Ditto for a familiar IQ question on what to do if you lose a friend's ball. Middle-class kids are more likely to say "buy him a new one" and working-class kids to say "tell the teacher". In fact, the latter are the least likely to tell the teacher, but more likely to believe that was the right answer.

Also interesting is how hard it is to guess whether items come from IQ or reading or math tests. I've tried random items out on audiences and gotten random answers.

Rosner's point, among others, was that if ETS (Educational Testing Service) has historically played around with items to ensure that male scores more properly matched or surpassed female scores, it could do so for race. Yet, he discovered, that the pre-tested items on which blacks more often got answers right were virtually never used—although they existed. The two items described were among that pool.

Democracy requires us to "act as if" the views of every single citizen are of equal importance. Not equally "correct" or "wise"—just equally important. And that each has a "logic" that the community would do well to acknowledge and make sense of.

That's what I rest my case on for democracy writ large, and writ small. I take it for granted, or tried to, that the stupidest questions or answers were important to understand. Doing so did not always lead to new truth. But it amazes me how often it broadened my understanding and that of others in the class. After a while it became possible to "outlaw" the phrase "it's obvious" in class! Jean Piaget’s work helped us see precisely how difficult it was to convince someone else of "the obvious" when they were viewing it from a different perspective of age, experience, temperament.

A youngster, studying a world map in the hall, asked me (shyly) why the West Indies was in the east and the East Indies in the west? It startled me into noticing an "obvious" misnomer stemming from the perspective of the maps we were accustomed to using. Ditto the 5-year-old who corrected me when I asked what the man on the moon might see if he looked “down” at us? She blurted out, “he’d look up!," setting my mind whirling on the concept of up and down!

Children aren't being "cute" when they say surprisingly wise things; they are just being both observant and accustomed to taking their observations seriously. Too often, schooling cures them of both. In contrast, while schooling based on both these qualities allows us to cover less in the here and now, it rests on the conviction that—in the long run—it will cover more. Yes, “less” can just be “less”, contrary to one of my favorite Coalition of Essential Schools principles. It all depends on…..

And that’s where the craft of teaching begins—with “it all depends on…”


A P.S. for future conversation. You and I remain unpersuaded by Murray and Herrnstein, Diane. I'm glad. But that a public bombarded with the apparent intractability of the "achievement gap" might fall back on centuries of racist/classist conclusions seems inevitable.

It's one of the potential unintended consequences of the well-intended battle cry to "close the gap" without more carefully examining what test scores as we know them can and can't tell us, and what other gaps we might start closing.


Dear Debbie and Diane,

I am so glad you address the issue of I.Q. and intelligence, for it remains a central–if sometimes covert–issue in educational policy, practice, and public attitudes.

As for Charles Murray, his work has been so undermined by measurement specialists, statisticians, and others (including nobel laureat economist James Heckman) –hardly Romantic types–that his continued presence in the public sphere says more about the state of our culture than the merits of his argument.

But beyond Murray, the issue of intelligence is important to discuss because, as you say, it’s woven in and out of various school reform efforts. We tend to operate with a pretty narrow definition of intelligence, one shaped by the I.Q. test. Even some of our seemingly less rigid notions–like “multiple intelligences” or “learning styles”–can quickly devolve into narrow notions of ability: “Oh, he’s a kinesthetic learner” or “girls learn best in collaborative settings; boys are individual learners.” We need a much richer discussion and understanding of intelligence than this. I’m glad you’re addressing the topic in Bridging Differences.

Mike Rose

Mike, if you are going to make an appeal to authority it would help if that authority supported your argument. Heckman on IQ:

There is ample evidence from economics and psychology that cognitive ability is a powerful predictor of economic and social outcomes. It is intuitively obvious that cognition is essential in processing information, learning, and in decision making. It is also intuitively obvious that other traits besides raw problem-solving ability matter for success in life.

"The Economics and Psychology of Personality Traits"(Borghans, Duckworth, Heckman, and ter Weel)

Heckman's criticism of the Bell Curve amounted to statistical quibbles with the underlying methodology, but fell far short of a discreditation. The American Psycological Association agrees with most of the points made in the Bell Curve.

Your criticisms should be directed to the entire field of researchers who agree with Murray instead of trying to marginalize Murray. Attack the views not the man.

I love the PS Deb...I hope you enjoyed the video...



Hi Debby,

I can understand how there are differences in the way people interpret questions. But that does not explain the differences between groups in areas such as the ability to read most texts and in the ability to do math. There are only so many factors of 56. They are not ambiguous. They have no relationship to experience. Either a child can figure them out or the child can not. The differences that bother me are of the latter type. We can remove questions that do not test what we want. But from my perspective this is an ancillary issue. I have seen smart kids in the upper grades who do not know that any quantity divided by itself evaluates to one. They were not taught that. A parrot sitting in a class for 12 years would know that if it were taught.

Tom Linehan


So do you think it's because no one ever taught them? They have the brains of parrots? Or possibly that intelligent people can be confused by your phraseology--"evaluates to one?"

My close attention to children's explanations of their answers has had a powerful impact, probably initiated by discovering that one of my three children (and decidedly not the least smart) had problems with both standardized reading tests and with an IQ test given to him by a colleague of mine. She had to stop half-way through because she was intrigued by the fact that he scored worst on the basic "life experience" questions! (I have another son who loves and instinctively does well on any multiple choice test, regardless of subject matter.)

These differences are partially idiocyncratic, but as I discovered in taping 7-8 year olds in Harlem, the different answers also had a lot to do with race and class in ways that said nothing about their capacity for logical or sensible thinking.

I think you are making a good point about why math tests should be easier to make culture-free; but not entirely so, especially perhaps because so much of math these days is told with "story" problems. But also because when we teach math we are doing two things at once: teaching them to think mathematically AND teaching them a specific mathematical language. Which is more important, especially for young children? Given time limitations, which would you put first?



I've always admired your work, but on this I think you are too pessimistic. Why do you think the intelligence issue is still a central issue? In my experienrence, racism comes from our fears not our misreading of stupid tests.

Debbie, when I got stuck teaching ACT prep for a month, I always got a 31 or a 32 on the tests when you would think that I would have always gotten a 35 or a 36 on a test that predicts whether you are prepared for the freshmean year of college. Sometimes I would read the explanations and agree with the testmakers that they had the better answers, but other times I remained convinced that my answer was better. It was no different than my experiences working with editors from New York vs. Oklahoma.

A great bookend to your concept of "it all depends on ..." is found in John Cassidy New York Review of Books analysis of the intellectual roots Obama's economics. Cassidy reviewed Thaler's and Sunstein's Nudge: Improving Decisions About Health, Wealth, and Happiness, arguing that old-fashioned economics used "the most highfalutin statistical techniques" but Obama's mentors borrow from experimental pychology. Cassidy wrote, "If you think you are too smart... then take the last three digits of your cell phone number and add two hundred to it ... and put the letters AD after it." Then answer when Attila the Hun invaded Europe. Unless you are an expert, those distractors will greatly influence your rational judgement.

I was struck by the way these economists learned from their students by performing cognitive experiments on them.

Then I realized why their efforts to "nudge" us to better decisions was so appealing. Every day, a bonus of my job is that I'm involved with so many informal cognitive experiments with my students. Together, we are exploring and reveling in the great human comedy. Even if there was always one "right" answer, I would not enjoy our intellectual journey, with its right and wrong turns, any less. We live in a world where "mental quirks" are "ubiquitous." Why are we investing so much energy into rearguard action to refine tests that are closer to the 19th century? Like Casidy wrote, "exploring the limits of human reason is interesting in its own right" so lets celebrate our human complexity. (which reminds me of a great lesson from the Utne Reader about why the Federal Reserve is studying the use of two dollar bills in strip clubs in Dallas. I love those sorts of lessons because they give me a chance to sing The Doors old song, "When You Are Strange.")


You write: "Democracy requires us to 'act as if' the views of every single citizen are of equal importance. Not equally 'correct' or 'wise'—just equally important."

I might agree with you if everyone put equal effort and care into the formation and refinement of said views. (I might not agree with you, even then--but I'll get to that in a second.)

I have only taught elementary school over a summer. (I have taught middle school and college for longer.) When teaching the younger children, I was struck by their sincerity of mental effort. I asked them to do something, and they would give it their best.

In middle school and onwards, as you know, this breaks up for all sorts of reasons. Some continue to give their best effort to school. Others give up, fully or partially, blatantly or subtly.

A teacher can address this in all sorts of ways--but the problem does not always manifest itself in poor performance. It exists even among the top students. There's a subtle failure of will. Perhaps students figure out that they can get away with less than their best. With that comes a certain bravado: "Look at me--I didn't do any work, and I passed!" (I was guilty of that in college to some degree.)

For that reason we cannot afford to take all views seriously. Nor should we be taken seriously if we say something that we haven't thought out well. One of the greatest favors our teachers and colleagues can do us is to call us on our bluff--to point out not the weaknesses in our argument, but the weakness in our effort.

If a violin student comes to the lesson without having practiced, the teacher can continue to give pointers on technique--but perhaps it would be more productive to have the student use the time as practice time, and give some pointers on practicing.

If we do not do this for each other, then those who put care into their ideas end up serving as janitors for the rest--sweeping up nonsense and fixing broken logic. The others, seeing themselves treated as royalty, have little urge to change. It turns into a reign of thoughtlessness.

Therefore it is our duty not to treat all views as equally important--that is, not until the effort is equal. Now, if everyone did put similar care and effort into ideas, should we then treat them as equally important?

Possibly. I have one reservation. You say, "Democracy requires us to 'act as if' the views of every single citizen are of equal importance." Now, if we have to "act as if," then perhaps we don't believe it? I wonder if you simply meant that we should respond courteously and thoughtfully to each other's ideas. I would agree with that.

Beyond that, we should be honest with ourselves and others. Some ideas or views will seem much more important to us than others, and it would be a mistake to pretend otherwise. We only have time in our lives to address so many ideas. We have to choose the ones we find most compelling.

I want to add to the above that we need knowledge and skill in order to form our ideas well. A person can put out tremendous effort and still arrive at a half-baked idea, if he or she does not possess the necessary background.

So, for all ideas to be equally important, they need not only effort and care, but a strong foundation.

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments