« Right Answers, Wrong Answers | Main | What Good are Tests? »

Scouring for Racial and Cultural Bias in Tests


Dear Diane,

You didn't suggest that good medical care was test-prepping—that was my translation. But it does seem to me that in effect that's not a bad way of looking at it—most standardized tests rest on experiences that don't come to us from school, but from the prepping that life has offered us. It may even account for the interesting story I just read—more next week—on a Chicago study that demonstrates that actual test-prepping may hurt scores!

As to testing itself: Yes, reviewing questions for bias and using judgment to determine "grade level" rather than statistics are important. But we disagree about whether, like judgment in general, we can be "neutral", devoid of politics as well as bias.

Agreed: there are "right and wrong" answers, and you give two examples of such. But I think we both agree that these examples are not at the heart of what we mean by being a "well-educated" person. Therein lies the dilemma. It shows up, we all assume, more on the language arts sections of the test than in disciplines like math and science. Yet one of the first critiques of testing I read was by physicist Banesh Hoffman in "The Tyranny of Testing". He noted that some answers on the SAT physics test were just plain wrong (although ETS insisted that they were still good "discriminators" between smart and less smart test-takers). Too few, he noted, helped him see who was of a "scientific" mindset. In our field, Diane, asking for the "causes" of WWI, or the Civil War, were an obvious effort to overcome this by asking for "reasoning", not just "facts." But in the end they rewarded conventional memory of conventional reasoning, not evidence of historical thinking. I recall that one history question required one to believe (or simply pretend to) that the cause of fascism was the Versailles Treaty. Surely a theory, but a fact?

Scouring for racial and cultural bias—and removing obviously biased items—is good practice. But the bias I'm describing is only possible to detect if one investigates it in the manner I did. What I was "judging" was the reasoning from the evidence that students demonstrated. They occur even on "factual" questions when the facts correlate with certain experiences that children are more or less likely to encounter.

On the NAEP, girls do better in reading; on the SAT, they do not. Who is "right"? Even the decision to rest the SAT on English and math is a decision that has an impact on results. I'm not necessarily condemning this decision, but pointing to the consequences and our ability to read too much into them.

You and I agree in seeing socioeconomic status as a powerful determinant, but logic alone does not get us to this conclusion. Charles Murray builds a powerful counter-argument, which some of our readers find compelling. He argues that, in large measure, the reasons whites and Asians do better than blacks and Native Americans, ditto for Rich vs. Poor, on tests is the same reason they are ahead in the world of wealth and power; it’s due to what’s "inside" their brain. Ditto perhaps for males vs. females?

Surely, beliefs of this sort are not only still around, but they have a long and sustained history in many parts of the world. They have an impact on those who come out ahead and those who come out behind. Living within a culture which takes your "lesser" abilities as the norm, and your signs of intelligence as the exception has a deep and abiding impact. It is one of the reasons I am so opposed to NYC's intention of testing all 5-year-olds for intelligence, and then (of course) reporting the results to teachers and parents. This is a step that must be stopped because, aside from its many pernicious effects, testing of 5-year-olds is notoriously unreliable as most independent testing experts and others affirm. (More on this very soon; along with the story of 160 out of 162 8th graders in the Bronx who refused to take a practice social studies Regents exam on the grounds that this was further wasting their time.)

Maybe there’s a paranoiac streak in my nature that makes me wonder whether it was sheer ignorance that led the authors of the NCLB legislation to require that all testing gaps be closed by 2014. NCLB was passed as much as a "sentiment" as a real vehicle for improvement. But setting the goal at 2014 was naughty. I think it expressed a coalition of the naïve and the clever. NCLB was hailed by many conservatives as a test of the capacity of public education to do its job properly. Setting an impossible goal has opened the doors for a lot of crisis-reorganization plans, including privatizing schooling. It may, in the end, also reinforce old racial and class prejudices. Surely that wasn't Ted Kennedy's idea—and in fact he put his weight behind NCLB precisely to stop worse measures. But in doing so he unleashed something he did not foresee. (He’s on all our minds these days.)



Readers interested in Deborah and Diane's discussion of IQ testing and its educational and social impact might also want to read Mike Rose's article in the May Phi Delta Kappan, "Intelligence, Knowledge, and the Hand/Brain Divide." It is available for free at http://www.pdkintl.org/kappan/k_v89/k0805ros.htm


Thanks for the link to Mike's article. I'm glad he emphasized the role of adults helping young people as they develop.

Back in the day, laboring people played an essential role in intitiating kids into adulthood. My first job as a ditch digger was to crawl through the mud under a house with a flashlight searching for leaks. Intitiations could be challenging (how does a kid react when the co-workers take a dump in your hat?), but they were a tradition designed to help not damage young people (for instance, my surprise was a Bull Snake, who tend to be mean, but not a rattler.) I don't know which was the better preparation for teaching - my phd courswork or "slingin iron in the oil patch." But both were beneficial for the same reason, they built confidence.

Our students and teachers both need all of the assistance we can get from the hand-brain intelligence of working people. Many of our kids need the challenge of vocational training, and all need the wisdom of working people.

I'm the only male in my famaily that can not put his hands on a kid, adjust his hands and feet, and thus help the young person slap a clean base hit or wrap up a take down. And sure enough, my brother the house painter (and occassional brawler) used alternative certification to become an award-winning art teacher and a coach who produced a national champion wrestler. (He also won an award the Most Courteous" teacher) Before I used alternative certification to become a teacher, I was a lobbyist who could explain complex concepts for the "good ol' boys." After I learned to clean up my jokes, I was able to use them in class.

Let's stop prejudging what is the ultimate instructional policy, and invite the full range of personalities into our schools.

And Debbie, I think the recruiting of so many different types of people into education would be an excellent way for adults to mentor the euducational principles that you, Diane, and your readers seek.

Great point about wrong or right answers, Deb. One could even take Diane's own example of an absolutely wrong or right answer:
--"If I asked you who was elected president of the United States in 1960, don’t you think there is a right answer?"-- Change the date to 2000 or 2004 and things get more complicated. Don't they?

That's a very unfortunate example you picked, Mike.

"There is no certainty that Nixon won both Texas and Illinois [which
he would have had to to do win the Electoral College vote]. What is
certain, however, is that massive voter fraud on Kennedy's behalf
occurred in both states. In Texas, Kennedy's margin of victory was
46,000 votes, but Lyndon Johnson's Lone Star state political machine
could easily have provided that number. In Illinois, Kennedy won by a
bare 9,000 votes, and Mayor Daley, who held back Chicago's vote until
late in the evening, provided an extraordinary Cook County margin of
victory of 450,000 votes. No thorough investigation of the massive
irregularities was ever conducted, and partisans of Kennedy and Nixon
still debate the bottom line."


Who won the Civil War?

Or would you prefer, "the War Between the States?" or "the recent unpleasantness?"


I deeply appreciate your raising questions about the translation of test scores into intelligence, especially in the context of the increased use of test scores since NCLB to evaluate the quality of schools.

As a former high school teacher (most recently at the Big Picture school University Prep HS in Detroit) and current employee of a charter school start-up organization (the Henry Ford Learning Institute - hfli.org), I think it's incumbent on those of us who argue that standardized tests are not the best measures of school quality, to develop transparent means of assessing the quality of schools, and the quality of student work that are both rigorous and reliable.

This is important for many reasons, but two of the most important are so teachers engage in a rigorous practice of their craft, focused on identifying and implementing practices (in the broadest sense) that help students develop skills and abilities deemed useful and important; the other is so that schools, as public institutions, can be accountable (in the general sense of public institutions in a democratic society, not in the narrow sense of how that word is used in ed circles as meaning higher test scores) to the public. Think of how the military is accountable for how it spends taxpayer money (Oh, sorry, bad example)...

So, my question is this: what practices do you recommend for assessing the quality of student work, and for assessing the quality of schools? What have your schools done? What are others doing? Where have people been successful in having these alternative means of assessment accepted by political bodies?

I'm sure you've addressed these questions elsewhere - could you point me in the direction of books articles you/others have written on the topic?

Many thanks.

Aaron Wilson-Ahlstrom

What's your point, John?


I had been growing increasingly uneasy about the tone of the debate on intelligence, and there I go. I throw a low blow.

I apologize.

The appropriate part of my jab is this. Your approach seems much more appropriate for 1908 than 2008.

My views are very consistent with those of the cognitive scientist George Lakoff who was again on NPR today. He argues that today's politics, especially those of the Democrats, are grounded in an 18th century, not a 21st century view of pyschology. To paraphrase, the Educational "Mind" needs to move away from the uni-dimensional view of the brain and recognize that cognitive science is showing us a more complex view of human learning. I think we should celebrate its richness.

Your approach seems much more appropriate for 1908 than 2008.

Care to elaborate?

I haven't advocated the misuse of IQ tests to rigidly track students.

I haven't advocated the use of IQ tests as a way for educators to excuse their inability to educate low-IW students.

In fact, I haven't advocated the use of IQ tests at all for educational purposes.

All I have advocated for is the need for better instruction targeted specifically to low-achieving students and an abandonment of educational policies based on the unsupported belief that environmental/SES interventions are going to have a significant effect on student achievement.

My views are very consistent with those of the cognitive scientist George Lakoff

Lakoff's opinions (not theories sonce he has no ressearch base and speaks with no authority) do little to inform education policies. Though I suppose such views could form the basis of a fallback position upon which a fighting retreat can be effected as the genome is eventually cracked and the 100% cultural/environemental theory of cognition and discredited in order to maintain the status quo. And I suspect that it will be to the continued detriment to students.

I think we should celebrate its richness.

Why don't we postpone the celeebration until we've actually learned how to teach students better than we currently do.

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments