Let's Explore How NAEP Could be Better Used
Thanks for being there at the venerable old Julia Richman high school building last week, Diane. So many old friends, some new ones, and a nice net profit for FairTest—the little David facing the Harcourts, McGraws, ETS's, et al Goliaths that sponsored the event. Actually, Diane, FairTest is not anti-test, but skeptical about the instruments used for mass standardized testing and, above all, the uses to which they are put. The same publishers that once warned us against any test prep, other than the practice tests they sent out with the real test, now probably make as much in preparing for tests as testing. They once said that it destroyed the psychometric reliability and validity of the data. Then they discovered that it was either (1) too hard to enforce and/or (2) that test-prepping was too lucrative a market to overlook.
It was only in the middle of my acceptance remarks last week that I realized that my son, Nick, and his father had polar opposite test responses. Nick always found ingenious ways to read test questions and interpret test directions that lowered his scores and Fred found equally ingenious ways to answer them right. Fred was good at the game of testing, but not schooling (he never got a high school diploma, but got into the University of Chicago via testing); Nick reversed it. Son Roger loved standardized tests and managed to avoid doing much schoolwork because he was so good at acing them. (The anecdote that I told, which those in the back like you may not have heard, was Nick’s insistence on covering up the reading passages before answering the questions because he considered it cheating to do otherwise.)
This story posted in an MSP working group by one Eric Jakobbson on June 5th is a delight and a reminder:
In the book of Richard Feynman's letters compiled by his daughter, she tells the story of being told by her mathematics teacher that a particular way of solving a problem that her father showed her, was in fact wrong. So Feynman and his daughter went to see the teacher, who proceeded to lecture Feynman on the mathematics. Feynman then revealed that he was a world-famous theoretical physicist, at which point the teacher was quite embarrassed. But the Feynman letter for which this was the context was the letter of apology that Feynman later sent the teacher, for invoking his stature to intimidate the teacher, rather than sticking to the mathematics to clear up misunderstandings.
Upon discovering the oddities of ”statistics, damn statistics,” I’ve enjoyed noticing every day how the media handles statistical information. Graphs, tables, etc. It takes a lot of perusal to be sure one gets them right. For example, if one graph bar looks enormously higher than the next one, it may mean very little because their starting point distorts the difference. And on and on. The headlines about our declining rate of unemployment offer another instance, and I’ll bet a large portion of the lay public confuses the declining rate of increase with a decline in unemployment. Probably if mathematics is to serve democracy (not just those going into mathematical professions) we need a drastic overhaul of how we teach and what we teach.
Exercising good judgment about complex matters—which is at the heart of education for democracy—includes knowing when to trust and who to trust since our expertise in any particular field is bound to be limited. There’s no escaping this, but it’s surely a serious weakness of democracy. Except for the fact that there’s no easy solution to it that isn’t worse. But it also requires knowing enough about some basic skills and subjects to apply good sense to the reported data or narrative account.
If we understood standardized testing better we’d know, as Daniel Koretz et al remind us in the New York Daily News piece you quote, that shifts in scores cannot and should not follow the bizarre patterns we’re seeing. Two tests of reading are only as valid as they are alike. If one test produces higher scores than another, one or both are misleading—defining reading in seriously different manners, being prepared for it in ways that distort the product, or plainly being cheated on. I claim all three are widespread. The higher the stakes, the greater will all three thrive.
Are there alternatives? This week at my 60th high school reunion, the old friends at my tables, while agreeing with me about tests, insisted that there was no alternative. (Disclosure: They were all good testers.) But there are alternatives. Let’s explore how NAEP could be better used, Diane, and also how schools and professional associations can help school communities—parents and teachers above all—better zero in on local information, down to “how’s my kid doing?” There are schools in NYC and nationwide that have solved the latter, including schools whose founders were in the room the other night—who started small and extraordinary high schools in the early '90s. These include the schools in Julia Richman where we met (Vanguard, International, and Urban Academy), and the schools started to take in kids otherwise served by Julia Richman, including Mary Butz’s The Manhattan Village Academy and Sylvia Rabiner’s Landmark. And many more. Our elaborate alternative to standardized testing is both more educative and more revealing, as a group of renowned psychometricians testified to 10 years ago in a remarkable document directed to the state Department of Education.
P.S. The “civil rights” trio of Klein, Sharpton, and Gingrich is falling apart—with Gingrich calling Sonia Sotomayor a “racist” because she thinks being Latino has been valuable to her judgment.