« Lies, Damn Lies, and Statistics | Main | The Obama Agenda is the GOP Agenda »

Let's Explore How NAEP Could be Better Used


Dear Diane,

Thanks for being there at the venerable old Julia Richman high school building last week, Diane. So many old friends, some new ones, and a nice net profit for FairTest—the little David facing the Harcourts, McGraws, ETS's, et al Goliaths that sponsored the event. Actually, Diane, FairTest is not anti-test, but skeptical about the instruments used for mass standardized testing and, above all, the uses to which they are put. The same publishers that once warned us against any test prep, other than the practice tests they sent out with the real test, now probably make as much in preparing for tests as testing. They once said that it destroyed the psychometric reliability and validity of the data. Then they discovered that it was either (1) too hard to enforce and/or (2) that test-prepping was too lucrative a market to overlook.

It was only in the middle of my acceptance remarks last week that I realized that my son, Nick, and his father had polar opposite test responses. Nick always found ingenious ways to read test questions and interpret test directions that lowered his scores and Fred found equally ingenious ways to answer them right. Fred was good at the game of testing, but not schooling (he never got a high school diploma, but got into the University of Chicago via testing); Nick reversed it. Son Roger loved standardized tests and managed to avoid doing much schoolwork because he was so good at acing them. (The anecdote that I told, which those in the back like you may not have heard, was Nick’s insistence on covering up the reading passages before answering the questions because he considered it cheating to do otherwise.)

This story posted in an MSP working group by one Eric Jakobbson on June 5th is a delight and a reminder:

In the book of Richard Feynman's letters compiled by his daughter, she tells the story of being told by her mathematics teacher that a particular way of solving a problem that her father showed her, was in fact wrong. So Feynman and his daughter went to see the teacher, who proceeded to lecture Feynman on the mathematics. Feynman then revealed that he was a world-famous theoretical physicist, at which point the teacher was quite embarrassed. But the Feynman letter for which this was the context was the letter of apology that Feynman later sent the teacher, for invoking his stature to intimidate the teacher, rather than sticking to the mathematics to clear up misunderstandings.

Upon discovering the oddities of ”statistics, damn statistics,” I’ve enjoyed noticing every day how the media handles statistical information. Graphs, tables, etc. It takes a lot of perusal to be sure one gets them right. For example, if one graph bar looks enormously higher than the next one, it may mean very little because their starting point distorts the difference. And on and on. The headlines about our declining rate of unemployment offer another instance, and I’ll bet a large portion of the lay public confuses the declining rate of increase with a decline in unemployment. Probably if mathematics is to serve democracy (not just those going into mathematical professions) we need a drastic overhaul of how we teach and what we teach.

Exercising good judgment about complex matters—which is at the heart of education for democracy—includes knowing when to trust and who to trust since our expertise in any particular field is bound to be limited. There’s no escaping this, but it’s surely a serious weakness of democracy. Except for the fact that there’s no easy solution to it that isn’t worse. But it also requires knowing enough about some basic skills and subjects to apply good sense to the reported data or narrative account.

If we understood standardized testing better we’d know, as Daniel Koretz et al remind us in the New York Daily News piece you quote, that shifts in scores cannot and should not follow the bizarre patterns we’re seeing. Two tests of reading are only as valid as they are alike. If one test produces higher scores than another, one or both are misleading—defining reading in seriously different manners, being prepared for it in ways that distort the product, or plainly being cheated on. I claim all three are widespread. The higher the stakes, the greater will all three thrive.

Are there alternatives? This week at my 60th high school reunion, the old friends at my tables, while agreeing with me about tests, insisted that there was no alternative. (Disclosure: They were all good testers.) But there are alternatives. Let’s explore how NAEP could be better used, Diane, and also how schools and professional associations can help school communities—parents and teachers above all—better zero in on local information, down to “how’s my kid doing?” There are schools in NYC and nationwide that have solved the latter, including schools whose founders were in the room the other night—who started small and extraordinary high schools in the early '90s. These include the schools in Julia Richman where we met (Vanguard, International, and Urban Academy), and the schools started to take in kids otherwise served by Julia Richman, including Mary Butz’s The Manhattan Village Academy and Sylvia Rabiner’s Landmark. And many more. Our elaborate alternative to standardized testing is both more educative and more revealing, as a group of renowned psychometricians testified to 10 years ago in a remarkable document directed to the state Department of Education.


P.S. The “civil rights” trio of Klein, Sharpton, and Gingrich is falling apart—with Gingrich calling Sonia Sotomayor a “racist” because she thinks being Latino has been valuable to her judgment.


It’s Manslaughter not Homicide in the Sotomayor Case

I have Sotomayor quoted as saying:

“I would hope that a wise Latina woman with the richness of her experiences would more often than not reach a better conclusion than a white male who hasn't lived that life.”

Is this statement racist? Should Sotomayor be condemned?

Rather, is this statement an example of a polylogism, meaning that certain classes, races or groups have their own deterministic logic? Major historical proponents are 1) Marxists: class determines ideology and action; 2) Nazis: race determines ideology and race is inherently inferior/superior.

It follows that if there is no universal logic valid for all humans then justice itself has no universality. It would be as if humans are broken down into logical species. It is, further, saying that humans cannot make up their own mind independent of these disparate categorical limitations.

The absurdity of polylogism in the social sphere is represented by Marx and Engels in that they were far from having proletarian backgrounds. The Nazis, in add, were quick to use the scientific discoveries of the inferior races, Jews even, and, in action, went against their own scientism.

Sotomayor does make the distinction between “Latina” and “white”, “woman” and “male”, and “her experiences” vs. (I assume) ‘his experiences’. Fair enough (aside from the contradiction that many Latinas are white etc.). However, Sotomayor then rates separate ability to “reach a better conclusion” based on these categorical differences. But Sotomayor slyly applies a softening agent and says she is only ‘hoping’ that a better conclusion could be reached. In addition, she uses qualifiers such as “wise”, “richness” and “more often than not” in front of her categorical distinctions. It is not clear as to which categories are determined or subordinate to which. Are experiences predominantly determined by being Latina? More importantly, is the understanding accrued via these experiences determined by Latina-ness? An answer in the affirmative by Sotomayor would provide evidence of her polylogist belief. (But would it make her hateful?)

If Sotomayor is engaging in strict polylogism then how could she preside over cases not referring to her categorical definitions? Cats do not know dog justice, so to speak. Albeit, she might be showing her polylogical hand by placing herself, her categorical spot, in a superior position to the white male (cats know cat and dog justice, but dogs do not know cat justice). Of course, this presupposition would be hateful. It is dificult to determine based on the statement by itself.

If one were to take away the qualifiers from Sotomayor’s language then it still would be difficult to charge her with intentional bigotry. Polylogical thinking has recently combined with relativism creating a belief that humans are wired to have separate structures of logic and justices that, oddly enough, are cosmically equal in substance. In other words, a cat cannot decide a dog case -and vice-versa- but the outcomes are equivalent.

It remains unknown what sort of case is being decided. She could be talking about extradition issues with Monaco for all we know. To note, “Latina” and other terms are not defined either. Books have been written on this subject. At any rate, it is unnecessary to know the exact meaning of such categories or any specific case, only that such categories have been established.

The real question is whether or not she is undermining the idea that there is logic valid for all humans regardless of race, class, place, time, culture or other conventions. Does Sotomayor threaten the long established truth that justice may be determined via common understanding?

Sotomayor, to her detriment, does engage in soft polylogism because the polylogistic elements of her statement are being qualified and not the other way around. It is this flirtation with polylogism, and not the more reaching charge of bigotry, that should raise enough solid warning flags to keep her out of any job requiring robes.

I understood Sotomayor to be speaking not of race, or gender, but rather of experience. The experience of injustice teaches us much about justice.

During WWII there were US citizens of Japanese origin thrown into camps because the Feds feared their loyalty might be to the Emperor. Meanwhile, the most decorated American unit of all WWII, I believe, was made up of Japanese Americans.

There is no determinism in human behavior.

A rape victim, given the power of presiding judge in a rape trial, may allow the prosecution too much leeway and violate the rights of the accused. Or this 'experienced' judge might be so damaged by his rape experience that he cowers to the defense. Or the judge could act coolly, fairly, reasonably, aware of the pitfalls in reaction to his bad experience.

Who is to say beforehand what the objective value of a particular experience is when actions require solid ratiocination based on the right theory?

Just because someone has experienced injustice does not mean they would then be a better candidate to adjudicate justice. Justice must stand on its own reasonableness.

Oh state testing… At the end of my “first-year” in NYC – I am one of the many Fellows who joined the ranks of the DOE – I am evaluating what I actually accomplished. Having had excellent teachers in my high school, I know that a teacher’s job is never really done and I can think of a million things that I will do differently next year. Despite my personal misgivings, I feel that I did okay – no one died/was maimed in class, I received a “Satisfactory” rating from my vindictive principal, and 88% of my students passed their Regents exam. While the first two accomplishments really feel like accomplishments, I’m not as proud of that last one. Why? Because the NYS Living Environment Regents exam is completely bogus.

Students only had to correctly answer 39 of 85 questions in order to receive a passing grade. That’s right: according to the conversion formula, correctly answering 46% of the questions means you receive a passing grade of 65. This in and of itself is deplorable, but then you have to take into account that the test questions rarely – if ever – test student knowledge of the content. Anyone with a 8th-grade reading level could pass this test (I’d wager that they could get a Mastery rating). Finally, the test barely scratches the surface of the 10 Key Ideas from the state standards, focusing mostly on reproduction/development, biotechnology, and ecology at the exclusion of everything else.

Having followed your blog, I know that this is not news to anyone who has anything to do with education. I can only imagine how my kids would do on the NAEP. The push for standardized testing under NCLB is maddening on so many levels. Personally, it means balancing quality instruction with test prep. Many people say that if you’re giving quality instruction, then student learning should translate to the tests. That’s true for a few of my students, but others desperately need test taking and reading skills – as I’ve already said, these skills would serve them better than actual knowledge of biology. I think of Deborah’s anecdote about her husband and son’s test-taking skills: one used tricks to up his score while the other got tricked into lowering his. This couldn’t be a more pertinent anecdote for my classroom. Students who never showed up to my class or were at best negligent in completing assignments received passing grades while one of my solid ‘B’ students got a 53 because she is not a “test taker”; that is, she can verbally explain most of the concepts discussed in class, but balks when asked to relate the information via a multiple-choice question. How is this fair? More importantly, how is this allowed to continue?

Just as you’ve said, the testing results on the state level are pure drivel. While the state and city’s scores are rising each year, so too are the scores in our department. The administration is ready to take credit for this phenomenon; however, the teachers who have taught during the last decade can all tell you that these increases are just a function of the test getting progressively easier. One of my colleagues dropped his 1998 Regents exam on my desk after our test results were completed. Not only was the older test more comprehensive in the content it tested, but it also put more emphasis on declarative rather than procedural knowledge. This didn’t surprise me, though. Our school purchased an expensive computer program to aid in test prep for our students. Compiling and sorting former Regents questions by key idea and year, the software is invaluable for test prep, but not much else. A cursory perusal of former test questions clearly delineates the decline of the test: questions before 2000 are deemed “irrelevant” because the Regents no longer asks questions at that level of difficulty.

Like Deborah, I too wish I could find a way to measure achievement and not test performance. It’s too much to watch my kids have a “light bulb” moment only to have a standardized test tell them they are “stupid.” It’s too much to have to eliminate biology content because it won’t be a multiple choice question, even though it is relevant to creating scientifically literate citizens. It’s too much to fool ourselves into thinking we’re making a difference when we’re only setting these kids up for defeat later on, creating students who are woefully unprepared for higher education or entering the workforce.

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments