Recently in NAEP Category

Dear Diane,

We can go on forever about why "testing as we know it" cannot lead to becoming a well-educated people. We agree, we need to invent a road test—which might in some cases look like AP exams. Or, it might look like the examination system used at Central Park East Secondary School or Mission Hill (my old schools) or other formats now used by Consortium schools in New York. On the federal level, what we need are deeper and better NAEPs—where sampling continues to be wise. (They can thus be cheaper and more authentic at the same time.) Since sampled tests do not all have to be identical to be comparable, we can afford to explore "what kids must all know" both deeply and broadly enough to take them seriously.

More important than why we test is "Why School?". It's the title of Mike Rose's latest wonderful book (The New Press). What I want to argue out with you, and our readers, is the nature of the kind of curriculum or subject matter for which schools in a democratic society, funded by public monies, should be held accountable. What can we demonstrate is essential for 100 percent of all voters—18-year-olds—to understand?

1. Reading the newspapers or non-fiction magazines—or their equivalent. Being able to report to others on stories, engage in a discussion about them, and write a letter to the editor and op-ed column on a few with which they disagree. Maybe on two levels—one at around ages 11-12 and the other at 16-18.

2. Sufficient mathematics to make sense of what they find in the media—statistics, probabilities, forms of graphing, percentages, et al to a high degree of sophistication by the time they are 16. Basic arithmetic computation by 13.

3. Then comes the subject matter, the stuff that is worth reading and writing about and for! Science, history, literature, all the arts, law, governance, philosophy/ethics, politics, and economics. The criteria? Whatever is needed to be a knowledgeable and powerful member of a democracy!

Literature. A learned academic book reviewer in The New York Times recently claimed that "no one disagrees that everyone should..." have read Dante, Shakespeare, Milton, and Homer. What's the "evidence" for such a claim—much less that "everyone" would agree? (Maybe one definition of well-educated people is that they know better than to make claims about "no one" or "everyone.") While great fiction can lay claim to take us into worlds we never could otherwise experience, including the personal worlds of people unlike ourselves, thus making one a better democrat—more empathetic, for example—I've found insufficient evidence that this is what requiring the reading of Milton does.

History. Everyone tells kids that those who don't know their (or our?) history are doomed to repeat it. In fact, there are no empirical studies to demonstrate this. Some nations certainly know their own history better than most Americans—even though theirs is often far longer. Still we were more, not less, innovative. In fact, knowing history can lead some to constantly repeat and relive old enmities. One might argue that America's success correlates with its disdain for history. (Students usually rate history as their most boring subject—it was my favorite.)

Science. While certainly U.S. citizens have traditionally been exposed to much more science education than their counterparts in other democratic western nations, superstition is far higher in the U.S. and suspicion (and misunderstanding) of science greater.

The arts. Most adults are sure they can't "do" or "appreciate" any art—and few and far between are those who visit art museums, concerts (except for teens, and it isn't quite the music they study in school), or live theater, or who write creative stories or poetry.

All this may sadden me, but it doesn't surprise me. And I haven't found studies that suggest that schools that accept "ordinary" kids have managed to get other results by age 16 or 18—or in most colleges. Alas, not even my schools.

I believe strongly that we shouldn't give up; but, meanwhile, we should rethink what is "essential." Because, more serious than students not having read Dante, is when they haven't been exposed to any tough examination about the wherefores of the world they live in, nor any understanding of a strong reason to care about the survival of the democratic process. They haven't experienced or practiced democracy—through literature or life. They haven't learned to argue in ways essential to a democracy—which requires empathy, respect, reasoning power, AND a half-way open mind to other possibilities—on matters hard to dismiss with, "Well, I have my opinion, you have yours." Politics remains a dirty word.

We don't need schooling to have opinions. Even strong ones. But we do need schools to begin to imagine arguments in favor of democracy that might keep citizens from letting it go whenever a crisis appears. (See rethinklearningnow.org for an interesting Web site on this topic.)

Julian Bell, in an article in this week's New York Review of Books entitled "Why Art?", quotes an art historian who suggests that the origins of art might simply be—to escape boredom! (I suspect that more students' art takes place while listening to a boring classroom lesson than happens in school art classes.)

Deb

Posted by Deborah Meier at 10:32 AM | Permalink | 29 Comments | TrackBacks (0)

Dear Diane,

Yes, it seems so obvious to you and me. Using their metrics, the boosters of mayoral control can hardly point to any trend that supports their claims. On NAEP data, the two biggie mayoral control cities show no change, and on graduation data, NYC shows some improvement, but Chicago shows none, even if we go by the city's data. If we look at the data in the recent RAND study, it confirms many others—charter schools do no better re. test scores than their comparable neighborhood schools—some better, some worse. The data on merit pay don’t exist, of course—and it’s already a national priority. But that most comparable jobs use “performance-based” pay just isn’t a fact. The Economic Policy Institute has published a report on the subject that hardly is conclusive, but the logic suggests serious trade-offs that none of the folks in D.C. seem to be even aware of. If you do X, there are “unintended” but inevitable consequences—especially if you’re not even aware of them. (As for the charge that local control leads to more corruption? Having the right friends in the right place may lead to millions under mayoral control vs. hundreds under local control.)

School reformers perhaps should be required to publish “cautions” the way drugs do—if you do this, watch out because….(In bigger print, however.) In fact, it’s an essential “habit of mind” of a well-educated person—paying attention to trade-offs.

Regarding the kind of national standards being proposed is really a national curriculum. One of the arguments for it is a give-away: with national standards, if a kid moves from school x to y, he’ll barely miss a heartbeat. Again, no concerns over unintended consequences—teacher burn-out at the very least. There are forms of standards that could avoid this—if we feel the need for them—such as those CPESS modeled (“habits of mind”) alongside some broad sweeping propositions in each field. More like the Advanced Placement English test (at least the way it used to be). It didn’t require everyone to read the same books or spout the same answers. It directed “attention” to certain themes and ideas so that students could use what they had read and studied to come up with their own responses to timeless issues in literature. Ditto for history or science. These wouldn’t be “specs” for test producers, but specs for schools to consider in designing curriculum and assessments.

The class-size debate isn’t on the “reform” agenda. But it has been assisted by two excellent books. An EPI (Economic Policy Institute) book in the form of a debate and a wonderful study and brief in favor of smaller classes by Garrett Delavan, "The Teacher’s Attention."

NAEP then could do the deeper and broader job of seeing how these play out over time, including a look eight years down the line—in college, on the job, as a voter.

I had a fascinating conversation with a teacher who has been teaching physics for 15 years in L.A. and the surrounding area. She teaches 200 students on most days (in groups of 40 or so), at least three of the classes have the same test-prep curriculum. She feels bored. Ready to “move on”—but to what? I asked her to describe what she wishes she could do—even if it were unrealistic. Her wishes? Small classes so she could explore science more deeply with kids, the opportunity to do some interdisciplinary teaching with colleagues, to be able to approach physics from directions that might not match the state exam, to expand her own intellectual horizons alongside of and separate from her students. What kid wouldn’t say, “amen”?

In such settings, a teacher with 15 years of experience wouldn’t be at the end of her career (and wits) in the classroom. Imagine schools, in collaboration with universities, as the site for teacher-training—prolonged apprenticeships. My friend would then also be teaching other colleagues and would-be colleagues and learning from some interesting scientists on campus.

This was the “reform” idea of the late '80s. But it’s getting harder, not easier, to imagine this happening today. But, you ask—how did we move so far from this vision in such a short time, so that we now have a bipartisan plan for schools that make the old factory-model look innovative? Partial answer: We left practitioners like me and educational scholars like you out of the loop and instead turned to financiers and lawyers!

Deb

Posted by Deborah Meier at 10:05 AM | Permalink | 26 Comments | TrackBacks (0)

Dear Diane,

Thanks for being there at the venerable old Julia Richman high school building last week, Diane. So many old friends, some new ones, and a nice net profit for FairTest—the little David facing the Harcourts, McGraws, ETS's, et al Goliaths that sponsored the event. Actually, Diane, FairTest is not anti-test, but skeptical about the instruments used for mass standardized testing and, above all, the uses to which they are put. The same publishers that once warned us against any test prep, other than the practice tests they sent out with the real test, now probably make as much in preparing for tests as testing. They once said that it destroyed the psychometric reliability and validity of the data. Then they discovered that it was either (1) too hard to enforce and/or (2) that test-prepping was too lucrative a market to overlook.

It was only in the middle of my acceptance remarks last week that I realized that my son, Nick, and his father had polar opposite test responses. Nick always found ingenious ways to read test questions and interpret test directions that lowered his scores and Fred found equally ingenious ways to answer them right. Fred was good at the game of testing, but not schooling (he never got a high school diploma, but got into the University of Chicago via testing); Nick reversed it. Son Roger loved standardized tests and managed to avoid doing much schoolwork because he was so good at acing them. (The anecdote that I told, which those in the back like you may not have heard, was Nick’s insistence on covering up the reading passages before answering the questions because he considered it cheating to do otherwise.)

This story posted in an MSP working group by one Eric Jakobbson on June 5th is a delight and a reminder:

In the book of Richard Feynman's letters compiled by his daughter, she tells the story of being told by her mathematics teacher that a particular way of solving a problem that her father showed her, was in fact wrong. So Feynman and his daughter went to see the teacher, who proceeded to lecture Feynman on the mathematics. Feynman then revealed that he was a world-famous theoretical physicist, at which point the teacher was quite embarrassed. But the Feynman letter for which this was the context was the letter of apology that Feynman later sent the teacher, for invoking his stature to intimidate the teacher, rather than sticking to the mathematics to clear up misunderstandings.

Upon discovering the oddities of ”statistics, damn statistics,” I’ve enjoyed noticing every day how the media handles statistical information. Graphs, tables, etc. It takes a lot of perusal to be sure one gets them right. For example, if one graph bar looks enormously higher than the next one, it may mean very little because their starting point distorts the difference. And on and on. The headlines about our declining rate of unemployment offer another instance, and I’ll bet a large portion of the lay public confuses the declining rate of increase with a decline in unemployment. Probably if mathematics is to serve democracy (not just those going into mathematical professions) we need a drastic overhaul of how we teach and what we teach.

Exercising good judgment about complex matters—which is at the heart of education for democracy—includes knowing when to trust and who to trust since our expertise in any particular field is bound to be limited. There’s no escaping this, but it’s surely a serious weakness of democracy. Except for the fact that there’s no easy solution to it that isn’t worse. But it also requires knowing enough about some basic skills and subjects to apply good sense to the reported data or narrative account.

If we understood standardized testing better we’d know, as Daniel Koretz et al remind us in the New York Daily News piece you quote, that shifts in scores cannot and should not follow the bizarre patterns we’re seeing. Two tests of reading are only as valid as they are alike. If one test produces higher scores than another, one or both are misleading—defining reading in seriously different manners, being prepared for it in ways that distort the product, or plainly being cheated on. I claim all three are widespread. The higher the stakes, the greater will all three thrive.

Are there alternatives? This week at my 60th high school reunion, the old friends at my tables, while agreeing with me about tests, insisted that there was no alternative. (Disclosure: They were all good testers.) But there are alternatives. Let’s explore how NAEP could be better used, Diane, and also how schools and professional associations can help school communities—parents and teachers above all—better zero in on local information, down to “how’s my kid doing?” There are schools in NYC and nationwide that have solved the latter, including schools whose founders were in the room the other night—who started small and extraordinary high schools in the early '90s. These include the schools in Julia Richman where we met (Vanguard, International, and Urban Academy), and the schools started to take in kids otherwise served by Julia Richman, including Mary Butz’s The Manhattan Village Academy and Sylvia Rabiner’s Landmark. And many more. Our elaborate alternative to standardized testing is both more educative and more revealing, as a group of renowned psychometricians testified to 10 years ago in a remarkable document directed to the state Department of Education.

Deb

P.S. The “civil rights” trio of Klein, Sharpton, and Gingrich is falling apart—with Gingrich calling Sonia Sotomayor a “racist” because she thinks being Latino has been valuable to her judgment.

Posted by Deborah Meier at 9:05 AM | Permalink | 4 Comments | TrackBacks (0)

Dear Deborah,

I enjoyed seeing you honored as a hero of education by FairTest last week, which established an annual award named for you. If anyone had told me five years ago that I would be at that event, I would have thought them mad. This is what “Bridging Differences” has done for me, I suppose.

At the event, I was surrounded, not surprisingly, by educators who have long believed that standardized tests are more wrong than right, or that they are a crime against children’s nature, or worse. I continue to believe that we can get valuable information from standardized tests and that they can help with diagnosing problems and needs. The information derived from testing can be useful, but lately I have begun to see how often test scores are being misused to punish kids, teachers, and schools and to mislead the public.

As it happened, New York state just released the results of its annual tests of English language arts and mathematics, and the scores soared across the state to an extent that was literally unbelievable.

The state Education Department released the math scores last week. From 2006, when the current testing regime’s trend line begins, to today, the percentage of kids meeting state standards (that is, scoring a 3 or 4 on a four-level proficiency scale) has gone from 65.8 percent to 86.4 percent. In 8th grade, the proportion meeting or exceeding standards leapt from 54 percent to 80 percent. The gains for black and Hispanic students across the state were huge—for black students, from 45 percent to 75 percent, and for Hispanic students, from 51 percent to 79 percent. White and Asian students are already close to the ceiling, at 92 percent and 95 percent respectively.

Some districts saw increases that defy anyone’s wildest dreams: In Buffalo, the proportion passing flew up from 28.6 percent to 63.3 percent; in Rochester, it went from 33.1 percent to 63.4 percent; in New York City, from 57 percent to 82 percent; and Syracuse nearly doubled from 30 percent to 58 percent. All in four short years! At this rate, everyone will be proficient well before NCLB's deadline of 2014.

The news media reported the dramatic gains with a straight face. The superintendents of Rochester and Buffalo basked in the limelight. Mayor Bloomberg said the scores proved the value of his one-man control of New York City's schools, although surely his reign had nothing to do with the even larger gains in other cities in the state. Only the Rochester newspaper asked in an editorial whether these gains made any sense.

Now the New York Daily News has done an analysis of the math tests and concluded that the state tests got progressively easier from 2006 to 2009. Kudos to reporters Meredith Kolodner and Rachel Monahan, who beat The New York Times to this statistical scandal. Kolodner and Monahan had the smarts to turn to Jennifer Jennings of Columbia University, who was formerly the blogger for Education Week known as eduwonkette; Jennings analyzed the tests and discovered that the state has been testing only a fraction of its math standards, and teachers are able to predict which standards will appear on the tests.

Jennings also found that nearly identical questions have appeared every year. “In 2009, at least 14 of the 30 multiple-choice questions on the seventh-grade exam, for example, had appeared in similar form in previous years,” said Jennings. Teachers and principals chimed in and agreed that the questions were predictable and students are taking frequent practice tests that teach them the format.

A teacher explained to me recently that “we drill down into the state test to predict what will be tested," and then students practice those questions, again and again.

My guess is that if the students in New York state were given a math test from another state—one that they had not been primed for—their scores would be much lower.

U.S. Secretary of Education Arne Duncan keeps telling states and districts that they are “lying” to kids when they tell them that they are doing just fine, but they really are not. Just last week, Duncan said that many states are "lying to children and their parents, because states have dumbed down their standards." New York is a perfect example of what Duncan means. The proportion of students who pass the tests keeps going higher and higher every year, but when the 2007 NAEP scores were released, the state had flat scores in everything but 4th grade math.

What we see in New York state is institutionalized lying, according to Secretary Duncan’s definition. The state is well on its way to becoming a national laughingstock if it keeps up this Ponzi scheme whose victims are its students.

Mark Twain wrote, “Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: 'There are three kinds of lies: lies, damned lies, and statistics.'"

The New York State Education Department is showing how easy it is to lie with numbers.

Diane

Posted by Diane Ravitch at 8:59 AM | Permalink | 82 Comments | TrackBacks (0)

Dear Diane,

It was wonderful to see you and listen to you last week at the lovely tribute in honor of some special people—including you. Class Size Matters and Leonie Haimson have done an amazing job and their Web site is must reading.

And while I’d like to pursue my last week’s letter—and get your responses—I have to say a few words about your Tuesday blog first.

Geoffrey Canada’s point in developing his Harlem Zone is, in fact, the Broader, Bolder point. He isn’t focused only on schools, but on the health and welfare of a community. Good for him. He fell for the “only tests count” strategy, as I read it, in part out of his own ignorance and in part because his wealthy board insisted on it. It’s the road to fame these days; nothing else will substitute. But, like you, I’ll settle for all schools having the resources and attention that Canada “lavishes” on his kids.

Where you and I are in serious disagreement is on the notion that good manners distinguish the poor from the middle-class or rich! If so—it’s in reverse order to what is too often suggested. The poor kids I encountered in kindergarten were accustomed to more formal and more consistent good manners—whether it was in how to address their elders or how to dress properly. They were less whiney and more obedient. It carried over to the doctor’s office—where middle-class kids acted like healthy brats while children of the poor sat with silent decorum. It breaks down much later.

Children of the poor get tougher and more unmannerly slowly. In time, they lose respect for authority. Perhaps because adults are rarely able (or willing) to protect them. Maybe because many public authorities quite openly treat them and their families disrespectfully. Over time, they come to depend on “the streets” and their “peer culture” for safety, and they imitate the public swagger offered on “middle-class” media of wealthy athletes, talk show hosts, et al.

You and I, Diane, want an intellectually feisty citizenry, and that’s what we don’t offer the poor. I discovered they enjoy it as much, if not more, than their richer peers.

Now, back to accountability.

Current forms surely fit well our age of distrust. Imagine pretending that the entire state of New York has earned unheard-of score increases over one year? Only someone ignorant about testing—or scornful of their audience—would dare release such absurd data. Any respectable scientist with equally startling results would avoid public announcement until engaging in an investigation—like doing a study using a different instrument on a sample of the population. We only need alternatives that do better.

We need to hold schools “accountable” in ways consistent with democracy. My default position (See Nov. 21, 2007, blog!) is “leave it to those closest to the action.” Every time we move a notch further away we need to beware. We lose something each step, so we better be sure what we gain is worth the loss.

Our schools should be exemplars, illustrative of what public scrutiny is all about. They should mirror the kind of thoughtfulness we hope goes on in our classrooms. I’d say schools are ahead of their reformers—politicians and business leaders—in this regard.

Last week, I suggested a more thoughtfully and deepened NAEP in literacy—starting no earlier than 4th grade. With math—which is both a routine skill and an academic subject, it’s more difficult. Maybe a math test at 8th grade that only covered “arithmetic” and some basics of mathematical reasoning, statistics, and odds could be arrived at? But I’m already exposing my prejudice. I’m more worried about the number of adults who can’t tell millions from billions from trillions easily (like me), who don’t know how to examine statistical claims, who are confused by “the odds,” etc. versus those ignorant of algebra and calculus. That’s why I like sampling—because it doesn’t pretend to be holding anyone accountable, but only hopes to be informative. Data can inform, it cannot drive! But, of course, the public is suspicious of sampling because we’ve never used schools to uncover their mystery even though we use the technique ad nauseam in both public life and business. Choices must be made—but none is perfect, and we have to make trade-offs all the time in life. But for the sake of our national health we shouldn’t all be required to make the same trade-offs—except for (I’d hope) demonstrating the connection between our mission and the mission of democracy.

In future weeks, I’d like to explore why we can’t ALL be doing something like “exhibitions” when it comes to “accounting” for individual learning (and external visitations re. individual schools). This month the Coalition of Essential Schools is highlighting a few schools' final exhibitions nationwide. That’s where the work really pays off—in helping teachers, students, and families be less dependent on test scores and more capable of knowing themselves well.

Such approaches allow us to measure the so-called effective skills and habits in the process of measuring the “hard” stuff. It’s a form of assessment that mirrors the values of a democratic society: the exercise of informed judgment and responsiveness to public critique.

Deb

Posted by Deborah Meier at 9:12 AM | Permalink | 28 Comments | TrackBacks (0)