« The Secret of Success and High Test Scores | Main | »

We're Relying on an Absurd Definition of Achievement


Dear Diane,

Yes, the ways in which standardized bubble-in tests are open to abuse are rampant—and gaming of test scores has increasingly important repercussions. In the long run it leads to an increasingly toxic education in the name of “standards.” I urge you to read the two chapters in "In Schools We Trust" on my own experience and subsequent investigation of standardized testing for 7-year-olds. I discovered, in my effort to “prep” them, that while reading tests may in part test the ability to read—comprehend (vs. re-code into sounds)—they are even better at detecting one’s class and cultural sub-group. Of course, there were exceptions. There are ways to raise scores via prepping, but it can only diminish the “gap” if those at the top get less effective prepping. I published the recorded interviews I conducted with students—in 1972, and Jay Rosner has shown how it works for the SATs, and many have followed. Between 1980 and 1990, it seemed we had won the battle. By 2000, they had returned with a vengeance. Thus many friends and allies suggested I give up on this particular line of reasoning.

Meanwhile, we’ve witnessed a massive redistribution of wealth. Everyone has their culprit. Too often, "lazy" children, parents, or educators/teachers? In a nation that ranks near the bottom in services for the young, not to mention a bigger gap between rich and poor than we’ve seen since the 1920s, maybe this is myopic? “Forget it,” friends suggest. You’ll just be accused of making excuses for bad schools.

What you have documented, Diane, is eerily close to the kind of data abuse that helped create our current financial crisis. We go blindly ahead. There seem equally few lessons learned from either crisis. It’s apparently easier to hold children and teachers’ “feet to the fire” than the creators of the greatest economic crisis of our lives—especially Harvard grads. Note: The latter did fine on their SATs.

Many reputable studies have also demonstrated that, based on such tests, no evidence exists for most of the Race to the Top-NCLB-like reforms. Just rhetoric. Just more shifting of power away from the public sector. The CREDO study you mention suggests that only 17 percent of the charter schools do better on tests than their comparable noncharters, and more do worse. Try that with a drug test, and how long would it remain on the shelf? (Even though, in fact, low scores is not one of my complaints against charters.)

What’s the alternative, critics argue! Is a bad drug better than none at all, I respond?

In fact, we have alternatives—some pioneered in the good old USA and others in those much-vaunted international comparisons. For example, there is no competitor that relies as much as much as we do on testing of our sort. None. None. None. (A light bulb should go on.)

Alas, there are no simple magic bullets even among the reforms I like. Partly because we don’t all have the same agenda when it comes to outcomes—our priorities differ, what we’re willing to trade off or risk differs. Also, any reform package depends on its implementation and few recipes are foolproof. Trying to copy the KIPP model or the Deborah Meier “model” won’t produce the same thing.

And, then there’s the ornery fact that when today’s reformers refer to proof of “achievement” they mean something different than you and I do, Diane. Achievement equals standardized test scores in reading and math; others add test scores in other subjects, including aptitude/IQ tests. Everything else gets called “soft skills.” It’s as absurd as calling the written driving test the real achievement and the road test a measure of “soft” skills.

I’d argue that this applies to most intellectual knowledge and skills. We’re relying on an absurd definition of achievement—at best. It’s not lack of alternatives, but a lack of interest in having real standards that take into account that we are all not the same and that we actually don’t want to be all the same. Our interests, passions, talents, and even our priorities differ. High standards can be met in honest and serious ways if we are prepared to take the harder route—starting with each child. If we care enough about both means and ends consistent with democracy it will not be easy and will not be based on the need to rank order individuals, schools, or nations. There are plenty of good alternatives.

Any publicly funded school must serve, for better or worse, the public good—the complex demands of a modern democracy. But the public good is also met best when each individual's private good (his/her passions and interests) are also met. But how we sort out what best serves both will always be an art, not a hard or exact science.

There is no single “best” model—but there are bad ones. Neither the common good nor personal good can come out of schools that do not treat all members of their community with respect. (Respect, of course, is no easier to define than achievement.) Perhaps the reason I love "Tales of Priut Almus*" by Robert Belenky—which describes the time he spent in a Russian shelter for homeless youth—is the unmeasurable respectfulness with which they responded to the young people in their care. Belenky says “offering a short-term, flexible, familial, community-based home may be the most useful gift (offered) these young people.” But he actually describes far more than that. So, too, in any respectful school there is more than that—but nothing without that.



"they [reading tests] are even better at detecting one’s class and cultural sub-group." But precisely.

The purpose of democratic schooling is to erase those class and cultural divides. Thus we can naturally assume that any test worth its salt will indeed show that the Polish American student who continues to think, read, and write in Polish language and workingclass Polish idioms will not be classed as making effective or "excellent" progress.

Of course, Polish is not the sub-class or culture you were referring to, Deb. It is, though, the culture that my grandparents managed to fight through and overcome. Are you familiar with Polish? Check out pp 282-283 of Michener's Poland for just a short peak at its complexity and perverse logic.

Now, Grandama, born in Poland, living on a 1 acre farmstead here, married to a slavic coal-miner-metal-worker, surviving the great depression, managed to learn English well enough to constantly whoop my butt on the NYTimes crossword puzzle. But that was before state education associations.

Today we have a cultural sub-group present in the country for 180 years. That's generally seven+ generations. (Families more recently immigrated show better assimilation).

Unlike the other major cultural sub-group--Hispanics--who have mostly arrived more recently, they (nationally) do not show the same type of progress in schools as Grandma and her generation of Poles, Slavs, Germans, Irish, Chinese did.

So, at this point, we can toss Ed in the stocks for racism and cultural insensitivity. Or, among more self-interested educators, we can say, 'yes! So don't blame the teachers!'

The bottom line remains: we need far more Blacks and urban youth to make it into the ranks of teachers, designers, medical professionals, engineers, scientists, technologists, new media professionals, etc.

The front line of that battle is the schools. There's nowhere else to look. Churches, maybe, but we as intelligentsia have too long over-berated and poo-poohed the value of that province of our society.

It is to teachers that we entrust the task of making that cultural sub-group undetectable over the phone in a business conversation.

I fear I know little of Kanye West. But I'm pretty sure he isn't going to help a lot in that goal.

The CREDO study you mention suggests that only 17 percent of the charter schools do better on tests than their comparable noncharters, and more do worse. Try that with a drug test, and how long would it remain on the shelf? (Even though, in fact, low scores is not one of my complaints against charters.)

This is completely incoherent and illogical. "Look at how bad charters are, they don't have high enough test scores! Now let me get back to my mantra that it's bad to talk about test scores."

If you want anyone to think that your arguments about test scores are sincere, don't make the only argument against charter schools be their test scores.

Problems with this argument, besides complete incoherence: The CREDO study was purporting to compare charter students to control studehts in public schools. But given that so many charter schools nationwide are "last resort" schools (i.e., meant for at-risk students), you can't really do such a comparison fairly. Students who were in such trouble that their parents tried the charter school can't really be matched with students who weren't in as much trouble.

Next, you're neglecting to mention the fact that students have higher test scores the longer that they're in charter schools, by the same study. The reason that the AVERAGE effect was lower was that students do worse in their first year, right after they transfer. No surprise there. But then they pull ahead. And when you average superior performance with slightly worse performance, the overall average looks rather, well, average. But it's misleading to quote the overall average as if that were the whole story.

Finally, you should take seriously your own complaints about test scores. And then admit the obvious: charter schools are often a good thing, not necessarily because they make kids into academic superstars, but just because they give people more options and choices. Someone around here, in a more sane time, wrote an article about why liberals should be pro-choice on schooling.

It is heartening to read this eloquent call to pay attention to individual children, to context, and to expose the political corruption at the heart of absurd definitions of achievement and reformers claiming a “best” model. How can we move beyond words shared on a blog to action? Before the Race to the Top funds arrive to begin sorting out teachers and principals, and the programs that prepare them, based on these absurdities, we must find a way to join forces and make the messier, artier alternatives take root in public education. What would you suggest? A march on Washington? We cannot stand idly by while young people are bored to death by school and deprived of a substantive education.

Standardized tests may have their problems, but I don't see the connection between reliance on school tests and the national income distribution. The academic achievement gap is smallest in countries that have a common curriculum controlled strictly by their Departments of Education, and that sort children to different school tracks based on test results in addition to school grades.

It is OK to prepare students for tests - as long as preparation has educational value in itself. But it's hard to get that done with multiple-choice tests, for which test preparation too often amounts to teaching in sound bytes. The solution is to write tests that rely instead on open-ended and multi-step questions.

The creators of the economic crisis, if it's possible to pin point them, may have passed their SATs with flying colors - it was not an ethics test.

The analogy to "derivatives" is very apt, Deb. Today test scores are converted to statistical scales through a process that very few understand. The scores are completely ungrounded. To make them "intelligible" arbitrary cut scores of the distribution are set and termed "Below Basic," "Basic," "Proficient" and "Above Proficient." ("Above proficient" is a mind-blower!)

The bubble will burst when people wake up to the fact that the reports of "proficiency" have nothing to do with proficiency.

As you say, the tests are sensitive to social class differences but not to instructional differences. There are papers that explain how this comes about and describe alternative methodology in papers at the Social Science Research Network:

Diane has pointed out that NCLB is not an instructional improvement program. It is a testing program.

In a very thoughtful paper,


Jim Popham, sketches the evolution of Federal involvement and points out that the tests currently in use do not provide a basis for instructional improvement.

This disconnect is the chief obstacle in achieving the very worthy aspirations of NCLB.

Thank you, John Doe, for doing the work of reading past the headlines on the CREDO report. Both Deb and Diane have lazily tossed this out as proof that charters are unnecessary and mostly unhelpful; better scholarship would be appropriate in this most critical of areas.

I'll re-quote you for effect; perhaps someone will further quote the study.

students have higher test scores the longer that they're in charter schools.. The reason that the AVERAGE effect was lower was that students do worse in their first year, right after they transfer. No surprise there. But then they pull ahead.

The CREDO study was a brutally honest investigation by a charter advocate. We should do them the justice of repeating the true result.

If we really want to be honest, we might then take the extra step of pro-rating charter's effectiveness per dollar granted. Everyone here knows that most charters are (1) working with less money per student and (2) forbidden by state laws to enjoy the benefits of scale that public districts can have. Moreover, most are not just teaching, they are still building up from scratch.

Finally, all new enterprises have their "gold-rush" period. When we look at charters, we have to project past that period to the day when the bad concepts and organizations have been driven out. There having passed insufficient time for that, the yet-to-be-weeded-out charters drag down the average scores.

We can expect lesser minds to misrepresent CREDO, but not Deb and Diane!

There may be no perfect model, but why aren't we working towards the best we can produce? As a scientist looking in, I don't see the same forces moving education forward in a continuously improving fashion like other fields - medicine for example. Look at medicine for a moment. What if educational R&D took place at embedded K-12 schools at education colleges and universities analogous to embedded hospitals? Isn't that the intellectually rich environment to develop and demonstrate new ideas? Don't colleges and universities have far more resources than charter schools to actively engage in R&D? I'm not against charter schools, but let's optimize the model.

The most telling comment in the article is the need to consider each individual child (who they are, where they have been, what they have experienced) and tailor the education to their interests. Children have a habit of taking up topics and exploring as much as they desire (some for a long time and others for a very short time), then moving on to other topics (including sports). Yet all reform wants to standardize the curriculum and expect everyone to move at the same pace. Why?
My personal experience shows that even though once an "average student" in third grade, and absence of 2 months from formal school and home tutored for 1.5 hours a day by my mother yielded an "A" student afterwards. I reverted again to an "average student" except for areas of interest in which I basically finished the course textbook at my own pace within the first 6 weeks of school. I and my friends picked up sports with a vengance for a period of time and moved on to others after a while. English was one of those courses that I had no interest in and saw no use for, until I moved to this country and had to use it every day. Was it hard? Yes, and no. I happened to start schooling here in the U.S. in 9th grade in March, and passed every subject except English Literature where the final exam was mostly - "What did the author mean by the passage ..." It was like trying to pick out the words in the lyrics of popular songs and getting the meaning of jokes. College performance was the same way, average except selected courses that evoked my interest. (I might mention that I also worked full time to pay for school and took certain classes during lunch as that was the only time offered.) Graduate school was different. There just about all of the course elicited my interest and I finished with an "A+ average." My career included doing research and my observation was that most of innovation and best ideas came from the infusion of different perspectives into describing a problem and identifying novel solutions by taking knowledge from different domains.
So what does this long exposition mean? That society needs clever mechanics as much as it needs mathematicians and scientists. That it needs erudite professors just like it needs broad minded politicians. That it needs inspired musicians as much as it needs gifted athletes. It needs compasionate ministers as much as it needs insightful doctors. Is it any wonder that we can not get agreement on standards when standards would make for a very boring society and end up with everyone working "inside the box." Let us get out of the box and develop choices in education that allows students to grow in the dimensions that they desire and contribute to all of society's needs.
One last thought - What works great in one setting may or may not work in a different setting because we are dealing with individuals interacting with others. Look at the mathematical representation of the interactions that take place in a school and you will see that they are exponential in nature as far as effects are concerned. Some will be as intended and some will not.

"54% of Mass. Students Fail to Meet Achievement Goals"

heAdlines an article in yesterday's Boston Globe:


As I recall, Diane and others have told us that MA has the best Content Standards and tests in the Nation and that MA students have made remarkable gains in the last several years (words to that effect).

No wonder teachers, parents, and citizenry are confused. Not only are we relying on an absurd definition of "achievement" but on an absurd definition of "achievement goals."

Deb and I don't agree on everything but when she talks about scantron bubble tests being a phoney and inadequate measure of achievment I am in complete agreement with her. I am horrified at the force march to 'stardards' based on multiple choice only tests. At present about 40 schoool days a year in my high school are spent administering 'bubble tests" and prepping for them. And these tests have two major problesm.
1) they are not accurate instruments of academic performance. For many immigrant students -even those who speak English- they find the language of these tests mystifying. "All are true EXCEPT..." "Which primary documnet is NOT ANALOGOUS? " I could go on. These tests ARE biased in favor of native English speakers who have better native cultural literacy and experience with these tests. For example, a local attornery hired me -at over $50 per hour- to tutor his child for the AP European History test, AP US History test and AP Spanish test. Other well-to do students enroll in expensive SAT prep classes. You can teach to those tests and inprove performance.

But at least AP tests are authentic instruments. They are almost all at least 50% essay and analysis and only about 50% multiple choice.

By contrast many other standarized tests are acaemic junk food. They don't say anything about the ability to learn. They are, above all, literacy tests. People who are not native speakers or who do not read well will perform poorly on these tests.

# 2 ( pushed the wrong button : I meant to preview not publish)

The second reasons many of these tests are not valid is that the students themselves have no buy in. Many are apatheic about the results because they have no influence on grades or graduation. We have students who score 400 or 450 on the CASHEE in English or Math (very good scores) who just blow off their core area English CST's (Subject area tests) as well as other tests. If we are going to spend millions on these tests they might as well go on the student transcripts and effect class standing at the very least.

As it stands now I am in favor of abolished NCLB and California CST's entirely. It is sad that we spend so much time teaching to these tests that we never do maps, go on field trips, do book reports, do term papers any more.

School was hard enough when I was in school but today it must be torture to many young people. I carry on and do my duty because I must but I will never pretend that I like it or this testing regime is doing kids and schools any good.

Ed Jones said:

"The bottom line remains: we need far more Blacks and urban youth to make it into the ranks of teachers, designers, medical professionals, engineers, scientists, technologists, new media professionals, etc.

MUNRO: I agree completely with this. It is the reason why we need good comprehensive high schools available to all youth who want to learn. But I think some students aged 15-16 are not mature enough or ready for high school. They should perhaps be enrolled in Vocational programs. That doesn't mean we have a two track system that locks out students forever. We have an Adult School here, GED programs, and the Junior College. For those who are finally mature enough to learn there are many avenues for self advancement. And that is not even considering Independent Study, Workforce and all the progams available in the public schools.

Ed Jones said

" The front line of that battle is the schools. There's nowhere else to look. Churches, maybe, but we as intelligentsia have too long over-berated and poo-poohed the value of that province of our society. "

There are many institutions which help educate our youth. The family is one.
The Boy Scouts and Girl Scouts are another. As are non-profit educational clubs. And the military does a wonderful job at training, assimilating, educating and motivating minority youth. I am fortunate in that I live in a semi-rural school district in which the majority of the children attend houses of God on a regular basis. It is even common for local ministers to work as substitute teachers. I myself work as a volunteer catechist (in English and Spansih) and some of my students attend my high school. The reading ability and cultural literacy of children who attend Church regularly are usually above average. People who can read the Bible and understand it can understand the allusions of classical literature and they can read Shakespeare and Cervantes. Part of the reasons for our educational decline is due to
1) the collapse of the family as an educational force.
2) the decline in church attendance (or attendence in the house of God of your own choosing.
3) the decline of literacy among adults and young adults in general. This generation is teh Vid-kid generation and addicted to cellphone, texting and the Internet.

DEB WRITES "Meanwhile, we’ve witnessed a massive redistribution of wealth. Everyone has their culprit. Too often, "lazy" children, parents, or educators/teachers? In a nation that ranks near the bottom in services for the young, not to mention a bigger gap between rich and poor than we’ve seen since the 1920s, maybe this is myopic? “Forget it,” friends suggest. You’ll just be accused of making excuses for bad schools. "

I don't want to make excuses for poor schools either but the idea that poor children are totally neglected and no money is spent on them is quite curious.
We spend a fortune of state, Federal and local tax dollars trying to see that Title I schools and other schools have a level playing field. Washington, DC public schools pay more per capita than any schools in the nation and perhaps the world but they have terrible results. I think we should spend more money on k-12 eduation than we do so that we can have smaller class size and offer more electives such as AP classes , art and music classes. But we are spending a lot of money for new books and the best support materials money can buy. At least that is true in our school district (Kern HS District). Yet the figures seem to show all this investment in new books and technology is not narrowing the education gap. This is a very complicated issue but if we are to be honest we must address student apathy and the anti-intellectualism that is so pervasive in our society. Even good students think their sports practices are more important than study so A students become C students. And is long as athletes are eligible that is all that matters. But student athletes in AP classes are as rare as lions in Egypt or Snow Leopards in the Himalayas. That is extinct or almost non-existent.

Hi All.... hope this finds everyone well and welcome back.

Would still like to suggest that we expand the menu of possible ideas that would improve educational outcomes...on many levels.

Hope and Despair in the American City
Why There Are No Bad Schools in Raleigh
Gerald Grant
In Hope and Despair, Gerald Grant compares two cities—his hometown of Syracuse, New York, and Raleigh, North Carolina—in order to examine the consequences of the nation’s ongoing educational inequities.

The school system in Syracuse is a slough of despair, the one in Raleigh a beacon of hope.
Grant argues that the chief reason for Raleigh’s educational success is the integration by social class that occurred when the city voluntarily merged with the surrounding suburbs in 1976 to create the Wake County Public School System.

By contrast, the primary cause of Syracuse’s decline has been the growing class and racial segregation of its metropolitan schools, which has left the city mired in poverty.

Hope and Despair is a compelling study of urban social policy that combines field research and historical narrative in lucid and engaging prose.

The result is an ambitious portrait—sometimes disturbing, often inspiring—of two cities that exemplify our nation’s greatest educational challenges, as well as a passionate exploration of the potential for school reform that exists for our urban schools today.

If interested in more info and a discussion feel free to check in here:

Odd how quickly we seem to believe that this could never occur in America.

be well... mike


I agree with you completely. Why aren't people organized to fight the current absurdities in education? I was hoping the Broader, Bolder Approach people would take the leadership, but they seem to be silent. I joined that group for this very purpose but I rarely hear from them. Surely the teachers' associations and the university professors of education could get together. It's all very strange.

Jnhn Doe claims that most charter schools are "last resort school." Does he have any data to support that? The data that I have read says quite the opposite. While some charter schools are aimed at serving low performing students, most serve a better off populations than there traditional counterparts, at least according to the last data that I read.

Hi All.... hope this finds you well!

From: School Resegregation and Civil Rights Challenges for the Obama Administration: A New Report from the Civil Rights Project at UCLA

Since the Supreme Court reversed course in 1991 and authorized return to segregated neighborhood schools, there has been an increase in segregation every year, particularly for black and Latino students.

The report shows that 40% of Latinos and 39% of blacks now attend intensely segregated schools.

The average black and Latino student is now in a school that has nearly 60% of students from families who are near or below the poverty line.

These doubly segregated schools by race and poverty have weaker teaching forces, much more student instability, more students who come to school not speaking English and many other characteristics related to family and neighborhood poverty and isolation that make for challenging educational environments.

These are the schools where much of the nation's dropout crisis is concentrated.

More than 40 years after passage of the Fair Housing Act, there continues to be almost no serious enforcement against widespread housing discrimination, which impacts the segregation in districts with neighborhood school policies, and is making it difficult to maintain integration in suburbia.

The report concludes that efforts to make separate schools equal, which have been the dominant approach since the federal government abandoned significant positive support for integration almost three decades ago, have failed.

This failure includes No Child Left Behind, which was supposed to quickly equalize achievement across racial lines but has fallen far short. Instead, it is sanctioning scores of segregated minority schools without providing them enough help to make a difference.

Go here for the full report:

be well...

The solution is to allow all children trapped in minority schools to apply to any public school with transportation provided. If the receiving school is paid enough money and has some say in accepting or rejecting students, I think parents might go for it. Keeping poor and minority children isolated is a terrible shame for our country.

No, there's no evidence that charter schools on average are "creaming" more advantaged students. And charter schools serve more "at risk" students than public schools, proportionately. See http://www.edreform.com/Archive/?Charter_Schools_Succeed_Where_Others_Fail_Serve_More_AtRisk_and_Minority_Students_and_Boost_Achievement (Spare me the tedious response that this is a link to an advocacy organization; its facts here are correct.)

And I'm dismayed to see this cropping up yet again from a supposedly intelligent person:

What you have documented, Diane, is eerily close to the kind of data abuse that helped create our current financial crisis.

Testing in schools has no parallel or analogy to what created our financial crisis. To expand a bit: Asking kids who were supposed to have learned long division problems to do a few long division problems on a test does not have anything whatsoever in common with improper valuation of mortgage-backed securities, overly low interest rates inflating the housing market, overuse of CDSs, etc.

You're arguing by prejudice and ignorance here, by referring to things that you plainly don't understand (the financial markets).

Checking--last time I responded it disappeared!

Well, lots of readers are responding to each other. Good! We won't change each others opinions--probably--but we expand our understanding. Possibly.

Precisely the disagreements here remind me of why we should not, as Andrei suggests, have a single curriculum nation-wide. (Of course, others may claim it's why we should!) Who are the philosophically neutral experts we'd rely on to invent it? Clearly some folks--like Ed and John, like the current definition of well-educated (scoring well on standardized bubble-in tests). I'm not for stopping them but I'd like there to be choices! Yes, I'm for choices where possible--and have written extensively about this; it's even one reason I'm for small schools.

No, most of the top ranking nations on tests do not have our kinds of tests or state controlled curriculum. Finland, for example, does virtually no testing. Even the international tests being used are themselves different than our usual fare. For one thing they rest on sampling. That in and of itself makes tests somewhat more reliable--reducing the pressure to teach to them and to cheat and to abuse the resultant data.

Ed. In fact, the earliest use of standardized tests was precisely to note that Poles and Jews (and the Irish, etc) were inferior. It took the Irish becoming "white" (there's a nice book with that title) to raise their scores for the reasons you suggest. It was harder to do if your ancestors were from Africa. And still is. And it took the GI Bill to break the barriers for Poles, Slavs, Italians, etc. The biggest beneficiaries of NYC's free college tuition was for white subgroups. By the way, Ed, when I mean to be talking about African-Americans I do so--but in this case I'm referring to the correlation between income and test scores--there are still more poor whites than blacks in the USA.

Yes, Dick, I am puzzled by Diane's praise for Massachusetts' MCAS tests--having worked there for 8 years. They have for a very long time, prior to NCLB and MCAS, ranked high in education - back when there were no state tests and almost 100% local control. But if 54% of Mass. schools are failing to meet AYP today, that tells us that their tests are harder than other states but not therefore necessarily better. They still out rank all other tests and most nations on NAEP and international comparisons.

Of course, we are learning 24 hours a day and 365 days a year--of which a very small percentage is in the schoolhouse itself. Work out the math. So even with a perfect assessment system it would not be surprising that the haves" outscore the "have nots". As a parent myself I find that having advantages has been an advantage to my kids.

Best, Deb

Charter schools "cream" students (sometimes) just by the fact that someone in the family has to take the time and have the where-with-all to fill out the forms and, sometimes, provide the transportation or deal with a change in schedule and many other "changes" that might be involved ... the poorest and those with the very weakest support systems, with exceptions of course, fall by the wayside.

Sometime last year, someone on this blog cited “Campbell’s Law” in a post. Campbell’s law is a critique of how quantitative social indicators come to corrupt whatever program they are intended to measure. It was published in a paper at http://www.wmich.edu/evalctr/pubs/ops/ops08.pdf in 1975. Campbell’s Law reads like this “The more any quantitative social indicator is us3ed for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

Campbell himself pointed out that this “law” applied to high stakes testing programs in schools, even in the 1970s. It also helps explain why crime rates are readily manipulable by new police chiefs, and why Soviet factories over-produced some items and under-produced others when given production goals. It even, Campbell points out, explains why “poor Lt. Calley was merely engaged in getting bodies to count for the weekly effectiveness report when he participated in the tragedy at My Lai.” Presented with a quantitative goal, the teacher, police officer, factory planner, and soldier will all seek to shape their actions to meet it, irrespective of broader more general goals that are not so easily quantifiable.

None of this is to say that quantitative indicators are not important for program administration. However, by themselves, when quantified measures are allowed to drive decision-making, rather than simply inform it they are likely to, as Campbell writes, corrupt the very social processes they are intended to monitor. A gut level appreciation of the irony embedded in Campbell’s Law is, I expect, the source of much of the anxiety over the new requirements that states compete for money based on whether they use student test scores to evaluate teachers. Given this pressure, scores will increase whether or not the underlying learning improves or not. Evidence-based decision making is ultimately ok, but only ok. Quantitative measures can inform wisdom, but they are never by themselves very wise.
Tony Waters

Deb writes: "Clearly some folks--like Ed and John, like the current definition of well-educated (scoring well on standardized bubble-in tests)."

Of course that's not true. I disdain the tests as you do; they are not at all a measure of well-educated.

What they are is the best your profession has managed to come up with as a way of shoring up the bottom.

What I imagine for education is continuous improvement processes, bringing in all of the tools and memes and balances that other professions use.

Until then, we have the tests. They're a too-blunt instrument. But they're an instrument that the state education associations couldn't defeat in legislature or court.

I wonder what readers think of this post at Common Core?

Immigrants must answer 6 out of 10 questions correctly on a basic civics exam in order to earn American citizenship. United States Citizenship and Immigration Services (USCIS) reports a 92.4% pass rate among citizenship candidates on their first try. Take a look at the question bank from which the 10 questions are drawn.

How well do you think American 12th graders who were born in this country can score on the exam? 90%? 80%? If only.

The Oklahoma Council of Public Affairs recently sponsored a survey of Oklahoma high school students to see how they would do on a citizenship test. 2.8% managed to pass.

Civic ignorance isn’t limited to the Sooner State. Our own Still at Risk showed that substantial numbers of students lack basic knowledge of history and literature. And in July, USAToday called attention to a recent Goldwater Institute survey of Arizona high school students who were administered the citizenship test. Arizona students put their peers in Oklahoma to shame: 3.5% of them passed the test.

Not to fear. State legislators are busily working to hide this ignorance by getting rid of social studies tests.

James Elias

Nev -- Anything that requires parents to fill out a form will draw parents who are willing to fill out a form as opposed to parents too stupid or lazy to do even that. So what?

Campbell's Law is not a warning about quantitative social indicators. It's a warning about their improper use. Campbell's professional career was focused on the appropriate use of quantitative indicators. He and Julian Stanley demonstrated that randomized controlled experiments seldom if ever have any external validity in education,
and both men must be spinning in their graves with this design presently considered the "gold standard", with a statistically impossible formula (AYP) being used to pound the Nation's kids and teachers, with untenable "gain" scores labeled "value-added" and mandated as a factor in determining teacher's "merit pay."

Determining whether a kid can read is a very transparent matter. It doesn't demand filling in bubbles on a paper-pencil test. The same is true of arithmetic, higher math, and other academic aspirations. The reliable accomplishment of these aspirations doesn't require either state or "national" "standards," that amount to little more than rhetorical wish lists.

El-hi schooling is not about tests, it's about instruction. Standardized achievement tests provide no information whatsoever that offers a basis for modifying instruction.

A poor workman blames his tools. It's not the measures, it's their use. But that's a bit like saying, "Guns don't kill people. People kill people." The unintended undesirable consequences of the misuse of standardized achievement tests have been well-documented. When are we going to quit beating up kids and teachers (with low SES kids being pounded the hardest)?

Hi All.... hope this finds everyone well.

From the LA Times:

Budget cuts push some classrooms way over capacity

Some L.A. Unified classes are crammed with about 50 students, leaving some pupils to sit on desks or the floor and their teachers to grade hundreds of papers while still focusing on improvement.
"I'm very frustrated," Collier said. "I mean, it's a good class -- it's an honors class, and the kids are really good. But it's unreasonable to ask me to teach a class of 48 kids and give attention to everybody."

Wonder.... in the school districts that you live in.... how many of your schools have class-sizes like this???

be well.... mike

No, I don't think the sum of education is doing well on standardized test. I nonetheless think they can be an improvement. Status quo: No one measures anything about how a child does in school. All you can tell from the outside is that a teacher says, "Little Junior here got good grades," which could be completely idiosyncratic and biased. With testing: At least we can tell if a student who was supposed to have learned a bit of algebra this year actually did learn a bit of algebra, or if a student can't parse a simple English sentence.

People get irritated with testing only because they prefer a model in which no one can ever figure out that a teacher is completely incapable of doing a good job.

There, how about that as a fair summary of your position? Wait, you say, there's something to your views besides the worst possible construal that an enemy can give? Do tell.

Who are you referring to here John (Doe)? I don't recognize your "summary" as a "construal" of any views presented in this thread--or anywhere else for that matter. The paragraph is a substantive mish-mash; with some bad grammar/typos thrown in.

And I didn't know that that enemy combat was involved. The blog is about "Bridging Differences" not "Building Enmities."

John Doe. A sixth habit of mind (to add to CPE's five) might well be: "don't attribute bad motives until you've exhausted good ones"--in short act "as if" those you are wasting your time arguing with have something worth saying.

Dick and others. What we pay too little attention to is what kids DO KNOW!

Campbell's Law, I thought, was about how high stakes indicators can cease to be measuring what they were intended to measure. My years as a NYC principal confirms this. I was guilty too. e.g. To improve attendance data the system allowed (maybe required) us to take attendance before lunch instead of at the start of the day. Miraculously attendance improved.

Best, Deb

"What we pay too little attention to is what kids DO KNOW!"

Amen and AMEN. We focus on filling deficits which is a bootless endeavor. With no feedback on their instruction, teachers inadvertently confuse many kids. When the "problem" becomes transparent at a much later time, it's a "big problem" so we slap a pejorative label on the kid, and the beat goes on.

Campbell's Law deals with the uses of social indicators rather than the indicators per se. Standardized achievement tests are not designed or constructed for the high stakes decisions for which they're being used

Jim Popham has concisely described the "slippery slope" the corruption has evolved.


Deb and Dick:

I think that you both summarize Campbell's Law well. Quantitative measures which have stakes attached are always subject to corruption. This is why NAEP is a better indicator of educational achievement than the various state measures to which jobs, funding, closures, etc., are tied.

Still, there is no perfect measure. All must be put in the qualitative context of what you are trying to do, and what the quantitative measure is actually measuring. This requires judgment. Just because something is quantitative does not make it particularly scientific.

There is a good line in Campbell’s paper which points out that that “for an administrator to take this attitude of scientific tentativeness constitutes a default of leadership. Conviction, zeal, enthusiasm, faith are required for any effective effort to change traditional institutional practice.” I would add courage as being a quality of leadership. One thing that is not on the list the current fad for “data driven decision-making”. I suspect that this is because Campbell believes that decision-making should be informed by data, not driven by it.


John Doe: "Anything that requires parents to fill out a form will draw parents who are willing to fill out a form as opposed to parents too stupid or lazy to do even that. So what?"

I don't need to add anything. Your ignorance comes shining through.

Nev, I was just asking you to elaborate on why the phenomenon YOU identified (and which I merely repeated) was worth caring about.

Dick -- since you seem to have no idea what my comment was about, I was merely responding to the silly claim from Ms. Meier that "clearly some folks--like Ed and John, like the current definition of well-educated (scoring well on standardized bubble-in tests)." in other words, all you have to do is say that tests, whatever their problems, are better than having no verifiable information about a child's learning, and Meier sneers that you think being well-educated is only about filling in bubbles.

Which reminds me: Meier is a champion violator of her own advice, i.e., "don't attribute bad motives until you've exhausted good ones."

OK. JD. That clarifies what you were trying to communicate.

"all you have to do is say that tests, whatever their problems, are better than having no verifiable information about a child's learning, and Meier sneers that you think being well-educated is only about filling in bubbles."

But the thing is, the issue is not bubble tests vs. no info and Deb was not saying anything about what constitutes "well educated." You're setting up and knocking down a straw man.

The thread is "We're Relying on an Absurd Definition of Achievement." Nothing you've said refutes that contention, or even speaks to the point. You've attacked Deb and introduced irrelevancies. If you can refute what she's said let's hear it. Personal attacks and irrelevancies do not bridge differences.

Of course Deb was not saying anything about what constitutes "well educated" herself; if you can read plain English, however, she was accusing anyone who favors testing of having a particular view of what it means to be "well educated."

I haven't made any personal attacks or irrelevant arguments. All I've done is point out the extraordinary degree of bad faith in the following arguments:

1) Charter schools are bad because they don't raise test scores enough (never mind that the other 99% of the time, Deb is bashing test scores);

2) Testing is bad because it somehow resembles a mortgage-backed security (this is as silly as, at the opposite extreme, saying that testing is like the NASA project to send a man to the moon, because both relied on data).

No one has any response to my points, because they're obviously true.

"No one has any response to my points, because they're obviously true."

That's not why "no one" has responded to your "points," JD.

Alright, wise guy, defend (if you can) the 1) use of test scores to bash charter schools right in the middle of a piece decrying the use of test scores; and 2) the inflammatory and completely inaccurate comparison to the financial markets.

Readers. You know it's better to get a heated response than none; and besides beneath Doe's belligerency, he has a point. My friends and I often struggle over whether it's legitimate to use test scores to make any point at all. I think it is. It's a fair response to those who think test scores are the best/only/preferred/least-evil et al measure.
If Duncan uses them to attack one set of schools and suggests charters are the answer to their poor performance he is just plain factually wrong. They are not an answer to low test scores. I'm happy to think we might agree on this. Are charters useful for other reasons? I thought so until they became "the answer" and mis-stated their test data to attack regular public education rather to bring innovative ideas into more prominence and offer radically different approaches to families interested in these--and from which we might learn.

As for the misuse of data by Wall Street and by Educators: when the public's view of how well businesses/schools are dependent on data that few understand or can check out, they both suffer. The "bottom line" data thinking at Wall Street is short-sighted and self-interested. The DOW goes up and it means all is well?? Scores go up and we cheer? It's why Ponzi's work. It's not a perfect metaphor, but it has merit--it resonates, or something like that.

I try to interpret criticism as having the best of intent--hoping to persuade me or at least to correct me. I learn as much in that process as I do from those who agree with me. But, let's try to modify our criticism of each other's intelligence or honesty. That's a critical rule in democratic debate, which we don't hve a strong inclination for in the U.S.: attacking ideas vs attacking the people who hold them. There's a thin line between these, of course.


OK, it's a fair point to say that even while you don't accept test scores as the measure of a school's worth, people promoting charter schools shouldn't promise too much test score improvement. That argument at least makes sense, even if, in the end, it's factually wrong. The best evidence actually does show that charter schools do help increase test scores and chances of graduation. See http://www.nber.org/~schools/charterschoolseval/ and http://www.ncspe.org/publications_files/OP169.pdf and http://www.tbf.org/utilitynavigation/multimedialibrary/newsdetail.aspx?id=9490

But it still leaves behind a mystery: Given that you don't accept test scores as the sole measure of worth, why don't you like plenty of other things about charter schools? Many of them focus on helping at-risk kids -- a bad idea? Many of them are aimed at kids in impoverished inner cities -- say, KIPP. Many of them are aimed at kids who have interests other than reading/math test scores -- say, arts-based curricula. With your supposed values, why aren't you in favor of all this?

On another note, the comparison to the financial markets still sheds way more heat than light.

That's a sensible point, even if factually wrong. That is, it at least makes sense for you to say that charter schools are being oversold on the basis of test scores. It's still a false premise, though, because the best studies -- Hoxby's new study out today, Booker/Sass/Zimmer/Gill, Kane/Angrist -- show that charter schools are likely to increase test scores and chances of graduation.

But that still leaves the gaping question: Why aren't you a big charter supporter? Leave aside test scores. Many charter schools focus on helping at-risk kids -- bad idea? Many focus on helping inner-cities (such as KIPP) -- bad idea? Some have a mission of serving kids who aren't obsessed with reading/math tests, such as arts-based schools -- bad idea? If your claimed values are sincere, you of all people should be broadly in favor of charter schools.

"They [charter schools]are not an answer to low test scores. I'm happy to think we might agree on this."

Count me in on that agreement. In part, this is due to the inherent characteristics of standardized achievement tests constructed using Item Response Theory. In part it's due to the fact that charter schools use the same instructional products/protocols as other schools. And they have no better (or worse) feedback information on the cumulative status/performance of students in acquiring expertise in reading and math.

There is wide variability in charter schools ostensibly following the same "model." (e.g. KIPP) and there is frequent mobility out of Charter Schools--back into public schools.

The thing is, aggregate parents are not dissatisfied with the school their child is attending. The child is a reflection of the parent(s) and the school the kid is being sent to is also a reflection of the parent(s). Few parents are willing able to admit that they are a "bad parent" or that the school is a "bad school." Aggregate schools do a neat job of attributing all instructional shortcomings to the kid, parents, or "society." So parents go along to get along. (There are exceptions, of course.)

School districts have had "magnet schools" "special schools" and such for a long time. Many school districts have "open enrollment." And there have been oodles of "school experiments," coalitions and such over the years. All of these have been better defined than current "charter schools" that run the gamut, with the characteristics typically defined by the selection of students and/or school personnel.

Charters were once hailed as a way to get around "union strangleholds" but charter schools are unionizing and unions are running charter schools.

Charter schools were hailed as examplars of "free market principles." But "free market principles" have been pretty much discredited and abandoned.

But with Secretary Duncan dangling what appear to be "big bucks" in front of the noses of state and local officials who sign on to establish more charter schools, more charter schools will be established.

But that brings us full circle to Deb's post: "We're Relying on an Absurd Definition of Achievement."

Okay. I took the bait.

Dick Schutz writes:

"Charter schools were hailed as examplars of 'free market principles.' But 'free market principles' have been pretty much discredited and abandoned."

Any entity funded via tax money is not a free market organization to the extent it relies on taxes (coerced money) for revenue.

It is this fallacious 'discrediting of free market principles' that foments the belief that market relations have destroyed the economy. It is quite the opposite, however. Some call it government failure.

Voluntary exchange based on property rights is a core basis for civilization itself. Any other economic relationship implies subjugation. Surely you don't support slavery?

I am curious, then, as to how you would go about a refutation and who would you cite in referencing your argument?

There are many unsupported and misleading statements in Dick Schutz post. Fallon handled one of them. Here are few more:

"due to the fact that charter schools use the same instructional products/protocols as other schools. And they have no better (or worse) feedback information on the cumulative status/performance of students in acquiring expertise in reading and math."

Incorrect. Charters can select any instructional materials they want, including materials that are unadopted by the state in adoption states. Many (most?) of them take advantage of this flexibility. Further, they can introduce variety of assessments w/o running into union rules or ed-code limitations, and they have no limitations on using assessment data to effect both instructional and personnel decisions, as many public schools do.

"All of these [magnet schools] have been better defined than current "charter schools" that run the gamut, with the characteristics typically defined by the selection of students and/or school personnel."

Incorrect. Both magnets and charters have equal ability to define themselves, but because charters need to get chartered--often by hostile organizations--the definition of their characteristics tends to be clearer. As to selection of students by charters, it is the one characteristics that does NOT define charters, as the student select them, or lottery must be held in case of oversubscription. This is not necessarily true for magnets, whose policies in case of oversubscription can vary from lottery, through racial considerations, and to qualification testing.

"Charters were once hailed as a way to get around "union strangleholds" but charter schools are unionizing and unions are running charter schools."

Incorrect. Charters were hailed by SOME as a way to get around unions. By many others they were (and are) hailed as a way of getting around education codes with their stultifying rules, and as a way to promote creativity, initiative, and accountability. Finally, while unionization is not a focus of charters, just a few of them are unionized. But, at least, unionization in charters is a choice of the school's staff rather than imposed by state laws or district policies.

Aw c'mon JD.

Read Hoxby's latest report on the NYC Charters.


The Charters are using the same genres of texts and tests that schools throughout the country are using. When you scrape away the fancy derivative statistics, they may nudge the distribution of test performance up a tad, but she has to struggle to get statistical significance, let alone practical significance.

A charter school does try harder. The NYC report provides useful information about the ways they try. All of these can be found in US schools around the country. The variable that seems to have the most robust effect is "time on task." But this has been well-known for 100 years.

Hoxby emphasizes that charter schools vary greatly. So do public schools. The el hi enterprise is running amok. Why? Read Deb's post. "We're Relying on an Absurd Definition of Achievement." For further reason, read Diane's subsequent post, "The NCLB Paradox Enters the Twilight Zone."

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments