« What NAEP Long-Term Trend Scores Tell Us About NCLB | Main | What 'The Harlem Miracle' Really Teaches »

Why Not Strengthen the NAEP as a Tool for Seeing Trends?


Dear Diane,

I think it’s unlikely that the fans of mayoral control are open to persuasion. But thanks, Diane, for relentlessly pursuing them. We’re in for lots of nonsense in the name of reform.

Arne Duncan is planning a contest for a new name to replace the unpopular No Child Left Behind. It’s the name, apparently, that he sees as the problem. Actually, it’s the only thing I like about the bill! Meanwhile Sharpton and Klein and Co. (EEP) are planning a rally in D.C. to end poverty (hurrah), by…"closing the testing gap.” And, careless reporters (see May 5th AP story by Libby Quaid) claim that NCLB has had success at doing so.

It was interesting also to read James Forman Jr. in the May-June issue of Boston Review (a publication worth getting). He explores the “gap” between those who want to focus just on schools to create equity (EEP) and those who see health and income gaps as part of the problem (Broader Bolder). But the heart of the article is about KIPP. He connects their work to that of CPESS and "The Power of Their Ideas"! The coupling was intriguing (and flattering) and relates to the theme of this letter: the confusing landscape of reform, reformers, and their messages!

The “gap focus” probably accounts for the renewed interest in early-childhood education. But, alas, it usually translates into just starting traditional schooling at an ever earlier age. Sunday’s NY Times Magazine had a piece ("Kindergarten Cram" by Peggy Orenstein) about what’s happening to our kindergartens—and I might add—to nursery schools and home life, too! And she’s talking about the elites! It’s not a new subject—David Elkind wrote "The Hurried Child" 20 years ago. We’re obsessed, pressing children at younger and younger ages in the name of raising standards, and now closing the achievement gap! So it was nice to read Orenstein’s sensible comments about the topic, based on a very important study done by Joan Almon and Ed Miller at D.C.’s Alliance for Childhood. Read it—and weep.

Alarm over an endless list of recurrent crises—some real and some questionable—is the style of the day. Unfortunately, being impatient is bad for child-rearing, and bad for society-rearing also—as Elkind reminded us. The latest issue of Educational Horizons (the magazine of Pi Lambda Theta) has a short piece questioning the significance of another crisis: the U.S. “lag” in TIMSS international math scores and one on how deciding to focus on four-year high school graduation rates has created a heightened “crisis.”

False alarmism is as dangerous as false complacency, I’m discovering. (Maybe it’s just my age?) Like red alerts, false alarms lose their sting.

I thought of you last week, Diane, at the Education Writers Association sessions in D.C. The one I participated in was largely focused on national standards. I found myself resisting the PR-induced consensus that is being invented. It’s an area where you and I have historically disagreed, but I suspect you’re a little suspicious, too.

The Constitution always requires reinterpretation and sometimes even amendment. But I’m an “originalist” in legislating school reform. The odds of harmful interference are just too great. And the thinning out of all local institutions is worrisome.

Of course, I have supported some shifts toward centralized decision-making re. schooling. Segregation was a cause important enough to err on the side of state and federal intervention. So was unequal funding for low-income schools. But much as the Boston integrators are among my heroes, the impact of their sweeping mandates—eliminating all neighborhood schools—did not turn out as they hoped. There are virtually no integrated schools in Boston today. In so far as NCLB was an equity law, the same might be said of it. In neither case did we pay much heed to those on the ground in considering how it might play out. I feel the same about national standards. We may unwisely rush into this, under the battle cry of “crisis” and end up reproducing the worst of NCLB.

I think if we do not want or believe it necessary to dumb down the best of schooling in the interest of equalizing it—there are wiser alternatives. I’d like to roll out a few. For starters:

1. We could strengthen the NAEP—as a tool that helps us see trends; that notes significant differences between states, regions, and subgroups; and that can put the spotlight on particularly interesting items that deserve closer attention. Especially re. literacy where, post-4th grade, there is perhaps 99 percent agreement defining it. We could go back to some of the ideas that old-timer Ralph Tyler had for NAEP —interviews, focus groups, projects, etc. that could provide more insight than bubbled answers can. My discovery that many 3rd graders who scored poorly on reading tests would do no better if the passages and items were read aloud is, for example, worth pursuing. As was my discovery that some wrong answers suggested good reading. Such an approach, based on sampled populations and using sampled items, can avoid high stakes use while also providing richer national data. (We could require states that want federal funds to join NAEP?)

After that it gets harder, so I’m going to roll out the next four ideas next week!


P.S. Our perennial disagreement about NAEP largely regards the wisdom in setting so-called benchmarks—and the labels attached to them. Otherwise, the exams have some of the limitations of any multiple-choice standardized instrument—but not its gross high-stakes misuse as a measure of individuals.


Good idea, Deb. But the thing is, NAEP is too flawed to be strengthened.

"Tyler took on the job of designing the assessment measures for the National Assessment of Education Progress (NAEP), which are federally mandated criterion-reference tests used to gauge national achievement in various disciplines and skill domains."

NAEP devolved rather than evolved from Tyler's conception. When "proficiency" is defined in terms of arbitrarily-set cutting scores on ungrounded statistical scales, the assessment clock is striking 13.

One has only to look at the NAEP categorization of "Reading" (Extracted below to see that, as defined, reading is not teachable. And if one tries to assign the labels to individual test items, it's a hopeless task.

(The same holds true for math.)

As you point out, there are many ways of investigating instructional matters. Even examination of NAEP item data produces much better Intelligence. The racial, SES "gap" is not the only reliable divergence. There are gender and demographic gaps. But all of the "gaps" are totally overwhelmed by the regularity of kids' bubble selection.

"National standards," "charter schools," "merit pay for teachers" "value added testing," and the fiction of "turnaround schools," coupled with "a race to the top" constitute tinkering.

Meanwhile, solid IES evaluations, showing that Reading First has had no impact and that the BEST 5th grade "Comprehension Instruction" has NEGATIVE impact are being overlooked.
Comprehends What Is Read

Comprehension is an interactive process by which the readers construct meaning both from the passage and from the various kinds of background knowledge they bring to the reading experience.
Comprehending what is read includes identifying specific information from texts as well as making simple inferences about the meaning of a text for a particular purpose.

Interprets What Has Been Read

Readers use a variety of skills to deepen their understanding of what they have read. These skills include relating the concepts to their own experiences, to other works they have read, and to their own initial reactions to a text.

Evaluates What Has Been Read

One part of a reader's reaction to any text is a judgment or evaluation of its usefulness or quality. Evaluating a text may include making informal judgments about a piece or making a critical evaluation about the success of a work based on generally accepted criteria for successful writing.

Analyzes What Has Been Read

When they analyze what they have read, readers may clarify their initial interpretations by explicating or explaining their views to others. Analysis can involve tracking the logic of the author's argument, identifying the emotional appeals underlying a political statement, explaining the motivation of a character in a story, or tracing the causes of a sequence of events.

Provocative essay, as always. But I wanted to call attention to a leap of logic that I believe hides behind much of the disagreement over testing.

You mention the “gap focus” accounting for "renewed interest in early-childhood education," then say that it "usually translates into just starting traditional schooling at an ever earlier age." You go pretty quickly from there to Elkind's "The Hurried Child."

Why does "traditional schooling" (by which I would assume you mean subject- and content-based learning) have to be so onerous?

I would argue that the problem with "traditional schooling" today is that it has simply lost touch with core subjects and is thus no longer comfortable with them. Thus the need to be "competitive" and to "teach to the test," to "push" your child, etc. Thus your scepticism about starting traditional schooling "ever earler."

Such tensions do not need to exist -- and would not exist if we simply embraced content early and taught children to embrace it. Practice not only makes perfect, but continuous practice at learning makes learning easier -- and more fun. If our tests were equally substantive -- and connected to a rich curriculum -- and equally familiar, we would go a long way toward elminating the problem of teaching to the test -- and taking it.

Deb, It is good to hear you talking about strengthening NAEP. I actually have no opinion on that worthy goal, save a general fear of things national. Yet to hear you call for strengthened testing has a certain ring of relief.

For too many years now, the focus of all street talk educational has been: if we could only get rid of the testing, if NCLB had never come along, if the evil George Bush hadn't been elected, education would be fine.

Its easy to get carried away with the details of what testing should do. For me, it should be an internal thing, created and sustained by the teaching profession in furtherance of its due diligence/best practices obligation. Let 3rd party groups like ISI create specialty studies if they want to examine our national awareness of history or compare our math ability to students of East Timor. Otherwise, we don't hear about whether software engineers are now using RSpec or Cucumber or whatever, why should we fight endlessly about evaluating schools?

What we needed the tests for was a legal instrument. To separate under the law those schools who were perennially not teaching kids to read, not teaching them to find 1/3 lb of lunchmeat on a digital scale. No one expected that legal instrument to be spot on; the idea was to at least give us power to make changes in some places; without every other school demanding equal resources. It also gave us leverage against schools who simply did not want to change.

No one expected tests to tell us if Johnny can out-synthesize Boris. Yet, that is much of the type of discussion which generally prevails in ed forums.

So, lets focus the testing to the end it was meant: getting help to the neediest.

Whatever testing you as education professionals want to do to focus that aid, be it to 1200 or 1700 schools; we don't care, so long as the first 1000 are really getting the attention they need.

While Ed has a fear of all things national, I tend to fear many things local. Personally I wouldn't mind a use of sampling in place of universal testing, IF we could ensure adequate numbers to be able to identify which kids (by building, district, group) are getting shorted on a regular basis. To my mind, such tests (used for accountability purposes) need not be incredibly specific. Such tests, in an atmosphere of competence and commitment to responsibility should provide sufficient rough guages to determine where resources (both financial and otherwise) are most needed.

What I have found to be truly frightening about the recent experience is the uncovering of both incompetence and lack of commitment. And by incompetence I am speaking less with regard to individual teachers (although not totally negating this either) than to systems. I think that these things have combined to bring about massive efforts to "game" the system (forgetting that it is children's lives that were are playing with) and buck passing--in the form of moving everything earlier in the grades (Senge terms this tendency a "learning disability" in the context of learning organizations)--or bemoaning poor parenting, inadequate nutrition or lack of health care.

Moving everything earlier in the grades has simply resulted in the kind of "mile-wide, inch-deep" content standards that Schmidt has described. We tend to pack way too much into a grade level, not teaching to mastery, in the hope that the kids who didn't get it this year will catch on in next year's iteration. I loved reading Singapore's math standards, because I, as a parent, could understand what was to be taught, and kids understand. And the truth of the matter is, many elementary school teachers have no more math instruction than I. And these are the people who are charged with responding to low scores. What do we expect them to do with that information. Certainly they break it down and review it by items (in our state this is pretty much possible--although we no longer release ALL of the items every year--due to the expense of creating all new ones every time the test is given). But, without a better understanding of what those huge volumes of standards mean--or some clearer standards that provide guidance in achieving mastery in fewer things before moving on, the move it earlier (and ever earlier) trend makes sense.

I absolutely agree with the folks who point to the necessity of strong teacher-developed formative assessments. I do not see a pervasive understanding of what this means, or looks like. I see instead desparate attempts to bring formative assessment in by buying smaller versions of the state assessments to assess quarterly "progress." Sad to say--these are the local responses.

Large scale accountability testing SHOULD be a key to enabling responsibility-based freedom to determine how the accountability measures are to be achieved. What I have see instead--at the local level--is rejection of accountability, or the accountability measures, calls for lowered expectation levels (for those poor kids who cannot make it), and lots of self-sabotaging (cutting out arts and recess to improve math, dulling/dumbing down the curriculum to aim at some weird notion of "teaching to the test").

It's hard to find allies, Deb, of the sort who understand what it is that you are talking about. I spent some time yesterday poking around for information on some "turn-around" schools in Chicago--and what they had done. What I found didn't deal a lot with strategies. It did deal with commitment. The strategies are there--they can be found, and learned, and taught. But without that commitment, that belief that every child can learn and deserves to learn--well, it's just to easy to make any endeavor a self-fulfilling prophesy.

Here is a simple assessment or reading which compares reading progress; it doesn't take an expert in covariance analysis to figure out that school 3 needs help.

While school 3 had a little more help from post-Katrina realities, I'm pretty sure we could find similarly needy schools in Detroit, East LA, and the Bronx, schools with staff not ready to meet the kids' early instructional needs.

How do we get the average teaching professional across the nation to recognize these kids' needs, to open their eyes to the 50-80% dropout rates, to stop tossing the blame over to health care, and start looking at the structure of the system?

I like Deb's peek at the Finnish pov on childhood. Yet lets also remember this when comparing at the country level: The entire population of Finland could be quartered along the I71 corridor inside Ohio. In reality, Finland spreads over an area the size of Ohio, Texas, and Florida. In all, they have but 1/60th our population.

National solutions to us are gonna look very different.

All of which is to say, perhaps, that I should be writing code; programming and developing learning/assessment content, not jabbering empty words here.

Joel Klein's latest nonsense ... in case you wanted to know: http://www.huffingtonpost.com/joel-klein/transforming-the-teaching_b_200616.html

Hi All... hope this finds everyone well as we begin the push toward the end of another school year.

Deb... just today was re-rading your bool In Schools We Trust and wanted to thank you for putting forth alternative visions of what schools could be.

My background has been working with troubled kids and as i re-read your work it certainly reminded me of the conceptual lense that i have found most helpful in the work i do...

THE CIRCLE OF COURAGE- developed by Larry Brendtro, Martin Brokenleg, and Steven Van Bockern 1990

The cirlce come from a Native American approach to child rearing and is divided into 4 quaderants:

BELONGING.... attachment

The circle is sacred and suggests the interconnectedness of life.

Done well...i certainly have seen the circle lead schools toward "an ethos hospitable to the promotion of learning" for everyone who is in connection with the schools community.

As i was re-reading your work.... listening to your thinking it was fairly easy to put your work into the Cirlce!

Thanks again from a long time reader of your work....

be well... mike

Hey.. a few other thoughts concerning school transformation after re-reading Deb.

“You can’t mandate what matters.”
( Fullan, 1997)

Trust is the “connective tissue” that holds improving schools together.
Trust is the “social lubricant” that makes difficult work possible.
Bryk & Schneider found:
Schools with a high degree of “relational trust” are far more likely to make the kinds of changes that help to raise student achievement.

Trust First vs Vision first
(Thomas Sergiovanni,2005)

Establish trust first
2. Set a vision
3. Develop a strategy
4. Move to action
5. Return to vision and strategy to modify in light of:
a) what works and what doesn’t
b) what assumptions are valid
c) what core values are compromised
6. Use accumulated “trust” to forge new strategies for improving effectiveness.

12 Healthy Cultural Norms
(Saphier & King, 1985)
High Expectation
Trust & Confidence
Tangible Support
Reaching out to the knowledge base
Appreciation and recognition
Caring & humor
Involvement in decisions
Protection of what is important
Open & honest communication

Wonder... in all the efforts to do things to schools how much time is spent in the area's that can really lead to deep change?

be well... mike

Ahhhhh... the debate over federalism... the shared power/authority of the national, state, and local governments.

1. Several states created "county" school districts in an effort to strike balance amongst the widely varied financial resources of the several communities within each county. This was touted as measures for "equity of opportunity."

2. Then the national government involved itself to try and bring "equity" of opportunity to the various "disadvantaged" parts of the nation... cities experiencing white flight, Appalachia, the South.

3. Then parents, who felt their childrens' special needs were not being met, pushed at the Congressional buttons and the courts. More national money. Along with it, more bureaucracy.

Is it appropriate that one child's LEAST RESTRICTIVE ENVIRONMENT costs the taxpayers $300,000 per year and $8,000 is spent on the next door neighbor's child?

Maybe the national government should expend resources for such extreme needs. However, to continually increase the national education budget with more of its red tape to monitor the monitors.... seems wasteful.

The larger the community, the greater number of competing interests... all jockeying for limited financial resources. Each will continue to create yet another "reason" why their particular special interest is MORE deserving of a greater piece of the pie.

Until communities once again, hold their schools up as beacons intended to outshine their neighbors (Hobbes clearly defined humans as having a need to compete for power, territory, and resources) there will be little progress.

The national government should issue standards. Each community should figure out how to accomplish them. Hold the mayors accountable - many of them want to run charter schools. Maybe that would increase community involvement.

Case in point:

I teach in an extremely large district (its size, being amongst the 10 largest in the nation, with over 250,000 students). We are a vacation destination with many large corporations headquartered here, or nearby. We were once a "bedroom" community for a larger city.. but are now swiftly becoming urbanized. On average... attendance at the school board (who are paid full-time salaries) meetings is 20.

Previously, I taught in a rural school district with 1 high school of roughly 400 students, a small feeder middle and 2 elementaries. The sons and daughters of mostly chicken farmers.
Average attendance at school board meetings (who are all elected volunteers).... 45

Which school district, do you think, had the following test for 4th grade elementary science?

(I am sure I have left out several steps.... but you'll get the idea)

1. Cut stalks of celery into equal lengths.
2. Weigh the celery and record it.
3. Take an empty cup and weigh it.
4. Put water in a cup. Measure the amount of water.
5. Weigh the cup once filled with water.
6. Predict what will happen if the celery is placed in the cup full of water. Write down your prediction.
7. Based on your prediction, what should you do before putting the celery in the cup of water?
8. Ok... try it.
9. What happened?
10. Do you need to try it again? Why, or why not?
11. Weigh the cup with the remaining water and the celery.
12. Lay the celery out to dry overnight.
13. Repeat the procedures again the next day.
14. Write down what you discovered. (2-page lab report - spelling and grammar count)

I am all for the national government creating a set of standards by which each community can measure the effectiveness of its schools. However, the responsibility for change is that of the smaller community, not the national government.

By the way.... a year ago.... the two districts had EQUAL numbers of national merit finalists.

When you play games, you really thought about the Department of your equipment well enough, your account than others, your gold enough ... There are better! What you want to have here.come on..Let's go!

When you play the game, you really think you equipment well enough, your account, you than other gold enough... Have a lot! Do you want to come in. Let's go!

Comments are now closed for this post.


Most Viewed on Education Week



Recent Comments