« The Proper Goal of Schooling | Main | Why We Need National Testing »

Mixing Politics and Testing

| 21 Comments

Dear Diane,

You suggest I needn't worry about annoying those "with more power". But I felt badly recently when (as I mentioned) somebody took after Mission Hill school as a way to attack me on another issue altogether. So they can "touch me"—but not stop me! Alas, my travels remind me that others have less wiggle room—even for saying what's on their minds.

You are right, our disagreements seem to lie in at least two places: (1) the role of standardized tests, and (2) national curriculum vs. local ones. Even in these two areas our views overlap considerably. Let's start with standardized tests.

When confronted by a disagreement between the judgment of a test vs. that of someone who knows a child well, I tend to rely on the latter. That's based on the experience of sitting down with kids and going through items one by one. I was shocked. Their wrong answers were often a sign of intelligence, possibly misapplied, not a sign of reading difficulties. (Read Chapter 6 of "In Schools We Trust".)

It's also based, however, on the arguments made by Steve Koss, in the Sept. 5th piece you sent me ("Tell me the results you want, and I'll find a way to make numbers show it."), who quotes test experts regarding the way these tests are designed. Since few understand p-values et al parents and laymen often believe that the scores rest on unquestionable assumptions.

But today we face a third dilemma. Standardized tests are no longer scored on a so-called national normal curve, but scored "politically." This is not an accusation but a fact. The level of item difficulty as well as scores rest on the judgment of an appointed group of people. This leads, for example, to scoring a 6th grader on how many right answers such "experts" believe he/she "should" or "could ideally" answer right rather than (as in the old days) scoring the test based on how many the median 6th grader does answer correctly—the center of the "normal curve" of scores. This led in the old days to a predictable curve. Today it leads to anything we want.

Opinion-wise—or otherwise—has replaced statistics. That's all I mean by "political" norming. At its worst it leads to the results the New York Daily News exposed with regard to N.Y. state's math test having been made easier this last year. It accounts for test scores all over the country that bear no resemblance to each other as different authorities make different decisions about how hard a test should be and how many right answers are needed. All test-makers know ahead of time how easy or hard it is to get the predetermined right answers for every single item. The selection of items decides the scores. The final scores are no surprise to them; if they were the test-makers would be incompetent.

The old form of norming is what has made such tests seem incorruptible and scientific. It led to other flaws (such as the fact that scores could never—en masse—go up or down for long—without renorming). The new system is living off the reputation of the old style tests, while in fact it rests on a much messier form of human judgment.

I like that messy form, but want to reserve it for when it's a sensible response. The kind done by individual teachers (grades on papers and exams done for a class, etc.), or by panels of experts of various sorts looking carefully at individual performances both acknowledge to being less and more than "scientific". The final "score" acknowledges this by being "signed off" on by real-live names (to whom one can appeal). Which is why the criteria for assessment is public and transparent. Such approaches (even PhDs) are much closer to the way we go about making decisions in "real" life; they respect the power of human reasoning in ways that standardized tests—normed or otherwise—cannot. But each has its flaws.

Some combination of these forms, including the judicious, occasional use of standardized norm-referenced tools (or a third variant we have yet to invent) could serve us well, with none having absolute power over the other, while all together inform our judgment. The argument for a more nuanced system of assessing young people shouldn't seem like anti-testing dogma.

How far apart are we on this, Diane?

Why do I fear lists of state-imposed required songs, historic dates, book titles, etc.? You have more faith than I do in our capacity to keep these "common" lists short. My experience suggests nothing ever gets taken out and much gets added in. I suspect we also disagree on how hard some things are to "learn"—and maybe what we mean by that. Besides I have an appalling rote memory, so memorizing the Gettysburg Address would require eliminating a lot more from the curriculum than it might for you. And finally, we may disagree about how likely they are to be used well vs. abused badly.

But there's no question that the folks making decisions for us now, at least in NYC, neither know the kids well, know much about what is easy or hard to learn, have never studied psychometrics, and haven't thought long and hard about the trade-offs involved in trying to create a form of education that serves democracy well. And they are not embarrassed about it.

Deborah

21 Comments

I would love to know (at some point) what each of you know and think about public school reform in Venezuela..

http://www.venezuelanalysis.com/news/2616

I keep reading reports to the effect that about 1/3 of kids entering college require remedial work. USDOE/OSEP's two national longitudinal studies show that there is almost a "zero correlation" between the grades that teachers give kids with disabilities and these kids' objectively-assessed reading and math scores (and most of these are kids with mild disabilities in regular classes). Then there's a NYS Comptroller audit, done back in the early '90's, showing that adults changed kids' high school test answer booklets to make marginally failing scores into marginally passing ones. This was way before NCLB.

So if we don't have some decent national test to use as an objective measuring rod, we wind up with subjective passing grades that apparently don't mean a large chunk of kids can do entry-level college work. They wanted to, which is why they applied to college and were evaluated as requiring remediation in the first place.

Perhaps one of the recommendations from the NY Campaign for Fiscal Equity litigation should be adopted on a state-by-state basis: an independent entity should be established to create tests, administer them, score them and do accountability reports. Like everything else in life, this process could certainly be corrupted, but at least the people who profit or lose by the testing and reporting system wouldn't also be the ones who ran it.

If this sounds like mini-NAEPs ... so be it.

Deb,

A great example of the politicization of the test results could be seen following the release of Chicago's high school test scores this week. Six years into the Mayor's Renaissance 2010 program, high school scores took their biggest tumble in 5 years. Some "Renaissance"!

Chicago public school officials said CPS scores from day one of the test, which is solely the ACT college admission test, were up. So they blamed the high school drop on day two, when reading and math skills needed in the workplace are tested.

CEO Arne Duncan responded to the drop, in today's Sun-Times, by calling on the state to change the test, "or do away with it.''

If they do either, look for scores to go up next year. They usually do in election years.

Dear everyone above,

Philip! Try again--there currently is no message in your comment space.

Tom. I know nothing about Venezuala, but does anyone else out there?

Mike. One could use your story, Mike, to claim that Chicago is an example of a nonpolitically normed test--or not a very powerful mayor (which seems unlikely). Is this re the state test? But of course, as Diane notes, test scores in mass don't actually go up and down this rapidly. What's amazing is how little technical information we get anymore on tests. In the bad old days (and I mean that) tests came with publicly available (every teacher got one!) manuals explaining how they were normed, what the reliability formula was (.e.g variation by random chance), etc, etc. Now this kind of information is a state secret.

Dee. Since schooling was invented every teacher and institution complained about the shape of the students they "got"--they always needed remediation. Read Rothstein on what Harard said about its freshman in 1898! I can offer you tons of examples of kids who failed NYC college entry tests (after completing high school) where the same students did just fine in the same college's actual courses! International High School pioneered that data.

I think the human race needs remediation. That's probably one reason I became a teacher, to see if I could figure out how we could use those 12 years better so that we'd all be less ignorant, foolish and narrow-minded. I think we're still a long way from the changes we need to make, and probably getting further away daily.

But, I think, Dee, that we base our conclusions on different forms of evidence.

Deborah

no message...just a link to a story about a Berkley study showing that grades determined college success better than SAT scores...of course...they only looked at 80,000 students...

Deb has written:

"When confronted by a disagreement between the judgment of a test vs. that of someone who knows a child well, I tend to rely on the latter. That's based on the experience of sitting down with kids and going through items one by one."

Though it might surpise some I tend to agree with her. I tolerate MC tests but have no love for them but they are a necessary evil.

I have long experience teaching ELL(English Language Learners) and the vocabulary of standarized tests is very difficult for the. Of particular difficulty is the fact that to avoid the appearance of 'wrote learning or strict memorization' the book tests use a vocabulary and language which IS NOT PRESENT in the course materials. This might not presnet problems for native speakers but it causes havoc with learners. But I have made my peace with local bench marks. I administer them so that they have practice but do not count them as part of the student's grade. I give the students tests that are srictly written so as to reduce the language problem for students. I also test student viva voce after school (which is a form of tutoring).

All my students must take high stake state tests.

I encourage them to do the best they can and to consider them learning experiences.

The thing to do is teach beyond these tests. They are only dipsticks.

just a link to a story about a Berkley study showing that grades determined college success better than SAT scores

You are getting confused by the restriction of range effect. When a college is selective with respect to SAT scores,there is a restriction in range of student SATs, so naturally other factors will be more pronounced predictors of academic success. In contrast, when colleges are not selective with respect to SATs, the SATs are good predictors of academic success.

Since schooling was invented every teacher and institution complained about the shape of the students they "got"--they always needed remediation.

It is one thing to complain that your calculus students are a bit dim, it is another thing when your calculus students are so dim that they can't pass a test of basic algebra and need to take a remedial algebra class as a result before the calculus teacher gets to call them dim.

Riccardo, thanks for reminding us of how this works for young people who are not fluent in English. It's an exggerated form of what happens to all test takers, except those who speak and think most like a test maker. It ought to be, at least, a caution in designing policy.

KDeRosa. Actually Harvard Univesity complaint in 1896 was much like yours: "...only 4 percent of student who applied for Harvard admission could write an essay, spell, or properly punctuate a sentence." And in 1898, the University of California found that 30 to 40 per cent of entering freshmen were not proficient in English." A hundred years later the U of C was still complaining about the declining competence of its students. (From Rothstein, The Way We Were) Rothstein couldn't find a period of history in which the complaints differed--anecdotally or otherwise.

Deborah, professorial bluster notwithsatnding, I think the real measure remains actual remediation ratesor some other plausibly objective standard, such as the famous decline in verbal SAT scores in the early 60s which occurred at the top of the scale, i.e., in Harvard land.

Is professorial bluster different than other kinds? I wondered since I;m actually am not and never have been...a professor.

I don't know whether remediation rates have gone up or down. I assume that when most kids weren't even graduating high school (when I was born most hadn't even started high school) they probably would have needed far more remediation at 18 than they do now. I honestly, K, don't know what to compare the kinds of comments or data that you have in mind to.

But we agree: most people are woefully ignorant. At least about some things--like the stuff that college professors teach and probably about a lot more.

Deb,

No rush but - still waiting for your reply on affective/cognitive domain balance for minority and inner city youngsters. Am vey curious what you think about this.

Hello all,
I teach in an inner city school in CT that is supposedly a "failing school" and of course we are made to feel like failing teachers as well.
The problem is that when parents are given a choice, they remove their child from our school ( most of the students who pass the State tests are the ones who leave.) We are left with the "failing" ( not myword) students. Will our scores ever go up?!
Be that as it may, we also have a great deal of students with ieps who are mandated to take the state test even though they cannot read it!!!!
This makes no sense at all!!! They are getting serviced on their level but still have to take the test!!!
The result is dismal.
Even so we are not making "cookies" out of the same mold. We need someone in the government to step up and see how unacceptable this is.
What about a "growth model"?
What about looking at the whole child?
Yes I agree there has to be some standards on which to measure students but lets be real and look at the whole child.
The worst part is, they keep raising the standards.

Hello all,
I teach in an inner city school in CT that is supposedly a "failing school" and of course we are made to feel like failing teachers as well.
The problem is that when parents are given a choice, they remove their child from our school ( most of the students who pass the State tests are the ones who leave.) We are left with the "failing" ( not myword) students. Will our scores ever go up?!
Be that as it may, we also have a great deal of students with ieps who are mandated to take the state test even though they cannot read it!!!!
This makes no sense at all!!! They are getting serviced on their level but still have to take the test!!!
The result is dismal.
Even so we are not making "cookies" out of the same mold. We need someone in the government to step up and see how unacceptable this is.
What about a "growth model"?
What about looking at the whole child?
Yes I agree there has to be some standards on which to measure students but lets be real and look at the whole child.
The worst part is, they keep raising the standards.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

just a few comments bcause apparently the other on didn't go through??? The educational system has beeen destroyed already by politians,i.e. Mayor Bloomberg, the lawyers that are making educatuional decisions, and the insane NCLB. I think education should never have been allowed to become a political footbal..legacy of Mayor Guillani. Lawyers need to do their own profession. Educators need to make auton
omous decisions that are educationally sound which rests on years and years of sucessfull practices: not decisions made by educrats which are politically expedient. They have been dummying down education now for years, trying to make themselves look successful. They Should be held ACCOUNTABLE.

Comments are now closed for this post.

Advertisement

Most Viewed on Education Week

Categories

Archives

Recent Comments