« Is Arne Duncan Really Margaret Spellings in Drag? | Main | What About 21st Century Skills? »

Confusing Test Scores With 'Being Well-Educated'

| 31 Comments

Dear Diane,

A great Tuesday letter. It has outraged some of my friends—whose strategic approach now is to applaud anything moving in our direction and speak quietly about anything we profoundly disagree with.

Yes, alas: Duncan’s office is not yet offering a change either of us can believe in.

To stimulate the economy, Obama’s education plan includes more focus on charters, and for teachers, schools, and districts that implement so-called “merit pay” based on student test scores. Aside from misdirecting the goals of education, it misdirects the path to a good education. By confusing test scores with “being well-educated,” and the motivation to do a good job as synonymous with financial reward, we undermine values essential to democracy.

The “rulers” of our economy had plenty of incentives to build a healthy economy. Trillions. They spent their smarts on making sure they increased their share of the pie. And if their pay had depended on decreasing the earning gap, I suspect they’d have hired statisticians to play with that data, just as they did with the economy—and just as their educational counterparts have done with education data. Diane, you have been a steady voice in alerting us to misinformation, especially on the NYC front. Who will be doing this for the new Duncan DOE? Hopefully it won’t be people who perhaps need to speak softly to avoid placing themselves outside the circle of power. Critics are needed as much when we win an election as when we lose one. One advantage of being older and retired is that we have less to lose.

The more high stakes the data, the more corrupt become the data—which I’m told is called Campbell’s Law. We poison the well once we promise folks more money for “better data.” When “data” (e.g. test scores) are in the driver’s seat, beware. We also need more independent “juries” to analyze and make recommendations based on independent information. The phrase itself “data-driven,” rather than “data-informed,” gives me the chills.

We also need sensible longitudinal research, to explore the connection between test scores, school models, etc., and “doing better” 10 years out. This is uncharted territory. We might explore, in short, what “doing better” could or should mean in real life.

I recently read about a high and mighty American who was lauded for “freezing” his pay—at $11 million a year. It’s not merely that such wages are a waste, maybe even bad for his particular business operation, but that they corrupt the concept of democratic society (one vote per citizen, et al). It reminds me of the story about Marie Antoinette offering to give the poor cake, or the joke about how even a poor man was guaranteed a place to sleep—if only under the bridges of Paris. (I have no doubt messed up both stories, but I suspect, Diane, you know their mythical sources.)

Whether we’re talking about schools that teach the academic disciplines or the interdisciplinary “habits of mind” and “heart” that underlie a complex democratic society, or even “2lst Century skills,” we should be alarmed at the direction the newly staffed Department of Education seems headed. The most heralded change is in finding a new title for NCLB, rather than tackling its basic hypocrisy. Despite (or because) those closest to our schools—school boards, parents, teachers—oppose the testing mania, those in D.C. seem as disposed as ever to ignore such “self-interested” opinions.

Tests, as we know them today, are not even good sources for knowing if Johnny can read. Does becoming “skilled” at the components of reading tests translate to becoming “whole” readers”? And, if it does, can we assume this translates into reading more and more wisely? There are ways to make for technically better readers that do not make for a better-educated citizen or employee, much less a creative and inventive one.

Being taught early, over and over, that making a predetermined “wrong answer” (out of a predetermined four or five) has serious intellectual and social consequences is dangerous. It leads to bad pedagogy. It’s precisely in school that it’s important to value the exercise of judgment based on evidence rather than being taught how to slyly “guess” at the one “right” answer.

Children, starting from birth, as well as at ages 3, 4, and 5, are still highly motivated to make sense of the world without any prodding. Regardless of their backgrounds. In fact, you have to prod children to stop doing so. Which is what we do at the average school—by state design. I can attest to this based on evidence from almost any source. So I am alarmed at hearing that we plan to stimulate the economy by doing this with kids younger and younger. Such schooling will, over time, undermine both our economy and democracy. We need funds for our youngest—including publicly supported child care of high quality and an end to conditions highlighted in The New York Times, Page One, "In Turnabout, Children Take Caregiver Role". It's referring to preteen caretakers!

My visits to Chicago and DeKalb kindergartens (with exceptions) scared me—the absence of playfulness has become so normal! I’d love to know where you stand on this, Diane. We could even use a little disagreement!

Deb

P.S. One and all, read Mike Rose’s blog—or did I already suggest that? Also Mike Klonsky’s, who disagrees with me for our sharp critique of Duncan. Finally, I'd also point our readers to your recent piece for Politico.com, Diane.

31 Comments

Dear Deb,

Here's more 'change':

"The omnibus spending bill now moving through the House includes language designed to kill the Opportunity Scholarship Program offering vouchers for poor students to opt out of rotten public schools. The legislation says no federal funds can be used on the program beyond 2010 unless Congress and the D.C. City Council reauthorize it. Given that Democrats control both bodies -- and that their union backers hate school choice -- this amounts to a death sentence.
Under the program,
Families receive up to $7,500 a year to attend the school of their choice. That's a real bargain, given that D.C. public schools spend $14,400 per pupil on average, among the most in the country. -WSJ
Meanwhile, education association opposition has ensured that only seven states have some form of vouchers.

Functioning Catholic schools around the country are closing from lack of funds; moral education has fallen to a point one can only truly appreciate by viewing CNBC's House of Cards; two parent families are old-skool, and we're adding another $140 Billion to the education budget. So the 1700 kids in DC vouchers need to be put in their place?

Ed,
There is no connection between the maximum amount of a voucher and the annual cost per student of an education in DC. It would be a bargain if there were schools in DC that cost $7500/yr.

I've never seen a voucher system that actually provides enough money for parents to have a choice. If every parent had $7500, they still couldn't save Catholic schools. The church is suffering from poor attendance and the decline of the working class, who can no longer afford to keep the church going. Small business owners have less time for church attendance since they don't have the 9-5 5 days a week.

Loren,
$$$: the point was that it's costing the taxpayers something less than the $24 million it would cost for these kids to attend failing public schools. And the kids and their parents obviously think they're getting a better education.

Church: Ahh, here is a topic for a complete other blog! If the middle class isn't attending, it may be because they have more money for pricier diversions. The restaurants seem to have no problem getting more Sunday attendees year after year.

Catholics attend more often than protestants; Americans more than our European friends.

Where the church does lose regular members, though, I think has much to do with the age of its priests; a diocese like this, where maybe 6 of 60 priests are under 55... makes it hard to generate some enthusiasm sometimes. Younger priests sometimes have more energy; or at least an ability to connect with certain younger people. Of course, the church is just plain short of priests.

And then there's the growing tolerance for the self-proclaimed geniuses who ridicule people of faith.

All that aside, there's certain evidence that Catholic schooling has been a cut above since before public schooling began. Having attended a public school myself, I'm not going to give that too much rope,..yet there are certain values that go along way toward academic achievement.

I always remember the stat that when I was a teen, Catholics comprised about 19% of the population, yet any elite group (Boys State was the venue I heard this) turned up about 30-33% Catholic. This statistic held up in the military, in higher education, in advanced professions, and low and behold when they asked for hands, at Boys State , a third of us were Catholic.

All of this is hardly an attempt to draw converts --or even enough to inspire me to get there more often ;-) . Each to his own.

What it is to say is that here is a good educational resource that government should consider using as it looks to school young, at risk, urban Americans.

Deborah,

I believe if you check back through the records you'll see Obama did, in fact, advocate for charter schools and merit pay for teachers.

I'm very much in favor of merit pay for teachers. My only reservation echoes one of yours - "The more high stakes the data, the more corrupt become the data." I'm probably naive on this but I still believe in today's world of advanced technology states and districts should be able to develop a protocol that should be able to prevent/minimize this.

My advocacy for merit pay; I saw too many senior teachers simply go through the motions for most of their careers and collect similar or larger salaries than their peers. Some of them were so cavalier they often derided people who put in all the extra time and effort. I also have a problem with years of service combined with advanced degrees or PD points as the sole determinants for how much a teacher gets paid. While seniority should have some standing in the educational marketplace, I do not believe it should be the primary variable in determining a teacher's ($) worth.

Yes, I know this is parallel to the business model if you award better performance with better pay. So! Is that a crime or a worthy paradigm buster?

Paul,

Some of the concerns about "tenure" et al--which you raise under the label of seniority-- have to do with the degree to which teachers are subject to local pressure of a broadly political sort--including just who they don't get along with. Thus, long before the power of unions, there were laws about how teachers could be hired and dismissed. We need to rethink, but not abandon these legitimate concerns.

On the question of pay for results. I'm surely not an expert on how this works for salesmen, but it was bad practice on assembly-line style work (piece-work) and it's bad in teacher-like work.

The slogan "data-driven" suggests that humans are capable of developing instruments that we can all agree are "neutral"--don't reflect my educational priorities vs yours, for example. Or that can't be "taught to" in the bd sense. Or manipulated in some way - even outright cheated on.

I'm influenced Paul by the fact that I grew up during great enthusiasm for the industrial developments in the USSR with their "five-year plans". It fooled a lot of people on the right and left. But what it meant, as does half the so-called reforms of current days, exercising our ingenuity over trying to influence the data, not the kids. Russia's stunning unpreparedness to fight the Germans in WWII (which was only finally overcome by the tenaciousness of its ordinary people) was due to the fact that Stalin believed the data.

Our reckoning--after 50 years of data-driven schooling (we test more than any nation on earth, and have done so since the 1950s)--should be obvious to one and all. But it isn't. Because it's judgment that has been shoved into the closet while we embraced data data data. As though the data spoke for itself. And more is not always better. But note that in my five "habits of mind" I consider evidence as the first and most critical issue in any field.

Keep me thinking. Thanks. Deb

>>confusing test scores with “being >>well-educated,”

How are we to show that students are well educated?

Is that even neccessary?

Should teachers be held accountable if they produce students who are not "well-educated"?

Deb,

The Soviets' stunning unpreparedness had more to do with the German - Soviet Non-Aggression pact of 1939 than Stalin's belief in the data. Even though Hitler invaded Poland one month later, many historians believe this pact lulled the Soviets into a false sense of security while Hitler was working furiously at developing his war machine to invade the Soviets two years later in June of 1941.

I also believe today's data is more immediate, more reliable, and less prone to compromise than the data of the past fifty years because today's technology has improved immeasurably.

Tests given to inner city kids or their suburban counterparts can all be used to study the growth of a student from one year to the next. It doesn't matter whether the learner is fast, average, or slow, it's the percentage of growth over the year that is calculated. It demonstrates each learner's progress (as opposed to a cohort of students as is done today) as well as the effectiveness of the teacher with that particular class of students.

Over time, a teacher's effectiveness can be examined to see whether there is a pattern or not. If a teacher appears to be producing weaker results year after year they should then receive guidance via professional development programs that could possibly get them headed in the right direction. Hopefully this would encourage and/or motivate some of these folks to want to alter their practice.

Deborah (& Diane) -

That you two are conducting this ongoing discussion, and involving interested outside parties as well, is so thoroughly encouraging that I can't help but think the overall effect will be very productive.

My concern is that the "grassroots" is not sufficiently engaged in the education issues and therefore the political debate is left to those who already have a hefty slice of the policy pie.

How do we fix that problem?

Deborah,

There is much to think about in your column. I found myself pondering something that you brought up at the very beginning and then a few paragraphs down. First, you mentioned your friends "whose strategic approach now is to applaud anything moving in our direction and speak quietly about anything we profoundly disagree with." Later you brought up "people who perhaps need to speak softly to avoid placing themselves outside the circle of power."

Why are people choosing to speak softly about their profound disagreements? It seems that a profound disagreement calls for speaking up. How can we do otherwise? Is it worth it to stay inside "the circle of power" if it means lowering one's voice to a hush?

Of course we can't and don't speak our minds in all places and times. If I disagree with a friend about something, I am likely to approach the friend directly instead of telling others about it. At work, I may guard some of my disagreements and exercise tact(though I have been speaking up more and more). We have to use our best judgment and consider the effects of our words on ourselves and others. But we do not need to confine ourselves beyond that.

Part of the reason I oppose merit pay is that no amount of money can compensate for the lack of my own time. In my own time I can read and write what I like. At work, even if I am relatively open, I must perform the tasks given to me. I may not take out a book and read. I may not work on my own projects.

You could offer me a doubled salary for, say, ten additional hours of official work, and I would say no. I work on my teaching from home anyway, but at least at home I am on my own time. I turn down "per session" jobs because I want the evenings and weekends to myself. If I lost that, I would lose the time when the mind can roam; when I can read and write about anything; when I am not beholden to an immediate "outcome."

If we give up our own time, we forget what it means to have it. What good is the extra money, if you have to give up your soul and mind?

Diana Senechal

P.S. Some would argue that merit pay has nothing to do with extra hours on the job. But it does. Proponents of merit pay typically also call for a longer school day and year. Also, let us not be fooled. Any merit pay based on test scores would result in additional test prep. Much of that would take place after school and on Saturdays.

Young people aren't stupid. Eventually, the country will get a chance to hear what young people think about NCLB. In 2012, the first generation of NCLB victims will get the chance to vote for President. If you realize that most states had NCLB testing in place by 2003, that means there was a group of students in the 3rd or 4th grade who, in 2012, will be 18 years old and able to vote. They will have a real insider's view of the law, having personally witnesses all of its effects.

They will have seen every item on every test so they will know exactly what the tests really measure; they will have gone to schools everyone knew were good, but were labeled "failing" year after year; they will have learned that the tests they took when they were very young, the tests that supposedly didn't count, were actually why they were not assigned to Ms. Smith's Advanced Algebra class with the rest of their friends; they will be able to search online databases to find out how much money their best teachers and worst teachers got in merit pay and wonder how the system could make so many mistakes.

They will have sat next to a friend from another country while they both took the reading assessment, knowing their friend was really smart but not very good with English yet. They will have had really smart friends who failed and not so smart friends who passed and wonder why. They won't need to see anyone's test scores, they will know because data-driven interventions will make clear to even a third grader who passed and who failed.

Some first-time voters will have been students who could have passed the NCLB tests on the first day of school, but were subjected to endless, mindless direct instruction drill and kill prep sessions. Some will have been students who struggled academically and were shuffled into even more idiotic interventions, enduring the intense, frantic, neurotic assistance of well-meaning teachers desperate to get them over the bubble.

They will have heard about charter schools, but not know anyone who went to one. Some of them will remember when the special education students got those checks to go to another school. Whatever happened to that kid? He was in our third grade and fourth grade class and then he just disappeared. They will remember the year that all of those students suddenly came to school out of nowhere when their school was closed. They will remember that everyone knew that every one of those kids had come from a school so bad they shut down the school. They will remember that a lot of older people were angry those kids were sent to their school and even some of the teachers were, too.

They will have sat through stupid pep rallies celebrating success on tests they knew were too easy... or Saturday tutoring sessions of endless A,B,C,D on tests with questions they were smart enough to know were deliberately confusing. Why don't they just ask one math question for each question? Why do they have to cram so many questions into one question just to get one answer?

Many students will have spent year after year being told they were proficient only to perform miserably on college entrance exams. Many will have spent year after year swinging back and forth between passing and failing, never knowing why. Many will have failed year after year, learning only to apologize to teachers who seemed to try to teach them everything in the whole curriculum while the teacher down the hall knew which 50 things were likely to be on the test.

Some students will be one of a group keeping a secret. How strange it was that Mrs. Jones asked us all of those questions the day before the test and then the test was just like the questions she asked us. How strange.

Apparently, if Arnie Duncan has his way, all of these new 2012 voters will remember how Obama changed the name of NCLB and then there was even more testing which was even worse. It will be Obama's law by then.

In 2008, Obama figured out how to use modern technology to reach young voters and get them to vote. If Obama succeeds at fixing the economy and keeping us safe, Obama's NCLB may be the most important issue for these new voters. If he doesn't want their votes, someone else will.


Ed,

Voucher programs haven't saved taxpayers a penny. They merely allow for some creative accounting making it look as though fewer education dollars have been spent. $7500 goes to the parents and the remaining $6500 is spent on the other services the student would have received in the public school. Someone will be spending $14,000 of taxpayer money either way.

Private schools will always look better because they have none of the bureacratic mandates that public schools do, and they can expel anyone who makes their stats look bad.

Catholic schools have historically served 2 major purposes: generate income for the church and allow children of working class white families an opportunity to attend whites only schools. Personally I don't think the government should be financially supporting any religion or school segregation scheme.

Loren,
Your faith in the efficiency of the DC public schools is certainly inspiring! One might ask why it costs so much more to educate a kid in DC than in any other large city? Do you have some data to justify both the inter-city discrepancy and t claim that voucher schools get $6500 in additional services?

--
The "Private schools will always look better" shibboleth has some small truth to it. If what we were talking small differences in performance, one could certainly make that claim.

Yet often we are talking private schools which may take only the rejects of public schools, or private schools with lottery drawings for admissions.

More to the point, there are often very large differences in performance. For example, moving from 20% literacy to 90+% literacy.

---
Still, Loren, quibbling over dollars and test percentages belies the more fundamental differences in approach. We two could never come to any honest discussion here. The reality is, I am a builder who looks for solutions. I am willing to tolerate a variety of solutions, none of which work perfectly. You however seem to tend more toward reactionary. You'll likely defend the worst lapses of your side, and quibble over the smallest shortcomings of anything new. You'll never go looking for innovative solutions; you'll always go looking to preserve and expand the present system you believe in--Government and union bureaucracies.

That's the fundamental problem with the system. Too many people in power with just that attitude.

Some people, Laura, can walk through America and never see all that we have; all that the private enterprise system has brought us. The compelling environment here at Panera where I am now working...the lady behind me who just made a new friend by offering a mom some assistance, the young man talking with Dad or a mentor about applying to Stanford and acting and PS2, the cool guy I just met who is advising the Senegal gov't on Education, the increased productivity a number of us get from being able to work and eat here...none of these things are, to reactionaries, a blessing; they are at best a tolerable side effect of the nasty capital markets which funded this building and the decor and the ovens and the oven manufacturer, and the grain mills, and the cargo planes which brought the fruit, the trains and trucks which brought the flour and napkins and drinks and soup ingredients.

People like me look at the public school we came from, say we love it, and we'd like to see it improve. We say, "who can we look to for better ideas?" We see some of the remarkable things like the incredible bookstore at one end of this complex, or the home store at the other. We ask, 'how could they bring all these things to so many people in so many places in so few years? What can we learn from their principles and processes?'

Your third paragraph I shouldn't dignify with a reply...but then, you might have learned your history in a public school, so I forgive you! ;-)

To tie this up, though, let me say that I believe that the troubles we have seen in the financial markets come from one source: two much dishonesty. Home buyers were lying about their income on applications. Loan originators were signing off, or in some cases, filling in the false data themselves. Politicians were pushing loans on people they knew could never sustain the payments. Banks were taking the loans knowing they had no verification of the buyers ability to repay. Credit ratings agencies were pushing through mathematical sleight of hand (via CDO's) knowing they were branding risky investments as AAA grade. Brokers were reselling these investments knowing they couldn't understand them at all. Leaders from university endowment managers to village mayors in Finland to private foundation managers were all investing in these things, knowing they had no understanding.

So, dishonesty came in many forms, but it finally did in the system of trust we all relied on. People were dishonest from top to bottom and back to the top again.

So, Loren, were I a Congressman or Ed Association leader, as I contemplated how to spend my hours each day and year,...leaping to oppose the assignment of vouchers to a few kids who weren't learning but are now, vouchers which might just go to a school where they post the Ten Commandments on the wall and focus on the basics like math, geography, history, and literature,...that's not a use of time I can say I'd adopt.

Some things to consider......

Eva Moskowitz Succeeds at "Harlem Success"

...In Raking in the Big Bucks.

Dangers in the market-based ed reform movement.

Juan Gonzalez in the Daily News reports on one of the more disgusting people in NYC education/politics. Charter schools have license to steal from the public coffers while private interests fuel a gravy train. What next, a Harlem Success corporate jet?

http://www.nydailynews.com/ny_local/education/2009/02/26/2009-02-26_former_city_council_member_eva_moskowitz.html

"Moskowitz, who makes no secret of her desire to create 40 charter schools across the city and run for mayor some day, raked in $371,000 in salaries in the 2006-2007 school year from organizations connected to her four schools," Gonzalez writes.

"Charter schools are free to use the money they raise from outside sources any way they see fit - even if that means huge salaries for the chief executive.

Given that Moskowitz routinely complains that the Department of Education has failed to provide a fair share of funding for her students, it's fair to ask why she's paying herself so much for educating so few.

Charters get about 90% of what it costs to teach each child and raise funds for additional money."

That's all for about 1000 kids from grades K-3 who attend Harlem Success.

Let's see now. At this rate, if Moskowitz was the chancellor, this would come to around $1,200,000,000 based on the per child rate. Hey Joel, you're underpaid.

see link...http://ednotesonline.blogspot.com/

be well... mike

Ed,

I would love to see that private school that takes only rejects from public schools and only costs $7500. BTW, what is a voucher school? I thought there were public, private and charter schools. I don't know about Panera Land, but here in NYC the schools with lotteries have strict entrance requirements to even enter the lottery. To suggest somehow that enrollment is random is disengenuous at best.

I'm not against vouchers. IMO no one has ever attempted to implement them fairly or adequately. Do them right or don't do them at all. I might add that none of the current crop of celebrity reformers has paid more than lip service to vouchers, preferring to fund private and charter schools with private money.


Where are the discussions of what we know we can test well, what we know we can only test poorly and what we know what cannot test at all? Shouldn't this be part of the debate?

Put this lists alongside the thing we want schools to teach or instill in children. Yes, reading and writing and 'rithmatic. But also Debbie's Habits of Mind (e.g. peristance, imagination, analysis, empathy, communication, committment, humility, joy).

If we know that we cannot -- at least not now -- test things that we think are really important, what should we do? Should we give up on the important things? Or should we give up on the idea of testing?

If we cannot measure it, does that mean that it is not worth teaching?

Tests, as we know them today, are not even good sources for knowing if Johnny can read.

More reflexive nonsense (I still can't believe that Diane Ravitch actually said that it's impossible to study the effect of uncertified teachers . . . tell that to Goldhaber, Brewer, Rockoff, Kane, Staiger, etc.).

A typical reading test for 4th graders (just for example) will provide a reading passage, and then ask several questions along the lines of: what is the theme of this passage, what did such and such person do, what did this sentence mean, etc.

No test is perfect, of course. But they're a lot better than nothing (and also a lot better than subjective "portfolios"). Do you want to know if Johnny can read? Then give him several passages to read and ask a few simple questions about the passages. If he can't answer them, then it's a pretty good bet that he's not an accomplished reader. And people need to know that, rather than continuing the old status quo in which teachers and schools were basically able to hide the fact that some number of kids weren't able to read.

The problem with using individual teacher judgments as the only indicator of student performance lies in their subjective nature. When I first began teaching in a high poverty mostly Latino urban school, I was convinced that almost all of my students were reading at or above grade level. I was shocked when the (dismal) results came back from the state assessments. While I realize that state assessments are not perfect, I didn't blow them off either. I rolled up my sleeves, went to work, and searched for every scrap of information I could get to improve my teaching effectiveness. My student's results have improved dramatically. I have been able to achieve up to 2 years growth with some of my lowest achieving students.

Consider Nebraska's experience with non-objective teacher assessments. Their STARS tests purported to show that Nebraska was one of the highest achieving states in the nation with 80% of their Latino population and 75% of their African American student population reading at or above grade level. But the NAEP tests showed something far more disturbing, only 12% of their Latino students and 9% of their African American students were actually reading at or above grade level. I'm not trying to criticize Nebraska teachers. I've been there. I made the same mistake they did. It's natural for teachers to assume that what is typical of students in their school is in fact what should be expected of that grade level.

Asking for objective data is not "teacher bashing". What is teacher bashing is unfairly comparing upper middle class suburban schools with schools that serve high needs populations. Sometimes even 2 years growth (and I don't claim to achieve this with all of my students) is not enough to lift my students to grade level in reading. Does this make me a bad teacher? According to NCLB it does. A growth model is sorely needed.

John Doe,

Under NCLB it is difficult for schools to keep uncertified teachers without facing consequences. Even those teachers in alternative certification programs have emergency certification; thus they are certified on paper. Many of them earn a masters degree in education while teaching. Some teachers teach out of license, but few lack a license altogether.

You mention Goldhaber. In 2007 Boyd, Goldhaber, Lankford and Wyckoff published a study titled "The Effect of Certification and Preparation on Teacher Quality" (Future of Children 17:1, Spring 2007, pp. 45-68). The authors observe on p. 53:

"In 2001 about half of all new teachers hired in New York City were uncertified; by 2004 that share had fallen below 10 percent, and it continues to fall. In short, New York City’s uncertified teachers have been largely replaced by the teaching fellows."

Thus it appears, at least in NYC, that the percentage of uncertified teachers is negligible. Of those who were hired before, I imagine many quit (as do many teaching fellows, even after they complete their masters). The revolving door is a serious problem.

Now, what are the effects of certification on student performance? The authors found some correspondence between teacher preparation in the subject area--in this case math-- at the graduate level and student achievement (p. 57), but little information regarding the effects of pedagogy coursework on student achievement (p. 58).

Moreover, certification requirements vary widely from state to state (p. 48): "As observers are increasingly aware, there is more variation within certification programs than across them. Thus traditional preparation in some states may look very similar to alternative preparation in others."

In sum, the authors found a disturbing "lack of an evidentiary base" for assessing the effects of certification (p. 63). There are too many complicating factors. But does this mean we should scrap certification? I say we should not; we should make it more meaningful instead. That is no simple matter.

It is true (at least in NYC) that we have very few uncertified teachers; thus it is true that we have no control group. However, we would do well to study the effects of teachers' subject matter knowledge on student achievement and overall school quality (over the long term).

But then we'd run into another complication. Do schools themselves offer the subjects that teachers come prepared to teach? We see the dilution of ELA and social studies; a teacher with strong preparation in literature or history may find this confounding at best. Many NYC Teaching Fellows come in to teach ESL, only to find that it is not a subject at all. Schools change their mind every year about how to handle it.

Now, I believe a teacher could teach well out of license if he or she has strong interest in the subject and a strong liberal arts education overall. Someone who majored in history but has a great love of math could conceivably be a superb math teacher. But such a teacher should still demonstrate competence in the subject.

We should examine not only teacher preparation in the subject they teach, but the schools' conception of the subject itself. That would do a lot to illuminate the situation.

Diana Senechal

“Wisdom hath built herself a house, she hath hewn her out seven pillars" (Prov. 9:1).

It was Rome which formulated and universalized the seven liberal arts -artes liberales- they adapted from Greek educational models. Liberal arts were considered liberal because they were for homo liber, the free man as opposed to the slave. This is idea of a "well-rounded education" that prevailed in classical and medieval times. Behind the Greek “enkyclios paideia” (εγκυκλιος παιδεια) or the Latin encyclios disciplina lies the image of the circle –the infinite cosmos itself- and the sense that the pursuit of knowledge is one of ever expanding limitless horizons . Cicero and many of the early Roman theorists tried to reconcile and synthesize the different Greek philosophies of education and adapt them to Rome. To study the Greeks, as Cicero advised his son to do, ite ad fontes – is to go to the source, the ultimate source, Homer, the philosophers Plato and Aristotle as well as the common sense soldier Xenophon.

The origins of the debate between utilitarian and those who favor a liberal education goes back at least to Plato’s Republic in the 4th century B.C. for it was at this time that Plato formed a new educational theory not based on Homer, Hesiod and Greek traditions but upon the theory that a strong ruler could use the state to teach virtues and so shape human nature so as to produce a more efficient and harmonious state.


Plato clearly saw that an advanced commercial and military state cannot leave education to chance and to private demand for the simple reason that the head of the household was often engaged in business of the state, at war, a prisoner in the mines of Sicily or dead. The Republic was written after the disastrous Syracusan Expedition and Athens’s defeat in the Peloponnesian War.

Who was going to educate the fatherless orphans of Athens? The demands of a more advanced commercial society required a new state supported compulsory educational system. For Plato this was the only way citizens could get the training and education they required for themselves so they could serve the state. There is no question that Plato’s insistence on a compulsory, state-run educational system was a direct attack on the old Athenian custom of home-schooling which left each citizen the freedom to purchase the slaves or teachers to provide an education for the household children and provide whatever education the market could provide. There is no question this led to an uneven, haphazard education for Athenians, particularly in times of hardship or war. Plato recognized this in his Protagoras when he satirized the Athenians for giving less thought to the education of their children than to the breaking in of a horse. Americans give less thought to the education of their own children, by and large, than they do to following the latest technical gadget, TV romance, spectator sports or video thriller.

Plato correctly came to understand that the whole question was not merely the training of orators or soldiers or even training itself.

Behind training lies the need for wisdom: knowing WHAT to teach and the AIM of education, which is WHAT to train students TO DO and how they SHOULD BE.

We can never assume that every one has the knowledge or cultural literacy which shall be taught; in fact what is critically needed is more knowledge (cultural literacy or numeracy).

Real education is not only for the classroom but should carry on beyond mere schooling and into later life. The greatest failing of our schools and colleges, in my humble opinion, is that so much time is wasted on testing and trivialities AND that so much of what is learned in school does not carry on into later life. A people without understand will come to ruin as we may be experiencing right now.

Its sad to see Mike join in on the charter bashing by focusing on something as trivial as a CEO salary.

Two related articles tell more of the story: Now Harlem parents can get an education and Parents pack education fair in Harlem.

"Rodney Jacobs, 49, from Harlem: "I'm looking for less pupils in each class and teachers who pay attention to kids' needs." Jacobs, who said he was unhappy with the level of discipline at the public school that his twin sixth-grade girls attend, was among thousands who showed up to check out some of the school choices in the Harlem area. "

"Samantha Parker, 27, said her daughter Tabitha, 8, wasn't challenged at her Harlem public school. Classes at her previous school at an Army base were much harder. "

"Education Department stats show that more than 70% of eighth-graders in District 5 aren't reading at grade level, and fewer than half passed the state math exam.

"At the elementary school level, more than half the third-graders in the district aren't reading at grade level."

Dear John (Doe),

You mention a "typical reading test for 4th graders." But few fourth grade NCLB reading tests are a measure of "reading." A proficient score can almost always be earned without reading any of the passages. These tests are riddled with problems.

- Questions about theme that can be answered by reading the title.

- Questions about the meaning of a word in context that could be answered if you knew the meaning before you read the passage.

- Questions about plot or character that appear to measure careful reading skills or critical thinking, but which point you to the paragraph you need to find the answer that is right there (marked in the text so you don't have to count down to paragraph number 6).

- Questions about the author's purpose which can always be answered by "to inform" (for nonfiction) or "to entertain" (for fiction).

Even if you had to read the passages, the passages are probably far too easy for most readers at that grade level. Can you really say a student is reading "on grade level" if most or all of the passages on the test are below any reasonable measure of grade level?

Elementary reading tests consist mostly of very easy questions, enough to get most students to proficient. No one complains because most students pass. Add in a group of test-savvy students able to decode the questions. They know to read all four answers before choosing one because sometimes A) is a trick and D) is better. Those students round out the upper part of the distribution between proficient and advanced. Finally, there are some questions that are usually very difficult, like "What is the author's tone?" Those items seem to challenge many students because not many answer correctly. But just because a question seems challenging doesn't mean that students with a better school/teacher/curriculum would do any better. Some items are "difficult" because they are highly subjective. Knowing that the answer is probably "satirical" or "ironic" and a lucky 50-50 guess will put you in the top category, "advanced readers."

The only real reading is when the students read the questions. The tests seldom if ever attempt to measure the most important reading skill in reading nonfiction -- integration of new facts with what you already know. If they did, the tests would rely too heavily on what students had already read before the test, maybe even what a student had read outside of school. The tests seldom if ever ask if a student liked reading a work of fiction or what he or she imagined while reading. More uncontrolled variance.

If you doubt any of this, you should go to your state's department of education website and download a sample copy (or request a sample) so you will have a better sense of the shortcomings of these tests. Get two or three old released editions, if you can. In some states, one year's test is so much like the next that a week's rehearsal on the sample would be enough to lift students up over the bubble.

So, why isn't every student proficient? Some students start so far behind that they will never be able to catch up, especially if they focus on reading instead of test taking strategies. Yes, we have all heard the hype about what might happen if students assigned to the very "worst" teachers three years in a row had instead been assigned to the "best" teachers. But since "best" and "worst" are defined based on the students' own test scores, it is entirely possible that if the "worst" students had been assigned to the "best" teachers for three years, those would no longer be the "best" teachers. They would be the "worst" teachers. There is no empirical evidence demonstrating that the "best" teachers are consistently "best" from year to year when assigned the same level students from the same school, so the idea of their remaining "best" with the "worst" students is, at best, naive.

Should we offer financial rewards to teachers smart enough to teach students what they need to do to pass a fourth grade reading test? Unless you have read the reading test, you aren't ready to answer that question.

JDET, I don't understand your point.

Plenty of parents know their kids aren't learning, regardless of test scores. That's why thousands lined up in Harlem Saturday for a shot at getting out of the public schools.

You can go to the Newshour archives to see videos of Johm Merrow's visit to classes where the students can read, vs. classes where they can't.

JDET:

I would suggest that while perusing your state's wesite for released versions of the test, one might also poke around for the technical report and any accounts of the test-building process. Most states (or at least my own) go through an extensive process of item review that includes quality control for the kinds of things that you are describing, generally carried out by committees of teachers and others. My own state also includes open-ended items, but they turn out to be among the least likely to be answered.

But, I also would have to underline Ed's point that despite what seems to be a simple solution to passing the tests (just review the released tests and teach test taking techniques), and the fact that most low-performing districts have build currciulum around doing just that, we still have too many schools where not enough students score proficient to meet minimum expectations (which are also frequently set quite low). Students who start out "behind," rather than benefitting from any planned intensification in order to "catch up," systematically receive less and fall further and further behind. Over and over we find that they have less access to experienced and highly educated teachers, to rich curriculum, to outside enrichment.

Can you imagine a health care system, upon finding a malnourished five year old, that would place that child on a restricted calorie diet because there are other children who are overweight, assuming that the child is just too far behind to catch up? These are the kinds of choices that we let slide in education all the time. And the answer is to throw out the scales.

JDET --

I'm looking right now at last year's Arkansas Benchmark test for 4th grade reading (located here: http://arkansased.org/testing/pdf/rib_gr4_spr07.pdf ). I don't see any reading questions that fall into any of your categories.

Maybe Arkansas is just that far ahead of the rest of the country.

Am I right in thinking that you're criticizing the fact that "The tests seldom if ever ask if a student liked reading a work of fiction or what he or she imagined while reading"? Surely you're not seriously suggesting that a test should ask such fluff questions?

A few thoughts in response to the above comments, by Deborah!.

Ed: Why do you assume that teachers are not in accord with ed association leaders? Of course politicians concern themselves with any voting group that has strong views. Farmers, small businessmen, etc--that's how the system works. Isn't that built-in to the politics of democracy?

I've been arguing with my private school friends (I went to one) about why they don't accept studebts via lottery--regardless of test cores, etc--even amongst the wealthy. And why do they "counsel out" kids at various stages along the way--thir failures--that than see this as their problem? , In NYC these selective schools charge $35,000 per child AND fund raise for extras. There's a ranking system among private schools--the most seletive down to the least.

I agree, Paul that schools with a coherent sense of their mission communicate such values to the young. And it's part of why they, to my mind, do a better job. All schools need to be able to do so. They needn't be religious value--although they may be influenced by such--of course. For example: the idea that democracy is a value we share should inform school missions, the way they operate and what and how they learn--and what and how they assess their work.

It's difficult because every choice we make has both positive and risky consequences. But is there anyone out there, Flint, who thinks that scores on standardized bubble reading and math tests are a good measure of being well-educated?

Diane. Additional responsibilities beyond those one is paid for should usually be compensated for--but I wouldn't call that merit pay. Like you, it was precisely the flexibility that teaching offered me re vacations, summertime and getting home for dinner that influenced my choice of professions! A staff that's got leeway to play with its own budget plays with time too--and individual preferences, as we did (do) at Mission Hill School staff with that kind of decision-making power and sense of "ownership" are inventive in the interest of considering both fairness and the needs of kids. Having a public Board is useful oversight.

Ceolaf. The hardest thing i've ever tried to do is convince people that reading tests are measuring many things but not--literally--reading. I found it out the hard way--by sitting with kids whose test answers I had access to and seeing what lay behind right and wrong answers. It turned out that reading it aloud to them made very little difference for most of the poor scoring kids. And, in fact, some wrong answers showed better, not less, understanding!

There are many ways to "measure" reading--including taping kids reading at various stages, including a short interview after, and "objectively" making judgment about what it tells us.

When parents are asked to discuss--not bubble in--why and what they value about their child's school (and what they don't), the three Rs actually don't appear at the top of the list. Odd?

Deb


JD,

The hypothesis is that most 4th grade NCLB reading tests are mostly very easy items with a few very, very difficult items thrown in to allow some students to appear to score very well, although they may be doing so by luck. Use the 4th grade 2007 Arkansas reading test you posted to prove me wrong. Ask the state department of education for a table that includes the raw score totals (actual raw points, not scale) from 0 to the maximum possible and the percent of students at each score point. Someone might say that the scale scores are more appropriate, but if we aren't comparing one test to any other, raw scores will suffice. Ultimately, scale scores come from raw scores and raw scores sometimes present a clearer picture of the facts. Also ask for the cut scores (basic, proficient, advanced) in raw score points. This is not confidential information and should be readily available, especially for an older test.

Assume that each raw point in the table represents a real test item or point for an open-response question. After all, a total score comes from real test questions. Easier items should be at the bottom scores and harder items at the top scores of the test total table. If not, you have complete chaos with low-scoring students answering difficult questions and high-scoring students missing easy questions. Let's assume that the students with the lowest scores generally answered the "easiest" questions correctly and the students with the highest scores generally missed only the most "difficult" questions.

If my hypothesis is correct, there will be a bunch of low raw score totals (0 up to X) for which no student earned that score. Those are points (multiple-choice items or open-response points) that not one student at all missed. Gimmes, as they say. Up from there will be a bunch of low raw score totals where very few students scored. Those scores came from super easy multiple-choice items or generously scored open-response. A big bunch of very easy items. How many are there on your example?

At the top of the test, you will see a similar pattern with many possible scores not obtained by any student. Those are the impossibly difficult questions that make the test look tough, although they may simply be impossibly difficult for anyone in any school with any teacher. How many are there on your example? There will be a bunch of students below that, maybe up to 5 percent, spread over several raw scores. Those are test-savvy students who could have simply made some lucky guesses.

Are there any score points with only 1 percent of students? How confident would you be in that score? How many scores are there with 1, 2, or 3% of students at that score or above? Is that enough questions to distinguish one student from another?

That is my prediction. So, if you have the time and energy to get the raw score table, I would be interested in six pieces of data.

1. The theoretically possible lowest and highest score on the test (0, X). That is the range the test claims to measure. The next data will tell us if it really does.
2. The lowest raw score obtained by any real student on that test. Below that are the gimme questions. How many are there?
3. The lowest raw score representing the bottom 5 percent of students (see where the percents from the bottom up sum to 5 percent). Those are more super easy questions. How many?
4. The raw score at basic, proficient, and advanced and the percent of students in each category.
5. The highest raw score obtained by 5 percent or fewer students (see where the percents from the top down sum to 5 percent). What is above them is an exceptionally difficult set of items or open-response points. If you had a multiple-choice item where only 5 percent answered it correctly, you would assume you had scored it incorrectly. Are those brilliant minds answering insanely tough questions or students skillfully guessing?
6. The highest raw score obtained by any student. This is the real ceiling on the test. How many impossible points are above it?

Does your test look like what I would predict? If so, how do you feel about using that scale for making high-stakes judgments about students, teachers, and schools? Could it really do a good job of measuring reading or academic progress?

If the data don't support this pattern, I humbly apologize in advance to everyone in Arkansas. I didn't pick Arkansas, but why not? Folks here are always talking about New York City. It's high time Arkansas got a turn.

JDET:

The information that you are looking for is readily available--I just checked New York and the cut scores, raw scores, conversion tables, etc were just one easy click from the home page. While I didn't go looking for them today, I know that the technical reports are also available and they include things like rasch scores that indicate level of difficulty, and explain the methodology by which cut scores are set. If New York is the state you are interested in, they have a certain advantage in that they have been in the testing business for a very long time. I understand also that they release every test item. This is an incredible level of transparency.

I am not at all certain what it is that you are going after. The odds of getting even 25% correct through random guessing are incredibly small (not one in four as some might assume, but a factorial that would represent a one in four chance for each item). As most tests purport to provide an indication of "levels" of achievement beyond merely crossing a "proficiency" line, it is appropriate that the questions reflect a variety of levels of difficulty from easier to more difficult. In fact, the kind of ordering that you describe is typically a part of the process of setting "cut scores."

So . . . you're basically asking me to do a huge amount of research in order to prove your point for you? Not that I mind taking a few minutes out of a work day to chat online (obviously), but that's a bit absurd.

In any event, if you go here: http://arkansased.org/testing/benchmark_iowa.html , the first two links seem to have a large amount of data on test scores in Arkansas. Why not spend the afternoon digging through those, and see if you can come up with some evidence for yourself? Thanks.

Judy: Sometimes even 2 years growth (and I don't claim to achieve this with all of my students) is not enough to lift my students to grade level in reading. Does this make me a bad teacher? According to NCLB it does.

Firstly, I thought NCLB labelled schools as bad, not individual teachers.
Secondly, why are you getting students who are more than 2 years behind grade level in the first place? Presumably you don't work entirely with the severely mentally-disabled as you are capable of moving many of your students up 2 grade levels in one year. So why did all their previous teachers fail to move them up one year's grade level in one year?

I don't think you're a bad teacher. I think that you're an exceptional teacher. But I think your school system's administration likely has serious problems if you are getting lots of students that far behind, excluding recent immigrants and the severely mentally disabled. Effective teaching requires the whole school system to work well, and that's what the NCLB tries to make happen.

My Response
After spending the last two weeks completing our state standardized testing, I have to agree with you. Just because a student ‘exceeds’ on a multiple choice test with minimal written response does not make this student eligible for being a successful adult. I have yet to take a multiple choice test in my career and I am a teacher. One would think if anyone took tests it would be teachers. Our students have become ‘word callers’ instead of being able to analyze a text for meaning or synthesize the reading to their personal lives. There is little to no connection between the standardized tests and being a ‘better-educated citizen’ as you stated. My own thirteen year old daughter refused to take the 8th grade standardized tests because it has no relevance to her life. Her math teacher told her ‘it is one thing to be smart, but another to prove it to others’. What kind of message is this sending to my daughter, that the only way to prove to others that she is smart is by participating in a standardized test? I support my daughter both as a parent and a teacher. Testing is not always the solution!

Comments are now closed for this post.

Advertisement

Most Viewed on Education Week

Categories

Archives

Recent Comments