« Guest Blogger Mica Pollock on: Everyday Antiracism: Getting Real About Race in School | Main | New Blogs To Check Out »

The Rhetoric of Reform: Does Research Count?


“Better schools. Higher scores. And satisfied parents. That's the record of the D.C. Opportunity Scholarship Program.”

Thus begins Secretary of Education Margaret Spellings’ column in yesterday’s Washington Post. In this piece, she seeks to rally public support to renew the DC Opportunity Scholarship Program (OSP), which provides scholarships up to $7,500 to use towards the costs of a participating private school, including tuition, fees, and transportation. The authorizing legislation stipulated that priority for scholarships was to be given first to students attending schools that were judged in need of improvement (SINI) under NCLB standards.

Last month, the Institute of Education Sciences, the research arm of the U.S. Department of Education which Spellings heads, released the results of the Congressionally-mandated evaluation of the OSP, which reports impacts after two years. As the first federally-funded private school voucher program in the U.S., the OSP is a political football, and this evaluation report and its predecessors have been pored over by policy wonks across the land. The statute that authorized the OSP mandated that it be evaluated in terms of its impact on student test scores and school safety, as well as a more ambiguous criterion of “success,” which was operationalized in the study as parents’ and students’ satisfaction with their schools. The evaluation used a randomized controlled trial (RCT) to assess the impact of the OSP.

The executive summary of the report tells the tale, in unambiguous terms. (a) After two years, there was no effect of the OSP on reading or math test scores either for students who were offered a scholarship or those who actually used a scholarship. (b) If we look at 10 different subgroups of students—girls or boys, students attending SINI or non-SINI schools at the time of application, elementary or high school students, those from application cohort 1 or cohort 2, or students performing relatively higher or lower at the start of the study—there were no statistically significant effects of participating in the OSP on math for any subgroup, and for reading, three subgroups (students attending non-SINI schools at the time of application, relatively high-performing students, and students from cohort 1) might have done better than their nonparticipating peers. But even here, the evaluators caution that the statistical significance of these effects did not hold up when conventional adjustments for multiple comparisons were made. In other words, these subgroup effects might be due to chance, given how many comparisons were being made at the same time. Notably, the subgroup specifically identified in the legislation—students who had attended a SINI public school under NCLB—did not do better either in reading or math.

skoolboy isn’t crazy about using public funds to support private schools, but he’s a big supporter of using public funds to support the education of children in D.C., who historically have been among the lowest performers in the nation. Congress authorized this program, it’s survived legal scrutiny, and it’s deserving of a fair shake. But distorting the results of an evaluation doesn’t serve the public good. If Ms. Spellings wants to argue that the program should be renewed by Congress because parents are more satisfied with their child’s school, or because they are less likely to report serious concerns about school danger, she’s welcome to make that argument. Those are good outcomes, and some might argue that they’re ample justification for renewing the program. (Others might point out that students who received scholarships did not report higher levels of satisfaction with their school, or better school safety.) Or, alternatively, one could argue that the program needs more time to mature in order to be successful. But let’s not kid ourselves, Madame Secretary: the evidence on the academic success of the D.C. Opportunity Scholarship program—measured on your preferred metric, scores on standardized reading and math tests—is far too weak to make a persuasive case. Misrepresenting the evidence does honor neither to education research nor to education policy.


I still am unsure how I feel about using federal funds to send children out of public schools and into private ones. Isn't this just a band-aid?

Also, I find the use of educational research (in this scenario) to be dangerous. As part of my doctoral studies, we spent HOURS (and I mean HOURS) discussing the need to read research critically, considering researcher bias, the methods used, etc. Unfortunately, reading research in this way (educational or not) is not typical and the average reader may simply look at the numbers. And I can't fault the average reader. I think researchers need to be more responsible with the presentation of their findings and, especially in educational research, remember that the experiences of children are at the heart of the matter, not our own agendas or biases.

It is both amusing and telling that when "Spellings'own metric" was used, demonstrating the voucher program had no effect on reading and math scores, the public schools are (as always) referred to as "failing" and "under performing" while the private schools are referred to as "better schools" and the "first opportunity to receive a high quality education".

It takes no small amount of gall for Spellings to blather about putting "student welfare above personal ideology" when she and her corporate-serving ilk are the living embodiment of such behavior.

When I put this study together with the pile of other luke-warm or conflicting findings regarding public vs private, public vs charter, public vs voucher, I largely conclude that we are pretty much barking up the wrong tree. While we may want public funding of public schools to provide the best outcomes, it would appear that alternate funding methods neither produce, nor impede good outcomes.

As a practical matter, as a parent in an urban district with very mixed results, I can appreciate having a greater number of (similarly mixed) schools to choose from in trying to get my kids educated. My liberal roots certainly lean in the direction of public funds for public schools, fair pay for teachers and anything that has a hope of prividing equitable access to education. But I have kids of my own--and know plenty of others. They all need education--and they all need it NOW. They will be grown too soon for our wanderings in the question regarding funding mechanisms to be resolved. I think it has become an enormous red herring.

I wish we could put as much enthusiasm into examining how to teach every kid as much as possible.

As a life-long DC resident, the proposed public-financing plan has my full support. In an ideal world - and city - DC's schools would be just good enough to not need this kind of crisis remedy, but the situation is so poor and demands immediate attention that such a move is necessary. It is, of course, a plan that might fail, but I think that's a chance DC and the gov't should take...the students in DC are getting an unfair shake as it is, so why not try something that could greatly benefit the students that have been given so few chances at success?

Mimi - Amen. That is, in part, why I started writing this blog.

Tauna - The metric shifting makes me crazy as well. I agree with the Madame that parent/student satisfaction are important, but if they are, why don't all of our policies recognize that? And as I explain below, the bumps in satisfaction would have to be huge for me to be willing to support vouchers.

Margo (I never know whether I should call you Margo or Margo/Mom) - I'm also interested in your response to the skoolboy proposition below. How big/how lifechanging would these effects have to be? Of course, given the current research on vouchers, this is a non-issue since the effects on achievement, if any, are very small.

skoolboy - Here's a thought experiment as I'm interested to hear more on your feelings about public funds going to private schools. What if vouchers did improve achievement, but just a little bit? Let's say that over the course of elementary school, a kid gains .1 standard deviations in reading/math achievement from moving from a public to a private school. In other words, how large an advantage would private schools have to confer to flip your position? Because of the weight I put on some of the other goals of public education, these effects would have to be pretty huge for me to accept a voucher program.

Paul - Re "why not try something that could greatly benefit the students that have been given so few chances at success?": It has been tried in DC, and with meager results.

If voucher programs did show significant results, one of the most plausible reasons would be the ability of private schools to exclude difficult/disruptive students to a much greater degree that public schools can.

But that opens up a question: public schools aren't able to isolate problem students because policy makers decided that -- out of a concern for educating all students -- they shouldn't be able to. But do we now turn around and allow the same result in through the back door with with vouchers?

It seems to me that if entities (such as private schools) are going to receive public funds, they should have to adhere to the same basic policy mandates that public entities do.

So far, I haven't seen any mention of the types of private schools the opportunity scholarships are being used for. While $7,500 is a significant amount, it covers only a third of the cost to attend some of the most academically rigorous schools in the D.C. area. If these scholarships are usually going to students attending schools that are judged to be in need of improvement, and the students are often from low-income families, doesn't it follow that the scholarships are often used at the lower-tuition private schools, such as the parochial schools (which are sometimes struggling themselves)? Can this account for the lack of improvement in test scores?

I agree with Rachel. You cannot compare outcomes when populations are so different--self-selected versus public.

We need early childhood education, vocational training, and teachers making policy.

Caroline: 80% of the students in the OSP attended religious schools, and only 11% of the program participants attended schools whose tuition exceeded the $7,500 provided by the scholarship. The schools could provide scholarship aid themselves to participants; the scholarship foundation administering the program reports that in Year 2 of the program, 164 participants paid out-of-pocket tuition over and above the $7,500 scholarship. It's clear that most scholarship recipients are not attending highly selective, tony private schools. This, along with the disruption that comes from changing schools, might account for the lack of effects on reading and math achievement.

Re: Caroline's comment...

One to remember in the voucher discussion is that most of the private schools involved are very different from the selective, high-tuition schools that come to mind when many people hear "private school." I don't know of any voucher proposal that would make those schools accessible to inner city students.

Everyone who has pointed out that you can't just compare public and private schools is right - but in this case and in some of the other voucher literature, estimates of voucher effects come from studies where students are randomly assigned to treatment (voucher) and control (non-voucher) groups.

It's not like private school or charter school teachers get any different training that public school teachers.

To me this is a no duh study.

It's basically the same result when you randomly place low SES students in middle class schools.

Vouchers or school choice are only useful if parents choose a school that uses effective teaching practices. Then again, it would be so much simpler if public schools improved their teaching techniques.

If our government is going to spend money on studies, someone needs to do a long term study on which education schools produce teachers who have the greatest effects on student achievement.

Who wants to bet that the results show that education schools that concentrate on the mechanics of teaching do much better than the schools that concentrate of edujargon?

Actually, hold of on doing the study, I smell a great thesis paper for my graduate studies in a few years.

Please forgive grammatical errors above.

The results, regardless of how the Secretary attempts to spin them, speak for themselves. The ONLY benefit of the vouchers is they offer choice to poor and minority students previously available only to students of wealth.

Being new to this debate, I'm curious how these voucher programs are getting around the separation of church and state rule since 75% of private schools are religious-based. Could someone please clarify the picture for me?


Other more lawyerly types may weigh in here, but my understanding is that the U.S. Supreme Court's Zelman decision in 2002 established a set of conditions in which a program like this would not violate the establishment clause of the First Amendment. The five-part test is:

(1) the program must have a valid secular purpose
(2) aid must go to parents and not to the schools
(3) a broad class of beneficiaries must be covered
(4) the program must be neutral with respect to religion, and
(5) there must be adequate nonreligious options.

Thanks skoolboy. Reason #2 seems the loophole which allows the whole thing to work. Although the money eventually goes to the school, it is laundered through the parents. Am I understanding the principle correctly?

Comments are now closed for this post.


Recent Comments

  • elton: Thanks skoolboy. Reason #2 seems the loophole which allows the read more
  • skoolboy: Elton, Other more lawyerly types may weigh in here, but read more
  • Elton: Being new to this debate, I'm curious how these voucher read more
  • Paul Hoss: The results, regardless of how the Secretary attempts to spin read more
  • rory @ parentalcation: Please forgive grammatical errors above. read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School