« LDH, IES and the Reign of Frogs | Main | The Skillful Publicist »

The Boston Pilot/Charter School Study: Some Good News, and Some Cautions

When Robert Pondiscio pages, I answer. Yesterday, the Boston Foundation released a study on the efficacy of charter and pilot schools, which had the advantage of including both observational estimates of these schools effectiveness (comparing the performance of students in these schools with those in traditional public schools, net of some control variables) as well as lottery-based estimates of their effectiveness (comparing those who won lotteries with those who did not).

Kudos to the research teams (Atila Abdulkadiroglu, Josh Angrist, Sarah Cohodes, Susan Dynarski, Jon Fullerton, Tom Kane, and Parag Pathak) - this is a well-done, careful study that provides us with a range of estimates of charter and pilot school performance. There is certainly enough positive evidence here to support the creation of more charter schools in Boston, but I want to offer two cautions.

1) Beware of extrapolating lottery-based estimates when only the most successful charter schools had oversubscribed lotteries to being with.The lottery-based estimates of the effects of charters on middle school math performance are simply huge - half of a standard deviation! But only a quarter of charter schools are part of the lottery sample. Zero of the 5 charter elementary schools, 4 of 13 charter middle schools, and 3 of 11 charter high schools are included in the lottery part of the study because other schools were not oversubscribed enough to require a lottery. To be sure, these lottery estimates are important to understanding what the most successful charter schools can do. But I think we will all be disappointed if we expect a charter school expansion to replicate the effects of these middle schools (Academy of the Pacific Rim, Boston Collegiate, Boston Prep, and Roxbury Prep), which are among the highest performing charter schools in the country.

2) Pay careful attention to what elements of these schools can and can't scale. Harvard graduates don't grow on trees. These schools have faculties drawn from the most selective colleges - faculties that are highly atypical of the public school teachers.

At a bare minimum, The Boston Foundation study demonstrates conclusively that the charter schools in the study are adding substantially more value to student achievement than district schools, thereby debunking once and for all the myth that their success is based on self-selection bias.

As is being demonstrated by high performing charter organizations across the country, such as Aspire, KIPP, Uncommon Schools, Achievement First, Mastery and Noble Network (the list goes on), it is possible to replicate successful charter schools without sacrificing quality. In addition, thoughtful authorizing and hard-nosed accountability -- for which Massachusetts is well known -- can and should ensure that quality is not sacrificed on the alter of scale.

The bottom line is the cap is an arbitrary and ultimately political artifact that cannot be justified on educational grounds.

It may be a little premature to say that this report debunks "once and for all" the notion that self-selection is a factor in charter school success. Given that a study of the DC voucher system from several months ago showed no appreciable difference between the achievements of kids who were accepted by a lottery and those who did not, we're left with dueling studies at the least. Also, given the high attrition rates at KIPP schools, how willing are we really going to be about saying that they are replicable on a wide scale? Certainly, this is a good and interesting study that hopefully spawns more of its kind. But it strikes me as more of a beginning than an end.


Plase explain further what you meant by this statetment: "Harvard graduates don't grow on trees. These schools have faculties drawn from the most selective colleges - faculties that are highly atypical of the public school teachers."


You seem to disparage the attrition rates at KIPP schools. Are you aware that 85% of KIPP schools are middle schools and their graduates naturally move on to other (high) schools?

Paul Hoss

Hey Paul,

My point was that these schools rely heavily on young teachers from selective colleges, and they are inherently limited in number. You can see a list of the universities these teachers attended here: http://www.aei.org/research/Education/subjectareas/projectID.31/default.asp

It's important to note that there's a difference between graduation and attrition (those who leave the school before graduating). A study of San Fransisco area KIPP schools found that 60% of the students who entered the schools in fifth grade left BEFORE they graduated on to a high school at the end of eighth grade.

The study shows that the KIPP schools do a good job for the kids who remain and stick with the program, but if that's only 40% of an already quasi self-selected group, I don't think we can honestly claim that KIPP is replicable on a large scale.

It is heartening to learn that students who got into Boston charter schools with higher levels of achievement (Table 4) themselves went on to achieve more per year. It could have been otherwise, as there has been some discouraging from public housing dispersal experiments. But, have these students closed the gap with their peers in the same charter schools? Have they closed the achievement gap with students from average schools in the Boston suburbs? Would the economists and econometricians all who prepared these regression standardizations enroll their own children in these charter schools?
They'd have to visit them to learn more about them, first? Don't they already know enough about them from the mean gains per year in performances to vote their own children? If they don't know, why would I 4 hours into study of their report?

I'll still choose schools on the basis of Ofsted.gov.uk -like audits and inspections, which begin where such as this leaves off.


eduwonkette is correct, of course, about the nature of the schools. To circumscribe the sample even further:

"On balance, our lottery-based findings provide strong evidence that the charter model has generated substantial test score gains in high-demand Charter Schools with complete records."

Because, as every qualified education inspector knows, lousy schools keep lousy records, which contributes to their problem in knowing where their students are in their development. (That is not a claim for more testing.)

There seems to be ready willingness here to acknowledge that the good/brand name schools do in fact get impressive gains compared to their district counterparts, even if we don't know much about the random schools. That's a pretty remarkable outcome by itself, but it also points to the success of the charter idea: let a thousand flowers bloom, some successful ideas will emerge, and then they'll proliferate. If some, or even most, schools aren't so good, that's to be expected. Since the first successful models (many of which were born in Boston and have since spread in one way or another to other cities) emerged, they've multiplied and have been mimicked. That's how a marketplace is supposed to work.

Oh, and John, your cheap shot on KIPP is unwarranted and unproven. There's no evidence that their attrition is any higher than their sending districts' is. Some of their schools have very high attrition, and some of them have very low attrition. Trying to take a bite out of this survey (the kind of survey the data buffs have been calling for for a long time) by making unsubstantiated accusations about KIPP just proves what I'd suspected all along: for some people, no survey, no matter how valid and how well administered, will be convincing.

It seems to me that the most difficult question these result pose is whether a precondition for a successful school is the ability to (in one way or another) avoid dealing with the most difficult students and the most disengaged parents.

This is a slightly different issue from simple self-selection. If the non-selected lottery kids worse in district schools than the their selected peers do in the charter school it may be because of the quality of the teaching, or the nature of the curriculum. Those would be potentially "fixable" issues.

But it also may be that the lagging outcomes for the non-selected kids is the result of the environment created by their fellow students with a higher concentration of discipline problems and dysfunctional families. That's a difficult problem to fix. Do you create charter schools for all the "good" kids and let the district schools become warehouses for the rest?

That seems to be the logic of the charter school movement at the moment. But though I can see the short-term logic of it, the longer term outcome seems to me to be the re-creation of a public school system segregated -- perhaps even fairly finely segregated -- on the basis of class and parental aspiration.

Rachel: I agree with your comments, above. I made similar remarks in a comment yesterday on Robert Pondiscio's blog through the link in eduwonkette's post. It's almost undoubtedly true that students benefit from being in schools only with other motivated and engaged students: The problem is how to make the rest of the students succeed as well. Therefore looking at charters as the panacea to all student achievement ills is not realistic.

The debate about the limitations of the study's methodology is very interesting, but it ultimately misses the point. The reality is there is no perfect study and there never will be. Charter opponents keep saying that they need more proof before agreeing to add more schools. But the proof of district failure is all around us. The TBF study and many others should provide policymakers with enough evidence of charter success to justify expansion. This does not mean policymakers should open the flood gates to any charter operator that comes along. High standards for granting charters and rigorous accountability for results ensure that any risk of weak charter performance is greatly mitigated. Why is it charters must prove beyond a shadow of a doubt that they are consistently and substantially superior to any alternative, when school districts don't have to prove a thing to maintain their protected franchise? I think we all know the answer to that question.

"But it also may be that the lagging outcomes for the non-selected kids is the result of the environment created by their fellow students with a higher concentration of discipline problems and dysfunctional families."

Rachel--certainly you pose a question worthy of more research. The first step in constructing such research, however, would be the careful definition of terms. As a social worker who grew up in an alcoholic (and upper middle class) family, I am very wary of the loose application of the term "dysfunctional family," particularly when looking at a population of students who might better be characterized as low-income or minority. I have certainly played the funding game long enough to have counted up percentages of minorities, single parents or other obvious factors and plopping them down into need statements (depending on what was in funding vogue at the time). And yet I know that the very real dysfunctions of the family I grew up in were nearly invisible to everyone who knew me as a child or youth (and others like me), while my current family might be mis-classified as dysfunctional based on several totally unrelated criteria (single parent, low-income neighborhood, minority kids, underachievement, discipline challenges).

As someone who has worked extensively with such families, as well as some of other income echelons, I would suggest that the assumption of rampant dysfunction in schools serving low-income and minority populations (particularly compared to majority and middle/upper class populations) has more fallacy than truth to it. I can, however, point to some real differences between schools serving those populations. A teacher who assumed that most of the kids were poorly supervised because their parents were addicted would be regarded as crazy in one school, but astute in another. In one school it is assumed that parents care about their offspring (perhaps too much) while in the other it is assumed that most care too little. To the extent that their is research into these things (the prevalence of addiction across socio-economic strata; the aspirations of parents for their offspsring), these things are at best far less true than they are assumed to be.

But, from a school perspective, I would really want to know what is going on in those classrooms to make a difference among similar populations. That is the place where we have the most opportunity to bring about changes.

Margo/Mom -- I agree that family dysfunction cuts across income line. But I think its likely that dysfunctional families of any income level are under-represented in schools that require commitments of parental involvement compared to their more "together" peers.

So one question is, if "schools of choice" work well for the students whose parents choose them, what do we do for the kids whose parents don't choose them?


So, one of your indicators of family function/dysfunction would be whether the family selects a school rather than going along as assigned? Is "commitment of parental involvement" a hallmark of charter schools? What specifically is required and does it do anything substantial for education, or does it primarily serve as a means of eliminating families whose parents are not good brownie-bakers? I don't see Barack and Michelle volunteering to wash cars on a Saturday. Does that make them underinvolved? Dysfunctional?

Again--I really believe it is necessary to tighten up this concept of family "dysfunction." What precisely do you mean by that? I would tend towards looking at things like nurturance and emotional availability. And, I would hold that these things are not well indicated by income level, ethnicity or "school involvement" as it is typically envisioned.

Margo/Mom wrote: "Is "commitment of parental involvement" a hallmark of charter schools? What specifically is required and does it do anything substantial for education, or does it primarily serve as a means of eliminating families whose parents are not good brownie-bakers?"

In reply, I would say that committment of parents is definitely a hallmark of charter schools (and private/other selective schools). Of course, not all committed parents send their kids to charter schools, but I'd bet that most kids in charter schools have at least one committed parent.

My guess is that the advantage is less that the parent is doing anything (e.g., baking brownies) for the school. It's that selecting for involved parents means that the kids are more likely to get help at night with their homework, have someone watching over them/making them get up for school in the AM, and just generally be more supported. I believe these characteristics of families raise the odds that the kids will perform well in school.

Attorney DC outlined my take on the issue pretty well.

At least in this area, parent involvement -- in terms of required volunteer activity -- is one of the most common features of charter schools. Some also strongly suggest a fairly high level of parent financial support (one in our area asks $3000 per child per year) though I doubt that's typical.

Dysfunctional is probably an overly broad word. But the bottom line is that charter schools are places where all the students are there because their parents made an active effort to get them there -- and that's going to be a different environment from one where most kids are there because they happen to live in the neighborhood.

I'd like to see much more discussion focus on what successful innovations from charter schools can be implemented in district schools, and less on "who's better."

Rachel: I agree with you that I'd like to see more discussion of what strategies or innovations (to use your word) charter schools are using that might be replicated in public schools. Simply saying "this charter gets better results than that public school" doesn't tell us much if we don't know what (if anything) the charter school is doing to attain its results. Similarly, when public schools outperform charter schools, what underlies their achievements? That is, if we can ever separate teacher and administrative strategies out from the underlying inherent characteristics of the students themselves (and their families) who attend each school.


What evidence do you have, beyond supposition, that charter schools have more committed/involved parents? And what's your evidence that the district schools are full of parents who don't care enough to commit the simple act of enrolling their children in a charter school? They enrolled their children in the district schools, didn't they?

Assuming your answer to the last question is that it's easier to enroll in a district than in a charter school, then what's your evidence of THAT?

If your answer is that district school-enrollment is compulsory, so what? If a parent is going to enroll his or her child in a school, I don't see why it isn't just about as likely to be a charter school as it is to be a district school.

I separate "parent involvement" into three types:

1. Involvement at the school/district level by attending general school meetings, activity in parent organizations, etc.

2. Involvement at the classroom level by providing support to teachers (eg. field trips), giving hands-on help during the school day (bake sales), etc.

3. Involvement at the home level by monitoring attendance and schoolwork, tracking grades, working cooperatively with teachers when issues arise, staying vigilant about peers and drug use, etc.

I've known families who have done plenty of #1 and #2, but have fallen down with some of the things in #3, and their children have struggled with school success.

I've also known families who never attend a single PTA meeting or volunteer in a classroom, but they do a lot of the things under #3. They are largely, but not always, Chinese families where I live.

These low-income families almost always have two parents and often have extended family support. They generally do a beautiful job of priming their childrens' educational pumps, despite not speaking English, working long hours, and living in dangerous neighborhoods.

The third type of parent involvement is by far the most powerful, in terms of benefit to the child. Any school that can get enough students from these types of families will succeed every time. Because these parents are the way they are, they are more likely to consider charter schools.

This looks to me a good study, though it doesn't attempt to analyze why Boston charter schools are more successful in raising student achievement. There are two possibilities that come to mind, and both may be operative.

1- Class Size -- an examination of the descriptions of a random selection of the charter schools included in the study, either garnered from reports from the MA state education dept. or self-reported by the charter school itself, shows that classes tend to be much smaller than those in the regular BPS system, with some classes as small as 15 (as in the Boston Collegiate Charter School). In most of these schools, classes appear to be limited to about 18-22 students; this is far smaller than class sizes in the regular BPS system of 25-28. In fact, many of these charter schools openly state that their class sizes are critical to their success. In general, it is easier for charter schools to reduce class size because they are able to cap enrollment, unlike traditional public schools.

2- Peer effects. Though the study finds that the randomly selected winners of the lottery who attend charter schools do better than who lost and thus remained in the BPS system, the study also states that "Charter Schools also serve a smaller proportion of special education students, free- and
reduced-price lunch students, and English learners than
do the traditional BPS schools. In addition, high school Charter students tend to come in with substantially
better math and ELA performance on the MCAS..."

So to the degree that the "winners" of the lottery are surrounded by better performers, this in itself is likely to help boost their achievement levels.

Tom Kane has been quoted as saying that "The next step is to identify what's working in charter schools that can be transferred back into the traditional public schools to improve student achievement." Let's hope they manage to achieve this.
I am aware of few such charter school studies in the past.

Comments are now closed for this post.


Recent Comments

  • Leonie Haimson: This looks to me a good study, though it doesn't read more
  • Pondoora: I separate "parent involvement" into three types: 1. Involvement at read more
  • Socrates: AttorneyDC, What evidence do you have, beyond supposition, that charter read more
  • Attorney DC: Rachel: I agree with you that I'd like to see read more
  • Rachel: Attorney DC outlined my take on the issue pretty well. read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City Budget cuts
New York City budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School