The way you describe this, there's really not any drama here.

- In the first situation, they pointed out that a study they don't think reports reliable information, based on methodology.
- In the second situation, they say that charter schools are doing well, based on a different study.

Not sure what you're aiming to convince me. If the problem is their tone in the first or second situations, I feel like it's kind of pot-meet-kettle ("too precious", "newsblast", "You can't make this stuff up", "Mr. Bogus" ...not really neutral tone).

I tend to think it's a good thing when people and groups show that they have the ability to change.

Hi Dave,
The methodology - a raw comparison of charter versus public average scores - is identical in both cases. The only difference is the finding.

Put simply, I find it troubling that interest groups mount massive campaigns against research findings that don't agree with their viewpoint, but do a 180 when an analysis with the same flaws does agree with their viewpoint. I don't think this is positive - or bodes well - for the public education debate.

Oh my, dear eduwonkette! We are so delighted by your coverage of the most important issues of the day. Alas, you broke the first cardinal rule of wonk-dom -- KNOW THY FACTS. We wonks and wonkettes must simply stick together, you know.

The extraordinary work of dozens of prize-winning researchers that signed the famous New York Times ad (including Nobel Laureates) spelled out the problem with bad journalism clearly. The New York Times had used an analysis of government data by a known opponent of charter schools to claim that charter schools were lagging other schools. It was akin to Philip Morris saying that tobacco is safe!

Besides the source, my dear, there is the sticky little issue of what the data really said. In this case, it was sample data of students in 4th grade compared across communities and states! In the case of the Washington Post's journalism this week, reporters -- not unions -- analyzed the results of test scores from ONE community, and amidst those test scores are data for both (EVERY) conventional and charter public school, not just a sample, allowing for apples to apples comparisons.

If you are saying you don't believe the Post, that's okay. Many others have felt that way for years. However, if you are saying we are biased or have a double standard, you are simply wrong, lovely little wonkette. We respectfully beg to differ.

In the coming days, in between other important elf duties, we will show you more. For now, we bid you a fond farewell and a wish that in addition to world peace, we all embrace the notion of digging deeper than just what seems to meet the eye.

Please join us over on Edspresso.

Hi Jeanne, Thanks for stopping by. That this is one community does not resolve one of the central critiques made in your ad and in that debate - that "The study in question does not meet current professional research standards" - because it does nothing to address the issue of selection into charters.

More concretely - your logic above about the methodological differences in the two studies (comparing within one city and sampling) suggests that you would be okay with comparing the average scores of charter and non-charter schools within DC, not just the scores of disadvantaged kids. In that case, DC charter schools are either no different or behind in 4th grade math and reading. It would be a mistake to make a facile comparison like this for all of the reasons that I discussed in the earlier post, and I suspect that charter advocates would be up in arms if the article instead foregrounded this point.

Unfortunately, Jeanne Allen is dead wrong on this one.

There are virtually no differences between the methods used by the AFT (and publicized by the Times) and the methods used by the Post reporters.

For one, both studies use NAEP. The NYT ad spends considerable space pointing out NAEP's limitations for these kinds of statistical inferences. (They are only "snapshots," with limited family background information attached, etc).

Second, both studies are straight comparisons of mean scores. The NYT ad criticized the "unsophisticated analysis" used by AFT, that "considers differences in only one family background characteristic at a time." The exact same thing can be said for the Post: they control only for economic disadvantage. Jeanne is correct in pointing out that this comparison is within one district (DC), which is more sensible than a nationwide comparison. But the advantages end there.

I can't speak for my economist colleagues who signed the NYT ad. But I am confident that all of them recognize that the Post analysis is NOT an "apples to apples comparison," by any stretch of the imagination.

This is why these very same researchers strive in their own work to base their causal claims on randomized experiments, or quasi-experimental evidence that approaches randomization. Only in these cases can we be confident of an "apples to apples" comparison.

Jeanne Allen and the NYT signers were correct in pointing out that we need higher quality research than the AFT study to make causal claims about charter schools. The same standard should be applied here.

Sean Corcoran
New York University

I am new to the discussion and have much to learn. I do have a thought about the point that has been made about comparing the success of students in charter schools versus the students in non charter schools.

It seems valid to say that students and parents that make the step of choosing to attend a charter school have more initiative, care more about their education, and are more likely to succeed wherever they attend school. The argument that comparing students that attend charters and students that don't is not a valid research experiment makes sense in that light.

However, doesn't the fact that people have accepted as fact the idea that an engaged, diligent, responsible parent that wants the best for their children will choose a charter school basically mean that we believe that charters are the better option? Or at least that the state of the traditional schools is so poor that responsible parents are right to want their children to have a better alternative?

Doesn't that logic make sense?

Chad: I think you're right that it would be reasonable for well-informed parents in DC to consider charters as an alternative for their children that may be better than the traditional DC public schools. My criticism of many articles on charter schools is that they falsely make the assumption that the charter schools themselves are somehow better schools (by dint of teacher quality or other variables) without taking into consideration this selection bias. I think it's quite possible that a school that accepts only motivated students with a demonstrated desire to improve their educational attainment may be a good place for its students. But I haven't seen any evidence that this is improvement is the result of anything the charter schools do over and above the effects of accepting only motivated students and holding them to higher levels of effort and behavior than traditional public schools are allowed by law to do.

Pretend the data were trouble free, the students were randomly assigned, the data were from elementary and high schools, not just middle schools, and that the test scores were the sole criteria of success.
Now, group the KIPP schools-- which are part of their own system, not independents --together with the single original DCPS school,enrollment 600, that went charter long ago to sustain its high achievement, and still serving about the same mostly black nad not well of demographic in the same building. (And here's a shout-out to Paul JHS principal Barbara Nophlin, (formerly with DCPS)!!)
Findings:
1)The students in KIPP and Paul JHS PCS outperform the rest of the charters
2) the Paul students are indistinguishable from the KIPP students; and
3)there is no statistical difference between the rest of the charters and the DCPS schools.

3 groups, 2df, & 1 statistically significant difference. Schools out.

I press the plunger to stop time running on my clock. No apology for excessive intimacy with the data. Over to anyone else to now talk about demonstrated charter school "effects" on achievement in DC

One of the most surprising things I learned when I became a teacher in 1964 was the close correlation between a parent's interest in his or her child's education and the level of achievement of the child. So even in the worst schools in Cleveland, a child with a very involved parent would do fairly well. Often that parent would look for ways to get the child into a "better" school. In 1964 that meant a parochial school or a school on the "west side." (more affluent part of town)

When I had children of my own I became acutely aware of the behavior of the other children. My top priority became the safety of my child. When things got too rowdy at the local public school I put my sons in a Catholic school. I had no knowledge of the teachers but I knew that the students were well-behaved; that's what I was after. My sons went on to the best universities in the country and are successful professionals today.

I shared this story because there are just so many variables involved in a child's education that it is close to impossible to say why one school is "better." Did the Catholic school I sent my sons to have better test scores because of better teachers or did they have a select student body? I'm almost certain it was the latter because all parents chose the school and paid tuition. Almost any school chosen by parents will appear superior to a school that accepts children who are assigned to it.

I have a question for Jeanne Allen. Do you think most citizens realize how powerful non-school factors are? I didn't have any idea until I started teaching. Thanks.

One of the most surprising things I learned when I became a teacher in 1964 was the close correlation between a parent's interest in his or her child's education and the level of achievement of the child. So even in the worst schools in Cleveland, a child with a very involved parent would do fairly well. Often that parent would look for ways to get the child into a "better" school. In 1964 that meant a parochial school or a school on the "west side." (more affluent part of town)

When I had children of my own I became acutely aware of the behavior of the other children. My top priority became the safety of my child. When things got too rowdy at the local public school I put my sons in a Catholic school. I had no knowledge of the teachers but I knew that the students were well-behaved; that's what I was after. My sons went on to the best universities in the country and are successful professionals today.

I shared this story because there are just so many variables involved in a child's education that it is close to impossible to say why one school is "better." Did the Catholic school I sent my sons to have better test scores because of better teachers or did they have a select student body? I'm almost certain it was the latter because all parents chose the school and paid tuition. Almost any school chosen by parents will appear superior to a school that accepts children who are assigned to it.

I have a question for Jeanne Allen. Do you think most citizens realize how powerful non-school factors are? I didn't have any idea until I started teaching. Thanks.

Almost any school chosen by parents will appear superior to a school that accepts children who are assigned to it.

A corollary to this -- that benefits charters when compared to district schools -- is that almost any impediment to enrollment, even having to sign up for a lottery, will select for motivated, involved parents, and change the student body in subtle ways that a simple comparison of income or parental education wouldn't illuminate.

The harder issue is that these school may actually provide a better educational environment simply by selecting for motivated parents. Where does that leave the kids of uninvolved, unmotivated parents? Should education be tracked that way, and if so, to what extent?

What is it reasonable to expect from parents? Signing up for a lottery? Checking homework every night? Coming in for a weekly conference? Volunteering 4 hours a month? Volunteering 4 hours a week?

What sort of financial commitment is consistent with public education? Buying uniforms? Voluntary purchases of wrapping paper and magazines? Voluntary $10 PTA dues? Voluntary $100 PTA dues? Voluntary $1000 PTA dues?

My experience supports Linda's view that an engaged, informed, and interested parent will find a way to make sure their child is in position to receive the education and training needed to be a responsible, contributing member of society.

Is there a consensus that increasing the amount of parents that fit this description is the best thing we can do to improve the education of as many children as possible?

If so, are there any success stories where such an initiative has taken hold?

My experience supports Linda's view that an engaged, informed, and interested parent will find a way to make sure their child is in position to receive the education and training needed to be a responsible, contributing member of society.

Is there a consensus that increasing the amount of parents that fit this description is the best thing we can do to improve the education of as many children as possible?

If so, are there any success stories where such an initiative has taken hold?

Chad -- I think parent outreach and education is a priority for many districts (it certainly is for ours) but I haven't seen much national-level of discussion about what is particularly effective.

"Almost any school chosen by parents will appear superior to a school that accepts children who are assigned to it.

A corollary to this -- that benefits charters when compared to district schools -- is that almost any impediment to enrollment, even having to sign up for a lottery, will select for motivated, involved parents, and change the student body in subtle ways that a simple comparison of income or parental education wouldn't illuminate."

Following this same logic, could we then conclude that adoptive parents--who have lots of hoops to jump through in order to become parents--are better, more involved and highly motivated parents than the biological kind?

Margo: I don't think that's the point of these comparisons. Obviously, many adoptive parents are very involved and caring, just as many biological parents can be involved and caring. The point is that articles comparing the academic achievement of charter schools vs. public schools do not always account for the selection bias that goes into the charter schools' student populations.

Articles often state that the SES characteristics of public vs. charter school students are roughly equivalent. They then conclude that the two groups of students are basically the same going into the school year - and further conclude that all differences in performance by the end of the school term are due to some innate characteristic of the charter schools themselves. Some of the posters on this site have pointed out that there is a selection bias in charter and private schools: Only families who are tuned into their kids' education to a certain degree will apply. Therefore, the student populations are NOT equal.

An appropriate analogy might be comparing students who try out for the school soccer team with those who don't attend tryouts - having them play in an end-of-year soccer tournment. The kids who take the time to try out for the team (even if they don't make it) will probably be, on average, better players than the students who don't try out for the team. Assuming the soccer team kids beat the non-soccer team kids in an end of year match-up, assigning all the credit to the soccer coach is not appropriate b/c the student groups are not the same at the beginning. Even if they are they have the same general athletic ability, there is obviously a level of motivation that the "tryout" kids possess.

Atty:

I get the point, and I was being a bit tongue in cheek with my suggestion--but not totally. Kids are pretty randomly assigned to bio parents, and generally stay with that randomly assigned family unless the parents are really profoundly deficient, over time.

Adoptive families have to go through a selection process that makes applying to a charter school like like a walk in the park on a sunny day. They have to be motivated to even go through all that--not to mention that fact that nobody ever becomes an adoptive parent by accident. They are frequently expected to be not only open to the demands of parenting, but frequently expected to be open to the demands of parenting the neediest of kids (those whose bio parents were deemed profoundly deficient, over time).

This comes to mind because adoptive parents are so frequently blamed by various school and other authorities for "having" children with so many problems--but also because it seems to be one of the least important questions to be asking. I understand the importance of taking into account selection bias in looking at school comparisons--but I would prefer to be looking at something more substantial. Are there critical differences in teaching methodology? In class size? In experience? In credentials? In per pupil expenditure? In administrative cost vs instructional cost? In any of the things that can be controlled within the school? We are so enamored of clumping together unlike school entities with a single commonality (charter, public, private, etc). Underlying the concern for parent selection bias, it seems to me, is a belief that somehow kids arrive at the school door with innately determined capabilities that schools can do little to impact.

Yes, families have an impact. Knowing that, and 4 bucks, will get you a cup of coffee from your local barrista. I think that the more critical question is what can we do within the schools that we have right now, and the students that we have right now to get closer to a universal level of basic literacy and numeracy?