The Full Page Ad That Won't Be in the Washington Post Tomorrow
To be sure, students are not randomly assigned to charter schools, so these critiques were not without merit. So keep your eyes on the Washington Post over the next week and see if charter school advocates again swoop in to defend educational research from the bad guys who would misuse it for their own purposes. Yesterday, the Post published an almost identical analysis claiming that DC charters, which currently enroll a third of DC students, "have opened a solid academic lead over those in its traditional public schools."
Again, for those who are willing to endure the occasional recycled rant on selection versus treatment effects:
First, that students selected into a charter lottery makes them different from those who did not. It may be that their parents are more involved in their education, that they are having a particularly bad experience at their neighborhood school, or that their parents can no longer pay for private school. Whatever the reason, families selecting in, even if they are all poor and minority kids, are different by virtue of choosing a non-neighborhood school.
Lots of choice advocates will spar on this point, and argue that everyone wants a better choice for their children, so there is no selection problem. While rhetorically effective, anyone arguing that families that choose into a charter school are the same as those who don’t is simply wrong. Saying that two-thirds of the kids are poor and that the overwhelming majority are African-American and Hispanic doesn't solve this problem. (Btw, in the small schools context, this is Joel Klein's favorite pasttime.) Even if charter kids had prior test scores identical to their neighborhood school peers, we still couldn't compare charter and neighborhood school kids who didn't opt in with any confidence because there is selection on unobservables - things like motivation and aspirations that are not measured by administrative datasets used to make these comparisons.
The only defensible approach here is to compare students who entered the charter lottery and won with those who entered the lottery and lost. But if the Post wants to extend its logic, here's a free story idea from your friendly neighborhood Secret Santa: go ahead and compare the NAEP results of charters and public schools nationally (and for extra credit, go one step further and compare the outcomes of free lunch kids in charter and traditional public schools), and let's get the Charter War of 2008 started.