Sol Stern and the SUTVA Shenanigans
What does interference mean? The idea is that Serena's outcome should not be affected by whether her peers Blair and Vanessa were assigned to the treatment or control condition. In other words, one subject’s outcome should depend only on the treatment to which that subject is assigned, not on the treatment assignments of other subjects. But in many social settings, there are peer effects and social spillovers and, as a result, others' treatment assignments likely do affect one's own outcomes. Vaccinations provide another clear example - my risk of contracting a disease is dependent on my exposure, which is in turn affected by whether others have been vaccinated. In the case of Reading First, one school's treatment likely affected what other schools in the district did, as Stern details in his article.
The lesson here is not just about the Reading First study. Experiments in social science are fundamentally different than experiments in medicine, and it turns out gold standard is often more silver or bronze than we would have hoped. Don't get me wrong - I still dig experiments - but I'm not counting on them to solve all of the problems that vex our schools.