Last week, MDRC released the latest in a series of careful evaluations that suggest New York City’s “small high schools” delivered some promising results. The new findings suggest that students at small high schools are more likely to enroll in college and that small schools boast a lower cost per graduate. This is smart, informative research. But I found some of the commentary on the results to be more than a little disconcerting in its suggestion that we now know that small schools “work” and that practitioners, policymakers, and funders should proceed accordingly. It all brought to mind the old fable of “stone soup.” If you don’t recall the tale, here’s Wikipedia’s (particularly dull) telling:

“Some travellers come to a village, carrying nothing more than an empty cooking pot. Upon their arrival, the villagers are unwilling to share any of their food stores with the hungry travellers. Then the travellers go to a stream and fill the pot with water, drop a large stone in it, and place it over a fire. One of the villagers becomes curious and asks what they are doing. The travellers answer that they are making ‘stone soup,’ which tastes wonderful, although it still needs a little bit of garnish to improve the flavour, which they are missing. The villager does not mind parting with a few carrots to help them out, so that gets added to the soup. Another villager walks by, inquiring about the pot, and the travellers again mention their stone soup which has not reached its full potential yet. The villager hands them a little bit of seasoning to help them out. More and more villagers walk by, each adding another ingredient. Finally, a delicious and nourishing pot of soup is enjoyed by all.”

For readers who haven’t yet had their coffee, I’ll make the link a little more explicit. Keep in mind that educational interventions like small high schools aren’t really “interventions” in any classic sense. Rather, as pursued in New York City, for instance, small high schools actually entail a gaggle of changes (most of which go unnoticed). The “smallness” factor is the stone that gets mentioned, observed, and casually cited as the causal factor, but many (or most) of these schools also were schools of choice, enjoyed support from external experts who were passionate about the small high schools, had waivers from the master contract, were staffed by faculty enthusiastic about the model, and benefited from initial philanthropic support. This does not mean MDRC’s findings are wrong, nor does it question the benefits of the small schools effort.

It does, however, mean that we ought to be careful about the takeaways if we conclude that small high schools “worked” (here we’re dealing with what researchers call “external validity”). It may be that smallness is responsible for all the observed benefits. But long experience with so many other initially promising reforms leads me to think that the above-the-marquee “reform” often amounts to the stone in the soup, and that the enabling conditions often play a big role in producing the outcomes. If imitators are to reap the benefits of the reform in question, they need to replicate the entire soup (carrots, seasoning, and everything else included). Unfortunately, it’s really hard to replicate the enthusiasm, foundation support, and intensive hand-holding from experts that accompany pilot efforts, or to broadly mimic the contract waivers and choice conditions that were intrinsic to the initial design. So, having heard about the wonders of stone soup, other villages go and obtain their own stones, add water, boil it, and then drink away--only to be disappointed to find that they have pebble-laden water rather than the promised broth.

There’s nothing here that’s unique here to the small schools effort. In fact, I wrote about this same phenomenon a decade ago in Common Sense School Reform, terming it the “law of the aluminum bullet” (e.g. when silver bullets turn out not to be so silvery after all). Sensibly interpreting the lessons of promising pilots has plagued school improvement for, well, pretty much forever. We need to be more thoughtful about looking for “omitted variables” when making sense of results--and that can be bloody difficult when there’s enormous pressure to find “what works” and “scale it up” in a hurry. A good first step is to ask ourselves whether we’re just eyeing the stone, or whether we’ve actually copied down the whole recipe--and are able and willing to provide everything it requires.