Over at Empirical Education yesterday, blogger Denis Newman made a case for replacing big, national experiments to find out what works in education with dozens of smaller—and less expensive—local experiments.
Newman, who is the president of the Palo Alto, Calif., consulting firm that hosts his blog, pitches the small-experiment idea in response to a blog post last month by Office of Management and Budget director Peter R. Orszag. If you’re a regular reader of Inside School Research, you’ll recall that Orszag shared some of this thinking in that post on the need for more evidence-based policymaking in government. (See my write-up from last month for details.)
The problem with Orszag’s plans, Newman says, is that he has “bought into the idea that a single, national experiment” will yield useful information on whether a government policy or program is effective. Borrowing from Donald Campbell’s concept of an “experimenting society,” Newman says 100 or more local experiments might be a better way to go. He writes:
First, the education domain is extremely diverse and, without the '100 locally interpretable experiments,' it is unlikely that educators would have an opportunity to see a program at work in a sufficient number of contexts to begin to build up generalizations. ... Second, the information value of local experiments is much higher for the decision-maker who will always be concerned with performance in his or her school or district. National experiments generate average impact estimates, while giving little information about any particular locale.
He also contends that local experiments are faster than national studies at as little as one-tenth the price.
This is no idle chit-chat. Newman and his colleagues recently completed a study on the feasibility of local experiments for the federal Institute of Education Sciences. You can request a copy of the study here.
.