Wish #3: Asking More "Why?" Questions
Earlier this month, a team of researchers at MIT and Harvard released a report contrasting the impact of charter schools, “pilot” schools, and traditional public schools on student achievement. The finding of charter school effects on achievement, using a random assignment research design, fueled the rhetoric of charter school advocates, some of whom saw the findings as a license for unlimited expansion of charter schools.
The researchers themselves were more cautious. They acknowledged that the study was not designed to discern why the effects were found. In fact, if the study had found that students in charter schools had shown less growth in achievement than students in traditional public schools, they wouldn’t have known why either.
Good public policy depends on compelling answers to “why” questions about both the observed effects and non-effects of policies and programs. And these “why” questions pertain both to the inner workings of policies and programs as well as the context in which the policies and programs are situated. Borrowing policies that have been found to be effective in one setting and expecting the same results in another setting makes sense only if we know why the policies were effective in that first setting. A research study showing that a policy or program “worked” in a particular setting doesn’t tell us that.
Our wish, then, is for asking “why?” more loudly, and earlier in the lifecycle of a policy or program. Why might achievement be higher in charter schools? Why do children learn more in smaller classes? Why are some teachers more successful in teaching low-achieving students than high-achieving students? Why don’t school expenditures have a stronger association with student outcomes? In skoolboy’s view, the real leverage in education policy comes from good answers to the “why?” questions. To paraphrase Jim March, research that addresses “why?” questions is more useful than research that addresses “what works?” questions because it has so many more applications.
One challenge posed by our wish is that the researchers who are skilled at addressing “what works?” questions are not necessarily the ones who are good at addressing “why?” questions. Even in large federal evaluations, there typically is a division of labor in which the study of implementation and context is segregated from the study of program impacts, and different research organizations or researchers are responsible for differing parts of the overall enterprise. Asking “why?” more often will require some hard thinking about research training and the infrastructure for education research in the U.S.
Comments
It seems to me, as a parent, and not a formal or professional educator, that the best way to figure out why a particular learning environment works is to go experience it. Be there for a day. Talk to the kids and the adults. It would seem to me that in most cases it will quickly become obvious why something works.
What I like about charter schools, or at least the promise of charter schools, is to bring some variety to parents' public school choices... the more the better. Problem is, the criteria to evaluate charters, as well as conventional schools are so 19th Century... the transmission of approved content, rather than 21st Century knowledge acquisition, creativity and collaboration skills.
Cooper Zale
www.leftyparent.com
Posted by: Cooper Zale | January 22, 2009 12:31 PM
skoolboy: Interesting questions. I'd love to hear the answers! If you get any good data on "why" definitely let us know.
Posted by: Attorney DC | January 22, 2009 12:32 PM
Jay Mathews new book "Work Hard. Be Nice: How Two Inspired Teachers Created the Most Promising Schools in America " is scheduled to be released today. KIPP's primary hurdle to large scale adoption is they have no magic bullets,they spend more focused time teaching. If you really want KIPP's excellent results, no more 180/6hr per day school years. The KIPP school year looks as if they read "The learning Gap" by Stigler and borrowed the day/week/year schedule (240 days per year)employed in Japan. I'm waiting for a graduate student in statistics to tell us how many KIPPs there could be and how many more Ghetto kid's would be succeeding if we wanted to actually close more of the achievement gap.
Posted by: John Stallcup | January 22, 2009 4:39 PM
Asking why questions often requires qualitative research methods and most evaluation studies in education are quantitative-focused. Quantitative methods are great at identifying patterns, but rarely good at identifying mechanisms (the reasons for those patterns). In a perfect world, evaluation and demonstration projects would include a theory-driven qualitative component to assess why. However, at this stage, I suppose we should be thankful that there is even financial support for any evaluation research.
Posted by: DJ Butler | January 22, 2009 9:33 PM
I agree with Cooper. Why don't we ask the people who teach in the KIPP schools why KIPP works? When I've done just that I've gotten answers that surprised me (e.g. the extended day and year is not as impactful as some other elements of their program).
Posted by: Socrates | January 23, 2009 8:07 AM
Why not listen to the advocates? They are advocates for a reason - and many of these people are educators, not Edison-style executives looking to exploit a market opportunity for monetary gain.
I would think that the educator-advocates, at least those whose primary motivation is improving public education - would have a great deal to say about "why," because they have stuck out their necks, in some cases risked their careers, to speak up for a reform that they believe in.
Posted by: KitchenSink | January 23, 2009 8:35 AM
Of course we should listen to the social scientists who do quantitative studies - and to the researchers who do qualitative studies. Of course, we should listen to KIPP teachers - and to education advocates who bring fresh eyes to education.
Of course we should listen to teachers.
Of course the answer is "all of the above." I have a problem with newcomers who "don't know what they don't know" and yet seek to impose their ideology on all of us, but that doesn't mean that they can't make useful contributions. They would be far more helpful, however, is they took the time to learn more about systemic problems. I suspect that many or most KIPP teachers are open-minded, but a KIPP teacher can also be like the blind man touching the trunk of an elephant, thinking they know it all.
Please don't take this wrong, but I was a professional scholar before I was an inner city teacher. The blogoshere has allowed me much better (but certainly not perfect by any means) awareness of how the limited knowledge of many, who haven't had enough practical experience, has distorted the analysis of many. In my experience, it is the people who buy into "expectations" and curriculum-driven reforms who are the most lacking in the background knowledge that is essential.
Posted by: john thompson | January 23, 2009 2:17 PM
A search of the Eduwonkette archive does not show that the April 08 report "Graduation Rate Watch" raised the bloggers' attention. It might have, and is worth hailing in the context of this day's lesson, (or another lesson on lessons from extremes...or not). When a policy-relevant field is already rich with ideas, notions, and small experiments, succinct but sufficiently detailed description of a large multi-year intervention answers "Why?"
In brief, author K. Carey uses the recently fully- operational statistical system on 6 year BA/BS completion ratios of US colleges to compare racial disparities, levels and changes, between schools and and within-schools over time. He describes the program (by argument and appeal to a large supporting literature) at one school, FSU, which eliminated the disparity. He then generalizes to other schools, some of which claim such programs, but which further examination has shown are often just lip service.
The statistical basis and reporting are excellent; and the writing on the statistics and on the "qualitative" elements is clear. Threats to validity (eg. transfers) are estimated and appropriately discounted. One need neither the training of an economometrician nor the old time statistical religion as still practiced by some educational psychologists* to ENJOY the report. Any reader here might find pleasure.
Would that we saw more similar studies at the ES level.
*This denomination can insist on rituals which make for very long services.
Posted by: incredulous | January 25, 2009 6:20 PM
Asking why is just as important as asking how...and qualitative approaches lend themselves to such questions. It's not just about KIPP's or other promising results and test scores. We have to ask, were there morally defensible means to the ends? What do we value, and what are our goals for our future democratic citizens?
Posted by: Alexandra Miletta | January 28, 2009 7:36 PM