« Wish #4: Better Alignment of Accountability Systems to School Outcomes | Main | Wish #2: The End of Proficiency Only Accountability Systems »

Wish #3: Asking More "Why?" Questions


Earlier this month, a team of researchers at MIT and Harvard released a report contrasting the impact of charter schools, “pilot” schools, and traditional public schools on student achievement. The finding of charter school effects on achievement, using a random assignment research design, fueled the rhetoric of charter school advocates, some of whom saw the findings as a license for unlimited expansion of charter schools.

The researchers themselves were more cautious. They acknowledged that the study was not designed to discern why the effects were found. In fact, if the study had found that students in charter schools had shown less growth in achievement than students in traditional public schools, they wouldn’t have known why either.

Good public policy depends on compelling answers to “why” questions about both the observed effects and non-effects of policies and programs. And these “why” questions pertain both to the inner workings of policies and programs as well as the context in which the policies and programs are situated. Borrowing policies that have been found to be effective in one setting and expecting the same results in another setting makes sense only if we know why the policies were effective in that first setting. A research study showing that a policy or program “worked” in a particular setting doesn’t tell us that.

Our wish, then, is for asking “why?” more loudly, and earlier in the lifecycle of a policy or program. Why might achievement be higher in charter schools? Why do children learn more in smaller classes? Why are some teachers more successful in teaching low-achieving students than high-achieving students? Why don’t school expenditures have a stronger association with student outcomes? In skoolboy’s view, the real leverage in education policy comes from good answers to the “why?” questions. To paraphrase Jim March, research that addresses “why?” questions is more useful than research that addresses “what works?” questions because it has so many more applications.

One challenge posed by our wish is that the researchers who are skilled at addressing “what works?” questions are not necessarily the ones who are good at addressing “why?” questions. Even in large federal evaluations, there typically is a division of labor in which the study of implementation and context is segregated from the study of program impacts, and different research organizations or researchers are responsible for differing parts of the overall enterprise. Asking “why?” more often will require some hard thinking about research training and the infrastructure for education research in the U.S.


It seems to me, as a parent, and not a formal or professional educator, that the best way to figure out why a particular learning environment works is to go experience it. Be there for a day. Talk to the kids and the adults. It would seem to me that in most cases it will quickly become obvious why something works.

What I like about charter schools, or at least the promise of charter schools, is to bring some variety to parents' public school choices... the more the better. Problem is, the criteria to evaluate charters, as well as conventional schools are so 19th Century... the transmission of approved content, rather than 21st Century knowledge acquisition, creativity and collaboration skills.

Cooper Zale

skoolboy: Interesting questions. I'd love to hear the answers! If you get any good data on "why" definitely let us know.

Jay Mathews new book "Work Hard. Be Nice: How Two Inspired Teachers Created the Most Promising Schools in America " is scheduled to be released today. KIPP's primary hurdle to large scale adoption is they have no magic bullets,they spend more focused time teaching. If you really want KIPP's excellent results, no more 180/6hr per day school years. The KIPP school year looks as if they read "The learning Gap" by Stigler and borrowed the day/week/year schedule (240 days per year)employed in Japan. I'm waiting for a graduate student in statistics to tell us how many KIPPs there could be and how many more Ghetto kid's would be succeeding if we wanted to actually close more of the achievement gap.

Asking why questions often requires qualitative research methods and most evaluation studies in education are quantitative-focused. Quantitative methods are great at identifying patterns, but rarely good at identifying mechanisms (the reasons for those patterns). In a perfect world, evaluation and demonstration projects would include a theory-driven qualitative component to assess why. However, at this stage, I suppose we should be thankful that there is even financial support for any evaluation research.

I agree with Cooper. Why don't we ask the people who teach in the KIPP schools why KIPP works? When I've done just that I've gotten answers that surprised me (e.g. the extended day and year is not as impactful as some other elements of their program).

Why not listen to the advocates? They are advocates for a reason - and many of these people are educators, not Edison-style executives looking to exploit a market opportunity for monetary gain.

I would think that the educator-advocates, at least those whose primary motivation is improving public education - would have a great deal to say about "why," because they have stuck out their necks, in some cases risked their careers, to speak up for a reform that they believe in.

Of course we should listen to the social scientists who do quantitative studies - and to the researchers who do qualitative studies. Of course, we should listen to KIPP teachers - and to education advocates who bring fresh eyes to education.

Of course we should listen to teachers.

Of course the answer is "all of the above." I have a problem with newcomers who "don't know what they don't know" and yet seek to impose their ideology on all of us, but that doesn't mean that they can't make useful contributions. They would be far more helpful, however, is they took the time to learn more about systemic problems. I suspect that many or most KIPP teachers are open-minded, but a KIPP teacher can also be like the blind man touching the trunk of an elephant, thinking they know it all.

Please don't take this wrong, but I was a professional scholar before I was an inner city teacher. The blogoshere has allowed me much better (but certainly not perfect by any means) awareness of how the limited knowledge of many, who haven't had enough practical experience, has distorted the analysis of many. In my experience, it is the people who buy into "expectations" and curriculum-driven reforms who are the most lacking in the background knowledge that is essential.

A search of the Eduwonkette archive does not show that the April 08 report "Graduation Rate Watch" raised the bloggers' attention. It might have, and is worth hailing in the context of this day's lesson, (or another lesson on lessons from extremes...or not). When a policy-relevant field is already rich with ideas, notions, and small experiments, succinct but sufficiently detailed description of a large multi-year intervention answers "Why?"

In brief, author K. Carey uses the recently fully- operational statistical system on 6 year BA/BS completion ratios of US colleges to compare racial disparities, levels and changes, between schools and and within-schools over time. He describes the program (by argument and appeal to a large supporting literature) at one school, FSU, which eliminated the disparity. He then generalizes to other schools, some of which claim such programs, but which further examination has shown are often just lip service.

The statistical basis and reporting are excellent; and the writing on the statistics and on the "qualitative" elements is clear. Threats to validity (eg. transfers) are estimated and appropriately discounted. One need neither the training of an economometrician nor the old time statistical religion as still practiced by some educational psychologists* to ENJOY the report. Any reader here might find pleasure.

Would that we saw more similar studies at the ES level.

*This denomination can insist on rituals which make for very long services.

Asking why is just as important as asking how...and qualitative approaches lend themselves to such questions. It's not just about KIPP's or other promising results and test scores. We have to ask, were there morally defensible means to the ends? What do we value, and what are our goals for our future democratic citizens?

Comments are now closed for this post.


Recent Comments

  • Alexandra Miletta: Asking why is just as important as asking how...and qualitative read more
  • incredulous: A search of the Eduwonkette archive does not show that read more
  • john thompson: Of course we should listen to the social scientists who read more
  • KitchenSink: Why not listen to the advocates? They are advocates for read more
  • Socrates: I agree with Cooper. Why don't we ask the people read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School