« Schools Restructuring under NCLB: Blow ‘em up Good? | Main | Let the Spin Begin »

Cool People You Should Know: Doug Downey

To many observers of public education, there is no doubt about which schools are failing - it's the schools with low rates of students passing state tests, stupid!

Of course, this assumes that students' achievement is a direct measure of school quality. "Yet we know that this assumption is wrong....It follows that a valid system of school evaluation must separate school effects from nonschool effects on children's achievement and learning" writes Doug Downey, a cool Ohio State sociologist of education you should know, in his recent paper (in collaboration with Paul von Hippel and Melanie Hughes), "Are 'Failing' Schools Really Failing?"

Analyzing data from the Early Childhood Longitudinal Study - Kindergarten Cohort, a national sample of 21,000 kindergarteners that were then followed through 5th grade, Downey and colleagues thus set out to isolate the effects of schools on student learning. The ECLS data are uniquely suited for this task because the study evaluated students in the fall and spring of kindergarten, and again in the fall and spring of first grade. It turns out that summers - a time when students are only affected by non-school influences - are the key to teasing apart school and nonschool factors.

Downey and colleagues look at schools' effectiveness in four different ways. First, they examine NCLB's method - overall test score levels. They then turn to 12-month learning rates; think growth models, which measure test score growth, for example, between a test given in April 2007 and a test given in April 2008. They contrast those rates with 9-month learning rates; imagine a test given in September, and then again in May. Finally, they introduce a measure called impact, which is the difference between the school year and summer learning rate.

"Impact" is attractive because it doesn't require us to measure and statistically control for all of the different aspects of children's nonschool environments that may affect school success, as do cardiac surgery report cards. It captures what we need to know about students' out-of-school environments without bogging us down in the methodological and political problems associated with introducing these controls. And it helps us adjust for "soft" factors like innate student motivation, for which it is difficult to measure and control. Moreover, it holds schools harmless for what happens to their students over the summer, which currently serves as a confounding factor in growth models.

What percent performing in the bottom 20% of overall achievement are actually in the bottom 20% for measures of impact and learning? Less than half! High-achieving schools are concentrated in more affluent communities, but "high impact" schools exist across the socioeconomic spectrum. And the opposite is true. There are plenty of school with good test scores that are skating by because simply because they had advantaged kids to begin with.

What does this all mean for NCLB? Downey and colleagues put it like this:
Our results raise serious concerns about the current methods that are used to hold schools accountable for their students' achievement levels. Because achievement-based evaluation is biased against schools that serve the disadvantaged, evaluating schools on the basis of achievement may actually undermine the NCLB goal of reducing racial/ethnic and socioeconomic gaps in performance. If schools that serve the disadvantaged are evaluated on a biased scale, their teachers and administrators may respond like workers in other industries when they are evaluated unfairly - with frustration, reduced, effort, and attrition. Under a fair system, a school's chances of receiving a high mark should not depend on the kinds of students the school happens to serve.
Crystal clear, creative thinking is the distinguishing feature of Downey's work - see, for example, his paper on school effects on child obesity, or his paper asking if schools are "the great equalizer."

Wonks can rest a little easier tonight with the knowledge that Downey's now turned his attention to NCLB.

Ohio has just started reporting value-added indicators. Schools and districts get an averaged rating of expected, below expected or above expected growth for the year. I don't know how much I trust the averaged rank, but the same ranks are given by grade/content, which a more helpful way of looking.

But I believe that these support Downey's assertion that actual "growth" is much more equitably spread across income levels than is achievement. And I suppose this makes some folks feel better (and others feel worse). And it is certainly helpful to look at where in the 3-8 grade span kids are getting lost and where they are picking up speed.

But NCLB really poses a different challenge than seeing that every kid gets the same amount of growth. It suggests that there is a minimum amount of knowledge or ability that we can shoot for providing to every kid. This doesn't say that some cannot get more (as we all know that they do). But we really have to focus PUBLIC education on ensuring that we have a public that leaves school at a certain level (or above). This implies that certain schools will need more resources to achieve this minimum level (if their environment isn't providing as much as some others). We seem to stall at this point--and all the arm-waving from teachers who feel blamed isn't helping us to look at a redistribution of resources.

I think that the Japanese get the point about ensuring a level of achievement for all. They don't seem to be troubled by "teaching to the middle," in order to assure the highest common good. This is a value that they hold. I don't know that we do. They rely on the joku to provide either tutoring or enrichment, and seem to feel that this is appropriate.

Measuring the impact of a school doesn't move me, if the impact is still insufficient to meet a minimum grade-level of achievement for all students. What this says to me is that some students need more. Why are we so loath to see that they get it?

Hi Margo,

I wholeheartedly agree with the social goals you set out around ensuring a basic level of achievement, and ensuring that kids, irrespective of their background, get there.

But we need to separate the issue of social goals from the issue of measurement, especially if our primary objective is achieving the societal goals we both agree upon. If a school is making exemplary progress based on a growth or impact measure but is not making proficiency targets, we don't serve the students well by sanctioning the school and its teachers.

Why not measure impact or growth to establish whether the school is performing worse than it should be first *and* then provide more support and additional interventions to students/schools if the school is not "low-performing" according to these more sophisticated ways of isolating school performance?

OK--so we measure impact or growth first and what do we do with the results? If the goal is the get to point A, and we suspect that we've got kids starting from B, C, D and F, what good does it do to know how fast they are moving?

But, I think there is something else that is crucial about Downey's methodology, which is that it doubles the amount of testing--with the only value-add that we can separate out the school contribution from the summer contribution (or loss). Which, again, seems to be a red herring, unless we are going to be able to tackle the environment (or the summer).

I get really nervous when people start talking about the "sanctioning" of schools. The "sanctions" in NCLB all relate to making improvements, beginning with enlarging the options available to students and providing tutoring, and moving forward with planning and implementing changes. While the press likes to focus on the drama of reconstituting personnel, this is one possibility mentioned along with improving curriculum, and extending the school day and year. I hear very little about districts who have gone this direction. In fact my own district has gone in the opposite direction (shortened school day--across the board. No one even mentioned the possibility of protecting any of the schools that were already doing poorly).

The other day, skoolboy mentioned an Ohio study that looked at funding disparities within districts. Just for fun, if someone has time, it would be interesting to see whether those disparities relate at all to the growth disparities.


The study by Condron & Roscigno of Columbus, OH does in fact show that within-district disparities in expenditures are associated with school-level achievement when controlling for prior achievement, which is essentially a growth model. But I have to caution that the effects of spending differences on achievement growth generally are neither large nor consistent, for a number of reasons: the unit of analysis is most commonly the district, which means inequalities among schools within districts are not taken into account; spending is a coarse proxy for exposure to resources that might matter for classroom teaching and learning; and how money is spent may be as important, and perhaps more important, than how much money is spent.

To respond to Margo/mom's earlier point about why don't we see that kids get what they need to end at the same minimum level, even if they are starting at positions of varying advantage of disadvantage?

It's a good question. Part of the problem may be that truly leveling the playing field may need to take place before a child hits kindergarten, with high-quality preK and very early childhood programs that include parent education. For an example of what might be possible, I suggest reading Paul Tough's new book, Whatever It Takes.

I am interested in collecting research testing conventional wisdom such as the childhood obesity study you cite. To the public it just seems obvious that schools with their vending machines, their terrible lunches, the reduction of PE and recess encourage obesity. Policy decisions are often made on the basis of faulty conventional wisdom.

I would like to make a collection of other such unexamined assumptions related to education and the pertinent research, or note the lack of research.

As Margo pointed out, if we were to universally apply Doug Downey's approach, we'd have to double the amount of standardized testing - not a desirable goal.

Further, using the test scores as the measure runs the risk of strengthening the illusion that test scores can tell us a lot about learning, especially in an era of intense teaching to the test. This is certainly not Downey's fault - it is mostly what we have for usable large-scale data - but it ignores untested subjects and vital aspects of the tested subjects themselves.

Questions: Do schools serving more affluent students provide more of these unmeasured attributes, such as ability to solve problems, engage in synthesis or evaluation? Or are they distributed more evenly, as Downey finds with school-year gains on reading and math tests? Or even more common in low-income schools? Which attributes would be want to know more about? How would be find out about them?

Years ago U Wisconsin researchers such as Fred Newmann reported studies in Chicago finding that student acquisition of more complex knowledge and skills was distributed across Chicago schools - but rare in most schools. They concluded this by looking at classroom-based work, also finding that less didactic, more interactive, instruction was found in the classrooms with greater evidence of students acquiring the higher order skills. NCLB has certainly tended to reinforce the dominant highly didactic approaches. That may explain the declining rate of improvement on NAEP - but even NAEP is not an adequate measure of complex, higher-order thinking.

Downey's work is indeed useful, but we have to get beyond the test scores in research as well as educational practice.

maritza: I'll probably get around to reading Paul Tough's book eventually, since I am reading so much ABOUT it. But I have some cautions about over-reliance on early childhood programs to level the playing field. I believe that some of early longitudinal studies of Head Start showed that it was effective in delivering students to school at a point closer to their upper income peers. But they lost ground again by second grade. Sorry--I cannot cite specific studies, perhaps someone else recalls what I am remembering.


It's true that many early childhood interventions, including Head Start, have had what are known as "fadeout effects," in which short-term gains quickly fade out over a few years. The conventional wisdom, as I understand it, is that (a) the quality and intensity of the early childhood intervention matter a great deal, and some programs are much better than others on these dimensions, and (b) the children who are the targets of early childhood interventions may need ongoing supports after the intervention is concluded and the children move into elementary schools which vary in quality and curricular alignment. The existing evidence on the potential of early childhood interventions, when coupled with ongoing support in elementary and high school, is what leads Nobel-laureate economist James Heckman to be such a staunch supporter of high-quality early childhood interventions.

I was not sure what your comment about Japanese schools was trying to get at. Japanese schools definitely do not teach to the middle. They do ensure that every student receives the delivery of the same educational content. For a primer, on Japanese schools, testing and the role of the juku in Japanese education, I suggest my article in the Oct 1993 issue of The Kappan, entitled The Secret of Japanese Education.

Skoolboy--I don't disagree a bit. It takes both appropriate early childhood care/education and high quality and appropriate elementary and secondary education. To me the value of the research was that it pointed out that it is insufficient to look to head-start-like programs to level the playing field. We have some real equity and quality issues to address within elementary schools as well.

Comments are now closed for this post.


Recent Comments

  • Margo/Mom: Skoolboy--I don't disagree a bit. It takes both appropriate early read more
  • sgoya: Margo: I was not sure what your comment about Japanese read more
  • skoolboy: Margo, It's true that many early childhood interventions, including Head read more
  • Margo/Mom: maritza: I'll probably get around to reading Paul Tough's book read more
  • Monty Neill: As Margo pointed out, if we were to universally apply read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School