« Bubble, Pony, or Lone Star?: Portraits of the Secretary of Education | Main | What Does It Mean to Be a "Gap Closer?" A Look at Three Boston Charter High Schools »

Early Warning Systems for School Dropouts

The recent flurry of attention to high school completion rates has revived interest in early warning systems designed to identify students at risk of dropping out of high school. The idea behind these early warning systems is that, through the analysis of administrative data, schools and school districts can develop models of risk factors which predict a high probability of dropping out of high school. If the models successfully distinguish probable dropouts from probable graduates, students at high risk of dropping out can be identified, and support resources can be focused on these students identified as at risk of dropout.

A good early warning system will have high sensitivity and high specificity. High sensitivity means that the early warning indicators will identify a very high percentage of those youth who will eventually drop out (i.e., a high percentage of "true positives"). High specificity means that the indicators will not identify many youth who are not destined to drop out (i.e., a low percentage of "false positives".) Phil Gleason and Mark Dynarski of Mathematica Policy Research showed in the federally-funded School Dropout Demonstration Assistance Program evaluation that most dropout prevention programs had disappointingly low sensitivity and specificity: they failed to serve youth who would eventually would drop out, and they frequently served youth who would likely have graduated in the absence of the program.

Early warning indicators have been developed in Chicago, by Elaine Allensworth and John Easton, and in Philadelphia, by Robert Balfanz and Ruth Curran Neild, as well as other cities. The Chicago indicator is an indicator of being "on-track" for high school graduation; a student is "on track" if he or she earns at least five full-year course credits and no more than one F in one semester in a core course during the first full year of high school. The Philadelphia measure relies on sixth-grade measures of academic performance and behavior. A student with at least one of the following four characteristics had at least a 75% chance of dropping out of high school: (a) a final grade of F in math; a final grade of F in English; attendance below 80% for the year; and a final behavior mark of "unsatisfactory" in at least one class.

It’s not exactly rocket science to show that students who fail courses and have low attendance have an elevated risk of dropping out of high school, but the architects of these systems argue that the specific indicators that students manifest warrant different responses. Low attendance may stem from a different set of sources than poor behavior, for example, and a key feature of these indicator systems is that they frequently rely on administrative representations of students’ behavior in the school and classroom (especially the incidence of failing a core course), rather than more distal status measures that are less amenable to a programmatic response. Finding that low-SES youth are more likely to drop out, for example, would not give a school or district much to work with.

One issue to consider is the way in which early warning indicators are used in medicine. They’ve become controversial in instances in which the indicators don’t prescribe a reliably successful course of treatment. In the absence of an effective treatment plan, critics argue, indicators of the heightened risk of conditions such as prostate cancer or breast cancer may simply upset patients and not improve outcomes. In contrast, cholesterol tests are much more valuable as early warning indicators for heart disease because the use of statins to reduce cholesterol levels is recognized as an effective treatment that improves cardiovascular outcomes.

The question we might ask about dropout prevention is: If we knew that particular students had an elevated risk of dropping out of high school, what would we do differently? The problem here is that we do not have a dropout prevention wonder drug that has shown to be reliable in lowering dropout rates in multiple contexts. The history of dropout prevention research is littered with poorly-designed, small-scale research studies that have failed to identify a set of program elements that consistently work. Moreover, the best-designed of such studies have found modest program effects on the probability of dropping out.

None of this is to say that local efforts to reduce dropping out are ineffective. Many talented and motivated people lead and staff such programs, and they may in fact reduce the risk of dropping out for some groups of youth. The problem is that we don’t really know if they work or not. And in the absence of such knowledge, skoolboy is just not sure that early warning systems to identify potential dropouts are all that useful.

The problem is that we don’t really know if they work or not. And in the absence of such knowledge, skoolboy is just not sure that early warning systems to identify potential dropouts are all that useful.

That gets a little circular... It's easier to test what dropout prevention programs work if you actually have a good idea of which kids are most at risk of dropping out, so it would seem to me to be worth some effort to figure out what the real risk factors and red flags are.

Hi Rachel,

Any effective program needs to be grounded in a good theory of the problem it's supposed to address, and also needs to ensure that the program is serving the target population and not ineligibles. Moreover, a program needs to take account of the local social, political and cultural context, including distinctive local causes of the problem.

But I don't think we need an early warning system to do that. There already is a great deal of research on the etiology of high school dropout that can serve as a basis for program design and evaluation.

My colleague Sherman Dorn also points out that the potential value and meaning of an early warning indicator differs according to the overall prevalence of the problem. In a district in which 15% of students drop out, an early warning system might allow the district to target individual students at great risk of dropping out. In a district in which 50% of students drop out, however, an individual case-management approach seems unlikely to work, as the problem appears more systemic. If 50% of the high school students in a district drop out, do you really need to know which 50% before developing a systemic approach to addressing the problem?

Good points... My district is at the "15%" end of the spectrum, so the idea of being able to help individual, particularly at-risk students seems relevant.

I agree that if 50% of students are dropping out, you've got problem that needs a systemic, not individualized, approach.

Regarding the appropriate intervention. I would not be surprised if the Hawthorne effect would have surprisingly effective results.

When observing improved productivity in a group of electrical workers, it turned out that every independent variable that increase the focus on the workers resulted in increased productivity.

It's similar, I think, to the placebo effect in medicine.

This is consistent with my observations that drop outs suffer from inattention. It would be interesting to know if there are any studies that have isolated the effect of attention, regardless of the content of the attention given.

If it turns out this is real, the problem might be mitigated by improved communication techniques, well within the control of managers. And not be the necessity of getting teachers to change their behavior, which is very slow and time consuming.


The term "Hawthorne effect" gets used to describe a range of social-psychological phenomena. I wouldn't equate it with a placebo effect. Many theories of school dropout emphasize students' engagement, withdrawal, participation and feelings of belonging or membership in the school as causal factors. The social ties that students have both to other students and to adults in the school are related to these factors, but it isn't always easy to tell which is a cause and which is an effect. The bigger problem is that disengagement and withdrawal often follow from histories of academic failure, and it's difficult to reverse such histories.


Don't get caught up in methodological details, but focus on the two or three main ideas.

Above all, prostate cancer isn't communicable. Disruption is contagious and there is nothing like teens with no hope of success in terms of spreading disorder, which robs their classmates of a chance of an education.

Secondly, the methodological problems of identifying and helping students, as much as we can, are nothing in comparison to the humonguous problems faced by VAM and the data-driven accountability hawks in order to make their systems work.

And what challenge do we want to tackle? Do we want to invest the best of educational talent to create a data-driven culture of accountability? Or should we invest those talents in identifying and directly helping our most vulnerable children? What if we used our digital skills to create systems where 1st graders' parents are contacted after their third absence and home visits begin after the 5th? What if we did something comparable during the equally dangerous transitions to middle school and high school? What if we used our talents to identify problems and experiment with solutions to directly help kids, as opposed to shaming adults into doing their jobs according to the beliefs of BloomKlein, Rhee, and the accountability hawks?


We may not have the highest level of specificity with regard to either identification or response. Nonetheless, I think that we know enough. I just checked the What Works Clearinghouse site to peruse what they have already looked at. I know that many challenge them for setting their standards too high to be of much use--the primary result being that the most reliable studies selected are frequently quite small.

However--I can pick out two from their list that would be good bets. One is accelerated Middle Schools, which is showing fairly strong results in progressing and staying in school (not completing). The other is Check and Connect, which has smaller numbers, but pretty good results again in progressing and staying. The completing school data looks to be more scattered across programs--and not necessarily the same programs that are good at staying and progressing.

You make a good--and frequently overlooked--point about the scope of the problem requiring different responses--the difference between responding to an epidemic and responding to a patient with a disease. The principles of Positive Behavior Support--which generally use the number of office referrals for discipline as a key indicator--apply a levelled approach, making the point that individual interventions are almost doomed to fail if the underlying system is not supportive. The major barrier that I see in implementing reforms in this way is that we tend to see the problems as residing in the students, and not within the system. We assume that we have to fix the students so they can fit into the system that we are comfortable with.

The grouping of key indicators is not data that is currently unavailable--and with unique student identifiers--which most states have or will soon have--looking at such data across indicators (grade failure, truancy and discipline, for instance) is fairly simple. This is helpful in identifying not only students, but buildings, in need of intervention. What we have to move away from is the current "shotgun" or "spray and pray" approaches. My district has begun a mentoring program with a goal of involving all students above a certain grade level. The cost per student for this volunteer program is high (although there are sponsors outside the system). It is being touted as "research based." While it is true that the program--a quasi branded approach, developed by a national non-profit--has done some good research, the program that they researched was different (more intensive, different selection process), the outcomes that they were looking at were different (reduction in risky behaviors, rather than increased graduation), and even though the results were significant--it was a reduction from 10% to 5% or their tested population engaging in risky behaviors. In short--I wouldn't bet much on this effort having much impact on the desired outcome of increased graduation rates--at least based on the existing research. But, it gets good press, and folks with money are willing to rally around and put their corporate logo on the effort. The non-profit benefits from the growth of their services for a few years. Kids probably aren't hurt--except in the sense that in the end there is the danger that this will contribute to the sense that "nothing works" for these kids.

We can even look earlier than the early warning systems proposed here, to the things that show some promise of keeping kids "on track" in math or reading through elementary school. While math is still pretty wide open (I believe), there is research aplenty with regard to what works in reading intervention--and I am thinking about Reading Recovery, which has been around quite some time and acquired a track record here and in other countries.

In short--I think that we have enough data to do a better job than we do (and implementing programs and evaluating them is certainly a good way to increase the available knowledge of what works).


I can't believe you did that to me! I've got a guest blog drafted on the Intervention Middle Schools. Another set of "treatments" can be found on Lynn Canady's web site. He says that every October we've identified students who are failing, and we can't think of anything better than letting them fail for the rest of the year? His solution is to have them "fail faster." Seriously, we wants alternative schedules so they can start over in October, and rejoin their peers by January if they take advantage of the second chances.

I also agree that we try to "fix" kids because we can't fix the system. But in reality, don't we always have to do some of that? If wait wait until utopia ...

So, we have to teach kids to function in an imperfect world.

"I also agree that we try to "fix" kids because we can't fix the system. But in reality, don't we always have to do some of that? If wait wait until utopia ..."

John--I think the issue is how many times are we willing to crash into the same wall before we try turning left (or right).

I look forward to reading about Accelerated Middle Schools. I don't really know anything about them except for how they appeared on the WWC list. Personally, I am in favor of taking all middle schoolers out into the woods to build their own shelters, as a beginning. Make use of all that social energy to teach them how to organize themselves as working groups. I also think that MS is a good place to try doing things differently, not only because they tend to be the weakest link in our current system, but also because so few adults are drawn to them unless they are really interested in responding to kids (who are really interesting).


I agree completely. We're on an agreeing streak.

But suppose it turns out that the fastest way to "fix" the kids is to fix the system?

Let's say that the main thing missing from the system is time to focus. Everybody is very busy, busy, busy.

Suppose it turns out that the specific content of the focus is much less important than the focusing itself.

I think John raised a good point when he said What if we used our digital skills to create systems where 1st graders' parents are contacted after their third absence and home visits begin after the 5th?

The hard part is to do it fast enough to make a difference. The effect of timely response is the value creator. It communicates that someone is watching and/or caring. The particular content of the message is much less important.

I got an 80% homework compliance in a 10th grade inner school high school class with a chart and form. The chart was who did their homework over the last two weeks. The form was the chart, a note that said three homework missing you get an F, and a place for mom to sign. The mere presence of the chart and form was enough to get most everyone to play the game. Then special attention could be given to the the 20%.

That's sort of what I meant by the power of the Hawthorne effect.

Just wanted to clarify the sense in which I see the similarity between placebo and Hawthorne.

One can look at the health of a physical body within a thought model that considers the communication ecology of a bilogical entity. The head brain talks to the gut brain talks to the head brain. There is lots of evidence that a change in the pathways of communication that are present in the head brain can increase healing.

A social system can also be looked at through the lens of its communication ecology. Changes in the speed, accuracy and control of communication in the information space can lead to tipping point changes in physical space, at low costs in internet time and with little "retraining" of either staff or students. Consider the changes in business and most recently in politics brought about by Google,YouTube, et al.

Given that most schools I've been in have seriously dysfunctional communication ecologies, it makes sense to me that similar tipping point changes can occur with small modifications of the communication ecology.

The secret sauce is that the communication channels themselves throw off the information that is needed for the feedback that can lead to constant improvement.

Imagine a system that sends an SMS to Mom's cell phone on the day that Junior doesn't show up to class. Not a difficult technical task. It gives Mom the information to ask the "where were you?" question ON THE DAY it has to be asked.

My bet is that as students understand that the such a system is part of the normal way of doing business, the amount of cutting would decrease significantly.

Or consider writing on a blog that is timed stamped and dated as a by product of clicking the post button. It would eliminate the wasted time in measuring. It would also keep the teacher's comments on record. My bet is that once everyone had the expectation of that way of doing business, students would write, on time and teachers would comment on time. It would also leave a transparent accessible record of who did what when, both for admins and parents.

My argument is that the content of the comment is much less important than the timeliness of the response.

Consider the effects of answering an email now, imperfectly...or in two weeks with just the right words.

Missing from the presentation, which may explain some softness of program benefits, is acknowledging that HS diplomas are being granted in contravention of putative rules.In times past HS students dropped out and found more congenial educational environments in GED programs. Now school systems offer credit recovery programs, quick classes with short hours, short semesters and no homework --basically a variation on the EdD paper mills by which schools superintendents get diplomas, which combined with higher graduation rates, earn them ever higher salaries. With programs like that in place, therapeutic programs directed at getting students back on the OLD TRACK will show shrunken benefits in who arrives at the station. The broom wagon of credit recovery means everybody, (superintendents included), arrives at the station and gets a ribbon.

Hey Incredulous,

If I understand your argument, you're suggesting that a high school diploma does not signify the same level of academic accomplishment today that it did in the past, due to academic shortcuts such as credit recovery. If that's so, then we might expect to see the earnings of high school dropouts and terminal high school graduates to converge over time, since the two groups will be increasingly similar on the knowledge and skills that the market rewards.

The institutional rules that assign higher rewards to higher credentials haven't caught up with the decoupling of credentials and learning that has characterized the last few decades.

Hey Skoolboy,

You say, "The institutional rules that assign higher rewards to higher credentials haven't caught up with the decoupling of credentials and learning that has characterized the last few decades."

The contention that more education causes higher income is not proven. I don't think the data support a casual relationship, merely an associated observation.

With a small number of exceptions, our schools were designed to identify and support the "smart" ones, bring rural workers into mass market industries and baby sit everyone else. Like any organization that naturally evolves, they became very good at that task.

The problem we're now facing is that for the first time, schools are being tasked with teaching every one to learn.

The problem is usually framed as about personnel retraining - teachers, managers, or the insufficient resources of "poor" people. One reason that it seems so intractable is that we don't focus on the crux of the matter.

A school can be seen as a set of job descriptions and the information exchanges between those job descriptions. It's what I would call the communication ecology of a school. We could get from here to where we have to be by investing in changing the ecology and investing less in the actors who are doing good enough in an increasingly extinct environment.

The certs and standardized tests are little like fighting a 21st century war with the weapons and strategy of the 20th century.


This is minor, but its a reminder about the problems of predicting, especially about the future.

I doubt there is much correlation between the booklearning of people who don't make it to college and their subsequent pay. The ability to get along is much more important in the working world than your algebra scores.

I only mention this minor issue because people who make many market arguments like Gates don't seem to understand they are addressing the wrong issue.

Create a great learning culture and we'll get more of both benefits. We'll get people better prepared for the 21st century, and we'll get people with better "people skills" and stick-to-it-ness. Education is at least as much an affair of the heart as the head. You know that. Too many "reformers" don't.

Also, address the whole human being and health care costs will decline, reducing another impediment to our competitiveness.


I don't know where the meaning of a HS diploma is becoming less clear. I am just mentioning one of the new ways by which it is changing for many students. "Credit recovery." There is no doubt that at different times and places educators and systems have incentives of varying power not to educate, but to certify.

In Washington, DC the number of credits needed to get a HS diploma has increased to the Nation's highest (or near it) even as rescue schemes by which to earn credits have increased. Some who tried to stop both claimed that increasing requirements would immediately cause the content of some courses, now required for a HS diploma, to be diluted.

As to your inference of shrinking future wage differentials? Perhaps. Look into the content of undergraduate education, including that at major State universities; you might surmise that its function is to sort on attendance and provide eligibility for further schooling.
But, that takes us too far into the topic of credentialing.
Returning to HS retention: Do the alternative programs, including credit recovery, twilight academies,etc. signify the same readiness and skills, and provide the same security as traditional HS programs? Enrollment in these alternatives shows more care and initiative than dropping out. (Though one could say that fear of stigma failure is the major cause of most HS sitting through their days in class). But, we should be honest that major driver right now is the desire to boost graduation rates. When the Armed Forces will accept increasing fractions of those who have not completed HS, we know HS attrition was just something kicked down the road.

I welcome the introduction of measures of effectiveness and efficiency from the epidemiological world. Beats hell out of standardized regression coefficients. But, in education /schooling we too often cannot confirm diagnosis, treatment, or case disposition.

You hit it on the head with "Create a great learning culture and we'll get more of both benefits. We'll get people better prepared for the 21st century, and we'll get people with better "people skills" and stick-to-it-ness. Education is at least as much an affair of the heart as the head. You know that. Too many "reformers" don't. Also, address the whole human being and health care costs will decline, reducing another impediment to our competitiveness."

To all on this thread,
I am not a ed professional. But there is an idea that I would like to hear the reaction from people who are. It goes to the question of measuring.

If one important goal is to make our students smarter, why not consider using the knowledge that has been earned in IQ testing to test conceptual "smart?"

Not exactly as the tests are now, but with the modifications needed to discriminate improvement over time. There is enough evidence that IQ does not measure innate ability anyway. As a non pro it seems so straightforward. Plus the inevitable teaching to the test, given the incentives in place, might turn out to be a feature, instead of bug.

Anybody have any thoughts on what I'm not understanding or have overlooked?

Comments are now closed for this post.


Recent Comments

  • Michael Josefowicz: John- You hit it on the head with "Create a read more
  • Incredulous: Skoolboy: I don't know where the meaning of a HS read more
  • john thompson: schoolboy, This is minor, but its a reminder about the read more
  • Michael Josefowicz: Hey Skoolboy, You say, "The institutional rules that assign higher read more
  • skoolboy: Hey Incredulous, If I understand your argument, you're suggesting that read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School