Researchers at the University of Southern California are converting one district’s evolving carousel of school turnaround reforms into a set of conclusions that administrators nationwide may be able to use to guide progress in their own schools.
The preliminary results suggest that the turnarounds introduced between 2010 and 2013 in Los Angeles had little overall effect on student achievement. But they did lead to better English/language arts results during a period in which turnaround strategies combined dramatic change with strong support. University of Southern California doctoral student Ayesha Hashim presented initial findings of the study earlier this month at the annual meeting of the American Educational Research Association in Philadelphia. Katharine O. Strunk and Julie A. Marsh, who are associate professors at the university, led the study.
The study focused on the Public School Choice resolution that members of the Los Angeles Unified Board of Education approved in August of 2009. That reform was similar to the changes that hundreds of districts nationwide are making in response to federal initiatives such as School Improvement Grants. Initial outcomes on the improvement grant initiative have been mixed, with two-thirds of schools showing gains in the first year, while another third slid backwards. The benefit of this most recent analysis is that the design permitted researchers to take a more in-depth look at turnaround reforms.
Under the Los Angeles reform described in the study, teams of educators, nonprofit organizations, and charter school operators submitted plans to run schools that the 655,000-student district identified as low-performing. It was a competitive process, with district officials selecting the winners. The selected plans took three main forms: “Re-started” schools re-opened with the same students and, generally, different staff. “Reconstituted” schools kept their students and replaced at least half their staff. And “transformed” schools kept their students and their staff but adopted a new operating plan.
This was a big study with a big scope. Researchers examined the English/language arts and math scores of more 400,000 students. Some of these students attended 27 low-performing schools that faced turnarounds between 2010 and 2013. The rest attended 190 schools that were also low-performing but differed in one key way: They were a hair too high-performing to have to face a turnaround.The researchers analyzed test results both from before (2003-2009) and after (2010-2013) the reforms. They also looked at suspension rates and demographics.
Overall, the researchers found that school turnarounds had no effect on the student achievement of the students attending the struggling schools that underwent the reforms. But when they peeled apart the layers of approaches implemented over time, they found varying results.
Cohort 1 schools mirrored the overall results: Their academic achievement was about the same as the achievement at similar schools that had not been turned around.
By contrast, the second wave of schools (cohort 2) got significantly better results than the comparison group in English/language arts. The turnaround advantage was equivalent to moving from the 50th to the 56th percentile over the course of two years.The wave 2 turnaround and comparison schools did about the same in math.
Although the turnaround advantage in wave 2 might seem small, Associate Professor Strunk noted that English/language arts test scores are particularly difficult to change. More so than math results, they are influenced by things that parents do at home. In addition, just five schools were turned around during wave 2. Strunk noted that the relatively small numbers of students involved made it more difficult to find statistically significant results. That’s because tests of statistical significance are more likely to be positive if sample sizes are large.
“The fact that we find English/Language Arts findings is really hopeful,” Strunk said. “These are schools who are high in their proportions of English-learners.”
As for cohort 3, they actually did worse than the comparison schools. The turnaround disadvantage was equivalent to moving from the 50th to the 46th percentile in English/language arts over one year. It was equivalent to moving from the 50th to the 44th percentile in math.
The turnarounds did not appear to effect suspension rates.
The researchers did not just analyze test scores and suspension rates. They also interviewed teachers and others involved with turnarounds.They conducted case studies. They observed meetings. They analyzed documents and surveyed principals and turnaround applicants.
Based on this information, they offered theories about why the turnaround had been most successful for the second cohort and least successful for the third cohort.
They suggested that wave 2 schools benefited from lessons learned during the first round of turnarounds. Educators at these schools got two weeks of paid professional development time during the summer before the turnaround began. In addition, in 2010, the district and its partners received about $6 million in federal Investing in Innovation grants and matching private funds.(An Investing in Innovation Grant also helped fund the study.) School officials used some of this money to help the wave 2 turnaround teams develop better plans. Another difference was that all of the five turnaround schools in the second cohort faced the two most dramatic turnaround options of re-constitution and re-start. This meant that they replaced at least half their staffs. By contrast, none of the wave 1 and 3 turnarounds used these two approaches.
As applicants wrote their plans for the third wave of turnarounds, the district changed the rules: External organizations like charter operators could still apply to turn around schools. But they had to hire employees covered by the district’s collective bargaining agreement with the United Teachers of Los Angeles. Researchers theorized that changing the plan in the middle of the school year led to widespread confusion that made turnarounds harder to implement. They also noticed that third-wave turnaround applicants were more likely to feel like they had been coerced into writing plans
Even the more successful wave 2 schools faced challenges. For example, wave 2 principals had trouble finding enough good employees in time for the first school year of the turnaround. However, to Associate Professor Strunk, the real takeaway of the study is that turnarounds can work if the conditions are right. According to her study results, these conditions include dramatic staffing changes, professional development, and technical assistance.