Data Fall Short When Evaluating Impacts of Principal-Prep Programs, Report Finds
In a two-year effort to examine how the graduates of five principal-preparation programs affected student outcomes once they were on the job, researchers came up against a major challenge: Good data were hard to come by.
But despite the dearth of good data, the researchers with the George W. Bush Institute and the American Institutes for Research stressed that understanding whether principal preparation programs are training principals for the work conditions and challenges of the job is an "urgent" policy concern. That's particularly true, the study's authors said, because of how much principals' jobs have changed in recent years, making it more important than ever to track principals during their careers and analyze the outcomes of their leadership on their schools and the students who attend them.
In their quest to do that, the data were a persistent problem. Sample size was an issue. Principal preparation programs graduate small numbers of students and not all go into the principalship right away. The researchers sought three years of student achievement data for their report, but because turnover in the profession is high—nearly half of new principals do not stay beyond three years, according to a study from the School Leaders Network—some schools and principals had to be excluded. Another limitation researchers encountered: districts that generally don't keep track of their principals' experiences before they arrive at the district or where those principals were trained.
Still, despite those challenges, the researchers found some illuminating things to note, including:
- While principal graduates provided generally positive reviews of the course work at their preparation programs and the hands-on experiences they received, they had a dimmer view of the support they received from the districts and their programs once they were on the job.
- The prep programs in the study produced graduates who were both high-performing and low-performing.
The findings, they noted, suggest that "focusing on reducing variation in the performance of graduates through training, selection, or other means and systematizing or better tailoring supports may be the keys to success in preparing effective school leaders."
The report, "Developing Leaders: The Importance—And The Challenges—Of Evaluating Principal Preparation Programs," comes as principal-prep programs are under more pressure to improve, superintendents say they are dissatisfied with their principals' readiness, and new, more rigorous education standards are being rolled out.
Title II ESSA guidelines, published by the U.S. Department of Education in September, provide new opportunities for states and districts to improve (and rethink) principal preparation and on-the-job support for school leaders.
Earlier this year, a report from the Wallace Foundation on improving university-based principal-preparation programs included this important nugget: 80 percent of school district superintendents thought that principal-prep programs need to be improved. Wallace recently announced a $47 million grant to districts and universities in seven states to revamp principal-prep programs and work with those states to take a more muscular role in improving program quality by, among other things, reviewing and revamping their licensing and certification requirements. (The George W. Bush Institute/AIR study looked at both university-based and non-profit prep programs.)
This latest prep-program evaluation is part of a series of school leadership research the George W. Bush Institute released last week in conjunction with AIR.
Another of its reports explored the importance of talent management, and provided examples of how school systems in Denver; Gwinnett County, Ga.; Houston; and Charlotte-Mecklenburg County, N.C., were using what it called a "framework" for talent management, a system that starts with preparation programs and continues with professional learning and incentives for principals on the job, in an effort to strengthen their principal pipelines.
Representatives from the Hillsborough County, Fla., school system and Charlotte-Mecklenburg district will be among the participants in a panel discussion at the American Institutes for Research in Washington on Nov. 3 on the "framework" and how their districts are using it.
Preparation programs—including the requirements and standards they must meet to stay open—vary from state to state, and few programs are evaluated based on their graduates' impact on improving student achievement, according to the new paper.
On the principal prep-study, the researchers' report suggests considering other factors such as teacher retention and school climate in addition to student achievement when seeking to measure a principal's impact. But the report also noted that even those measures come with their own set of problems. Teacher-retention, for instance, is an incomplete measure because one way that a principal can improve his or her school is to get rid of ineffective teachers.
It is also difficult to "disentangle" how much of a principal's impact is due to his or her preparation program or skills he or she picked up along the way, from their principal supervisors or through district support and professional development, the report noted. And, the researchers noted, it takes time for principals to have an impact on their schools.
Again, the researchers returned to shortcomings in the data.
The report makes recommendations, stressing that prep programs need to know how their graduates were affecting student achievement in order to make continuous improvements and that districts need to know which programs were producing the best principals.
Among the recommendations: States and districts should collect more systematic data on student outcomes in addition to standardized tests; embrace programs that use best practices recommended by experts; evaluate and collect more data on the programs that are producing the best graduates; conduct additional research on developing and retaining the most effective principals; and continue to draw lessons from leadership research in other fields.
You can dig into all of the George W. Bush Institute reports here: