This post is by Susan Fairchild, chief knowledge officer for New Visions for Public Schools.
Accountability systems like those mandated by the federal No Child Left Behind act, as well as increasingly sophisticated technologies, have resulted in exponential amounts of data. But can data alone guide our education system into a new phase of student success?
Shane Hall, data strategist for the Dallas Independent School District, introduced the “data as a flashlight” metaphor at a convening of the College Readiness Indicator System (CRIS) initiative this past December. Hall went on to note that using data to enlighten, however, might actually be less about data and more about mindset. While we have lots of data, helping educators focus data to form insights is no easy task.
One reason why it is so hard to actualize the “data as flashlight” metaphor is because data are often decoupled from the school structures and mental models that generate them. John Shibley suggests organizations fail to convert data into meaningful change when the people within them “leap to action"--going from the observation of events to the implementation of solutions--without fully understanding the mental models and structures producing those events. Shibley, as well as a recent Spencer Foundation funded report, recommend that we take a much more serious look at systems thinking as a means of bridging this gap.
Systems thinking is a framework that shapes the way we understand and interpret data. Barry Richmond’s definition is particularly helpful: “Systems thinking is the art and science of making reliable inferences about behavior by developing an increasingly deep understanding of underlying structure.” Systems thinking is a particular way of seeing the world. Systems thinkers are trained to look at the structures (e.g. how departments are organized, how students are programmed for courses), the interdependence of those structures, the ensuing dynamics of those structures, and the mental models that hold them together. As a result, systems thinkers use data differently.
Data tools that are firmly rooted in systems thinking are rare. New Visions’ Stock and Flow tool is one data tool we know of in the field of education that is entirely informed by systems thinking and that is designed to encourage school leaders to consider how structures in schools produce patterns of student outcomes. Our Stock and Flow tool is the product of two core principles of systems thinking: 1) structure determines behavior and 2) systems are dynamic.
Structure Determines Behavior. At New Visions, we work primarily with high schools in New York City. The goal of high schools is to provide students with sufficient and valuable opportunities to earn credentials within four years that will prepare them for postsecondary work and/or meaningful careers. To achieve this, schools must be designed or structured to direct the flow and speed of students’ academic trajectories from the moment they enter ninth grade. School staff need to be organized to manage three key factors: the different levels of skills and knowledge students bring when they enter high school; the different rates at which they accumulate credits; and the narrowing opportunities for earning credits and passing required exams for students who fall behind.
Consider the example of Student A. She enters high school performing at grade level, hopeful that she will complete eight semesters of coursework, accumulate credits, pass all required exams, and graduate four years later. If, however, she fails to meet certain requirements and falls off track at the end of ninth grade, she will carry a certain number of lost credits or failed exams. To graduate on time, she will need to make up those shortfalls over the next three years while still maintaining her forward momentum.
This is a challenging situation for the student, but it also places extra demands on her school. When Student A falls behind, her school will need to invest additional resources to move her back on track. The further behind she falls, the more resources will be needed. At the same time, the window to intervene narrows as she approaches senior year. The challenges are even greater, for both student and school, for a ninth grader who arrives in high school performing below grade level.
Patterns of students progress including when they stumble, how quickly they recover, and the amount of time they are on track provides clues into school responsiveness and school structures. When visualized correctly and framed the right way, these data patterns shed light on whether or not schools are organize to optimize student trajectories.
Systems are Dynamic. At the same time, systems are dynamic. John Sterman points out that, as a consequence of this dynamism, “Doing and undoing have fundamentally different time constraints.” For example, the amount of time it takes to move lower performing students into higher performance categories can take a long time. And, substantially more resources are required to support lower performing students.
Educators also have to worry about sustaining high performing students. Just because a student is high performing at one moment in time does not mean she will continue to maintain that level of performance. The effort it takes to create high performing students and to keep them that way generally takes long periods of time and resources.
The speed with which a student can stumble, on the other hand, can happen in the blink of an eye.
Our video below, created in collaboration with Andrew Garcia Phillips, aims to put the complexity and dynamism clearly into perspective and it highlights the important role of systems thinking.
Our Stock and Flow tool emerges from our understanding of structure and dynamism. The tool visualizes how progress to graduation is shaped over the course of eight semesters and helps schools locate the structures influencing that movement.
Student progress can be measured as both a “stock” (a point-in-time accumulation) and as a “flow” (a dynamic movement across time that encompasses directionality). Visualizing both dimensions of student achievement produces a unique map of student performance. This interactive graphic allows you to explore the features of our “Stock and Flow Tool.”
What is different about this tool is that it shows how students’ performance categories change not just from one semester to the next but between semesters. In this way, we are visualizing how students drain out of and fill up different performance categories at different moments in time. The width of each color band represents the percentage of students comprising each performance category at the beginning of the semester. We use six performance categories. “On track to college readiness” (blue) is the highest level of performance. “Off track” (red) represents the lowest level of performance.
The movement (or lack of movement) between semesters is just as much about school responsiveness as it is about student progress. For example, when we see increasing numbers of students flowing into the off track, red category, over eight semesters, this suggests that the school may not have sound systems (e.g. attendance, programming and scheduling, academic intervention systems) in place.
When we see larger percentages of almost on track and off track students move into higher performance categories between the 7th and 8th semesters, this suggests a school may be putting out fires and intervening too late.
The Stock and Flow Tool helps schools pinpoint the “leaky valves” in their school structures by illustrating how students flow toward and away from graduation benchmarks over the course of their four years. Subsequently, school leaders need to explore the specific structures eliciting this behavior and fix them, potentially dismantling assumptions and belief systems that enable these structures in the first place. To know what works and what doesn’t, New Visions has been investigating our highest performing schools. Our latest report, Design and Data In Balance, is a comprehensive case study that explores school structures and their resulting impact on student trajectories. We see first hand how student progress unfolds within extraordinarily complex and dynamic schools.
What is clear is how high performing schools leverage that dynamism and minimize the complexity. And with the right mindset and the right data visualizations these educators are never in the dark.