The Daily News reports that Cambridge Education Associates is getting a 9% pay raise, even as NYC schools face budget cuts. The average cost of reviewing a school will jump to $4,856, up from $4,427. NYC taxpayers are dishing out 1.1 million for their travel expenses - looks like you and I are paying for our cross-pond friends to fly business class and eat warm chocolate chip cookies. Meanwhile, 8th graders who face retention have lost out on tutoring opportunities. Awesome!

With $2,375,649 spent on the 30 staff working in NYC Department of Education public relations via the "Communications Office," the "Office of Public and Community Affairs," the "Strategic Response Unit," as well as "Community Education Council" PR, can't these wizards keep pay for the Cambridge Ed punters out of the news? You'd think the folks pulling $175,250, $158,603, and $127,776 (top 3 earners in NYC DOE PR) could bring it. NYC Educator provides a clue - were all hands on deck prepping the Ed Next debutante ball?

Posted by eduwonkette at 11:11 AM | Permalink | Comments (3) | TrackBacks (0)

Scott McLeod, a professor of Educational Leadership and Policy Studies at Iowa State, blogs at Dangerously Irrelevant. Many thanks to Scott for this guest post!

When eduwonkette asked me to guest blog about data-driven decision-making in schools, I eagerly agreed. Why? Because in my work with numerous school organizations in multiple states, I have seen the power of data firsthand. When done right, data-driven education can have powerful impacts on the learning outcomes of students.

Unfortunately, most school districts still are struggling with their data-driven practice. Much of this is because they continue to think about using data from a compliance mindset rather than using data for meaningful school improvement. An uninformed model of data-driven decision-making looks something like this:



This is the NCLB model. Schools are expected to collect data once a year, slice and dice them in various ways, set some goals based on the analyses, do some things differently, and then wait another whole year to see if their efforts were successful. Somehow, this model is supposed to get schools to 100% proficiency on key learning outcomes. This is dumb. It's like trying to lose weight but only weighing yourself once a year to see if you're making progress. Compounding the problem is the fact that student learning data often are collected near the end of the year and given back to educators months later, which of course is helpful to no one.

A better model looks something like this:



The key difference in this model is an emphasis on ongoing progress monitoring and continuous, useful data flow to teachers. Under this approach, schools have good baseline data available to them, which means that the data are useful for diagnostic purposes in the classroom and thus relevant to instruction. The data also are timely, meaning that teachers rarely have to wait more than a few days to get results. In an effective data-driven school, educators also are very clear about what essential instructional outcomes they are trying to achieve (this is actually much rarer than one would suppose) and set both short and long-term measurable instructional goals from their data.

Armed with clarity of purpose and clarity of goals, effective data-driven educators then monitor student progress during the year on those essential outcomes by checking in periodically with short, strategic formative assessments. They get together with role-alike peers on a regular basis to go over the data from those formative assessments, and they work as a team, not as isolated individuals, to formulate instructional interventions for the students who are still struggling to achieve mastery on those essential outcomes. After a short period of time, typically three to six weeks, they check in again with new assessments to see if their interventions have worked and to see which students still need help. The more this part of the model occurs during the year, the more chances teachers have to make changes for the benefit of students.

It is this middle part of the model that often is missing in school organizations. When it is in place and functioning well, schools are much more likely to achieve their short and long-term instructional goals and students are much more likely to achieve proficiency on accountability-oriented standardized tests. Teachers in schools that have this part of the model mastered rarely, if ever, complain about assessment because the data they are getting are helpful to their classroom practice.

NCLB did us no favors. It could've stressed powerful formative assessment, which is the driving engine for student learning and growth on whatever outcomes one chooses. Instead, it went another direction and we lost an opportunity to truly understand the power of data-driven practice. There are hundreds, and probably thousands, of schools across the country that have figured out the middle part of the model despite NCLB. It is these schools that are profiled in books such as Whatever It Takes and It's Being Done (both recommended reads) and by organizations such as The Education Trust.

When done right, data-driven decision-making is about helping educators make informed decisions to benefit students. It is about helping schools know whether what they are doing is working or not. I have seen effective data-driven practice take root and it is empowering for both teachers and students. We shouldn't unilaterally reject the idea of data-driven education just because we hate NCLB. If we do, we lose out on the potential of informed practice.



Thanks for the guest spot, eduwonkette!

Posted by eduwonkette at 10:14 AM | Permalink | Comments (4) | TrackBacks (0)

Thank you, Scott, for providing insight into how schools are using data to improve learning, not just test scores. Unfortunately, I’ve witnessed less cheerful data-driven decision making. Some schools are using benchmark tests and other newly available data to play the system and up their numbers. Let me mention a few of these bad and the ugly uses of data.

When I was teaching, my school ran a Saturday program for kids who were close to passing state tests. At the time, I patted myself on the back and thought we were helping our students. Now I understand that our principal was simply trying to increase our passing rates the fastest way she knew how. Other students were nowhere close to passing, and we didn’t roll the red carpet out for them.

The practice of focusing on kids who are close to passing has been well-documented by now. A RAND report on D3M identified this practice as one of the most common forms of data-driven decision making. In one of their studies, more than 75% of principals reported that their school or district encourages teachers to focus on these students, and between one-quarter and one-third of teachers said they actually do focus on these students.

A more expedient way to use data is to select out lower performing students before they even enter your school. In a system like New York City, where all students must apply to high school, even unscreened schools – schools that are prohibited from selecting students based on their test scores, prior grades, etc - have used data to screen out students. Until last year, all unscreened schools had access to individual students’ prior attendance, grades, their test scores, their date of birth, their address, their sending junior high schools, and their special education and English language learner status.

Interestingly, the Department of Education stopped providing this information beginning with admission for the 9th grade class of 2007. Why? One can imagine that the Dept of Ed finally figured out what many of us already knew – that some unscreened schools were using these data to pick off the best students. (For more info on the issue of creaming in NYC, see here, here, and here.)

Certainly data-driven decision making has a bright side, but it has a dark side as well, especially when schools feel intense pressure to quickly improve their scores. Scott makes the important point that we shouldn't throw out the baby with the bath water, and I agree. But as more schools implement D3M-based approaches, we should be aware that its uses are not uniformly positive.

Posted by eduwonkette at 9:40 AM | Permalink | Comments (0) | TrackBacks (0)

I hate leftovers, too. But there is a lot left to say about last week's theme of data-driven decision making, so I'll tie up loose ends this week. Forthcoming posts include: How are data currently being used in schools, and who's entered the business of providing data solutions? What are some of the technical challenges with value-added models of teacher effectiveness? And what are their potential unintended consequences?

Posted by eduwonkette at 9:56 PM | Permalink | Comments (0) | TrackBacks (0)

skoolboy returns to weigh in on data-driven decision making:

I’m as much a fan of data as the next guy. But I worry that proponents of data-driven decision-making are understating just how hard it is to use data thoughtfully.

I’d like to describe the strategy championed by the New York City Department of Education, and point out the difficulties involved. The logic that the DOE is promoting is (a) use data to identify an area where a school is lagging, either in relation to some absolute standard or to other similar schools; (b) use the available data systems to identify similar schools that are doing better in this area; (c) ask these more effective schools what they are doing that accounts for their success; and (d) adapt their suggestions for use in the school.

It’s not as easy as it looks to determine which schools are doing better than others. Two different criteria are relevant: is the difference in performance between two schools large enough to matter, which is sometimes termed educational significance or practical significance; and is the difference in performance between two schools real, or could it just be due to chance, which is typically described as statistical significance. Ideally, we are interested in differences that are both practically and statistically significant. But a difference could be large, but not statistically significant (which is often the case when we have a small sample of information about performance), or statistically significant, but very small (in which we are pretty sure that the difference is real, but it’s just not very important). (Yes, statistical significance does matter!)

This is kind of abstract, so here’s an example, drawn from the NYC Department of Education’s Survey Access tool, which reports the results of the system’s first round of Learning Environment Surveys in the spring of 2007. The Department’s spiffy PowerPoint presentation imagines the principal and a group of teachers in (mythical) IS 402 identifying teacher engagement as an issue. In particular, teachers in this school generally disagreed that “Obtaining information from parents about student learning needs is a priority at my school.” Using the Survey Access tool, it’s possible to identify 12 similar NYC schools (i.e., middle schools with an enrollment over 700 and at least 25% ELL students), seven of which have more positive scores on this question. In the top school, the Eleanor Roosevelt School, 71% of the teachers strongly agreed or agreed with the statement, whereas in the bottom school, 13% of the teachers strongly agreed or agreed. (In mythical IS 402, 36% of the 31 teachers who responded to the survey strongly agreed or agreed.)

So why not just look at the seven schools above IS 402? Because the percentages of teachers strongly agreeing or agreeing is an estimate of the true percentage that would be observed if all teachers in the school responded to the survey. (In these 12 schools the teacher response rate ranged from 26% to 53%; in mythical IS 402, 40% of the teachers responded.) Our interest is in the population of teachers in the school, not just the sample that chose to respond. And there’s a degree of uncertainty in these estimates. If a different group of 31 teachers in IS 402 responded, just by chance, we might not have obtained an estimate of 36% strongly agreeing or agreeing. In fact, with a sample of 31 teachers responding and a sample estimate of 36%, the percentage of all of the teachers in IS 402 agreeing or strongly agreeing could plausibly range from 23% to 49%. (There’s a finite population correction in there, for those who care about such things.) That’s a pretty big range, and the range of possible values is pretty large for the other dozen schools as well.

Of the seven schools above IS 402, just one of them, the Eleanor Roosevelt School, is really head-and-shoulders above it in a statistical sense. The other six are statistically indistinguishable, because there’s so much overlap in the intervals in which the true percentage of all of the teachers strongly agreeing or agreeing in each school lies.

Would the principal and teachers in IS 402 learn something from asking the staff in these seven other schools how they do things? Sure! It doesn’t hurt to think about new ways of doing business. Will doing so raise performance in IS 402? Probably not. Because an assessment of statistical significance suggests that, with the exception of Eleanor Roosevelt, these other schools really aren’t doing better, and therefore there’s no reason to think that adopting their practices will yield genuine improvements.

Data-driven decision makers, beware of spurious comparisons.

Posted by eduwonkette at 6:15 PM | Permalink | Comments (8) | TrackBacks (0)

In many ways, data-driven decision making (D3M) in education is an old idea packaged as a new one. As far back as anyone can remember, teachers have given their students regular quizzes, projects, and tests. When students performed poorly, "data-driven" teachers retaught the material or tried to figure out what went wrong. Without the benefit of spreadsheets or data displays, teachers have attempted to tailor their instruction to different groups of students. To be sure, there have been assumptions, blindspots, and kids overlooked, but the fundamental idea of teaching, assessing, figuring out what works for whom, and re-teaching is as old skool as Tupac.

What's new is the formalization of this process. Student learning is now quantified in test scores, stored in data warehouses, and made available for teachers to analyze. What's also new is the creation of two very different camps of data-driven reformers, which Sol Stern recently referred to as instructionists and incentivists.

Instructionists like the authors of the book DataWise see data as a useful tool for identifying problems of teaching practice, investigating them, and addressing them. Their focus is on improving student learning, not just test scores, and they are very clear about this distinction. They worry about the dangers of test score inflation and gaming the system, and advise schools to take steps to ensure that their improvements are not simply the result of shortcut practices that do not improve student learning.

Instructionists define data broadly - student work, student attitudes, and more are all relevant. In this view, data are not a replacement for expertise, and data don't make decisions. Rather, data are a useful tool for educators to harness to improve instruction. Instructionists stress that D3M requires a collaborative learning process, and are concerned about approaches that use data to blame individuals rather than support educators' professional growth. For example, Boudett, City, and Murnane wrote in Data Wise:

Agreeing on norms like "no blame" is an essential first step in creating an atmosphere that supports productive data discussions. It is important to emphasize from the beginning that data will not be used to punish teachers, but to help them figure out how to teach their students more effectively.

Incentivists' view of D3M is different. Perhaps best captured in adversarial approaches like CompStat (represented in The Wire as CityStat), D3M is a way to hold people's feet to the fire. This approach is agnostic about the "how" of fixing the numbers, and thus rewards better statistics with little attention to how these numbers were produced. As such, incentivists see little need to study diverse kinds of data - in fact, non-quantified data are dismissed as anecdotal. In this view, data, not educators, are the experts.

Unfortunately, D3M is being (has been?) hijacked by incentivists. The focus is on the numbers, not the process of arriving there, which invites all kinds of mischief. Most of the D3M that I've observed has involved schools figuring out how to cut corners in order to make test score ends meet. And to those who are ready to crucify the teachers for doing so, let me reiterate that it is unrealistic to expect schools to ignore pressure from city, state, and federal muck-a-mucks to rapidly improve test scores, not learning. As long as we only reward end game numbers and ignore the process through which schools get there, I predict that the instructionist approach to D3M will have a hard time getting off the ground. This is regrettable, as the ongoing, systematic analysis of multiple forms of data is an immensely promising strategy for improving teaching and learning.

Tomorrow I'll provide examples (good, bad, and ugly) of how teachers are currently using data. On Friday, I'll investigate the corporations that have entered the K-12 marketplace to provide "data solutions" for school districts.

Posted by eduwonkette at 12:08 AM | Permalink | Comments (2) | TrackBacks (0)

Walk into any school's faculty meeting, and you'll think you've stumbled into a tongue twister competition. The push for data-driven decision making, DDDM, D3M - whatever you prefer to call it - is everywhere. This week I'll explore what data-driven decision making can and can't do for education and share some of the research on how data are currently used in schools.

Got something to share about how data are used in your school? Email me at eduwonkette (at) gmail (dot) com. In the meantime, you can check out some resources on DDDM over at Scott McLeod's site (the writer of Dangerously Irrelevant).

Wednesday: D3M Box Scores: Incentivists:10, Instructionists:1

Thursday: D3M Gone Wild: How Do We Know What Data To Trust to Inform Decision Making?

Posted by eduwonkette at 6:49 PM | Permalink | Comments (1) | TrackBacks (0)

Spoiler alert: I'm going to write about data-driven decision making next week, so who better to profile than Kathryn Boudett, who teaches at the Harvard Grad School of Ed and is a co-author of the book Data Wise: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning. Note that the book is about improving teaching and learning, not just test scores! And that's why I like it. Here is a little snippet about the book, which I will say more about next week, and the syllabus for her course.

Posted by eduwonkette at 11:30 PM | Permalink | Comments (5) | TrackBacks (0)