« August 2008 | Main | October 2008 »
We're here to thank our president,No matter how catchy the ditty, a song can't carry a fundamentally flawed law. That's where Tom Toch and Doug Harris come in. They've penned a thoughtful commentary in this week's Ed Week about the future of NCLB (Salvaging Accountability). It's an important one, because it recognizes that NCLB conflates the school's contribution to student learning with what students bring to the school to begin with. Essentially the argument is that:
For signing this great bill,
That's right! Yeah,
Research shows we know the way,
It's time we showed the will!
Betsy Gotbaum is the Public Advocate for the City of New York. The Public Advocate is an independently elected citywide official who serves as a public ombudswoman.
The soul-crushing aspect of Fryer's theoretical framework....is that it lets the curriculum and the teacher and the school entirely off the hook. It's not a matter of creating learning experiences that connect with a child, her culture, her community; or creating curriculum that intrigues her or teaching that respects her and her family; or about creating schools where families and communities can find support and education and develop skills of active citizenship....No, it's a much more cynical view on students living in poverty. They don't care, they are only motivated by material objects that they don't have, they have to be bribed into "learning" (or at least learning to get a better score on a bubble sheet)....RRRRR! Get me out of here.COWAbunga, indeed.
Sometimes it feels as if the forces in the universe are alligning to make this job as difficult as possible, just to see if I have the balls to stick with it. Other times, it feels as if teachers (as people) are the absolute last priority on everyone's list...that we will just suck it up and deal with ridiculous situations "for the kids."2) Quiz Show: Celia Oyler puts together her second New York City Progress Report Quiz.
If one more person tells me to do it "for the kids", I might throw a kid at them. Seriously. Stop playing on our good intentions and altruistic dedication to the future and treat us like the professionals you so desperately claim you want us to be. It just seems at times as if this job teeters on the brink of being inhumane.
eduwonkette and I have been blogging about the School Progress Reports released last week by the New York City Department of Education. We’ve shown that, although the performance and environment scores of schools were pretty consistent from last year to this year, the student progress scores were virtually unrelated—knowing a school’s progress score from last year didn’t predict which schools would demonstrate a lot of progress this year. This, we argued, demonstrated that the progress part of the School Progress Report—representing 60% of the letter grade each school received—wasn’t really telling us which schools consistently are promoting student progress, but rather was mostly random error.
The problem was particularly acute in the domain of English Language Arts (ELA). The stability in the student progress scores from 2007 to 2008 was so low that it led skoolboy to wonder if a monkey could actually do a better job predicting which schools show progress in students’ ELA performance in 2008 than relying on the DOE’s 2007 student progress score. The particular measure I examined was the percentage of students in the school making at least one year of progress on the ELA test from last year to this year. (As we've noted in earlier posts, the calculation of this measure changed slightly from 2007 to 2008.)
In the interest of full disclosure, skoolboy didn’t actually rent a monkey to pick the schools. Animals scare him, and he wouldn’t have been able to record the picks while hiding under his bed. What I did instead was use a random number generator to assign each school to the top or bottom half of the distribution of schools on last year’s peer and citywide measures of the percentage of students making a year of progress in English Language Arts.
The DOE got credit for a correct prediction if it correctly predicted that a school would be in the top half of this year’s schools, based on the school being in the top half on the DOE’s 2007 measure, or correctly predicted that a school would be in the bottom half of this year’s schools, based on the school being in the bottom half last year. The monkey got credit for a correct prediction if the randomly-selected location of a school as being in the top half of the 2007 distribution correctly predicted that a school would be in the top half of this year’s schools, or the random pick of being in the bottom half of last year’s distribution correctly predicted that a school would be in the bottom half of this year’s schools. These predictions were done separately for the 570 elementary schools, 128 K-8 schools, and 289 middle schools which received overall letter grades last year and this year.
Round 1. We begin with the peer horizon score for the 570 elementary schools. The DOE’s peer horizon progress score from last year correctly predicted the progress status of 46% of the elementary schools this year. The monkey correctly predicted the status of 51% of this year’s schools.
Score: Monkey 1, DOE 0.
Round 2. We next turn to the citywide horizon score for the 570 elementary schools. The DOE’s citywide horizon progress score from last year correctly predicted the progress status of 47% of the elementary schools this year. The monkey correctly predicted the status of 52% of this year’s schools.
Score: Monkey 2, DOE 0.
Round 3. In this round, we examine the peer horizon scores for the 128 K-8 schools. The DOE’s peer horizon progress score from last year correctly predicted the progress status of 45% of the K-8 schools this year. The monkey correctly predicted the status of 55% of this year’s schools.
Score: Monkey 3, DOE 0.
Round 4. Next, we look at the citywide horizon progress scores for the 128 K-8 schools. The DOE’s citywide horizon progress score from last year correctly predicted the progress status of 43% of the K-8 schools this year. The monkey correctly predicted the status of 47% of this year’s schools.
Score: Monkey 4, DOE 0.
Round 5. The final stage of the competition examines the 289 middle schools. The DOE’s peer horizon progress score from last year correctly predicted the progress status of 40% of the middle schools this year. The monkey correctly predicted the status of 50% of this year’s middle schools.
Score: Monkey 5, DOE 0.
Round 6. The last round looks at the citywide horizon progress scores for the middle schools. The DOE’s citywide horizon progress scores from last year correctly predicted the progress status of 45% of this year’s middle schools. The monkey correctly predicted the status of 49% of this year’s middle schools.
Score: Monkey 6, DOE 0.
skoolboy will forego the cheap jokes about how a monkey could do a better job of managing New York City’s accountability system than the people currently in charge. On the whole, they’re smart, hard-working people, and ridiculing them is not likely to persuade them to change their behavior (as satisfying as it may be at particular moments.) But the system that they have designed and implemented is profoundly flawed, as this comical example illustrates, and it needs to change. eduwonkette and I are going to keep hammering on this point, because it has such important consequences for students and for schools.
And besides: I bet the DOE would beat the monkey in predicting school progress scores in math. (But it wouldn’t be a rout.)
Today marks the one-year anniversary of eduwonkette's bold entry into blogging about education. A lot has happened here over the past year, across 487 different posts, and thousands and thousands of comments. (Heck, back then, eduwonk and eduwonkette were BFF.)
eduwonkette has tackled a remarkably diverse set of education policy issues: teacher quality, No Child Left Behind, gender differences in academic performance, myths about small schools, New York City's School Progress Reports, the "it's being done/no excuses" argument, the achievement gap and "acting white", value-added assessment, choice, incentives, unions ... the list goes on and on. And she's done it all with great style and wit, first with and now without the mask.
Today is an opportunity to revisit the principles that brought her to the blogging world:
Are you tired of listening to the usual suspects on education policy? So am I. Education policy debates are dominated by a small number of very loud voices. In these debates, ideological claims, rather than research, data, the experience of educators, and common sense, are wielded as weapons. What are some of the problems I see with these debates?
• A selective reading of educational research: The loudest outlets pick and choose which studies are relevant, often leading to a skewed view of what we know and don’t know about how to improve schools.• An inattention to the costs and benefits of policies: Policy solutions are endorsed as if they have no downside. But we know that all actions have positive and negative consequences. The education policy debate would benefit from such an acknowledgement.
• A fundamental disrespect for the knowledge of teachers and principals who work in public schools: Too often, teachers and administrators are dismissed as “self-interested” or “protecting the status quo” when they question what policymakers wreak on their classrooms and schools. In no other profession are we willing to discount the opinions of those closest to the work at hand. Education should be no different.
Rather than stepping into this ideological boxing ring, this blog takes a different approach.
And so she has. Happy anniversary, eduwonkette!
* That test scores alone are sufficient to evaluate a teacher, a school, or an educational program.EW: I'm intrigued by your third point about alignment. For example, we often hear that because state testing systems are directed towards a particular set of standards, we should primarily be concerned with student outcomes on tests aligned with those standards. This is the common refrain about a "test worth teaching to." What's missing from this argument?
* That you can trust the often very large gains in scores we are seeing on tests used to hold students accountable.
* That alignment is a cure-all - that more alignment is always better, and that alignment is enough to take care of problems like inflated scores.
1) The poorly constructed progress measure is simply measuring noise.New Yorkers are left with three courses of action:
2) The DOE somewhat tweaked the progress measure for this year, so the results are not comparable.
3) The receipt of and publicity around last year’s progress measures fundamentally changed how New York City’s elementary schools do business, so that schools that were more successful in raising student achievement in 2007 suddenly became less so, and schools that were less successful in raising student achievement in 2007 suddenly became more so.
* If explanation 1 is correct, we should ignore these report cards altogether because they are primarily (60%) measuring error.Thanks to skoolboy’s masterful analysis of the data, we present evidence below the fold to suggest that the likely culprit is measurement error. The evidence is not conclusive, because every single element of the progress measure—and there are 16 of them in this year’s student progress measure—changed slightly from last year to this year. The strategy that we pursue below is to compare those elements of the progress measure that were used in both years - for example, the percentage of students making at least one year of progress, or the average change in proficiency scores. Again, we stress that these measures were not identical across years, but one would expect them to be moderately related. Needless to say, that is not what we found. We think it extremely unlikely, given these analyses described in detail below, that this is simply due to a tweaking of the progress report measures.
* If explanation 2 is correct, we should not compare schools' grades in 2007 with their grades in 2008, because they are measuring fundamentally different dimensions of school performance. In this case, the collective hysteria that has ensued in NYC schools last week about why grades are up or down is all for naught.
* And if explanation 3 is correct, eduwonkette and skoolboy should shut up and get out of the way of the silent revolution that has transformed public schooling in New York City.
You have to feel sorry for the brain trust working on [the Progress Reports]. Up until this week, if you told the American public that you had hired a group of really smart economists to develop some complicated statistical models that explain the entire universe, many people would have been very impressed. But that was before those really smart economists crashed the entire global economy.
(...) To be blunt, the results make no...sense to anyone. Schools shouldn't be jumping all over the place in terms of progress scores or achievement. That is simply not reality. A school that moved up should be proudly proclaiming how they spun gold from wet straw, but the principal says nothing changed. What strange statistical model is spitting out this nonsense?....recoding statistical noise into a letter grade does not an accountability system make.
Now that the results are out and the validity in question for a second year, critics are being told: "Statistics don't lie." But ordinary people know that companies shouldn't be allowed to borrow 30 times more than what they own so they can gamble the money on other stocks. You don't need a Ph.D. in economics to know what that smells like. New York's "Progress Reports" have the same foul stench. Maybe everyone will feel differently about it in 10 or 20 years when we have forgotten the damage done, but right now something about value-added models seems just a bit too...risky.
Philissa Cramer totally geeks out over at GothamSchools, and posts a great figure showing that smaller schools were more likely to experience wild swings in their school grades. Head over and check it out.
Some days, skoolboy feels bad for the hard-working folks in the New York City Department of Education. They’re caught between a political rock and a statistical hard place. The political rock is the New York State accountability system, which complies with No Child Left Behind’s requirements to test students annually in grades 3-8 in Mathematics and English Language Arts, and to classify students, based on their test scores, as either Not Meeting Learning Standards (Level I), Partially Meeting Learning Standards (Level II), Meeting Learning Standards (Level III), or Meeting Learning Standards with Distinction (Level IV), and then aggregate the performance of students, and subgroups of students, to assess the school’s progress toward the goal of 100% proficiency for all students by the year 2014. The mechanism for this is a series of grade-specific exams, with a broad (but arbitrary, as Dan Koretz explains in Measuring Up) standard-setting process that define the scores on the exam that correspond to the four proficiency levels. Whatever a student’s scale score on the exam, he or she is classified into a particular proficiency level.
The statistical hard place is that the proficiency levels are only part of the story. The NYC DOE has found that the scale scores matter, such that a student whose scale score is halfway between the cutoffs for Level II and Level III, and therefore whose proficiency level is Level II, has a higher probability of graduating from high school on time than a student whose scale score is right at the cutoff for Level II. The scale scores have predictive validity—that is, they predict educational outcomes that we think of as important—but they don’t have the political currency of the proficiency levels specified by the state and the federal government.
There’s no evidence, to skoolboy’s knowledge, that achieving a proficiency level on NCLB-style exams has any predictive validity over and above the scale scores on which they are based. (Another regression discontinuity design study waiting to happen.) But I’ll wager that they don’t.
Whether or not the state/NCLB proficiency levels matter, the NYC DOE is stuck. They have to pay homage to the state standards, even though their internal evidence shows that partial progress—“learning quite a bit,” in skoolboy’s terms—really does matter for students’ futures, and therefore is something that schools should be held accountable for.
And I don’t disagree. I would be comfortable (though not ecstatic) with school progress reports that used changes in scale scores to quantify how much students had learned from one year to the next, under two conditions: (a) if the exams were vertically linked, and (b) if the uncertainty in the estimates of school-level effects on the average change were taken into account. Neither of these conditions is met in the current New York City School Progress Reports.
Navigating the political rock and the statistical hard place is definitely a challenge, both rhetorically and in the construction of the School Progress Reports. Rhetorically, the DOE is obliged to argue that a student who is Level III in fourth grade and Level II in fifth grade has lost ground—that student has fallen off of the sharp Level III cliff—because the state and federal accountability metrics treat this as a sharp discontinuity. But as a practical matter, the student may not have fallen off a cliff; rather, she may be just a little bit lower on a gradual hill in fifth grade than we’d like, but still higher on the hill than she was in fourth grade--and the DOE’s internal analyses document that anyone who is higher on the hill is better off than someone lower.
What’s the DOE to do? Well, it could continue to escalate the rhetoric directed toward its critics. (I note with alarm that the DOE went from calling me by my blogging name “skoolboy” on Monday to calling me “Professor Pallas of Teachers College” on Wednesday—whose proclivity to giving A’s to all of his students will come as a surprise to many of them—what’s next? Examining my teeth?) Or it could speak honestly and openly about the challenge of incorporating political and technical realities into the School Progress Reports. I think readers know which path skoolboy recommends.
A week ago, skoolboy encouraged readers to predict schools' upward and downward grade mobility. Here's how that shook out. When 26% of elementary and middle schools that received Fs last year - 9 schools - climb from a F to an A, it does make you wonder what exactly it is that we are measuring. Likewise, 26 schools cascaded from As or Bs to Ds or Fs. Readers, stare into the table and tell me what you see...
Some of you have asked what fraction of NYC schools receiving each Progress Report grade are in good standing with NCLB. As a refresher, NCLB labels schools in need of improvement based on overall proficiency. NYC's system is based 60% on year-to-year growth, 25% on proficiency, 5% on attendance, and 10% on surveys.
Given these differences, perhaps you won't be surprised to find that a higher fraction of F schools are in good NCLB standing than are A schools:
* 74% of A schools are in good standing with NCLB
* 67% of B schools are in good standing with NCLB
* 69% of C schools are in good standing with NCLB
* 48% of D schools are in good standing with NCLB
* 89% of F schools are in good standing with NCLB
What if we just look at the "performance grade", aka the proficiency grade, that each school received, and see how that maps on to NCLB good standing? Recall that this year, schools also were given separate grades for the performance, progress, and environment categories. I guess the peculiar results below are a function of the fact that schools are being compared to peer groups, but here's what I've got:
* 86% of A schools based on proficiency on the are in good standing with NCLB
* 60% of B schools are in good standing with NCLB
* 60% of C schools are in good standing with NCLB
* 51% of D schools are in good standing with NCLB
* 75% of F schools are in good standing with NCLB
Suppose that your fourth-grader takes a state test that shows that she understands the associative property of multiplication, can multiply two-digit numbers by two-digit numbers, and can find the perimeter of a polygon by adding up the length of the sides. A year later, as a fifth-grader, she takes a test that shows that she can compare fractions and decimals using <, > or =; identify the factors of a given number; simplify fractions to their lowest terms; and knows that the sum of the interior angles of a quadrilateral is 360 degrees—but she cannot yet create algebraic or geometric patterns using concrete objects or visual drawings (e.g., rotate and shade geometric shapes). Would you say that your child had lost ground in proficiency, or actually gone backward?
Jim Liebman would. Liebman, the Columbia University law professor on leave as Chief Accountability Officer at the New York City Department of Education, is quoted and paraphrased in an article by Jim Dwyer in Saturday’s New York Times on the F grade that P.S. 8 in Brooklyn Heights will receive in this year’s School Progress Reports—a grade that many are finding hard to believe, given that 80% of the students tested in the school are judged proficient in math, and two-thirds are judged proficient in English Language Arts. Doubly embarrassing, in that Chancellor Joel Klein and Mayor Mike Bloomberg have publicly declared the school to be successful and worthy of emulation.
So the spinmeisters are out, and the spin here is justifying the grade of F by arguing that the children in P.S. 8 are going backward. “You drop them off at the beginning of the year, and on average, by the end of the year, your child lost ground in proficiency,” Dwyer quotes Liebman as saying. “Where was the child last year, and where is the child this year?” Liebman asked. “You’re comparing them to themselves.”
A gentle reminder to Mr. Liebman, who was hired in January, 2006: the state math and ELA tests which children take, and are the primary basis for assigning these lovely letter grades, are not vertically equated. (See skoolboy's testing primer here.) This means that there is no basis for comparing performance on the fourth-grade test with performance on the fifth-grade test. For each test, there is a subjective judgment about what level of performance constitutes proficiency, but the tests are independent. There is no basis for claiming that children are going backward; there’s no justification for claiming that a child “lost ground in proficiency,” since proficiency doesn’t exist in the abstract, but rather in grade-specific skills; and the children are not being compared to themselves, but rather their location in the distribution of children’s performance in one year is being compared to their location in the distribution of children’s performance the following year.
Perhaps Jim Liebman simply misspoke, as perhaps did Chancellor Joel Klein when he referred to statistical significance as “playing something of a game.” Such missteps might arise from the tremendous pressure to justify a particular high-stakes evaluation of a school when there are multiple sources of information about school performance that point in different directions—NCLB status, achievement levels, gains, school quality reviews, not to mention the public pronouncements of Liebman’s boss, and his boss’s boss.
There’s nothing wrong, in skoolboy’s view, in looking at students’ achievement growth as one of several criteria for judging how well a school is doing in relation to other schools. But I would never think of using year-to-year changes in proficiency levels on just two tests as the primary basis for evaluating a school’s performance. And neither would most people who study testing and assessment for a living.
Our results raise serious concerns about the current methods that are used to hold schools accountable for their students' achievement levels. Because achievement-based evaluation is biased against schools that serve the disadvantaged, evaluating schools on the basis of achievement may actually undermine the NCLB goal of reducing racial/ethnic and socioeconomic gaps in performance. If schools that serve the disadvantaged are evaluated on a biased scale, their teachers and administrators may respond like workers in other industries when they are evaluated unfairly - with frustration, reduced, effort, and attrition. Under a fair system, a school's chances of receiving a high mark should not depend on the kinds of students the school happens to serve.Crystal clear, creative thinking is the distinguishing feature of Downey's work - see, for example, his paper on school effects on child obesity, or his paper asking if schools are "the great equalizer."
This morning, the Center for Education Policy in Washington, DC is issuing the latest in a series of state-level reports on the fate of schools restructuring under NCLB policy. Today’s report, authored by Brenda Neuman-Sheldon (a one-time student of skoolboy’s, but I hear that she’s back on solid food), examines restructuring schools in Maryland. In 2007-08, Maryland had 38 schools in restructuring planning, a huge increase over the four schools the preceding year, and 64 schools in restructuring implementation, a 7% decline from the preceding school year. The restructuring schools are concentrated in a small number of Maryland’s 24 school districts, with 61% of the restructuring schools in Baltimore City, and an additional 30% in Prince George’s County, which adjoins Washington, DC. This concentration has stretched the capacity of the state and these districts to support restructuring planning and implementation. Prince George’s County, for example, soared from one school in restructuring planning in 2006-07 to 21 in 2007-08.
Neuman-Sheldon identifies a major shift in the form that restructuring schools in Maryland is taking. Whereas 58% of the schools in restructuring implementation in 2007-08 relied primarily on the appointment of a school “turnaround specialist” as the engine of restructuring (already a decline from the 73% using this option in 2005-06), all of the schools in restructuring planning that had submitted a plan at the time the report was written were proposing some form of “zero-based staffing”—i.e., replacing most or all of the staff in the school or asking all staff to reapply for their positions. It’s the neutron bomb theory of school reform!
But is it a good theory? That remains to be seen. What mechanism will bring highly-qualified teachers to these failing schools? Where will the tenured teachers who leave the schools go? In schools that replace only some of their staff, how will decisions about who stays and who leaves be made?
Beyond these logistical questions, though, lies another fundamental challenge: will changing the staffing—including the principals, who, Neuman-Sheldon reports, are often surprised to learn that when they select zero-based staffing as an option, they’re placing their own jobs on the line—fundamentally alter the context for teaching and learning in the school, when other powerful forces shaping teaching and learning aren’t changing at all?
I think the reason we don't want to inject the idea that student achievement is based partly on what [students] come to school with (parent support, poverty rates, etc.) into the NCLB debate is because it comes too close to admitting that our public education system doesn't help everyone equally. And that education does give everyone the same advantages is one of our cherished public ideals....We don't want to admit that there are problems that are too big for education as it exists right now to fix.The second winner is Erin Johnson, who, in a series of comments, made compelling arguments about the differences in the evidence bases for educational and medical practice. Read them all here, and here's a tasty morsel from one of them:
I've often wished that if I am going to be held so accountable for student performance that we had a boarding school system, so I could make sure my students had a quiet place to do homework, a good dinner and breakfast, etc....I like the idea of value-added assessments; we get value-added scores for each classroom teacher in my state. I wish that NCLB took those scores into account....I know one year, our value-added scores were great, yet we still didn't make AYP because our students were so far behind to begin with. It's very demoralizing to be labeled in the news as a failing school when you've made so much progress based on where the students started.
The development of medicine and education was not random. Both were a function of very specific decisions made by key opinion leaders and laws passed both on the state and federal levels....We take for granted the the scientific, evidentary basis of our medical system, but it was not pre-ordained to be so.
I wish that you -- you personally and every creative individual and organization you know -- will find a way to directly engage with a public school in your area, and that you'll then tell the story of how you got involved, so that within a year we have 1,000 examples of innovative public-private partnerships.Now there's a website called "Once Upon A School" that's tracking project ideas for engaging in local schools. If you're looking for a little shot of edu-Red Bull this week, check them out.
skoolboy doesn’t fancy himself a particularly political creature, although some readers would likely argue that I’m kidding myself, in that blogging is an inherently political activity. In any event, I haven’t chosen to do a close analysis of the positions or proposed policies of the finalists in our Presidential derby. I’ll make a brief exception today, not to make political hay, but rather to try to illuminate an enduring sociological challenge.
Yesterday, Barack Obama issued a new plan for school reform, emphasizing choice and innovation, investments in technology, enhanced college readiness, incentives for improved classroom teaching, and heightened responsibility from parents and from the federal government. The last piece of this agenda calls for the creation of quarterly parent report cards to support individual learning plans. Press reports of this component of the Obama agenda conveyed the impression that such report cards would simply be a fancy repackaging of the periodic report cards that parents already receive itemizing how their children are doing in school. But the Obama plan has something more ambitious in mind, including “the concrete information [that parents] need to help improve their child’s performance each year and plan for post-high school education”:
Is more information inherently better than less information? No, skoolboy thinks, not if more information is overwhelming. This is a remarkably diverse set of objectives, and each of them would require at least a term paper’s worth of material to convey what’ s important. Providing parents with the information necessary to enable them to choose between their child’s current school and alternatives? What’s the right metric here? Value-added models of school effects? I've seen highly-educated professionals struggle to understand them. Concrete information on how a child is expected to perform at the child’s grade level? You can find this on most state department of education websites, but it’s not something that can be summarized in a page or two.
The more serious problem, though, is the assumption that providing information in and of itself creates a logic for action. The available evidence calls into question both the inclination and the ability of parents to use information to make decisions regarding their children’s schooling. Moreover, these orientations and predispositions are linked to social class. skoolboy’s long-time colleague Annette Lareau, noted here as a cool person you should know, has written extensively about the differing childrearing and schooling practices of middle-class and working-class parents. Her analyses show that middle-class parents are predisposed to see family and school as connected, and to be proactive in seeking out and evaluating educational opportunities for their children. Working-class parents care just as much about their children’s education, but they see family and school as separate, and are less likely to intervene in what they view as the responsibility of the school.
Provision of this information, therefore, could have the unintended consequence of exacerbating social class differences in schooling. Middle-class parents may be better able to make sense of the information, and will be more prepared to act on it. Working-class parents may be overwhelmed by it, and will not necessarily know how to translate the information into concrete action steps. It wouldn’t be the first policy initiative to founder by assuming that everyone behaves like the middle class.
And finally: “quarterly”? Maybe that’s just rushed copyediting…
The characteristics that Medicare patients bring with them when they arrive at a hospital with a heart attack or heart failure are not under the control of the hospital. However, some patient characteristics may make death more likely (increase the ‘risk’ of death), no matter where the patient is treated or how good the care is. … Therefore, when mortality rates are calculated for each hospital for a 12-month period, they are adjusted based on the unique mix of patients that hospital treated.If you replace the word "hospital" with "school" above, you can imagine the reception this statement would receive in the educational accountability debate. Soft bigotry of low expectations, and you probably kill baby seals for fun, too.
Sometime soon, with great fanfare, the New York City Department of Education will release this year’s School Progress Reports. (Word on the street is that schools already know their grades.) The School Progress Reports, for better or worse, are the centerpiece of the NYC accountability system. (skoolboy thinks for worse, but more on that later.)
The DOE has made a number of changes to the Progress Reports for this second iteration, and I think that eduwonkette had something to do with that (as did other critics and analysts outside of the Tweed inner circle.) We can expect to see separate letter grades for the three major dimensions on which the Progress Reports are based: school environment (including attendance, and parent, teacher and student surveys), student performance, and student progress. But the overall format appears to be unchanged: most of the grade is based on student progress on test scores, and such gains are not very reliable from one year to the next. There is, in skoolboy’s opinion, a false sense of precision conveyed by these letter grades, as they are based on components that are measured with error, but that measurement error is not reflected in how the grades are calculated. And I’m particularly annoyed at the misuse of social surveys for accountability purposes.
Nevertheless, the DOE is marching onward, and we’ll have this year’s grades to pore over in the near future. (And you can bet that eduwonkette will put on the green eyeshade for this, even though it clashes with her cape and mask.) How many schools will improve their grade from last year to this year? How many will fall? It’s time to make some predictions. What do you think, readers?
Here's a five-by-five table designed to show how this year’s grades are associated with last year’s grade. Each column represents last year’s grade, and each row represents a possible outcome for this year. The column percentages will add up to 100%. Try to fill in the blanks: What percentage of the schools that received A’s last year will receive an A this year? What percentage of A’s will decline to B’s? What fraction will fall further to C’s, D’s, and F’s? At the other end of the spectrum, what percentage of last year’s F’s will remain F’s? What percentage will climb out of the cellar to obtain a D? Will any make the leap from F to A?
As a reminder, last year, about 23% of schools received an A; 38% received a B; 26% received a C; 8% received a D; and 4% (i.e., 53 schools) received an F.
A caveat: The DOE knows that the legitimacy of the School Progress Reports depends on the grades not being too volatile from year to year. If 75% of last year’s A’s became F’s this year, no one would take this scheme seriously. (And if schools that everyone views as exemplary or high-performing got middling grades, this too would call the scheme’s legitimacy into question. So don't expect Stuyvesant High School to get a C.) There may not be very much fluctuation from last year to this. You can be sure that the DOE has constructed this year’s scores so that there’s not too much instability from last year to this year.
But since we believe in incentives on this blog, the reader who comes closest to the actual association between last year and this year shall receive a prize to be selected by eduwonkette—and we know how creative she can be. Be sure to fill in all 25 blanks.
*Employees of Tweed Courthouse, KPMG Consulting, and the Parthenon Group are ineligible for this contest.
No, there's no convention commentary here (or else skoolboy would have to shoot himself). This week’s “Comment of the Week Award,” also known as the COWAbunga Award, goes to NYC Educator, for a comment on yesterday’s Coffee Talk question about which big-city school district is the worst-managed. NYC Educator wrote:
I see the system in which I work on a daily basis, and I don't always see its reality reflected in the press--although they've made great strides over the last few years.
Really, when you're a teacher and you find blatantly preposterous statements in the NY Times, you have to wonder about the reporting from other cities. Who knows whether or not they're telling the truth, or whether they've sent anyone to find out what was really happening. Certainly it's easier to just ask City Hall what's going on and write whatever they tell you.
Big-city school districts are notorious for turning inward—transparency has never been their strong suit. A vigorous press is one of the ways that those in charge of these districts can be held to account for their responsibilities as public servants. This is one of the reasons why yesterday’s announcement that the New York Sun may be folding at the end of the month was so disappointing. skoolboy didn’t often agree with the editorial pages of the Sun, but I always felt better knowing that there was a venue for opinions different from mine to be aired and debated.
Even more importantly, though, the shutdown of the Sun would mean less daily beat reporting on New York City schools. eduwonkette has said repeatedly, and I agree wholeheartedly, that Sun reporter Elizabeth Green has been breaking important stories since she arrived on the scene last year, and it would be a shame if those of us with a stake in New York City schools were to be deprived of her investigative skills. (And yes, she wrote a feature on eduwonkette, and I’ve assisted her in a story or two, but the quality of her work speaks for itself.) Alexander Russo over at This Week in Education has also lamented the recent transitions of a number of well-regarded education writers to new positions that remove them from day-to-day beat reporting. Really, is it possible to have too much high-quality reporting on public education? Maybe … but we have a long way to go before that’s a serious question to consider.
In the meantime, the gap between the person-power devoted by school systems to transmitting messages about public schools to the public and the person-power available in an independent press to interpret these messages in a critical and thoughtful way for the public continues to widen. This, in skoolboy's view, does not serve the public interest.
Long-time followers of skoolboy (hi, Mom!) know that his first posts on eduwonkette’s blog were about class size. I argued for championing class size reduction as the right thing to do for children and for teachers—an argument grounded in the moral content of public schooling more so than in the technical consequences of class size reduction for standardized test scores.
Over the past year, I’ve observed a number of trends in the operation of big-city school districts. I’ll use New York City as my key example, because it’s my hometown, but the issues are sufficiently general to warrant posting here.
First, large districts are increasingly trying out innovative policies and practices for which there is little or no pre-existing research support. In New York City, the issuing of school report cards and conduct of school quality reviews are high-stakes evaluative practices for which there’s no prior evidence showing beneficial outcomes. In Washington, DC and New York City, school officials are offering incentives in the form of cash and cellphones to students in exchange for meeting academic performance targets. Some of these innovations have evaluations built into their design, whereas others do not.
Second, the arguments in support of these innovations often rely on claims that other innovations have not been successful. The best example is the juxtaposition of teacher quality and class size reduction. All kinds of policies regarding teachers—value-added assessment, merit pay, new recruitment strategies—are being justified on the grounds that teacher quality has much larger consequences for student achievement (read: test scores) than other policy choices, such as class size reduction.
Third, a lot of the claims about these effects take the form of “Research shows…”, which eduwonkette has derided as glib and poorly documented. There are, of course, important studies of both teacher quality effects and class size effects on student outcomes, but different studies yield different estimates of the magnitude of these effects. In part, this is because the impact of a particular innovative policy or practice is contingent on how the policy or practice is implemented and the features of the local organizational and institutional context for the new intervention. (We might expect, for example, that class size reduction would have different effects in classrooms with novice teachers than in classrooms with experienced teachers, or in classes that differ in the amount of prior student misbehavior.)
So when a policymaker confidently says that we should prefer innovations designed to influence teacher quality rather than class size reduction in a particular local setting—say, New York City—what’s the evidence for such a claim? Specifically, what does research tell us about the consequences of a well-designed class size reduction intervention in New York City?
The answer is, we don’t know—because there has never been a carefully-controlled study of class size reduction in New York City.
So at this point, skoolboy throws down the gauntlet: If we’re serious about data-driven decision-making, we should put our money where our mouth is, and demonstrate the relative effectiveness of class-size reduction and other policy initiatives. I call on the New York City Department of Education to carry out a well-designed study—ideally, a randomized experiment—of class size reduction in New York City public schools. View it as a small-scale pilot, as is true for some of the other initiatives, such as the student incentive plans, and look for some private funding (if it’s not feasible to draw on the operating budget). It will not be hard to pull together some of the leading researchers on class size to inform the design (and it wouldn’t kill anybody to have a couple of knowledgeable parents and teachers at the table too.) There's nearly a full year to get this off the ground for the start of the 2009-10 school year.
skoolboy is willing to live with the findings of a well-designed and well-implemented study of class size reduction in New York City, whether they support or refute claims about the efficacy of class size reduction. What I cannot support are claims that “research shows” that teacher quality is more important than class size reduction for student outcomes in New York City—or any other local education setting—in the absence of research that actually does show this.
skoolboy was having a spirited discussion with some of his students the other night, who have taught in school systems such as New York City, Detroit, LA, New Orleans, Washington, DC, Newark, Oakland, and elsewhere. The topic of the day: what's the worst-managed big-city school system--and why? Readers, what do you think? Discuss.
skoolboy has been worrying about how he was going to make this week's COWabunga award. There haven't been any comments to his posts! Hard to believe that such witty and incisive remarks would draw nary a "well done!" or "you're full of it, skoolboy!" Turns out that the website woes that Ed Week has endured the past few days include a disabling of the comment features here. The good people at Ed Week are now aware of this, and I look forward to hearing what readers have to say when the problem is resolved. It's not the first time that technology has kicked skoolboy in the butt, and I'm sure it won't be the last.
If you can't wait for the site to get fixed to get something off your chest, feel free to e-mail me at skoolboy2 (at) gmail.com.
Yesterday, State Senator Rev. James Meeks engineered a boycott of the Chicago Public Schools, urging CPS students to travel with him to high-spending districts in Chicago’s suburban North Shore to try to register for school. The objective of the protest was to draw attention to inequalities in school funding in Illinois. Rev. Meeks sought to contrast the Chicago Public Schools, which annually spends a bit over $10,000 per student, with New Trier High School, which spends in the neighborhood of $18,000 per student. Publicity stunt, or principled protest?
Probably a bit of both, in skoolboy’s view. Illinois still relies heavily on local property taxes to fund its schools, and the variability in income and wealth across school districts means that different districts have differing capacities to raise money to support the schooling of the children who reside in them. State and federal funds are supposed to compensate for these inequalities, and they do, but not completely. The available evidence suggests that total per-pupil spending on students in the wealthiest 20% of school districts in Illinois is considerably higher than total per-pupil spending on students in the poorest 20% of school districts—a difference on the order of $2,500 per pupil per year.
The chart below shows these dynamics. skoolboy divided Illinois’ school districts into national deciles based on the median family income of the district in 2000. Districts with a median family income of $30,000 or lower were in the lowest decile, whereas those with a median family income of $66,000 or higher were in the highest decile. I looked at three different revenue streams: per-pupil local revenues; per-pupil state revenues; and per-pupil federal revenues. The sum of these three is reported as total per-pupil revenues. (I use revenues because they’re reported by source in the federal data, and expenditures are not. The data are also weighted by the number of students enrolled in each district, so smaller districts count less than larger ones. I also excluded districts in which the total per-pupil revenues exceeded $40,000 per year. The story is pretty much the same whether one looks at median family income or the percentage of children living in poverty within a district.)
You can see just how strongly district median income and local per-pupil revenues are correlated in Illinois (r=.68). It’s also clear that state and local funds flow disproportionately to lower-income districts. But when the three funding streams are added together, there is a moderate positive correlation (r=.38) between a district’s median family income and its total per-pupil revenue. Although federal and state revenues do help to close the gap between wealthy and poor districts in Illinois, the remaining inequalities in spending are not trivial.
Having said that, a comparison between the Chicago Public Schools and New Trier is fundamentally misleading. By skoolboy’s calculations, the average total per-pupil revenue in New Trier in 2006 was nearly $22,000, which is way, way above the average total per-pupil revenues for the 113 Illinois districts in the top national income decile ($11,400). Moreover, CPS is in the 5th median income decile, not one of the lowest, and its total per-pupil revenues are a tad above the average for the 87 Illinois districts in that decile.
Not all states show this pattern; some have been more successful in reducing the association between a school district’s total per-pupil spending and the characteristics of the students in that district. (For example, like Ken DeRosa, I also find no correlation between per-pupil revenues and the percentage of children in poverty among Pennsylvania school districts. However, in Pennsylvania, as in Illinois, districts with higher median family incomes do spend more than those with lower median family incomes. ) How schools are funded has a lot to do with the inequalities across districts, but funding formulae don’t change easily. Don't expect high-spending districts to be happy with policies that ask them either to spend less or to subsidize the spending on children in other districts.
Even though eduwonkette and skoolboy have been unmasked, skoolboy plans to continue to refer to himself in the third person. Why? If I did it at school, my students would laugh me out of the classroom. If I did it at home, my wife would kick my butt. So let me (er, skoolboy) have some fun, OK?
And for the record: both skoolboy and eduwonkette are lower case. Only proper nouns warrant capitalization, and it should be clear by now that skoolboy isn't very proper.
It’s back to school! Today, more than one million schoolchildren will get up from the breakfast table, strap on a backpack, and trundle off to … the living room. Home schooling has been expanding rapidly over the course of this decade, according to data from the National Center for Education Statistics, representing approximately 2.2% of the student population in 2003. (The NCES definition of home schooling is children who are schooled at home instead of in a public or private school for at least part of their education, and whose part-time enrollment in public or private schools does not exceed 25 hours per week.) skoolboy hoped to be able to report some new evidence from the Parent and Family Involvement (PFI) module of the National Household Education Survey’s 2007 sample, but those data have not yet been released. Unfortunately, that means that the best available information is from 2003, the prior wave gathering information on the incidence of home schooling. Moreover, only 239 homeschooled children were included in the PFI module of the 2003 NHES, and thus our knowledge about their characteristics isn’t very precise.
There are a lot of misconceptions about home schooling, such as homeschooled children lack normal social graces due to isolation from peers, or they’re all very well-prepared for college. skoolboy has seen no persuasive evidence of any problems of social adjustment among homeschooled children. The reality is that most homeschooled children and youth are not isolated from others; they often participate in homeschooling networks, may participate in extracurricular activities sponsored by public and private schools, and, for a significant fraction, are part of religious communities that provide opportunities for interaction with peers and adults. Homeschooled children and youth probably have fewer opportunities to interact with other youth with differing social characteristics than do students who attend public school; but you don't need to be a homeschooler to select yourself into settings where you engage almost exclusively with other people who are like you.
It’s challenging to assess the impact of home schooling on children who are home schooled, because families self-select into home schooling, and the kinds of families that choose to home school differ, on average, from those who do not. (And don’t hold your breath waiting for the definitive randomized experiment!) Homeschooled children are more likely to be white than Black or Hispanic; to be in a household with three or more children than one with fewer children; to live in a two-parent household with one parent in the labor force than in another configuration; and to have college-educated parents.
One of the most interesting features of home schooling, from skoolboy’s view, is its implications for defining teaching as a profession. For the most part, parents who home school their children are subject to very little oversight by the state. Contrast this with the rules for licensing teachers who teach in the public schools. Although eduwonkette pooh-poohs my “1950’s” thinking about what defines a profession in the sociological sense, I think she would agree that the fact that the state will allow parents with no formal training, and who are not accountable to other teachers for what they do, to teach weakens the case for teaching as a profession.
In February, a California appeals court held that parents can be prosecuted for failing to ensure that either (a) their children attend a full-time public or private day school, or (b) their children are instructed by a tutor who holds a state credential for the child’s grade level. The case alarmed home schoolers and their supporters across the country. On appeal, that same court ruled last month that “(1) California statutes permit home schooling as a species of private school education; and (2) the statutory permission to home school may constitutionally be overridden in order to protect the safety of a child who has been declared dependent.” The court made clear that it was not taking a stand on whether or not home schooling should be allowed, and blamed the California legislature for a lack of clear legislation on the issue. What counts as a threat to the safety of a dependent child is not inscribed in the law, but physical and sexual abuse (which were alleged in the California case) surely count; skoolboy’s guess is that mediocre instruction would not. (If it did, there’d be an awful lot of usual suspects to round up!)
