Last week's teacher effects brouhaha brings me back to where this blog started - not eduwonk channeling Britney, but rather how to measure teacher effectiveness. We know a lot more about estimating teacher effects on student test scores than we did 10 years ago. (Readers know well that I am as concerned with academic and social outcomes of education that are not measured by test scores, but that is for another post.) Nonetheless, big picture questions linger, and Mary Lou Retton-worthy technical gymnastics won't make teachers feel comfortable with value-added until these questions are answered. Here's what I'd like to know before moving forward:

1) How do schools affect teachers' ability to be effective in the classroom? The current assumption about teacher effects is that they reside within the teacher - i.e that a teacher is "good" or "bad" independent of the school context in which s/he is working. But we don't know if a teacher is equally effective across multiple schools, or if some component of a teacher's effectiveness is "firm-specific." For example, Harvard health economist Robert Huckman has examined doctors' effectiveness across hospitals and found that human capital isn't entirely portable. (The Effect of Organizational Context on Individual Performance). Is this also true in education?

2) How, and how much, do colleagues matter? Having higher quality colleagues may make you a better teacher yourself. We need to know whether "teacher peer effects" exist, and if so, how important they are. (For more, see No Teacher is An Island). Colleagues matter in a second way in middle and high school, where kids have different teachers for different subjects. That your students have an exceptional English teacher makes it easier for your kids to write lab reports in science, and prior year teachers may matter as well. We need to know how these crossover effects operate, and how large they are.

3) Are the same teachers that are effective in promoting short-term score gains effective in promoting longer term academic growth?: We currently estimate teacher effects on what happens on a year-end test - but what we're really after is teachers' long-term effects on their students. We're not interested in short-term score inflation, but in improved learning that lasts. (See this New Yorker article about the trouble with hedge fund bonuses.) A new paper, "How Long Do Teacher Effects Persist?" by Spyros Konstantopoulos provides some insight here.

4) Are the same teachers that are good at promoting math skills good at promoting reading skills? Does being an "effective teacher" mean that you are good at one or good at both? Current estimates of the correlation between teachers' math and reading effects are in the neighborhood of .50-.60.

5) How large are student peer effects, and how does the existence of peer effects complicate our ability to estimate teacher effects? Classrooms are interactive organisms, not individuals sitting in separate cells. Teachers are well aware of this fact, and talk about classes from hell/heaven. Peer effects can be random - i.e. a couple of kids who chemically react and pull the class down with them - or socially patterned. For example, classes with a higher proportion of girls result in both girls and boys performing better (See More Girls=More Learning). How should our knowledge of peer effects in the classroom affect the way we model teacher effects?

6) What about non-random assignment? Non-random assignment may be the biggest threat to value-added systems. (See The Great Sorting Machine for more.) It's important from a technical perspective (see Do Value-Added Estimates Add Value?), but also from a legitimacy perspective. Teachers know that principals can bury them by sticking them with tough kids.

7) Are all gains created equal?: Should gains for high performers be treated differently than gains for low performers? In other words, should a gain of 10 scale score points for a high scoring kid be treated the same as a gain of 10 points for a low scoring kid?

Why do these big picture questions matter? Each has modeling implications. More importantly, they matter because teachers have these concerns about value-added estimates and they deserve to have their questions answered. From following the use of value-added in Dallas at the Dallas ISD Blog, it appears that few teachers actually understand how their CEI scores are calculated. Researchers and wonks interested in trying value-added need to do a better job of explaining these systems to teachers, of making them comprehensible, and of addressing concerns like those raised above.

My one line position on value-added? It's not ready.

Posted by eduwonkette at 2:05 PM | Permalink | Comments (5) | TrackBacks (0)

Though the Two Americas campaign is kaput, let’s talk about a new Maxwell School poll on Americans’ attitudes towards inequality. (Juju to Andrew Leigh for the link.) What continues to surprise me is how, despite a rising tide of inequality, a high proportion of Americans still believe that everyone has a fair shot.

Here are the key findings from the September 2007 survey: Though 67.4% of Americans think that we’re becoming a society of haves and have nots and only 33.4% believe that everyone in America has an opportunity to succeed, less than half (45.7%) see the current extent of income inequality in our society as a serious problem.

And most Americans still believe in the American Dream. When asked “Do you think what you achieve in life depends largely on your family background, or on your abilities and hard work?”, only 12.3% chose family background (32.6% said both). 85% agreed that, “While people may begin with different opportunities, hard work and perseverance can usually overcome those disadvantages."

These responses varied across the income spectrum, but not as much as one might think. 31.1% of those with family incomes under $50,000 said that “just some” Americans have opportunities," while 22.4% of those with family incomes over $100,000 did. 73.8% of those with family incomes under $50,000 said we are becoming a society of haves and have nots, while 59% of those making over $100,000 did.

What does this have to do with K-12 education? Back in November, I responded to Jim Horn’s post “Work Hard, Be Nice, and Other Lies My KIPP Teacher Told Me.” Basically, the question was whether educators should preach that hard work and effort yield success irrespective of one's racial or class background. Some observers have argued that they shouldn’t. For example, in Ain’t No Makin’ It, Jay McLeod wrote:

The familiar refrain of “Behave yourself, study hard, earn good grades, graduate with your class, go on to college, get a good job, and make a lot of money” reinforces the feelings of personal inadequacy and failure that working-class students are likely to bear as a matter of course. By this logic, those who have not made it have only themselves to blame. Because it shrouds class, race, and gender barriers to success, the achievement ideology promulgates a lie, one that some students come to recognize as such.

On the other hand, what does a rejection of the “achievement ideology” look like in the classroom? Readers, please chime in.

Want to read more about research on inequality? Check out a new magazine on poverty, inequality, and social policy called Pathways. Think Ed Next, but without editorial dogma about "the establishment." The first issue includes articles by Clinton, Obama, and Edwards on the "new War on Poverty," as well as short articles about income inequality by folks like Robert Frank, Charles Murray, and Tim Smeeding. There are also ed related articles on the gender gap and housing vouchers. You can sign up for a free subscription here.

Posted by eduwonkette at 8:38 PM | Permalink | Comments (7) | TrackBacks (0)

1) Prelude to a Post: A few weeks back, Robert Pondiscio, eduwonk, Charlie Barone, and I went at it about the impact of NCLB/accountability systems on curriculum and instruction. Now, Ms. Frizzle explains how NYC's Progress Report system is affecting her school (A Fairy Tale), and the Tempered Radical contemplates the costs of cutting out higher order thinking skills to up his scores (Tricks or Trash?). Stay tuned for my very tardy follow-up on this debate.

2) When Bad Clothes Happen to Good People: I noticed, too. Fashionista A-Rus talks wardrobe (Spellings, Jessica Alba, and the Polar Bear from the Golden Compass).

3) Superfreaky: What would Steve Levitt say to Nabokov over beer? See this follow-up to the Facebook literary SAT graph (What Do Lolita and Freakonomics Have In Common?).

4) Dunking Booth and Cotton Candy Not Included: The Carnival of Education is up at Creating Lifelong Learners.

Posted by eduwonkette at 5:54 PM | Permalink | Comments (0) | TrackBacks (0)

What's worse: evaluating college quality using standardized tests (Madame Secretary's pet project), or relying on Rate My Professors? At Rate My Professors, students rate their professors on "educational" qualities like their hotness, their easiness, their helpfulness, and their clarity. (Here's a nice Village Voice article about RMP; hat tip: Mike Arnzen). Now MTV has kicked off a spoof called "Professors Strike Back," in which profs respond to comments ranging from "I want to be her slave" to "Eats children for breakfast."

A mocking blog called Rate Your Students has emerged in response - you can read about some unbeloved students in this post (Head-Nodders, Laptop Kids, Winter Flip Floppers, and Some Nefarious Wannabe Gangsters. Where is that Walmart Application?).

Don't get me wrong - I'm all for the course evaluations that are typical at most campuses. Because everyone (who shows up) completes one, you have a full sample of students - not just the angry and elated - and narrative sections allow students to provide meaningful feedback on how to tweak the course in the future. Propositioning is generally not included, though students still throw in the occasional pediatric temper tantrum.

I'm undecided on whether and how colleges should make course evaluations public. On one hand, the public release of formal evaluations would help students decide among many courses. On the other hand, a student-driven evaluation system creates incentives to pander to Gen Facebook, and further encourages the "I'm paying, so I deserve an easy A" consumerism of many students.

So I'm on the fence about the role of course evaluations in assessing college teaching. Readers, what's your take? How should profs' teaching be evaluated?

Update: To clarify, there are at least 4 questions raised by this post:

1) How should learning be evaluated in college?

2) Are course evaluations a fair and comprehensive measure of college teaching? (Of course not, in my opinion.)

3) What should universities do with student course evaluations?

4) What are the potential risks/benefits to students and profs of making them public?

Posted by eduwonkette at 12:18 AM | Permalink | Comments (5) | TrackBacks (0)

Yo yo yo
Word up to Dan Brown
For showing how to break
A billionaire down

Shameful practice?
DOE, you're just like a cactus
Soaking up data but ya head is all dry
Read the research, yo
I'm telling you why

My boy Brian Jacob and his main man Lars
Wrote a paper saying you're down from Mars
For holding back kids when they're 14 years old
Check their results, J.K., then see if you're sold

Next time someone argues that rapping doesn't require talent and skill, direct them to this post. You'll handily win the argument.

Seriously, the Jacob and Lefgren paper, based on analysis of Chicago's similarly structured 8th grade retention program, found that Chicago's 8th grade retention policy increased the proportion of 8th grade retainees dropping out of high school. (However, Jacob and Lefgren found no effects of the 6th grade retention policy on students' likelihood of dropping out of high school.) It's a very thorough paper - take a look.

Posted by eduwonkette at 12:02 AM | Permalink | Comments (0) | TrackBacks (0)

Let's give it up for guest blogger Dan Brown, the author of the Bronx teacher memoir, “The Great Expectations School: A Rookie Year in the New Blackboard Jungle.” You can email him at danbrownteacher@gmail.com.

It’s a rough time to be a struggling student in New York City.

Mayor Bloomberg has now pledged to end the “shameful practice of social promotion” for eighth-grade students who fail either of their two state tests or any core classes. This means nearly 17,000 more eighth-graders than last year may be retained. For his tough position on boosting standards and student accountability, Bloomberg has received much praise.

But what about those kids who will be left back? Who are these socially promoted hangers-on that each year skate by with sub-par marks, undermining the achievement of a serious educational institution?

The answer is innocent kids who aren’t getting the help they need and don’t know how to demand it.

Students come to school with low academic skills for a variety of reasons. In New York, many are faultless victims of the ever-present crush of poverty and its far-reaching tentacles. The school system’s obsession with high-stakes testing— a game struggling students are poorly equipped to play— exacerbates their frustration. Their self-esteem levels are rock bottom and oppositional behavior often takes root. Can you blame them?

Blindly pushing struggling students forward (social promotion) is not the answer, but neither is holding them back for another lap around a failed track. Retaining low-achieving students does not improve their academic future; in fact it often does quite the opposite.

The struggling student conundrum can’t be solved with false choices like the ones offered in the social promotion political debate, but with serious assessments of the short-term and long-term needs of students.

The short-term answer for failing students is a major investment in remediation and individualized support. Clearly, the traditional classroom set-up isn’t working for these students.

The long-term solutions, ones that deal with the root issues of why kids fall behind early on, are more complicated, and more important, for the future of New York City. Students don't spontaneously combust in middle school. When students fail in eighth grade, something has been wrong for a long time. We need our mayor to address how those students can be rescued before hitting a seemingly irreversible frustration level.

Today's middle school students have lived with high-stakes testing in the No Child Left Behind era for virtually their entire scholastic lives. Mayor Bloomberg and Chancellor Klein have worked unrelentingly to conflate school accountability with test performance, a practice with myriad negative consequences. Rather than making it a priority for school to be a nurturing and personal experience, our system sees many kids denied preschool, packed into overcrowded classrooms, denied support services like fundamental skills tutoring, denied much-needed counseling, and supervised by administrators often more worried about test scores than their real needs. It's no wonder that some students eventually give up.

Many of Bloomberg and Klein’s school reforms are dynamic and exciting, but the ones that they have not yet made are essential. A more substantial up-front investment in supporting all students will pay manifold dividends.

Bloomberg is an expert of the business sphere, but bottom-line-driven business models are an ill fit for the education of young human beings. Focusing on holding struggling students back rather than intensively attending to their academic needs is tantamount to blaming the victims. Many socially promoted students have unwittingly suffered the collateral damage of suffocating poverty at home and a depersonalized, test-obsessed regime at school. It’s time they had some doors opened for them, not slammed in their faces.

Posted by eduwonkette at 12:18 AM | Permalink | Comments (0) | TrackBacks (0)

Irrespective of which candidate you're gunning for, who isn't happy that it's W's last State of the Union? Here's my retrospective of the Bush years via recent education posts:

1) Doomed to Repeat It: Chicago is intent on resurrecting a school closing strategy that didn't work the first time, says Mike Klonsky (More Shock and Awe in Chi-Town).

2) You Get What You Pay For: Robert Pondiscio at Core Knowledge quips about haggling over the price on student incentives, while the Quick and the Ed (If You Pay Them, They Will Pass) and Ed Sector (per USA Today) pleasantly surprise me - they aren't holding their breath for results. Neither am I.

3) Bourne Again: Ms. Frizzle weighs in on the NY Teacher Experiment (I've Always Resisted Conspiracy Theories). What if teachers were ranking principals instead? Successful Teaching has the 411 on what administrative support means. And what if principals were sizing up their higher ups? Diane Ravitch points out that they're too worried about retribution to be honest.

4) Plant Life as Weaponry: Kids throw stuff. Scenes from the Battleground takes us inside the War Room (Ammunition).

5) Ability Grouping (or Not?): Over at Authentic Education (a group blog including Grant Wiggins), educators are debating its role in schools.

Image credit: lewrockwell.com

Posted by eduwonkette at 6:17 PM | Permalink | Comments (0) | TrackBacks (0)

Now that the spring semester is in full swing, I’ve concluded that excuse writing deserves its own genre. College-level excuses are a) painfully specific and b) include details better kept to oneself.

Profgrrrrl’s recent post sparked a personal mission to track down the best excuse ever given. (Hat tip: Sherman Dorn.) There are excuse generating websites (if you need to squeeze out of a wedding or work, click here), but my colleagues can beat them all. Some candidates included missing the final exam because of a heroin overdose, having “totally THE WORST cramps ever,” winning last minute tickets to Spamalot, and getting stuck in an earthquake in Pennsylvania. But this one beats them all, and even ended up in the Chronicle last year:

I will be unable to be in class today because every year we have a Jell-O wrestling competition on campus, and it has just come to my attention that the 50 gallons of Jell-O that we previously made has spoiled. So now I have to remake the 50 gallons before 9 o'clock tonight. ... I understand this is a really weird circumstance, but without the Jell-O we have no competition, and without the competition we lose all of our fund-raising. Thanks you, and have a good weekend.

Image credit: trpaulsen.com

Posted by eduwonkette at 1:22 PM | Permalink | Comments (1) | TrackBacks (0)

Between Kanye West's role modeling (post here) and "I will steal your car," ED in '08 has struggled with commercials. As primary season gets into full swing, I figured they could use my pro bono help:

Wouldn’t you know it? One moment we’re on the road to reform, when suddenly it gets interrupted. And when I looked at our outcomes, I was embarrassed.

That’s why for guys like me with E.D., there’s E.D. in '08. E.D. in '08 is clinically proven to go to work fast and to have effects that last up to 10 months. The fact that E.D. in ’08 can work in the primaries is great, but having up to 10 months gives me the option to be ready once the moment is right. Unlike other solutions, E.D. in '08 frees me from counting on ed reform on their schedule.

What are you waiting for? Ask your doctor if E.D. in '08 is right for you. Don’t take E.D. in ’08 if you’re skeptical about the effectiveness of NCLB, performance pay, or mayoral control.

E.D. in '08: when the moment is right, you can be ready.

Posted by eduwonkette at 10:40 PM | Permalink | Comments (0) | TrackBacks (0)

I hate leftovers, too. But there is a lot left to say about last week's theme of data-driven decision making, so I'll tie up loose ends this week. Forthcoming posts include: How are data currently being used in schools, and who's entered the business of providing data solutions? What are some of the technical challenges with value-added models of teacher effectiveness? And what are their potential unintended consequences?

Posted by eduwonkette at 9:56 PM | Permalink | Comments (0) | TrackBacks (0)

Facebook detective Virgil Griffith has cooked up a clever graph plotting students' favorite books against their institutions' average SAT scores. The result is a cheeky (non-causal) cultural portrait of American college students. Books at the top of the SAT food chain include Lolita, One Hundred Years of Solitude, Crime and Punishment, and Freakonomics. Books hugging the bottom include the The Color Purple, Flyy Girl, Fahrenheit 451, and books by Zane. Here's what I want to know - will Lolita get an Amazon bump from crazed parent collegeseuirs?

Posted by eduwonkette at 5:14 PM | Permalink | Comments (3) | TrackBacks (0)

Freakonomics and Marginal Revolution face off on unintended consequences - it's timely food for thought about the potential consequences of adopting value-added as the primary measure of teacher effectiveness. As I've noted before, value-added as one of many measures works for me; value-added as the master measure - which I fear it would become - does not. Why? Teaching is a multifaceted task, and value-added measures use a simplistic evaluation rubric to monitor a complex task. Alex Tabarrok sums up the potential problem here:

The law of unintended consequences is what happens when a simple system tries to regulate a complex system. The political system is simple, it operates with limited information (rational ignorance), short time horizons, low feedback, and poor and misaligned incentives. Society in contrast is a complex, evolving, high-feedback, incentive-driven system. When a simple system tries to regulate a complex system you often get unintended consequences.

An unintended consequence of blogging is that I am about to miss a deadline, so I've got to bounce. Stay tuned for more value-added debate next week. Enjoy the weekend, everyone!

Posted by eduwonkette at 12:46 PM | Permalink | Comments (2) | TrackBacks (0)

skoolboy returns to weigh in on data-driven decision making:

I’m as much a fan of data as the next guy. But I worry that proponents of data-driven decision-making are understating just how hard it is to use data thoughtfully.

I’d like to describe the strategy championed by the New York City Department of Education, and point out the difficulties involved. The logic that the DOE is promoting is (a) use data to identify an area where a school is lagging, either in relation to some absolute standard or to other similar schools; (b) use the available data systems to identify similar schools that are doing better in this area; (c) ask these more effective schools what they are doing that accounts for their success; and (d) adapt their suggestions for use in the school.

It’s not as easy as it looks to determine which schools are doing better than others. Two different criteria are relevant: is the difference in performance between two schools large enough to matter, which is sometimes termed educational significance or practical significance; and is the difference in performance between two schools real, or could it just be due to chance, which is typically described as statistical significance. Ideally, we are interested in differences that are both practically and statistically significant. But a difference could be large, but not statistically significant (which is often the case when we have a small sample of information about performance), or statistically significant, but very small (in which we are pretty sure that the difference is real, but it’s just not very important). (Yes, statistical significance does matter!)

This is kind of abstract, so here’s an example, drawn from the NYC Department of Education’s Survey Access tool, which reports the results of the system’s first round of Learning Environment Surveys in the spring of 2007. The Department’s spiffy PowerPoint presentation imagines the principal and a group of teachers in (mythical) IS 402 identifying teacher engagement as an issue. In particular, teachers in this school generally disagreed that “Obtaining information from parents about student learning needs is a priority at my school.” Using the Survey Access tool, it’s possible to identify 12 similar NYC schools (i.e., middle schools with an enrollment over 700 and at least 25% ELL students), seven of which have more positive scores on this question. In the top school, the Eleanor Roosevelt School, 71% of the teachers strongly agreed or agreed with the statement, whereas in the bottom school, 13% of the teachers strongly agreed or agreed. (In mythical IS 402, 36% of the 31 teachers who responded to the survey strongly agreed or agreed.)

So why not just look at the seven schools above IS 402? Because the percentages of teachers strongly agreeing or agreeing is an estimate of the true percentage that would be observed if all teachers in the school responded to the survey. (In these 12 schools the teacher response rate ranged from 26% to 53%; in mythical IS 402, 40% of the teachers responded.) Our interest is in the population of teachers in the school, not just the sample that chose to respond. And there’s a degree of uncertainty in these estimates. If a different group of 31 teachers in IS 402 responded, just by chance, we might not have obtained an estimate of 36% strongly agreeing or agreeing. In fact, with a sample of 31 teachers responding and a sample estimate of 36%, the percentage of all of the teachers in IS 402 agreeing or strongly agreeing could plausibly range from 23% to 49%. (There’s a finite population correction in there, for those who care about such things.) That’s a pretty big range, and the range of possible values is pretty large for the other dozen schools as well.

Of the seven schools above IS 402, just one of them, the Eleanor Roosevelt School, is really head-and-shoulders above it in a statistical sense. The other six are statistically indistinguishable, because there’s so much overlap in the intervals in which the true percentage of all of the teachers strongly agreeing or agreeing in each school lies.

Would the principal and teachers in IS 402 learn something from asking the staff in these seven other schools how they do things? Sure! It doesn’t hurt to think about new ways of doing business. Will doing so raise performance in IS 402? Probably not. Because an assessment of statistical significance suggests that, with the exception of Eleanor Roosevelt, these other schools really aren’t doing better, and therefore there’s no reason to think that adopting their practices will yield genuine improvements.

Data-driven decision makers, beware of spurious comparisons.

Posted by eduwonkette at 6:15 PM | Permalink | Comments (8) | TrackBacks (0)

Over at the Ed Sector, there's some confusion about my concern with the ethics of the NYC teacher experiment (see here). To be clear, my problem is not that NYC is collecting value-added data. As I have written before, standardized tests have a role to play in teacher assessment alongside holistic evaluation of teachers' effectiveness. But as eduwonk himself noted, the methodological issues are hairy and as of yet unresolved.

The concern expressed in my earlier post was how this experiment was conducted in secret and, in my opinion, in violation of generally accepted human subjects policies. The entire enterprise of social science relies on potential study participants trusting researchers to minimize risks and fully disclose the purpose of their study. Every time a gaff like this happens, it undermines researchers' ability to build trust with study participants in the future. Let's review the chronology:

1) In September, an academic experiment headed by two very talented researchers, Jonah Rockoff (Columbia Business School) and Tom Kane (Harvard Grad School of Ed), was announced. It was presented as an experiment intended to generate academic knowledge, not to inform human resources decisions in real time. (You can watch a video of a study recruitment session here.)

2) Academic research is bound not only by common sense research ethics, but by the conventions of university Institutional Review Boards. What this means is that when academic researchers conduct research intended to produce generalizable knowledge - i.e. if researchers want to publish off of these data - the experiment has to proceed within generally accepted research ethics and a university IRB has to approve it. (Even if this was not an academic research project, the DOE should have notified teachers of an intervention of potential consequence for them. After all, the data are not just being collected, but distributed to principals in the experiment's treatment group.)

IRBs are primarily concerned with the harm that researchers could do to subjects by intervening in their lives, and applicants to IRBs must demonstrate that their project poses minimal risks, that participants have been notified of these risks, and that participants have consented to the research. Teachers did not need to consent in this case, as they are government employees and their employers can collect whatever data they want.

However, it is difficult for me to understand how one could justify not notifying teachers in the study. After all, the information given to their principal - which, given the ongoing methodological problems with value-added, may or may not be accurate - has the potential to permanently change their principals' perceptions of them and their future employment prospects. Moreover, this treatment is not being applied universally to NYC teachers. By simply having the bad luck to be selected into the study's treatment condition, some teachers are affected and others are not.

It is important to note that a "live experimental" study like this one is different from the secondary data analysis studies that eduwonk cites. He wrote:

By that logic, all these various studies with panel data, choice studies using lotteries, etc...all constitute human experimentation and are wrong.

Studies based on secondary data analysis are fundamentally different - and are treated differently by IRBs - because researchers are analyzing "dead" data that have no effect on real people's lives. Ongoing research projects in which interventions are made in real people's lives are held to a different standard. And should be.

3)According to Edwize and the NYT article, teachers were not notified of the study. What went wrong is that at some point this went from an academic study to a human resources project that Chris Cerf wants to take prime time. Perhaps he mispoke, or the NYT article had this wrong, but it appears that these data, collected under the auspices of an academic research study, may be used as early as June. As eduwonk noted, simply gathering the data is not a problem. The problem is that under the cover of "academic research," data are being given to princpals in ways that affect teachers' future employment without teachers' knowledge.

The irony, of course, is that none of this would be a big deal if the project had been announced to teachers. When I watched the recruitment session video back in September, it didn't seem like a big deal at all. I bookmarked that this was an interesting experiment conducted by two reseachers whose work is first rate, and assumed that the experiment would proceed under normal conditions (i.e. full disclosure of the study). For reasons I don't fully understand, it didn't. And here we are.

There's much more to say about the methodological and broader philsophical issues with value-added measures. I'll follow up with a post on these issues later.

Update: eduwonk and I continue our bridging differences exercise. He wrote:

Her position here would be a lot more compelling if (a) this were an actual experiment in the way she and other anti-Klein partisans are seeking to describe it rather than what it is. In addition --and again-- the fact is that we don't know what they are doing with the data so at this point all these leaps to various consequences are unfounded.

But we do know what they are doing with the information, at least in the context of this experiment (and, as I have explained above, it is an experiment). Principals in the treatment group are given value-added data reports on each of their teachers. These principals' perceptions of teachers' academic effectiveness are thus affected - correctly or incorrectly - by this information. Saying "principals can't use it" is like trying to strike evidence from the record in a courtroom. Jurors' perceptions are already influenced, and the damage is done.

Posted by eduwonkette at 1:08 AM | Permalink | Comments (3) | TrackBacks (0)

Helen Ladd, an economist at Duke, has turned in an exceptional commentary about rethinking the ways we hold schools accountable. Ladd has spent a decade studying the effects of North Carolina's accountability system. Here's an excerpt:

The bottom line is clear: Test-based accountability has not generated the significant gains in student achievement that proponents — however they perceived the problem to be solved—intended. Nor is the country on track to meet either the high proficiency standards required under the No Child Left Behind law or the equity goals suggested by its name.

As a reform strategy, test-based accountability falls short in at least three ways. First, it pays too little attention to the social factors that affect student achievement....Second, the approach pays too little attention to the broader system within which individual schools operate. Where is the accountability for state, county, or district officials who fail to provide the resources and support services needed to make the schools function better?....Third, test-based accountability tends to be punitive and pays too little attention to promoting effective process and practice within schools.

Ladd goes on to outline an alternative system - one that maintains realistic test score goals but incorporates inspection-based reviews of schools. This approach deserves serious consideration.

Posted by eduwonkette at 10:14 AM | Permalink | Comments (1) | TrackBacks (0)

In many ways, data-driven decision making (D3M) in education is an old idea packaged as a new one. As far back as anyone can remember, teachers have given their students regular quizzes, projects, and tests. When students performed poorly, "data-driven" teachers retaught the material or tried to figure out what went wrong. Without the benefit of spreadsheets or data displays, teachers have attempted to tailor their instruction to different groups of students. To be sure, there have been assumptions, blindspots, and kids overlooked, but the fundamental idea of teaching, assessing, figuring out what works for whom, and re-teaching is as old skool as Tupac.

What's new is the formalization of this process. Student learning is now quantified in test scores, stored in data warehouses, and made available for teachers to analyze. What's also new is the creation of two very different camps of data-driven reformers, which Sol Stern recently referred to as instructionists and incentivists.

Instructionists like the authors of the book DataWise see data as a useful tool for identifying problems of teaching practice, investigating them, and addressing them. Their focus is on improving student learning, not just test scores, and they are very clear about this distinction. They worry about the dangers of test score inflation and gaming the system, and advise schools to take steps to ensure that their improvements are not simply the result of shortcut practices that do not improve student learning.

Instructionists define data broadly - student work, student attitudes, and more are all relevant. In this view, data are not a replacement for expertise, and data don't make decisions. Rather, data are a useful tool for educators to harness to improve instruction. Instructionists stress that D3M requires a collaborative learning process, and are concerned about approaches that use data to blame individuals rather than support educators' professional growth. For example, Boudett, City, and Murnane wrote in Data Wise:

Agreeing on norms like "no blame" is an essential first step in creating an atmosphere that supports productive data discussions. It is important to emphasize from the beginning that data will not be used to punish teachers, but to help them figure out how to teach their students more effectively.

Incentivists' view of D3M is different. Perhaps best captured in adversarial approaches like CompStat (represented in The Wire as CityStat), D3M is a way to hold people's feet to the fire. This approach is agnostic about the "how" of fixing the numbers, and thus rewards better statistics with little attention to how these numbers were produced. As such, incentivists see little need to study diverse kinds of data - in fact, non-quantified data are dismissed as anecdotal. In this view, data, not educators, are the experts.

Unfortunately, D3M is being (has been?) hijacked by incentivists. The focus is on the numbers, not the process of arriving there, which invites all kinds of mischief. Most of the D3M that I've observed has involved schools figuring out how to cut corners in order to make test score ends meet. And to those who are ready to crucify the teachers for doing so, let me reiterate that it is unrealistic to expect schools to ignore pressure from city, state, and federal muck-a-mucks to rapidly improve test scores, not learning. As long as we only reward end game numbers and ignore the process through which schools get there, I predict that the instructionist approach to D3M will have a hard time getting off the ground. This is regrettable, as the ongoing, systematic analysis of multiple forms of data is an immensely promising strategy for improving teaching and learning.

Tomorrow I'll provide examples (good, bad, and ugly) of how teachers are currently using data. On Friday, I'll investigate the corporations that have entered the K-12 marketplace to provide "data solutions" for school districts.

Posted by eduwonkette at 12:08 AM | Permalink | Comments (2) | TrackBacks (0)

Tomorrow's Carnival of Education will be hosted at The Median Sib. Til then, check out some of these posts:

1) Bloomberg Betting and Bingo: Read NYC Educator's thoughts on a Bloomberg run, Dave Bellel's post on predicting Bloomberg's probability of running via Candidate Bingo, and a new blog called "Respite From Mike." If you're not electioned out by then, check out this interview with Ed in '08's Roy Romer.

2) Steven Covey for Principals: A Shrewdness of Apes provides life lessons for principals .

3) Laugh Out Loud: BellRingers, which I discovered last week and really like, has two funny posts about how having a sense of humor helps in the classroom (here and here). Bonus prize: Teacher adapted lyrics of Justin Timberlake's Sexyback.

4) New Teacher First Aid: Check out these podcasts directed at new teachers (courtesy of ABCTE), including an interview with New York's Teacher of the Year.

Posted by eduwonkette at 4:20 PM | Permalink | Comments (0) | TrackBacks (0)

The NY Times reported yesterday on an ongoing experiment on teacher effectiveness in NYC schools. Principals in the treatment group (140 schools) receive extensive value-added information on each teacher, and then are asked to evaluate the teachers. Principals in the control group do not receive these reports but also provide evaluations of their teachers. As far as I can tell, the goal is to determine how principals' evaluations are affected by having access to value-added data. By the summer, the NYC DOE will decide how these data will be used, and Deputy Chancellor Chris Cerf has even suggested releasing individual teachers' effectiveness data publicly. You can watch this video for more information about the experiment.

While much could be said about the challenges of estimating reliable value-added measures for teachers or the move to use test scores as the primary measure of assessing teacher effectiveness, I'll save those for later. (See more posts about measuring teacher effectiveness here.) Instead, I want to talk about the issue of research ethics in scientific experiments. It turns out that many teachers in participating schools have not been notified of the study.

Secret experiments have an odious history in science. The most notable example is the Tuskegee experiment, in which African-American men with syphilis were recruited into a study but not told of the purpose of the study or notified of their diagnosis. Their disease was left untreated so that researchers could track its progression. Once this experiment broke publicly, Congress passed legislation that, many commissions and administrative changes later, ultimately required universities receiving federal grants to form Institutional Review Boards to oversee all research. Human subjects policies require university researchers to receive the consent of all subjects and to make them aware of the potential risks of the study.

My point is not that the NYC experiment's secrecy is the moral equivalent of the Tuskegee Experiments. The Department of Education is not bound by any university's human subjects policy, and it is their right to examine whatever data they please to produce new knowledge. (Note that the university researchers involved are bound by IRB standards if they plan to publish off of these data.) But the Hippocratic Oath of the research community - that subjects should be aware that they are part of a study - has been grossly violated. And it does not help the reputation or future of "scientifically based research" in education when studies are conducted in secret. Even if this was not a research study, a decent boss notifies employees when they change the criteria on which employees are evaluated.

Where is this going next? Notably, Cerf's suggestion that individual teachers' data should be publicly released has precedent in New York. The New York State Department of Health started collecting similar data on doctors' effects on mortality in the early 1990s. In 1991, New York Newsday filed a Freedom of Information request, which forced the Department of Health to publicly release doctor level data. Since then, individual doctors' data have been publicly reported. Assuming the same Freedom of Information statutes apply to education, it may not be long before we can examine the "value-added scores" of NYC teachers while waiting for the C train to show up.

Back to data-driven decision making tomorrow.

Posted by eduwonkette at 9:47 AM | Permalink | Comments (10) | TrackBacks (0)

Tune in to the Brian Lehrer Show, which is hosting its annual MLK Tribute.

Posted by eduwonkette at 10:04 AM | Permalink | Comments (0) | TrackBacks (0)

Walk into any school's faculty meeting, and you'll think you've stumbled into a tongue twister competition. The push for data-driven decision making, DDDM, D3M - whatever you prefer to call it - is everywhere. This week I'll explore what data-driven decision making can and can't do for education and share some of the research on how data are currently used in schools.

Got something to share about how data are used in your school? Email me at eduwonkette (at) gmail (dot) com. In the meantime, you can check out some resources on DDDM over at Scott McLeod's site (the writer of Dangerously Irrelevant).

Wednesday: D3M Box Scores: Incentivists:10, Instructionists:1

Thursday: D3M Gone Wild: How Do We Know What Data To Trust to Inform Decision Making?

Posted by eduwonkette at 6:49 PM | Permalink | Comments (1) | TrackBacks (0)

Let me pile on to the eduwonk-Barone-Pondiscio debate. I'm no fan of the "NCLB: The Silent Killer" melodrama that blames the No Child Left Behind Act for all of our schools' problems, and there's obviously plenty of it to go around. This is what Charlie Barone and eduwonk reacted to yesterday when they pointed to a NYT article about college prep to argue that NCLB is not forcing schools to become drill and kill test-prep factories. (See eduwonk's post here.) Robert Pondiscio responded at Core Knowledge by providing an insider's view of currriculum narrowing and test prep. He concluded, "Dismiss it at your own peril."

I'm with Robert on this one. In my view, NCLB is creating very real problems by leading some schools to focus primarily on reading and math and to zero in on a small set of tested skills in these subjects at the expense of the full range of skills we want kids to have. I also think this response is too pervasive to ignore. While we can argue whether it "works" or not, it's happening.

The much blogged about Center on Education Policy Report (available here) released last summer found that 44% of districts had reduced time spent on social studies, science, arts and music, lunch and recess to fit in more time for reading and math. Comparing districts that had at least one school not making AYP with those who had none reveals starker contrasts: 51% of districts with at least one identified school decreased time in social studies, while 31% in districts with no identified schools did. (See Table 4 in the CEP report for more.) You can look at the numbers above in a glass half full way - it's not all schools, after all. To me, it's enough schools to cause concern. (It's also worth noting that district-based surveys probably understate how much narrowing there is in schools struggling with AYP.)

eduwonk and Barone are arguing that not all schools have responded to NCLB's incentives this way, so the problem isn't with NCLB. The underlying assumption is that good educators can resist these pressures. But eduwonk and Barone both support NCLB, I think, because they believe schools need incentives to improve. If you believe that incentives can have strong impacts on behavior, it doesn't make sense to argue that schools can (and should) just turn their backs on these incentives. Schools get no credit for teaching science and social studies, and schools that cut back on untested subjects and do lots of test prep are playing by NCLB's implicit rules.

There are a number of ways to address this issue that would seem acceptable to NCLB proponents - i.e. by "right sizing" the school day, as Paul Reville suggested, or testing all subjects, as the Center on Education Policy advised - but supporters of NCLB would do well to acknowledge and address the problem.

(Image credit: nataliedee.com)

Posted by eduwonkette at 11:11 AM | Permalink | Comments (4) | TrackBacks (0)

New York City posted the nomination narratives from its "Thank a Teacher" awards program. Here's the first one, about a physics teacher named Sidney Harris:

Mr. Harris’s expertise was in physics but what he taught me went far beyond science. He pushed me. He shaped the way I thought about my future. And he set expectations for me that were, before then, unimaginable.

What was his value-added on this kid's Physics Regents? We'll never know, but Mr. Harris' former student Joel Klein says: "I really believe I am chancellor today in no small measure because of Sidney Harris." Read a handful of these narratives and then ask yourself if we should evaluate teachers primarily based on their students' test scores.

Posted by eduwonkette at 12:00 AM | Permalink | Comments (0) | TrackBacks (0)

Spoiler alert: I'm going to write about data-driven decision making next week, so who better to profile than Kathryn Boudett, who teaches at the Harvard Grad School of Ed and is a co-author of the book Data Wise: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning. Note that the book is about improving teaching and learning, not just test scores! And that's why I like it. Here is a little snippet about the book, which I will say more about next week, and the syllabus for her course.

Posted by eduwonkette at 11:30 PM | Permalink | Comments (5) | TrackBacks (0)

Test your skills with this word problem:

A comprehensive high school in New York City has an enrollment of 900 9th graders. The NYC Department of Education decides to close the school and replace it with 5 new small schools, each of which will enroll 108 9th graders. How many 9th graders are left over?

Extra credit, Part I: Imagine that the NYC Dept of Ed closes 2 comprehensive schools in one year with enrollments identical to those above. Now how many 9th graders are left over?

Extra credit, Part II: Where will the displaced kids go to school?

If you've got your noggin on, you know that the answer is 360 kids, and that if we close two schools, we now have 720 displaced kids who need a place to go to 9th grade. This is, in part, the subject of Sam Freedman's NYT column yesterday. His column provides a hint on Extra Credit, Part II:

More broadly, the problem is the outcome of Department of Education decisions to open scores of small, niched schools in the area, close large ones perceived as academic failures and leave the excess students to land in traditional schools like Richmond Hill that, while relatively successful academically, were often overcrowded to begin with. In this version of education reform, it is never hard to tell the winners from the losers.

I know what you're thinking - doesn't anyone have a calculator? The NYC Department of Ed seems to have forgotten that matter is neither created nor destroyed in a chemical reaction - and the kids don't disappear, either.

Update: For more on high school reorganization hiccups, see the Gotham Gazette's Wonkster and this article in the Village Voice.

Posted by eduwonkette at 10:21 AM | Permalink | Comments (3) | TrackBacks (0)

"Welcome to the 154th Carnival of Education. Out on the red carpet, emotions are running high. Margaret Spellings took off her Sexy Librarian Glasses to wipe her brow. Rod Paige clung to Reg Weaver's arm to steady himself. And rumor has it that Bill Gates has even stopped talking about education and the election. I'll be hosting you this evening with my colleague - "

"Ryan Seacrest."

"Ryan Seacrest?! I asked for Stephen Colbert. Or Patrick Dempsey."

"Stephen has to write his own show now, eduwonkette. And McDreamy is all about solidarity. Let's get started with Number 40 on this week's billboard charts..."

"Wrong show, Ryan."

"Right. Who will be the next American Idol?"

"Um, Ryan, this is the Carnival Of Education. But it's just a press conference because of the writers' strike. So here's the list of winners:"

The "Barbara Walters Edu-Interviewing" Award:
* Robert Pondiscio at Core Knowledge, who asks the education cognoscenti to predict NCLB's future
* SharpBrains, for an interview with the author of "The Adolescent Brain" (terrifying specimen that it is)
* Alexander Russo, for a quirky interview in which Democrats for Ed Reform's Joe Williams confesses that math is hard for journalists
* The Line, for asking us to contribute our "burning questions" about education

The "Saturday Night Live Would Be Proud" Award (funny posts reflecting on classroom life):
* BellRingers, for a post on the perils of the yearbook's color printer when hormonal teenagers are involved
* Ms. Frizzle, for "That's My Pen! A Play in Three Acts"

The "Election: It's Not Just a Movie Starring Reese Witherspoon" Award:
* Seth Pearce at NYC Students, on Obamarama
* Scripted Spontaneity, for supporting K-12 teachers who want to get active in education reform debates
* Successful Teaching, for a post on how to teach the election now
* Campaign K-12, for reminding us that all (education) politics are local

The "Romper Room, No More!" Award (classroom management musings and tips):
* Scenes from the Battleground, for "Zen and the Art of Going to the Lavatory"
* The Elementary Educator, for his call to eliminate extrinsic rewards
* It's a Hardknock Teacher's Life, for asking her kids to ante up

The John Dewey Award (serious thoughts on big teaching issues):
*frogs-swans and swans-frogs, for reminding us that teaching from a script is as good as shellac
* A Shrewdness of Apes, for a post on high school kids (not) aging out of high school
* Principled Discovery, for revealing how colleges treat mentally ill students
* Teacher in a Strange Land, for loving Ellen Page and writing about how schools and society treat pregnant teens

The "Show Me the Money!" Award (posts on the role of money at various levels of the education system):
* NYC Educator, for noting that kids who get credit cards for the tee shirt need to know how to manage their finances
* Money Blue Book, for giving "My Super Sweet Sixteen" the roughing up it deserves
* College Information for Smart Students, for teaching us how to live large while spending little money (hint: follow your friends with meal plans into the dining hall)
* DoE, who wants to cut his kid in half to help with the district's budget cuts

The "I Heart New York" Award: (posts about NYC education policy and politics)
*Debbie Meier at Bridging Differences, for her post, "An Absurd Grading System and Lessons Unlearned"
* Edwize, for telling it like it is on NYC School Progress Reports
* Gary Babad at NYC Parents, for giving Eli Broad a new idea to fund (a Degree Program in High-Stakes Testing)
* Under Assault, for highlighting the UFT's internal politics

The "Skooling the Ed Schools" Award (ed schools and their discontents):
* Colossus of Rhodey, who thinks ed schools are awful
* I Thought a Think, who is shocked by ed schools' meager offerings on classroom management
* Going to the Mat, who examines the course requirements at Maryland ed schools
* EduDiva, for comparing medical and ed school approaches

The "Research This!" Award (posts on educational research):
* Sherman Dorn, who wrestles with economists on education and the economy
* Stuart Buck, who questions the findings of a new paper on school choice

The "Ripley's Believe It or Not" Award (about ridiculous things that happen in education):
* Education in Texas, for a post about a teacher denied an award because he refused to miss school to accept it
* Dangerously Irrelevant, for being too dangerous to make it through schools' software filters
* Friends of Dave, for a post quoting a parent who wants to stand on her car and throw rocks to protest budget cuts
* Mamacita, for a post on choosing valedictorians

The "Blogging + Teaching Cocktail" Award (how blogging can improve teaching):
* So You Want to Teach, for "8 Ways Blogging Makes Me a Better Teacher"
* The Tempered Radical, for reflecting on the blogging/teaching nexus and sharing a new blogging toy with us

The Writers' Strike Award (posts on teaching writing and reading):
* Teaching in the 408, for "Be the Molotov Cocktail" (strategies for getting kids into reading)
* Right Wing Nation, who wants to bring grammar back to writing education
* Larry Ferlazzo, for a list of the best websites for supporting writing instruction

The Blackboard Award (on teaching and learning):
* Joanne Jacobs, for a post on teaching the classics (or not)
* Nucleus Learning, for thoughts on gender and science (& bonus pics of cool science projects)
* In Practice, for a post asking if technology in the classroom improves learning
* My Ten Thousand Friends, for "Why the Hell Would I Want to Teach?"
* Creating Lifelong Learners, for thoughts about design and storytelling in film (& bonus pics of adorable kids)
* Circle Time, for finding meaning in teaching kids to blow bubbles
* Life Without School, for a post about unschooling while killing chickens

Thanks to all of the contributors and to the EdWonks for giving me the opportunity to host the Carnival!

Posted by eduwonkette at 2:19 AM | Permalink | Comments (13) | TrackBacks (1)

Earlier today, Diane Ravitch drew attention to American education's growing faceoff between non-educators and educators. She writes: How did American education fall so effortlessly into the control of Know Nothings from the world of business, law, and politics? Now, John Merrow releases a podcast with NYC's past and present Chancellors (moderated by Jay Mathews) that squarely hits on this philosophical divide. Some highlights:

*Joel sums up his job with a song: "Give a little, take a little, let your poor heart break a little…"

* Joel identifies leadership and attracting new and different teachers into teaching as his top two improvement initiatives. He questions whether the principal should primarily be an instructional leader. Rudy disagrees, saying, "the core of this business is how children learn."

* Joel and Rudy spar on the role of charter schools in urban ed reform.

Definitely worth a listen.

Posted by eduwonkette at 5:03 PM | Permalink | Comments (0) | TrackBacks (0)

On Sunday, the NYT wrote about admissions anxiety stemming from a larger than average senior class in Connecticut. It turns out this isn't just another case of helicopter parent mania - economists John Bound and Sarah Turner analyzed 50 years of data and found that the size of the cohort in a state actually does affect the percentage of students getting a BA (Paper here). After ruling out competing explanations for this outcome – for example, that larger cohorts are less prepared for college – Bound and Turner concluded that a 10% increase in the size of the college cohort within a state leads to a 4% decrease in the college completion rate within that state.

If there's a lesson here, perhaps it is that luck, and states' failure to fully adjust for these bulges, matter.

Posted by eduwonkette at 12:17 PM | Permalink | Comments (0) | TrackBacks (0)

Last week, Madame Secretary unveiled a shiny new toy called the "National Dashboard." I pooh-poohed it, saying that most of these data were already available in the National Center for Education Statistics' Common Core of Data or elsewhere. After checking it out (and seeing how pretty it is!), I like it. If you need a tidbit of data quickly, this is helpful, and most data consumers aren't going to take the time to navigate the Common Core. Score one for the Madame.

What does the dashboard include? Demographics, percentage of schools by state making AYP and in restructuring, NCLB funding, percentage of teachers that are highly qualified in low and high poverty schools, percentage of students proficient by subgroup on the state test and NAEP, graduation rates, percentage of students taking AP exams, and the percentage of students using tutoring and choice options. Check out your state at the link above.

Posted by eduwonkette at 10:48 AM | Permalink | Comments (1) | TrackBacks (0)

Edwize is pulling in a gaggle of guest bloggers to comment on the NYC Progress Reports - check out Sherman Dorn's post on "Bundling Accountability," Seth Pearce's post on "The Importance of the School Progress Debate," and my post, "The NYC Progress Report Catch-22".

Posted by eduwonkette at 6:12 PM | Permalink | Comments (0) | TrackBacks (0)

Number of Golden Globe nominations for “Juno”: 3
Percent of teenage girls who received abstinence-only sex education with no information on birth control (2002): 21%
Percent in 1995: 9%
Increase in teenage birth rate for 15-17 year olds in 2006: +3%
Percent of all births to unmarried mothers in 2006: 38.5
Extent to which Ellen Page got shafted at the Golden Globes: Majorly

Number of viewers of 4th Season’s opening "Grey’s Anatomy" episode: 20.5 million
Number of "Grey’s Anatomy" Golden Globe nominations: 1
Mortality rate due to homicide, white 15-17 year olds (per 100,000, 2002-04): 1.7
Mortality rate due to homicide, black 15-17 year olds (per 100,000, 2002-04): 20.7
Number of students in the US expelled for bringing a firearm to school (2002-2003): 2,143

Ratio of Golden Globe nominations to wins for “Atonement:” 7/1
Keira Knightly’s estimated weight in lbs (US Magazine): 100
Percent of non-poor 2-19 year olds overweight, (average:1988-1994): 8.4
Percent of non-poor 2-19 year olds overweight, (avg: 2001-2004): 14.9
Percent of poor 2-19 year olds overweight, (avg: 1988-1994): 12.6
Percent of poor 2-19 year olds overweight, (avg: 2001-2004): 17.9

Stats courtesy of NCES, CDC, and the Guttmacher Institute.

Posted by eduwonkette at 11:15 AM | Permalink | Comments (1) | TrackBacks (0)

Last week, A-Rus asked if "The Wire" is just "poverty porn." This week, Dave Bellel puts up a clip from last night's episode that hits the "no excuses" debate in less than 2 minutes. Is the Ed Trust or Richard Rothstein doing the ghost writing? You decide.

Posted by eduwonkette at 7:17 AM | Permalink | Comments (2) | TrackBacks (0)

eduwonkette is going theme-less this week - we'll have the Carnival of Education and a mishmosh of other posts. For those unfamiliar with the Carnival, every Wednesday, one blogger aggregates posts from around the edu-blogosphere. Send your posts to me at eduwonkette (at) gmail (dot) com by 7pm Eastern Time on Tuesday, January 15th. You can also use this submission form.

Posted by eduwonkette at 6:08 PM | Permalink | Comments (0) | TrackBacks (0)

Rewind to the conversation you overheard on Sunday, where I’m on the other end of the line with the mother of Madison (MOM), a friend on the brink of a school shopping meltdown. Here’s what I had to say:

1) The school Madison attends is not going to make or break her test scores. When it comes to academics, the more your home is like a high quality school (particularly in the early grades), the less the school matters. That means that if you own Baby Einstein and have oodles of books, maps, and science kits cluttering up the bottom shelf of your coffee table, you are probably worrying more than you need to.

MOM is sure I’ve got this wrong. I tell MOM that she should be more worried about which teachers Madison is assigned than which school attends. More importantly, given that Madison is in school for only 6 hours a day, MOM and I should be talking about what Madison’s going to be doing after school, on the weekend, and during the summer, since they’re going to make up the bulk of Madison’s K-12 life anyway.

2) MOM is down to five schools, and the differences between the schools she’s choosing – whether on teacher quality, approaches to instruction, and peers - are negligible. MOM is drawing strong contrasts, i.e. the school in Dangerous Minds versus the school in Gossip Girl. This masks a more nuanced reality. The schools Madison might attend are not that different in terms of academic quality. I remind MOM that very few parents are choosing between schools on either end of the continuum, though many of them think they are.

MOM brings up my previous post about class size, where I note that there are kindergarten classes in New York with 15 kids and others with 31. I remind her that the some of the schools she’s looking at have classes of 16, and others have classes of 18. (In cities, there are families who are choosing between a wide range of schools, but this is certainly not representative of the parents of America’s school age kids.)

3) MOM sees the Final Five as hierarchically ranked – i.e. School A is better than School B is better than School C – and I suggest that we think about choosing a school as a matching process rather than one where some people win admission to “the best school” and everyone else is screwed. Madison is a delightful kid who’s on the shy side, and probably would be most comfortable in a place that is attuned to the social dimensions of schooling. MOM concedes that all five schools are alike in this regard.

100 million parents aren’t wrong. But particularly in some circles, they are more stressed about schools than they need to be. I buy MOM, and myself, a copy of Judith Warner’s Perfect Madness: Motherhood in the Age of Anxiety . I’ll let you know how that book is soon.

Enjoy the weekend, everyone!

Posted by eduwonkette at 1:09 PM | Permalink | Comments (3) | TrackBacks (0)

From time to time, my colleague skoolboy will pop in and say a word. (You can check out his holiday posts about class size here):

Greene and Catherine Shock, writing in the Winter issue of City Journal, contend that ed schools care more about the political and social ends of education than basic academic skills. In a survey of U.S. News and World Report’s top 50 ed schools and 21 other flagship state universities, they examined course titles and descriptions in order to calculate a “multiculturalism-to-math ratio”—the ratio of courses that emphasize multiculturalism to those that focus on math. At the average education school, they contend, the multiculturalism-to-math ratio is 1.82, but at some schools, the ratios are much higher. At UCLA, for instance, 47 courses include the words “multiculturalism” or “diversity,” whereas only three contain the word “math,” for a ratio of almost 16 to 1.

skoolboy likes this kind of research, because he doesn’t have to leave the comfort of his office to figure out what students are expected to know before they are admitted to their degree programs; what courses they are required to take for their degrees; and what they actually take. It’s so much more convenient to look at course catalogs. I decided to do the same kind of analysis for Harvard Medical School, looking at the course offerings for the 2007-08 academic year. Did you know that there’s not a single course that mentions the word math?! But there are two that mention either diversity or social justice. Why, that’s a ratio of … hmm, I’m in an ed school, I guess I’m not sure. But I think it’s outrageous that the faculty of Harvard Medical School don’t care if their students know anything about math.

I decided to take a closer look at UCLA, which offers a Mathematics for Teaching B.S. degree. The preparation for the major requires seven courses in mathematics, and courses in physics, computing, and chemistry or biochemistry. The major itself requires 13 mathematics courses. (UCLA operates on a quarter system.) None of these courses is offered in UCLA’s Graduate School of Education and Information Sciences, but I don't think you can say that the school doesn't care about the mathematical preparation of its prospective math teachers.

The villains of Greene and Shock’s story are familiar: ed school professors accountable to no one but themselves, and blindly allegiant to multiculturalism and diversity; students shying away from math because it’s hard; and spineless accreditation bodies such as NCATE that care more about multiculturalism and diversity than subject matter teaching. Little wonder we’re getting our butts kicked by Slovakia in international assessments!

Too bad that the story is so distorted. One of NCATE’s constituent organizations is the National Council for Teachers of Mathematics, which has explicit standards for prospective math teachers’ content knowledge, field experiences, and mathematics teaching processes. Most states now regulate teacher preparation programs in ways that are intended to insure that teachers have adequate subject matter knowledge. And a little-known piece of legislation called No Child Left Behind has sought to promote this as well. We’re still some distance from agreement on how to discern teachers who know their subjects and who know how to enable students to master them; but no one’s proposing using course catalogs for this purpose.

Posted by eduwonkette at 9:21 AM | Permalink | Comments (3) | TrackBacks (0)

Some people protest war. Others protest hunger and suffering. Less discussed, but no less common, is a special class of protest reserved for parents: conscientious objection to their children’s troublesome friends. When parents look out into the world, they see peers whose values and attitudes are contagious. And they are notorious for circling the wagons to keep out unwanted intruders.

Which brings us back to the question of whether the school your kid attends matters as much as you think it does. On Monday and Tuesday, I pointed out that the differences between schools in improving test scores are actually quite small. However, I argued that schools do offer different kinds of opportunities to learn, and parents’ anxiety about where to send their kids to school is partially about their kids’ academic futures.

Amidst all our wonky talk about school choice and academic quality, it’s easy to forget that parents are acutely concerned about what kind of kids are going to be over for play dates. Most parents intuitively buy into the maxim, “You are the company you keep,” and believe that peers are going to affect the person their child turns out to be. Ignoring the emotional dimension of choosing schools leads us to a cookie cutter - and ultimately myopic - understanding of this process.

I’ll give you the bright side of parents’ worries, and then the dark side. Parents reason that their kids are going to spend most of their waking hours surrounded by their peers. They want their child to be flanked by kids who are well-behaved and respectful of the learning process. Parents would also prefer that the other parents at their school share their approach to parenting. For example, they’d like to know that their five year old isn’t watching "Showgirls" and hitting the bottle after they’ve dropped him off.

Parents also know, especially when the kids are young, that the parents of their kids’ friends are going to become their friends. So parents need to be able to see themselves in the other parents at the school. As Ryan, the husband of a San Francisco mom who’s blogging about her school search, said, “I liked the [parent] tour guides. I could see myself being their friends. We're going to be spending a lot of time at Alice's school, and we want to be in a place where we feel like we can connect with the other parents.”

Here’s the dark side. NCES Commissioner Mark Schneider tracked parents' use of a school search website in DC (DCschoolsearch.com), and documented which features of the schools parents looked at, and in what order. Guess what the heaviest hitter was? Demographics. Socioeconomic and racial composition play a large role when parents are choosing schools. (More on whether parents choose school quality or school racial/class composition; see also Mark Schneider's book - Charter Schools: Hope or Hype?).

Economists might argue that this is “statistical discrimination” – essentially using group averages about performance when we have insufficient information. Certainly, on average, learning conditions are worse at schools with high proportions of poor and minority kids. A less cheerful take, of course, is that this is animus-based discrimination.

Regardless of your interpretation, my point is that parents’ choices are as much about "the company" as they are about school quality.

Image credit: 4th grade, Linwood Elementary School.

Posted by eduwonkette at 12:28 PM | Permalink | Comments (4) | TrackBacks (0)

Madame Secretary's rolling up at the National Press Club for lunch today, vowing to take matters into her own hands on NCLB (USA Today article here). She's expected to chat about expanding growth models, differentiating sanctions, and requiring states to adopt a uniform definition of high school graduation. More details here.

I'm all for growth models. But growth models that don't ditch the 100% proficiency fantasy are not much of an improvement. Stay tuned for the GWG's talk.

Posted by eduwonkette at 9:49 AM | Permalink | Comments (2) | TrackBacks (0)

Cool people you should know #14! Russ Rumberger teaches at the University of California - Santa Barbara's School of Education. Not only does he have the most zany academic website I've ever visited (animation + music), he's done a lot of NCLB-relevant work on dropouts, English language learners, and student mobility.

Check out his recent study on a critical question - how does the high school a student attends affect her test scores and likelihood of dropping out or transferring? Despite the recent media frenzy about "dropout factories," Rumberger determined that after you control for student background characteristics, schools don't vary substantially in their dropout rates. However, there are big differences between schools in their transfer rates. Perhaps most relevant for NCLB, he found that, "Schools that are effective in promoting student learning (growth in achievement) are not necessarily effective in reducing dropout or transfer rates. " Rumberger advises that school performance should not be judged only by test scores, but by dropout and transfer rates as well. Multiple measures, anyone?

Rumberger and Greg Palardy's paper, "Test Scores, Dropout Rates, and Transfer Rates as Alternative Indicators of High School Performance," published in the American Educational Research Journal in 2005, is available at his website. California readers, you might also be interested in his study on the California dropout problem.

Posted by eduwonkette at 12:02 AM | Permalink | Comments (0) | TrackBacks (0)

We've now entered a P.F. (Post Freakonomics) age, and talk of incentives is everywhere. Education is no exception - there's rising interest in the idea of paying kids for upping their scores (more on this idea here). See the New Yorker's pithy take on incentives and the afterlife here:

Eternity lasts a very long time. Our resources, though “infinite,” are not unlimited....Focus groups have suggested that offering a mere year or two of heavenly bliss, coupled with the threat of a single hour spent bathing in hot pitch and being harassed by demons, would generate ninety-seven per cent of the current program’s salutary effect on mortal behavior. (Interestingly, eternity itself is now perceived as a disincentive by blessed souls with more than two years of college education.) This suggests that severely scaling back Our incentive plan—and its attendant costs—would not lead to a significant diminution in faithfulness, obedience, repentance, or other benefits accruing to Ourself.

Posted by eduwonkette at 12:57 PM | Permalink | Comments (0) | TrackBacks (0)

We've all played Six Degrees of Kevin Bacon. The idea that we are all more connected than we think took off after psychologist Stanley Milgram's small world experiment. In the late 1960s, Milgram sent a chain letter to residents in Omaha and Wichita. The challenge for participants was to return the letter to a designated person in Boston by handing it off to the fewest people possible. Channeling Disney, Milgram found that it's a small world after all. It turns out that it took an average of six exchanges to get the letters back to Boston.

What does this have to do with you? Once a week, I profile a public school teacher nominated by his or her peers for doing exceptional work. (See previous "cool teachers you should know" here.) To date, I've profiled teachers in Indiana, Massachusetts, New York, and New Jersey.

How many degrees of separation are there between you and a teacher in Iowa, New Hampshire, or South Carolina? You can find out by either directly nominating a teacher from one of these states or forwarding this message to someone most likely to know a great teacher there. To nominate a teacher, just email me at eduwonkette (at) gmail (dot) com. My hope is to hit all 50 states before the election.

Posted by eduwonkette at 12:16 AM | Permalink | Comments (0) | TrackBacks (0)

Today is NCLB’s 6th birthday. NCLB is, at its core, a policy predicated on the idea that schools vary widely in their ability to improve students’ test scores. By holding schools accountable, the hope is that “bad” schools will become more like “good” ones. (Note - this is a post about NCLB on NCLB's terms, so I'm going to focus on test scores. For more posts on NCLB, take a look here.

However, as I wrote yesterday, once we take into account students’ background characteristics, school effects on standardized test scores are pretty small. The good news is that teacher effects on test scores are quite large (you can find more posts on teacher effectiveness here). In short, the differences between teachers in improving test scores are much larger than the differences between schools. This finding has significant implications for the potential success of school-based efforts to improve test scores, as Barbara Nye, Spyros Konstantopoulos, and Larry Hedges wrote in their paper, “How Large Are Teacher Effects?”:

Many policies attempt to improve achievement by substituting one school for another (e.g. school choice) or changing the schools themselves (e.g. whole school reform). The rationale for these policies is based on the fact that there is variation in school effects. If teacher effects are larger than school effects, then policies focusing on teacher effects as a larger source of variation in achievement may be more promising than policies focusing on school effects.

(You can click to enlarge the picture above - courtesy of the Halloween Edu-Parade, Rod Paige is Armstrong Williams.)

Posted by eduwonkette at 11:43 AM | Permalink | Comments (1) | TrackBacks (0)

On NCLB's birthday, Diane Ravitch suggests that we're prancing around in our birthday suits (Grading Schools):

I find myself (once again) in the uncomfortable position of seeing ideas that I have supported as part of a broader set of reforms turn into unhealthy obsessions. I feel like someone who said that people should wear hats and then turned around to discover that people were talking about nothing else but their hats and walking around naked.

Deb, I think that one of the things that has occasionally drawn us together is that we both have a vision about education, what it might be, even when we disagree about this or that detail. Now I find that no one seems to talk about education anymore, just testing and accountability.

More on NCLB in a bit.

Posted by eduwonkette at 10:57 AM | Permalink | Comments (2) | TrackBacks (0)

I can't offer all you can eat shrimp or bottomless margaritas, but Ed Week has installed two handy new functions that you should check out. On the right hand bar, you'll find an RSS feed, as well as a gadget that delivers posts to your email. Enjoy.

P.S. - I am the last living person without an RSS reader (or, OMG - a Facebook page), so I am directing you to do things I don't totally understand. What I forgot to say: if you subscribed to the RSS feed before, can you *resubscribe*? Many thanks.

Posted by eduwonkette at 12:25 AM | Permalink | Comments (1) | TrackBacks (0)

In October, I awarded the first "Gold Star Book Award" to Mitchell Stevens' Creating a Class: College Admissions and the Education of Elites. (You can read more about the book here.)

In this week's Chronicle of Higher Education, Stevens turns in an incisive op-ed that serves as a powerful rejoinder to the absurdity bug that bit the Wall Street Journal last week:

By the time upper-middle-class 17-year-olds sit down to write their applications, most of the race to top institutions has already been run, and they already enjoy comfortable leads....For those kids, the big question is not whether they will be admitted to an elite institution, but which ones will offer them spots. Even while the fate of individual applicants at particular colleges remains uncertain until decision letters are mailed, the overall distribution of outcomes is heavily skewed in favor of affluent applicants. That is not the result of discrimination by admissions officers, but rather the consequence of privileged families deftly playing by the rules of the meritocratic game.

My research convinced me that the ever-more-frenzied activity surrounding selective admissions is essentially ceremonial — an elaborate national ritual of just desserts. The fact that the fates of particular applicants at particular colleges remain uncertain until the end enables us to believe that the winners earn their victories in a fair game. That is how the anxiety that attends the application season is deceptive: It encourages those who experience it to believe that the outcomes of the process are considerably more uncertain than they actually are.

Posted by eduwonkette at 7:28 PM | Permalink | Comments (0) | TrackBacks (0)

It's no "Juno," but this video of a Caroline Hoxby talk on charter schools in NYC is well worth watching. Hoxby discusses her evaluation of NYC charter schools, which compares students who win charter lotteries with those enter but don't win. The charter effect on math scores is .09 standard deviations, while the effect on reading scores is .04 standard deviations (for a year spent in a charter school). Of particular interest in her description of the programmatic differences between these schools, the most central of which is a longer school day and school year. More on this study coming soon.

Posted by eduwonkette at 2:59 PM | Permalink | Comments (0) | TrackBacks (0)

While you were watching the NFL playoffs this weekend, economists converged on New Orleans for the American Economic Association's annual meeting - think Mardi Gras, but without the fun.

For the early risers yesterday morning, there was a panel called "Student Incentives in Action: Experimental Evidence from Offering Money for Educational Achievement." Roland Fryer and Ceci Rouse, originally scheduled to present, were no shows, but there were three other papers presented: the first by Case Western's Eric Bettinger called, "Paying to Learn: The Effects of Financial Incentives on Elementary Test Scores" (policy brief available here), a second by Josh Angrist, Daniel Lang, and Philip Oreopoulos, "Lead Them To Water and Pay Them to Drink: An Experiment With Services and Incentives for College Achievement," and a third by Angrist and Victor Lavy called "The Effect of High-Stakes High School Achievement Rewards: Evidence from a Group Randomized Trial" (the latter two are available here).

Bettinger's study was based on a randomized experiment where students were paid for performance on periodic math, reading, writing, social studies, and science tests. These incentives increased test scores only in math, but not in any other subject. And the kids who gained the most from receiving the incentive were those already performing at higher levels, not the lowest performing students. Here's the kicker: The study was multi-year, such that some students were given incentives in one year and not in the next. Advocates of incentives argue that while students will react to the cash at first, when the incentive is taken away, they will learn "for learning's sake." Yet Bettinger found no carry over effects when the incentive was taken away, writing, "This may suggest that the existence of external motivation has a negative effect on the intrinsic desire to learn." What's worse, kids reverted back to their initial achievement level, suggesting that the incentives affected not permanent learning, but short-term effort.

Bettinger also shared two funny stories about how teachers used the cash as motivation: in one case, a teacher had the kids chant "Show me the money!" In another case, a teacher hung a giant $100 bill in her classroom. For writing practice, kids were asked to write about how they would spend their money.

In the other K-12 paper, Angrist and Lavy found that offering financial incentives to pass a high-stakes test in Israel improved outcomes for girls, but not boys. The effects on girls were largely driven by an increase in passing rates among those who had a relatively high chance of passing these exams to begin with.

Taken together, these studies don't bode well for the current drive to improve outcomes for the lowest performing students by paying them.

Posted by eduwonkette at 12:01 PM | Permalink | Comments (6) | TrackBacks (0)

Ask your companions at a dinner party about their elementary or high school, and you will learn that everyone has a theory about what made it “good” or “bad.” The amazing teachers. The decrepit building. The souped up science labs. The pungent cafeteria food. Unique extracurricular activities. The football team’s reign of terror. And the lists go on. When it comes to our schools, we all fashion ourselves as mini-experts. Most of us are convinced that some schools are better and others worse. And above all, we are certain that which school our kids attend matters.

What does it mean to say that schools matter, i.e. to claim that there are “school effects?” Essentially, this is a claim that, all else equal, going to one school versus another makes a substantial difference in a child’s outcomes. We all suspect that there are real quality differences between schools. But the trouble is that many studies find that differences between schools are dwarfed by differences within schools.

When the outcome in question is test scores, researchers have found that school effects are quite small. For example, once family characteristics are taken into account, private schools don’t come out ahead of public schools. (Catholic schools are a notable exception, though the most convincing studies find test score effects only on the students who are least likely to attend these schools.) Though city parents fight for their kids to get into selective elementary schools precisely because they are assumed to be “better schools,” economists Julie Cullen and Brian Jacob found that kids winning a kindergarten lottery to attend selective schools in Chicago don’t end up with higher test scores. (More details here.)

Does this mean that all schools are the same? Could 100 million parents be wrong? I don’t think so. These parents are only wrong if their sole goal is to pump up their kids’ test scores. But parents have a broad range of goals for their kids, and it’s not clear that test scores are the top priority. For example, Richard Rothstein and Rebecca Jacobsen found that parents, when asked to prioritize the goals of public schools, collectively value social skills and work ethic, citizenship and community responsibility, and emotional health more than the acquisition of basic academic skills. If we researchers took our heads out of the sand and studied the many goals of education, we might find that schools matter more than we think.

This is not to say that parents aren’t in it for academics – they are – but perhaps that parents see academic growth more broadly than the acquisition of test scores. Parents visit schools, and they discover that some kids are dissecting pig hearts, while others read out of textbook. They see that some schools require their students to write frequently in a variety of different styles, and provide their teachers with reasonable workloads that allow them to provide meaningful feedback. They notice that some schools offer art and music, while others have cut out these “extras.” And they know that some schools get their kids excited about learning, while others are passing out worksheets.

We’re so used to equating test scores with educational quality now that it’s easy to forget the big picture. Schools may not matter much for test scores. But that doesn’t mean that schools don’t matter.

Posted by eduwonkette at 9:32 AM | Permalink | Comments (3) | TrackBacks (0)

Every year, the parents of 55.1 million American schoolchildren fret about where to send their kids to school. They inspect potential schools from top to bottom. They wonder if the private school up the road might give their kid an edge. And they talk - to their friends, other parents, and their colleagues - to get the skinny on the local schools. In a bloggable age, parents' searches have been split open for all of us to watch - and even participate. (My favorite blog in the school shopping genre is the San Francisco K Files, written by a mom who's searching for a school for her daughter.)

This week's posts came out of conversations with my fertile friends about where to send their kids to preschool, how to buy into the best school district for their future spawn, or how to pick a good private high school. Here's a peek into these exchanges:

"Hey, eduwonkette. Thank God you're home. I went to tour this school, and - "

"The answer is that it probably doesn't matter much."

"WHAT?!?!? You study education! You do this all day! What do you mean that the school Madison goes to doesn't matter?"

"Look, researchers can't find much evidence that going to this or that school makes a big difference. Especially since the schools you're choosing between are pretty much identical. Did I tell you about the shoes I just won on ebay?"

"That's impossible."

"I know, right? They were mad cheap. And pink."

"I don't give a damn about your shoes. I need to find a school for my daughter. So seriously..."

Here's the puzzle. Parents are convinced that the right school will make all the difference for their child. Researchers are not so sure. What's going on here? Is this a story about parenting in an era of anxiety, or do researchers just have their heads in the sand?

On tap for this week:

Monday: Do Schools Matter?

Tuesday: Evidence on School Vs. Teacher Effects

Thursday: Parents' "You are the Company You Keep" Hypothesis

Friday: Why My Friends Still Like Me: My Answer to the "Where Do I Send My Kid?" Question

Posted by eduwonkette at 12:01 PM | Permalink | Comments (4) | TrackBacks (0)