August 2010 Archives

Value-added gauges of teacher effectiveness are highly error-prone and shouldn't make up more than a minimal part of a teacher's evaluation, contends a report released this week authored by a high-powered list of academics.

Problems with the gauge include the non-random assignment of students and teachers to classrooms, as well as the fact that value-added can't distinguish between the contributions of multiple teachers over time and seem to be unstable from year to year.

"Such scores should only be a part of an overall comprehensive evaluation," the authors wrote. "Some states are now considering plans that would give as much as 50 percent of the weight in teacher evaluation and compensation decisions to scores ... Based on the evidence, we consider this unwise."

The report comes as the latest salvo in the ever-increasing debate about the appropriate use of the value-added measures in making judgments about teachers' performance.

Many of the report's authors are measurement experts: Eva Baker, a co-director of the National Center for Evaluation Standards and Student Testing, at UCLA; Paul Barton, an associate director of the National Assessment of Educational Progress; Edward Haertel, a former president of the National Council of Measurement in Education; Helen Ladd, a professor at Duke University; Robert Linn and Lorrie Shepard, both professors at the University of Colorado; and Richard Shavelson, a former president of the American Educational Research Association, among others.

Some of these academics also contested the Race to the Top guidelines for similar reasons last year.

In this report, the researchers say even though value-added measures purport to take socioeconomic factors into account, other differences can skew the estimates—like inequitable access to health, special services, smaller classes and better resources. "Each of those resource differences may have a small impact on a teacher's apparent effectiveness, but cumulatively they have greater significance," the report states.

The report presages ill effects if value-added is made too much a part of evaluation systems. Among other things, the report argues such pressure would narrow the curriculum further; cause teachers to focus more heavily on areas that are likely to improve scores; and make it less likely that teachers would want to work with low-income students with lower test scores.

In interviews, other academics who work on value-added methods said that the report's caution about the misuse of value-added information is wise. Douglas Harris, a professor at the University of Wisconsin who's written extensively about value-added, said that it makes sense to use other measures, like classroom observations of teacher practice, to counterbalance or offset test scores so that teachers aren't unduly pressured to focusing exclusively on topics likely to appear on tests, or to instruct in a way that emphasizes factual recall over higher-order analysis.

But Harris also had some reservations about the report's conclusions.

"The only reason we can have this debate is that value-added gives us observations of lots of different students, and we can think of them being a sample, and have a confidence interval around the results," he said. "You can't even estimate the error with other measures [such as teacher observations], so we're holding value-added to a higher standard than the other approaches."

The study of the correlation between teacher observation measures and student achievement is still in its infancy. One of the few studies to make this link, based on Cincinnati's teacher-evaluation system, did find a difference in teacher effectiveness that was picked up by classroom scorers.

And since most uses of value-added have never been studied in depth, it's hard to say definitely what effects they'd have on teaching and learning, he added.

Daniel Goldhaber, a professor at the University of Washington, in Bothell, said that the report didn't note that there are few alternatives for measuring teachers' influence on instructional outcomes.

"I think people are right to point out the potential flaws of [value-added modeling], but it should be compared against what we have, not some nirvana that doesn't exist," he said.

But both researchers took pains to note they don't endorse the public reporting of teachers' value-added scores in Los Angeles as part of a controversial Los Angeles Times series.

Some of the states that have won Race to the Top grants, such as Rhode Island and Florida, have committed to basing at least 50 percent of a teacher's evaluation on student achievement. Those states will have to confront the problem that only a portion of teachers currently instruct in tested grades and subjects, however.

Posted by Stephen Sawchuk at 3:21 PM | Permalink | Leave a comment | Recommend

The National Staff Development Council, a membership group working to improve post-preparation professional development for teachers, will be rebranding itself beginning Sept. 1. Its new name: Learning Forward.

The name "represents that what we do today in schools affects lives far into the future," the group's deputy executive director, Joellen Killion, wrote in a release. "We will continue to serve our members with the same dedicated level of attention, provide leadership in the field of professional learning, develop resources and information to support educator learning, and advocate for policies and effective practices that ensure that professional learning leads to student achievement."

Whatever you think of NSDC's new name, perhaps you'll join me in breathing a sign of relief that it doesn't reference the term "job-embedded."

Posted by Stephen Sawchuk at 10:05 AM | Permalink | Leave a comment | Recommend

Most beginning teachers now appear to be receiving induction services, but teachers overall are spending less time in some kinds of sustained professional development activities than just a few years ago, according to a new analysis of federal data.

Released this morning by the National Staff Development Council, a membership group supportive of school-based teacher training, the report was penned by three researchers at the Stanford Center for Opportunity Policy in Education. It is the second of a three-part research study on professional development.

The study draws on data from the 2000, 2004, and 2008 administrations of the federal Schools and Staffing Survey, a nationally representative data set. As of 2008, the scholars found that 78 percent of beginning teachers report having had a mentor, though not always in the teacher's content area. That's a big leap from 71 percent of teachers in 2004 and just 62 percent in 2000.

"We seem to have broken through and come to an understanding of the importance of induction," Linda Darling-Hammond, one of the report's authors, said in an interview.

(Not all mentoring is of equal quality, although highly structured and intensive formats seem to produce more effective teachers, according to a recent Mathematica study.)

Even so, the report found that the intensity of other types of professional development decreased between 2004 and 2008. Training of at least nine to 16 hours on the use of computers for instruction, reading instruction, and student discipline all declined notably, while training of up to eight hours in those areas shot up. That could be a sign that teachers are back in the infamous and much-maligned one-shot workshops. (Time spent on P.D. in teachers' own content area improved slightly over this time period.)

That finding is particularly discouraging given two other studies on professional development. One analysis of nine rigorous scientific studies, released in 2007, found that professional development with fewer than 14 hours of training had no statistically significant effect on student achievement, in comparison to those with at least 30 hours.

And this study's first report found that teachers in high-performing countries generally spend less time on instruction and much more time each week meeting, planning, and constructing lessons with other teachers. Read more about it in this EdWeek write-up.

The new analysis also found that, almost a decade after the No Child Left Behind Act put an increased emphasis on special populations, only 42 percent of teachers reported having special-education-focused professional development, and 27 percent reported professional development for working with English-language learners.

"It's a disturbing thing," Darling-Hammond said. "With all the emphasis on high standards for all, there is still not nearly enough professional development going on for [working with] special education students and English-language learners. Teachers continue to say they need that."

The causes for the declines aren't specifically stated in the report, but Darling-Hammond contended that states have let their infrastructures for professional development dwindle of late.

Other notable findings from the report include the following:

• Participation in P.D. that was focused on aspects like technology, reading instruction, working with students with disabilities, and so forth varied widely from state to state. To account for this, the authors created an index of 11 indicators against which to judge state provision of P.D. Arkansas and Utah were the standouts. Darling-Hammond said that the third report generated from this study would delve more deeply into the structures at work in these states that seem to be driving these results—and that hopefully are also producing higher-quality P.D.

• On state-to-state variability, reading stands out in particular: 52 percent of teachers participated in reading P.D. in Oklahoma to 83 percent in Florida—a bit of a surprise given the $6 billion federal Reading First investment and its focus on practices grounded in "scientifically based reading research."

• The percentage of teachers who perceived a culture of "cooperative effort" in their schools dropped from 34 percent in 2000 to just 16 in 2008. But, the percentage of new teachers who said they had common planning time increased from 49 percent in 2004 to 56 percent in 2008. Per this apparent contradiction, the study postulates the mere provision of common planning time is not enough to support collaborative work among teachers.

The report has one drawback common to many studies that look at P.D. through a time or credit-hour-based lens: It lacks context on the type of activities gauged. Although the drop in overall hours spent in certain areas is suggestive, there's no way to tell whether the activities teachers engaged in reflected workshops, or the types that many educators say matter most: Lesson study, professional learning communities, inquiry teams, and content-based "coaching."

One of the reasons for that here concerns the 2008 SASS. For cost reasons, officials dropped some questions that dealt with the specific type of development offered ("job-embedded" vs. workshop) and the structures built in to support it (reimbursement, release time, etc.)

Darling-Hammond said she's lobbying the Institute of Education Sciences to restore them in the next SASS survey. She'll no doubt be joined by other scholars hoping to make inroads into this massive and still-understudied area.

Posted by Stephen Sawchuk at 1:00 AM | Permalink | Leave a comment | Recommend

The Phase 2 Race to the Top picks are here, and they're an interesting and eclectic bunch.

The effective teachers and leaders section of the competition represented the largest percentage of overall points in the competition, but based on a cursory review of the winners, there does not appear to be a clear thread among the teacher proposals that explains why some states won and others didn't.

A big theme has been incorporating student achievement into teacher evaluations. But despite the rumors, the idea of making student growth count for up to 50 percent of an evaluation does not seem to have been a major factor in determining the winners, after all. New York State officials, with union support, agreed to use 40 percent, of which only a portion will be based on standardized test scores and the remainder on local measures. But a bunch of the winning states—Massachusetts, Ohio, North Carolina—didn't specify a percentage in their teacher evaluations. Instead, they promise that student achievement will make up a "significant" percentage of the evaluation protocol. As I've noted time and time again, that term is in the eye of the beholder.

Two states that passed aggressive teacher laws, Colorado and Louisiana, came up empty-handed. In Colorado, that change cost the state the significant support of the Colorado Education Association. Just 5 percent of eligible unions signed on.

A few of the winning states had strong legal backing for their plans. Maryland's teacher plans, including a longer time-to-tenure bar and evaluation procedure are essentially codified in a bill and regulations; Rhode Island has based its changes in a series of regulations approved by its Board of Regents; and New York City set new details in legislation, including the provision that a teacher deemed ineffective for two years could be subject to an expedited dismissal hearing. In D.C., meanwhile, the district maintains the sole legal control over teacher evaluations.

But several of the winning states do not have as foolproof of a legal framework to back up their ambitious teacher plans. Florida is well known for its model Memorandum of Understanding, which codifies its 50-percent evaluation figure. But, some districts and unions appear ready to circumvent that, as Ron Matus, of the St. Petersburg Times, has ably reported.

Hawaii's teacher plans are contingent on a statewide letter of agreement with its teachers' union. In that letter, the parties agree to conduct evaluations annually and to incorporate student growth, but most of the details on how that will inform tenure and compensation haven't yet been worked out.

So what's the story behind these diverse results? Some folks, like the American Federation of Teachers' Randi Weingarten, are pointing to the stakeholder buy-in factor. Even then, that doesn't appear to have been foolproof: Only two union affiliates signed off in Maryland, for a total of 8 percent union buy-in.

Correction, 8/25/2010, 9:47a.m.: An earlier version of this blog post incorrectly stated that no National Education Association affiliate signed onto the state's Race to the Top bid. The Prince George's County Education Association, an NEA affiliate, did so.

In any case, lack of union support is a legitimate reason why peer reviewers may have downgraded these applications. But if that turns out to be the case, it would contradict U.S. Secretary of Education Arne Duncan's promise earlier this year that bold plans would trump buy-in.

More for you soon when we have the final score sheets and reviewers' comments.

Posted by Stephen Sawchuk at 4:40 PM | Permalink | Leave a comment | Recommend

The president of the American Federation of Teachers, Randi Weingarten, has weighed in on the Los Angeles Times' value-added project. She agrees that parents, teachers, and principals have the right to access this kind of performance data, but argues making such data public to the public at large isn't appropriate.

In some ways, the L.A. Times series is really a litmus test about data, its usage, and who gets to access it. And it's unclear whether other journalists or districts are going to follow the paper's lead and try to obtain these data or to make them public. In the past, getting access to such information hasn't been easy: Researchers have long complained about blocked access to value-added data, even without teacher names attached to them.

Another issue that occurs to me is that the push to incorporate these data into personnel evaluations is likely to help define who is and who isn't eligible to access such data. Evaluations of public employees are often protected by state or district policy. So if value-added measures become a formal part of teacher evaluations, as is the case in several states' Race to the Top proposals, that change would probably limit the accessibility of such data to a few groups.

UPDATE, 11:11 p.m.: John Fensterwald makes the same point at The Educated Guess.

It isn't an issue right now in Los Angeles, but it could be in the future: The Times reports that district officials want to formally make value-added measures part of teacher evaluations in an upcoming contract with United Teachers Los Angeles.

Posted by Stephen Sawchuk at 12:46 PM | Permalink | Leave a comment | Recommend

Not too long ago, I promised you more analysis on the action at the two national teachers' unions conventions. I just filed a story for Education Week on this topic. The punchline: the unions' different governance structures have a lot to do with the paths they've pursued in these areas. You can read the story here.

(For daily coverage from the conventions, click on the archives for July 2010; entries begin July 2.)

Posted by Stephen Sawchuk at 5:44 PM | Permalink | Leave a comment | Recommend

Washington state pays its math and science teachers, on average, lower salaries than other teachers, in what appears to undercut plans by that state's leaders to invest more heavily in the quality of math and science teachers, a new analysis concludes.

The analysis, released today by the Center for Reinventing Public Education, faults the structure of the salary schedule for the disparity. It hypothesizes that math and science teachers may be leaving the teaching profession for other, more highly paid opportunities, thus resulting in a less-experienced and less highly paid cadre of educators in those subjects.

Like most salary schedules, Washington's rewards teachers for longevity and credentials, but not for other factors.

Researchers Jim Simpkins, Marguerite Roza, and Cristina Sepe analyzed data from the state's 30 largest districts, from 122 high schools in which more than 7,150 teachers worked.

They discovered that in 19 of the 30 districts studied, the average base pay of math and science teachers lagged that of their peers. And in 21 of the 30 districts, math and science teachers' years of experience fell behind their peers'. That's problematic because research has shown that teachers typically grow in effectiveness their first five years in the classroom.

"Washington state, despite its earnest commitment to high school math and science teaching, actually ends up spending less per teacher in the two subject areas it wants to emphasize," the authors write. "If a salary schedule instead tied wages to some measure of labor market value, ... we might expect to find that math and science teachers routinely ended up with higher pay than their peers."

The paper doesn't deal with other states, but it's logical to assume that similar patterns could sprout up in other locations that don't offer premiums for high-need fields.

Meanwhile, the issue of subject-specific pay represents one of the big differences in the way the two national teachers' unions approach compensation issues. The policy of the National Education Association—the parent of the Washington state affiliate—eschews higher pay for math and science teachers (though some affiliates have bargained it). The American Federation of Teachers, meanwhile, remains open to the practice.

Posted by Stephen Sawchuk at 12:11 PM | Permalink | Leave a comment | Recommend

Reporters for my hometown paper, the Los Angeles Times, just published a fascinating, potentially explosive story based on the use of teacher "effect" data that indicates which teachers seem to be producing the strongest gains for their students.

Using value-added student test-score-growth data gathered from a public-records request, the newspaper created a database with information about how well teachers in tested grades and subjects fare in terms of boosting student achievement. It will even release data on individual teachers' growth scores because, the paper writes, "they bear on the performance of public employees who provide an important public service, and in the belief that parents and the public have a right to the information."

The district had the ability to generate such information but never pursued it, partly fearing the reaction from the local teachers' union.

Some of the analyses' early findings stand in contrast to the conventional wisdom about teacher distribution. For example, they find that the most- and least-effective teachers are not actually concentrated in the most- or least-affluent schools. That's counter to most other measures of teacher quality; for instance, low-income students tend to be assigned more out-of-field teachers.

Meanwhile, there's been precious little information on the distribution of effective, rather than qualified, teachers. (Mathematica Policy Research, as part of the federally funded Talent Transfer Initiative, is releasing an analysis on this topic sometime in the near future. See a recent story of mine on this issue of "equitable teacher distribution" for details.)

Other surprising findings: Qualifications had few effects overall on effectiveness, in contrast to other studies suggesting that "bundles" of certain qualifications might have an impact. And class size was unrelated to effectiveness as well.

United Teachers Los Angeles is, let's just say, really, REALLY peeved about this. It's fighting back, asking members to send letters to the editor to protest the story, and I've even heard it wants members to boycott the paper and cancel subscriptions. The union argues that test scores aren't an appropriate measure of student learning and are even worse for judging teacher effectiveness, citing problems with the estimates.

Frankly, though, I wonder whether UTLA's protests will do much to change things now that the genie is out of the bottle. Parents, teachers, and students are likely to make full use of this database, and the consequences will be interesting to watch.

Way back in 2008, when I first wrote about value-added data for EdWeek—presciently, it seems— the president of the National Council on Teacher Quality told me she felt that the data, when unveiled, would create a demand for more such information. "If the public were given half a chance to learn about the power of value-added-data systems, I doubt they'd remain unengaged and tolerate depriving schools and teachers themselves of the data," she said. She may be about to be proved correct.

Posted by Stephen Sawchuk at 9:30 AM | Permalink | Leave a comment | Recommend

State lawmakers and district officials should revise local collective bargaining contracts and state laws so that they support high-quality professional development, urges a report released recently by four groups.

The National Staff Development Council, the National Education Association, the American Federation of Teachers, and the Council of Chief State School Officers banded together to produce the report. It incorporates the work of six state teams charged with examining local collective bargaining language and state codes that shape professional development.

The report covers a wide swath of topics, including whether the states have standards for professional development; specify budgetary policies for professional development; compensate teachers for professional development; and provide for teacher-designed PD and collaboration. It also discusses how they address a few specific types of training, such as the PD required for relicensure, induction, national-board certification, and career ladders.

The examples they discuss in the report aren't perfect, the authors write, but are at least a jumping-off point for further discussion.

In all, the report's authors deem the current scope of policies disappointing. "From our study in these six states, it seems that professional learning does not have a significant place in policy and collective bargaining language," they write.

One of the big problems: a lot of these examples are still oriented toward a model of PD based on seat time and credit hours, rather than focusing on whether the training involves rigorous activities measured for their effect on student learning. What's more, the policies typically don't even specify parameters for the content of the professional development.

And school schedules often don't support collaboration among educators. Not one of the six states had a district with a formal policy about the effective use of time set aside for PD. For example, in North Carolina, the state school calendar sets aside eight mandatory days for staff development, but teachers can take annual leave for seven of those days.

"The harder, more productive challenge is to reorganize the school day to allow teachers to have time for collaborative professional learning, thereby benefiting all students," the authors write. (For an example of what this might look like, check out what the Brooklyn Generation school is doing in New York City.)

I"ll post a link to the report as soon as I have it, and once you've digested the report, please write in here with your thoughts. To get the dialogue started, here are a couple of things that occurred to me as a I read the report:

Specificity: What is the proper grain size for these policies? Specificity has the advantage of laying out clear expectations, but also risks adverse consequences. Take California's Beginning Teacher Support and Assessment induction program and Florida's professional-learning-community requirements, for instance: Over the years I've heard from some teachers who have found these programs helpful and from other teachers who say the initiatives have generated more paperwork than results.

Implementation: Even a clearly written, thoughtful policy is just that—a policy. What really matters is how well it is put into place. If you know of a district that does a particularly good job translating one of these pathways into effective PD, why not write in and tell us where it is and why you think it's exemplary?

Cost: When you add up all these different forms of development—induction, relicensure, release time for teachers to collaborate, extra compensation for engaging in PD, salaries for teacher coaches, and so on, you're talking significant costs—a topic the report glosses over. What does all this actually get us in terms of student learning? That's a hard question to answer, because the research literature is pretty thin overall on this question.

EdWeek reporters, including yours truly, will dig into a number of these issues in coming stories, so stay tuned. We'll have more for you soon.

Posted by Stephen Sawchuk at 9:00 AM | Permalink | Leave a comment | Recommend

Fully 24 percent of the i3 winners plan to use their grants to help improve teacher effectiveness. You may have heard of some of the big-ticket winners, like Teach For America, which will use its $50 million grant to grow the size of its teacher corps by 80 percent by 2014.

But here's a rundown of some of the other winning teacher plans:

• The American Federation of Teachers' own Innovation Fund will use its grant to scale up one of its projects, a plan to pilot and implement a new teacher-evaluation and -development system in select New York and Rhode Island affiliates.

• The Boston Teacher Residency will expand its program to prepare and then place a total of 130 teacher residents in "turnaround" schools in Boston.

• Jefferson County schools, in Louisville, Ky., will move to a trimester system that, among other things, will allow teachers more time to collaborate on professional development.

• The Exploratorium in San Francisco will devise a professional-development program that integrates English-language development and science instruction to be used in the district's elementary schools.

• The Miami-Dade school district and several project partners will create an early-learning specialization within the district's on-the-job master's degree program, and the school system will also expand a "master teacher" program in which select teachers lead professional learning communities to additional grades and subjects.

• The Chicago-based Erickson Institute will establish a whole-school professional-development program for math teachers of pupils in K-3 classrooms in Chicago.

• The New Teacher Project will set up an effectiveness-based teacher-preparation program, using a student-achievement- focused curriculum, for prospective teachers in the District of Columbia; Chicago; Fort Worth, Texas; and Nashville, Tenn.; among other sites. It plans to prepare up to 2,400 new teachers by 2015 and certify only those teachers who have demonstrated classroom effectiveness.

Posted by Stephen Sawchuk at 11:52 AM | Permalink | Leave a comment | Recommend

The American Federation of Teachers has joined several other groups in filing a class action today on behalf of 350 Filipino teachers who were allegedly subjected to usurious fees, substandard housing, threats of deportation, and other harassment by a teacher-recruitment firm, USA Today reports.

The teachers came to Louisiana on H-1 guest-worker visas arranged by Universal Placement International, a Los-Angeles based company. The lawsuit is the culmination of AFT's investigations into the company's practices.

AFT, the Southern Poverty Law Center, and the firm of Covington & Burling filed the suit together. UPI's owner, Lourdes Navarro, and two other employees of the recruitment company face charges of racketeering and human trafficking, the newspaper reported.

Much of the coverage of the teachers' unions here at Teacher Beat and in news media in general has focused of late on how they engage in issues of policy. This lawsuit serves as an important reminder that unions spend a lot of time on workplace issues like discrimination and harassment.

Posted by Stephen Sawchuk at 11:03 AM | Permalink | Leave a comment | Recommend

The Chicago Teachers' Union is suing to prevent the school district from laying off hundreds of classroom teachers and instructional "coaches," the Associated Press reports.

The district is trying to close a huge funding shortfall, but the union says that the district violated the teachers' due process and constitutional rights in firing about 240 teacher coaches and sending pink slips to 600 classroom teachers and other support personnel.

However, one of the interesting wrinkles is that about a third of the pink-slipped teachers had an unsatisfactory evaluation rating and were identified for layoffs for that reason, rather than under reverse-seniority rules.

It remains an open question whether the district's attempt to terminate poorly rated teachers before others passes legal muster.

Posted by Stephen Sawchuk at 12:37 PM | Permalink | Leave a comment | Recommend

A group convened by the Council of Chief State School Officers recently released a draft of professional teaching standards that outline what practices, essential knowledge, and dispositions teachers should embody to help students succeed.

Developed by the CCSSO's Interstate Teacher Assessment and Support Consortium, the new set of standards contains some notable differences from the prior set, released in 1992. For one, the standards are no longer limited to beginning teachers; instead, they're meant to guide teachers at all levels of their career, with more-experienced teachers exhibiting the practices in more-sophisticated ways.

In addition, the standards put more emphasis on teachers' ability to use assessment data to support instruction, to address cultural and linguistic diversity in the student population, and to harness technology as a tool to support learning.

The authors also wove interdisciplinary themes of communication, collaboration, critical thinking, and technology use—sometimes called the 21st-century skills—throughout the standards.

Of course, as with the common academic-content standards movement, there's a huge difference between adopting standards and actually creating the infrastructure to translate them into action. Will states and districts align induction supports, evaluation systems, and professional development to these guidelines?

The standards are open for public commentary until Oct. 15. Feel free to post your thoughts on the InTASC standards here, too.

Posted by Stephen Sawchuk at 11:23 AM | Permalink | Leave a comment | Recommend