« Quality Reviews and the Fetishization of Data: A Fantasy | Main | There Won't Be Blood »

Do Quality Reviews Lead to Increased Student Achievement?

skoolboy wraps up his posts on Quality Reviews. His first two posts can be found here and here.

Do quality reviews lead to increased student achievement? There’s been surprisingly little research that addresses this question. Most research on quality reviews has examined the school inspection process in Great Britain managed by the Office for Standards in Education (Ofsted), a national agency which reports to the Parliament. Since school inspections for primary and secondary schools were instituted in 1993, there have been several iterations in the school inspection process. But I haven’t found any persuasive evidence that inspections improve student achievement. Some teachers and administrators report that they intend to change their practices in response to the inspection report, but I’ve not seen studies which examine whether those intentions translate into improved practice.

You might get the impression from my postings this week that I think that quality reviews are a bad idea. Not necessarily! But there are some things that I think are essential for quality reviews to be a good idea. Here’s a brief list:

The purpose of the review must be clear. Sociologist Gary Natriello has written about four potential purposes for evaluations in schools: motivation, direction, certification and selection. The first two can contribute to school improvement, whereas the latter two are more concerned with regulation, accountability, and control; and it’s desirable to confront the tensions between improvement and control directly. If the purpose of a quality review is to improve how schools work, then all phases of the review process need to be oriented towards this purpose.

Definitions of quality must be clear and transparent. If there are clear criteria and standards for what constitutes school quality, then both educators and inspectors can orient their activities towards these criteria and standards. Unclear standards and definitions undermine the legitimacy of the quality review process. My impression is that the Ofsted criteria are a lot clearer than those that I’ve seen stateside. Quality teaching is a particularly challenging phenomenon to articulate; but if the goal is to improve teaching, we’ve got to be able to do it.

The quality review process must be designed to collect a sufficient amount of data on quality. If, for example, the purpose of the quality review is to improve teaching, then presumably there should be sustained collection of data on teaching quality, primarily through direct observation, but perhaps in other ways as well. Ms. Frizzle recently commented that in her New York City school, the quality reviewer was planning to observe 9 different classrooms in 30 minutes. Not much data on teaching quality will come from such a process. The intensity of data collection is a recurring challenge in evaluation research that involves site visits, because they are labor-intensive. “Drive-by” site-visits just aren’t very useful, even if conducted by well-trained observers, because they don’t gather enough data on the things that matter.

The frequency of quality reviews should be synchronized with a theory of how fast school quality is changing. This is Social Research 101: phenomena that change more quickly need to be measured more frequently to detect such changes, and phenomena that change more slowly don’t need to be measured as often. How frequently should we assess school quality? The school year is an arbitrary metric, and it may be wasteful and counterproductive to conduct school quality reviews on an annual basis. (In Great Britain, Ofsted inspects primary schools every three years.) Given a choice, I’d rather have less frequent, but more intensive, quality reviews.

Question: does accountability lead to increased student achievement? You point out that quality reviews can have multiple purposes. Clearly the quality reviews you've been describing are about accountability, not school improvement. They are designed to evaluate overall school performance, not individual teachers or programs. While I think it's hard to comprehensively evaluate a school based on a 1 day visit, in that time you can learn a lot, particularly about low-performing schools. For instance, in 30 minutes you can tell if there's chaos in the hallways and if kids are participating in learning activities. In 6 hours you can get a pretty good sense of whether instruction involves purposeful lessons, engages students in meaningful learning, and assesses for understanding. But note that this is still a broad brush and does not evaluate individual teachers. The purpose is to add qualitative data to the quantitative data to form a more nuanced picture of the overall school for the purpose of accountability. That said, the information obtained from this type of quality review can also contribute to school improvement, but that's generally left to the school leaders or districts. And you rightly point out that school improvement requires the ongoing collection of this type of data, but more importantly the effective use of that data to make regular adjustments to instruction, curriculum, professional development, etc. (at both the classroom and school levels).

Gideon is right. The school or district must actually choose to read and make changes based on an evaluation for it to be effective. I have seen both the good and the bad. One example is the "retired educator" who came in and rated an urban school "pretty good" (there must have been a context of "for an urban school" to have missed some major climate and achievement problems), although it was noted that the school's solution to kids pulling the fire alarm (a sign hung on it that said "Out of Order") was a bad idea. The same school was treated to a much more expensive (at district expense) outside evaluation that actually delved into climate (with some not too surprising findings that while the kids weren't terribly respectful it was hard to teach respect in an atmosphere in which teachers demonstrated disrespect), and curriculum delivery (again not surprisingly the overwhelming instructional method was teacher talk and work-sheet). While I think the out of order signs came down, as far as I can tell the other report was just accepted and filed.

Comments are now closed for this post.


Recent Comments

  • Margo/Mom: Gideon is right. The school or district must actually choose read more
  • Gideon: Question: does accountability lead to increased student achievement? You point read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School