« Miscellany: NYC Teaching Fellows Conundrum, Absenteeism, & Dorn-onomics | Main | 2nd Annual Halloween Edu-Parade! »

Educational Malpractice? Why NYC School Progress Reports Deserve an F

Apologies for being AWOL, folks - both skoolboy and I are on the road. Below, you can find the beginning of an op-ed about the NYC Progress Reports that the two of us wrote for our local West Side Spirit - the link to the full text is below.
Each fall, many New Yorkers head to family physicians for an annual physical. Doctors record some standard measures—body temperature and blood pressure, for example—and perhaps draw some blood to send to the lab. Doctors will also ask about changes in health over the last year. Only after considering all of this information will they make a holistic assessment and recommend an appropriate treatment plan.

This fall, the New York City Department of Education is releasing its version of the annual check-up for schools: the School Progress Reports. September brought the reports for approximately 1,000 elementary, K-8 and middle schools, with the high schools coming shortly. The progress reports assigned these schools a letter grade ranging from A to F, based mostly (60 percent) on their contribution to students’ test scores from last year to this year. The progress report letter grades drive the “treatment plan” for the schools: schools which receive an A or a B are eligible for cash rewards, whereas those receiving a D or F face eventual restructuring or closure.

In doctors’ offices, we count on lab tests, X-rays and other reliable measures of our health. Can we count on the progress reports in the same way?

Our analyses of last year’s and this year’s progress reports suggest that we cannot.
Click here to read the rest.

There is an old joke about philosophers who argue endlessly about whether the King of France is wise. Some argue yes, he is wise; and others argue no, he is not. But they are all wrong because there is no King of France.

Much has been written about New York's Progress Reports: Do they really measure what students are learning in school? We now know that question is ridiculous. We now know that many children in New York do not attend school, at least not in the way most people think about attending school. About 20% of elementary school students missed 20 or more days of school last year. For high school students, 40% missed 20 days or more of school. Overwhelming numbers of students simply do not reliably attend school. The absenteeism issue raises a fundamental question that all the deep thinkers have overlooked: What can any statistical model say about the contribution of a school to students' academic achievement and/or growth if the students do not consistently attend the school? How can the king be wise if there is no king?

This is a fatal flaw in the NY accountability system. It has nothing to do with software systems, statistics, or politics. And no PR spin can correct for it. All of the New York data are meaningless in light of this issue. Merely counting attendance as 5% of an overall school score does not account for such significant absences. It is like an experiment in which you give the experimental group the drug but forget to give it to 25% of them. The effect of this "mistake" in the design is not easily corrected. To put it in testing terms, imagine if 25% of students had skipped 20% of the questions on the test for no reason you know. One out of four students has 20% random blanks throughout the test, as if they had never even seen the questions. Would you think those test scores are useful?

It is a catch-22. If missing weeks and weeks of school doesn't hurt your test score, does your score actually measure what you are learning in school? How could it? On the other hand, if missing weeks and weeks of school does hurt your test score, how could any measure of the school -- or a teacher -- be valid unless it adjusts for the individual attendance of each student? Do we really believe that these gross absences are distributed randomly, equally dispersed over every school and every classroom? Of course not. In fact, even two students who were both absent 20 days from the same classroom may be differentially impacted depending on when they missed class and what was taught those days. So, how could you possibly hold teachers accountable for the progress of students if, in one classroom, attendance is high and, in another classroom, one out of every four fifth grade students has missed 20 or more days of school?

As the city moves from school reports to teacher reports this fall, teachers should demand that data on the attendance of their students be printed in the reports so they have complete, fair documentation of their performance. Given all of the other disaggregated analyses the city is providing, teachers should demand that they have at least one comparison to teachers in the city whose students were absent an equal number of days. At the very least, teachers should be allowed to see a simple graph for the city for each test (grade, subject) that shows attendance in days on the X axis and student test scores on the Y axis. That would then immediately answer the question of how much one affects the other.

For years, we have had laws requiring "opportunity to learn." Under those laws, high stakes tests cannot be used as a graduation requirement for a student unless a state can document that all students have had a fair chance to learn the curriculum. Hopefully, sometime soon we will have a law based on "opportunity to teach" under which teachers cannot be held accountable for their work unless the state (or city) can prove that the teacher was given a fair chance to teach, including: students who attend class, reasonable class sizes, safe working conditions, and appropriate curricular materials. When the city allows an absenteeism rate as it currently stands, many teachers have been denied an "opportunity to teach." Judging them based on students they never had a chance to teach is obviously unfair.

It is also counterproductive. How many proponents of pay-for-performance would be willing to award cash prizes to teachers if they learned that the difference between the winners and losers was whether the students were actually in the classroom? Are they really paying the teacher for instructional excellence or rewarding the teacher for student attendance? Just like the teachers, proponents of those plans may want to ask about the attendance data before they take out their checkbooks. Certainly, if taxpayers are funding the system, we have a right to ask and a right to an answer.

Does attendance affect student test scores? If not, what are test scores measuring? If so, what are we measuring when we fail to measure that?

Very interesting comment, and one that deserves a post in response. It may take a few days for me to work one up.

As a first step, though: A critical issue is whether we treat student attendance as exogenous--that is, predetermined, or outside of the control of the school--or endogenous, i.e., determined by a student's experience in a school. Some absences are likely outside of a school's control, as when a parent decides to pull a child from school for a trip to Disneyworld. Others may result from a student feeling that the school doesn't care about him or her, which may be more subject to school influence.

Thanks for this posting (and your more in-depth article!)

Excellent point as I sat in massive confusion over my school getting an "A" on the report card. While one year we weren't meeting our growth goals, the next we were above target. How kids do on a standardized test (with only looking at the short-term measurement of growth from one year to the next) hardly looks at the fundamental growth (or lack thereof) that may be happening in the school.

While "test sophistication" also plays a strong role in altering test scores, I think more indicators need to be included (such as reported reading growth) in determining whether or not a school is "growing" enough or not.

Comments are now closed for this post.


Recent Comments

  • mr. s: Thanks for this posting (and your more in-depth article!) Excellent read more
  • skoolboy: Very interesting comment, and one that deserves a post in read more
  • The Absent from School Effect: There is an old joke about philosophers who argue endlessly read more




Technorati search

» Blogs that link here


8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
AERA annual meetings
AERA conference
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
picking a school
press office
principal bonuses
proficiency scores
push outs
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
small schools
small schools in New York City
social justice teaching
Sol Stern
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing and accountability
Texas accountability
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-added assessment
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School