Apologies for being AWOL, folks - both skoolboy and I are on the road. Below, you can find the beginning of an op-ed about the NYC Progress Reports that the two of us wrote for our local West Side Spirit - the link to the full text is below.
Each fall, many New Yorkers head to family physicians for an annual physical. Doctors record some standard measures—body temperature and blood pressure, for example—and perhaps draw some blood to send to the lab. Doctors will also ask about changes in health over the last year. Only after considering all of this information will they make a holistic assessment and recommend an appropriate treatment plan.Click here to read the rest.
This fall, the New York City Department of Education is releasing its version of the annual check-up for schools: the School Progress Reports. September brought the reports for approximately 1,000 elementary, K-8 and middle schools, with the high schools coming shortly. The progress reports assigned these schools a letter grade ranging from A to F, based mostly (60 percent) on their contribution to students’ test scores from last year to this year. The progress report letter grades drive the “treatment plan” for the schools: schools which receive an A or a B are eligible for cash rewards, whereas those receiving a D or F face eventual restructuring or closure.
In doctors’ offices, we count on lab tests, X-rays and other reliable measures of our health. Can we count on the progress reports in the same way?
Our analyses of last year’s and this year’s progress reports suggest that we cannot.
There is an old joke about philosophers who argue endlessly about whether the King of France is wise. Some argue yes, he is wise; and others argue no, he is not. But they are all wrong because there is no King of France.
Much has been written about New York's Progress Reports: Do they really measure what students are learning in school? We now know that question is ridiculous. We now know that many children in New York do not attend school, at least not in the way most people think about attending school. About 20% of elementary school students missed 20 or more days of school last year. For high school students, 40% missed 20 days or more of school. Overwhelming numbers of students simply do not reliably attend school. The absenteeism issue raises a fundamental question that all the deep thinkers have overlooked: What can any statistical model say about the contribution of a school to students' academic achievement and/or growth if the students do not consistently attend the school? How can the king be wise if there is no king?
This is a fatal flaw in the NY accountability system. It has nothing to do with software systems, statistics, or politics. And no PR spin can correct for it. All of the New York data are meaningless in light of this issue. Merely counting attendance as 5% of an overall school score does not account for such significant absences. It is like an experiment in which you give the experimental group the drug but forget to give it to 25% of them. The effect of this "mistake" in the design is not easily corrected. To put it in testing terms, imagine if 25% of students had skipped 20% of the questions on the test for no reason you know. One out of four students has 20% random blanks throughout the test, as if they had never even seen the questions. Would you think those test scores are useful?
It is a catch-22. If missing weeks and weeks of school doesn't hurt your test score, does your score actually measure what you are learning in school? How could it? On the other hand, if missing weeks and weeks of school does hurt your test score, how could any measure of the school -- or a teacher -- be valid unless it adjusts for the individual attendance of each student? Do we really believe that these gross absences are distributed randomly, equally dispersed over every school and every classroom? Of course not. In fact, even two students who were both absent 20 days from the same classroom may be differentially impacted depending on when they missed class and what was taught those days. So, how could you possibly hold teachers accountable for the progress of students if, in one classroom, attendance is high and, in another classroom, one out of every four fifth grade students has missed 20 or more days of school?
As the city moves from school reports to teacher reports this fall, teachers should demand that data on the attendance of their students be printed in the reports so they have complete, fair documentation of their performance. Given all of the other disaggregated analyses the city is providing, teachers should demand that they have at least one comparison to teachers in the city whose students were absent an equal number of days. At the very least, teachers should be allowed to see a simple graph for the city for each test (grade, subject) that shows attendance in days on the X axis and student test scores on the Y axis. That would then immediately answer the question of how much one affects the other.
For years, we have had laws requiring "opportunity to learn." Under those laws, high stakes tests cannot be used as a graduation requirement for a student unless a state can document that all students have had a fair chance to learn the curriculum. Hopefully, sometime soon we will have a law based on "opportunity to teach" under which teachers cannot be held accountable for their work unless the state (or city) can prove that the teacher was given a fair chance to teach, including: students who attend class, reasonable class sizes, safe working conditions, and appropriate curricular materials. When the city allows an absenteeism rate as it currently stands, many teachers have been denied an "opportunity to teach." Judging them based on students they never had a chance to teach is obviously unfair.
It is also counterproductive. How many proponents of pay-for-performance would be willing to award cash prizes to teachers if they learned that the difference between the winners and losers was whether the students were actually in the classroom? Are they really paying the teacher for instructional excellence or rewarding the teacher for student attendance? Just like the teachers, proponents of those plans may want to ask about the attendance data before they take out their checkbooks. Certainly, if taxpayers are funding the system, we have a right to ask and a right to an answer.
Does attendance affect student test scores? If not, what are test scores measuring? If so, what are we measuring when we fail to measure that?
Very interesting comment, and one that deserves a post in response. It may take a few days for me to work one up.
As a first step, though: A critical issue is whether we treat student attendance as exogenous--that is, predetermined, or outside of the control of the school--or endogenous, i.e., determined by a student's experience in a school. Some absences are likely outside of a school's control, as when a parent decides to pull a child from school for a trip to Disneyworld. Others may result from a student feeling that the school doesn't care about him or her, which may be more subject to school influence.
Thanks for this posting (and your more in-depth article!)
Excellent point as I sat in massive confusion over my school getting an "A" on the report card. While one year we weren't meeting our growth goals, the next we were above target. How kids do on a standardized test (with only looking at the short-term measurement of growth from one year to the next) hardly looks at the fundamental growth (or lack thereof) that may be happening in the school.
While "test sophistication" also plays a strong role in altering test scores, I think more indicators need to be included (such as reported reading growth) in determining whether or not a school is "growing" enough or not.