Should Data Matter?
As you know, Mark Twain (or Disraeli or someone) once wrote that there are three kinds of lies: Lies, Damn Lies, and Statistics. Everyone in education, or so it seems, has learned how to present them in ways that bolsters whatever they want to do. Either, the sky is falling, or things have never been better.
Let me suggest that the statistics you offer are open to different interpretations, to be generous. I don't really know how anyone can say how students in Singapore or Sweden would perform IF they took a NAEP test, which they did not take. Why not just look at the international tests that were taken by students in the United States and many other nations? At least one need not hypothesize about what would have happened, but can look at the results of taking a common test (by the way, this makes my point about the value of having a common national test, rather than 50 different state tests).
On the TIMSS test of fourth grade science in 2003, our students scored significantly behind Singapore, Chinese Taipei, Japan, and Hong Kong. They scored significantly ahead of children from the Netherlands, Australia, New Zealand, and 13 other countries.
On the TIMSS test of fourth grade mathematics in 2003, our students scored significantly behind Singapore, Hong Kong, Japan, the Netherlands, Latvia, Lithuania, the Russian Federation, England, and Hungary. They outscored Cyprus, Moldova, Italy, and 10 other countries (including Iran and Tunisia).
On the TIMSS test of eighth grade science in 2003, our students scored significantly behind Singapore, Chinese Taipei, Korea, Hong Kong, and Japan. Our students outscored most other countries, but many of them were not comparable, industrialized nations (e.g., Jordan, Iran, South Africa).
On the TIMSS test of eighth grade mathematics in 2003, our students scored significantly behind Singapore, Korea, Hong Kong, Chinese Taipei, Japan, Belgium-Flemish, the Netherlands, Estonia, and Hungary. Our students were tied with several other nations, including Malaysia, Latvia, the Russian Federation, and the Slovak Republic, and outperformed 25 others (including many Third World nations, such as Botswana, Lebanon, Ghana, and Morocco).
On the PISA tests (Program in International Assessment), which assesses 15-year-old students, the U.S. scores about average among the 27 nations on reading literacy. The highest-scoring nations in reading are Finland, Canada, and New Zealand. Among the nations that do better than the U.S. are Australia, Ireland, Korea, the United Kingdom, and Japan.
On the PISA tests of math and science literacy, the U.S. students are average. In math, our students score significantly behind Japan, Korea, New Zealand, Finland, Australia, Canada, Switzerland, and the United Kingdom. In science, our students score significantly behind Korea, Japan, Finland, the United Kingdom, Canada, New Zealand, and Australia.
The American Institutes for Research reviewed the TIMSS and PISA data and concluded that if one looked only at the 12 nations that participated in all the assessments, then American high school students' performance is consistently "mediocre," consistently ranking 8th or 9th among the 12 nations. That study can be found here.
Our fourth-grade students begin well (although the AIR authors say their performance is also "mediocre"). By the time they are 15, however, they are smack dab in the middle (or worse, according to AIR). Is that okay? Presumably we want to know whether our students are learning as much as their peers in similar nations. Maybe we can learn something from the results of these tests and improve our own curriculum and pedagogy. Maybe we can just shrug and say we don't care how they score in comparison to any other nation and that test scores don't tell us anything that we need to know.
Again, what I take away from all this is the value of a common test at different grade levels. And by the way, NAEP is not a "low-stakes test." It is a no-stakes test. It tests a sample of students. No student learns his or her score. No teacher learns his or her students' scores. You can't get closer to the definition of no-stakes than that.
As for Campbell's Law, I find it confusing. I take it to mean that any measure or information that is used for policymaking tends to be corrupted by its use. What should we base policymaking on then? Hunches, ideology, hopes, fears? I think we have to rely on some sort of reliable data. We should be talking about how to get data that can improve decisionmaking, how to make it better, how to refine it, how to make it more reliable, rather than disparaging any possibility of ever measuring anything meaningful.