Guest post by Jack Hassard
Who the #@!% would make such a statement? Why would such a statement be made about America’s youth?
If you go the Broad Foundation Education page you will find the answer to the first question. This is the first of four statements about American youth, followed by “stark” statistics. The Broad Foundation says:
“We have low expectations for American students.”
Shame on them!
This is the foundation that has channeled over $400 million into education, primarily in charter schools, training of administrators, and online education. It’s a very good time to be in the business of influencing and undermining public education these days, especially if you run a very well-endowed foundation or corporation.
For years now, these same foundations and corporations are using statistics that misrepresent and pervert what is actually the case. Data from tests, especially international test results, are used by politicians, foundation heads, the media, and even the U.S. Department of Education to make proclamations about the status the country’s educational system. Needless to say, American youth are beat over the head for not meeting someone else’s expectations.
TIMSS and PISA: The Super Bowls of Education
Two international assessments are: Trends in International Mathematics and Science Study (TIMSS) and the Program for International Assessment (PISA). Each of these international organizations test students in mathematics, reading and science. PISA studies 15 year-olds, while TIMSS assesses students in grades 4 and 8. TIMSS claims to assess students’ performance on the curriculum, whereas PISA claims to test student’s abilities to apply what they have learned to real-world problems. But please keep in mind, these are low stakes bubble tests comprised of a pool of questions that in general are without a context.
Since about 65 countries participate in these assessments, there is the general feeling that the results are important, and provide us with a glimmer of the nature of science education in these various nations. Some would agree, others would argue that the real issues facing any nation’s educational system are masked by looking at average scores, and simple rankings. Still others report that the findings are inconsistent. For example, a country might score low on TIMSS, yet higher on PISA. Most researchers urge that we use caution when interpreting the results, and not rely of simple averages (now someone’s thinking) to make judgements about student performance.
That said, Dr. Svein Sjoberg, Professor of Science Education, University of Oslo, and Director of the ROSE project (The Relevance of Science Education), an international comparative research project that gathers information about attitudes of students toward science & technology, makes this point regarding PISA:
the main focus in the public reporting is in the form of simple ranking, often in the form of league tables for the participating countries. Here, the mean scores of the national samples in different countries are published. These league tables are nearly the only results that appear in the mass media. Although the PISA researchers take care to explain that many differences (say, between a mean national score of 567 and 572) are not statistically significant, the placement on the list gets most of the public attention. It is somewhat similar to sporting events: The winner takes it all. If you become no 8, no one asks how far you are from the winner, or how far you are from no 24 at any event. Moving up or down some places in this league table from PISA2000 to PISA2003 is awarded great importance in the public debate, although the differences may be non-significant statistically as well as educationally.
If a team doesn’t win the Super Bowl, is that team a failure? What do you think? What does the public think?
Are our schools failing? Is is a fair claim to say we have low expectations for American students? The answer is no!
Let’s take a look.
The Math and Science Conundrum
It is easy to make a quick decision about what you think about math and science education when you read headlines in the newspaper that report that the sky is falling on our educational system, or that we are experiencing another Sputnik moment. But the teaching and learning of mathematics or science, as seen by practicing teachers and collaborating researchers is much more complex (and interesting) than the questions that make up the tests that PISA or TIMSS uses to assess mathematics and science in more than 60 nations.
The conundrum is this. The vision of science that each of these tests measures gives meaning to scientific literacy that looks inward at the canon of orthodox science--the concepts, processes and products of science. Science is seen through the lens of the content of science. But added to this the fact that we have a second vision of science. This vision of science includes public understanding of science and science literacy about science-related situations. In this vision we are more interested in the context of learning, as well as the meaning that students attach to science and mathematics, and how it relates to their world. The lens we use here to view science is within the framework of socioscientific issues (SSI).
TIMSS, because it is tied to the current traditional curriculum, is likely measuring the outcomes of vision I. PISA claims to be measuring students’ abilities to apply what they learned to real situations. But science education researchers Troy Sadler and Dana Zeidler disagree with this, and suggest that the test items that have been released publicly seem quite removed from the intent of the SSI movement.
Given this analysis, we are quite safe to claim that these tests are measuring Vision I of science education, and do not provide a full picture of what actually is happening in many classrooms, schools and districts. Science education is more than learning terms, and concepts. It should include problem-solving and inquiry, and investigations into problems that are relevant to students lived experiences.
Standings
Where do we stand?
PISA and TIMSS are favorite sources of data for foundations and corporations, and especially the U.S. Department of Education (ED) to use to show how poorly American students are doing in mathematics and science. The Program for International Student Assessment ( PISA) is a system of international assessments that tests 15-year-olds in reading, math and science in 65 countries every three years. The latest results are available for 2009. The next will be administered in 2012.
Using scores from tests such as PISA or TIMSS to evaluate and assess science education misleads the public into thinking that science learning has been assessed in the first place. For instance, in the United States there are more than 15,000 independent school systems, and to use a mean score on a science test, such as PISA, or TIMSS does not describe the qualities or inequalities inherent in the U.S.A.'s schools. Furthermore, as we showed above, there are at least two visions for teaching science, and these tests seem to assess Vision I, ignoring perhaps more relevant and interesting science learning that is taking place in many science classrooms. That said, let’s look at two interpretations of data from these international tests.
Interpretation 1.
For example, take a look at these statistics that you can find here on the Broad Foundation website, most of which were based on PISA results from past years.
American students rank 25th in math and 21st in science compared to students in 30 industrialized countries.
America's top math students rank 25th out of 30 countries when compared with top students elsewhere in the world. [PISA Math Assessment, 2006)]
By the end of 8th grade, U.S. students are two years behind in the math being studied by peers in other countries. [Schmidt, W., 2003 at a presentation]
Sixty eight percent of 8th graders can't read at their grade level, and most will never catch up.
The Broad Foundation paints a picture of American education as a broken system, with little hope for many students, especially those who the Broad Foundation claims cannot read at their grade level.
Interpretation 2.
Let’s take a look at another way to examine these data. I have gone to the ED site that presents PISA data, and downloaded Highlights from PISA 2009 in reading, math and science to provide another view of the results. Here is another interpretation, point by point.
1. In mathematics, the only country of similar size and demographics that scored higher than the U.S. was Canada. Most of the other countries that did score significantly higher were small European or Asian (Korea, Japan) countries. The U.S. score was above the average score of OECD countries. Although there were 12 countries that scored significantly higher, there were only three that are similar to the U.S. in size and demographics. We are not ranked 25th in math and 21st in science. (source: PISA Data 2009)
2. America’s top students’ performance place near the top of all students tested by PISA. For example Dr. Gerold Tirozzi, Executive Director of the National Association of Secondary Schools, analyzed the PISA data from the lens of poverty, as measured by the percentage of students receiving government free or reduced lunches. For example, Tirozzi found that in schools where less than 10% of the students get a free lunch, the reading score would place them number 2 in the ranking of countries. This is very far from being 25th as reported by the Broad Foundation.
3. Are we two years behind in the content of math that is being studied by 8th graders? There is no data that would support such a claim in the form of statistical analysis. Curriculum differences have great variance from one country to another. As in other countries, curriculum is implemented in American schools based now on the Common Core State Standards in mathematics, and the high-stakes tests that are used in each state.
4. It is not true that 68% of 8th graders can’t read at their grade level. In the 2009 NAEP reading achievement-level results, 76% of American 8th graders were above the basic level of performance. The graph below shows 8th grade reading results, 1969 - 2011. Yes, we have work to do, but the claim that 68% of 8th graders cannot read is not justified.
NAEP Eighth-Grade Reading Achievement Results 1969 - 2011
Trends in Performance
Here is the truth.
I have provided graphs showing trends in science, mathematics and reading for American students as measured by National Assessment of Educational Progress (NAEP). You will find that the trends reported by NAEP do not support the Broad Foundation’s opinions of American youth.
Science. U.S. students have significantly improved on the PISA test from 2006 to 2009, as shown in the graph below. This trend is a positive sign, and disputes the claim that expectations for American students is low. One of the ways in which data is perverted is to claim that American education, including science education is broken, and that the cause probably has to do with poor performance of “bad” teachers. It is an unsubstantiated claim.
Average scores of 15-year olds in the U.S. and OECD countries in science
Source: Fleischman, H.L., Hopstock, P.J., Pelczar, M.P., and Shelley, B.E. (2010). Highlights From PISA 2009: Performance of U.S. 15-YearOld Students in Reading, Mathematics, and Science Literacy in an International Context (NCES 2011-004). U.S. Department of Education, National Center for Education Statistics. Washington, DC: U.S. Government Printing Office.
Student performance is affected by a number of factors including gender, race/ethnicity, type of school, family income level. The figure below shows Grade 4 results on the 2009 NAEP science assessment. The graph shows relationship between family income (as measured by eligibility for reduced-price or free lunch). Note that students of families with lower incomes perform lower than students from families with higher incomes. This is an important factor when we interpret test scores, as Dr. Gerold Tirozzi found when he analyzed the PISA data from the lens of poverty.
Grade 4 Science Results, NAEP 2009 by Family Income. Click here to explore this data in more detail.
Mathematics. According to NAEP results, mathematics scores for 9- and 13-year olds were higher in 2008 when compared to previous years. There was no significant change in the White - Black or White - Hispanic score gaps compared to 2004. However, since 1973, Black and Hispanic students have made greater gains than White students.
Trend in Mathematics scores for 9- and 13-year olds 1973 - 2008. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), various years, 1973-2008 Long-Term Trend Mathematics Assessments.
Reading. Overall, the national trend in reading showed improvement from 2004 to 2008 for students at three ages (9, 13, and 17). The average reading score for White, Black and Hispanic students was higher than in previous assessments.
Trend in fourth- and eighth-grade NAEP reading scores 1992 - 2011
Have you visited any of the educators in your community that teach science? Have you heard about any of the projects that they doing with their students? What do you think about the Broad Foundation’s crummy assessment of American students’ performance in math and science and that we should have low expectations.
What are they thinking?
Jack Hassard is Professor Emeritus of Science Education, Georgia State University. He is author of The Whole Cosmos Catalog of Science, Science Experiences, Adventures in Geology, The Art of Teaching Science (2009), Second Edition, Routledge, and most recently, Science As Inquiry (2011), 2nd Edition, Good Year Books. Specialities include science teaching & learning, global thinking & education, geology, web publishing, blogging, writing, and antiquing. His blog is The Art of Teaching Science.
All graphs are used with permission.