A game-based, blended learning math instruction program increased California elementary students’ math scores significantly on state tests compared with students not using the program, a new study has found.
The evaluation of Spatial-Temporal Math, or ST Math, which uses an animated penguin named JiJi to guide children through mathematics lessons, found that students using the program improved their scores on the California Standards Test, or CST, in a statistically meaningful way. The independent, non-profit research organization WestEd, which conducted the study, looked by grade at 209 2nd through 5th grades which fully implemented the program—more than 19,980 students in 129 California public and charter schools.
WestEd defined grades as “fully implementing” ST Math if 85 percent of students had logged in to the program during the school year and at least 50 percent of the grade-level material in the program was covered by the students.The study, released earlier this month, also looked at a wider group of grade levels “provided” with ST Math, but not necessarily fully implementing it. That group included 463 grades spanning 2nd through 5th grade. Even in that larger group, researchers found improvements—though smaller—on the CST compared to control grades not provided with the program.
Andrew R. Coulson, the chief strategist at the MIND Research Institute, which created ST Math, said he hoped the study “helps show how publishers and districts can look at efficacy in a cost-effective way, and hold the content accountable.”
Researchers gauged ST Math’s impact across grades 2 through 5, finding an overall “effect size” of .42, when it came to California state test scores during the 2010-11 school year. The federal Institute of Education Sciences’ What Works Clearinghouse, describes 0.25 as a “substantively important” effect size in research. The study also looked at student achievement by individual grades, finding, for example, that students in 5th grades fully implementing ST Math scored an average of 17.5 points higher on the CST than students in 5th grades not using the program.
Also, the grades fully implementing the program had a larger proportion of students in the advanced and proficient math categories on the California state test than the grades not using ST Math, an effect size of .47. For example, 2nd grades that fully implemented the program had students considered advanced or proficient at an average rate of 8 percentage points higher than 2nd grades not using the program. Those trends were consistent across the grades generally, except for in the 4th grade, where the research found no statistically significant differences in any outcomes for students. That may be because a smaller number of 4th grades fit the study criteria here, said John Rice, a senior research associate at WestEd and a co-author of the study.
Diane J. Briars, the president of the National Council of Teachers of Mathematics, noted that the study underscores the importance of faithful execution of such programs, since it looks at grades which fully implemented the program and grades that were just provided with access to the program.
“One of the big takeaways is that the effect you get is also a function of the quality of the implementation,” she said.
WestEd eliminated from the study pool grades that had a significant number of students already performing at a high level in math, instead focusing on schools with more proficient or lower performing students, Rice said. ST Math preferred to focus on lower performing schools, and including grades with disproportionately higher performing students could have skewed the results differently, by showing less of an impact by the program, he said.
The study authors were also careful to say that the improvements could not be wholly attributed to the ST Math program, since schools that volunteered to implement the program may have already had an emphasis on mathematics.
Briars said that though the study gauged the effect of a technology-based intervention, she noted that non-tech interventions have had similar levels of impact on student achievement. “What we really want to learn are the features that make programs effective and how they’re supporting student learning so we can continue to improve both the technology tools and the non-technology tools we have,” she said.
Some aspects of the ST Math program make it particularly effective, Coulson said, including an effort to provide informative feedback to students when they get a problem wrong as well as when they answer correctly. In addition, the program is highly visual, using virtual manipulatives to introduce math concepts instead of relying on math vocabulary students may not yet have yet learned. Coulson said the study could be helpful to other programs using technology with students. Currently 800,000 students in 40 states are using ST Math, the company said.
“We’re not overflowing with proof points for large-scale, credible, rigorous programs that show ed-tech working,” he said. “We’re hoping this can be...an example of the kind of methodology that can be applied to lots of education programs.”
See also:
- How I Blended My Math Class
- Educators, Researchers Look for Lessons in Blended Learning Algebra Program
Photo credit: MIND Research Institute