Video Games as 'Petri Dishes' for Assessment Data
For the current issue of Education Week, I wrote about University of Wisconsin-Madison researchers who believe video games could revolutionize the assessment field. I had way too much thought-provoking material to cram into 1,500 words...here's Part 1 of what got left out. To read more about video games, noncognitive skills, and the federal government, check out Part 2.
One of the big reasons some people are so excited about video games' potential as tools to assess student learning?
"Games are interesting assessment environments because they're like little petri dishes," said Constance Steinkuehler, an associate professor of education and the co-director of the Games+Learning+Society center in Madison. "You can put people into games and actually watch what they do and get the 'data exhaust' of their choices, what they repeat, what they struggle on, and where they spend the time. And you can use that [data] to draw inferences about how they're learning."
In Crystals of Kador, the center's new role-playing iPad game prototype about a damaged robot stranded on a distant planet, here's how that process works:
While playing the game, a child will make roughly 3,000 decisions—for example, to move her robot, to talk to an alien, or to Taser an interplanetary creature. Each decision is made by tapping on the iPad screen. The game will automatically log each of those "events," along with about 15 pieces of contextual information. By the time one kid is done playing Crystals, the researchers will have a database with nearly 50,000 records.
Multiply that by hundreds or thousands of students playing the game, and you have a monstrous load of data.
Businesses gather and analyze this "telemetry" or "clickstream" information all the time, said Rich Halverson, a former high school math teacher in Chicago who helped found Games+Learning+Society and is now an associate professor of education at Wisconsin-Madison.
But "those data are now being used for marketing, to sell you in-game apps," Halverson said. "We want to change that discussion. We believe we can [use telemetry data] to determine your learning patterns."
For proponents of video games-as-assessments, the ideal scenario is when "winning" a game would in and of itself constitute solid evidence of content mastery.
James Paul Gee, another founder of the Games+Learning+Society and a leading theorist of video games for learning, likes to use Halo as an example.It would be nonsensical, argued Gee, to give a player who successfully completes the popular first-person shooter game a separate written assessment of his or her Halo knowledge. "The game is the test," said Gee. "If we could design teaching algebra as well as Halo is designed, we'd say the same thing."
But so far, said Halverson, using telemetry data to empirically demonstrate that real learning has occurred has proven more complicated than that. "It's not simply a matter of can you master the [game] mechanics," he said. "In commercial games, that's all there is. But in learning games, there's a content model you're trying to get across."
Using games' telemetry data to assess that kind of academic content mastery is a brand new field, one that still has traditional assessment experts a bit wary.
"How do you put it all together? How do you know the conclusions you're making about students are accurate?" wondered Gregory Cizek, a professor of educational measurement and evaluation at the University of North Carolina at Chapel Hill.
The Games+Learning+Society researchers say they're aggressively trying to answer those questions.
With Crystals and Tenacity, another new iPad game from the center, the researchers are also looking to validate signs of learning found in telemetry data by examining corresponding changes in the actual structure and functioning of children's brains.
The researchers say there's reason for optimism: they recently had success mining the telemetry data of another game they developed in-house. In Progenitor X, players are asked to assume the identity of a scientist who has been infected by zombies. In order to save themselves, they must grow new organs from scratch. Underneath the gore, the game, developed with the help of researchers at the Morgridge Institute for Research and the Wisconsin School of Medicine and Public Health, is designed to teach science students about stem cells.
After combing through the data generated by the game, the researchers compared the patterns they found with students' performance on pre- and post-tests of their knowledge of stem cells. A compelling finding emerged: Players who engaged with Progenitor X's toughest challenges improved their scores on the traditional assessment, even if they failed at the challenges and "lost" the game.
"Designing a game is not designing a test. It's designing an experience," said Halverson. "But if we can confirm [what we found with Progenitor X] with thousands of players, then we could start to say the game itself is a form of assessment."
Not just any form of assessment, argued Gee, the Arizona State professor. A form that represents a major upgrade over what he called our current "Cro-Magnon" testing regime.
"With standardized testing, you're sampling your population two or three times a year, because you don't have a better way to get data," Gee said. "Now, we can collect a lot more data, in the kid's real environment. We can mine it. And we can represent it in ways that help the kid as well as grade the kid."
"It just seems like a better way," Gee concluded.
And as for likely concerns about invasions of privacy, inappropriate surveillance, and other issues that have arisen in conjunction with the prevalence of "big data" in other sectors of society?
Gee acknowledged a potential "dark side" to collecting and analyzing all this data on student learning.
Halverson did, too—but then the former math teacher grew exasperated.
"The fear of misuse is one of the big reasons why technology in schools is so far behind [the rest of society]," Halverson said. "Kids have developed these rich digital lives outside of school, while schools are still stuck in the 20th century."