Washington
CORRECTED
If you’re riding your bike home from school and it breaks, how do you figure out what’s wrong? What should you do to fix it?
A new interactive test from the National Assessment of Educational Progress aims to measure how well students do just that kind of real-life problem-solving.
At the U.S. News STEM Solutions conference Wednesday, representatives of NAEP—better known as the Nation’s Report Card—detailed the Technology and Engineering Literacy assessment, which looks at applied technology and engineering skills. The new test, TEL, was recently administered to 8th graders across the country.
TEL bears little similarity to the multiple-choice tests of days past and, in fact, was developed with and by video-game designers. It leads students through 10- to 30-minute computer-based modules in which they confront a problem—perhaps a device is broken or needs to be improved. The test-taker then interacts with the module and uses a range of supplied resources to troubleshoot and find a solution.
Lonnie Smith, an assessment specialist at ETS who worked on TEL, explained that, just as a reading test assesses comprehension and not prior content knowledge, TEL assesses students’ problem-solving skills, not what they already know about engineering or technology.
“What we’re looking at is not are you able to solve the problem, but how do you go about solving the problem? What do you do first, what do you do second ... are you able to do it efficiently?” he said. Every student will eventually finish the troubleshooting activity, he said. “We’re trying to assess not whether students can arrive at a solution, which in my opinion is not that telling of their abilities. But what is interesting is how do they get there?”
The National Center for Education Statistics, the group that administers NAEP, has released one sample module to the public so far. In it, students are asked to play the role of an engineer and figure out why a remote community’s water pump isn’t working. They watch a tutorial on how a water pump functions, ask a community member questions to determine if there has been a water shortage, and use a repair manual to help figure out what needs to be fixed.
The sample task is on the website here, along with a video that takes viewers through the problem in just a few minutes.
Smith assured attendees that the task is an authentic one. “We did research on community organizations in southeast Asia that use this technology,” he said.
Melisse May, the community engagement director for the Greater Cincinnati STEM Collaborative, who attended the presentation and has a background in engineering, said she thought the module was “remarkably accurate.” However, in addition to testing literacy and problem solving, May said, the problem also seems to test “affinity for mechanics. If you’ve not ever seen or used a pump, that’s a lot of data to assimilate.”
The skills TEL aims to measure parallel many of those emphasized in the Next Generation Science Standards, which 11 states and the District of Columbia have adopted. Peggy Carr, associate commissioner of NCES, said the test was already in the works before those standards were finalized. Now, though, NCES is doing an “alignment study” between the TEL and the Next Generation Science Standards frameworks. According to Carr, the results will be released in the next six months.
It’s worth noting that, as yet, there is no assessment aligned with the common science standards. (Does NCES see an opening here, perhaps?)
In March, NCES finished administering the new test to 15,000 8th grade students. Results are expected in the summer of 2016. The test will eventually be given to 4th, 8th, and 12th grade students, although no date has been announced yet.
At this stage, the group reports that the test has been a hit with students. “The students loved it,” said William Ward, a senior research scientist at NCES who also worked on TEL. “The 8th grade students—within the 60-minute testing session, they were engaged.”
[CORRECTION: The original version of this post incorrectly described the NAEP testing earlier this year as a pilot. The pilot testing took place in 2013. The testing from January to March of 2014 was the first round of formal testing of the new assessment.]