New York City schools are bringing together classroom teachers, software and app developers, and university researchers to conduct rapid-fire evaluations of new education technology products.
The project, known as the “Short Cycle Evaluation Challenge,” is part of an emerging national push to get teachers more involved in the development of ed-tech tools—and to provide better information to the school purchasing officials who buy those tools.
“It’s about using a process to evaluate who the products work for, when, and under what circumstances,” said Kara Chesal, the director of the 1.1 million-student district’s InnovateNYC Schools office, commonly referred to at the iZone.
Here’s how the concept looks in practice:
Working classroom teachers identify a problems they’d like technology to help address.
Officials from the iZone pair the teachers with companies offering products that might help.
Then researchers from Johns Hopkins University in Baltimore document how the tools are used in the classroom, conducting focus groups, administering surveys, and doing other analyses that ultimately lead to a report that is offered to both the developer and the district.
A major side benefit, participants in the challenge say, is the conversation and feedback that occurs between teachers and developers—two groups that typically operate in separate worlds.
The iZone’s $350,000 effort is one of three “test-bed networks” being funded via a $2 million worth of grants from the Bill & Melinda Gates Foundation. The others are in California’s Bay Area (the Learning Innovation Hub, launched in 2014 by the Silicon Valley Education Foundation, in cooperation with the NewSchools Venture Fund) and Chicago (LEAP Innovations.)
A ‘life-changing experience’
Five companies participated in the first round of New York’s short-cycle evaluation challenge last fall: two providers of educational math software (BuzzMath and Motion Math), a digital-reading platform (LightSail), and two digital student-portfolio management tools (Edusight and Open School ePortfolio.)
“We were at a stage where we had a basic product, but were looking to build new features, and we weren’t sure what to prioritize,” said Garros Li, the founder and president of Edusight, a one-and-a-half year old ed-tech company with offices in Toronto and San Francisco.
“It was a life-changing experience,” he said of participating in the challenge.
Li’s background is in business. After graduating from the University of Western Ontario, he says he worked for two years as a management consultant, helping Fortune 500 companies analyze data.
He left to found Edusight along with three others, based on the premise that teachers need better tools to collect, store, and organize signs that their student are learning.
“The systems they were using hadn’t changed in over 10 years,” Li said. “That really bugged me.”
The company set about building “an intuitive grade book with a built-in student-learning portfolio” that would allow teachers to see both quantitative measures (grades, student test scores, etc.) and qualitative measures (student work samples, teacher notes, videos of students in class) in the same place.
But what Edusight was missing, it turned out, was a good understanding of the low-tech workflow that teachers already use to gather and analyze such information.
“They had no clue what goes on in the life of a teacher,” said 12-year veteran Maria Arcodia, now a kindergarten teacher at Brooklyn Arbor Elementary School, in the borough’s Williamsburg section.
In her kindergarten class, Arcodia said, she has typically kept a running pen-and-paper checklist of how each of her students were doing on everything from letter recognition to phonemic awareness, bringing home charts and highlighters each weekend to group students in different skill and reading levels.
Arcodia and four Brooklyn Arbor colleagues applied last summer to be part of the iZone project because they wanted to make that process of analyzing student data more efficient.
She and her colleagues were pleasantly surprised to see how responsive Li and his fellow developers were to teachers’ experiences and concerns.
Their dialogue resulted in Edusight focusing its efforts on building a new mobile app that allowed teachers to use iPads to take and store photos of student work, as well as to organize students’ information by specific skills and academic standards (and not just by the type of assessment the teacher had given).
Quickly, Edusight became a useful tool the Brooklyn Arbor teachers wanted to use.
“It was really great when parents came in to be able to actually show them, ‘This is how your child sorted shapes,’” Arcodia said.
And for the company, the feedback was invaluable.
“We got lucky,” Li said. “The concept of having dedicated teachers testing your product and giving you detailed feedback is great.”
Systemic impact?
The impact of the iZone effort on the New York district’s ed-tech procurement process is yet to be seen.
Chesal, the iZone director, said the office is still working to figure out how to best bring the reports that resulted from the challenge to the principals and other district decision makers in a format that is useful.
The usefulness of the quick-turnaround “test-bed” evaluation also remains to be determined.
While it’s important to get real-world feedback as quickly as possible, so developers can improve their products and purchasing decisions can be made, there are also limits to the types of information that be collected and analyzed on such a short time frame.
The iZone’s efforts to determine the products’ impact on student achievement, for example, are based on entirely on self-reports from teachers about how their students are doing.
Nevertheless, Chesal said she’s optimistic that the process will lead to meaningful insights about how effective, fast evaluations of ed-tech products are best done. She noted that the New York, Chicago, and California efforts are all taking slightly different approaches.
And Emily Dalton Smith, the senior program officer at the Gates Foundation who is supporting the three networks, said the work is still early, but promising.
“We have been really thrilled to see a lot of activity around making sure kids and teachers are really driving ed-tech product development,” she said.
Photo: Garros Li, center, founder and president of ed-tech company Edusight, listens to teachers from New York City’s Brooklyn Arbor Elementary School. -- Courtesy of Edusight and InnovateNYC Schools.
The Bill & Melinda Gates Foundation provides support for Education Week’s coverage college- and career-ready standards. Education Week retains sole editorial control over the content of this coverage.
See also:
- K-12 District Leaders Evolving Into Smarter Ed-Tech Consumers
- Ed-Tech Vendors Often in Dark on District Needs, Study Shows
- Who Calls the Shots in School Purchasing? Vendors, K-12 Officials Disagree
- Better District Ed-Tech Purchasing Strategies Explored