Cross-posted from the Inside School Research blog. By Sarah D. Sparks

As the first grants issued under the federal Investing in Innovation, or i3, program wind down, we’re taking stock of what we’ve learned from this politically popular program. Here at Inside School Research, I’m also exploring some of the quirkier results coming out of these grants. You can dive deeper into the program with an explanation of the Investing in Innovation structure and political legacy, a round-up of some of most recent i3 evaluations, and on-the-ground-looks at what’s changed in schools in Maine and North Carolina as a result of the program.

The Investing in Innovation development grants were billed as a way to quickly test ideas and interventions with a lot of potential, to find the ones worth expanding to other schools. But as schools in Forsyth, Ga., and Corona-Norco, Calif., discovered, educators can learn a lot even from programs that don’t become superstars under i3.

Forsyth and Corona-Norco each won one of the first 49 development grants under i3 in 2010. Forsyth schools won $4.74 million to implement a digital content and data-management system called EngageME P.L.E.A.S.E., while Corona-Norco won $5 million to implement an online student writing program called WriteUp! But both programs ran into problems during implementation and by the end of the grant, neither program had found evidence of significant benefits for students participating.

In that, the districts are in the same boat as many of the i3 grants, particularly those in the category of “development"—the smallest grants designed to test ideas with the least initial evidence. In an interview with Education Week, Nadya Chinoy Dabby, the U.S. Department of Education’s assistant deputy secretary for innovation and improvement, estimated more than 1 in 4 i3 projects found significant results, higher than the average for venture capital projects, but still leaving many ideas on the cutting-room floor.

But that doesn’t mean the educators and researchers who participated didn’t find any benefit from making the attempt. Jason Naile, Forsyth’s project director, said the district “absolutely” gained from the i3 planning and research process. “We’ve had a number of things as a result of the grant to accelerate what we are doing or bring in new things we wouldn’t have tried.”

Implementation Issues Slowed Evaluation

Forsyth originally envisioned building EngageME from scratch as a data system to track students’ progress through a variety of academic and behavioral indicators, and match it to a comprehensive digital library that would suggest potential remedial or enrichment activities targeted at the student’s interests and learning style to provide “flipped classroom” options.

It was an ambitious task, and in the first year of development, the district had to drastically cut back and change vendors three times in the search for someone who could do the work. By the time the system was up and running, the district had only a half-year, rather than a year, of implementation time with the treatment group before adding the control group for the evaluation. “Like anything else rolled out midyear, it can be difficult to get going,” he said.

Looking back, Naile said said he wished the Education Department had provided more guidance for small-scale grantees on how to change the plan in response to early problems. “We had a change midstream or otherwise we would not have met the goals of the grant, and there wasn’t a lot of guidance on how to do that,” he said. “Navigating [the change in vendors and scope] within a grant environment, with private sector and public school” was challenging.

For other small grantees, Naile recommended getting involved in the networking community that has built up around i3, including discussion groups, meetings, and coaching calls. “Maybe if we had gotten involved in some of those activities earlier, some things would have gone smoother,” he said.

Similarly, the Corona-Norco Unified school district planned a hybrid online system for giving students immediate feedback and support on their writing. The plan was to use the written part of state assessments to evaluate whether the program helped students, but in the third and fourth years of the grant, California suspended student testing as it switched over to tests alligned with the Common Core State Standards. The district was left with only baseline data to draw from in time for the grant evaluation.

That was a huge difficulty. The lack of data was one of the biggest issues, and it was out of our control,” said Charla Capps, Corona-Norco’s director of educational services for curriculum and instruction. “The high schools were telling us they were instantly able to tell which kids had come from treatment schools, because their writing was dramatically better. The soft data we had was amazing, the improvement teachers were seeing in writing was amazing—but it was all anecdotal.”

Schools Persevere Beyond i3

Both Corona-Norco and Forsyth continue to focus on building on the evidence they did glean.

Forsyth now has an individualized education plan for every student, and a digital content library tagged with metadata that allow teachers to search by the content standard, grade, description, and media type, as well as to provide ratings and feedback. For any given assignment, a teacher can get recommended content from the system for each of her students, and assign it to them with one mouse click. The library now has 15,000 items built by district teachers or collected from open resources online and vetted by the district. “A lot of teachers want to use good content, but they don’t have time to search for it,” Naile said. “We’re hoping this lets them get it without having to go to page 35 of a Google search.”

“In our grant, we focused on flipped classes and the technology in personalized student learning,” Naile said. “Now we are taking that back and seeing that ‘personalized learning’ is such a buzzword right now, but there are so many things involved in personalized learning beyond technology.”

For example, “the personalized task button is being used, but not as much as we would like,” Naile said. “Teachers don’t trust the content. For one assignment they could potentially have 100-200 pieces of content recommended to those kids, so they say, ‘How do I know it is meeting their needs?’”

Still, he said the platform developed for the grant has come into everyday use. In fact, during a series of snow days last year, the district continued to hold class, as teachers pulled together online classes from the digital library and assigned them to students at home.

Based on its school-level results from teachers, Corona-Norco decided the WriteUp! program was worth pursuing for elementary and middle school students, as well as for remediation for 9th graders. It has expanded WriteUp! to all of its treatment and control schools and all of its high poverty Title I schools, serving about 26,000 students in grades 4-8 out of 54,000 students in the district.

“The i3 project really helped us to be more thoughtful and more strategic in how we roll out technology throughout the district,” said Barbara Wolfinbarger, Corona-Norco’s administrative director, who wrote its i3 grant in 2010. “As we look at moving forward tech to implement [Smarter Balanced testing], our professional development model [created for WriteUp!] has served us well in classroom coaching, make sure the teachers know what we are planning and have some time to plan how to implement it.”

And now that the state has settled into its new assessment, “when our students had to take the [Smarter Balanced] assessment, they were head and shoulders above the other kids because they had been writing to prompts [on WriteUp!] that ended up being a lot like those on the test,” Capps said.

It’s worth it for districts to try to use i3 or other grants to pilot and study new programs, even if they don’t produce results within the grant period, Capps said. “Ensure you have multiple measures, because things will happen, ... and really document the growth that you are making, both the statistical and anecdotal,” she said.