Every little brother must know the big brother line: “Yes means no, and no means yes. Do you want me to hit you?”
No Child Left Behind was intended to encourage the private sector to develop a class of products and services with demonstrated efficacy. Providers and investors who believed this determined that if they could prove their offerings improved student performance, they could surmount the formidable barriers to entry that mainstream publishers built up over a century. This group welcomed the definition of “Scientifically Based Research” in NCLB that emphasized comparing the academic performance of similarly situated students who do and do not receive the program under study. Supplementary Educational Services providers, believing that they could not meet this standard, lobbied successfully for a lower, “Research Based” standard. That approach permitted evidence as limited as reference to bodies of related research that suggest a particular program might be effective.
The policy problem facing legislators and the Department of Education is how to balance three requirements.
First, to assure that the government does not waste its money, teachers’ time, or students’ education, on programs that simply don’t work. We’d like a standard high enough to at least weed out products and services benefiting solely from the placebo effect.
Second, understanding that the state of the art in k-12 program evaluation is nowhere near advanced enough to be confident that a very high bar will not screen out more useful programs than those lacking utility, to create a standard that leads to a market large enough to give educators real choice among offerings. We don’t want a standard no one can meet.
Third, recognizing that the growth of private sector offerings in school improvement depends on new investment in new providers, assuring that the standards are simple, clear and stable. Investors will not put money into something when they don’t understand the criteria of technical success or can’t rely on it to hold for the period during which they expect to see their financial return. We don’t want an arbitrary, capricious or ever-changing standard.
For roughly the past year, I have provided readers of School Improvement Industry Week (see explanation below) with the standards states are beginning to employ to assess the efficacy of new and current Supplementary Educational Service (SES) providers. Yesterday I excerpted the relevant Kansas regulations:
The Kansas State Department of Education (KSDE) will select providers of supplemental educational services who adequately meet the eligibility requirements described in the application. Applicants who meet the criteria will be included on an approved list maintained by KSDE and made available to local districts.... Providers will be removed from the list if they are unable to sustain the requirements or meet the achievement goals within two years of consecutive service....
Evidence of Effectiveness.... Demonstrate that the program is of high quality and that the applicant has been effective in raising the achievement levels of student who have received services in Kansas. If new to Kansas provide evidence in locations with similar demographics as Kansas. Use evidence from standardized tests, or student grades, teachers’ assessments, improved student attendance, retention/promotions rates, or other measures which can be correlated to Kansas data points. Strongest consideration will be given to evidence of positive impact on student achievement, particularly low-income underachieving students to include SPED and ESL students. Include details to explain whether this evidence was gathered from services that your organization provided or from another entity’s use of the program. Please note that priority will be given to third party independent research.
Some months earlier I offered the new rules promulgated by the Rhode Island Department of Education:
To be included on the approved list of supplemental educational services providers, applicants must meet the following criteria:
• Have a demonstrated record of effectiveness or have a high probability of increasing student academic achievement….
• Provide instruction that is of high quality, research-based, and specifically designed to increase academic achievement of eligible children on state assessments and attain proficiency in meeting the RI academic achievement standards. Reading instruction must be scientifically based and proven to be effective and include the National Reading Panel Report criteria
Reviewing the federal government’s implementation of NCLB’s SBR/RB provisions I think it’s fair to say that the states have been moving towards the legislation’s Scientifically-Based Research standard to define Research Based under Supplemental Educational Services. I am as adamant about the need for SES providers to demonstrate efficacy as anyone, not because it’s required by law – but because it is essential for the SES program’s future in a reauthorized NCLB. As Radio Moscow commentators used to say during the Cold War, “it is no coincidence” that states have adopted a very high standard of evidence for a program most don’t support.
But it is also fair to say that the Federal government has used the NCLB’s Research-Based standard to define SBR for most school improvement programs. The obvious example here is Reading First, where programs were judged on the basis of a third or fourth level employee’s view of their relationship to a body of knowledge on phonics, rather than specific evidence of program efficacy. It is also no coincidence that the big winners here were programs sold by the big publishers, and authored by the Department’s Reading First advisors.
If one goal of NCLB is to encourage new investment in new firms with new effective school improvement products, services and programs, the implementation of the law’s SBR and RB provisions could hardly be less helpful. No investor can look at this history and feel any confidence in the government’s interest in real choice or what works. The real message has to be “watch what we do, not what we write.”
Regarding School Improvement Industry Week
School Improvement Industry Week (SIIW) is one of the information services I provide to my clients. SIIW summarizes or excerpts five documents I consider important professional reading for leaders in this field. The five-page report generally includes something on federal or state regulation of No Child Left Behind, education research, marketing data, something specific to one state, and program evaluation. The document is delivered by email Tuesday as a web-enabled pdf, with every item hotlinked to its source.
I developed the service because it’s something I wished I had when I was New American Schools COO and President of its Education Entrepreneurs Fund. As important as this kind of information was to my responsibilities for organizational strategy, my in box was too full for me to take the extra time required to find and read the right materials. With SIIW I could have relied on someone who was doing this for a living, printed out the report, and put it in my briefcase or suit coat pocket to read when I was in a cab, on a plane or with a few minutes to spare. If I read one page a day for 52 weeks I’d have the gist of 260 documents covering our industry. That’s a lot better than what I actually was able to read.
