« Following Up on John Doerr: The Private Sector in Space | Main | Friday Guest Column: SBR and RB – Measures of Efficacy or Red Herring? »

The Letter From... SBR is Really RB, and RB is Really SBR: Do You Want Me to Apply SBR to Your Program?

| 1 Comment

Every little brother must know the big brother line: “Yes means no, and no means yes. Do you want me to hit you?”

No Child Left Behind was intended to encourage the private sector to develop a class of products and services with demonstrated efficacy. Providers and investors who believed this determined that if they could prove their offerings improved student performance, they could surmount the formidable barriers to entry that mainstream publishers built up over a century. This group welcomed the definition of "Scientifically Based Research" in NCLB that emphasized comparing the academic performance of similarly situated students who do and do not receive the program under study. Supplementary Educational Services providers, believing that they could not meet this standard, lobbied successfully for a lower, “Research Based” standard. That approach permitted evidence as limited as reference to bodies of related research that suggest a particular program might be effective.

The policy problem facing legislators and the Department of Education is how to balance three requirements.

First, to assure that the government does not waste its money, teachers' time, or students' education, on programs that simply don’t work. We’d like a standard high enough to at least weed out products and services benefiting solely from the placebo effect.

Second, understanding that the state of the art in k-12 program evaluation is nowhere near advanced enough to be confident that a very high bar will not screen out more useful programs than those lacking utility, to create a standard that leads to a market large enough to give educators real choice among offerings. We don’t want a standard no one can meet.

Third, recognizing that the growth of private sector offerings in school improvement depends on new investment in new providers, assuring that the standards are simple, clear and stable. Investors will not put money into something when they don’t understand the criteria of technical success or can’t rely on it to hold for the period during which they expect to see their financial return. We don’t want an arbitrary, capricious or ever-changing standard.

For roughly the past year, I have provided readers of School Improvement Industry Week (see explanation below) with the standards states are beginning to employ to assess the efficacy of new and current Supplementary Educational Service (SES) providers. Yesterday I excerpted the relevant Kansas regulations:

The Kansas State Department of Education (KSDE) will select providers of supplemental educational services who adequately meet the eligibility requirements described in the application. Applicants who meet the criteria will be included on an approved list maintained by KSDE and made available to local districts.... Providers will be removed from the list if they are unable to sustain the requirements or meet the achievement goals within two years of consecutive service....

Evidence of Effectiveness.... Demonstrate that the program is of high quality and that the applicant has been effective in raising the achievement levels of student who have received services in Kansas. If new to Kansas provide evidence in locations with similar demographics as Kansas. Use evidence from standardized tests, or student grades, teachers’ assessments, improved student attendance, retention/promotions rates, or other measures which can be correlated to Kansas data points. Strongest consideration will be given to evidence of positive impact on student achievement, particularly low-income underachieving students to include SPED and ESL students. Include details to explain whether this evidence was gathered from services that your organization provided or from another entity’s use of the program. Please note that priority will be given to third party independent research.

Some months earlier I offered the new rules promulgated by the Rhode Island Department of Education:

To be included on the approved list of supplemental educational services providers, applicants must meet the following criteria:
• Have a demonstrated record of effectiveness or have a high probability of increasing student academic achievement….
• Provide instruction that is of high quality, research-based, and specifically designed to increase academic achievement of eligible children on state assessments and attain proficiency in meeting the RI academic achievement standards. Reading instruction must be scientifically based and proven to be effective and include the National Reading Panel Report criteria

Reviewing the federal government’s implementation of NCLB’s SBR/RB provisions I think it's fair to say that the states have been moving towards the legislation’s Scientifically-Based Research standard to define Research Based under Supplemental Educational Services. I am as adamant about the need for SES providers to demonstrate efficacy as anyone, not because it's required by law – but because it is essential for the SES program’s future in a reauthorized NCLB. As Radio Moscow commentators used to say during the Cold War, “it is no coincidence” that states have adopted a very high standard of evidence for a program most don’t support.

But it is also fair to say that the Federal government has used the NCLB’s Research-Based standard to define SBR for most school improvement programs. The obvious example here is Reading First, where programs were judged on the basis of a third or fourth level employee’s view of their relationship to a body of knowledge on phonics, rather than specific evidence of program efficacy. It is also no coincidence that the big winners here were programs sold by the big publishers, and authored by the Department’s Reading First advisors.

If one goal of NCLB is to encourage new investment in new firms with new effective school improvement products, services and programs, the implementation of the law’s SBR and RB provisions could hardly be less helpful. No investor can look at this history and feel any confidence in the government’s interest in real choice or what works. The real message has to be "watch what we do, not what we write."

Regarding School Improvement Industry Week

School Improvement Industry Week (SIIW) is one of the information services I provide to my clients. SIIW summarizes or excerpts five documents I consider important professional reading for leaders in this field. The five-page report generally includes something on federal or state regulation of No Child Left Behind, education research, marketing data, something specific to one state, and program evaluation. The document is delivered by email Tuesday as a web-enabled pdf, with every item hotlinked to its source.

I developed the service because it’s something I wished I had when I was New American Schools COO and President of its Education Entrepreneurs Fund. As important as this kind of information was to my responsibilities for organizational strategy, my in box was too full for me to take the extra time required to find and read the right materials. With SIIW I could have relied on someone who was doing this for a living, printed out the report, and put it in my briefcase or suit coat pocket to read when I was in a cab, on a plane or with a few minutes to spare. If I read one page a day for 52 weeks I’d have the gist of 260 documents covering our industry. That’s a lot better than what I actually was able to read.

1 Comment

This is the first time I have seen in print that the intent of NCLB was to encourage the development of SBR or RB products, although it is not a surprising point of view. But the SBR or RB dichotomy leaves me a bit cold. I think either can be fairly rigorously defined. I think the larger problem is that there is insufficient state resource or incentive to do much in the way of enforcement. While the public schools have been almost universal in their insistence that the private providers offered an inferior product by any measure (and again, I would submit that regardless of state requirements, there is little resource for actually evaluating against either standard), I have found that the district offerings are typically no better. As a parent, I have tried to track down any research base behind the district (required) intervention offerings. I have gotten the full range of run-arounds within district. Generally when I locate someone with data it is weak (along the lines of those who choose to show up on Saturday mornings for the intervention did better on tests than those who did not--which points to a large selection bias), or else I am referred to the product vendor (apparently there is an assumption that a product purchased and implemented by the district is inherently superior to one selected by parents)--where the "research base" is likely to be a claim that the materials are aligned to state standards.

As a parent, I can drive a truck through the holes--and if the state had resources to follow closely, likely they would as well. There were also some requirements that students who received SES (from outside vendors) be evaluated on improvement goals set by the district. But this did not account for the lack of cooperation from the district in setting any such goals, or providing any meaningful individual student data on which to base such goals.

Comments are now closed for this post.


Recent Comments

  • Margo/Mom: This is the first time I have seen in print read more




Technorati search

» Blogs that link here