Good Research and Good Marketing: Marzano and the Evidence
I spoke with Michael Toth of Learning Sciences International last Monday, after he called EdWeek to discuss my recent post on Marzano's new "causal" evaluation system.
LSI is partnering with Marzano to help states and districts implement the model, and Toth expressed some concern that I was misrepresenting it. He claimed that the model has indeed been validated by new research, and took umbrage at my statement that
Marzano is simply drawing on his previous meta-analyses of teaching techniques to throw more weight behind his framework. There is no specific research validating the framework itself, much less its godlike causal power. It's probably a decent framework, but it has no more ability to improve student learning than any other framework for evaluating teaching practice.
I think teacher evaluation is an extremely important issue, and I certainly have no problem with companies like LSI helping states and districts do it well. Marzano's research is a terrific foundation for teacher evaluation, because it has the potential to guide professional growth in areas identified for improvement.
One of my concerns with all evaluation systems is that they will be used as prescriptive checklists of teacher behaviors, rather than descriptions of professional practice. Any system that penalizes teachers for failing to use a specific strategy is wrongheaded. Strategies have a place—helping us figure out how to improve our practice—but are not evaluation criteria. To turn strategies for achieving excellence into measures of excellence is to place the cart in front of the horse.
Mr. Toth was quick to point out that the Marzano system is not checklist-oriented, and that it contains a robust set of descriptions of practice. These descriptions of practice overlap a bit with instructional strategies (moreso than, say, in Charlotte Danielson's framework), which means that they are more proven by research, but also that they risk being wrongly turned into evaluative criteria.
For example, take a look at Marzano's Domain 1: Classroom Strategies and Behaviors. Design Question #5 asks "What will I do to engage students?" Several strategies are listed, including "Using academic games" and "Using friendly controversy."
Should teachers be held accountable for engaging their students? Absolutely. Should they be penalized if they fail to use academic games or friendly controversy in any particular lesson? Of course not. Those are strategies to use if you need help engaging your students.
I'm sure Dr. Marzano would agree that the problem with having a widely known evaluation system is that people will not always interpret or implement it in the way you intend. It is of the utmost importance, then, that marketing materials avoid exaggerating, overpromising, or creating misconceptions about how the system should be used.
While I have the utmost respect for Marzano the person, Marzano the brand is being marketed in ways that are unhelpful and unflattering to Dr. Marzano's reputation. On the front page of MarzanoEvaluation.com, it says
Learn more: Read more about Marzano's Causal Teacher Evaluation Model and it's [sic] Proven Cause & Effect Relationship.
What exactly is this "proven cause & effect relationship"? The link points to this page, which states (emphasis in original):
The first of its kind, this causal teacher evaluation model is not only based on studies that correlate instructional strategies to student achievement, but is also grounded on experimental/control studies that establish a direct causal link between elements of the model and student results.and
Each domain builds on the previous with direct links to create a causal chain that results in the increased learning and achievement of all students. This direct causal effect between elements of the model and student achievement is validated by data analysis from experimental/control studies.
While it's true that many studies have been done to establish links (even causal links) between elements of the model (that is, particular teaching practices) and student learning outcomes, it is not true that the model itself has been proven to cause student learning to increase. That type of study would be very difficult, expensive, and time-consuming to conduct, and while it's certainly conceivable, it simply hasn't been done yet for this model, as far as I can tell. Mr. Toth tried to convince me otherwise, but declined to cite any studies other than those mentioned in this document, none of which are actually studies of the Marzano evaluation system.
There's a subtle sleight-of-hand taking place here: While specific practices within the model have been proven to increase student learning, the full causal chain represented in the model has not been validated by any studies that have been brought to my attention. What we're really looking at in the diagram on this page is a theory of action. And it's a good one. But it is not a proven one. Strong links do not necessarily form a chain; even one weak or missing link can break the chain.
Why does this matter? Because anyone could develop a teacher evaluation system based on the same research Marzano cites, and that wouldn't necessarily make it a good system. Marzano's system is probably very good, but no system can live up to the claims that are being made in its marketing materials.
Dr. Marzano, it's time to hold your marketing people to the same high standard as your research. Clarity, honesty, and modesty will do more for your evaluation system's credibility than all the bold and italicized text in the world.