skoolboy returns for part II on Quality Reviews. You can find his first post here.
The year is 1975. Coach John Wooden of UCLA has just won his 10th NCAA men’s basketball championship in 12 years, a record that will likely never be matched in collegiate sports. Cambridge Associates sends Clive Wingtip to conduct a Quality Review of the UCLA program. Over the course of a day and a half, Wingtip talks with Coach Wooden, his assistant coaches, the players, and other staff, and observes the team practices. He also observes a collaborative activity: a meeting between Coach Wooden and his assistant coaches. The program is evaluated on five quality statements, each scored as either underdeveloped; underdeveloped with proficient features; proficient; well developed; or outstanding. Here’s a summary of the report he filed:
Quality statement 1: “The coach and staff consistently gather and generate data and use it to understand what each player knows and is able to do, and to monitor the player’s progress over time.”
There is evidence that Coach Wooden studies each individual, and his strengths and weaknesses, very carefully. But he does not rely on statistics collected during practice sessions or games to inform his judgment, and his observations are subjective, not objective. He does not measure performance and progress based on comparisons to similar schools. Overall Score: Underdeveloped
Quality statement 2: “The coach and staff consistently use data to understand each player’s next learning steps and to set suitably high goals for accelerating each student’s learning.”
The Coach and staff convey consistently high expectations to the players, and set specific goals for the team and for individuals. But Coach Wooden regularly played only 7 of the team’s 12 in games, and these same 7 practiced as a unit, suggesting that the reserves were not as important as the regulars. Coach Wooden occasionally displays the soft bigotry of low expectations. He states, “There is nothing wrong with the other fellow being better than you are, as long as you did everything you possible could to prepare yourself for the competition. That is all you have control over. It may be that the other fellow’s level of competency is simply higher than yours. That doesn’t make you a loser.” Overall Score: Underdeveloped with Proficient Features
Quality statement 3: “The program aligns its work, strategic decisions and resources, and effectively engages players, around its goals and plans for accelerating players’ learning.”
Although there was evidence of individualized instruction, the Coach and staff did not use objective team and individual data to plan for and provide this instruction. Team members trusted and respected Coach Wooden. Overall Score: Proficient
Quality statement 4: “The program has structures for monitoring and evaluating each player’s progress throughout the year and for flexibly adapting plans and practices to meet its goals for accelerating learning.”
Coach Wooden relies heavily on repetition during the season. One player said, “He never talks about strategy, statistics, or plays but rather about people and character.” The program did not rely on objective measures to assess progress towards goals, such as the final score of games or written tests, and did not have a playbook. Coach Wooden frequently adjusted the plans for each practice, and made notes after each practice about adjustments; but these notes were based on subjective judgments, not hard data on performance. Overall Score: Underdeveloped with Proficient Features
What’s wrong with this picture? How does the man voted Coach of the Century by ESPN receive a quality review rating of “underdeveloped”? It’s all about the quality criteria, which privilege using data to make decisions about how to help student/athletes to learn and develop over simply making good decisions about teaching and learning. I refer to this as the “fetishization of data.” In the Quality Review game, quantitative performance data have become an end in themselves, rather than a means to an end. One of the most cutting insults that one social scientist can hurl at another is to label another’s research as a “data dump.” In some school districts which have embraced external quality reviews, compiling notebooks full of undigested data has become a substitute for thoughtful analysis of a reasonably small number of important themes and problems.
What’s the solution? I’d start by broadening the definition of data beyond quantitative performance measures. There’s no doubt in my mind that Coach John Wooden relied heavily on data to inform the design of his practices, and his approach to cultivating the talents of his players and his team. But those data took the form of the systematic observations and judgments of an expert practitioner. I’d also seek to evaluate schools on the basis of the quality of the teaching within them, not whether the educators in a school arrived at their teaching practices via the analysis of quantitative performance data.
“Don’t mistake activity for achievement…If you spend too much time learning the tricks of the trade, you may not learn the trade.” (John Wooden)