Data and Lore: Schools, Reputations, and Facts
I wrote about data and some new kinds of assessments being used in independent schools a while back. I've been giving the subject a lot more thought, especially as July 1 approaches, when the National Association of Independent Schools presidency falls to John Chubb, a former Hoover Institute guy who proclaimed "I love data!" at our annual convention back in February,
Much of the rest of the conference had introducers and keynotes assuring us that they, too, loved data. It began to feel a bit like a Star Trek convention, with everyone proclaiming how well they thought of Brent Spiner's cybernetic character, Data, in The Next Generation series.
Star Trek's Data, for non-Trekkies, is an entirely computerized being, brilliant, strong, and even wise, but who yearns (like the Tin Man in The Wizard of Oz) for the human feelings omitted by his mad scientist creator. Occasionally we cross paths with his "brother," who was given Data's brilliance but also a slightly dodgy "emotion chip." The brother, named Lore (you get it, right?) is brilliant but also self-aggrandizing and malevolent, wreaking havoc on any community with whom he comes in contact. If Data celebrates the cerebral, even with his tragic flaw, Lore warns us of the dangers of amoral, egocentric passion.
Independent schools are market-driven; brands, images, and reputations matter a great deal--it's an industry driven in large measure by lore. The great prep schools bask in the reflected glory of their revered graduates (Kennedys, Roosevelts, Bushes, and more), while lesser-known schools bask in the reflected glory of (and often do their best to emulate) the more revered ones. In all, a tiny portion of the American education sector--a few score independent schools and the even smaller number of hyperselective universities to which legend (but not fact) sends the bulk of their graduates--creates an aura well out of proportion to its population and that influences the behavior of the rest of the education sector well beyond its actual due. Reputation matters, disproportionately.
Now, there is at least some "data" to support the primacy we give to these hyperselective universities. Based on grades and standardized test scores alone, we know that the ten percent or so of applicants admitted to the Ivy League and a handful of other ultra-prestige colleges are at least, by those measures, at the top of the academic heap. Of course, there is plenty to dispute here: evidence continues to accumulate that standardized testing favors the already privileged, and school performance can be helped along by family lives in which scrabbling for the next month's rent is not an issue. Certain minorities and the poor continue to be significantly underrepresented, despite the colleges' expressed wish and efforts that this not be so.
For many years, independent school graduates have made up a disproportion of the students accepted to these colleges and universities. The syllogism is obvious, and on the surface it even looks like applied data: if independent school students wind up winning more than their statistically predictable share of seats in highly selective colleges, there must be something inherently superior about the schools that prepare them.
Here my analysis of the data must end, although there is "supporting" evidence that independent school students have somewhat higher standardized test scores than the generality of students. This only makes sense when one considers that most independent schools are themselves at least somewhat selective in their admissions policies--not as an industry anything near like the selectivity of Yale or Stanford, but certainly more selective than public schools, which by definition aren't and can't be selective at all. But further hard data supporting the superiority of independent schools dries up at this point; you can look it up (or not).
College matriculation lists, then, often serve as a proxy for actual data, despite many schools' righteous disclaimers that such lists represent the range of talents, interests, and even financial capacity in their student bodies--that, sure, maybe two kids are going to Princeton and one to M.I.T. and seven to State U., but this is because these are the colleges that "fit" the kids. Mostly this is kind of true, but it is hard for the eyeballs of interested parties (prospective families, say, or alums) not to flit across those college lists and alight on the Ivy names (how nice!) or raise a frown (too many going to State, maybe?). It's an ugly business, and many schools actually wrestle with how to present their college lists in ways that don't reflect, as important but misleading lore, disproportionately well or poorly on the quality of the school and its students. Some just embrace it, it's true, listing on their websites only "a few of the many schools our graduates attend." No surprise that those hyperselectives tend to make up their own disproportion on these "best of" lists.
Washington Post education commentator Jay Mathews and a few other journalists who pay attention to these things regularly excoriate independent schools for failure to release more data (they would like to see average SAT or ACT scores, for example), and perhaps they have a point. However, for good reasons--you can read an example here--many industry organizations urge schools not to release such information, at least when it is being sought to generate some kind of ranking. Schools are so different from one another, missions are so different, student bodies are so different, is the argument. And it's true.
We're stuck then, at least for the external world, with more lore than data in determining much of anything about a single school or even about independent schools as a whole.
As I have suggested, this may change in years to come, if only because accreditors are now requiring schools to show evidence of ways in which they have used data internally to make decisions about academic programs. The challenge for schools, then, will be how to collect useful data, how to analyze it in productive ways, and how to apply it well. It may be that some of the new kinds of assessments, like the College and Work Readiness Assessment or even the international standard PISA (now available for individual school use, but reputedly expensive), will wind up generating data that is both useful internally for schools and actually meaningful when presented and explained to external audiences.
Lore has been good to independent schools, at least on a superficial level, but data is likely to be better, for schools and above all for students.
Engage with Peter on Twitter: @pgow