A pair of Israeli researchers have re-opened the question of “what does the Fox say?”
In an article published earlier this year in the peer-reviewed Journal of Educational Psychology, Eyal Peer and Elisha Babad reached new conclusions when they recreated an old but famous psychological experiment known as “the Dr. Fox lecture.” Their findings contradicted the original study by suggesting that, even though students may enjoy an engaging but meaningless lecture, they are under no illusion that they have actually learned anything.
In the original study, a professional actor, introduced as “Dr. Fox,” used a humorous and engaging tone to deliver a nonsensical lecture on game theory and physician education. The three groups of psychiatrists, psychologists, social work educators, and educational administrators who were his audience subsequently gave him extremely favorable ratings. Nine even said they had read his publications.
"[S]tudent satisfaction with learning may represent little more than the illusion of having learned,” authors Donald H. Naftulin., John E. Ware, Jr., and Frank A. Donnelly concluded in the July 1973 edition of the Journal of Medical Education.
The so-called “Dr. Fox effect” has long been ammunition for critics of student evaluations of teachers. (It was even been mentioned in a 2012 Education Week opinion blog to suggest that observers can be fooled into believing that teaching is effective when it’s not.) This is despite the fact that researchers began raising serious questions about the study’s methodology and interpretation soon after it first appeared in print.
In their recreation, Peer, a senior lecturer at the Graduate School of Business Administration of Bar-Ilan University in Ramat-Gan and Elisha Babad of Hebrew University of Jerusalem, repeated the original statistical analysis, used the original questionnaire, and recreated the original setting, right down to showing the actual 1970s video footage. However, they also addressed critiques that Dr. Fox did not apply to student teacher evaluations because his audience consisted of professionals. They did so by limiting participation to university students. Unlike the original study, theirs included both treatment and control groups.
Peer and Babad also explored critiques by introducing multiple smaller variations of the original study. For instance, they offered participants opportunities to agree or disagree on two types of six-point scales rather than merely saying “yes” or “no” as they had originally been asked to do. They eliminated the original study’s introduction, in which Dr. Fox was described as a world expert in his field. They tried out the lecture on a class in which graduate students had been taught about game theory, the topic butchered by Dr. Fox.
They changed the wording of the questions from positive to negative in order to address criticisms that audience members had rated Dr. Fox favorably on the original yes/no questionnaire as a result an “acquiescence” bias in which survey respondents are more likely to agree than disagree with a statement, regardless of its content. In some conditions, they even explicitly warned students of such biases by writing: “Please pay attention. Past research demonstrated a type of bias in which questionnaire respondents tend to agree with any statement presented to them. Please try as best as you can to avoid this bias in filling out the questionnaire, and respond according to your true feelings.”
None of this made a difference. The students still rated Dr. Fox favorably.
Then, Peer and Babad introduced a new question that had not appeared on the original survey. Using a variety of formats and conditions, they asked students whether they had actually learned anything from the lecture.
Finally, they got different results. Regardless of the wording and condition, almost all of the students indicated they had not learned from the lecture. In essence, though they might have enjoyed watching Dr. Fox, they had not been duped into believing they had learned from him.
Peer and Babad suggested that future research on student evaluation of teachers should concentrate more on students’ ability to judge their own learning in various environments rather than focusing so much on their perceptions of the behaviors of their teachers.
They also questioned why the Dr. Fox study has continued to resonate in academia, despite the significant body of research critiquing it. In doing so, they mentioned a disturbing finding from another set of research studies: Namely, “80% of scientific citations had not been read by the citing authors but copied from the lists of references that appeared in other publications.”
The lead author of those citation studies, Mikhail Simkin, actually wrote to “70 of the hundreds of authors who cited the 1973 Dr. Fox article.” Of those 70, 48 responded. Just five had viewed the footage from the original study.
Here’s the kicker. In a footnote, Babad, second author of the recreation, noted that he had cited the 1973 Dr. Fox article without actually viewing the video footage. He defended himself by adding that such footage was not as accessible in the past as it is today. And he noted that he did read the article multiple times before citing it for the first time.
“However,” he concluded, “viewing the tape changed my opinion about the research and actually triggered the current research.”
You be the judge! Original footage from the Dr. Fox experiment.