« Violation of the Iron Law of Qualitative Research in Education, #1,321 | Main | Suggestions for a Summer Reading List? »

skoolboy's Platinum Law of Educational Research

| 6 Comments
spiffboy2.jpg
eduwonkette's "Iron Law of Qualitative Research in Education" is that the number of participants in the study should exceed the number of authors on the paper. Ha-ha, very funny, but the subtext is that (a) we cannot learn anything of value from studies that have small sample sizes; (b) qualitative research often has small samples; (c) therefore, we can't learn very much from qualitative research. Eduwonkette would protest that that's not what she's saying at all—"qualitative research is critical to educational research and policy," and I know that she does believe this. But poking fun at a paper reporting qualitative data without explaining why does her readers, and those who believe that qualitative research can be of great value, a disservice. I'd like to upgrade eduwonkette's Iron Law to skoolboy's Platinum Law of Educational Research: Poorly designed and conceived research is poorly designed and conceived research, regardless of the sample size.

I'll leave a defense of research using small samples for another day, and focus on why I think that the paper eduwonkette drew to our attention is poorly designed and conceived. I don't want to go on too long about this—there's a lot more to say than will hold the attention of casual readers—but here's the gist. The authors claim that teaching for social justice evokes a range of emotions in novice teachers, and they seek to understand the strategies that teachers use to navigate their emotional responses, and the implications of those strategies for their self-understandings and practices. I found the concept of socially just teaching confusing, but I'll accept the possibility that there are teacher education programs and novice teachers that are committed to the idea of teaching in ways that promote the life chances of members of marginalized groups in society, such as the poor and racial/ethnic minorities. In this paper, teaching for social justice is taken for granted as a good thing, which I know vexes some readers here, and the study seeks to build on previous work on emotions and emotional navigation in teaching. It's not news that teachers often express ambivalence about their work, and that they might struggle with how to respond to feelings of ambivalence.

The authors introduce the term critical emotional praxis to characterize the role of emotions in socially just teaching. This is not an analytic term emerging from their analysis of data on how teachers manage emotions in their work; rather it is a normative term—that is, a term that describes what the authors think the role of emotions in teaching for social justice should be. In their view, critical emotional praxis involves understanding the role of emotions in engaging with unequal power relations in classrooms and society; acknowledges the interplay between a teacher's local context and her emotional responses; and moves from a theoretical understanding of emotion to a practical set of relationships and teaching practices that promote teaching for social justice. I find this concept to be of minimal value for research purposes, since it has no apparent relationship with observations of teachers' practices and emotional states.

The purpose of the study is to describe how a novice teacher seeking to teach for social justice navigates her ambivalent emotions. The authors don't offer an explanation of why a case study of a single teacher is appropriate to address the questions they pose about emotional navigation in teaching. In this particular study, one of the authors observed the teacher for 80 minutes per day during the final 9-week period of her first year of teaching, and interviewed the teacher six times for two to three hours at a clip. The teacher's department chair and 10 students were interviewed as well. A year later, an author interviewed the teacher once for three hours, and did two more 80-minute classroom observations. Although the authors acknowledge some of the problems associated with the fact that the subject of the study was a former student of one of the authors, a teacher educator who taught her about socially just teaching, these problems are not adequately addressed in the research design.

What are some of the key findings of the research? One pertains to the teacher's mode of response to her emotions. The teacher, Sara, began seeing a professional counselor in December of her second year of teaching. She also enrolled in a course on nonviolent communication, and began sponsoring her school's forensics team. These three concrete modes of response, the authors contend, gave her insight into her self and emotions, and provided concrete strategies for relaxing, having fun, and balancing her feelings of sadness stemming from her observations of social injustice. With what consequences? She quit teaching, leaving her school and volunteering at an orphanage and school in a developing country.

What's wrong with this picture? I think the authors lacked a theory of when novice teachers might develop feelings of ambivalence and seek out strategies for coping with them. In this study, most of the action took place in the teacher's second year of teaching, and the primary source of data on these strategies is a retrospective interview conducted at the end of the second year. Therefore, the authors missed most of the action, and can only provide a bare-bones understanding of even this one case. Moreover, the fact that this teacher left the field of teaching raises serious questions about whether this case can inform teacher education in the ways that the authors hope. One reading of the results is that the teacher's leaving of the field is prima facie evidence that her strategies for coping with the feelings of ambivalence associated with seeking to teach for social justice didn't work; and although we can certainly learn from strategies that don't work, a study that shows strategies that do work would likely be more valuable.

The problem with this paper is that the intellectual payoff is nowhere near commensurate with the amount of space it took up in a major journal—45 journal pages, from start to finish. I agree with eduwonkette that it doesn't reflect well on the field of education research to have papers which make marginal contributions taking up so much airtime, and the time I spent reading this paper is lost forever—time that I could have spent in other, more valuable ways, like updating my Facebook page or grading papers.

But: the take-away message here is not that a study with a small sample—even an N of 1!—cannot contribute new knowledge to the field of educational research. It's that a badly designed and executed study won't contribute much. And bad design and execution have to do with a lot more than sample size.
6 Comments

I'm happy to accept the smackdown on all of your points but one - the sample size of 1 person.

Maybe I am just thinking about the wrong topics - can you give me some examples of research questions that could be answered with a sample size of 1?

I'm assuming that you're not talking about single-case experimental designs using quantitative analyses, which your nerdy IES friends are so fond of.

One form of question that a single case can address takes the form, "Does the phenomenon X exist in the world?" A single case study can serve as an "existence proof," although it might not be the most efficient design for doing so.

I worry, though, that casting this as a matter of "research questions that could be answered" is overly narrow, because generating research questions via a case study is a perfectly honorable form of research. It only takes a single case to create some intellectual tension between what we think we know and what we might like to know.

Personal favorites of single-case studies of individuals: David K. Cohen's classic "Mrs. Oublier" study of instructional reform in California, and Deborah Loewenberg Ball's studies of her own practice as an elementary math teacher.

But you are talking about case studies of individuals, and not the broader array of single case studies, aren't you? Because there's a much longer list of case studies of schools, reforms, etc. at our beck and call.

The Platinum & Iron Laws are not mutually exclusive. I agree with both, with one qualification: like the Platinum Law, the Iron Law applies to both qualitative & quantitative research. To put my prejudice out in the open, I am a number-cruncher, but I recognize the value of good qualitative research. I do not, however, care much for what I call "personal history as research." Too many of those "studies" belong under the heading of essays rather than research.

Yes on hypothesis generation - but maybe the follow-up study is what belongs in the journal, not the study attempting to generate hypotheses.

Yes, my concern is with study of individuals, who I would guess are more idiosyncratic than organizations - though many of the same issues can be raised.

I like the Oublier (Outlier?) study too, but wouldn't it have been even better if he had two other teachers? How hard is it to study three teachers?

"Yes on hypothesis generation - but maybe the follow-up study is what belongs in the journal, not the study attempting to generate hypotheses."

Great point eduwonkette. The study of outliers is an area that could use more attention especially in Ed Research but i really don't think a theory generated from study of one person is a real theory. There has to be more to why this study is so small.
I could even see three separate studies conducted by three separate research teams but using concurrent methodology as a valid contribution to the field. Of course then we might talk even more about the researchers and less about the research. Hmmm maybe I should re-do this study for my dissertation? Nah I don't think it would fly... Our school is trying to gain credibility not lose it.

I think this discussion of a study of a single individual is a bit of a red herring. A case study is a study of a single case of some phenomenon, and the phenomenon isn't the individual. The study we've been talking about, as flawed as it is, is a study of a case of a teacher managing her emotions while attempting to engage in socially just teaching. There is a venerable tradition of case study research in the social sciences, and lots of thoughtful writing about how to set the boundaries of a case.

Would a study of three cases be better than a study of a single case? Everything else being equal, sure, but everything else rarely is, as there are inevitable tradeoffs that are rooted in the limits to researcher time and energy. Studying three teachers not very intensively over the course of a year versus studying one teacher very intensively? That's a tradeoff, and it's hard to tell in advance which design is better.

I will say, however, building on something that eduwonkette noted in an earlier post, that the more diverse the phenomenon that a researcher is studying, the more desirable it is to have multiple cases of that phenomenon to study. If we think that teachers have diverse ways of negotiating a response to the emotion work of teaching -- and I think we do think this -- then we should sample more cases of this phenomenon. This is another very serious limitation to the AERJ study under discussion.

Comments are now closed for this post.

Advertisement

Recent Comments

  • skoolboy: I think this discussion of a study of a single read more
  • J.M. Holland: "Yes on hypothesis generation - but maybe the follow-up study read more
  • eduwonkette: Yes on hypothesis generation - but maybe the follow-up study read more
  • Georgia Sam: The Platinum & Iron Laws are not mutually exclusive. I read more
  • skoolboy: I'm assuming that you're not talking about single-case experimental designs read more

Archives

Categories

Technorati

Technorati search

» Blogs that link here

Tags

8th grade retention
Fordham Foundation
The New Teacher Project
Tim Daly
absent teacher reserve
absent teacher reserve

accountability
accountability in Texas
accountability systems in education
achievement gap
achievement gap in New York City
acting white
admissions
AERA
AERA annual meetings
AERA conference
AERJ
Alexander Russo
Algebra II
American Association of University Women
American Education Research Associatio
American Education Research Association
American Educational Research Journal
American Federation of Teachers
Andrew Ho
Art Siebens
ATR
Baltimore City Public Schools
Barack Obama
Bill Ayers
black-white achievement gap
books
books on educational research
boy crisis
brain-based education
Brian Jacob
bubble kids
Building on the Basics
Cambridge Education
carnival of education
Caroline Hoxby
Caroline Hoxby charter schools
cell phone plan
charter schools
Checker Finn
Chicago
Chicago shooting
Chicago violence
Chris Cerf
class size
Coby Loup
college access
cool people you should know
credit recovery
curriculum narrowing
D3M
Dan Willingham
data driven
data-driven decision making
data-driven decision-making
David Cantor
DC
Dean Millot
demographics of schoolchildren
Department of Assessment and Accountability
Department of Education budget
desegregation
Diplomas Count
disadvantages of elite education
do schools matter
Doug Ready
Doug Staiger
dropout factories
dropout rate
dropouts
education books
education policy
education policy thinktanks
educational equity
educational research
educational triage
effects of neighborhoods on education
effects of No Child Left Behind
effects of schools
effects of Teach for America
elite education
ETS
Everyday Antiracism
excessed teachers
exit exams
experienced teachers
Fordham and Ogbu
Fordham Foundation
Frederick Douglass High School
Gates Foundation
gender
gender and education
gender and math
gender and science and mathematics
gifted and talented
gifted and talented admissions
gifted and talented program
gifted and talented programs in New York City
girls and math
good schools
graduate student union
graduation rate
graduation rates
guns in Chicago
health benefits for teachers
High Achievers
high school
high school dropouts
high school exit exams
high school graduates
high school graduation rate
high-stakes testing
high-stakes tests and science
higher ed
higher education
highly effective teachers
Houston Independent School District
how to choose a school
IES
incentives in education
Institute for Education Sciences
is teaching a profession?
is the No Child Left Behind Act working
Jay Greene
Jim Liebman
Joel Klein
John Merrow
Jonah Rockoff
Kevin Carey
KIPP
KIPP and boys
KIPP and gender
Lake Woebegon
Lars Lefgren
leaving teaching
Leonard Sax
Liam Julian

Marcus Winters
math achievement for girls
McGraw-Hill
meaning of high school diploma
Mica Pollock
Michael Bloomberg
Michelle Rhee
Michelle Rhee teacher contract
Mike Bloomberg
Mike Klonsky
Mike Petrilli
narrowing the curriculum
National Center for Education Statistics Condition of Education
NCLB
neuroscience
new teachers
New York City
New York City bonuses for principals
New York City budget
New York City budget cuts
New York City Budget cuts
New York City Department of Education
New York City Department of Education Truth Squad
New York City ELA and Math Results 2008
New York City gifted and talented
New York City Progress Report
New York City Quality Review
New York City school budget cuts
New York City school closing
New York City schools
New York City small schools
New York City social promotion
New York City teacher experiment
New York City teacher salaries
New York City teacher tenure
New York City Test scores 2008
New York City value-added
New York State ELA and Math 2008
New York State ELA and Math Results 2008
New York State ELA and Math Scores 2008
New York State ELA Exam
New York state ELA test
New York State Test scores
No Child Left Behind
No Child Left Behind Act
passing rates
Pearson
picking a school
press office
principal bonuses
proficiency scores
push outs
pushouts
qualitative educational research
qualitative research in education
quitting teaching
race and education
racial segregation in schools
Randall Reback
Randi Weingarten
Randy Reback
recovering credits in high school
Rick Hess
Robert Balfanz
Robert Pondiscio
Roland Fryer
Russ Whitehurst
Sarah Reckhow
school budget cuts in New York City
school choice
school effects
school integration
single sex education
skoolboy
small schools
small schools in New York City
social justice teaching
Sol Stern
SREE
Stefanie DeLuca
stereotype threat
talented and gifted
talking about race
talking about race in schools
Teach for America
teacher effectiveness
teacher effects
teacher quailty
teacher quality
teacher tenure
teachers
teachers and obesity
Teachers College
teachers versus doctors
teaching as career
teaching for social justice
teaching profession
test score inflation
test scores
test scores in New York City
testing
testing and accountability
Texas accountability
TFA
The No Child Left Behind Act
The Persistence of Teacher-Induced Learning Gains
thinktanks in educational research
Thomas B. Fordham Foundation
Tom Kane
Tweed
University of Iowa
Urban Institute study of Teach for America
Urban Institute Teach for America
value-addded
value-added
value-added assessment
Washington
Wendy Kopp
women and graduate school science and engineering
women and science
women in math and science
Woodrow Wilson High School