Earlier this month, the Spencer Foundation announced $2 million worth of grants to five different teams to develop new measures of the quality of civic and political engagement among young people. I’m on a team, along with Brandon Stewart of Princeton and Dustin Tingley at Harvard, that is investigating the degree to which students with different political beliefs engage with ideologically diverse peers in MOOCs. Here’s the introduction to our grant application, that describe the opportunity that we see in open online learning, and our proposals for new measures to better take advantage of those opportunities.
Political theorists have long argued that exposure to diverse perspectives is vital to a robust civil society and to the development of individuals within those societies (Kahne, Middaugh, Lee, & Feezell, 2012). Democratic discourse requires engaging with people who hold different perspectives on issues, and differing perspectives are often a function of different life circumstances. One serious threat to contemporary civic education, therefore, is the growing homogeneity of school populations. As residential neighborhoods are increasingly de facto segregated by race and class, students increasingly find themselves in schools with other students similar to themselves (Orfield, Kucsera, & Siegel-Hawley, 2012).
Online education offers one pathway for students to join communities of diverse learners beyond the bounds of geography (Reich, Romer, & Barr, Forthcoming). Consider, for instance, the JusticeX open online course taught by Michael Sandel through the edX platform (Reich et al., 2014). JusticeX examines political philosophy and moral reasoning, and in its first run in 2013 nearly 80,000 students from over 180 countries registered for the course. Students watched lectures, read cases, and participated in online discussion forums where they debated moral dilemmas with peers around the world.
A globally-diverse online learning community, however, does not guarantee that students encounter and consider different perspectives. Internet researchers have posed two competing theories for how people confront differences on the Web (Gardner & Davis, 2013). One theory holds that the Internet is a series of silos where individuals seek out media and communities that conform to their established beliefs (Parser, 2012). Another theory holds that the Internet contains many interest-driven spaces that serve as ideological bridges (Rheingold, 2000), where people attracted to these interest-driven spaces can be diverse in many dimensions. At present, we know little about which of these theories best characterizes open online courses like JusticeX. We know that many diverse students enrolled in JusticeX; we do not know whether those students directly engaged with others holding different beliefs.
In this study, we seek to develop new measures that would allow instructors to evaluate in real-time whether students in online course forums are building siloes or bridges. These measures will build upon previous work examining open classroom climate (Campbell, 2008; Kahne, Crow, & Lee, 2013), but differ in that our measures will draw upon event logs from online platforms rather than survey self-reports. We will use the extremely granular data collected by open online courses to closely examine two specific dimensions of engagement across difference: network shape and thematic convergence.
First, we will use social network analysis to describe the shape of interactions among people with diverse political beliefs in HarvardX courses. From analysis of the connections between posts and replies in forum threads, we will evaluate the degree to which students in forums respond directly to students with differing opinions versus forming separate conversational siloes containing only like-minded students.
Second, we will use text analysis to evaluate whether the use of language in online forums converges or diverges among students with different political beliefs. Even if students with different beliefs form integrated networks, it may still be possible for students to use siloed language and to talk past one another. For instance, in a conversation about guns in American society, conservatives might address Second Amendment rights whereas liberals might discuss issues of public health. We hypothesize that in high-quality discussions where students directly address each other’s ideas, computational text analysis will reveal more modest differences in the language used by people with opposing political beliefs, and we hypothesize that in lower-quality discussions, these gaps in language use will be wider.
From these two analyses, we propose a two-by-two matrix to summarize possible dimensions of engagement across difference in online discussion forums, shown in Figure 1. The bottom left quadrant includes forums where people with different political beliefs separate into different network clusters and use different language; these are the echo chambers of Internet discourse. The top left quadrant includes integrated threads where partisans use different language; these are spaces where students with different beliefs talk past one another. In the bottom right quadrant, students with different political beliefs discuss topics using a shared language, but they divide themselves into conversational siloes with like-minded others. In the top right quadrant is our hypothesized ideal condition of deliberative discourse, where people with diverse beliefs engage in discussion using a common vocabulary.
Figure 1: Two-by-two schematic of dimensions of engagement in online discussion forums based on whether thread participants include students with diverse political beliefs (integrated threads) or only those with like-minded beliefs (siloed threads) and whether language use in the thread differs substantially by students’ political beliefs (divergent use of language) or if a shared language emerges among students with different beliefs. (convergent use of language).
We hypothesize that these two computational measures of discursive practices will correlate with established measures of open classroom climate as well as instructor perceptions of the quality of online discussions. We predict that in courses with high quality discussions, event log analysis will show that students in diverse networks communicate directly with each other using shared language, surveys will show that students report open classroom climates, and instructor analyses will report positive perceptions of discussion quality. If our computational measures prove to be useful indicators of high quality discussion, then our research will point the way towards the development of instructor dashboards that can provide real-time indicators of the civic health of online discussions.
This research opportunity comes at a pivotal moment in the short history of massive open online courses (MOOCs) and the much longer history of online education. Although online courses have been available to students for decades, the scale of MOOC enrollment and their adoption by elite universities has dramatically raised both public and institutional attention to online learning. As occurs in all sectors of education, there is constant tension between the many possible purposes of MOOCs: as online job and workforce skill training, as a stimulating leisure activity for lifelong learners, as an entry point into advanced education for the underserved, or as a modern Agora where citizens from a global commonweal can learn and share together. Especially given the vast venture and philanthropic capital invested in MOOC platforms, there is tremendous pressure among MOOC providers to “pivot” (in the parlance of Silicon Valley) towards workforce training courses with a high likelihood of near-term financial return. A major research investment in the civic purposes of open online courses, and the development of new measures for the civic health of MOOCs, will support the advocacy of those who work to ensure that the civic mission of schools remains vital as higher education evolves in the digital age.
References
Allen, I. E., & Seaman, J. (2014). Grade change: Tracking online education in the united states. Working Paper. Babson Survey Research Group.
Brace, P., Arceneaux, K., Johnson, M., & Ulbig, S. G. (2004). Does state political ideology change over time? Political Research Quarterly, 57(4), 529-540. doi:10.1177/106591290405700402
Campbell, D. E. (2008). Voice in the classroom: How an open classroom climate fosters political engagement among adolescents. Political Behavior, 30(4), 437-454.
Christakis, N. A., & Fowler, J. H. (2009). Connected: The surprising power of our social networks and how they shape our lives New York: Hachette.
Gardner, H., & Davis, K. (2013). The app generation: How today’s youth navigate identity, intimacy, and imagination in a digital world. New Haven, CT: Yale University Press.
Goldenberg, A., Zheng, A. X., Fienberg, S. E., & Airoldi, E. M. (2010). A survey of statistical network models. Foundations and Trends in Machine Learning, 2(2), 129-233.
Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, Fall 2012-Summer 2013. ( HarvardX & MITx Working Paper No. 1).
Kahne, J., Crow, D., & Lee, N. (2013). Different pedagogy, different politics: High school learning opportunities and youth political engagement. Political Psychology, 34(3), 419-441. doi:10.1111/j.1467-9221.2012.00936.x
Kahne, J., Middaugh, E., Lee, N., & Feezell, J. T. (2012). Youth online activity and exposure to diverse perspectives. New Media & Society, 14(3), 492-512. doi:10.1177/1461444811420271
Kane, T. (2013) Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73.
Knoke, D., & Yang, S. (2008). Social network analysis Thousand Oaks, CA: Sage.
Newman, M. E. (2002). Assortative mixing in networks. Physical Review Letters, 89(20), 208701.
Orfield, G., Kucsera, J., & Siegel-Hawley, G. (2012). E pluribus... separation: Deepening double segregation for more students. Working Paper. Los Angeles: Civil Rights Project/Proyecto Derechos Humanos.
Parser, E. (2012). The filter bubble: How the new personalized Web is changing what we read and how we think. New York: Penguin.
Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., Chuang, I., & Ho, A. D. (2014). JusticeX: Spring 2013 course report. (HarvardX Working Paper No. 4). Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2382248
Reich, J., Romer, A., & Barr, D. J. (Forthcoming). Dialogue across difference: A case study of Facing History and Ourselves’ Digital Media Innovation Network. In E. Middaugh, & B. Kirshner (Eds.), Becoming political in a digital age. Charlotte, NC: Information Age.
Reich, J., Tingley, D., Leder-Luis, J., Roberts, M. E., & Stewart, B. M. (2014). Computer-assisted reading and discovery for student generated texts in MOOCs. Working Paper. Retrieved from http://scholar.harvard.edu/files/dtingley/files/educationwriteup.pdf
Rheingold, H. (2000). The virtual community: Homesteading on the electronic frontier. Cambridge, MA: MIT.
Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder‐Luis, J., Gadarian, S. K., . . . Rand, D. G. (2014). Structural topic models for Open‐Ended survey responses. American Journal of Political Science