The First Year of edX: Research Findings to Inform Online Learning
My colleagues and I from HarvardX and MITx research are very pleased to announce the release of 16 working papers and course reports that summarize our research from the first full academic year of courses on edX. We have one multi-course report that summarizes findings across all of the courses. I was the lead author for three papers that extensively explore the first two HarvardX courses from the Harvard School of Public Health, HeroesX and JusticeX. These reports combine analysis of edX platform log data, with interviews with faculty and course development teams and a close examination of course content. We also released 11 additional course reports for each of the MITx courses from 2012-2013. Finally, Terry Fisher was kind enough to re-release his appraisal of his CopyrightX course as part of our series.
I'll be giving a livestreamed talk at 6pm eastern tonight (UPDATE: this link now has the recording), which will be an opportunity to walk people through some of the findings from these reports. In particular, I'll try to emphasize how MOOCs are best understood from the bottom up rather than from the top down: it's better to know something about each course and its intentions and context before making generalizations across courses. I'll also show some of the new approaches that our team developed for comparing and analyzing courses that get us past simple metrics like numbers of registrants or completion rates.
My colleagues and I are hopeful that this research won't be easily placed in either the camps of MOOC optimists or skeptics, but rather points towards some successes that courses are now enjoying and challenges and opportunities that lay ahead.
Below is the executive summary from the multi-course report, and I hope it whets your appetite to read the other reports. Send along some questions and comments to @bjfr and #harvardx as you read!
- In the year from the fall of 2012 to the summer of 2013, the first 17 HarvardX and MITx courses launched on the edX platform. In that year, 43,196 registrants earned certificates of completion. Another 35,937 registrants explored half or more of course content without certification. An additional 469,702 registrants viewed less than half of the content. And 292,852 registrants never engaged with the online content. In total, there were 841,687 registrations from 597,692 unique users across the first year of HarvardX and MITx courses. (See Table 2.)
- The most typical course registrant is a male with a bachelor's degree who is 26 or older; however, this profile describes fewer than one in three registrants (222,847, 31%). A total of 213,672 (29%) registrants report their gender as female; 234,463 (33%) report a high school education or lower; 45,884 (6.3%) report that they are 50 or older; and 20,745 (2.7%) have IP or mailing addresses from countries on the United Nations list of Least Developed Countries. Small percentages are not small numbers. The diversity of registrants resists singular profiles; registrants are notable for their differences. (See Table 3 and Table 4.)
- Course certification rates are misleading and counterproductive indicators of the impact and potential of open online courses. (See Figure 1.)
- Large numbers of non-certified registrants access substantial amounts of course content.
- Open online registration is not equivalent to enrollment in conventional courses, where traditional enrollment generally entails monetary costs, opportunity costs, and accountability.
- Certification rates can be useful indicators when enrollments are limited. For a fixed number of registrants, higher certification rates accurately reflect larger numbers of certified registrants. For open online courses that support large-scale enrollment, there is no forced tradeoff between numbers of certified and non-certified registrants--both numbers can increase freely by design. In these circumstances, focusing on certification rates alone penalizes desirable activities like browsing and exploring courses, which open online courses are generally designed to support.
- Pressure to increase certification rates may decrease the impact of open online courses, by encouraging instructors and administrators to suppress or restrict registration, lower certification standards, deemphasize recruitment of target subpopulations, or disregard interventions that may disproportionately increase numbers of non-certified registrants over certified registrants.
- There are considerable differences in average demographics across courses, in terms of gender (13%-49% female), college degree attainment (54%-85%), median age (23-30), and percentage from the US (16%-36%). These differences are best appreciated in the context of the diversity of course offerings, the intentions of the instructor teams, and the outreach and dissemination efforts of course teams. In spite of average differences, all large-scale courses had hundreds of registrants with only high school degrees or who are under 15, and also had hundreds of registrants with postdoctoral degrees or who are over 50. (See Figures 3-8, and Tables 2-6.)
- Unlike conventional courses, open online enrollment occurs continuously throughout courses, with enrollment rates rising as course launch dates approach and then declining more quickly after launch dates pass. Exploration and certification is more likely among registrants who enroll near the launch dates, but viewing likelihood is stable through the run of the courses. Course exploration and certification may benefit from synchronous course schedules and the cohorts that they build. Managing asynchronicity to maintain registrant involvement regardless of enrollment date is an ongoing challenge for instructors and a fertile area for future research. (See Table 5, Figure 8, and Figure 9.)
- New metrics, far beyond grades and course certification, are necessary to capture the diverse usage patterns in the data. A simple comparison of grades and viewed content shows thousands of users who fit a range of profiles. Of particular interest may be those students who accessed substantial course content but did not participate in assessments. Metrics include course chapters accessed, forum usage, total numbers of "clicks," and numbers of active days in the course. (See Table 6 and Figure 13.)
- The average percentage of registrants who cease activity in these open online courses is highest in the first week at around 50%. The average percentage of registrants who cease activity in the second week declines sharply to 16% for registrants who persist to that point, and these percentages continue to decline over subsequent weeks. This indicates that registrants who are active after the first week have a relatively high chance of visiting again in subsequent weeks. (See Figure 12.)
- Over four thousand registrants earned more than one certificate across HarvardX and MITx, including 1,912 who earned at least one certificate from both institutions. A total of 76 registrants earned 5 or more certificates from the first 17 courses.