Much has been heralded about the research possibilities inherent in the massive clickstream databases produced by MOOC platforms. If these databases, however, fail to capture important data about students’ learning experiences, then having vast quantities of data will not compensate for having the wrong data. Even in the era of big data, good research depends on good design.
To draw attention to these design considerations, in this presentation I will present a taxonomy of four kinds of MOOC research:
• fishing in the exhaust: post-hoc analyses of courses designed and taught without any particular research agenda, yielding descriptive, observational insights
• experiments in the periphery: A/B type experiments of particular pedagogical approaches that test domain-independent teaching strategies (e.g. commitment devices, priming experiments, or motivational messages) that can be conducted
• observations in the field: qualitative research from interviews and observations that seeks to investigate the contexts in which MOOC students learn and the ways in which they make meaning of their experiences
• design research in the core: design research conducted in partnership with course faculty, instructional designers, and learning scientists to investigate pedagogical questions that attend to core issues in the design of a learning environment (e.g. the effect of using block-based programming languages in introductory computer science courses).
The past year of MOOC research has been characterized by “fishing in the exhaust,” and while some useful initial descriptions of user behavior have emerged from these approaches, there are strict limits to what we can learn from observational research (even observational research with lots of data). The coming year of MOOC research will see more fishing, but also a growth of experiments in the periphery. These kinds of experiments are typically dreamed up by educational researchers and not by course faculty, and as a result they explore issues that are potentially interesting but not central to the learning goals of any particular course. These studies are easy to facilitate in the complex ecosystem of MOOC production, but they may not investigate the most important domains of large-scale online learning.
In order for MOOC research to reach its full potential, the field needs stronger lines of research in both qualitative research and in design research in the pedagogical core of our courses.
Powerful design processes in education begin with empathizing with the learner, and at present we have trillions of cells of data about what people have clicked and we know very little about what has changed in their heads. While the best qualitative research will be conducted by trained specialists, a wide variety of roles in the MOOC production ecosystem—producers, developers, instructional designers, teaching assistants, and researchers—can and should contribute to this important work.
With a better sense of the needs of our learners, the most important and impactful research that will be done with MOOCs will be design research in the core. This difficult work will require close collaboration among teams of diverse researchers and practitioners, but it will investigate questions that pertain directly to the central pedagogical theory in a course or discipline.
Participants will leave the session with both a richer understanding of the importance of research design, and ideas about how to create MOOC production workflows and ecosystems that support ambitious, collaborative research.