Understanding Student Engagement in MOOCs

Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume Iii and Lise Getoor

The large number of students participating and the availability of interaction data in massive open online courses (MOOCs) provides an opportunity to study and uncover student engagement patterns, develop technology that can help improve student engagement and facilitate instructor interventions. Our work looks at several problems in MOOCs: 1) modeling student engagement to predict course completion, 2) analyzing changes in engagement patterns, and 3) understanding discussion forum content and the relationship to course completion.

When users interact on a MOOC, they leave behind cues suggestive of their engagement with the course and intention to complete the course. We have developed a data-driven model of student engagement in MOOCs using features from users’ interaction with the MOOC, and use that to predict course completion (course survival) [3]. Our model uses behavioral cues (course related activities such as viewing lectures, giving assignments, participating in discussion forum), forum content (po- larity and subjectivity of forum posts), and forum interaction structure to distinguish between forms of student engagement (active and passive). The engagement types are represented as latent variables in our model and are learned from observed data. We then use the latent variables to predict student survival. We use probabilistic soft logic (PSL) [2], to represent ob- served features, (latent and target) variables as logical predicates and construct rules over these to capture domain knowledge. We evaluated our models on predicting learner survival across three MOOCs—Surviving Disruptive Technologies, Women and the Civil Rights Movement and Gene and Human Condition. We demonstrated that incorporating latent engagement variables helps in predicting student survival.

In order to design effective interventions, we need to identify students at a risk of dropping out early-on in the course. We conducted experiments to predict student survival early-on in the course, by training our models on data from the initial part of the course. Our experiments show that our models, especially the latent model, is able to predict student survival reliably at an early stage when compared to the model without latent variables. In addition to improved prediction accuracy, our latent engagement model also unveils interesting patterns in student engagement. Analyzing latent engagement estimates predicted by our model, we find that passive engagement dominates in the beginning and there is an increase in active forms of en- gagement toward the end. Examining the transition between engagement types, we observed that most passive users show an increase in active engagement levels prior to dropping out. This is suggestive of help-seeking/complaining/expressing dissat- isfaction or difficulty in following course materials in discussion forums before dropping out. Probing these forum-posts, we can uncover reasons leading to student disengagement and dropout and identify students that can be helped via intervention. This leads us to perform a more close analysis of forum content.

Our second contribution is looking more closely at discussion forum content for student survival indicators. MOOC discussion forums are the principal means of interaction among MOOC participants. Negative sentiment can indicate dissatisfaction with the class, however it can also be used to express an opinion showing high levels of engagement. Negative sentiment in course related discussions does not imply attitude toward the course and does not mean disengagement, but in logistics or feedback posts it signifies disengagement. Discerning between the two types of sentiment is vital as we are trying to identify students at a risk of dropping out. We make use of recent improvements in topic modeling, Seeded Topic Modeling (SeededLDA) [1], to extract posts corresponding to course-logistics, feedback and course-related material [under review]. We leverage the knowledge of course syllabus and general nature of logistics and feedback posts to seed our model. We enhance our survival models described above with topic distribution from SeededLDA. Our rules capture sentiment and topic of posts to assess signs of engagement and disengagement. We demonstrate that inclusion of features from topic distribution in our survival models helps in predicting student survival.

Our current research focuses on applying our models to courses as they progress and identify possible instructor interventions.

[1] Jagadeesh Jagarlamudi, Hal Daume ́, III, and Raghavendra Udupa. Incorporating lexical priors into topic models. In Proceedings of EACL, 2012.
[2] Angelika Kimmig, Stephen H. Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. A short introduction to proba- bilistic soft logic. In NIPS Workshop on Probabilistic Programming: Foundations and Applications, 2012.
[3] Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume ́, III, and Lise Getoor. Learning latent engagement patterns of students in online courses. to appear in AAAI 2014.

MOOC Learner Motivation and Course Completion

Yuan Elle Wang and Ryan Baker

In this talk, we will discuss our work to investigate the relationships between the motivations of MOOC learners and completion rates, to study the motivational differences between students who complete the course and those who do not.

The research context is the Coursera MOOC “Big Data in Education”, with a total enrollment of over 48,000. As of the end of the course, the pre-course survey (consisting of subscales of the PALS and MOOC-specific items) received 2,792 responses. Our research results show that this combination of survey measures can be useful for studying students’ motivational directions early in a MOOC. Specifically, we find 4 relationships with the potential to inform future intervention for both instructors and learning designers:
• Students who were more interested in the MOOC learning environment as a new platform of learning than aspects related to the course content area will less likely to complete the course. In specific, non-completers rated higher on items such as “Course is offered by a prestigious university”, “Geographically isolated from educational institutions” when asked their reasons for enrolling in this course.
• Mastery-goal orientation and academic efficacy were not useful predictors of whether a learner will successfully complete the course or not. Learners were generally high in mastery-goal orientation, a finding potentially generally characteristic of students who choose to take a MOOC for no formal credit.
• Early self-reported confidence in completing the course was a successful indicator of whether a student will complete the course or not, with more confident students being more likely to complete.
• Students self-identified as non-English native speakers exhibited higher confidence in completing the course than native speakers, from the beginning of the course. However, this difference was not linked to a difference in course completion.

In considering these findings, it is worth noting that course completion is only one of many metrics that can be used to study persistence and learning in the context of MOOCs; many students enter MOOCs with the goal of learning a specific subset of course material rather than completing the course. Nonetheless, our results suggest that initial student motivation can play an important role in whether students persist in and complete a MOOC.

Achieving Learning Objectives Online: Not All Platforms are Equal

Nick Feamster

Over the past twelve months, I have taught two Massive Open Online Courses (MOOCs):
● A Coursera course on Software Defined Networking, to over 50,000 enrolled students, 4,000 of whom
completed all programming assignments; and
● A Udacity course on graduate computer networking, as part of Georgia Tech’s initial offering of the Online
Masters in Computer Science program

In both cases, I also used the videos as content for “flipped” versions of the on­campus versions of the courses at Georgia Tech (a graduate seminar for the SDN course; and a large graduate course for the latter).

Although the syllabus of each course differed, the courses have several overlapping assignments and elements, particularly on topics related to software defined networking. Despite the overlap in content and assignments in both courses, the modes of delivery and the amount of support that each platform offers for course development differ significantly. I have noted some of the following salient differences:
● Video delivery. Coursera uses a “slides and talking head” model for preparing videos that is largely “do it yourself”; Udacity uses a full production team to produce letters with a “voice overlay over whiteboard/writing hand”.
● Forums. Coursera uses a “home brew” forum setup, whereas Udacity relies on more polished third­party forum software, such as Piazza.
● Pacing. The courses are paced differently: Coursera uses a “synchronous” model, whereas Udacity uses a “go at your own pace through all of the course material” model.
● Demographics. The makeup of the Coursera course was largely students who enrolled in the course for their own enrichment, as no other credit was offered other than a “completion certificate”. On the other hand, as the Udacity course is part of a (paid) online degree program, I found students to be more “conventional” in their demands (and sense of entitlement).

I am still forming my conclusions about each of these platforms, and I think I can offer some fairly strong (and probably controversial) opinions about these platforms, and the pedagogical challenges with offering a degree online. In summary, I have found the Coursera course I have offered infinitely more enjoyable (and I think more effective as a course), for many reasons:
● One of the features of a MOOC that allow it to scale is that students help each other out, rather than relying excessively on TA or professor support. In the Coursera course, students enrolled for their own enrichment and thus “self selected” because they were there to learn the topic. This type of learning attitude allows a MOOC to scale much better than when students have more conventional “entitled” attitudes of paying for a course (and, hence, expecting instructor response). Independent of how MOOCs are financed, I have concluded one thing: students who pay “tuition” expect a level of service and responsiveness that may not scale. There need to be other, better ways of financing MOOCs.
● Along the lines of the previous point, the asynchronous “go at your own pace” model does not scale: Students can’t help each other out, and (busy) professors cannot context­switch quickly enough to provide in depth technical answers to student questions. In the Coursera course, students were all in the same place and could help each other out. But, perhaps even more importantly, my head was at one point in the course and I was much more capable of providing detailed answers to questions (often even reading code). Such a level of support is simply not possible in the asynchronous go­at­your­own­pace model.
● Content is king, and substance trumps style. I made all of the Coursera videos from a laptop camera, from the comfort of my own home. I edited them myself with Camtasia. I “scripted” each lecture the morning that I recorded it. I had no production staff. I am firmly convinced that the output from those videos was just as good as the over­produced content that was required from the in­studio Udacity lecture recordings. One thing in particular was that I found the Udaicty recording setup overly cumbersome: the tendency to encourage the “whiteboard style” made it particularly hard to do “on screen demos”, something that is critical in the type of things that we teach in networking courses.

In keeping with the theme of the workshop, I can provide examples of the same content delivered in both settings. We can look at a Udacity video and a Coursera video describing the same content. I can show examples of the forums from both courses and how the organization and response to questions different in each case. I can also show some examples of recording and production. One takeaway I would like to emphasize is that we should not be generalizing so much about what MOOCs can (or cannot) do. We should be paying particular attention as well to some of the finer points, such as the design of the platform and modes of delivery. My experiences may not generalize: it may be the case that other instructors might have exactly the opposite experience when comparing different platforms. However, I offer my two different experiences as a concrete starting point for discussion.

Diana Oblinger

Dr. Diana G. Oblinger is President and CEO of EDUCAUSE, a nonprofit association whose mission is to advance higher education through the use of information technology. The current membership comprises over 2,400 colleges, universities and, education organizations. Previously, Oblinger held positions in academia and business.

Oblinger is known for her leadership in teaching, learning, and information technology. For example, Oblinger created the EDUCAUSE Learning Initiative (ELI), known for its innovation in learning and learning technologies, as well as launching the Next Generation Learning Challenges with the Bill Melinda Gates Foundation. She is the author or co-author of many books, articles and monographs, including Game Changers: Education and Information Technologies, Educating the Net Generation, and What Business Wants from Higher Education.

Oblinger serves on a variety of boards and has received several awards and honorary degrees.

Mark Lester

Mark Lester is Global Head of Partnerships at FutureLearn, the UK-based massive social learning platform, and a member of its Executive team. Prior to joining FutureLearn, Mark headed the strategy development unit at the British Open University, has held senior management positions in the financial services sector and central Government, and was a managing consultant for several years at Monitor Group advising organisations on innovation, industry competitiveness, business strategy and healthcare policy. Mark holds a Masters of Science degree and a Bachelor of Science degree from the LSE and trained as a teacher at the Institute of Education, London. He is married with two children.

Melissa Loble

Melissa Loble is senior director of Canvas Network, a platform where academic institutions can offer open, online courses, including MOOCs. Melissa oversees strategy, design, and implementation, with a focus on ensuring highly engaging and effective learning experiences.

Previously, Melissa was associate dean for Distance Learning at the University of California, Irvine where she provided leadership in curriculum development, instructional design, and the selection and use of educational platforms and technologies. She led multiple projects resulting in the delivery of 13 MOOCs, including a course focused on themes from a popular television show, Society, Science, Survival: Lessons from AMC’s The Walking Dead and her own course, Emerging Trends and Technologies in Virtual K12 Education.

Melissa has held senior leadership roles for a number of educational technology companies and has taught in Pepperdine University’s Masters in Learning Technologies program (MALT) for the past ten years. Her classes have included Technology and Curriculum, Managing Technology for Change, and Mentoring Team Leadership.

She holds master’s degrees in business administration and educational policy from Columbia University and a bachelor’s degree in political science from the University of California, Los Angeles.

Vivek Goel

Dr. Goel is responsible for for working with partner institutions to foster development of course content for the platform. Prior to joining Coursera he was the Founding President and CEO of Public Health Ontario, a government agency dedicated to protecting and promoting the health of Ontario residents. Previously he was Vice-President and Provost at the University of Toronto where he continues as a Professor at the Dalla Lana School of Public Health and the Institute for Health Policy, Management and Evaluation. He has extensive experience in health care evaluation and research, health policy and management, and information systems.

Dr. Goel received his medical degree from McGill University, and completed specialty training in Public Health and Preventive Medicine at the University of Toronto. He received his Master of Science in health administration from the University of Toronto and his Master of Science in biostatistics from the Harvard School of Public Health. He is a Fellow of the Canadian Academy of Health Sciences and has been involved in a broad range of governance activities with academic and public sector organizations.

Anant Agarwal

Anant Agarwal is the CEO of edX, an online learning destination founded by Harvard and MIT. Anant taught the first edX course on circuits and electronics from MIT, which drew 155,000 students from 162 countries. He has served as the director of CSAIL, MITs Computer Science and Artificial Intelligence Laboratory, and is a professor of electrical engineering and computer science at MIT. He is a successful serial entrepreneur, having co-founded several companies including Tilera Corporation, which created the Tile multicore processor, and Virtual Machine Works.

Anant won the Maurice Wilkes prize for computer architecture, and MITs Smullin and Jamieson prizes for teaching. He holds a Guinness World Record for the largest microphone array, and is an author of the textbook Foundations of Analog and Digital Electronic Circuits.

Scientific American selected his work on organic computing as one of 10 World- Changing Ideas in 2011, and he was named in Forbes list of top 15 education innovators in 2012. Anant, a pioneer in computer architecture, is a member of the National Academy of Engineering, a fellow of the American Academy of Arts and Sciences, and a fellow of the ACM.

He hacks on WebSim, an online circuits laboratory, in his spare time. Anant holds a Ph.D. from Stanford and a bachelors from IIT Madras. Anants twitter handle is @agarwaledu.

Open Edx Annotation tools: breaking the unidirectionality of online course content

Philip Desenne, Leah Reis-Dennis

The open Annotation and Tagging Framework offers multiple means of engagement with the edX course material and introduces new models of online, peer-to-peer and student-instructor interactions inside the platform. Digital annotation tools allow contextual commentaries and conceptual tagging of media fragments inside MOOCs, thereby transforming the unidirectional delivery of the online course content.

In 2013-14, HarvardX produced three interoperable media-rich annotation tools to annotate text passages, video clips, and deep-zoom, high definition images inside the edX platform. All annotations can be aggregated under one section of the platform, where students can browse and review their own notes, as well as the instructors’ and other students’ contributions. All notes maintain hyperlinks to their original annotated fragment on the corresponding target page.

HarvardX initially piloted the annotation framework for the poetry course module AI 12.2x “Poetry in America: Whitman.” The module sought to introduce students to the poetry of Walt Whitman, exposing students to Whitman’s iconic long form poetry in the context of his life and times in the mid-19th-century United States. Students had opportunities to practice and refine the way they read, and write about, poems, through interpretive expeditions and interactive exercises.

The capstone of student engagement in this module was digital annotation, allowing students to virtually annotate assigned poems much in the way that students would be ask to annotate poems by hand in a brick-and-mortar classroom. Unlike traditional pen-and-paper annotation, the Annotation Tool enables students to view and interact with each other as they annotate and explore a poem, making the study of poetry a conversation instead of a solitary endeavor.

The pedagogical goals for the poetry annotation exercises included active student reading instead of passive reading and a deepened understanding of analytical modes and poetic devices. Students were invited to actively annotate the poems by noticing and tagging remarkable poetic devices, highlighting and reflecting on unusual or challenging passages, and, throughout, recording their own processes of interpreting language.

Annotation data from the poetry module revealed close participant engagement with the poems, to the tune of nearly 60,000 annotations on passages from 14 poems, generated by 1,000 students, over a span of seven weeks. On average students were very active, creating over a 1000 annotations a day and over 3500 annotations per poem passage, with some individuals contributing up to 700 total annotations.

The Annotation Framework lays the groundwork for a much-needed method of assessing the Humanities at scale. It helps monitor quality and level of engagement of the individual, of groups, and of the entire student population. At the individual level it empowers mechanisms for formative assessment (just-in-time teaching). At the massive scale, it promotes community- building and fosters crowdsourcing to enhance learning though collaborative research and discovery.