Testing Theories of Goal Progress within Online Learning

Joy Lu, Eric T. Bradlow, J. Wesley Hutchinson

Online educational platforms like Coursera and Khan Academy make education accessible by allowing learners to progress at their own pace within opt-in and on-demand structures. An advantage of this online learning setting is that we can track learner activity within different courses, including watching lectures and taking quizzes, and look at how the motivation to consume course content varies over time. Using data from thousands of learners across four introductory online business courses, we wanted to investigate three main questions in our research: (1) Do any distinctive patterns or “learning styles” emerge? (2) How do different learning styles relate to important downstream outcomes such as final performance and enrollment in additional courses on the platform? (3) Are there any differences in behavior between learners who had paid or not paid for the option of a course certificate?

First, to detect different learning styles, we estimated a model that captured how individual learners chose to consume content over time. Specifically, by allowing the utility of watching lectures to vary as a quadratic function of current progress, our model reveals whether learners consume lecture content faster, slower, or non-monotonically as they approach the end goal. Increasing utility is consistent with the classic goal-gradient hypothesis that predicts people are more motivated as they approach an end reward (Kivetz, Urminsky, and Zheng 2006), while decreasing utility corresponds to decreasing motivation or enjoyment due to satiation (Nelson, Meyvis, and Galak 2009). A U-shaped pattern is consistent with the “stuck-in-the-middle” theory that predicts individuals are motivated towards the beginning and end of a task when there are salient reference points (Bonezzi, Brendl, and De Angelis 2011).

While these 3 shapes have been well-documented in prior literature, we instead find that most learners exhibit an inverted-U or bell-shaped pattern, with one explanation being that when learners first committed to taking the course, they may have overestimated their future “slack” for time resources (Zauberman and Lynch 2005), resulting in an initial rise and later fall in motivation (see attached Figure). We also determine the number of weeks of learner data required to detect this inverted-U pattern, demonstrating that our model may serve as an early detection system of individuals who are likely to exhibit a decrease in engagement towards the end of the course and may benefit from motivation reminders.

Second, we examine how different learning styles relate to performance outcomes and future activity. For example, while learners who exhibit a monotonic increase in utility (i.e., goal-gradient) get lower final grades compared to those who exhibit the more common inverted-U (i.e., resource slack), they make more total lecture progress and take more additional courses later on. One interpretation is that the ending increase of the goal-gradient pattern leads to momentum to take additional courses in the future.

Finally, at the time of the data, all lectures and quizzes were available to both learners who had paid and not paid for the option of a course certificate. Interestingly, we find that non-paying learners avoided taking quizzes and, consistent with these findings, the most recent versions of many online courses are designed such that quizzes are only available to paid learners, thus adding differentiating value to the paid course option via the “extrinsic” motivation to take quizzes to earn the certificate.

In summary, due to the self-pacing of online courses, online learners may exhibit a variety of learning styles, depending on their own preferences and the course structure. We find that online learners in four introductory business courses tend to exhibit a rise and fall in motivation (inverted-U) when consuming course content. Our model may be used as an early detection system to anticipate changes in engagement, as well as to determine the learning styles that lead to the best performance outcomes or continued engagement on the platform. Changes in learner behavior may also inform the design of personalized nudges or restructuring course content and assessments to improve motivation throughout the course.