deconstructing disengagement: analyzing learner subpopulations in moocs
TRANSCRIPT
Deconstructing Disengagement:Analyzing Learner Subpopulations in
Massive Open Online Courses
René
Kizilcec
Chris
Piech
Emily
Schneider
MOOCs (in this paper) are
instructionist + individualised
• 6-10 weeks long
• 2-3 hours of video lectures/week
• autograded assessments with regular
deadlines
• discussion forum
Massive Open Online Courses
Heterogeneous population:
Learners join from anywhere in the
world, at any age, for any reason
Defining Success for Open-Access Learners
Assessment scores are problematic:
• not comparable across courses
• not available for all learners because
test-taking is not aligned with learner
goals
Defining Success for Open-Access Learners
Completion rates are highly problematic:
• numerator = certificate earners, i.e. learners who take assessments
• denominator = o total enrolled? overestimate; indicator of
interest and not participation
o total active? how defined?
• ignore plurality of learner intentions
• no nuance about subpopulations to help us design interventions or customized course features
Process measures hold promise:
• conceptualize learning as an ongoing
set of interactions with learning objects
and other humans
• allow early detection and prediction
• indicate points for intervention
Defining Success for Open-Access Learners
Defining Success for Open-Access Learners
Completion rates
Assessment scores
Process measures
How to classify learners into
meaningful subpopulations?
Classification Criteria
Classification methods for MOOC subpopulations:
Universal – valid across multiple courses
Theory-driven – reflect the processes of learning
Parsimonious – based on small, meaningful feature set
Predictive – suggest likely outcomes
Dynamic – account for new information over time
Lens for Analysis
• Compare subpopulations
• Compare courses
The Data
Analyzed Three Courses
Who took these MOOCs?
A lot of data!
Gender skew
Interesting age group
HDI skew
Clustering
Sub-populations basis?
Engaged
Not Engaged
Engagement Ideal
Time
Enga
gem
ent
Coarse Engagement Labels
(T) On Track: Did the weekly assignment on
time
(B) Behind: Did the weekly assignment, but
finished after the due date
(A) Auditing: Watched videos but did not do
the assignment
(O) Out: Did not interact with the course,
either through videos or assignments
We were able to predict who would take the final AUC = 0.96
The Aggregate Class A = AuditingO = OutT = On TrackB = Behind
In this picture Out Is not to scale!
The Aggregate Class A = AuditingO = OutT = On TrackB = Behind
5k
In this picture Out Is not to scale!
The Aggregate Class A = AuditingO = OutT = On TrackB = Behind
In this picture Out Is not to scale!
7k
Example Student 1
Example Student 2
Example Student 3
Clustering Methodology
There were 21,108 paths in the
GS class
Four Prototypical Trajectories
Cluster!
(k-means of L1 norm)
Four Prototypical Trajectories
And?
The Four Prototypical Trajectories
Prototypical Trajectory 1: Completing
Prototypical Trajectory 2: Auditing
Prototypical Trajectory 3: Disengaging
Prototypical Trajectory 4: Sampling
Four Prototypical Trajectories
Consistent across three courses:
Auditing learners watch lectures throughout course, but
attempt very few assessments
Completing learners attempt majority of assessments offered
in course
Disengaging learners attempt assessments at beginning of the
course, but then sparsely watch lectures or disappear entirely
Sampling learners briefly explore course by watching a few
videos
Four Prototypical Trajectories
The other courses?
Four Prototypical Trajectories
Four Prototypical Trajectories
<suspense>
Four Prototypical Trajectories
Four Prototypical Trajectories
Same pattern in all classes
HS Composition [46k]C
om
ple
tin
g
Completing
Sampling
Disengaging
Auditing
UG Composition [27k]C
om
ple
tin
g
Dis
en
ga
gin
g
Sampling
Auditing
MS Composition [21k]C
om
ple
tin
g
Dis
en
ga
gin
g
Auditing
Sampling
Validation
Cluster Validation
• Different values of k (split by time)
• Including “assignment pass” (95%
overlap)
• Excluding “behind” (94% overlap)
• Silhouette of 0.8 (that’s pretty good)
• Pass the common sense test
High Level
Clustering
Engagement in
MOOCs
Four
Prototypical
Patterns
Results &
Recommendations
Comparing Trajectories
between Courses
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
HS
UG
GS
3.0 3.5 4.0 4.5 5.0
Overall Experience
Completing (and Auditing)
have best experience
Overall Experience
Identify subpopulations early
to customize course features
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
HS
UG
GS
0.1 0.51.0 2.0 4.0 7.0 10.0
Average Forum Activity
Completing learners are
most active on the forum
Discussion Forum
Reputation systems &
Social features
Causal relationship?
Geographical Distribution
Trend confirmed by top four participating countries
United States, India, Russia, United Kingdom
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
Sampling
Disengaging
Completing
Auditing
HS
UG
GS
2 4 6 8 10 12 14 16
Odds Ratio (Male/Female)
Female Completing learners underrepresented in advanced courses
Gender
Frame assessments to
minimize stereotype threat
Stereotype threat? Spencer et al., 1999
Future Directions
Future Directions Experiments
Collaboration and Peer Effects
Interface Customization
Targeted Interventions
Nuanced Analytics
Auditing: MOOC-as-a-resource vs. MOOC-as-a-class
Disengaging: Early prediction for intervention
Reasons to enroll and trajectories
Engagement trajectories for real-time analytics in MOOCs
Dashboard visualizations
Thank you!
Stanford Lytics Lab lytics.stanford.edu
Office of the Vice Provost for Online Learning
Roy Pea, Clifford Nass, Daphne Koller
Our LAK reviewers
Reference
S. Spencer, C. Steele, and D. Quinn. Stereotype threat and women’s math
performance. Journal of Experimental Social Psychology, 35(1):4–28, 1999.
More info?
René Kizilcec [email protected]
Chris Piech [email protected]
Emily Schneider [email protected]
Stanford’s Learning Analytics Group:
Lytics Lab lytics.stanford.edu
Paper: http://goo.gl/OSX72