Examining engagement:

analysing learner subpopulations in

massive open online courses (MOOCs)

Rebecca Ferguson, Doug Clow

What are MOOCs?

● Massive

thousands may sign up

● Open

no payment is required

● Online

resources on the Internet

● Courses

time-bounded cohorts

Commonalities of scale, economic/philosophical perspective,

location and structure – but not pedagogy

Current context

% complete from: www.katyjordan.com/MOOCproject

Students seek not merely

access, but access to success“ ”John Daniel, 2012

Patterns of engagement: Coursera

● Sampling

learners explored some course materials

● Auditing

learners watched most videos, but

completed assessments rarely, if at all

● Disengaging

learners completed assessments at the

start of the course and then reduced

their engagement

● Completing

learners completed most assessments

Kizilcec, R., Piech, C., and Schneider, E., 2013. Deconstructing disengagement:

analyzing learner subpopulations in massive open online courses. LAK13

MOOC designers can apply this

simple and scalable categorization

to target interventions and develop

adaptive course features“”

5

Replication

FutureLearn data

6

Calculating an activity profile

Replicating the method

● T = on track (3)

undertook the assessment on time

● B = behind (2)

submitted the assessment late

● A = auditing (1)

engaged with content but not assessment

● O = out (0)

did not participate

7

Replication

Identifying dissimilarity between engagement patterns

Assigned numerical value to each label

• On track = 3

• Behind = 2

• Auditing = 1

• Out = 0

Calculated L1 norm for each

engagement pattern

Used that as the basis for 1-dimensional

k-means clustering

Repeated clustering 100 times and

selected solution with highest likelihood

Focused on extracting four clusters

8

Replication

Coursera and FutureLearn results were different

● Sampling

learners explored some course materials

● Auditing

learners watched most videos, but completed assessments rarely, if at all

● Disengaging

learners completed assessments at the start of the course and then reduced

their engagement

● Completing

learners completed most assessments

√

√

x

x

9

Exploring the method

Trying different approaches

● Different values for k

explored values for k between 3 and 8 (silhouette width was at a minimum for k=4

– suggesting this was the least suitable value)

● One-dimensional approach

might discard potentially useful information about patterns of engagement before

they can be used by the clustering algorithm

● Ran k-means on the engagement profiles directly

treating them as 6- or 8-dimensional vectors (some courses were six weeks long,

and some courses were eight weeks long).

Explored with k=4 and again found Samplers and Completers

Explored with k=3 to k=8 – less successful than one-dimensional approach

10

FutureLearn is different

Pask, Gordon. (1976).

Conversation Theory:

Applications in Education

and Epistemology.

New York: Elsevier.

11

FutureLearn is different

Conversation is a central feature

12

Revising the numeric values

13

Patterns of engagement

Patterns vary with pedagogy and learning design

On an eight-week MOOC:

● Samplers visit only briefly

● Strong Starters do first assessment

● Returners come back in Week 2

● Mid-way Dropouts drop out mid-way

● Nearly There drop out near the end

● Late Completers finish

● Keen Completers do almost everything

14

Typical engagement profiles

These profiles apply to an eight-week course

● Samplers visit only briefly

[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content

● Strong Starters do first assessment

[9, 1, 0, 0, 0, 0, 0, 0] – 9 means they visited content and did assessment on time

● Returners come back in Week 2 [9, 9, 0, 0, 0, 0, 0, 0]

● Mid-way Dropouts

[9, 9, 9, 4, 1, 1, 0, 0] – 4 means they submitted assessment late

● Nearly There drop out near the end

[11, 11, 9, 11, 9, 9, 0, 0] – 11 means full engagement, 8 means submission on time

● Late Completers finish

[5, 5, 5, 5, 5, 9, 9, 9] – 5 means they viewed content and submitted late

● Keen Completers do almost everything [11, 11, 9, 9, 11, 11, 9, 9]

15

Samplers

[1, 0, 0, 0, 0, 0, 0, 0] – 1 means they visited content

● The largest group in all MOOCs

● Typically accounted for 37% – 39% of learners

● Visited the materials, but only briefly

● Active in a small number of weeks

● 25% – 40% joined after Week 1

● Very few Samplers posted comments (6% – 15%)

● Almost no Samplers submitted any assessment

Highly stable across all MOOC and across most values of k

16

Strong starters

[9, 1, 0, 0, 0, 0, 0, 0]

● All Strong Starters submitted the first assignment

● Engagement dropped off sharply after that

● A little over a third of them posted comments

● Typically posted fewer than four comments

Highly stable across all MOOCs and across most values of k

17

Returners

[9, 9, 0, 0, 0, 0, 0, 0] – came back in Week 3

● Completed the assessment in the first week

● Completed the assessment in the second week

● Then dropped out

● Over 97% completed those two assessments, although some submittted late

● No Returner explored all course steps

● Average amount of steps visited varied (23% – 47%)

This cluster did not appear on MOOC3, which had three widely spaced assessments, so this

engagement pattern was not possible

18

Mid-way Dropouts

[9, 9, 9, 4, 1, 1, 0, 0]

● A much smaller cluster (6% of learners on MOOC1, 7% on MOOC4)

● These learners completed three or four assessments

● They dropped out around halfway through the course

● Mid-way dropouts visited about half the steps on the course

● Just under half posted comments

● Posted just over six comments on average

This cluster did not appear on MOOC2 and MOOC3,

because of the spacing of their assessments

19

Nearly There

[11, 11, 9, 11, 9, 9, 0, 0]

● Another small cluster (5% – 6% of learners)

● Consistently completed assessments

● Dropped out just before the end of the course

● Visited around 80% of the course

● Submitted assignments consistently (>90%) and typically on time until Week 5

● Activity then declined steeply

● Few completed the final assessment

● None completed the final assessment on time

This cluster appeared for all four MOOCs, but was variable with varying k

20

Late Completers

[5, 5, 5, 5, 5, 9, 9, 9]

● Submitted the final assessment

● Submitted most other assessments

● However, either submitted late or missed some assessments

● Each week, more than 94% of this cluster submitted their assessments

● More than three-quarters submitted the final assessment on time (78% – 90%)

● Around 40% of them posted comments (76% did so on MOOC3)

This cluster was fairly stable across all MOOCs and across most values of k

21

Keen Completers

[11, 11, 9, 9, 11, 11, 9, 9]

● Accounted for 7% – 23% of learners

● All the Keen Completers submitted all assessments

● More than 80% of these were submitted on time

● Typically, Keen Completers visited more than 90% of course content

● Over two-thirds contributed comments (68% – 73%)

● Mean number of comments varied from 21 to 54

This cluster was highly stable across all MOOCs and across all values of k

22

Improving learning and learning environments

Closing the loop

● Previews of course material would allow Samplers to make a more

informed decision about whether to join the course

● Sign-up pages could draw attention to the problems experienced by

those who are out of step with the cohort

● Discussion steps for latecomers could support those who fall behind at

the start

● Prompts might encourage flagging learners to return and register for a

subsequent presentation

● Bridges between course weeks could stress links and point learners

forward

23

View these slides at www.slideshare.net/R3beccaF

Rebecca

Ferguson

@R3becca

F

Doug

Clow

@dougclow