Learning Analytics - From theory to practice

Annelies Raes, WimVan den Noortgate, StijnVan Laer

Educational Technology Day

Learning Analytics

May 16, 2018

URL: www.pollev.com/smarted

Innovation trends: Outlook 2020

Fundamental societal transitions

© Center for Curriculum Redesign, 2018

Non-routine tasks

Routine tasks

Tasks not

supported by

ICT

Tasks supported by ICT

Changing focus in the curriculum

© Center for Curriculum Redesign, 2018

Non-routine tasks

with high level use

of ICT

Routine tasks

With high level use of ICT

Non-routine tasks

with low level use

of ICT

Routine tasks with

low level use of ICT

Changing focus in the curriculum

• Focus on ‘foundational skills’, 21st Century skills

Today’s challenge

• Valuing Practicing Assessing

Learning Analytics is in the air…

LA is supporting from different perspectives

(Janssen, Molenaar, & Van Leeuwen, 2018)

Different dimensions of LA

1. What? What kind of data does the system gather, manage, and use for the

analysis?

2. Who? Who is targeted by the analysis?

3. Why? Why does the system analyze the collected data?

4. How? How does the system perform the analysis of the collected data?

(Chatti et al, 2012)

Applying LA – three use cases

I. Using log data for adaptive item selection

(Wim Van den Noortgate)

II. Using log data for instructional design purposes (Stijn Van Laer)

III. Using multimodal data to monitor engagement during both face-

to-face and remote learning (Annelies Raes)

Using log data for adaptive item selection

Dmitry Abbakumov, Frederik Cornillie, Sean Joo, Trevor

Kadengye, Ellie Park, Kelly Wauters, Wim Van den Noortgate

Example: item-based e-learning

In-video formative assessment Summative assessment

Example: MOOCs

Example: educational game

DATA STRUCTURE

1 1 1 0 0 1 1 0 1

0 0 1 0 0 0 1 0

1 1 1 0 0 0

1 0 1 1

0 0 0 0 1

Using IRT models

Ability

Item 3 Item 1 Item 2

Testing persons using a ‘calibrated’ scale

Testing persons using a ‘calibrated’ scale

Comparing persons

Not necessarily same or equivalent test!

Progress testing

Evaluate learning

• Using intermediate assessments?

o often tests too short

o obtrusive

• Using responses on all tasks and exercises

requires dynamic models

Describe learning

N = 3803

I = 610

Describe learning

1 1 1 0 0 1 1 0 1

0 0 1 0 0 0 1 0

1 1 1 0 0 0

1 0 1 1

0 0 0 0 1

• Content

• Type (e.g., MC)

• Including video/audio

• …

• Age

• Previous courses

• Motivation

• …

• Time

• Location

• Previous behavior

• …

Explain learning

• How is learning influenced by:

o user characteristics,

o item characteristics,

o context characteristics,

o previous behavior, and

o interactions of these!

Theoretical interest

Practical interest: adaptation of the learning environment to the learner

Explain learning

Track learning ‘on the fly’

Personalisation of the learning environment!

Cold start

problem!

Cold start problem• New users + users that were absent for a long time

• Improvements:

o Use explanatory models to help the tracking algorithm

o Use previous trajectory

Without

background

information

With background

information

Complex tasks imply multiple skills …

Example Monkey Tales Educational game

EDUCATIONAL GAMETRACING MATH & GAMING SKILLS

Using log data for instructional design purposes

Stijn Van Laer & Jan ElenCentre for Instructional Psychology and Technology

Operationalization of SRL

cyclical, influenceable, and covert in nature

Operationalization of SRL

cyclical, influenceable, and covert in nature

Self-regulated learning and log files

• Investigating self-regulated learning

o Behaviour: Event sequence analysis (log files)

o Outcomes: Changes in cognitive, motivational, and meta-cognitive variables

(behavioural consequences)

o Behaviour + outcomes: Goal-directed learning behaviour or self-regulated

learning

• Data structure

o Computer log files (Learning Management System)

o Time Stamped Event (TSE) Data

o Data-driven approach (no recoding, transforming, etc.)

Example study

Research question

“How do cues for reflection in blended learning environments impact both learners’ (a) products and (b) processes of learning?”

Context and sample

Context

• Flemish centre of adult education – Second chance education;

• “AAV Wiskunde M2” – Basic statistics (surveys, frequency tables, etc.);

• Two eight week courses – courses co-designed by 2 instructors.

Sample

• 41 adult learners (25 woman – 16 man) and age ranging from below 20 to maximum 50 years

of age;

• No diploma secondary education – often plenty of experience;

• Exposed to mathematics and computers before.

METHOD (I) – Pre and post test

Description internal conditions: cognition and motivation

• Cognitive conditions

o Performance-based prior domain knowledge test – Test exam

o Performance-based domain knowledge test – Test exam

• Motivational conditions

o Achievement Goal Questionnaire - Revised (AGQ-R) (1) – Likert type scale

o Academic self-concept (ASC) scale (2) – Likert type scale

The test and questionnaire was piloted and tested for internal consistency to ensure the scale reliability. (3)

(1) Elliot, & Murayama, 2008; (2) Liu, & Wang, 2005; (3) Cronbach, 1951

METHOD (II) – Intervention

Description external conditions: design of environments

METHOD (III) – Products of learning

2 x 2 Mixed ANOVA

o Within-subject factor: Time (pre and post test)

o Between-subject factor: Condition (experimental and control)

Interaction effect Time x Condition?

Affordances and constraints

o Missing data: time point vs list wise deletion

o Possibility for post hoc tests

o Tracking and interpreting interactions

o Need for larger sample sizes

* SPSS

Event

E

Event

C

Event

B

Event

B

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Method (IV) – Processes of learningLog File dataset

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Event

A

Event

D

Event

C

Event

A

Event

D

Event

A

Event

E

Event

C

Event

B

Event

B

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Identification

* TraMineR in R Statistics

Event

E

Event

C

Event

B

Event

B

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Method (V) – Processes of learning

Event

A

Event

D

Event

C

Event

A

Event

D

Event

A

Event

E

Event

C

Event

B

Event

B

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Identification

SCORE A

SCORE B

SCORE C

SCORE DInte

rnal co

nd

itio

n Observed A

Observed B

Observed C

Observed D

Significant

difference?

* TraMineR in R Statistics

ANALYSIS

• Internal conditions: cognition and motivation (1)

o Prior domain knowledge - Test score - Cronbach alpha (.76)

o Motivation - Cronbach alpha’s (all between .73 and .95)

o Domain knowledge test - Test score - Cronbach alpha (.83)

• Products of learning (2)

o Tests of assumptions for normality (Shapiro–Wilks’ test), sphericity

(Mauchly's Test of Sphericity), and homogeneity of variances (Levene’s test)

o 2x2 Mixed ANOVA (Time as within-subject & Condition as between-subject)

• Processes of learning (3)

o Exploratory visualization of reflection cue use

o Frequent event sub-sequences (pMinSupport=.25 and K=undifined)

o Discriminant event sub-sequences per variable based on Chi Square (χ2)

(1) Cicchetti, 1994; (2) Elliot, & Murayama, 2008; Liu, & Wang, 2005; (3) Müller, Studer, Gabadinho, &

Ritschard, 2010

Results – Products of learning

• Within-subject effect of time:

o Performance approach orientation (F (1, 8) = 6.564, p = .034, ηp2 = .45)

o Learning confidence (F (1, 8) = 7.498, p = .026, ηp2 = .48)

o Domain knowledge (F (1, 8) = 46.716, p < .001, ηp2 = .85)

• Within-subject effect of timexcondition:

o Performance avoidance orientation (PAV) (F (1, 8) = 7.374, p = .026, ηp2 = .48)

Regardless the intervention, both learners in the control and experimental condition

increased in performance approach, learning confidence and domain knowledge.

In the experimental condition learners increased in performance avoidance

orientation where learners in the control condition decreased.

Results – Processes of learning

• Effect of condition on significant discriminant sub-sequence :

o Sig. more sequences including communication tools (χ² (2) = 5.49, p = .02)

o Sig. more sequences with regard to task submission (χ² (2) = 8.33, p = .003)

o Sig. more sequences relating to tests (χ² (2) = 6.27, p = .01)

* Discriminant sub-sequences

In the experimental condition learners used significantly more sequences related to

features that enable testing and comparison.

Discussion (I)

Products of learning• Increase over time on performance approach, learning confidence and domain knowledge.

• The reflection cues provided only increased learner’ performance avoidance orientation.

Processes of learning• Significantlymore sequences related to features that enable testing and comparison.

o Mastery goals focus on gaining competence while performance goals focus on demonstrating performance to others andoneself (e.g., Kovanovi, Gasevic, Joksimovi, Hatala, & Adesope, 2014).

o Cues might have evoked learners to test their own performance compared to others and so help reduce the anxietyassociated with a feeling of a potential looming failure (e.g., Crippen, Biesinger, Muis, & Orgill, 2009).

o Learners seek to demonstrate that they are not incompetent and hence not doingworse than others (Collazo, Elen, & Clarebout, 2015).

Discussion (II) – Cue use

* Tableau 10.3

No significant impact of (prior)

learner characteristics on the

distribution, frequency, or

lifecycle of use of the

reflection cues.

Conclusion (I) – Reflection cues

Increase (over time) in performance avoidance orientation – products

Cues used, no significant differences among learners – cue use

Significant behavioural changes (use of sequences containing test, assignments, and

communication tools) – processes

Although the cues were used, the intervention presented did not

seem to support the learners sufficiently toward increased

(favourable) learning outcomes, implying that supporting reflection

only is insufficient to evoke goal-directed learning behaviour.

Conclusion (I) – Reflection cues

Increase (over time) in performance avoidance orientation – products

Cues used, no significant differences among learners – cue use

Significant behavioural changes (use of sequences containing test, assignments, and

communication tools) – processes

Although the cues were used, the intervention presented did not

seem to support the learners sufficiently toward increased

(favourable) learning outcomes, implying that supporting reflection

only is insufficient to evoke goal-directed learning behaviour.

Elaborated insights through Learning Analytics(opportunities and challenges)

Continuous, unobtrusive, and predictive

Event

A

Event

D

Event

E

Event

C

Event

B

Event

C

Event

A

Event

D

Event

BProduct

Event

A

Event

D

Event

E

Event

C

Event

B

Event

C

Event

A

Event

D

Event

BProduct

Internal and external

conditions

Intervention

Intervention Intervention

Opportunities

(but also…)

Event

E

Event

C

Event

B

Event

B

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

OperationalizationLog File dataset

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Event

A

Event

D

Event

C

Event

A

Event

D

Event

A

Event

E

Event

C

Event

B

Event

B

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Event

A

Event

D

Event

E

Event

C

Event

B

Event

B

Event

C

Event

A

Event

D

Event

A

Event

B

Event

…

Identification

Challenges

Unit of analysis and grainsize

Feature A Feature B Feature C Feature …

Level 1 Level 2 Level 3 Level …

External conditions

Internal conditions

Learn

er

beh

avio

r

Measurement

Challenges

Level of Inferences

State? State?

Data level

Interpretation

Inferences

Event

1

Event

2 State? State?Event

3

Event

…

State …

Co

gn

itiv

e

pro

ces

ses

State 1 State 2 State 3

Self-regulated learning behavior

Challenges

Using multimodal data to monitor engagement during face-to-face and

remote learning

Annelies Raes, Fien Depaepe, Pieter Vanneste,

WimVan den Noortgate, Ine Windey

Learning analytics for educational decision making

(Janssen, Molenaar, & Van Leeuwen, 2018)

Cartoon: Joris Snaet

New challenges in future learning spaces

• Student perspective:

Remote/online students have different

experiences How to make sure students

still feel connected and stay engaged?

o Teacher perspective

increased workload Cognitive

(over)load of teacher when teaching

and overseeing the smart classroom“We want to know when we are loosing our students”

Remote classroom Hybrid virtual classroom

Enhancing interactivity

Interactive lecture: Quizzes, polls & silent questions

Passive (Inter)active

Main objectives

• The effect of interactive quizzes on students’

engagement?

o How to measure engagement?

Engagement = complex & multidimensional concept

Behavioral

E.g. attending lectures, asking questions,

participating in quizzes …

Emotional

E.g. students’ feelings of interest,

happiness, anxiety, sense of belonging

Cognitive

The skills, and the strategies they

employ to master their learning

Aspects of student engagement within taught contexts

(Dobbins & Denton, 2017)

Methodological challenge & opportunities

• Self-report measures (e.g. motivation scales) have many shortcomings and biases:

a snapshot articulation of engagement rather than examining how emotion & behavior

unfolds in an interactive context (Schrerer, 2004)

• New technologies have the opportunity to observe, measure, and understand learning

and assessment processes more objectively, online and determined in real time.

(Gomez & Danuser, 2007; Sakr, Jewitt & Price, 2016)

Po

ste

st

Process data

0 1 1 0 1

log data Psycho-

physiological

data

Audiovisual

data

Multimodal learning analytics to capture ‘engagement’ cognitive

validation?

Self-report

ENGAGEMENT

METER

1) Optimal use of quizzes within lectures?

2) The effect of quizzes on students’ engagement?

1.

Pilot (2

quizzes)

2.

Quizzes

3.

1 quiz

4.

4 quizzes

5.

0 quizzes

6.

3 quizzes

1. Self-report

4.424.22

4.364.11 4.11

4.4

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

6

Intrinsieke motivatie: geboeid / cognitieve prikkeling

Intrinsic motivation

Les 1 (2Q) Les 2 (2 Q) Les 3 (1Q) Les 4 (4 Q) Les Les 5 (0 Q) Les 6 (3 Q)

N = 17

1. Self-report

4.65 4.744.55

4.794.6

4.97

1

1.5

2

2.5

3

3.5

4

4.5

5

5.5

6

Expectations Lecture 1 (2quiz.)

Lecture 2 (2quiz.)

Lecture 3 (1quiz)

Lecture 4 (4quiz.)

Lecture 5 (0quiz.)

Lecture 6 (3quiz.)

Expected & experienced effect on learning (PU) of quizzes

No novelty

effect

1. Self-report

83%

17%

Lecture 5 without quizzes: Have you missed the quizzes?

Yes, it would have more fun with 1 or more quizzes No

2. Audiovisual dataSTEP 1 – Recording interactive courses STEP 2 – Annotating individual students’ behaviour

STEP 3 – Use computer vision techniques to

automatically recognize these actions in real-

time

STEP 4 – Use machine learning techniques to discover

patterns between the behaviour and manual annotations

Po

stte

st

Process data

Log data

3. Log dataParticipation in quizzes and accuracy

0 1 1 0 1

Po

stte

st

Process data

Log data

3. Log dataWeb browser activity (‘Engagement Monitor’

by Distrinet)

0 1 1 0 1

Po

stte

st

Process data

4. Psychophysiological dataWeb browser activiteit

Q1 Q2 Q3 Q4

Po

ste

st

Process data

4. Psychophysiological data

Q1 Q2 Q3 Q4

Work in progress

• Modelling dis)engagement

• Visualising (dis)engagement in a dashboard educational decision making

• Implementation & testing within the context of multilocation learning

Thank you for your attention.

Suggestions or questions?