Implementing Predictive Learning Analytics on a Large Scale: The Teacher's Perspective

Dr Christothea Herodotou

Authors: Christothea Herodotou, Bart Rienties, Avinash Boroowa, Zdenek Zdrahal, Martin Hlosta , Galina Naydenova. Proceedings of LAK2017.

Predictive Learning Analytics (PLA)

o PLA identify students at risk + inform teacherso Mixed-effects of providing PLA to teacherso Difficulty in understanding and interpreting PLA data

and visualisationso Difficulty in identifying specific interventions o Promising outcomes (van Leeuwen et al., 2014;

McKenney and Mor, 2015) o Identification of participation problems and interventiono Development of curriculum materials

2

Aim of this study

Evaluate whether providing teachers with PLA data would empower them to identify and assist students in need for additional support

3

OU Analyse (OUA)

•OUA: Early identification of students at risk of failing•Available to tutors and student support teams •Aim: Improve retention of OU students

https://analyse.kmi.open.ac.uk/

4

OUA Dashboard: Module view

5

OUA Dashboard: Student view

6

OUA Dashboard: Student view

Nearest neighbours, Predictions with real scores, Personalised recommender

7

Research Questions

1.How did 240 tutors within 10 modules made use of OUA predictions and visualisations to help students at risk?

2.To what extent was there a positive impact on students' performance and retention when using OUA predictions?

3.Which factors explain tutors' uses of OUA?

8

Methodology

1.Statistical comparisons: 240 tutors OUA access Vs 613 tutors with no access to OUA predictions formal withdrawal rates by the end of the

module, completion and pass rates.

2.Usage data of OUA dashboard (N=70)3.Qualitative data: Semi-structured interviews

(N=6)

9

Results: Withdrawal rates

Formal withdrawal rates by the end of the module

  2015 -OUA students 2015 - nonOUA students

count % count %

Arts 55   1728  

did not reach the end of the module 15 27.27% 253 14.64%

reached 40 72.73% 1475 85.36%

Technology 239   1809  

did not reach the end of the module 56 23.43% 311 17.19%

reached 183 76.57% 1498 82.81%

Law 246   1820  

did not reach the end of the module 33 13.41% 359 19.73%

reached 213 86.59% 1461 80.27%10

Results: Completion & pass ratesModule completion and pass rates

  2015 -OUA students 2015- nonOUA students

Completion rates count % count %

Education 260   2841  

did not complete 69 26.54% 931 32.77%

completed 191 73.46% 1910 67.23%

Law 245   1815  

did not complete 79 32.24% 732 40.33%

completed 166 67.76% 1083 59.67%

Pass rates        

Law 246   1820  

did not passed 86 34.96% 797 43.79%

Passed 160 65.04% 1023 56.21%11

Results: Usage of OUA dashboard

12

Results: Interviews (1/2)Actual uses of OUA: •Most useful OUA features:

o colour-coded students o VLE activity

•Teachers define what features to use and how often (time management)

Usefulness of OUA•Teachers checking on students often (emails, forums)•Teachers more proactive/enhance teaching practices

o “on top” of studentso “where I need to put my e orts" ff

• Predictions: complement teachers’ intuition + additional insights

13

Results: Interviews (2/2)

Approaching students at risko Varied interventions - referral to student support

services, sending emails, texting, calling, doing nothing

o Varied approaches: persistent VS less pro-activeo Personalised approach

Future intentionso Interested in future use yet with improvements to

the system (e.g., “sensitive” to students’ activities online)

14

Conclusions

o Blurred picture (withdrawal, completion and pass rates)

o Variation in teachers’ degree and quality of engagement with learning analytics.

o Lack of consensus about intervention strategieso Predictive data - enhance and facilitate teaching

practice, especially within distance learning contexts

o Research directions: Identify how, when, and what interventions to trigger to support students adequately

15

Thank you, @herodotouc

16

Questions?