keynote presentation oofhec2016: bart rienties

Keynote address

Analytics4Action Evaluation Framework: a review of evidence-based learning analytics interventions at the Open University UK

20 October 2016

@DrBartRienties

Reader in Learning Analytics

A special thanks to Avinash Boroowa, Aida Azadegan, Shi-Min Chua, Simon Cross, Rebecca Ferguson, Lee Farrington-Flint, Christothea Herodotou, Martin Hlosta, Wayne Holmes, Garron Hillaire, Simon Knight, Nai Li, Vicky Marsh, Kevin Mayles, Jenna Mittelmeier, Vicky Murphy, Quan Nguygen, Tom Olney, Lynda Prescott, John Richardson, Jekaterina Rogaten, Matt Schencks, Mike Sharples, Dirk Tempelaar, Lisette Toetenel, Thomas Ullmann, Denise Whitelock, John Woodthorpe, Zdenek Zdrahal, and others…A special thanks to Prof Belinda Tynan for her continuous support on analytics at the OU UK

(Social) Learning Analytics“LA is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (LAK 2011)

Social LA “focuses on how learners build knowledge together in their cultural and social settings” (Ferguson & Buckingham Shum, 2012)

Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.

So what does the OU UK do in terms of interventions on learning analytics?

1) Exemplar of “small intervention”

2) Large scale adoption of predictive analytics to help teachers to identify “students at risk”

Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: a review of evidence-based learning analytics interventions at the Open University UK. Journal of Interactive Media in Education, 1 (2) 1-13.

7

Analytics4Action framework

Implementation/testing methodologies

• Randomised control trials• A/B testing

• Quasi-experimental• Apply to all

Communityof inquiry

framework:underpinning

typology

Menu of response actions

Methods of gathering data Evaluation Plans

Evidence hub

Key metrics anddrill downs

Deep dive analysis and

strategic insight


9

Menu of actions Learning design (before start) In-action interventions (during module)

Cognitive Presence Redesign learning materials

Redesign assignments

Audio feedback on assignments

Bootcamp before exam

Social Presence Introduce graded discussion forum activities

Group-based wiki assignment

Assign groups based upon learning analytics

metrics

Emotional questionnaire to gauge students’

emotions

Introduce buddy system

Organise additional videoconference sessions

One-to-one conversations

Cafe forum contributions

Support emails when making progress

Teaching Presence Introduce bi-weekly online videoconference

sessions

Podcasts of key learning elements in the module

Screencasts of “how to survive the first two weeks”

Organise additional videoconference sessions

Call/text/skype student-at-risk

Organise catch-up sessions on specific topics that

students struggle with

Large scale adoption of predictive learning analytics

• 10 modules used predictive learning analytics• 240 teachers had access to OUA vs. 613

teachers who did not • 4320 students with OUA, 12713 without• 70 teachers received a weekly reminder (email)

notifying them that the OUA predictions were available through OUA dashboard

• 170 received the OUA weekly predictions via email in excel

Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M. (Submitted: 19-10-2016). Using Predictive Learning Analytics to Support Just-in-time Interventions: The Teachers' Perspectives across a large-scale implementation.

So did it make a difference?

• In 7 out of 10 modules there was no difference

• In 3 pass rates were higher

So did teachers use PLA?


So what did teachers experience?Emerging themes Explanation

Actual uses of the OUA dashboard ₋ Features of the OUA dashboard that are used by teachers

₋ Frequency of use

₋ Ease of use

Perceived usefulness of OUA ₋ Teachers' perceptions of the system as being useful or not

₋ In what respect the system support teaching practices

Approaching students at risk ₋ How teachers react to students flagged as being at risk

₋ What intervention strategies they devise to support students

Informing learning design ₋ How OUA could be used to inform the design of a module

Improvements to OUA ₋ Teachers’ suggestions for improving OUA

Intention to use OUA in the future ₋ Teachers’ intentions in terms of using OUA in the future

Table 2: Emerging themes identified through thematic analysis


Conclusions (Part I)

1. Learning analytics can help teachers to find mismatches in design and learners’ needs, behaviors, and expectations

2. Teachers can make a difference when intervening and LA can help to track the success of those interventions.

Conclusions (Part II)

1. Power of predictive analytics depends on how teachers (and students) are using data

2. Lack of technology acceptance and implicit/explicit academic resistance might limit power of LA

3. Professional development of staff and strategic support from senior management needed to make LA a success

Keynote address

Analytics4Action Evaluation Framework: a review of evidence-based learning analytics interventions at the Open University UK

20 October 2016

@DrBartRienties

Reader in Learning Analytics

A special thanks to Avinash Boroowa, Aida Azadegan, Shi-Min Chua, Simon Cross, Rebecca Ferguson, Lee Farrington-Flint, Christothea Herodotou, Martin Hlosta, Wayne Holmes, Garron Hillaire, Simon Knight, Nai Li, Vicky Marsh, Kevin Mayles, Jenna Mittelmeier, Vicky Murphy, Quan Nguygen, Tom Olney, Lynda Prescott, John Richardson, Jekaterina Rogaten, Matt Schencks, Mike Sharples, Dirk Tempelaar, Lisette Toetenel, Thomas Ullmann, Denise Whitelock, John Woodthorpe, Zdenek Zdrahal, and others…A special thanks to Prof Belinda Tynan for her continuous support on analytics at the OU UK