Assessment data

@glynnmark

Overview

• Introduction

• Learning Analytics

• LMS analytics example and issue

• Assessment analytics

Audience participation

Who am I?

Teacher

Blogger

Tweeter

Learner

Student

Father

husband

Chemist

Dr

Sports fan

BrotherFirst aider

Public relations

executive

Learning technologist

National Institute for Digital Learning

National

Institute for

Digital

Learning

Open

Education

Unit

Digital

Learning

Research

Network

Teaching

Enhancement

Unit

Teaching Enhancement Unit

TEACHINGENHANCEMENT

UNIT

Onlineand Blended Learning

Support

Awards and Grants

Credit Earning Modules

Professional Development

Workshops

Learning Analytics

Learning analytics is the collection and

analysis of data generated during the learning

process in order to improve the quality of

both learning and teaching.

So much student data we could use …

Academic Performance

• CAO and Leaving cert, University exams, course preferences, performance relative to peers in school

Demographics

• Age, home/term address, commuting distance, socio-economic status, family composition, school attended, census information, home property value, sibling activities, census information

Online Behaviour

• Mood and emotional analysis of Facebook, Twitter, Instagram activities, friends and their actual social network, access to VLE (Moodle)

Physical Behaviour

• Library access, sports centre, clubs and societies, eduroam access yielding co-location with others and peer groupings, lecture/lab attendance,

Levels of Analytics

Descriptive• What has happened?

Diagnostic• Why this happened?

Predictive• What will happen?

Prescriptive• What to do?

Learning analytics is a moral practise which should align with core organisational principles

The purpose and boundaries regarding the use of learning analytics should be well defined and visible

Students should be engaged as active agents in the implementation of learning analytics

The organisation should aim to be transparent regarding data collection and provide students with the opportunity to update

their own data and consent agreements at regular intervals

Modelling and interventions based on analysis of data should be free from bias and aligned with appropriate theoretical and

pedagogical frameworks wherever possible

Students are not wholly defined by their visible data or our interpretation of that data

Adoption of learning analytics within the organisation requires broad acceptance of the values and benefits (organisational

culture) and the development of appropriate skills

The organisation has a responsibility to all stakeholders to use and extract meaning from student data for the benefit of students

where feasible

This study was carried out following core principles

Outlined in PLS

Intervention is interactive

Students are able to change their data by engaging with Moodle more

All data is calculated using the same algorithm therefore cannot be biased

Data used is from Moodle usage alone and therefore does not in any way

define an individual

All researchers have the appropriate skills required to handle the data at

hand

Results of the study will be used to better understand how ro increase

studen engagement

Guiding Principles

Total Moodle Activity

Modules which work well …

• Have periodicity (repeatability) in Moodleaccess

• Confidence of predictor increases over time

• Don't have high pass rates (< 0.95)

• Have large number of students, early-stage

One example module – ideal !

Notes on model confidence

• Y axis is confidence in AUC ROC (not probability)

• X axis is time in weeks

• 0.5 or below is a poor result

• Most Modules start at 0.5 when we don't have much information

• 0.6 is acceptable, 0.7 is really good (for this task)

• The model should increase in confidence over time

• Even if confidence overall increases, due to randomness the confidence may go up and down

• It should trend upwards to be a valid model and viable module choice

LG116: Introduction to Politics

Students / year = ~110

Pass rate = 0.78

LG116 – Predictor confidence (ROC AUC)

Discussion

• Good points

• Bad Points

http://padlet.com/markglynn/cxbh3efm9e0z

Assessment data

What data?

• Module title

• Assessment Title

• Opening and closing date

• Description

• Category i.e. weighting

• Student grade i.e. category total

• Student overall grade i.e. Course total

Privacy versus Visibility

-3 +30

Relative grade

Relative Grade (close up)

Relative assessment grade – Lecturers view

-3 +30John

-3 +30

Paul

-3 +30

Ringo

-3 +30

George

57%

86%

65%

43% ˇ

Coordinators view

-3 +30Chemistry

-3 +30

Physics

-3 +30

Biology

-3 +30

Maths

57%

86%

65%

43% ˇ

Student name: John Lennon

CRM project

Discussion

• Good points

• What next?

http://padlet.com/markglynn/cxbh3efm9e0z

What else?

• Module title

• Assessment Title

• Opening and closing date

• Description

• Category i.e. weighting

• Student grade i.e. category total

• Student overall grade i.e. Course total

Teachers Perspective

Calendar

Rubric Analysis

Rubric Analysis

Track individual student engagement with video

video engagement analytics: measure attention during playback

Discussion

• Good points

• What next?

http://padlet.com/markglynn/cxbh3efm9e0z

Conclusions and next steps

• Huge but simple opportunity

• Minimal investment

• Please share

• Mark.glynn@dcu.ie

Acknowledgements

Moodle ModificationsLearning Technology Services

www.lts.iesomerandomthoughts.ie

@ghenrick

Team behind the Moodle Analytics project

John BrennanOwen Corrigan

Aly EganAlan F. Smeaton

Sinéad Smyth