Finding your way through the mist: Analytics in Learning and Teaching

Simon Bedford

UOW @ a glance

University of Wollongong, an

institution ranking among the

top 2% of universities in the

world, with an enviable record

in teaching and research.

Drive for change @UOW

I. Curriculum Transformation Process – 2015 to 20181. FYE@UOW

2. Capstones@UOW

3. MyPortfolio@UOW

4. Connections@UOW

5. Hybrid Learning@UOW

II. TEL Strategy (DLT’s) & Assessment & Feedback Principles

III. 2017-18 TEQSA Re-Registration:• New Higher Education Standards Framework -2017

• Teaching & Assessment Policy Suite (TAPS) -2016

• Assessment Quality Cycle and External Referencing of Standards

HESFTAPS

PolicyT&A

PracticeTEQSA

Government Institution

IMPACT

CTP

A&FP

TEL/DLT

Put changes into practice and measure it?

Learning Analytics @ UOW

• Motivation is to assist with:

– Student retention

– Personalising student learning

– Continuous improvement of teaching & learning

• Narrowed focus 2015@UOW

– Near real-time delivery of information….

– …to teachers and students.

– Maximising the student learning experience

“learning analytics is the measurement,

collection, analysis and reporting of data

about learners and their contexts, for

purposes of understanding and optimising

learning and the environments in which it

occurs.”

Learning Analytics @ UOW

Data Mining

Impact of

CTP

A&FP

TEL/DLT

Student Learning

Inputs

Subject

Level

Course

Level

Learning Analytics@UOW

Maximising the teaching and learning

opportunities for higher education students

– a learning analytics case study within the

sciences

Science Medicine & Health – Case Study1. Chemistry Enabling Science: 5 YR1 subjects, <1000 students

2. Curriculum Transformation (@FYE)

3. Focus was student retention and interventions

4. LA Reports – Week 3, 6, 9, 12, Post Declaration of results

5. Meeting with Subject Coordinator & LA team

6. Interpretation of data – and modification of the model

7. List of actions for the next report.

• Bringing together multiple data sources to provide a more holistic picture of student resource

utilisation and performance

• Analytical insights can inform more tailored student communications;

• Caution required when interpreting data to avoid making assumptions;

• Learning analytics can serve as a catalyst for deeper understanding of students learning and

support needs;

• Improvements to data quality (e.g. attendance records) that informs evidence based decision

making

Multiple dimensions

to learning analytics

at UOW

SMP – Data Source

• Collected by teaching staff on SSHEETS

• Amalgamations of marks

• Not in real time – completed at the end

• Data of little use for interventions

• Lacks other inputs e.g. FA or Attendance

Moodle – Data Source

• Academic, professional, PT staff added to subject site

• Staff Dev to input data into grade book and on time

• All activity tracked – e.g attendance, formative & summative,

Number of logins, time on site etc.Moodle Logs etc

Moodle Data+

Library Data +

PASS, + SOLS,

Etc… =

Student Activity

Outcomes - Early Interventions

Students

Week 3 Week 6

Students

No Moodle? No PASS? Post Intervention

Outcomes – c/w other subjects

Science Students

Week 6Week 6

Law Students

Predicted FINAL

GRADE

78 D

79 D

85 HD

68 C

77 D

86 HD

97 HD

65 C

86 HD

78 D

Outcomes – Detail Report

Week 9

Students

Week 12

Students

1. Doing FA/Feedback = Did better SA

2. HD C Drop Off (lots reasons e.g Biology)

HD

C

Outcomes - Predictors

Post-Declaration

Students

Outcomes - Predictors

>5 PASS Sessions (FA&FFB)1-5 PASS Sessions

Data to Students - LA Dashboard

Impact on motivation

and moderation of

assessment?

Case Study - Conclusions

1. Not all data is useful data – e.g SOLS data not broken down

2. Academic Considerations: Causes data fluctuations

3. No yet able to “see” across all subjects taken in a semester ….

4. … and need to have coordinated approach for interventions.

5. To interpret models you need LA and Subject Specialists.

But overall LA has given us a far greater understanding of what students are

engaging in as they move through our subjects – and this will be of value in

measuring the impact of curriculum transformation in the future.

Data driven decision making for quality

assurance purposes

“…if you measure

something you

change it..”

Heisenberg's

uncertainty principle

Data for Continual Improvement

Focused data for Quality Enhancement:

1. Assessment board meetings

2. Subject Evaluation Reports (SC/HoS/ADE)

3. Course annual health check (APD)

4. Course comprehensive reviews (5Yrs)I. External Referencing of Student Attainment to comparable courses of study

II. and benchmarking (attrition, retention, pass rates)

Data for Review of Teaching (DaRT)

Assessment Quality

Cycle (AQC)

1 | P a g e

Subject Results – Current Session - Wollongong

Campus Comparison*

*The results presented are the latest set of results for each campus / delivery mode for which the subject is taught that has occurred within the last 12 months.

12% 9% 4%

9%

3% 7%

17%

6% 15%

7%

26%

3%

18%

11%

9%

3%

15%

7%

11%

6%

6%

15%

11%

15%

6%

19%

14%

3%

15% 15%

24%

6% 4% 9%

6% 4%

12% 3%

7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

On-Campus Distance On-CampusBega

On-CampusPSB Singapore

IPC

WD

WS

WH

TF

F

PS

P

C

D

HD

Average Mark 62.23 Median Mark 65.35 Highest Mark 92.00 Lowest Mark 30.56 Standard Deviation 15.89 Passed 85%

Wollongong

Assessment Committee Report SUBJ123; Wollongong, Autumn 2016 – DD/MM/YYYY

Student Count: 200 320 280 240 Last Day of Session: 30 June 16 30 June 16 30 June 16 1 Sept 15

3 | P a g e

5% 15%

2%

10% 15%

20%

15%

35%

20%

30%

28%

25%

30%

28%

45%

6%

6%

14%

20%

7% 8% 5% 9%

2%

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Wollongong SIM RegionalCampuses

IRI Hong Kong

Sub

ject

Me

an

Gra

de

Dis

trib

uti

on

HD D C P PS F TF Mean

Student Outcomes – International & Domestic – Grade Distribution

Subject Student

Type Campus

Student Count

HD% D% C% P% PS% F% TF%

SUBJ123 Domestic Wollongong 3 66.7 33.3

SUBJ123 Inton Wollongong 7

28.6 57.1 14.3

SUBJ123 Intoff Singapore

SUBJ123 Total 10

40.0 40.0 20.0

School Total 722 8.7 24.9 35.2 24.2 0.3 6.5 0.1

Faculty Total 9572 8.4 26.5 34.2 24.3 0.9 4.9 0.8

University Total 47057 8.8 24.4 31.2 23.6 1.3 8.2 2.6

Student Outcomes – Comparison by Location

Wollongong UOW Singapore Onshore Centres IRI Hong Kong

Count Mean Count Mean Count Mean Count Mean

SUBJ123 400 62.4 30 73.5 100 72.3 36 69.3

School 3060 67.2 94 71.2 1487 69.7 203 66.8

Faculty 7785 68.0 94 71.2 1490 69.8 203 66.8

University 53259 66.9 3038 63.5 3370 71.1 203 66.8

SUB

12

3, 6

8.4

SUB

12

3, 6

7.3

Sch

oo

l, 6

9.5

Sch

oo

l, 7

0.3

Facu

lty,

70

.2

Facu

lty,

75

.2

UO

W, 7

2.4

UO

W, 7

2.5

Domestic International

62

64

66

68

70

72

74

76Average Student Results

31%

7% 14% 13%

23%

14% 5%

20%

15%

14% 23%

30%

8%

21%

27%

10%

8%

14%

23% 13%

15%

29%

9% 13%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Assessment 1 Assessment 2 Assessment 3 Assessment 4

Student Outcomes - Across Assessments

HD D C P PS F

Assessments that

assure learning

outcomes

Student Type

Campus

Location

1 | P a g e

Subject Results – Current Session - Wollongong

Campus Comparison*

*The results presented are the latest set of results for each campus / delivery mode for which the subject is taught that has occurred within the last 12 months.

12% 9% 4%

9%

3% 7%

17%

6% 15%

7%

26%

3%

18%

11%

9%

3%

15%

7%

11%

6%

6%

15%

11%

15%

6%

19%

14%

3%

15% 15%

24%

6% 4% 9%

6% 4%

12% 3%

7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

On-Campus Distance On-CampusBega

On-CampusPSB Singapore

IPC

WD

WS

WH

TF

F

PS

P

C

D

HD

Average Mark 62.23 Median Mark 65.35 Highest Mark 92.00 Lowest Mark 30.56 Standard Deviation 15.89 Passed 85%

Wollongong

Assessment Committee Report SUBJ123; Wollongong, Autumn 2016 – DD/MM/YYYY

Student Count: 200 320 280 240 Last Day of Session: 30 June 16 30 June 16 30 June 16 1 Sept 15

1 | P a g e

This report provides data on student demographics and comparative student outcomes (CSO) for your subject. It is designed to support a self-evaluation and subject-level quality enhancement process.

Student Profile

2013 2014 2015

Student Demographics

Studying in the Faculty 73% 72% 71%

Sex (% Female) 56% 57% 55%

Residence (% Illawarra) 60% 62% 61%

Domestic Student % 94% 80% 72%

Average Age 20.8 19.5 19.5

Average EFTSL 0.8 0.8 0.8

Credits Completed

0 35 28 52

1-48 578 465 420

48-96 21 52 62

96+ 9 16 24

Students Repeating the Subjects

Yes 14 10 5

No 629 521 642

Course Enrolments

Bachelor of Commerce 250 220 260

Bachelor of Communication and Media Studies 100 150 135

Bachelor of Commerce (Dean’s Scholar) 50 10 60

Bachelor of Arts 20 30 25

Bachelor of Business 5 0 10

Bachelor of Engineering (Honours) 2 25 15

Bachelor of Social Sciences 1 0 0

Yearly Subject Evaluation Report Prototype - SUBJ123 2015

Historical Student Profile

Historical Subject Mark

Historical % Fails

2 | P a g e

Student Outcomes - Entry Level

Session Enrolments Prior to Census

Withdrawn at Census

% Change

2015 – Autumn 219 4 -1.82%

2015 – Summer 20 5 -25.00%

2014 – Autumn 250 12 -4.80%

2013 – Autumn 240 18 -7.50%

2013 – Summer 60 10 -16.67%

WAM Student Count Average Mark

<50 35 45.9

50 to 65 231 58.9

65 to 75 203 67.8

75 to 85 128 82.3

85+ 25 87.1

WAM Unknown 21 65.2

ATAR Student Count Average Mark

<50 7 52.4

50 to 64 27 65.8

65 to 74 108 63.2

75 to 84 155 72.6

85+ 137 84.2

UAC Unknown 32 62.5

Direct Entry 177 74.8

IELTS Student Count Average Mark

<5.0 7 52.4

5-6 27 65.8

6.5 137 84.2

7 2 62.5

7.5 8 74.8

8+ 5 84.2

Required part of my program 400

Relevant to my career 50

Fitted my personal timetable 20

The reputation of the subject 40

Seemed an interesting subject to do 65

Only subject available 10

Subject 2015 2011 2008

SUBJ123 2.5 1.8 1.7

School Total 2.7 2.7 2.6

Faculty Total 2.6 2.7 2.7

University Total 2.8 2.9 2.7

Reason for Taking the Subject

Average Satisfaction

Student Feedback – Subject Evaluation Survey Results as at DD/MM/YYYY

Please note that this feedback represents the data that was collected the last time the Subject was surveyed.

240 220 215 210 210 205

0 0 0 0 0 20

0

100

200

300

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Stu

de

nt

Enro

lme

nts

Subject Enrolment Headcount

2013 2014 2015

Subject Headcount

Subject Survey Data

Combined Subjects at Risk Data for ADE/HoS/APD (Faculty of Business)

Criteria / Weighting:

Enrolments

Repeating Students

Student Performance

Student Type

Location

Student Satisfaction

Questions