Nottingham Trent University Student DashboardEd Foster, Student Engagement Manager

NTU’s use of learning analytics

• Learning analytics

• 2013-14 pilot

• Analysis of 2013-14 pilot year

• Next developments

What is learning analytics?

• “Analytics marries large data sets, statistical techniques, and predictive modelling. It could be thought of as the practice of mining institutional data to produce ‘actionable intelligence’. – Campbell, DeBlois & Oblinger (2007)

• Use in MOOCs– Personalization of learning experience

• When to communicate to learners• Which questions are most difficult• Personalized course design

• Use in retention and success– Manchester Metropolitan University

• Outcomes for students who use the library after 9pm – 9am were … • Who was more likely to do their coursework at the last minute?

– Males or females?• But what’s the really hard part?

Future adoption of learning analytics?

http://cdn.nmc.org/media/2015-nmc-horizon-report-HE-EN.pdf

Learning analytics at your institutions

• Please say hello to the people around you

• Are your universities working on learning analytics?

• What are you working on?

• Do you have any fears?

Should we be worried about learning analytics?

Yes• Privacy

• Free will• Learning analytics is potentially very behaviouristic

• How will this data actually be used?

• So what if we can tell to 0/7% accuracy what the problem is? • Can we change the outcomes?• Should we change the outcomes?

• Sophisticated stereotyping/ blaming the student for the problem?

Why were we interested in learning analytics?Ed Foster, Student Engagement Manager

Mike Kerrigan, Strategic Data & Intelligence Manager

Why were we interested?

• Three different work streams:

• HERE Project (part of “What Works?” Student Retention & Success, 2008-2012)– Student doubters, weaker connections to their course, tutors and peers– Difficult to bring about institutional change

• Internal audit– NTU overall very good at retention, but weak at sharing data with those staff

who need it

• IS were working speculatively with a range of providers– Can you demonstrate a working model?

Primary interests in the Dashboard

Attainment

Belonging

Student Retention

• Reduce barriers to being known by a tutor• Space for tutors to make notes & plan

• Students as agents• Can see own grades, and compare self to peers • Feedback• Tool for academic use

• Promotes virtuous behaviour• Early warnings• Relationship building• Information about support

What does the dashboard do?

NTU Student Dashboard

Student biographical info, e.g. enrolment status

Evidence of student engagement

• Door swipes (where appropriate)

• Library book loans

• VLE use

• Dropbox submissions

• Future goals• Attendance• E-resources

Staff view

Studentview

Compares student engagement across the cohort & gives rating

Can make comments in free text box

Raises alerts!!

Progress

• Project Group– Students’ Union, quality assurance, Equality & Diversity, planning, legal

services, registry, Information systems, Student Support Services, Schools, Colleges & Community Outreach

• Pilot programme (2013-14)– 4 first year courses, 22 staff, 400 students– Understand how students and staff use it – Explore reactions to using the dashboard – Test methodology – does it work?– Student focus groups, student surveys

• Full implementation (2014-15)– By Spring 2014, it was felt that there was sufficient benefit to consider full

implementation, further evaluation conducted in Nov 2014 – Jan 2015– Agreed to implement in all academic Schools– Undergraduate & postgraduate

The NTU Student Dashboard

Student Engagement Rating - Cumulative

Student Engagement Rating – week-week

Engagement based on student behaviour

Three examples of withdrawn first year students 2013/14

Different patterns of withdrawal, some may be more amenable to support/ early warning

Patterns of disengagement & then withdrawal

Tutor NotesArranged to see [student] as attendance was poor. We had an open and honest discussion about this and I explained to him the relationship between attendance and success! We talked about the barriers to his attendance, which related to too many late nights! Will meet again to review his attendance [tutor name][student] has not attended my lectures for a few weeks. Emailed him to ask him to attend a one to one tutorial [tutor name]

Evidence from the 2013-14 pilot

Checking the efficacy of the dashboard model

Snapshot of engagement 2013-14

• A need to explore the relationship between engagement as recorded by the dashboard and retention and success

• Average daily engagement from 22nd Sept – 6th June – Derived from daily engagement classifications: low, satisfactory, good, high– Includes weekends, Christmas & Easter– For students who departed early, stopped counting on the last day they were

fully enrolled

• Looked at all undergraduate & postgraduate students (n=25,216)– Smaller samples for progression from first to second year & final year

attainment

• Overall student engagement

• Engagement for priority groups

• Engagement by entry qualification

• Engagement by date

Overall average engagement 2013-14

• Engagement trends differed by mode of study and level of study• For example, 50% of part-time students had low engagement average• To avoid confounding bias, analysis restricted to full-time

undergraduates

Relationship between engagement and progression

Low engagement Satisfactory engagement Good engagement High engagement

Other (n=76) 3.1% 1.2% 0.7% 0.2%

Withdrawn (n=310) 18.5% 4.0% 2.8% 4.0%

Transfer (n=143) 5.3% 2.2% 1.1% 1.3%

Repeating (n=364) 12.4% 6.2% 2.0% 2.5%

Academic failure (n=281) 36.5% 3.1% 0.8% 0.4%

Progressed (n=5,836) 24.2% 83.4% 92.5% 91.7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Progression status by engagement rating (year 1 full time UG students)

low satisfactory good high

0-220 points 23.7% 82.9% 95.4% 89.8%

221-240 points 34.7% 87.4% 95.6% 95.7%

>240 points 40.9% 92.6% 95.3% 94.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Students progressing to second year of study by capped pre-entry A Level tariff and engagement rating

We tried to pick specific points in time that were relevant to the student experience. Therefore the dates are not evenly distributed. The trend line ignores the Christmas/ early January period. In future analyses we need to separate the Christmas and January.

Relationship between engagement and attainment

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

low

satisfactory

good

highEn

gage

men

t cla

ssifi

catio

n

low satisfactory good high1st Class Honours 13.2% 15.2% 20.9% 28.0%

2nd Class Honours-1st Division 28.6% 47.0% 55.2% 52.5%

2nd Class Honours-2nd Division 28.6% 30.5% 21.5% 17.0%

3rd Class Honours 11.0% 4.9% 1.4% 1.8%

Other - Ordinary Degree 18.7% 2.4% 1.1% 0.6%

Final degree awards by engagement classification

Impact of engagement upon attainment

• Strong association between engagement and attainment• Although not as clear cut as engagement and progression

Where next?

Student Transition Survey

• First year survey (23rd Feb – 7th March)

• This time using questions from the UKES survey for comparison purposes with the sector

• Questions about Dashboard– How frequently have you used it– Has it changed the way you work at all– Checking against questions about – Belonging– relationships with staff– Academic performance

• Early findings– 90% of students had used the Dashboard at least once– 27% of students reported that they had changed their behaviour in response to

the data provided by the Dashboard

Focus groups

• Seven with students & staff (January – February)– Mixture of academic, administrative and professional services staff

• Main themes– What to do with your engagement score - today– Explain Dashboard methodology in the dashboard– Revise part-time/ placement/ exchange student methodology– Far more communications– Access to the dashboard (from emails/ NOW)– Additional data sources (electronic resources & attendance data)– Further data sources for day to day operations: NECs, AA– Relationships between named staff and students – Dashboard categories– Assessment view– Design/ layout/ possible work on visualization– Staff notes page – Possible periodic reporting to staff/ students

NTU Dashboard team

Existing data in warehouse

Developments to improve the methodology: new data sources/part time

Other developments: e.g. NECs, user interface, feedback, student support services

NTU: more communications, staff using the tool, students using the toolTutorials, 1-1s, online support

Stuff we’ve not thought of/ lessons learnt

Thank you

• Any questions/ thoughts/ ideas?