implemententing analytics part 1 - niall sclater
TRANSCRIPT
Implementing analytics:Learning analytics and business intelligenceNiall Sclater, Consultant
Session outline
» Overview of the Learning Analytics Service » Overview of the Business Intelligence and Analytics Labs
project» Presentation 1: Moriamo Aduyemi, head of corporate
information systems, Abertay University» Presentation 2: Niall Sclater, learning analytics consultant,
Jisc
Effective learning analytics challenge
Rationale» Universities and colleges wanted help to get started and
have access to a standard set of tools and technologies to monitor and intervene
Priorities identified» Code of Practice on legal and ethical issues» Develop a basic learning analytics service including an app
for students» Provide a network to share knowledge and experienceTimescale» 2015-2016 - Test and develop the tools and metrics» 2016-2017 - Transition to service (Freemium)» Sept 2017 - Launch. Measure impact on retention and
achievement
What do we mean by Learning Analytics?
» The application of big data techniques such as machine based learning and data mining to help learners and institutions meet their goals:
» For our project: › Improve retention (current project)› Improve attainment (current project)› Improve employability (future project)› Personalised learning (future project)
Toolkit and community
» Blog: http://analytics.jiscinvolve.org
» Reports› Code of practice for learning
analytics› The current state of play in UK
higher and further education› Learning analytics in Higher
Education: a review of UK and international practice
» Mailing: [email protected] » Network meetings
Learning analytics architecture
Current engagement
» Expressions of interest: 85» Engaged in activity: 35» Discovery to Sept 16: agreed (28), completed (18),
reported (17)» Learning analytics pre-implementation: (12)» Learning analytics implementation: (7)
Current engagement
» From Sept 2016» “Readiness Toolkit” with a diagnostic set of questions and
support materials leading to implementation» Start-up guidelines to get ready for learning implementation
Further details will be announced via [email protected]
HESA and Jisc Business Intelligence
Heidi Plus The new business intelligence service for UK Higher Education Replaces Heidi (which will be decommissioned in November
2016) Launched in November 2015 offering:
Improved data content and functionality Delivery of data sets through commercial data explorer tool New visualisations and dashboards New training programme and support materials
Available to HE institutions with a full HESA subscription Over 80% of current Heidi subscribers have started the Heidi Plus
application process (40% completed)
Heidi Lab overview
Take a thin vertical sliceDon’t try to waterfall each stepWant quick release of product and steer
Dev Team1
Dev Team 2
Dev Teams
Dev Teams
Dev Teams
Dev Teams
Analysis
Data orders
User stories
Dashboards
Process overview
Scrum team activity
Library Analytics labs
» Teams working on Library BI Stories at 0.2 FTE, total estimated effort 15 days from July - Oct 2016
» Both product owners and sector data eperts invited:› Product owner from the sector to steer which stories are
of interest› Sector experts to understand what data sources are
available and what is in the data› Jisc Contracted data transformation specialist (CETIS)› Jisc Agile scrum master and Tableau user
» Teams receive experience and guidance of Agile working» Option for Tableau desktop training to help with creating
visualisations» Apply at http://bit.ly/jisc_library_data_labs_applications» Queries to [email protected] or
Learning analytics
Retention
» 178,100 students aged 16-18 failed to finish post-secondary school qualifications they started in the 2012/13 academic year› costing £814 million a year - 12 per cent of all
government spending on post-16 education and skills (Centre for Economic and Social Inclusion)
» 8% of undergraduates drop out in their first year of study › This costs universities around £33,000 per student
» Students with 340 UCAS points or above were considerably less likely (4%) than those with fewer UCAS points (9%) to leave their courses without their award
Attainment
» 70% of students reporting a parent with HE qualifications achieved an upper degree, as against 64% of students reporting no parent with HE qualifications
» In all disciplines except Computer Science, Medicine and Dentistry, and Physical Science, students with a parent with an HE qualification were more likely to have achieved an upper degree
» Overall, 70% of White students and 52% of BME students achieved an upper degree
Marist college – Academic early alert system
Approach» Supported by Bill and Melinda Gates Foundation.
Investigated how use of Academic Early Alert systems impact on final course grades and content mastery
Outcome» Analysis showed a statistically significant positive
impact on final course grades» The most important predictor of future academic
success was found to be partial contributions to the final grade
» The predictive models developed at one institution can be transferred to very different institutions while retaining most of their predictive abilities
» Simply making them aware that they are at risk may suffice
Signals at Purdue University
Approach» Signals’ predictive algorithm is based on performance,
effort, prior academic history and student characteristics
Outcome» Problems are identified as early as the second week in
the semester» Students are given feedback through traffic lights – and
from messages tailored by their instructors» Students using Signals seek help earlier and more
frequently» One study showed 10% more As and Bs were awarded
for courses using Signals than for previous courses which did not use Signals
UK Open University – developing an analytics mind-set
Approach» Investing in strategic learning analytics programme to
enhance success of over 200,000 studentsOutcome» Availability of data – macro-level aggregation of all data
relating to student learning and experience» Analysis of that data – multiple dashboards and tools
under development, including student views» Interventions and processes that enhance success –
including input into learning and assessment design at module level
Predictive analytics at Nottingham Trent University
Approach» Most prominent UK: university-wide dashboard for
30,000 students, enhancing retention, community belonging, and attainment
Outcome» ‘Doubters’ less likely to complete targeted via tutor and
peer support» 27% of students changed learning behaviour because of
analytics feedback» Engagement in studies shown better predictor of final
grade then entry qualifications or demographics
25
Jisc’s effective learning analytics programme
ECAR Analytics Maturity Index for Higher Education
UK Learning Analytics Network
26
ECAR Analytics Maturity Index for Higher Education
Group Name QuestionMain type
Importance Responsibility
2 Consent Adverse impact of opting out on individual
If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress?
Ethical 1 Analytics Committee
7 Action Conflict with study goals
What should a student do if the suggestions are in conflict with their study goals?
Ethical 3 Student
8 Adverse impact
Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?
Ethical 1 Educational researcher
86 issues in 9 groups
Available from Effective learning analytics blog: analytics.jiscinvolve.org
Group Name Question Main type Importance Responsibility2 Consent Adverse impact of
opting out on individual
If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress?
Ethical 1 Analytics Committee
7 Action Conflict with study goals
What should a student do if the suggestions are in conflict with their study goals?
Ethical 3 Student
8 Adverse impact
Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?
Ethical 1 Educational researcher
86 issues
jisc.ac.uk/guides/code-of-practice-for-learning-analytics
jisc.ac.uk/guides/code-of-practice-for-learning-analytics
jisc.ac.uk/guides/code-of-practice-for-learning-analytics
Times Higher, 25 Feb. 2016
32
Jisc’s effective learning analytics programme
ECAR Analytics Maturity Index for Higher Education
Discovery Phase
34
ECAR Analytics Maturity Index for Higher Education
Architecture
Project partners
Learning Analytics architecture
Unified data definitions
The case for Learning Analytics
Service: dashboards
Visual tools to allow lecturers, module leaders, senior staff and support staff to view: » Student engagement» Cohort comparisons» etc…
Based on either commercial tools from Tribal (Student Insight) or open source tools from Unicon/Marist (OpenDashBoard)2/03/2016
The case for Learning Analytics
Service: Alert and intervention system
Tools to allow management of interactions with students once risk has been identified:
» Case management» Intervention management» Data fed back into model» etc…
Based on open source tools from Unicon/Marist (Student Success Plan)
2/03/2016
Service: Student app
Comparative – social – gameified – private by default – usable standalone - uncluttered
Profile Aim Activity Data sourcesRussell Group Retention of widening
participation + support for students to achieve 2.1 or better
Discovery + Tribal Insight + Learning Locker
Moodle + Student Records
Research led University
Retention, improve teaching, empowering students
Discovery + OpenSource Suite + Student App
Moodle + Attendance+ Student Records
Teaching led University with WP mission
Retention - requirement to make identifying students more efficient so they can focus on interventions
Tribal Insight + Learning Locker
Blackboard + Attendance + Student Records
Research led University
Student engagement Discovery + Student app + Learning Locker
Moodle + Student Records
Teaching lead Understanding of how Learning Analytics can be used
Discovery + Technical Integration
Moodle
