analytics are everywhere
Post on 18-Dec-2021
3 Views
Preview:
TRANSCRIPT
Innovate • Design • Enhance
“Analytics for Action – enabling in-presentation evidence-based change on Open University modules”
16 Octubre de 2017
• Rafael Hidalgo - Senior TEL Designer
LTI – The Open University
Innovate • Design • Enhance
What is Analytics for Action (A4A)?
• A4A is a project to enable evidence-based change in presentation –
Active Presentation
• It started as part of the Learning and Teaching Centre’s Analytics Project
(2014-2016), encompassing:
• Real-time Student Feedback, Student progression reporting, SAS
Visual Analytics, Ethics, Predictive modelling (OU Analyse) and
Analytics 4 Action
• TEL Design provided a natural long-term home for Analytics for Action .
• A4A is run by TEL LTI since 2016/2017- Data is business as usual
Innovate • Design • Enhance
“Using the data as prompts, module teams can initiate a range of response actions, including pedagogic change and student support”
- Bart Rienties, IET, The Open University
Innovate • Design • Enhance
The Active Presentation Toolkit
• Based on the A4A
framework
• Available to all staff
in the Open
University intranet
• Step-by-step
instructions for
running an active
presentation
Innovate • Design • Enhance
What support is available?
For all modules
• Toolkit provides for a DIY approach
• Training enables staff to get started with this,
• Surgery support at Learning Systems drop-ins
• Mailbox support: LTI-TEL-Design@open.ac.uk
For selected modules:
• Data support meetings: 3 per presentation
Innovate • Design • Enhance
Which modules receive full support?
• Agreed with Faculties
• Modules identified as underperforming by the AQR – Z scores
• High student population
• Pedagogic challenges and/or innovations
• Qualifications needs
• Faculty strategic interest
Innovate • Design • Enhance
Training
• 2 hours hands-on training sessions run by Senior TEL Designers, using
real and current data.
• Priority places for MTC’s and CM’s of modules selected for formal support
• Main focus: Active Presentation Toolkit and Student Progression
Dashboards
• Small groups to allow enough discussion and specific questions
Innovate • Design • Enhance
Data support meetings
• Senior TEL Designers will support the relevant faculty modules
• Academics provide in-depth knowledge of the module,
• STELDs provide wider view as we are exposed to other data,
• Together we agree on the issues to be investigated,
• STELDs suggest actions and help in the implementation (this may
involve other units)
• Comprehensive reports on discussions and agreed actions,
• Regular updates to the faculty stakeholders about progress
Innovate • Design • Enhance
What sort of data do we look at support meetings?
Student progression dashboards
• Student profile
• Concurrency
• Assessment submission rates
• Withdrawer profiles
• Online engagement
• Online tools and resource
usage
• Retention
• Pass and retention rates
Innovate • Design • Enhance
Student profiles and concurrency
• 267 students registered at FLP25
• Roughly half registered for Q72 and a quarter for Open Degree
• 66% male, most common age group is 30-39 (37%)
• 16.5% have declared a disability
• 73.8% were studying at least another module, mostly T313, with 38 students studying 120 credits
• 79% are employed part or full time
T219
• 187 students registered at FLP25
• Q72 is the most popular qualification (30%)
• 64% male, most common age group 30-39 (37%)
• 11% have declared a disability
• 55% were studying at least one other module with 20 students studying 120 credits
• 87% are employed part or full time
T319
Innovate • Design • Enhance
Assessment Submission Rates
93.1
87.0
96.5
83.6
94.9
62.155.2 63.9
65.1 59.6
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
S104 16J SDK100 U116 S111 S10415J
% of TMA Submissions - Level 1 Science
TMA01 TMA02 TMA03 TMA04 TMA05 TMA06
Innovate • Design • Enhance
Assessment Submission Rates
87.3
70.9
59.357.1
87.6
76
60.3
0
10
20
30
40
50
60
70
80
90
100
TMA01 TMA02 TMA03 Project
% of TMA Submissions - T219 – by year
13J 14J 15J 16J
Innovate • Design • Enhance
Retention
80
82
84
86
88
90
92
94
17 18 19 20 21 22 23 24 25
% Retention
T219 T213 DST206 AVE
Innovate • Design • Enhance
Data insight: Conversations and actions
• “Significant numbers of students are studying other modules concurrently. The module team
don’t advise on multiple module study.”
Student Support Teams need to be made aware of significant number of students taking
120 points at once. A faculty discussion is also needed at qualification level.
• Are extensions any good?
• Is the gap between TMA02 and Project too long?
TMA02 weight and timing needs to be assessed
• More support from the tutor at the start of collaborative work
Module Team to consider an intro collaborative activity
• Tutors should be trained in handling the forums, are they?
Innovate • Design • Enhance
What can we do about it?
• Based on our findings we could decide:
• A change to the materials
• Some additional support to everyone
• Some targeted support
• Improvements upstream to get the right people onto the module
Innovate • Design • Enhance
What can
we do
about it?
http://article.iet.open.ac.uk/Q/QE%20report%20series/2015-QE-
learning-analytics.pdf
Innovate • Design • Enhance
Success examples
KJY113 16B: Improved pass rates
• Registered students dropped 12-13 %
from start to 25FLP. Pass rates under
faculty average for 3 years
• Additional information via dedicated
website and A5 postcard
• AL’s contacted students at risk
• Pass rate up 12%. Completion rate up
10%. TMA submissions improved. Drop
between start and FLP25 was only 3%
Innovate • Design • Enhance
Success examples
S350 16J: Managing extensions
• Concurrency levels very high: 73.4%
• 75% of students were employed
• Students unexpectedly taking Sxx390
• AL’s were given list of high workload
students and their TMA dates for
concurrent modules, to manage
extensions.
• No historical data to compare but early
intervention by AL’s was enabled by
data support meeting
Innovate • Design • Enhance
Success examples
T192 16J: design to evaluation
• In the design, Module Team addressed
a key mathematical concept –
“rearranging equations”-
• TEL Design helped to set up an
adaptive learning package of iCMAs
with support built-in via the website and
module tutors.
• Monitoring the data allowed to provide
the module team chair with the
information required to decide whether
to take action on these individual
quizzes.
-10
0
10
20
30
40
50
60
70
80
90
100
0 2 4 6 8 10 12 14 16 18 20
Number of students with scores under 40% and 50%
Scores < 40 Scores < 50
Innovate • Design • Enhance
Analytics for action: Data is business as usual
In 2016/17
• 29 OU modules fully supported – reaching almost 25,000 students
• Over 100 staff trained in 23 training sessions
• 75 data support meetings held
• Coverage expected to grow in 2017/18
Innovate • Design • Enhance
Analytics for action: The future
For 2017/18: growing numbers and quality
• Up to 49 OU modules fully supported
• Improved tools and systems
• More than 120 staff to be trained and over 100 data support meetings to be held
• Increased scholarship
• More/better data from the VLE?
Innovate • Design • Enhance
Analytics for action: What are they saying? About Data Training Tool sessions:
“The initial question I had was answered and I found out other useful things also”
“The trainers were able to support individual learners with their unique issues”
“It was informative and with help easy to navigate”
“Hands on appropriate support and guidance using real live data”
About Data Support Meetings:
“Being able to go “off grid” and actually – finally! – get some insights and answers into presentation questions we have had for ages”
“Thank you that gives us something to think about for the future presentations”
“I was updating VB on the great work that has been happening with T219/T319”
“We were enjoying ourselves so much in the world of data that we almost called up for a pizza to keep going!”
Innovate • Design • Enhance
The Open UniversityLTI TEL Design teamLevel 7 Perry B
lti-tel-design@open.ac.uk
@TEL_Design_OU
Get in touch!
top related