using learning analytics to assess innovation & improve student achievement
TRANSCRIPT
Using Learning Analytics to Assess Innovation & Improve Student Achievement
John Whitmer, [email protected]@johncwhitmer
UK Learning Analytics Network Event (JISC) March 5, 2015
http://bit.ly/jwhitmer-jisc
Quick bio
15 years managing academic technology at public higher ed institutions (R1, 4-year, CC’s)
• Always multi-campus projects, innovative uses of academic technologies
• Most recently: California State University, Chancellor’s Office, Academic Technology Services
Doctorate in Education from UC Davis (2013) with Learning Analytics study on Hybrid, Large Enrollment course
Active academic research practice (San Diego State Learning Analytics, MOOC Research Initiative, Udacity SJSU Study…)
Meta-questions driving my research
1. How can we provide students with immediate, real-time feedback? (esp identify students at-risk of failing a course)
2. How can we design effective interventions for these students?
3. How can we assess innovations (or status quo deployments) of academic technologies?
4. Do these findings apply equally to students ‘at promise’ due to their background (e.g. race, class, family education, geography)
Outline
1. Defining & Positioning Learning Analytics2. A Few Empirical Research Findings
• Understanding Contradictory Outcomes in a Redesigned Hybrid Course (Chico State)
• Creating Accurate Learning Analytics Triggers & Effective Interventions (SDSU)3. How we’re Applying this Research @ Blackboard4. Discussion
4
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.5
200MB of data emissions annually
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.6
Logged into course within 24 hours
Interacts frequently in discussion boards
Failed first exam
Hasn’t taken college-level math
No declared major
7
What is learning analytics?
Learning and Knowledge Analytics Conference, 2011
“ ...measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding
and optimizing learning
and the environments
in which it occurs.”
Strong interest by faculty & students
From Eden Dahlstrom, D. Christopher Brooks, and Jacqueline Bichsel. The Current Ecosystem of Learning Management Systems in Higher Education: Student, Faculty, and IT Perspectives. Research report. Louisville, CO: ECAR, September 2014. Available from http://www.educause.edu/ecar.
Source: Educause and AIR, 2012 (2012), http://goo.gl/337mA
2. A Few Empirical Research Findings
Study 1: Understanding Contradictory Outcomes in a Redesigned Hybrid Course (Chico State)
Course redesigned for hybrid delivery in year-long program
Enrollment: 373 students (54% increase largest section)
Highest LMS usage entire campus Fall 2010 (>250k hits)
Bimodal outcomes:• 10% increased SLO mastery• 7% & 11% increase in DWF
Why? Can’t tell with aggregated reporting data
Study Overview
54 F’s
Grades Significantly Related to Access
Course: “Introduction to Religious Studies” CSU Chico, Fall 2013 (n=373)
Variable % VarianceTotal Hits 23%Assessment activity hits 22%Content activity hits 17%Engagement activity hits 16%Administrative activity hits 12%
Mean value all significant variables 18%
LMS Activity better Predictor than Demographic/Educational Variables
Variable % Var.HS GPA 9%URM and Pell-Eligibility Interaction 7%Under-Represented Minority 4%Enrollment Status 3%URM and Gender Interaction 2%Pell Eligible 2%First in Family to Attend College 1%Mean value all significant variables 4% Not Statistically Significant
Gender Major-College
At-risk students: “Over-working gap”
Activities by Pell and grade
Grade / Pell-Eligible
A B+ C C-
0K
5K
10K
15K
20K
25K
30K
35K
Measure Names
Admin
Assess
Engage
Content
Not Pell-Eligible
Pell-Eligible
Not Pell-Eligible
Pell-Eligible
Not Pell-Eligible
Pell-Eligible
Not Pell-Eligible
Pell-Eligible
Activities by Pell and grade
Grade / Pell-Eligible
A B+ C C-
0K
5K
10K
15K
20K
25K
30K
35K
Measure Names
Admin
Assess
Engage
Content
Not Pell-Eligible
Pell-Eligible
Not Pell-Eligible
Pell-Eligible
Not Pell-Eligible
Pell-Eligible
Not Pell-Eligible
Pell-Eligible
Extra effortIn content-related activities
Study 2: Creating Accurate Learning Analytics Triggers & Effective Interventions (SDSU)
20
Study Overview
• President-level initiative
• Goal: identify effective interventions driven by Learning Analytics “triggers”
• Multiple “triggers” (e.g., LMS access, Grade, Online Homework/Quiz, Clicker use)
• At scale & over time: conducted for 3 terms, 5 unique courses, 3,529 students
• “Gold standard” experimental design (control / treatment)
21
Focus on High Need Courses
ANTH 101 COMPE 270 ECON 101 PSY 101 STAT 1190%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
84%
62%69%
78.5%70.8%
16%
38%31%
21.5%29.2%
Repeatable Grades
Non-Repeatable Grades
1. Identify courses and recruit instructors 2. Prior to course start, review syllabus, schedule meaningful
“triggers” for each course (e.g. attendance, graded items, Blackboard use, etc.)
3. Run reports in Blackboard, Online Homework/Quiz software to identify students with low activity or performance (~ weekly)
4. Send “flagged” student in experimental group a notification/intervention
5. Aggregate data, add demographic data. Analyze.
Study Protocol
22
Key Questions
1. Are triggers accurate predictors of course grade?
2. Do interventions (based on triggers) improve student grades?
3. Do these relationships vary based on student background characteristics?
Frequency of interventions (Spring 2014)
# Students Receiving >0 Interventions: PSY: 177 (84%) STAT: 165 (70%)
14%
19%
11%
17%
10%
6% 5% 6%
2% 1%3% 2%
30%
17%
13% 12%7%
6% 6%
3%
4%
1% 2%
4%
0%
5%
10%
15%
20%
25%
30%
35%
0 1 2 3 4 5 6 7 8 9 10 >10
Stud
ents
Interventions
PSY
STAT
Frequency of interventions (Spring 2015)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 140%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Triggers Activated per StudentS(Spring 2015)
Anthro1Anthro3Comp EngrEcon4Psych1Psych2Stats3Stats4
Triggers Activated
Stud
ents
25
Interventions
Spring 2014
Fall 2014
Fall 2015
26
27
A Typical Intervention: “Concerned Friend” tone
… data that I've gathered over the years via clickers indicatesthat students who attend every face-to-face class meeting reduce their chances of getting a D or an F in the class from almost 30% down to approximately 8%.
So, please take my friendly advice and attend class and participate in our classroom activities via your clicker. You'll be happy you did! Let me know if you have any questions.
Good luck,Dr. Laumakis
Poll question
A Not significant
<10%, significant .05 level
20%, significant .01 level
30%, significant .001 level
Did triggers predict achievement? What level significance? How much variation in student grade was explained?
B
C
D
E 50%+, significant .0001 level
28
Poll question
A Not significant
<10%, significant .05 level
20%, significant .01 level
30%, significant .001 level
Did triggers predict achievement? What level significance? How much variation in student grade was explained?
B
C
D
E 50%+, significant .0001 level (Spring 2014, Fall 2014)
29
Statistics
Learning analytics triggers vs. final course pointsSpring 2014: 4 sections, 2 courses, 882 students
Psychology
p<0.0001; r2=0.4828 p<0.0001; r2=0.6558
Fall 2014 results: Almost identical5 Sections, 3 Courses, N=1,220 students
p<0.00001; r2=0.4836
Spring 2015 Results (tentative): lower relationship
8 Sections, 5 Courses, N=1,390 students
p<0.00001; r2=0.28
32
Spring 2015 Results (tentative): lower relationship
8 Sections, 5 Courses, N=1,390 students
p<0.00001; r2=0.28
33
Explained by differences between courses (Spring 2015 Results by Course)
34
So did the interventions make a difference in learning outcomes?
Control (STAT)
Exp. (STAT)
Exp. (PSY) Control (PSY)
0%10%20%30%40%50%60%70%80%90%
100%
79% 79%
89%82%
21% 21% 11% 18%
Experimental Participation vs.
Repeatable Grade (Spring 2014)
Repeati-ble Grade
Passing Grade
36
No Interventions _x000d_(n=87, PSY,
Pell-Eligible)
Interventions _x000d_(n=81, PSY,
Pell-eligible)
0%10%20%30%40%50%60%70%80%90%
100%
77%
91%23%9%
Experimental Participation vs. Repeatable Grade (Pell-El-
igible) (n=168, Spring 2014, PSY
101) Passing Grade
Repeati-ble Grade
24 additional Pell-eligible students would have passed the class
if the intervention was applied to all participating students.
37
Fall 2014 / Spring 2015 Intervention Results:
No Significant Difference Between Experimental/Control Groups.
38
One Explanation: Low Reach
39
Fall 2014 (n = 1,220)
Row Labels # Triggers Message Open Rate Clickthrough Rate
Econ1 8 76% 36%Psych1 6 70% 29%Psych2 7 69% 35%Stat3 9 62% 25%Stat4 8 65% 27%Grand Total 38 68% 30%
Spring 2015 (n = 1,138)
Row Labels # Triggers Message Open Rate Clickthrough Rate
Anthro-In Person 17 57% 10%Anthro-Online 7 71% 35%Comp Engineering
15 52% 14%
Econ 15 44% 13%Psych1 17 60% 13%Psych2 17 63% 13%Stat3 21 64% 9%Stat4 20 55% 5%Grand Total 129 58% 12%
Proposed Next Steps
Add interventions that move “beyond informing” students to address underlying study skills and behaviors
Supplemental Instruction <http://www.umkc.edu/asm/si/>
Adaptive Release within online courses (content, activities)
40
1. Data from academic technology use predicts student achievement; diverse sources provide better predictions.
2. Tech use > demographic data to predict course success; adding demographic data provides nuanced understandings and identifies trends not otherwise visible.
3. Academic technology use is not a “cause” in itself, but reveals underlying study habits and behaviors (e.g. effort, time on task, massed vs. distributed activity).
4. Predictions are necessary, but not sufficient, to change academic outcomes. Research into interventions is promising.
5. We’re at an early stage in Learning Analytics; expect quantum leaps in the near future.41
Conclusions and Implications
4. How we’re Applying this Research @ Blackboard
42
Improved instrumentation for learning activity within applications
Blackboard’s “Platform Analytics” Project
A new effort to enhance our analytics offerings across our academic technology applications that include
Applied findings from analysis (inc. inferential statistics and data mining)
Integrated analytics into user experiences(inc. student and faculty)
Aggregated usage data across cloud applications (anonymized, rolled-up, privacy-compliant)
Blackboard Analytics for Learn
3. Wrap-Up and Discussion
Factors affecting growth of learning analytics
Enabler
Constraint
WidespreadRare
New education models
Resources ($$$, talent)
Data governance (privacy, security, ownership)
Clear goals and linked actions
Data valued in academic decisions
Tools/systems for data co-mingling and analysis
Academic technology adoption
Low data quality (fidelity with meaningful learning)
Difficulty of data preparation
Not invented here syndrome
Call to action [with amendments](from a May 2012 Keynote Presentation @ San Diego State U)
You’re not behind the curve, this is a rapidly emerging area that we can (should) lead... [together with interested partners]
Metrics reporting is the foundation for analytics [don’t under or over-estimate the importance]
Start with what you have! Don’t wait for student characteristics and detailed database information; LMS data can provide significant insights
If there’s any ed tech software folks in the audience, please help us with better reporting! [we’re working on it and feel your pain!]
