Download - EDUCA Leveraging Analytics FINAL
![Page 1: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/1.jpg)
Leveraging Analytics to Improve Student Success
Karen Vignare, University Maryland University College
@kvignare
Ellen Wagner, PAR Framework@edwsonoma
![Page 2: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/2.jpg)
Session Description
• This session shows how analytics can be used to identify opportunities for improving student success.
• By the end of the session, participants will make connections between predictions about risk, and the interventions most likely to work best under varying conditions and with different populations.
![Page 3: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/3.jpg)
Setting the Context:
Data Are Changing Everything
![Page 4: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/4.jpg)
“But education researchers have
always worked with data.”
• We do qualitative research with data
• We do quantitative research with data
• We do evaluations with data
• We develop surveys and instruments and experiments to collect more data
• We pull data from LMSs, SISs, ERPs, CRMs …
• We write reports, summaries, make presentations, develop articles and books and webcasts….
![Page 5: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/5.jpg)
From Hindsight to Foresight
![Page 6: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/6.jpg)
6
![Page 7: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/7.jpg)
Analytics in Higher Education
Learning Analytics
Best way to teach and learn
Learner Analytics
Best way to support students
Organizational Analytics
Best ways to operate a college
Academic Analytics
![Page 8: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/8.jpg)
Create new insights and opportunities for
data in our practices
• Enrollment management
• Student services
• Program and learning experience design
• Content creation
• Retention, completion
• Gainful employment
• Institutional Culture
![Page 9: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/9.jpg)
How Are We Doing So Far?
• Data is the number 1 challenge in the adoption and use of analytics. Organizations continue to struggle with data accuracy, consistency, access.
• The primary focus of analytics focuses on reducing costs, improving the bottom line, managing risk.
• Intuition, based on experience, is still the driving factor in data-driven decision-making. Analytics are used as a part of the process.
• Many organizations lack the proper analytical talent. Organizations that struggle with making good use of analytics often don’t know how to apply the results.
• Culture plays a critical role in the effective use of data analytics. 9
![Page 10: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/10.jpg)
GROUP DISCUSSION
• Is your institution using (or planning to use) academic analytics specifically to improve student success?
• What kinds of questions are you trying to answer?
• What kinds of data are you planning to use?
• What kinds of barriers are you encountering?
![Page 11: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/11.jpg)
Getting to the right answer takes work
• Analysis and model building is an iterative process
• Around 70-80% efforts are spent on data exploration and understanding.
SAS Analysis/Modeling Process
![Page 12: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/12.jpg)
Link Predictions to Action
• Predictive analytics refer to a wide varieties of methodologies. There is no single “best” way of doing predictive analytics. You need to know what you are looking for.
• Simply knowing who is at risk is simply not enough. Predictions have value when they are tied to what you can do about it.
• Linking behavioral predictions of risk with interventions at the best points of fit offers a powerful strategy for increasing rates of student retention, academic progress and completion.
![Page 13: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/13.jpg)
Collaborative
National
Multi-institutional
Non-profit
Institutional Effectiveness +
Student Success
![Page 14: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/14.jpg)
What PAR does
PAR uses descriptive, inferential and predictive analyses to create benchmarks, institutional predictive models and to inventory, map and measure student success interventions that have direct positive impact on behaviors correlated with success.
![Page 15: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/15.jpg)
Linking Predictions to Action
• Identify obstacles and remove barriers from student success pathways.
• Provide actionable information so students and advisors can build informed opportunity pathways.
• Know where to invest in student success leveraging collaborative insight that determine return on investment in interventions and support.
![Page 16: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/16.jpg)
Benchmarks & Insight Predictive Analytics Intervention Inventory and ROI Tools
Diagnostics
PAR analytic toolset
![Page 17: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/17.jpg)
Benchmarks & Insight Predictive Analytics Intervention Inventory and ROI Tools
Web Tools
Student Success Matrix (SSMx)
![Page 18: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/18.jpg)
PAR by the Numbers
• 2.2 million students and 24.5 million courses in the PAR data warehouse, in a single federated data set, using common data definitions.
• 48 institutions, 351 unique campuses.
• 77 discrete variables are available for each student record in the data set. Additional 2 dozen constructed variables used to explore specific dimensions and promising patterns of risk and retention.
• 343 discrete interventions filtered on predictor behaviors, point in student life cycle, student attributes, institutional priorities and ROI factors in the growing SSMx dataset.
![Page 19: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/19.jpg)
Structured, Readily Available Data
• Common data definitions = reusable predictive models and meaningful comparisons.
• Openly published via a cc license @ https://public.datacookbook.com/public/institutions/par
![Page 20: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/20.jpg)
Speak the same
language
![Page 21: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/21.jpg)
PAR Puts it All Together
Determine students probability of failure
(predictions)
Determine which students respond to interventions (uplift
modeling)
Determine which interventions are most effective (explanatory
modeling)
Allocate resources accordingly (cost benefit analysis)
![Page 22: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/22.jpg)
Findings from aggregated dataset
Positive Predictors
High school GPA (when available)
Dual Enrollment – HS/College
Any prior credit
CC GPA
Credit Ratio
Successful Course Completion
Positive completion of DevEd
Courses
Negative Predictors
Withdrawals
Low # of credits attempted
Varies but can be significant
PELL Grant Recipient
Taken Dev Ed
Age
Fully online student
Race
![Page 23: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/23.jpg)
• Measurement resources are usually located separately from intervention planning & implementation resources
• Lack of connection of predictors to interventions and interventions to outcomes
©PAR Framework 2015
Common Challenges for
Intervention Effectiveness
![Page 24: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/24.jpg)
PAR Student Success Matrix (SSMx)
• An organizational structure that helps institutions inventory, organize and conceptualize interventions aimed at improving student outcomes.
• A common framework for classifying interventions
• Provides a basis for intervention measurement
©PAR Framework 2015
![Page 25: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/25.jpg)
learner characteristics
learner behaviors
fit/feelings of belonging
other learner support
course/program characteristics
instructor behaviors
time connection entry progress completion
predictors
©PAR Framework 2015
![Page 26: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/26.jpg)
SMALL GROUP DISCUSSION
How Are You Measuring
Interventions at YOUR Institution?
![Page 27: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/27.jpg)
Specific Examples of
Data Driven Improvements
• UMUC / U of Hawaii – replication of community college success prediction studies
• U of Hawaii – “Obstacle courses”
• University of North Dakota – predictives tied to student watchlist data
• Intervention measurement at Sinclair CC and Lone Star CC
• National online learning impact study on student retention (in press, based on results from >500,000 students taking onground, blended and online courses)
![Page 28: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/28.jpg)
Intervention Measurement –
Student Success Courses Results
• 12 month credit ratio: Only 1 of the 8 Student Success Courses analyzed showed a statistically significant positive effect for students taking the course vs. those who did not.
• Retention: 7 of the 8 courses showed a significantly positive effect
• Retention higher by 14% to 4X
![Page 29: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/29.jpg)
Intervention Measurement –
Student Success Courses
Course Component Summary:
![Page 30: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/30.jpg)
Public university offering online degree
programs to a diverse population of
working adults
Largest open access public online
university in U.S.
Premier provider of higher education to
U.S. military since 1949
Part of the University System of Maryland
About UMUC
![Page 31: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/31.jpg)
20th Century
Historical
Longitudinal
Warehouse
Siloed
External
Reporting
21st Century
Predictive
Real-Time
Dashboards
Integrated Institutional Insights
Continuous Improvements
Evolution of Data for Retention
![Page 32: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/32.jpg)
Institutional Research
Institutional Effectiveness
Business Intelligence
Civitas Learning, Inc.
PAR Framework, Inc.
Retention Resources at UMUC
![Page 33: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/33.jpg)
Pre-enrollment
Demographics
Enrollment
LMS Engagement
Student Performance
Transfer
Military
Factors Included in Predictive Model for Retention at UMUC
![Page 34: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/34.jpg)
Campus
Class Load
Military Status
Academic Performance
Payment Method
Key Factors for Retention at UMUC
![Page 35: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/35.jpg)
One year retention (year over year measured with a cohort)
Re-enrollment (term to term metric that includes all students)
Successful course completion (percentage of students receiving a successful grade)
Graduation (1,2,3,4,5, and 10 year rate tracks the graduation status of the starting cohort over time)
Metrics at UMUC
![Page 36: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/36.jpg)
Curriculum Redesign (2010)
8-week Standard Sessions (2010)
Community College Transfer (2010)
Registration Policy (2013)
Onboarding (2014)
Just-in-Time Messages (2014)
Retention Initiatives
![Page 37: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/37.jpg)
71.2 72 71.6 73.2
60.5 59.5 61.566
0
10
20
30
40
50
60
70
80
Fall 2011 Fall 2012 Fall 2013 Fall 2014
Stateside
Overseas
Retention Rates and Headcounts
47,416 46,213 41,197 41,356
![Page 38: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/38.jpg)
Operationalize
successful tests;
“Lessons
Learned” fed
back
to body of
knowledge
Student Retention Enterprise Framework
Diagnosed
from internal
data and
external
research
Root cause
analysis
performed
and search
of existing
body of
knowledge
solutions
Work within
Governance
Structure
Levers pulled
here;
Measure
success &
ROI;
Quarterly
Reviews
Retention Root Cause Identification & Analysis
Retention
Opportunity
Problem
AnalyzedHypothesis
Generated
Test &
Learn Cycle
Operationaliz
e or Re-
create
![Page 39: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/39.jpg)
Discussion
How will you begin, or improve, your
analytics journey at YOUR institution?
![Page 40: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/40.jpg)
Elements of a Data Model
Use modeling to
Test likely impact on retention when new
initiatives or planned interventions are
undertaken
Create models that build out retention
impact by segments, e.g., demographics,
academic programs, persistence, etc.
![Page 41: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/41.jpg)
Continual Improvement
Design Intervention
Collect Data
Analyze Data
Refine or Sunset
![Page 42: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/42.jpg)
![Page 43: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/43.jpg)
DISCUSSION
![Page 44: EDUCA Leveraging Analytics FINAL](https://reader036.vdocuments.site/reader036/viewer/2022081323/5a66a4277f8b9ac5128b5f37/html5/thumbnails/44.jpg)
THANK YOU FOR YOUR INTEREST