Moving Forward: Lessons Learned From The Last 10 Years of
Risk Modeling
Presentation Agenda1) Provide a framework for defining risk 2) Examine the role of theory and personal bias in the modeling process 3) Help outline possible partnerships needed for successful processes (data extraction, intervention design, student outreach) 4) Review a list of data sources, variables and tools needed for successful program implementation 5) Guide participants in identifying next steps in their own Risk Identification Programs 6) Examine the impact individualized reports and ad-hoc reports play in day to day lives of advising administrators
● Top-tier public research university● Founded by Russell H. Conwell in 1884● Located in Philadelphia, PA● Average Combined SAT (1170)● Percent Admitted (79.9%)● Undergraduate Enrollment: 29,416
Retention Rate: 89.7%
Graduation Rates: (2012 Cohort)
4-year: 48%6-year: 70%
Framework:
Defining Risk
● At risk for what?● Defining Retention and Academic Success● 20% Rule
Data Mining Definition
70 - 90% - Spent Cleaning and Understanding data
The Role of Theory and Personal Bias in the Modeling Process
● Dataset Selection● Variable Inclusion & Exclusion● Tinto, Astin, Chicoring + Others
It Takes a Village (campus)Business IntelligenceInstitutional ResearchBanner Competency CenterColleges & Advising OfficesHousing OfficeDisability Resource CenterRegistrar Office Office of OrientationCareer Center
Tools● Cognos● Query Studio● Tableau● SPSS (Modeler)● Good Old Friend Excel● Canvas LMS
PastRegression Model
Present - Existing Structure4 Models updated each year for all newly admitted Freshman & Transfer students
○ Admission - Placement ○ Orientation - Add/Drop ○ Add-Drop - W date ○ End of Term
Present: Configural Analysis (Classification or Segmentation)
Databases Variables Used - Student Life Cycle○ Frozen Cohort Files○ Admissions - (High School Info, demographic)○ College/Major (Change of either)○ Financial Aid (EFC, Pell, Unmet Need)○ New Student Questionnaire (Q1 - Q83)○ Placement Scores and Levels (Eng, Math, Aleks Sub Scores, SATs)○ Orientation Information○ Housing + LLC○ Term Balance (Start - #of Hrs Registered & End of Term), T1 - T8○ Mid Semester Grades/Alerts (Cumulative)○ Hold Indicators, Multiple W Grades○ Special Populations○ Flyin4 Participation○ Delta GPA Term 1 GPA - HS GPA * Most Predictive
Models and Key Variables
Model 1: Admissions - NSQ Placement (June 15th)● Admit Type● # of Days between Deposit and First Day of Class or September 1 (Calculated)● # of Days between Orientation and First Day of Class or September 1 (Calculated)● Race*● HS GPA, Rank, Percentile, Size of HS● HS Code (GPA of all students who went to it before)● Admissions Ranking● Unmet Need/Offer Amount● Academic Major/Program*● Parental Education● Admissions Letter Code*● SAT/ACT● NSQ ~ 15 Variables● Placement (Math, ENG, Aleks Score)
Models and Key Variables
Model 2: Orientation - Registration - Change of College/Major (September 12th)● Census Date - Day after Add/Drop period ends● Change in college since Orientation (Y/N) as of Census Date● # of Credits Registered (FT/PT)● Date of Orientation (Late Orientation for TR)● LLC vs Non LLC● Balance as of Add/Drop● Housing assignment (Name of the Building)● Unmet Need● EFC● Pell Amount
Models and Key Variables
Model 3: First Semester Half Mark (6 Weeks)● 1 week after the “W” date● # of UG Mid semester Alerts● # of W grades (1 week after 2 date)● Balance as of W date● EFC & Unmet Need ● Total Amount Borrowed ● Student’s College/Major● Special Populations
Models and Key Variables
Model 4: End of Term 1 Marker
● January 2nd● Hold Indicator● Term 1 GPA (Academic Standing)● GPA Drop HS - Term 1 (1.25 drop from HS, Honors = .85)
Balance as of Final Grades > $200● Housing assignment On or Off● #of Credits Passed <12● Math & Eng Grades● # of UG MidSemester Alerts
Risk Liaison Intervention Training Summer Meetings – July 11th and August 9thJuly: The Ones that Got Away: Canvas Review, Risk List Review & Group IntroductionsAugust: Designing Fall Interventions and Strategic Communications for students at Risk, on Probation and on Warning
Fall Meetings (Fourth Wednesday of the Month @3PM)September Working with Data & Final Freshman & Transfer Risk List Review (Bring your laptop and Excel)October CSRDE Comparative Reports Review & Retention WorkshopNovember Failed to Launch: Mid-Semester Alerts, LOA, WE StudentsDecember Designing Spring Interventions and Strategic Communications for students at Risk and on Warning
Spring Meetings- Fourth Wednesday @3PMFebruary The Murky Middle 2.0 – 2.5March Intervention Discussion (What Worked & What Didn’t)April No MeetingMay Intervention Assessment
InterventionsAddition of mentoring and peer mentoring supportsSpecialized Workshops (Financial Aid, Housing, etc)Mid Semester Initiatives (Letters, phone outreach)Outreach is done by Financial Aid, Housing, DRS, other supporting units not just advisingJobs on campus and work study positions○ (Need to identify students eligible earlier)
InterventionsAdvisor ratios reviewSpecialized advisors focused on riskiest of studentsAdvisor intervention design and data trainingShift from intrusive to developmental & appreciative advisingShift from group to individual advising for riskiest of students
Retention Canvas Site
Advising Director Feedback● Results of Qualitative interviews with 12 Schools/Colleges● Qs Asked
○ How have you used Risk List?○ What challenges have you faced in working with at-Risk students?○ What data/reports would make your life easier?○ What were the most successful interventions done by your department?
Future - Individualized Risk ModelsAdditional Information to be added to the models
● Retention Defined by College (i.e. Retained in Engineering not TU) ● Special Populations Tracked and Monitored
a. Off/On Campus, first gen, low income, adult 24+, LLC, Pre-Business, Pre-Health, Undeclared with 60+ credits)
● Examining impact of Grades/Mid Semester Alerts in specific Courses○ (First Semester ENG & MATH, STAT, ENG, etc.)
● Focus on Freshman, Sophomore & Transfer ● Focus on Specific Majors (Biology, Undeclared (all colleges), Psychology, Biochemistry, Business
Management, Pre-Pharmacy, Communication Studies, Computer Science, Early Childhood Education, Engineering, Political Science)
● June, September, January review of models and at-risk student lists/dashboards ● Monthly Advisor/Risk Liaison Workshops
Examining Internal University Student Migration
Examining Program Level Retention and Migration
CSRDE - National Student Consortium for
ReportsWebinars and Advisor TrainingComparative data
Future Partnerships
QuestionsAlexandra Yanovski-BowersAssistant Director for Undergraduate Strategic Initiatives Office of the Vice Provost for Undergraduate StudiesTemple University500 Conwell Hall1801 N. Broad StreetPhiladelphia, PA 19122 [email protected] p: 215-204-7596https://www.linkedin.com/in/alexyanovski