risk modeling moving forward: lessons learned from the ... › sites › default › files › files...

Moving Forward: Lessons Learned From The Last 10 Years of

Risk Modeling

Presentation Agenda1) Provide a framework for defining risk 2) Examine the role of theory and personal bias in the modeling process 3) Help outline possible partnerships needed for successful processes (data extraction, intervention design, student outreach) 4) Review a list of data sources, variables and tools needed for successful program implementation 5) Guide participants in identifying next steps in their own Risk Identification Programs 6) Examine the impact individualized reports and ad-hoc reports play in day to day lives of advising administrators

● Top-tier public research university● Founded by Russell H. Conwell in 1884● Located in Philadelphia, PA● Average Combined SAT (1170)● Percent Admitted (79.9%)● Undergraduate Enrollment: 29,416

Retention Rate: 89.7%

Graduation Rates: (2012 Cohort)

4-year: 48%6-year: 70%

Framework:

Defining Risk

● At risk for what?● Defining Retention and Academic Success● 20% Rule

Data Mining Definition

70 - 90% - Spent Cleaning and Understanding data

The Role of Theory and Personal Bias in the Modeling Process

● Dataset Selection● Variable Inclusion & Exclusion● Tinto, Astin, Chicoring + Others

It Takes a Village (campus)Business IntelligenceInstitutional ResearchBanner Competency CenterColleges & Advising OfficesHousing OfficeDisability Resource CenterRegistrar Office Office of OrientationCareer Center

Tools● Cognos● Query Studio● Tableau● SPSS (Modeler)● Good Old Friend Excel● Canvas LMS

PastRegression Model

Present - Existing Structure4 Models updated each year for all newly admitted Freshman & Transfer students

○ Admission - Placement ○ Orientation - Add/Drop ○ Add-Drop - W date ○ End of Term

Present: Configural Analysis (Classification or Segmentation)

Databases Variables Used - Student Life Cycle○ Frozen Cohort Files○ Admissions - (High School Info, demographic)○ College/Major (Change of either)○ Financial Aid (EFC, Pell, Unmet Need)○ New Student Questionnaire (Q1 - Q83)○ Placement Scores and Levels (Eng, Math, Aleks Sub Scores, SATs)○ Orientation Information○ Housing + LLC○ Term Balance (Start - #of Hrs Registered & End of Term), T1 - T8○ Mid Semester Grades/Alerts (Cumulative)○ Hold Indicators, Multiple W Grades○ Special Populations○ Flyin4 Participation○ Delta GPA Term 1 GPA - HS GPA * Most Predictive

Models and Key Variables

Model 1: Admissions - NSQ Placement (June 15th)● Admit Type● # of Days between Deposit and First Day of Class or September 1 (Calculated)● # of Days between Orientation and First Day of Class or September 1 (Calculated)● Race*● HS GPA, Rank, Percentile, Size of HS● HS Code (GPA of all students who went to it before)● Admissions Ranking● Unmet Need/Offer Amount● Academic Major/Program*● Parental Education● Admissions Letter Code*● SAT/ACT● NSQ ~ 15 Variables● Placement (Math, ENG, Aleks Score)

Models and Key Variables

Model 2: Orientation - Registration - Change of College/Major (September 12th)● Census Date - Day after Add/Drop period ends● Change in college since Orientation (Y/N) as of Census Date● # of Credits Registered (FT/PT)● Date of Orientation (Late Orientation for TR)● LLC vs Non LLC● Balance as of Add/Drop● Housing assignment (Name of the Building)● Unmet Need● EFC● Pell Amount

Models and Key Variables

Model 3: First Semester Half Mark (6 Weeks)● 1 week after the “W” date● # of UG Mid semester Alerts● # of W grades (1 week after 2 date)● Balance as of W date● EFC & Unmet Need ● Total Amount Borrowed ● Student’s College/Major● Special Populations

Models and Key Variables

Model 4: End of Term 1 Marker

● January 2nd● Hold Indicator● Term 1 GPA (Academic Standing)● GPA Drop HS - Term 1 (1.25 drop from HS, Honors = .85)

Balance as of Final Grades > $200● Housing assignment On or Off● #of Credits Passed <12● Math & Eng Grades● # of UG MidSemester Alerts

Risk Liaison Intervention Training Summer Meetings – July 11th and August 9thJuly: The Ones that Got Away: Canvas Review, Risk List Review & Group IntroductionsAugust: Designing Fall Interventions and Strategic Communications for students at Risk, on Probation and on Warning

Fall Meetings (Fourth Wednesday of the Month @3PM)September Working with Data & Final Freshman & Transfer Risk List Review (Bring your laptop and Excel)October CSRDE Comparative Reports Review & Retention WorkshopNovember Failed to Launch: Mid-Semester Alerts, LOA, WE StudentsDecember Designing Spring Interventions and Strategic Communications for students at Risk and on Warning

Spring Meetings- Fourth Wednesday @3PMFebruary The Murky Middle 2.0 – 2.5March Intervention Discussion (What Worked & What Didn’t)April No MeetingMay Intervention Assessment

InterventionsAddition of mentoring and peer mentoring supportsSpecialized Workshops (Financial Aid, Housing, etc)Mid Semester Initiatives (Letters, phone outreach)Outreach is done by Financial Aid, Housing, DRS, other supporting units not just advisingJobs on campus and work study positions○ (Need to identify students eligible earlier)

InterventionsAdvisor ratios reviewSpecialized advisors focused on riskiest of studentsAdvisor intervention design and data trainingShift from intrusive to developmental & appreciative advisingShift from group to individual advising for riskiest of students

Retention Canvas Site

Advising Director Feedback● Results of Qualitative interviews with 12 Schools/Colleges● Qs Asked

○ How have you used Risk List?○ What challenges have you faced in working with at-Risk students?○ What data/reports would make your life easier?○ What were the most successful interventions done by your department?

Future - Individualized Risk ModelsAdditional Information to be added to the models

● Retention Defined by College (i.e. Retained in Engineering not TU) ● Special Populations Tracked and Monitored

a. Off/On Campus, first gen, low income, adult 24+, LLC, Pre-Business, Pre-Health, Undeclared with 60+ credits)

● Examining impact of Grades/Mid Semester Alerts in specific Courses○ (First Semester ENG & MATH, STAT, ENG, etc.)

● Focus on Freshman, Sophomore & Transfer ● Focus on Specific Majors (Biology, Undeclared (all colleges), Psychology, Biochemistry, Business

Management, Pre-Pharmacy, Communication Studies, Computer Science, Early Childhood Education, Engineering, Political Science)

● June, September, January review of models and at-risk student lists/dashboards ● Monthly Advisor/Risk Liaison Workshops

Examining Internal University Student Migration

Examining Program Level Retention and Migration

CSRDE - National Student Consortium for

ReportsWebinars and Advisor TrainingComparative data

Future Partnerships

QuestionsAlexandra Yanovski-BowersAssistant Director for Undergraduate Strategic Initiatives Office of the Vice Provost for Undergraduate StudiesTemple University500 Conwell Hall1801 N. Broad StreetPhiladelphia, PA 19122 ayanovski@temple.edu p: 215-204-7596https://www.linkedin.com/in/alexyanovski