Data to Inform and Assess Curricular Innovations

MARCO MOLINARO, UC DAVIS | BRIAN SATO & RAY VADNAIS, UC IRVINE

Essential Questions & Data Sources

§Provides a set of questions that can be used at multiple levels to assess progress

§Identifies data sources and analytical tools to help answer these questions

§Recommends ways to address challenges to institutional and multi-institutional evaluation

mmolinaro@ucdavis.eduCEE.ucdavis.edu

Supporting Undergraduate Instructional Improvement

Our Changing Student DemographicsFall 2017 Continuing and New Student Demographics

Ong

oing

Stu

dent

sN

ew S

tude

nts

international

Example Work Focused Campus-wide

*CA residents only

GPA Averages in All first year Courses & Opportunity – One AY*

AFg

0=NotFG FG

2.65

2.70

2.75

2.80

2.85

2.90

2.95

3.00

3.05

3.10

3.15

3.20

3.25UCDOverallGpa

1,691

2,103

SIsUrm

NotURM URM

2.65

2.70

2.75

2.80

2.85

2.90

2.95

3.00

3.05

3.10

3.15

3.20

3.25

UCDOverallGpa

1,126

2,595

FG Status URM Status

Gap with URM or FGAppears similar in GPA points

TRADITIONAL APPROACHGP

ASC

ALE

GPA

SCAL

E

URM_FG_LIcombinations

NotTriple Triple FGonly LIonly URMonly FG&LI URM&FG URM&LI other

2.65

2.70

2.75

2.80

2.85

2.90

2.95

3.00

3.05

3.10

3.15

3.20

3.25

UCDOverallGpa

311

other

68

URM&LI

228

URM&FG

483

FG&LI

233

URMonly205

LIonly

362

FGonly

550

Triple

1,400

NotTriple

IntersectF15entrySp16GPA

*CA residents with valid zipcode only

GPA Averages in All first year Courses & Opportunity – AY 2015-16*Averages vs Box Plots

URM_FG_LIcombinations

NotTriple Triple FGonly LIonly URMonly FG&LI URM&FG URM&LI other

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

4.0

TUcdOverallGpa

IntersectF15entrySp16GPAboxplot

NOTE

SCA

LE C

HANG

E

GPA

SCAL

E

DIFF

EREN

T G

PA S

CALE

What Might CauseThese Differences?

Regional Opportunities Index (ROI)http://interact.regionalchange.ucdavis.edu/roi/about.html

Regional Opportunities Index (ROI)

http://interact.regionalchange.ucdavis.edu/roi/webmap/webmap.html

PEOPLE PLACE

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

Avg.ZPeopleMean

other

URM&LI

URM&FGFG&LI

URMonly

LIonly

FGonly

Triple

NotTriple

Peoplemean

only

37.5

38.0

38.5

39.0

39.5

40.0

40.5

41.0

41.5

42.0

42.5Avg.ZPlaceMean

other

URM&LI

URM&FG

FG&LI

URMonly

LIonly

FGonly

Triple

NotTriple

Placemean

onlyURM_FG_LIcombinations

NotTriple Triple FGonly LIonly URMonly FG&LI URM&FG URM&LI other

2.65

2.70

2.75

2.80

2.85

2.90

2.95

3.00

3.05

3.10

3.15

3.20

3.25

UCDOverallGpa

311

other

68

URM&LI

228

URM&FG

483

FG&LI

233

URMonly205

LIonly

362

FGonly

550

Triple

1,400

NotTriple

IntersectF15entrySp16GPA

GPA Averages in All first year Courses & Opportunity – One AY*

*CA residents only

GPA

SCAL

E

PLAC

E SC

ALE

PEO

PLE

SCAL

E

Exploring How We Can Approach Change

Cycle of Progress

Awareness Understanding

ActionReflection

Fostering a Cycle of Progress

Cycle of Progress

Awareness Understanding

ActionReflection

One Person’s Action can spread awareness and understanding

Conscious effort to focus on Faculty, Professional Staff and Administrators to impact Student & System outcomes

Cycle of Progress

Awareness Understanding

ActionReflection

One Person’s Action can spread awareness and understanding, a group expands

Cycle of Progress

Awareness Understanding

ActionReflection

Awareness Understanding

ActionReflection

One Person’s Action can spread awareness and understanding, a group expands, the system lingers

Now to UC Irvine

Now to UC Davis

Tools & Processes to Support Faculty

Faculty DepartmentalLeadership

College/UnivLeadership

Know Your Students Department Diagnostics Dean’s Undergrad Metrics

Self-ServiceCommunity

Targeted Support

Self-ServiceRoutine Analyses

Deep Dives

OngoingFocused Support

Systemic Studies (Intersectionality, WLD, …)

Custom Analyses

Example Work Focused at Department/College

Level

Retention in a Major – Multiple freshmen cohorts

Female bothMale nFG

Male FG

major

Freshmen Student Migrations – Multi-year starting cohorts

DFW and GPA Gap Variables

Gender

First GenerationLow IncomeURMURM/FG/LI

ESLInternational

Gender

SES Opportunity

Language

OVERALL

Gender Socio-Economic Opportunity Language1st Gen Low Inc URM URM/FG/LI Int’l ESLGender

All Variable Gaps (DFW, GPA) for a College

AWARENESS ONLY

Making Dashboard ActionableApproaches to foster greater understanding?

INITIAL IDEAS Absolute vs weighted impactRaw vs adjusted for prior characteristics Specialness of first year courses Course orderCourse sizeGrading practices – norm vs criterion referencedInstructional variability

UNDERSTANDING?

And then we need to move on to ACTION and REFLECTION

Example Work Focused at Course/Instructor

Level

Multidimensional Instructional Development for Achievement and Success (MIDAS) UCD Inclusive Excellence project funded in part by HHMI – Facciotti/Molinaro Co-PIs

Major findings related to emergent research on: low-income/SES, first generation, URM & transfer student success

Academic preparedness, motivation, and persistence

Implications for pedagogy

Instructional & Support Community

BibliographyTeaching Strategies

Kno

w Y

our S

tude

nts

Das

hboa

rd

Teaching Portfolio

AWARENESSUNDERSTANDINGACTIONREFLECTION

Faculty

Instructional Exploration& Data Gathering

Know Your Students– FG and Repeaters

Course taught by HHMI Project Co-PI Prof. Marc Facciotti

Know Your Students– ESL Example

Know Your Students– First Generation Just in Time Teaching

JiTT (Just in Time Teaching)Promoting First Generation Student Academic Success

JiTT (Just in Time Teaching)Academic Success, Teaching Strategies, Social/Community Integration

Connecting to the Instructional Support CommunityFaculty, campus professionals, campus services

Know Your Students– UCD Experience and Majors

Know Your Students– Concurrent/Prior Courses and Gaps

Know Your Students– TAs, Grade Distributions

Know Your StudentsFuture Course Performance (past courses only – not predictions)

Next Course

Junior Level course

Enhancing Know Your Students dataNew sources for greater understanding

Data Sources- DART, COPUS, LOPUS, Clicker use, LMS use- Surveys - student check-in, time on task- Teaching practices, inclusivity- Bloom’s analysis of learning objectives- Student Course Evaluations (NLP)- Use of campus tutoring resources

Gradebook analysis for gaps (ESL) and targeted supports

Instructional Exploration& Data Gathering

Enhance Individuals to System Effectiveness Towards Students

Awareness Understanding

ActionReflection

Undergraduate Education Initiatives

List of all People Involved in this Work – Expanding constantly

BIOLOGYSusan KeenKen BurtisMark GoldmanAnne BrittMitchell SingerMichelle IgoJoel LedfordMarina CrowderNatalia CaporaleMona ManfredBIS2 TAsErin Easlon

INST. RESEARCHErika JacksonJason StewartEric RothgarnAnnamalai NatarajanKamal LemsefferBob Loessberg-ZahlGillian ButlerHelen PaikJessica Loudermilk

PsychologySteve LuckVictoria CrossCynthia Pickett

REGISTRARBrad HardingThom PonElias Lopez

ADMISSIONSDona Owfook

IET/ATSDavid LevinAndy Jones

PHYSICSMax ChertokAndreas AlbrechtMarusa BradacDavid WebbRandy HarrisWendell Potter

CHEMISTRYSusan KauzlarichNeil SchoreAnneliese FrantzDelmar LarsenCarlito LebrillaOzcan GulacarJulia ChamberlainTing GuoBryan EnderleA. ToupadakisMatt AugustineDean TantilloAndrew FisherXi ChenPeter KellyGreg Allen

CEEMarco MolinaroKem SaichaieMatt SteinwachsStephanie PulfordMeryl MotikaPatricia TurnerCecilia GomezKatie MartinezMonica EsquedaKavi TanKatie MartinezBarbara MillsCandice BoulwareCara TheisenXinyi HouRaymund AlksynisChris PagliaruloCatherine UvarovAlberto Guzman-AlvarezErin BeckerAlisa LeeQiwei LiHaomiao Meng

MATH & STATSJesus De-LoeraTim LewisKerry Wong

ENGINEERINGMark Facciotti (bio)Jean VanDer GheynstJP DelaplanqueJames SchaafMike HillDavid SpightJason MooreJennifer Choi

ECONOMICSScott CarrellJanine WilsonDerek StimelLester Lusher

EDUCATIONCindy PassmoreMichal KurlaenderDerek Dockter

AGRICULTUREDiane UlmanSue EbelerDave RizzoLily BynesTracy Grissom

SPECIAL THANKS to:Provost Hexter

+ Ideas and approaches from colleagues in:AAU, AAC&U, APLU, BVA, ALMAP, TEA, TRESTLE

UECarolyn ThomasHelen FrasierBrett McFarlane

§What data and information are most important? Is it available? What additional information would you find valuable?

§What strategies and approaches would you use to facilitate productive conversations within your department about these types of data?

Table Top Exercise WORKSHEET