Statistical Validation of Prerequisites and Corequisites: Approaches from the Field

DAY L E N E M E U SC H K E , D I R E C T O R , I N S T IT U T I O N AL R E S E A RC H , C O L L E G E O F T H E C A N YO N S

J I M F I L L P O T , D E AN , I N S T I T U T I O N AL R E S E A RC H & R E S O U RC E D E V E LO P M E N T , C H A F F E Y C O L L E G E

K E I T H W U RT Z , D E A N , I N S T I T U T I O N AL E F F E C T IV E N E S S , R E S E ARC H & P L AN N I N G , C RA FT O N H I L L S C O L L E GE

AN N E DA N E N B E R G , R E SE A RC H A N ALY S T P L AN N IN G , R E SE A RC H A N D I N S T I T U T I O N A L E F F E C T I V E N E S S O F F I C E , S AC RAM E N T O C I T Y C O L L E GE

RP Conference April 11, 2014

Session Objectives Briefly review the Suggestions for CCC institutional Researchers Conducting Prerequisite Research

Review approaches for locally validating prerequisites pre-implementation and post-implementation

◦ Research methodologies◦ Data sources◦ Statistical techniques

Provide strategies for communicating prerequisite findings to internal audiences and facilitating evidence-based decision making

Overview of the Prerequisite Validation Guidelines

Intended to help researchers ◦ Execute the statistical analyses◦ Support faculty

Developed by the RP Group with input from the field

Input provided by various groups: matriculation, faculty, researchers

Incorporated information from Academic Senate and Chancellor’s office guidelines

The Prerequisite Research Framework

Post-Implementation

Pre-Implementation Content Analysis

The “Warm Up” (a.k.a. Preliminary Work)

Content Review/Validity (Pre-Implementation)◦ Develop a systematic but manageable approach◦ Researchers can help facilitate, if needed◦ Identify what is acceptable level of success◦ Ensure faculty have sufficient experience with the pre-requisite and target courses◦ Consider using a rating template

Content Review: Sample Rating Template

Skill: Ability to…Rater

Mean Rating#1 #2 #3 #4 #5

Skill 1: Solve radical, quadratic equations. 4 3 4 4 4 3.8

Skill 2: Solve logarithmic equations.

4 4 5 4 5 4.4

Skill 3: Solve exponential equations. 3 4 4 5 4 4.0

Skill 4: Solve a variety of problems by applying the definitions, postulates and theorems of plane geometry.

5 4 4 3 5 4.2

Skill 5: Graph linear, quadratic, simple polynomial, exponential logarithmic functions and conic sections.

5 4 5 4 4 4.4

Number of skills with a mean rating of ≥ 4.0: 4

Percent of skills with a mean rating of ≥ 4.0: 80%

The “Warm Up” (a.k.a. Preliminary Work)

Preliminary Analyses (Pre-Implementation)◦ Is the pre-requisite reasonably likely to improve student success?◦ What enrollment/access issues might arise?◦ What impact might this have on other programs?

Information researchers need from faculty◦ Changes in curriculum◦ Inclusion/exclusion of different delivery modes◦ Outcome measures

Pre-Implementation Statistical Analysis

Initiating a Statistical Validation Request

Local Discussion Among:◦ Discipline Faculty◦ Faculty from Prerequisite Discipline◦ Student Services (Matriculation) Personnel◦ Curriculum Committee◦ Administration

•Knowing What the Process Entails

•Institutionally Approved Requestor/Process◦ Curriculum Committee◦ Approved Research Request Form with Appropriate Sign-Off

Some Useful Data Elements for Statistical Validation of Prerequisites

MIS Data Elements◦ GI03 (Term Identifier)◦ CB01 (Course-Department-Number)◦ SX01 (Enrollment-Effective-Date)◦ SX04 (Enrollment-Grade)◦ SB, SD, STD Data Elements (for Disproportionate Impact)

Assessment Data◦ Placement Recommendation (Communication and

Computational Skills Courses)

Sample Statistical Options for Researchers

Test Questions Answered with Prerequisite Validation Approach

T-Test Is their a statistically significant difference between the average GPA/success rate in the prerequisite and target courses? The observed difference may not be substantial.

Chi-Square Is there a statistically significant difference between GPA in the target course and completion of the prerequisite course? The observed difference may not be substantial.

Pearson Correlation

Is there a significant relationship between GPA in the prerequisite and target courses?

Sample Statistical Options for Researchers

Test Questions Answered with Prerequisite Validation Approach

Odds Ratio How likely is it that students who meet the prerequisite will succeed in the target course compared to those who do not meet the prerequisite?

Effect Size What is the strength of the relationship between successfully completing the prerequisite course and successfully completing the target course?

Average Percent Gain

What is the average percent gain in success in the target course of students who met the prerequisite over those who did not meet the prerequisite?

Chaffey CollegePrerequisite Validation Approach

Three-Pronged Approach:1. Comparison of Performance in the Target Course of Students Who Did

and Did Not Meet the Prerequisite

2. Effect Size (accounts for influence of sample size) and Average Percent Gain

3. Restricted Bivariate Correlation Coefficient and Corrections for Restriction of Range◦ Pearson’s r (Rule of Thumb: r ≥ .35, assuming p < .05)◦ Chaffey also recalculates to correct for restriction of range

Chaffey CollegePrerequisite Data Table

Examination of Disproportionate Impact Prior to Prerequisite Enforcement Disproportionate impact occurs when “the percentage of persons from a particular racial, ethnic, gender, age or disability group who are directed to a particular service or placement based on an assessment instrument, method, or procedure is significantly different from the representation of that group in the population of persons being assessed, and that discrepancy is not justified by empirical evidence demonstrating that the assessment instrument, method or procedure is a valid and reliable predictor of performance in the relevant educational setting.” [Title 5 Section 55502(d)]

When there is a disproportionate impact on any such group of students, the district shall, in consultation with the Chancellor, develop and implement a plan setting forth the steps the district will take to correct the disproportionate impact.” [Title 5 Section 55512(a)]

Disproportionate Impact Guide (CCCCO and RP Group, 2013)

Examination of Disproportionate Impact Prior to Prerequisite Enforcement Classification and Regression Trees (CART)

◦ divide a population into segments that differ with respect to a designated criterion.

◦ identifies the best predictor variable, conducting a splitting algorithm that further identifies additional statistically significant predictor variables and splits these variables into smaller subgroups

◦ ensures that cases in the same segment are homogeneous with respect to the segmentation criterion, while cases in different segments tend to be heterogeneous with respect to the segmentation criterion

Classification and Regression Tree (CART)Example Tree

Node 0Category % n

34.3 172Did Not Meet Prerequisite65.7 329Met Prerequisite

Total 100 .0 501

Age RangeImprovement=0.022

Disproportionate Impact: COMSTD-72, ENGL-1A prereq

Node 1Category % n

54.8 57Did Not Meet Prerequisite45.2 47Met Prerequisite

Total 20.8 104

40 to 49; 25 to 29; 30 to 34; 50 or Older

Node 2Category % n

29.0 115Did Not Meet Prerequisite71.0 282Met Prerequisite

Total 79.2 397

35 to 39; 19 or Younger; 20 to 24

Did Not Meet PrerequisiteMet Prerequisite

Chaffey CollegeDecision to Implement a Prerequisite

Green – Sufficient evidence exists to enforce prerequisite (at least two out of three measures are supported)

Yellow – Although evidence exists, only one out of three measures supports enforcement of the prerequisite. Further discussion should occur within the department and the Curriculum Committee before the prerequisite is enforced

Red – Data does not exist to support enforcement of the prerequisite. None of the measures explored meet pre-established criteria

Insufficient Data – While evidence may point to the efficacy of the prerequisite, the sample size is too small to render a reliable decision

Post-Implementation Statistical Analysis

Post-Implementation Statistical Analysis: Research Questions

The following questions were examined to determine the impact of implementing READ-078 as a prerequisite for EMS-020:

◦ Did the EMS-020 course success rate increase after the READ-078 prerequisite was implemented?

◦ What is the racial/age/gender/disability makeup of the course post implementation compared to pre implementation?

◦ Does the increased success of students in each protected category support the implementation if indeed the percentages of students in each group have changed?

◦ Was there disproportionate impact?◦ What effect did the implementation have on overall course enrollment?

Post-Implementation Statistical Analysis: Methodology

An effect size statistic was used to indicate the size of the difference between course success for students who met and did not meet the prerequisite

At the time of the study the prerequisite had been enforced from Spring 2011 to Spring 2013 (i.e. 5 primary terms)

The performance of students who had to meet the prerequisite prior to taking EMS-020 was compared to students who earned a GOR in EMS-020 from Fall 2008 to Fall 2010

Post-Implementation Statistical Analysis: Findings

1. Did the EMS-020 course success rate increase after the READ-078 prerequisite was implemented?

a. Yes, students who met the reading prerequisite were statistically significantly (p < .001) and substantially (ES = .21) more likely to successfully complete EMS-020 (62%) than students who had not completed the prerequisite (51%).

Course

Success Rate Pre-

ImplementationPost-

Implementation

# N % # N % ES P Value

EMS-020 440 857 51.3 349 565 61.8 .21 < .001

Post-Implementation Statistical Analysis: Findings

2. What is the racial/age/gender/disability makeup of the course post implementation compared to pre implementation?

a. Gender, ethnicity, age, and disability status were not substantially different prior to or after the implementation of the READ-078 prerequisite.

GenderPre-Implementation Post-Implementation Total

# N % # N %Female 142 16.6 103 18.2 245 17.2Male 710 82.8 462 81.8 1,172 82.4Unknown 5 0.6 0 0.0 5 0.4Total 857 100.0 565 100.0 1,422 100.0

Post-Implementation Statistical Analysis: Findings

To identify disproportionate impact can use the 80% rule or segmentation modeling (see page 17 in Wurtz & Riggs, 2010 for an example of segmentation modeling).

80% Rule

.80 * .828 (proportion of males pre-implementation) = .6624 or 66.2%. Is the proportion of males less than 66.2% post-implementation?

.80 * .166 (proportion of females pre-implementation) = .1328 or 13.3%. Is the proportion of females less than 13.3% post-implementation?

GenderPre-Implementation Post-Implementation Total

# N % # N %Female 142 16.6 103 18.2 245 17.2Male 710 82.8 462 81.8 1,172 82.4Unknown 5 0.6 0 0.0 5 0.4Total 857 100.0 565 100.0 1,422 100.0

Post-Implementation Statistical Analysis: Findings

3. Does the increased success of students in each protected category support the implementation, if indeed the percentages of students in each group have changed?

a. Yes, male students, Hispanic Students, and students 24 years old or younger were substantially (ES >= .20) and statistically significantly (p < .01) more likely to successfully complete EMS-020 if they had met the reading prerequisite than students who had not met the prerequisite.

b. In addition, female, African American, and Native American students were slightly more likely to successfully complete EMS-020 post-implementation.

EthnicitySuccess Rate Pre-Implementation Post-Implementation

# N % # N % ES P ValueAsian 11 22 50.0 11 24 45.8 -.08 .783African American 30 57 52.6 14 26 53.8 .02 .919Hispanic 116 304 38.2 121 211 57.3 .39 < .001Native American 14 24 58.3 9 14 64.3 .12 .726Caucasian 258 429 60.1 191 286 66.8 .14 .070Unknown 11 21 52.4 3 4 75.0 .44 .425Total 440 857 51.3 349 565 61.8 .21 < .001

Post-Implementation Statistical Analysis: Findings

4. Was there disproportionate impact?◦ Used Classification and Regression Tree (CART) modeling to analyze

disproportionate impact by gender, ethnicity, age, and disability status

Post-Implementation Statistical Analysis: Findings

5. What effect did the implementation have on overall course enrollment?◦ The overall course enrollment in EMS-020 did not decrease as a result of the

implementation of the prerequisite◦ The decline in enrollments and section offerings was due to the statewide budget

cuts and comparable to the cuts that occurred college wide.

Pre-Implementation Post-Implementation

Section Fall 2008

Spring 2009

Fall 2009

Spring 2010

Fall 2010

Spring 2011

Fall 2011

Spring 2012

Fall 2012

Spring 2013

1 40 40 40 40 42 39 39 43 39 422 40 39 42 30 43 38 37 39 37 423 39 45 37 41 36 43 40 35 29 234 40 36 41 41 0 0 0 0 0 05 37 35 0 0 0 0 0 0 0 06 33 0 0 0 0 0 0 0 0 0Total 229 195 160 152 121 120 116 117 105 107

Sacramento City College Example

History Department, 2011—2014

17 courses

40-50 sections per semester offered pre-implementation, with 2,300 to 2,700 Census enrollments from 2009-2011

Studied largest-enrolled (U.S. HIST) courses based on Fall 2009 HIST enrollments and prior English preparation. A department-wide pre-req. of English 1-level-below-transfer was implemented in Fall 2012, based on a number of factors, including:Same SLOs for all SCC HIST coursesSame minimum writing word count requirement for all SCC HIST courses

Sacramento City College Example (continued…)

Enrollment impact estimation=16% (based on 2009 data)

Able to calibrate estimates against actuals to inform planning for other departments. How close were we? From Fall 2011 to Fall 2012 = 16.37%

No measurable change is success rates

How does enrollment composition look?

SCC Example (continued…) Now that prerequisite has been in placefor 3 semesters…

86.183.0

99.8

89.9

0.0

20.0

40.0

60.0

80.0

100.0

120.0

Percentage

SCC History Department Fall and Spring Fill Rate (Fall 2004 to Fall 2013)

Fall 2012: Pre-requisiteimplementation

SCC Example (continued…) Ethnicity before and after prerequisite

0

5

10

15

20

25

30

35

Fall 2009 Spring 2010 Fall 2010 Spring 2011 Fall 2011 Spring 2012 Fall 2012

African American

Asian

Filipino

Latino

Multi-

Native American

Other, non-wht

Pacific Islander

Unknown

White

Communicating Results and Evidence-Based Decision Making

Tips for CommunicatingPrerequisite Results

Affirm your common interest in supporting and enhancing student success

Affirm that you are there to support and assist faculty

Affirm that the research is not meant to substitute for faculty’s professional judgment (i.e. evidence-based decision making)

Making data accessible (i.e. do not use big statistical term, know your audience)

Recognize and communicate the limitations of the data/research in the beginning

Consider different approaches for how data is presented

Questions

Resources: College Examples

Cabrillo College◦ Cabrillo College. (2002). Validation of English 1A as a prerequisite for Psychology 1A. Aptos, CA: Borden, R. C.

Chaffey College◦ Chaffey College. (2011). Prerequisite Validation Studies: Impact of a Reading Prerequisite on HIST-1, HIST-2, and HIST-7.

Rancho Cucamonga, CA: Institutional Research◦ Chaffey College. (2010). Philosophy 76 Prerequisite Validation Study: English 1A Prerequisite. Rancho Cucamonga, CA:

Institutional Research.

Sacramento City College◦ Danenberg, A. (2011). Methodological and Data Considerations for a Communication or Computation Prerequisite

Implementation Study. Sacramento City College: Planning, Research, & Institutional Effectiveness.

Crafton Hills◦ Wurtz, K. A. (2014). Relationship of the EMS-020 Reading Prerequisite to EMS-020 Course Success. Retrieved March 25, 2014

from http://www.craftonhills.edu/~/media/Files/SBCCD/CHC/About%20CHC/Research%20and%20Planning/Research%20Briefs/Academic%20Success%20Studies/2013_July_EMS20_PrereqEval_Post2.pdf

◦ Wurtz, K. A., & Riggs, M. (2010). Prerequisite Validation Study: Examination of Reading as a Prerequisite to EMS-020 (Emergency Medication Technician-I / EMT – Basic). Retrieved November 16, 2012 from http://www.craftonhills.edu/~/media/Files/SBCCD/CHC/About%20CHC/Research%20and%20Planning/Research%20Reports/0910_EMS_Read_PrerequisiteStudy.ashx

RP Group for California Community Colleges◦ RP Group. (2013). SUGGESTIONS FOR CALIFORNIA COMMUNITY COLLEGE INSTITUTIONAL RESEARCHERS CONDUCTING

PREREQUISITE RESEARCH. Retrieved March 25, 2014 from http://www.rpgroup.org/sites/default/files/RPGroupPreqreqGuidelinesFNL.pdf