Michal Kurlaender, University of California - Davis
Jacob Jackson, Public Policy Institute of California
California’s Early Assessment Program: Assessing Intended
and Unintended Consequences
1
Policy Analysis for California Education Seminar Series March 13, 2015
• Funding Agencies
– Institute of Education Sciences, U.S. Department of Education
– Bill and Melinda Gates Foundation
– Spencer Foundation
• State Agencies
– California State University Chancellor’s Office
– California Department of Education
– California Community College Chancellor’s Office
• Collaborators
– Jessica Howell, The College Board
– Eric Grodsky, University of Wisconsin
2
Acknowledgements
Presentation Outline
1. Background on the EAP
2. The Intended Impacts of the EAP
3. Unintended consequences and mechanisms of the EAP
4. Future Research on the EAP and College Readiness
3
BACKGROUND ON THE EAP
Section 1
4
High remediation and low graduation rates at CSUs
Percent of students requiring some remediation at CSU system and six-year graduation rates by cohort
Data from CSU Analytic Studies: http://www.asd.calstate.edu/performance/proficiency.shtml
Remediation Need
6 year grad rate
0%
10%
20%
30%
40%
50%
60%
70%
2000 2002 2004 2006 2008 2010
Pe
rc
en
t
First Time Freshman Cohort
5
Remediation need varies by race
Data from CSU Analytic Studies: http://www.asd.calstate.edu/performance/proficiency.shtml
Percent of students Requiring remediation at CSU in 2013
6
65%
55%
51%
49%
44%
37%
36%
21%
43%
0% 20% 40% 60% 80%
African American
Mexican American
Other Latino
Pacific Islander
American Indian
Filipino
Asian
White
Total
College Readiness by High School
0.2
.4.6
.81
Pro
port
ion R
ead
y in E
ng&
Ma
th
2.5 3 3.5 4 4.5HS GPA
All Schools Average
Readiness at Entry by GPA
7
“Readiness” Differs by High School
0.2
.4.6
.81
Pro
port
ion R
ead
y E
ng&
Math
2.5 3 3.5 4 4.5HS GPA
All Schools School A School B
Readiness at Entry by GPA
8
Early Assessment Program Overview
• Goals of EAP:
Provide an early signal to students about their college readiness
California State University collaboration with K-12
Provide 12th grade interventions
• Components of EAP:
11th grade testing (early assessment)
Professional development for teachers
Supplemental preparation for students
9
The EAP sends a college readiness signal to 11th grade students
• Optional 11th grade test in English and Mathematics
Optional 15 questions on 11th grade CST
Additional items developed by CSU faculty
Score based on CST augmented with EAP items
• Potential college readiness signals
Exempt from remedial courses at a CSU
Conditional Exempt from remedial courses at CSU
Not exempt from remedial courses at a CSU
10
Most students are not Ready for College Level Work
2013 EAP Test Results
11
55%
18%
18%
51%
21% 12%
7% 19%
0%
100%
Math English
Ready
Conditional
Not Ready
Did NotParticipate
THE INTENDED IMPACTS OF THE EAP
Section 2
12
Intended Consequences: Research Questions
• How does participation in California’s Early Assessment Program affect a student’s probability of requiring remedial coursework in college?
• Does the effect of the program differ for different students?
13
Intended Consequences: Research Questions
• How does participation in California’s Early Assessment Program affect a student’s probability of requiring remedial coursework in college?
• Does the effect of the program differ for different students?
14
Data
• CSU Chancellor’s Office
– Six cohorts of first-time freshman applicants (2000/01 – 2005/06)
• California Department of Education
– Matched CSU applicants (enrollees) to CST scores, EAP participation, other student and high school characteristics
15
Creating a Counterfactual
Pre-EAP Cohorts Post-EAP Cohorts
Non-participants
Non-participants
Participants
16
Intent to Treat
Pre-EAP Cohorts Post-EAP Cohorts
Non-participants
Non-participants
Participants
17
Post-EAP Cohorts
Non-participants
Non-participants
Participants
Pre-EAP Cohorts
Non-participants
Impact of Participation
18
Pre-EAP Cohorts
Non-participants
Post-EAP Cohorts
Non-participants
Participants
Participants Non-Participants
90% Participation Schools
Impact of participation at high participation schools
19
The implementation of the EAP is related to lower likelihood of requiring remediation
20
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
English Math
Pr
ob
ab
ilit
y o
f N
ee
din
g
Re
me
dia
tio
n
Pre-EAP Post-EAP
Fitted Values for Remediation Rates (ITT)
-.018
-.020
Covariates, High School and College FE, 6 year sample
21
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
English Math
Pr
ob
ab
ilit
y o
f N
ee
din
g
Re
me
dia
tio
n
Pre-EAP Post-EAP
-.022
-.027
Fitted Values for Remediation Rates (For 90% Participation Schools)
Covariates, High School and College FE, 6 year sample
Participation in the EAP is related to lower likelihood of requiring remediation
Intended Consequences: Research Questions
• How does participation in California’s Early Assessment Program affect a student’s probability of requiring remedial coursework in college?
• Does the effect of the program differ for different students?
22
Students with the lowest probability of needing remediation seem to benefit the most
-.0
4-.
02
0
.02
Est
imate
d E
ffe
ct
0 .2 .4 .6 .8 111th grade remediation need
Estimated EAP Effect at Different Levels of Need for English Remediation
23
Students with the lowest probability of needing remediation seem to benefit the most
Estimated EAP Effect at Different Levels of Need for Math Remediation
-.1
-.0
5
0
.05
Estim
ate
d E
ffe
ct
0 .2 .4 .6 .8 111th grade remediation need
24
Summary: Impacts of the EAP Program
25
• The EAP policy led to a modest reduction in the probability that CSU freshmen require remediation.
• EAP participation leads to a modest (1 to 2 percentage point) reduction in the probability that CSU freshmen require remediation.
• The program had larger effects for students who were less likely to need remediation
UNINTENDED CONSEQUENCES AND MECHANISMS OF THE EAP
Section 3
26
Unintended Consequences and Mechanisms: Research Questions
27
1. Does the EAP discourage CSU enrollment? 2. Does the EAP encourage math course-taking in 12th grade? 3. How does the EAP impact high school accountability measures? 4. Has there been an impact of the EAP on community colleges?
Unintended Consequences and Mechanisms: Research Questions
28
1. Does the EAP discourage CSU enrollment? 2. Does the EAP encourage math course-taking in 12th grade? 3. How does the EAP impact high school accountability measures? 4. Has there been an impact of the EAP on community colleges?
Most students are not Ready for College Level Work
2013 EAP Test Results
29
55%
18%
18%
51%
21% 12%
7% 19%
0%
100%
Math English
Ready
Conditional
Not Ready
Did NotParticipate
How did the EAP reduce the need for remediation?
Option 1: Students are motivated by a not-exempt signal to improve in their senior year.
Option 2: Students receiving a not-exempt signal are discouraged from attending a CSU.
30
Testing the impact of the college readiness signal
• Outcomes:
– CSU Application
– CSU Enrollment
• Predictor:
– Readiness signal (exempt status)
31
Example regression discontinuity effect
0.1
.2.3
.4.5
.6
P(A
pply
to C
SU
)
-60 -40 -20 0 20EAP English Score (0=Exempt)
32
Hypothetical Probability of Applying to a CSU by EAP English Score
Hypothetical 8 percentage point
difference
The English readiness signal has a small impact on CSU application
(only among math nonparticipants)
33
0.1
.2.3
.4.5
.6
P(A
pp
ly to
CS
U)
-60 -40 -20 0 20EAP English Score (0=Exempt)
N= 671,861 Effect =.015 ** (.006)
Probability of Applying to a CSU by EAP English Score
The English readiness signal has no impact on CSU enrollment
34
0.1
.2.3
P(a
tte
nd
to C
SU
)
-60 -40 -20 0 20EAP English Score (0=Exempt)
N= 671,861 Effect =.003 (.004)
Probability of Attending a CSU by EAP English Score
Early negative information does not discourage students from enrolling
• The English readiness signal has no impact for EAP Math participants
• For math non-participants:
– There is a small negative impact on application to a CSU
– There is no impact on CSU enrollment
35
Unintended Consequences and Mechanisms: Research Questions
36
1. Does the EAP discourage CSU enrollment? 2. Does the EAP encourage math course-taking in 12th grade? 3. How does the EAP impact high school accountability measures? 4. Has there been an impact of the EAP on community colleges?
Students are no more likely to enroll in advanced math when receiving conditional exempt signal
37
Unintended Consequences and Mechanisms: Research Questions
38
1. Does the EAP discourage CSU enrollment? 2. Does the EAP encourage math course-taking in 12th grade? 3. How does the EAP impact high school accountability measures? 4. Has there been an impact of the EAP on community colleges?
02
04
06
0
# o
f S
cho
ols
0 20 40 60 80 100% Participation in English EAP
Using EAP participation quartiles
Percent of students taking EAP English Exam per school (2005 Cohort)
39
Higher EAP participation is associated with higher levels of CST Proficiency
1.8
0
10
20
30
40
50
Q1 Q2 Q3 Q4
% P
ro
fic
ien
t o
f A
dv
an
ce
d
Pre-EAP Post-EAP
*
Fitted Values for CST Proficiency from Difference in Difference
Pre-Post Differences in School-wide CST Proficiency by Participation Quartiles
40
0
2
4
6
8
10
12
14
16
18
Q1 Q2 Q3 Q4
% A
dv
an
ce
d
Pre-EAP Post-EAP
Higher EAP participation is associated with higher levels of CST Advanced
Fitted Values for CST Advanced from Difference in Difference
1.3
*
*
Pre-Post Differences in School-wide CST Advanced by Participation Quartiles
41
200
300
400
500
600
700
800
Q1 Q2 Q3 Q4
AP
I
Pre-EAP Post-EAP
Higher EAP participation is associated with higher API
Fitted Values for API from Difference in Difference
5.3
*
Pre-Post Differences in School-wide API by Participation Quartiles
42
Conclusions: EAP and college readiness at high schools
• School characteristics are mostly unrelated to EAP participation
• Schools in all levels of participation see gains in test scores, accountability measures, and college-level outcomes
• Higher levels of participation in EAP are associated with higher gains across tests scores and school accountability measures
43
Unintended Consequences and Mechanisms: Research Questions
44
1. Does the EAP discourage CSU enrollment? 2. Does the EAP encourage math course-taking in 12th grade? 3. How does the EAP impact high school accountability measures? 4. Has there been an impact of the EAP on community colleges?
Very Few CCC students are exempt via
the EAP
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
English Math
Pe
rc
en
t o
f a
ll E
nte
rin
g S
tud
en
ts
Percent of CCC Entering Students Scoring Exempt in 2009
Eligible Take EAP Conditional Exempt Exempt
45
EAP Exempt students take more demanding courses
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
CSU Transferable UC Transferable Basic Remedial
Pe
rc
en
t o
f C
las
se
s
Percent of Transferrable Classes for EAP Exempt Students
ALL Students English exempt Math Exempt
46
EAP at Community Colleges
47
40%
45%
50%
55%
60%
65%
70%
75%
80%
1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013
Pe
rc
en
t o
f F
irs
t-T
ime
Stu
de
nts
English Developmental Course Participation
Early Adopters
Late Adopters
No EAP
Summary: Unintended Consequences of the EAP
• The EAP does not dissuade students from applying to or enrolling in a CSU
• The EAP does not impact student 12th grade course-taking
• EAP participation may benefit high school accountability measures
• EAP adoption at community college is associated with lower rates of developmental coursetaking
48
FUTURE RESEARCH ON THE EAP AND COLLEGE READINESS
Section 4
49
Future work on college readiness
• Long term outcomes of EAP
– Graduation
– Time to degree
– Persistence
• College Readiness in the era of Common Core and Smarter Balanced Assessments
• Early Start at California State University
50