we would like to thank the institute for education sciences for their support and funding
DESCRIPTION
Principal Investigators: Brian Jacob & Susan Dynarski University of Michigan Barbara Schneider & Ken Frank Michigan State University Thomas Howell Center for Educational Performance & Information Joseph Martineau Michigan Department of Education. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/1.jpg)
We would like to thank the Institute for Education Sciences for their support and funding. Please direct any questions to [email protected]
735 S. State Street | Ann Arbor, MI 48109
Principal Investigators:
Brian Jacob & Susan DynarskiUniversity of Michigan
Barbara Schneider & Ken FrankMichigan State University
Thomas Howell Center for Educational Performance & Information
Joseph MartineauMichigan Department of Education
![Page 2: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/2.jpg)
What is MCER?
Michigan Consortium for Educational Research
IES-funded collaboration between: University of Michigan Michigan State University Michigan Department of Education Michigan Center for Educational Performance
and Information.
![Page 3: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/3.jpg)
Objectives of MCER
Engage stakeholders and education experts in research for the benefit of public education in Michigan
Provide research-based evidence to policymakers in Michigan
Inform national policy initiatives for improving education
![Page 4: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/4.jpg)
Inaugural Research Questions
What is the effect of the Michigan Merit Curriculum on… course-taking patterns, student achievement high school graduation, postsecondary attendance
What was the effect of the Michigan Promise Scholarship on… college entry, college choice, college completion
Do these effects vary by school or student characteristics?
![Page 5: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/5.jpg)
MCER Data Goal
Detailed Student-Level Longitudinal Dataset
Attainment:
HS graduation,
GED, college degrees
K12:
test scores, enrollment,
demographics, GED, ACT
scores
Postsecondary: NSC, MI public
college transcripts
![Page 6: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/6.jpg)
Michigan Merit Curriculum Study As of 2011, all graduating MI HS students must pass
16 rigorous courses (e.g., Algebra II, Biology, Chemistry, Physics) and complete end-of-course exams to measure content mastery
The evaluation will compare student outcomes before & after the MMC
Standardized test scores, HS graduation College entry, choice & completion
Random sample of 100+ schools: assess fidelity of implementation
End-of-course exams Student transcripts Interviews
![Page 7: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/7.jpg)
Michigan Promise Study Intervention
$4,000 college scholarship for students with qualifying score on Michigan Merit Exam
Students must maintain 2.5 college GPA
Methods Regression discontinuity: compare outcomes
just above/below MME threshold College enrollment/choice/persistence (NSC) Scholarship receipt (Treasury)
![Page 8: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/8.jpg)
Emerging Research Questions
• Has mandatory ACT (now part of MME) improved college attendance and choice?
• What is the “value-added” of individual schools, once we control for student characteristics and initial achievement?
• Are Michigan’s charter schools raising student achievement?
![Page 9: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/9.jpg)
Example studyThe Impact of the Threat of School Sanctions: A
Regression Discontinuity Study of Being on a Probationary List
Guan K. Saw, I-Chien Chen, Barbara L. Schneider, Kenneth A. Frank
Michigan State University
![Page 10: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/10.jpg)
• This study analyzes the effect of the first stage of school sanctions in Michigan, being on a probationary list. We pay attention to its possible impact on student achievement in high-stakes and low-stakes subjects at school level.
Purpose of Study
![Page 11: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/11.jpg)
Background
• School sanctions, increasingly used as instruments of education policy, have been the focus of debate at federal and state levels.
• The goal of sanctions is to incentivize schools that fail to meet academic standards to improve their students’ educational performance.
• How does it work to make school change?
![Page 12: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/12.jpg)
Two possible explanations1. Probations serve as a Social Stigma
– Stigmatizing or labeling is a potent tool for guiding individuals to conform to social norms.
– For failing schools, stigmatization becomes a motivating factor to make change (Figlio & Rouse, 2006; Ladd & Glennie, 2001; Mintrop, 2004; Sim, 2007, 2009).
– “Being on a probationary list” can be seen as a social stigma, which may have a “labeling effect”.
![Page 13: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/13.jpg)
2. Effects of Sanction Threats– Not only imposing sanctions, but also
threatening sanctions can change an individual’s behavior (Lacy & Niou, 2004).
– In economic sanctions literature, some argue that sanctions threatened are often more effective than those that are deployed (Drezner, 1999; Drury & Li, 2006; Lacy & Emerson, 2004; Smith, 1996).
– In education, the effects of sanction threats on low-performing schools are mixed (Chiang, 2009; Figlio & Rouse, 2006; Springer, 2008; Winters et al., 2010).
![Page 14: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/14.jpg)
Crowding-out hypothesis– There is a growing concern that test-based
accountability may cause schools to shift inputs from low-stakes subjects (Corbett & Wilson, 1991; Kohn, 1999; Nichols & Berliner, 2007; Whitford & Jones, 2000).
– This crowding-out hypothesis was supported by some qualitative evidence (Au, 2007; Groves, 2002; King & Mathers, 1997; Murillo & Flores, 2002).
– In contrast, some quantitative studies report that high-stakes testing policies led to significant gains in low-stakes subjects (Jacob, 2005; Winters, Trivitt, & Greene, 2010).
![Page 15: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/15.jpg)
Michigan Context: PLA and Watch List
• Since 2009, Michigan Department of Education (MDE) has annually published a list of the lowest performing 5% schools, named the Persistently Lowest Achieving list (PLA list).
• The PLA list is established by certain criteria: (a) 2-year average percent proficiency in math and reading;(b) 4-year slope of percent proficiency in math and reading; (c) whether a school made Adequate Yearly Progress (AYP) status over the past two years; and
(d) whether a school had a 4-year graduation rate below 60% for three years in a row.
![Page 16: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/16.jpg)
• The PLA list schools have to make significant gains in student achievement within a short time to get off the list.
• If not, further sanctions may be imposed including turnaround, restart, and closure of schools.
• With labeling and sanction threat effects, we hypothesize that the PLA list schools tend to positively affect student achievement.
![Page 17: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/17.jpg)
• In addition to the PLA list (bottom 5%), MDE also publishes a “Watch list” of schools in the lowest quintile (6-20%), which were identified as being in danger of falling under the 5% mark.
• This does not affect the PLA ranking but provides an alert to these schools to keep them out of the PLA category.
• Without a strong threat of further sanctions, we expect that the labeling effect of being on the watch list is relatively limited.
Michigan Context: Watch List
![Page 18: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/18.jpg)
Data Sources• Our longitudinal school-level data
constructed with:(1) Michigan Educational Assessment System (MEAS);(2) Common Core of Data (CCD).
• We only focus on a sample of regular high schools (only 333 schools ranked by state-wide achievement base percentile ranking).
![Page 19: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/19.jpg)
Measures• Treatment 1: Being on 2008-09 PLA list
(<5%)Treatment 2: Being on 2008-09 Watch list (6-20%)
• Forcing variable: State-wide achievement base percentile ranking
• Outcomes: % of students met proficiency level in (a) high-stakes subjects: math, reading, writing,(b) low-stakes subjects: science, social studies
• Covariates: % of free/reduced lunch students, % of black students, school size, and pupil teacher ratio.
![Page 20: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/20.jpg)
Analytic Method• We employ a regression discontinuity
design (RDD) method. This is a quasi-experimental design, in which treatment status depends on whether an observed covariate exceeds a fixed threshold(Lee & Card, 2008; Shadish, Campbell, & Cook, 2002).
• In our case, the fixed threshold is the cutoff (5% or 20%) of percentile ranking for being on the PLA or watch lists.
![Page 21: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/21.jpg)
• RD Analysis of PLA List
• RD Analysis of Watch List
0 5
0 205 35
10 100
100
PLA list
Watch list
Percentile ranking
Percentile ranking
![Page 22: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/22.jpg)
![Page 23: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/23.jpg)
RDD with covariates
• Given the imbalance of covariates between treatment and control groups, we include the covariates in the RD models, which can (Imbens & Lemieux, 2008): (1) reduce small sample bias;(2) improve precision if covariates correlated with potential outcomes (as in analyses of
randomized experiments)
![Page 24: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/24.jpg)
Findings: PLA List0
2040
6080
% o
f stu
dent
s m
et m
ath
prof
icie
ncy
leve
l in
2010
-201
1
0 5 10Tier 2 percentile rank in 2008-09
Figure 1a. Percentage of Students Met Proficiency Level in 2011, by Tier 2 Percentile Rank in 2008-09
020
4060
80%
of s
tude
nts
met
read
ing
prof
icie
ncy
leve
l in
2010
-201
1
0 5 10Tier 2 percentile rank in 2008-09
020
4060
80%
of s
tude
nts
met
writ
ing
prof
icie
ncy
leve
l in
2010
-201
1
0 5 10Tier 2 percentile rank in 2008-09
020
4060
80%
of s
tude
nts
met
sci
ence
pro
ficie
ncy
leve
l in
2010
-201
1
0 5 10Tier 2 percentile rank in 2008-09
020
4060
80%
of s
tude
nts
met
soc
ial s
tudi
es p
rofic
ienc
y le
vel i
n 20
10-2
011
0 5 10Tier 2 percentile rank in 2008-09
Mathematics Reading Writing
Science Social Studies Cutoff = 5%<5% (PLA list) = 19 schools
5-10% (Control group) = 19 schools
![Page 25: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/25.jpg)
Effects of Being on the PLA List (2008-2009)
![Page 26: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/26.jpg)
• RDD analyses show a positive “list” effect on all subjects 2010-2011.
• In models with presence of covariates, only the positive “list” effect on writing achieves statistical significance.
• No negative effect on low-stakes subjects of science and social studies was observed.
![Page 27: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/27.jpg)
020
4060
80%
of s
tude
nts
met
sci
ence
pro
ficie
ncy
leve
l in
2010
-201
1
0 10 20 30 40Tier 2 percentile rank in 2008-09
Findings: Watch List
Figure 1b. Percentage of Students Met Proficiency Level in 2011, by Tier 2 Percentile Rank in 2008-09
Cutoff = 20%5-20% (Watch list) = 54 schools
20-35% (Control group) = 51 schools
020
4060
80%
of s
tude
nts
met
mat
h pr
ofic
ienc
y le
vel i
n 20
10-2
011
0 10 20 30 40Tier 2 percentile rank in 2008-09
020
4060
80%
of s
tude
nts
met
read
ing
prof
icie
ncy
leve
l in
2010
-201
1
0 10 20 30 40Tier 2 percentile rank in 2008-09
020
4060
80%
of s
tude
nts
met
writ
ing
prof
icie
ncy
leve
l in
2010
-201
1
0 10 20 30 40Tier 2 percentile rank in 2008-09
020
4060
80%
of s
tude
nts
met
soc
ial s
tudi
es p
rofic
ienc
y le
vel i
n 20
10-2
011
0 10 20 30 40Tier 2 percentile rank in 2008-09
Mathematics Reading Writing
Science Social Studies
![Page 28: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/28.jpg)
Effects of Being on the Watch List (2008-2009)
![Page 29: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/29.jpg)
• We found no effect of being on the watch list for all subjects in 2010-2011.
• This finding holds across estimation models using different bandwidths (5%, 10%, and 15% below and above cutoff).
![Page 30: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/30.jpg)
Robustness Test• We created a pseudo PLA list using the
previous school year data (2007-08), before the state policy mandating assignment of low-performing schools to a probationary list had been enacted.
• Then, we compare the results of 2008-09 PLA list effects to 2007-08 pseudo PLA list effects.
• We expect that there would be no effect of being on the pseudo PLA list since these schools did not receive a labeling treatment or real threat of sanctions.
![Page 31: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/31.jpg)
Findings: Pseudo PLA List
Figure 1c. Percentage of Students Met Proficiency Level in 2010, by Pseudo Tier 2 Percentile Rank in 2007-08
Cutoff = 5%<5% (Pseudo PLA list) = 18 schools
5-10% (Control group) = 17 schools
020
4060
80%
of s
tude
nts
met
mat
h pr
ofic
ienc
y le
vel i
n 20
09-2
010
0 5 10Pseudo tier 2 percentile rank in 2007-08
020
4060
80%
of s
tude
nts
met
read
ing
prof
icie
ncy
leve
l in
2009
-201
00 5 10
Pseudo tier 2 percentile rank in 2007-08
020
4060
80%
of s
tude
nts
met
writ
ing
prof
icie
ncy
leve
l in
2009
-201
0
0 5 10Pseudo tier 2 percentile rank in 2007-08
Mathematics Reading Writing
![Page 32: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/32.jpg)
Effects of Being on the Pseudo PLA List (2007-08)
![Page 33: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/33.jpg)
• Results from the 2007-08 pseudo list analysis indicate no “list” effect on all subjects in 2009-2010.
• This finding further testifies the robustness of the effects of 2008-2009 PLA list.
![Page 34: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/34.jpg)
Conclusion• Being on PLA list as a social stigma
combined with possible following incremental sanctions may spur school improvement.
• Without a real and strong threat of further sanctions, just being labeled by the watch list does not stimulate school performance.
• Crowding-out hypothesis was not supported by our data.
![Page 35: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/35.jpg)
Limitation and Discussion• Limitations:
(1) given three years of data, we only can claim a short term effect of PLA list.(2) the limited variables in the data set provide no information to uncover real changes being implemented in the schools.
• Puzzles:(1) Are the PLA effects stronger for writing? Why? (2) What organizational processes occurred in the
schools placed on the PLA list?
![Page 36: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/36.jpg)
…take home message…
Within the school sanctioning context:• The labeling effect is limited. • The threat of sanctions does motivate low-
ranked probationary schools to make changes.
• Crowding-out effects may not occur but spillover effects may be present.
![Page 37: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/37.jpg)
“On the List” Example of Evaluation of Institution
• State implemented• Involves state data as well as school
composition (common core)• Apply to Michigan Merit Curriculum
– Graduate students– Data– Collaborative faculty
![Page 38: We would like to thank the Institute for Education Sciences for their support and funding](https://reader036.vdocuments.site/reader036/viewer/2022070423/56816703550346895ddb69a3/html5/thumbnails/38.jpg)
We would like to thank the Institute for Education Sciences for their support and funding. Please direct any questions to [email protected]
735 S. State Street | Ann Arbor, MI 48109
Principal Investigators:
Brian Jacob & Susan DynarskiUniversity of Michigan
Barbara Schneider & Ken FrankMichigan State University
Thomas Howell Center for Educational Performance & Information
Joseph MartineauMichigan Department of Education