opportunities and challenges in a multi-site regression discontinuity design stephen w. raudenbush...
TRANSCRIPT
Opportunities and Challenges in a Multi-Site Regression Discontinuity Design
Stephen W. RaudenbushUniversity of Chicago
Presentation at the MultiLevel Theory and Research Conference
The Pennsylvania State University University Park, PA, May 17, 2015
The research reported here was supported by a grant from the WT Grant Foundation entitled “Learning from Variation In Program Effects: Methods, Tools, and Insights from Multi-site Trials.”
Outline
Counter-factual account of causation
The “drug-trial paradigm” for causal inference
An alternative paradigm for social interventionsHeterogeneous agentsSocial interactions among participants
Curriular reform in chicago
Conventional RDDIncorporating Agents and Social InteractionsIdentification: School-specific IV
Conclusions
Counter-factual account of Causation
In statistics (Neyman, Rubin, Rosenbaum)
In economics (Haavelmo, Roy, Heckman)
Drug trial paradigm for causation
Y(1): Outcome if the patient receives Z = 1
(the “new drug”)
Y(0): Outcome if the patient receives Z = 0
(the “standard treatment”)
Y(1) – Y(0): Patient-specific causal effect
E (Y(1) – Y(0)) = : Average causal effect
Stable Unit Treatment Value Assumption (Rubin, 1986)
• Each patient has two potential outcomes• Implies
– Only one “version” of each treatment– No “interference between units”
• Implies the doctor and the other patients have no effect on the potential outcomes
Formally…
)();,...,,( 11211 zYdzzzY n
Failure of SUTVA in Education
• Teachers enact instruction in classrooms– Multiple “versions of the treatment”
• Treatment assignment of one’s peers affects one’s own potential outcomes– EG Grade Retention
– Hong and Raudenbush, Educational Evaluation and Policy Analysis, 2005
– Hong and Raudenbush, Journal of the American Statistical Association, 2006
Group-Randomized Trials
Potential outcome
Thus, each child has only two potential outcomes – if we have “intact classrooms”– if we have “no interference between classrooms”
controltoassignedisjiftY
treatmenttoassignedisjiftY
tzzzY
jj
jj
jnjjjj
);0,...,0,0(
);1,...,1,1(
);,...,,(
1
1
211
Limitations of cluster randomized trial
Mechanisms operate within clusters
* Example: 4Rs
teachers vary in response
classroom interactions spill over
We may have interference between clusters
* Example: community policing
Alternative Paradigm for social interventions
Treatment setting (Hong, 2004):
A unique local environment for each treatment composed of * a set of agents who may implement an intervention and* a set of participants who may receive it
Each participant possesses a single potential outcome within each possible treatment setting
Causal effects are comparisons between these potential outcomes
);,...,,( 21 jnjjij tzzzYj
Example: Community Policing (Verbitsky-Shavitz and Raudenbush, 2012)
• Let Zj=1 if Neighborhood j gets community policing
• Let Zj=0 if not
• Under SUTVA
)0()1( jjj YY
“All or none”
)0,0()1,1(
jjj YY
1
1
1
1
10
0
0
0
0
Do it only in high-crime areas: effect on low-crime areas
)0,0(),0( ''''
jjjj YZY
1, HC
1, HC
0, LC
0, LC
1, HC
0, HC
0, HC
0, LC
0, HC
0, LC
Results
Having community policing was especially good if your surrounding neighbors had it
Not having community policing was especially bad if your neighbors had it
*** So targetting only high crime areas may fail***
Application: Double-dose AlgebraNomi and Raudenbush (2015)
Requires 9th-graders to take Double-dose Algebra if they scored below 50 percentile on 8th-grade math test
12,000 students in 60 Chicago high schools
Double-dose Algebra enrollment rate by math percentile scores (city wide)
Enro
llmen
t Rat
es
ITBS percentile scores
Conventional Mediation Model (T, M,Y model)
Cut off (T)Double-Dose Algebra (M)
Algebra Learning (Y)
• γ = average effect of cutoff on taking Double Dose• δ=average effect of taking Double Dose on Y for compliers (“CACE” effect)• Assume no direct effect of T on Y (exclusion restriction)• β= γδ (“ITT” effect)• So δ= β/ γ
Nomi, T., & Allensworth, E. (2009)
γ δ
Conventional Model is Founded on SUTVA
CACE
BYY
E
MM
1)1)(Pr(|E(
Y"oneffectITT"E(B)
Yoncutofimpact)0()1(
Y[M(0)]-Y[M(1)]
M"oneffectITT")(
DDtakingoncutofimpact)0()1(
Results of Conventional Analysis
• Large average impact of Cut on taking DD (ITT effect on M)
• Modest average impact of Cut on Y (ITT effect on Y)
• Modest CACE (Average Impact of M on compliers)
ITT effect on Y
-1.5
-1-.
50
.5A
lgeb
ra S
core
s
-50 -40 -30 -20 -10 0 10 20 30 40
Math Percentile Scores
observed values
fitted values (Lowess)
But the policy changed classroom composition!!
Classroom average skill levels by math percentile scores
Pre-policy (2001-02 and 2002-03 cohorts)
Post-policy (2003-04 and 2004-05 cohorts)
Implementation varied across schools in---
• Complying with the policy • Inducing classroom segregation
Exclusion Restriction RevisedT-M-C-Y model
Cut off (T) Double-Dose Algebra (M)
Algebra score (Y)
Classroom Peer skill (C)
1
2
21
Identification Problem
We have one equation, two unknowns:
Strategy is school-specific
21
21
/
CACE
ITT
)(
21
21
jjj
jjj
jjjjj
u
A simple two-level model
At level 1
At level 2
jijijijjijjjij TXfTTYY .),().(.
jjjj u 211
-.6
-.4
-.2
0.2
Eff
ects
of
the
cut-
scor
e on
cla
ssro
om p
eer
abil
ity
0 .2 .4 .6 .8 1
Effect of the cut-score on double-dose enrollment
Derivation of assumptions using potential outcomes
CACECACECACE
EBECACE
CMYCMYB
CC
MM
21
21
21
)1|()1|(
)]0(),0([)]1(),1([
)0()1(
)0()1(
Parameter Estimate SE
ITT impact on M 0.72 0.03
ITT impact on C -0.28 0.03
ITT impact on Y 0.07 0.03
CACE of M 0.20 0.05
CACE of C 0.22 0.09
-.6
-.4
-.2
0.2
Eff
ects
of
the
cut-
scor
e on
cla
ssro
om p
eer
abil
ity
0 .2 .4 .6 .8 1
Effect of the cut-score on double-dose enrollment
5. Conclusions on DDThe reform• Increased instructional time• Changed class composition
Median skill kids• Gained a lot if not tracked into low-skill classes• Gained little if they were
Conclusions on Causal Inference
Conventional causal paradigm:* a single potential outcome per participant under each treatment
Alternative paradigm:* a single potential outcome per participant in each treatment setting
RDD as a means-tested program
Potentially large policy implications of causal paradigm