opportunities and challenges in a multi-site regression discontinuity design stephen w. raudenbush...

Opportunities and Challenges in a Multi-Site Regression Discontinuity Design 

Stephen W. RaudenbushUniversity of Chicago

Presentation at the MultiLevel Theory and Research Conference

The Pennsylvania State University University Park, PA, May 17, 2015

The research reported here was supported by a grant from the WT Grant Foundation entitled “Learning from Variation In Program Effects: Methods, Tools, and Insights from Multi-site Trials.”

Outline

Counter-factual account of causation

The “drug-trial paradigm” for causal inference

An alternative paradigm for social interventionsHeterogeneous agentsSocial interactions among participants

Curriular reform in chicago

Conventional RDDIncorporating Agents and Social InteractionsIdentification: School-specific IV

Conclusions

Counter-factual account of Causation

In statistics (Neyman, Rubin, Rosenbaum)

In economics (Haavelmo, Roy, Heckman)

Drug trial paradigm for causation

Y(1): Outcome if the patient receives Z = 1

(the “new drug”)

Y(0): Outcome if the patient receives Z = 0

(the “standard treatment”)

Y(1) – Y(0): Patient-specific causal effect

E (Y(1) – Y(0)) = : Average causal effect

Stable Unit Treatment Value Assumption (Rubin, 1986)

• Each patient has two potential outcomes• Implies

– Only one “version” of each treatment– No “interference between units”

• Implies the doctor and the other patients have no effect on the potential outcomes

Formally…

)();,...,,( 11211 zYdzzzY n

Failure of SUTVA in Education

• Teachers enact instruction in classrooms– Multiple “versions of the treatment”

• Treatment assignment of one’s peers affects one’s own potential outcomes– EG Grade Retention

– Hong and Raudenbush, Educational Evaluation and Policy Analysis, 2005

– Hong and Raudenbush, Journal of the American Statistical Association, 2006

Group-Randomized Trials

Potential outcome

Thus, each child has only two potential outcomes – if we have “intact classrooms”– if we have “no interference between classrooms”

controltoassignedisjiftY

treatmenttoassignedisjiftY

tzzzY

jj

jj

jnjjjj

);0,...,0,0(

);1,...,1,1(

);,...,,(

1

1

211

Limitations of cluster randomized trial

Mechanisms operate within clusters

* Example: 4Rs

teachers vary in response

classroom interactions spill over

We may have interference between clusters

* Example: community policing

Alternative Paradigm for social interventions

Treatment setting (Hong, 2004):

A unique local environment for each treatment composed of * a set of agents who may implement an intervention and* a set of participants who may receive it

Each participant possesses a single potential outcome within each possible treatment setting

Causal effects are comparisons between these potential outcomes

);,...,,( 21 jnjjij tzzzYj

Example: Community Policing (Verbitsky-Shavitz and Raudenbush, 2012)

• Let Zj=1 if Neighborhood j gets community policing

• Let Zj=0 if not

• Under SUTVA

)0()1( jjj YY

“All or none”

)0,0()1,1(

jjj YY

1

1

1

1

10

0

0

0

0

Do it only in high-crime areas: effect on low-crime areas

)0,0(),0( ''''

jjjj YZY

1, HC

1, HC

0, LC

0, LC

1, HC

0, HC

0, HC

0, LC

0, HC

0, LC

Results

Having community policing was especially good if your surrounding neighbors had it

Not having community policing was especially bad if your neighbors had it

*** So targetting only high crime areas may fail***

Application: Double-dose AlgebraNomi and Raudenbush (2015)

Requires 9th-graders to take Double-dose Algebra if they scored below 50 percentile on 8th-grade math test

12,000 students in 60 Chicago high schools

Double-dose Algebra enrollment rate by math percentile scores (city wide)

Enro

llmen

t Rat

es

ITBS percentile scores

Conventional Mediation Model (T, M,Y model)

Cut off (T)Double-Dose Algebra (M)

Algebra Learning (Y)

• γ = average effect of cutoff on taking Double Dose• δ=average effect of taking Double Dose on Y for compliers (“CACE” effect)• Assume no direct effect of T on Y (exclusion restriction)• β= γδ (“ITT” effect)• So δ= β/ γ

Nomi, T., & Allensworth, E. (2009)

γ δ

Conventional Model is Founded on SUTVA

CACE

BYY

E

MM

1)1)(Pr(|E(

Y"oneffectITT"E(B)

Yoncutofimpact)0()1(

Y[M(0)]-Y[M(1)]

M"oneffectITT")(

DDtakingoncutofimpact)0()1(

Results of Conventional Analysis

• Large average impact of Cut on taking DD (ITT effect on M)

• Modest average impact of Cut on Y (ITT effect on Y)

• Modest CACE (Average Impact of M on compliers)

ITT effect on Y

-1.5

-1-.

50

.5A

lgeb

ra S

core

s

-50 -40 -30 -20 -10 0 10 20 30 40

Math Percentile Scores

observed values

fitted values (Lowess)

But the policy changed classroom composition!!

Classroom average skill levels by math percentile scores

Pre-policy (2001-02 and 2002-03 cohorts)

Post-policy (2003-04 and 2004-05 cohorts)

Implementation varied across schools in---

• Complying with the policy • Inducing classroom segregation

Exclusion Restriction RevisedT-M-C-Y model

Cut off (T) Double-Dose Algebra (M)

Algebra score (Y)

Classroom Peer skill (C)

1

2

21

Identification Problem

We have one equation, two unknowns:

Strategy is school-specific

21

21

/

CACE

ITT

)(

21

21

jjj

jjj

jjjjj

u

A simple two-level model

At level 1

At level 2

jijijijjijjjij TXfTTYY .),().(.

jjjj u 211

-.6

-.4

-.2

0.2

Eff

ects

of

the

cut-

scor

e on

cla

ssro

om p

eer

abil

ity

0 .2 .4 .6 .8 1

Effect of the cut-score on double-dose enrollment

Derivation of assumptions using potential outcomes

CACECACECACE

EBECACE

CMYCMYB

CC

MM

21

21

21

)1|()1|(

)]0(),0([)]1(),1([

)0()1(

)0()1(

Parameter Estimate SE

ITT impact on M 0.72 0.03

ITT impact on C -0.28 0.03

ITT impact on Y 0.07 0.03

CACE of M 0.20 0.05

CACE of C 0.22 0.09

-.6

-.4

-.2

0.2

Eff

ects

of

the

cut-

scor

e on

cla

ssro

om p

eer

abil

ity

0 .2 .4 .6 .8 1

Effect of the cut-score on double-dose enrollment

5. Conclusions on DDThe reform• Increased instructional time• Changed class composition

Median skill kids• Gained a lot if not tracked into low-skill classes• Gained little if they were

Conclusions on Causal Inference

Conventional causal paradigm:* a single potential outcome per participant under each treatment

Alternative paradigm:* a single potential outcome per participant in each treatment setting

RDD as a means-tested program

Potentially large policy implications of causal paradigm