studying the mean and variation in the effect of program participation in multi-site trials the...
TRANSCRIPT
![Page 1: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/1.jpg)
Studying the Mean and Variation in the Effect of Program Participation
in Multi-site Trials
The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “Building Capacity for Evaluating Group-Level Interventions.” Thanks to Takako Nomi for her collaboration on these ideas.
![Page 2: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/2.jpg)
Outline1. Pervasiveness of
• Multi-site trials• Non-compliance
2. Potential outcomes framework • Observed data• Potential outcomes• Causal effects
3. Instrumental variables in a single-site study
• Under homogeneity of impact• Under heterogeneity of impact• Complier-Average Causal Effect (CACE)
4. Instrumental variables in multi-site studies
– Estimating the Average CACE– Estimating the Variation in CACE
![Page 3: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/3.jpg)
1. Non-compliance in Multi-Site Trials
Examples
* National Head Start Evaluation (US Dept of HHS, 2010)
* Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006)
* School-based lottery studies (Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009).
* Tennessee STAR (Finn and Achilles, 1990)
* Double-Dose Algebra (Nomi and Allensworth, 2009)
* Small Schools of Choice (MDRC)
![Page 4: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/4.jpg)
2. Observed Data (Single Site)
Observed variables
outcomeY
notif
treatmentineparticipatifM
controltoassignedif
treatmenttoassignedifT
i
i
i
0
1
0
1
![Page 5: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/5.jpg)
2. Potential Outcomes and Causal Effects
YonMofimpactMYM
MonTofimpactMM
YonTofimpactYY
i
iii
iii
)0()1(
)0()1(
)0()1(
![Page 6: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/6.jpg)
2. Average Causal Effects
)(
)(
)(
i
i
i
i
i
i
E
E
E
effectscausalAverage
YonMofimpact
MonTofimpact
YonTofimpact
![Page 7: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/7.jpg)
T=Random assignment M=Participation
Y=outcome
Conventional Instrumental Variable Model (Homogeneous Treatment Effects)
0/
:EffectITTTotal
so
![Page 8: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/8.jpg)
What happens if impacts are heterogeneous?
![Page 9: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/9.jpg)
Single site, heterogeneous treatment effects
TM
Y
Person-specific Causal Model
)(E
)(E
/),(/
),()(
Cov
CovBE
B iii
![Page 10: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/10.jpg)
Assume away “Compliance-Effect Covariance”??
![Page 11: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/11.jpg)
Alternative Approachfor binary M
“Complier Average Causal Effect” (CACE)
or
“Local Average Treatment Effect”
(Bloom, 1984; Angrist, Imbens, and Rubin, 1996)
![Page 12: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/12.jpg)
Principal Stratification(Frangakis and Rubin, 2002)
Stratum M(1) M(0) Ф=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction of pop
Average
Effect
Compliers 1 0 1 Y(1)-Y(0) πcompliers δcompliers
Always-takers
1 1 0 Y(1)-Y(1)=0 πalways 0
Never-takers
0 0 0 Y(0)-Y(0)=0 πnever 0
Defiers 0 1 -1 Y(0)-Y(1) 0 0
![Page 13: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/13.jpg)
Complier-average causal effect
compliers
compliers
complierscompliers
neveralwayscomplierscompliers
so
BE
/
*0*0)(
![Page 14: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/14.jpg)
In Sum
We can estimate the Population-Average Effect of Participating if we assume Cov(Φ,Δ)=0
We can estimate CACE if we assumePr(Φ ≥0)=1
The latter is a much weaker assumption
![Page 15: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/15.jpg)
2. Causal Effects in Multi-site Trials
,,meansOverall
,,meansSpecificSite
:SpecificPerson
jjj
ij
ij
ij
YonMofimpact
MonTofimpact
YonTofimpact
![Page 16: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/16.jpg)
TM
Y
Site-specific Causal Model
jjj
j
j
![Page 17: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/17.jpg)
Multiple Sites:Causal Effects
)430.0,194.1:( 2 valuestrue
2)(,)(
.).(.
jj
jijjijjjij
VarE
MMYY
![Page 18: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/18.jpg)
Combine 2 ITT analyses
Step 1: Estimate the Impact of Treatment Assignment on the Outcome
Results
111
1
)(,)(
.).(.
VarE
eeTTYY jijjijjjij
)75.1,21.(499.*96.1770.
forintervalvalueplausible%95
499.249.0ˆ
53.0,770.0ˆ
1
2211
1
j
se
![Page 19: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/19.jpg)
Step 2: Estimate the Impact of Treatment Assignment on Program Participation
Results
2
1
)(,)(
.).(.
VarE
TTMM jijjijjjij
)86,55(.
forintervalvalueplausible%95
078.0061.0ˆ
703.0ˆ
22
jG
![Page 20: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/20.jpg)
Step 3: Combine Results: mean
095.1703./770.ˆ/ˆˆ
/
caseourIn
![Page 21: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/21.jpg)
Step 3: Combine Results: variance
)248.1,267.(483.*96.1095.1)(%95
483.)703(.)078(.
)078(.*)095.1()249(.ˆ
caseourIn
)(
22
222
22
2222
222222
jPV
so
if
![Page 22: Studying the Mean and Variation in the Effect of Program Participation in Multi-site Trials The research reported here was supported by a grant from the](https://reader036.vdocuments.site/reader036/viewer/2022070413/5697bfee1a28abf838cb945e/html5/thumbnails/22.jpg)
In sum
True Values
Our estimates
430.0,19.1: 2 valuesTrue
487.0ˆ,10.1ˆ 2