marginal treatment effects and the external validity of the oregon health insurance experiment...
TRANSCRIPT
![Page 1: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/1.jpg)
Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment
Amanda Kowalski
Associate Professor, Department of Economics, YaleFaculty Research Fellow, NBERVisiting Associate Professor, SIEPR
October 2015
![Page 2: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/2.jpg)
Experiments Address Internal Validity,But What About External Validity?
• Explosion in use of experiments• Are they externally valid?• Quandary: what to do if the results from
randomized experiments do not conform to results of seemingly convincing natural experiments?
• Policy-relevant example: Oregon Health Insurance Experiment vs. Massachusetts Health Reform
• My results show that there may not be a conflict after all…
![Page 3: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/3.jpg)
Oregon Health Insurance Experiment (OHIE) Shows ER Visits Increased
• Do these results generalize to other settings?• Massachusetts Health Reform showed – ER visits decreased • Miller (2012), Smulowitz et al. (2011)
– ER visits stayed the same• Chen, Scheffler, Chandra (2011)
– Admissions from ER (proxy for ER visits) decreased• Kolstad & Kowalski (2012)
![Page 4: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/4.jpg)
I Apply MTE Methods to Examine External Validity of the OHIE
• Prior literature– Bjorklund and Moffitt (1987)– Heckman Vytlacil (1999, 2005, 2007)– Brinch, Mogstad, Wiswall (2015)
• Extended Heckman Vytlacil to allow discrete instrument
• MTEs in experiments (binary instruments) are now possible• To my knowledge, these methods have not been applied to experiments
(Brinch et al. application is to quantity-quality of children in IV framework)
• Key insight: unless everyone assigned to lottery gets intervention (and everyone else does not), then external validity will be affected by selection
![Page 5: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/5.jpg)
I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon
• Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization
• Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate
• MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments
![Page 6: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/6.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 7: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/7.jpg)
Selection into treatment given lottery is key model element
![Page 8: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/8.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 9: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/9.jpg)
MTE is treatment effect for individuals indifferent to treatment
• Can recover LATE from MTE:
![Page 10: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/10.jpg)
Can identify linear MTE from standard LATE assumptions, nonlinear MTE requires additional separability
• Assumption 1 sufficient for linear MTE– Standard LATE conditional independence and monotonicity– Key is additional moment: separation of compliers from always takers
and never takers• Assumption 2 yields nonlinear MTE
– Additive separability between observed and unobserved heterogeneity often imposed in applied work
![Page 11: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/11.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 12: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/12.jpg)
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10)
All
Always Takers and
Treated Compliers
Always Takers
Never Takers
Never Takers and
Control Compliers
Treated Compliers
Control Compliers
All Compliers
D = 1 D = 0Z = 1 Z = 0 Z = 1 Z = 0 Z = 1 Z = 0 Z = 1 Z = 0
Female 0.56 0.55 0.56 0.58 0.66 0.50 0.52 0.51 0.55 0.53Age in 2009 40.7 40.7 40.7 41.4 40.2 39.6 40.9 42.5 42.6 42.6English 0.91 0.91 0.91 0.92 0.91 0.90 0.91 0.92 0.92 0.92Y (Any ER visit) 0.37 0.38 0.37 0.43 0.44 0.31 0.34 0.41 0.37 0.38Y (Number of ER visits) 1.12 1.16 1.09 1.36 1.38 0.85 0.97 1.35 1.13 1.20Number of Observations 19,643 6,755 12,888 4,096 3,747 2,659 9,141 2,132 4,068 6,200D = 1 if any insuranceZ = 1 if randomized into lottery
D = 1 D = 0
Source: Oregon Administrative Data, 1 Lottery Entrant in Household.
Compliers, Always Takers, Never Takers
• Standard formulas for average characteristics
Always takers: D=1 regardless of Z=1 or Z=0Compliers: D=1 if Z=1 but D=0 if Z=0Never takers: D=0 regardless of Z=1 or Z=0Defiers: D=1 if Z=0 but D=0 if Z=1 (not present by assumption)
![Page 13: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/13.jpg)
New Test of Internal Validity
![Page 14: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/14.jpg)
Test of External Validity(1) (2) (3) (4) (5) (6) (7) [(4) - (5)] - [(6) - (7)]
All
Always Takers and
Treated Compliers
Always Takers
Never Takers
Never Takers and
Control Compliers
Difference-in-difference Test of External Validity
Z = 1 Z = 0 Z = 1 Z = 0 Z = 1 Z = 0Female 0.56 0.55 0.56 0.58 0.66 0.50 0.52 -0.058
(0.016)†††(0.014)***
Age in 2009 40.7 40.7 40.7 41.4 40.2 39.6 40.9 2.586(0.388)†††(0.383)***
English 0.91 0.91 0.91 0.92 0.91 0.90 0.91 0.013(0.009)(0.009)
Y (Any ER visit) 0.37 0.38 0.37 0.43 0.44 0.31 0.34 0.007(0.015)(0.017)
Y (Number of ER visits) 1.12 1.16 1.09 1.36 1.38 0.85 0.97 0.103(0.081)(0.087)
Number of Observations 19,643 6,755 12,888 4,096 3,747 2,659 9,141††† p<0.01, †† p<0.05, † p<0.1; asymptotic standard errors*** p<0.01, ** p<0.05, *p<0.1; bootstrapped standard errorsD = 1 if any insuranceZ = 1 if randomized into lottery
D = 1 D = 0
Source: Oregon Administrative Data, 1 Lottery Entrant in Household.
![Page 15: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/15.jpg)
The Geometry of the LATE and MTE
Always takers Always takers and
treated compliers
Never takers
Never takers and
control compliers
![Page 16: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/16.jpg)
The Geometry of the LATE and MTE
Always takers Always takers and
treated compliers
Never takers
Never takers and
control compliers
![Page 17: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/17.jpg)
The Geometry of the LATE and MTE
![Page 18: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/18.jpg)
The Geometry of the LATE and MTE
![Page 19: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/19.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 20: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/20.jpg)
Any Emergency Room Visits(1) (2) (3) (4) (5) (6) (7)
Medicaid 0.0697(0.024)†††
(0.024)***
Any insurance 0.0610 0.0665 -0.0124 0.0633 0.207 0.0453(0.021)††† (0.026)†† (0.022) (0.022)††† (0.087)†† (0.023)††
(0.021)*** (0.023)*** (0.022) (0.022)*** (0.094)** (0.022)**
Control variablesPre-visits,
Lottery Entrants
Pre-visits, Lottery
Entrants
Lottery Entrants
No Controls
Lottery Entrants, Common Controls
Common Controls
Common Controls
Regression sampleFull sample Full sample Full sample Full sample Full sample
2+ Lottery Entrants
1 Lottery Entrant
Observations 24,646 24,646 24,646 24,646 24,646 5,003 19,643R-squared 0.151 0.146 0.023 0.036 0.050 0.033 0.033Mean of Y in control group 0.34 0.34 0.34 0.34 0.34 0.21 0.37
Bringing MTE to OHIE
• Control for lottery entrants matters• Otherwise, replicate well with covariates• Results for number of visits similar
![Page 21: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/21.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 22: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/22.jpg)
MTE Results
![Page 23: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/23.jpg)
MTE Weights
![Page 24: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/24.jpg)
Weighted MTE
![Page 25: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/25.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 26: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/26.jpg)
Propensity Scores in MA
![Page 27: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/27.jpg)
Propensity Scores in MACompliers Only, Rescaled
![Page 28: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/28.jpg)
Massachusetts Weights
![Page 29: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/29.jpg)
Oregon MTE/Massachusetts WeightsNegative ER Impacts Plausible in MA
![Page 30: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/30.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 31: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/31.jpg)
Treatment Effect HeterogeneityBinary Subgroup Analysis
• Point estimates all positive except non-English speakers
![Page 32: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/32.jpg)
Treatment Effect HeterogeneityMultivariate Subgroup Analysis
• Many negative point estimates
![Page 33: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/33.jpg)
38
Who Are the People with Reductions in ER Utilization?
![Page 34: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/34.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 35: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/35.jpg)
Experimental Design for External Validity
• Collect data on treatment (endogenous variable)• Collect a wide array of covariates– Use standard definitions
• Only stratify on covariates that can be obtained elsewhere
• Collect data on experimental subjects randomized out of treatment who get treatment via other means
• Design experiment so that always takers and never takers are possible
![Page 36: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/36.jpg)
Lessons for Experimentalists
• Going to great lengths to encourage “attendance” could decrease external validity– In the limiting case, if full attendance, cannot
estimate MTE• Going to great lengths to encourage
“attendance” could decrease estimated effect size
![Page 37: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/37.jpg)
Outline
• Model• Marginal Treatment Effects• New Tests of Internal and External Validity• Bringing MTE to OHIE• Results• Bringing Oregon MTE to Massachusetts• Treatment Effect Heterogeneity• Experimental Design for External Validity• Conclusions
![Page 38: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/38.jpg)
I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon
• Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization
• Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate
• MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments
![Page 39: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/39.jpg)
Appendix Slides
![Page 40: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/40.jpg)
Appendix: Bringing OHIE to MTENumber of ER visits
• Return to similar results for any visits
![Page 41: Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics,](https://reader035.vdocuments.site/reader035/viewer/2022070411/56649f435503460f94c63cb6/html5/thumbnails/41.jpg)
Appendix:Formulas for Complier Characteristics
• Treated Compliers
• Control Compliers
• Return to results with complier characteristics