![Page 1: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/1.jpg)
New research methods for the evaluation of policy changes
Sanjay Basu, MD, PhD
O L D P R O B L E M S , N E W S O L U T I O N S
![Page 2: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/2.jpg)
Three new methods to discuss
• If you have individual-level data, but an imperfect control group:• Near-far matching
• If you have population-level data, but an imperfect control group:• Synthetic control analysis
• If you have either type of data, and want to estimate disparities:• Distributional decomposition
![Page 3: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/3.jpg)
Example: Is the school meal program “worsening child health”?
In a recent analysis: “Rural children in the meals program had a significantly higher probability of
being stunted than those not in the program, even after controlling for income differences.” (IIPS, 2014)
![Page 4: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/4.jpg)
Our typical solutions
Propensity score matching • Problem: unobserved confounders
![Page 5: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/5.jpg)
Our typical solutions
An instrumental variable Problem of “weak” instruments
![Page 6: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/6.jpg)
Method: near-far matching
Baiocchi, et al., Health Serv Outcomes Res Method 2012
![Page 7: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/7.jpg)
Re-analysis of food programs
Re-analysis of India school mean program and stunting Without matching: OR = 1.28 (1.12,1.44) With propensity score matching: OR= 0.96 (0.80, 1.12) With near-far matching: OR = 0.84 (0.70,0.98)
For a worked example, see: Lorch et al, Pediatrics, 2012
![Page 8: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/8.jpg)
The ‘individualistic fallacy’
Some of our policies are designed to focus on a population-level outcome, not just an individual-level one
And many of our most interesting policies are ‘case studies’ of one group performing a policy, with no natural ‘control group’
![Page 9: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/9.jpg)
Typical solution: difference-in-differences analysis
![Page 10: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/10.jpg)
Example of synthetic control
For worked example, see Abadie et al., Am J Pol Sci, 2014 In Stata: ssc install synthIn R: install.packages("Synth")
![Page 11: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/11.jpg)
Decomposition
As compared to standard regression
![Page 12: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/12.jpg)
Distributional decomposition
For proof and worked example, see Basu et al., Am J Epi, 2015 In Stata: download distdecomp package from sdr.stanford.edu
![Page 13: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/13.jpg)
References
Near-far matching Mike Baiocchi, http://web.stanford.edu/~baiocchi/ See: Baiocchi, et al., Health Serv Outcomes Res Method 2012
Synthetic control Jens Hainmueller, http://web.stanford.edu/~jhain/ See: Abadie et al., Am J Pol Sci, 2014
Distributional decomposition Sanjay Basu, http://web.stanford.edu/~basus/ See: Basu et al., AJE (in press), 2015
![Page 14: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/14.jpg)
Additional slides
![Page 15: New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f495503460f94c6b261/html5/thumbnails/15.jpg)
Regression discontinuity
Advantages:
only post-policy data
needed
Disadvantages:
people can ‘cheat’
only informs the margins