balancing external representativity and ... - world...
TRANSCRIPT
![Page 1: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/1.jpg)
ABCDE, World Bank 2016
{ noriega, pentland } @ mit.edu
Balancing External Representativity and Interference in Large-Scale Rural Experiments
Alejandro Noriega and Alex Pentland
MIT Media Laboratories MIT Institute for Data, Systems and Society (IDSS)
![Page 2: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/2.jpg)
Field RCTs in Data-Rich Contexts
SAMPLE TARGET
POPULATION
Interference (SUTVA) Extrapolation
![Page 3: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/3.jpg)
SAMPLE TARGET
POPULATION
Interference (SUTVA) Extrapolation
Field RCTs in Data-Rich Contexts
![Page 4: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/4.jpg)
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
![Page 5: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/5.jpg)
National Social Program -- PROSPERA
• National-Wide system: – families below poverty line – Based on engagement and co-
responsibility
• 2nd Largest Conditional Cash Transfer program in the world – +8M families, +25M beneficiaries
• Services: – Health – Cash transfer – Nutrition – Productivity
Interference and Representativity in Large Rural Experiments
![Page 6: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/6.jpg)
PROSPERA – Traditional communication means
Interference and Representativity in Large Rural Experiments
![Page 7: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/7.jpg)
PROSPERA – Commutes
Interference and Representativity in Large Rural Experiments
![Page 8: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/8.jpg)
PROSPERA – Commutes
Interference and Representativity in Large Rural Experiments
![Page 9: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/9.jpg)
5,000 Approx. 5,000 women Jan 16
Interference and Representativity in Large Rural Experiments
![Page 10: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/10.jpg)
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
![Page 11: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/11.jpg)
Interference and Representativity in Large Rural Experiments
PROSPERA – All Clinics
![Page 12: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/12.jpg)
Interference and Representativity in Large Rural Experiments
PROSPERA – Eligible Clinics
![Page 13: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/13.jpg)
Interference and Representativity in Large Rural Experiments
Extrapolation
PROSPERA – Eligible Clinics
![Page 14: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/14.jpg)
Estimation error decomposition and non-random sampling
Interference and Representativity in Large Rural Experiments
Sampling Error
Treatment Assignment Error
![Page 15: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/15.jpg)
Estimation error decomposition and non-random sampling
Interference and Representativity in Large Rural Experiments
Sampling Error
Treatment Assignment Error
non-random sampling
![Page 16: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/16.jpg)
Interference and Representativity in Large Rural Experiments
Interference
Extrapolation
PROSPERA – Eligible Clinics
![Page 17: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/17.jpg)
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
![Page 18: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/18.jpg)
Modeling Interference –- Road-and-Walk Route Distance
Interference and Representativity in Large Rural Experiments
![Page 19: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/19.jpg)
𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝
Gravity Model of Interference
𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝
Interference and Representativity in Large Rural Experiments
![Page 20: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/20.jpg)
Inter-clinic Interference Network
Interference and Representativity in Large Rural Experiments
![Page 21: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/21.jpg)
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
![Page 22: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/22.jpg)
Sample search: Minimizing Interference
Greedy
Interference and Representativity in Large Rural Experiments
![Page 23: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/23.jpg)
Sample search: Minimizing Interference
Algorithm 1
Interference and Representativity in Large Rural Experiments
![Page 24: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/24.jpg)
Interference Minimization
Interference and Representativity in Large Rural Experiments
![Page 25: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/25.jpg)
Interference vs. Representativity Tradeoff
Interference and Representativity in Large Rural Experiments
![Page 26: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/26.jpg)
Interference vs. Representativity Tradeoff
Interference and Representativity in Large Rural Experiments
![Page 27: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/27.jpg)
Representativity – KS Covariate Distance (Kolmogorov-Smirnov)
Interference and Representativity in Large Rural Experiments
Target Population
Sample
KS Distance For relevant covariates :
• Social lag • Illiteracy • Education • Indigenism • Newborn Weight • Diseases • Distance to clinic • Clinic infrastructure
![Page 28: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/28.jpg)
Interference vs. Representativity Tradeoff
Interference and Representativity in Large Rural Experiments
![Page 29: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/29.jpg)
Interference and Representativity in Large Rural Experiments
Bi-Objective Sample Search @k=25
Interference (SUTVA) Representativity (KS)
![Page 30: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/30.jpg)
Interference vs. Representativity: Bi-Objective Sample Search
Interference and Representativity in Large Rural Experiments
![Page 31: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/31.jpg)
• Evaluate direct and ‘spillover’ health effects (experiment is on!)
• Evaluate models of interference: gravity model, road network vs. spatial, communications (CDRs)
• Bi-objective sample search: optimization bounds, better bi-objective heuristics.
Research Paths
Forward
Interference and Representativity in Large Rural Experiments
![Page 34: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/34.jpg)
Modeling Interference (SUTVA violations)
Interference and Representativity in Large Rural Experiments
![Page 35: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/35.jpg)
Interference vs. Representativity
Bi-Objective Sample Search (Algorithm 1)
Interference and Representativity in Large Rural Experiments
![Page 36: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/36.jpg)
Innovating the Mexican social health and financial system
Interference vs. Representativity
Bi-Objective Genetic Algorithm
![Page 37: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/37.jpg)
Innovating the Mexican social health and financial system
Interference Minimization
![Page 38: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/38.jpg)
Modeling Interference –- Road Network + Walk
Distance
Innovating the Mexican social health and financial system
![Page 39: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/39.jpg)
Health Intervention
Potential Projects with Mexico – National and Mexico City
Geospatial Analysis: • Macro spatial dynamics of health clinics, rural & urban
populations of beneficiaries, and mobile phone coverage
![Page 40: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/40.jpg)
Potential Projects with Mexico – National and Mexico City
Geospatial Analysis: • Macro spatial dynamics of health clinics, rural & urban
populations of beneficiaries, and mobile phone coverage
![Page 41: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/41.jpg)
Potential Projects with Mexico – National and Mexico City
The collaboration
![Page 42: Balancing External Representativity and ... - World Bankpubdocs.worldbank.org/pubdocs/publicdoc/2016/6/58544146671481… · PROSPERA – National Social Program A Large Scale Experiment](https://reader034.vdocuments.site/reader034/viewer/2022052613/5f1b76832f467e162b05912b/html5/thumbnails/42.jpg)
Potential Projects with Mexico – National and Mexico City
The collaboration
Free SMS and CDRs
RapidPro plattform