issues in impact evaluation in design
DESCRIPTION
Issues in impact evaluation in design. Howard White International Initiative for Impact Evaluation (3ie). Impact evaluation. An impact evaluation seeks to attribute all, or part, of the observed change in outcomes to a specific intervention. Impact in the log frame: basic education. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/1.jpg)
Howard WhiteInternational Initiative for
Impact Evaluation (3ie)
![Page 2: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/2.jpg)
An impact evaluation seeks to attribute all, or part, of the observed change in outcomes to a specific intervention.
![Page 3: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/3.jpg)
Level IndicatorsInputs Resources: Funds and
personnelActivities Teacher training
School improvementsDecentralized management and local school management committees
Outputs Trained teachersBetter school facilitiesFunctioning school management committees
Intermediate outcomes Higher school enrolments at all levelsTeacher and parent satisfactionBetter managed schools
Final outcomes Improved learning outcomesImpact Higher productivity and
earningsEmpowerment
![Page 4: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/4.jpg)
Focus on final welfare outcomes, e.g. Infant mortality Income povertySecurity
Usually long-term, but not necessarily so (but then sustainability is an issue)
![Page 5: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/5.jpg)
Projects (or specific interventions) Individual projects are the ‘back bone’ of impact
analysis But even then may only be able to do rigorous
impact analysis of some components Programmes
Sector wide programs can be conceived of as supporting a range of interventions, many of which can be subject to rigorous impact evaluation.
Policies In general different approaches are required, such
as CGEs – these are not being discussed today
![Page 6: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/6.jpg)
Pick a named intervention for an impact evaluation and make a short list of indicators (using the log frame) for evaluation of this intervention
![Page 7: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/7.jpg)
Before AfterProject (treatment)
66
Control
![Page 8: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/8.jpg)
But we don’t know if they were similar before… though there are ways of doing this
Before AfterProject (treatment) 66Control 55
![Page 9: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/9.jpg)
Sometimes this can work … but usually not
Before AfterProject (treatment)
40 66
Control
![Page 10: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/10.jpg)
Before AfterProject (treatment) 40 66Control 44 55
![Page 11: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/11.jpg)
Ex ante design preferred to ex post: impact evaluation design is much stronger if baseline data are available (but may still be possible even if they are not)
Means collecting data before intervention starts, and can be affecting the design of the intervention
But can sometimes use secondary data, that is an existing survey
![Page 12: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/12.jpg)
1. Confounding factors
2. Selection effects
3. Spillovers and contagion
4. Impact heterogeneity
5. Ensuring policy relevance
![Page 13: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/13.jpg)
Other things happen – so before versus after rarely sufficient
So get a control group… but different things may happen there
So collect data on more than just outcome and impact indicators
And collect baseline data But …
![Page 14: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/14.jpg)
Program placement and self-selection Program beneficiaries have particular
characteristics correlated with outcomes – so impact estimates are biased
Need to use experimental or quasi-experimental methods to cope with this; this is what has been meant by rigorous impact evaluation
But it is just one facet of impact evaluation design
Other things can also bias impact estimates
![Page 15: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/15.jpg)
Experimental (randomized):Limited application, but there are
applications and it is a powerful approachMany concerns (e.g. budget and ethics) and
not valid Quasi-experimental design (regression
based):Propensity score matching is most commonRegression discontinuity Interrupted time series Regression modelling of outcomes
![Page 16: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/16.jpg)
Spillover – positive and negative impacts on non-beneficiaries
Contagion – similar interventions in control areas
Need to collect data on these aspects and may need to revise evaluation design
![Page 17: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/17.jpg)
WHAT ARE THE MAJOR CONFOUNDING FACTORS FOR YOUR OUTCOME AND IMPACT INDICATORS?
HOW MIGHT SELECTION BIAS, SPILLOVER AND CONTAGION AFFECT THE EVALUATION OF THE INTERVENTION YOU HAVE SELECTED?
![Page 18: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/18.jpg)
Impact varies by intervention (design),
beneficiary and context
‘Averages’ can be misleading
Strong implications for evaluation
design
![Page 19: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/19.jpg)
Is the impact of X and Y, bigger, equal to or less than the impacts of doing X and Y separately?
For example, hygiene promotion and sanitation facilities
Evidence suggestions they are substitutes- either one reduces incidence child diarrhea by 40-50%, but not by more if the two are combined
![Page 20: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/20.jpg)
Irreparable damage to physical and cognitive development results from nutritional deprivation in the first two years of life
Hence interventions to infants have greater long-run impact on many outcomes than do those aimed at older children (such as school feeding programs)
![Page 21: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/21.jpg)
Expected impactGood year LowAverage-bad year HighVery bad year None
![Page 22: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/22.jpg)
What sort of differences in impact would you expect for your intervention with respect to intervention (design), context and beneficiary?
![Page 23: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/23.jpg)
ProcessStakeholder engagementPackaging messages
DesignTheory-based approachMixed methodsCapture all costs and benefits, including
cross-sectoral effects Cost effectiveness and CBA
![Page 24: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/24.jpg)
Make explicit underlying theory about how inputs lead to intended outcomes and impacts
Documents every step in causal chain Draws on multiple data sources and
approaches Stresses context of why or why not
working
![Page 25: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/25.jpg)
Assumption FindingsProvide nutritional counselling to care givers
Mothers are not decision makers, especially if they live with their mother-in-law
Women know about sessions and attend
90% participation, lower in more conservative areas
Malnourished and growth faltering children correctly identified
No – community nutrition practitioners cannot interpret growth charts
Women acquire knowledge Those attending training do soAnd knowledge is turned into practice
No there is a substantial knowledge-practice gap
Supplementary feeding is additional food for intended beneficiary
No, considerable evidence of substitution and leakage
Adopted changes are sufficient to improve intended outcomes
Only sometimes (not for pregnant women)
![Page 26: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/26.jpg)
Need to collect survey data at the unit of intervention (child, firm etc)
Will need also facility/project data Need data across the log frame and for
confounding factors – and for your instrumental variables (lack of valid instruments is the major obstacle to performing IE)
Designing data collection instruments takes time and should be iterated with qualitative data
![Page 27: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/27.jpg)
Study Data sourcesRural electrification 3 rural electrification surveys
11 DHS2 LSMS
India irrigation and rural livelihoods
Own surveyDistrict-level government dataCensus data
Bangladesh maternal and child health and nutrition
DHSProject data + national nutrition survey
Ghana basic education 1988/89 GLSS (LSMS)Own follow up survey
Kenya agricultural extension 2 previous rural surveysOwn follow up survey
![Page 28: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/28.jpg)
OUTLINE YOUR PROPOSED EVALUATION DESIGN (TIMING OF DATA COLLECTION, IDENTIFICATION OF CONTROL, IF ANY)
WHAT DATA SOURCES WOULD YOU USE FOR YOUR PROPOSED EVALUATION?
![Page 29: Issues in impact evaluation in design](https://reader036.vdocuments.site/reader036/viewer/2022062310/568167b5550346895ddd05a9/html5/thumbnails/29.jpg)
VISIT
WWW.3IEIMPACT.ORG