introduction to impact evaluation - world...
TRANSCRIPT
![Page 1: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/1.jpg)
Patrick Premand, Laura Rawlings
SSN Course, May 2016
This material builds on Gertler, P. J.; Martinez, S., Premand, P., Rawlings, L. B. and Christel M. J. Vermeersch, 2010, Impact Evaluation in Practice: Ancillary Material, The World Bank, Washington DC (www.worldbank.org/ieinpractice). The content of this presentation reflects the views of the authors and not necessarily those of the World Bank.
Introduction to
Impact Evaluation
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
Pub
lic D
iscl
osur
e A
utho
rized
![Page 2: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/2.jpg)
OutlineMonitoring and Evaluation
The Main Concepts of Impact Evaluation
Choosing the best IE design for your project
Randomization as an operational tool
Evidence Frontiers on Safety Nets?
Evaluation in SSN Projects
Monitoring and Evaluation
Results Chains
Ethical Considerations
![Page 3: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/3.jpg)
Monitoring and Evaluation
![Page 4: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/4.jpg)
Monitoring
A continuous process of collecting and analyzing
information,
o To compare how well a project, program or policy is
performing against expected results, and
o To inform implementation and program management.
![Page 5: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/5.jpg)
Evaluation
A systematic, objective assessment of an on-going or
completed project, program, or policy, its design,
implementation and/or results, asking
o Descriptive Questions to seek to determine what is taking
place and describe aspect of a process.
o Normative Questions to compare what is taking place to
what should be taking place.
o Cause-and-Effect Questions to examine outcomes and
assess what difference the intervention makes in outcomes
![Page 6: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/6.jpg)
Complementary Roles of
Monitoring and EvaluationMonitoring Evaluation
√ Clarifies program objectives √Analyzes why intended results were or
were not achieved
√Links activities and their
resources to objectives √Assesses specific causal contributions
of activities to results
√Translates objectives into
performance indicators and
set targets√ Examines implementation process
√Routinely collects data on
these indicators, compares
actual results with targets√ Explores unintended results
√Reports progress to
managers and alerts them
to problems√
Provides lessons, highlights significant
accomplishment or program potential,
and offers recommendations for
improvement
![Page 7: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/7.jpg)
Descriptive Process Evaluations -- Assess whether a program is being implemented as plannedo Tailored to program’s institutional arrangements and
components
o Often include quantitative and qualitative approaches
o Particularly useful at early stages of program implementation
Normative Targeting/Incidence Analysis – Determine whether the program is reaching its intended beneficiarieso Can be applied at the geographical and household levels
o Includes errors of inclusion and exclusion
o Needs a reference from national measures of poverty (usually direct or proxy measures of income or consumption) against which to benchmark program performance
o Can use national surveys with ID of program beneficiaries, and oversampling if needed and/or regular program registration process
Evaluations in SSN Projects
![Page 8: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/8.jpg)
Causal Impact Evaluations – An assessment of the causal effect of a project , program or policy on beneficiarieso Uses a counterfactual obtained from a control or comparison
group to estimate the state of the beneficiaries in the absence of the program
o Relies on baseline and follow-up data on treatment and comparison groups
Useful for:
-- Determining intermediate or final outcomes attributable to the intervention
• Often used to examine questions with less clear answers such as changes in behavior or outcomes with a range of drivers
-- Testing program design options• For example, different outreach strategies or the relative effectiveness
of different benefit packages
Evaluations in SSN Projects
![Page 9: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/9.jpg)
Evaluations are derived from the question posed and should be tailored accordingly
Evaluations can benefit from…
o Combining quantitative and qualitative data
o Cost – benefit analysis
o Ensuring timeliness of measuring results, producing information to inform key decisions
o Early planning!
Keep an eye on costs and take advantage of available data, national surveys
Structuring Evaluations
![Page 10: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/10.jpg)
Results Chains
![Page 11: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/11.jpg)
Using a Results Chain
What are the intended results of the program?
How will the intended results be achieved? Are there any critical
assumptions? Sufficient resources?
How will we know we have achieved the intended results?
A Results Chain maps out a theory of change
for any project:
![Page 12: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/12.jpg)
Typical Results Chain
Inputs
• Financial, human, and other resources mobilized to support activities
• Budgets,
staffing, other
available
resources
Activities
• Actions taken or work performed to convert inputs into specific outputs
• Series of
activities
undertaken to
produce goods
and services
Outputs
• Products resulting from converting inputs into tangible outputs
• Goods and
services
produced and
delivered , under
the control of
implementing
agency
Outcomes
• Changes resulting from use of outputs by targeted population, (behavior, utilization, conditions)
• Not fully under the control of implementing agency
Final
Outcomes
• The final
objective of the
program –
benefits
• Long-term goals
• Changes in outcomes with multiple drivers
14
Implementation (SUPPLY SIDE) Results (DEMAND + SUPPLY)
![Page 13: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/13.jpg)
Public Works Program
Results Chain Example
Inputs
• Budget for PW
Program
• Ministry of Labor
staff
• Staff from
participating
municipalities
Activities
• Setting of sub-
minimum wage
• Information
campaign
• Enrollment
• Selection of sites,
contracting and
training of PW
operators
Outputs
(Annual)
• 50,000 jobs
• $1,000,000 in
wages
• > 75% of
program costs
transferred as
wages
• 2,000 PW
subprojects
produced
Outcomes
• Net income
transfer to
households
• Skills acquired
• Utility,
maintenance of
PWs
Final
Outcomes
• Income,
employment
• Beneficiary
households:
-income, assets
-health, nutrition
- education
• Aggregate
unemployment,
poverty
15
Implementation (SUPPLY SIDE) Results (DEMAND + SUPPLY)
![Page 14: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/14.jpg)
The Main Concepts of Impact
Evaluation
![Page 15: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/15.jpg)
Impact Evaluation needs to be
distinguished from other “evaluations”
The objective of impact evaluation
is to estimate the causal effect or
impact of a program on outcomes
of interest.
![Page 16: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/16.jpg)
The Objective
Estimate the causal effect (impact)
of intervention (P) on outcome (Y).
(P) = Program or Treatment
(Y) = Outcome Indicator, Measure of Success
Example: What is the effect of a cash transfer program (P)
on Household Consumption (Y)?
![Page 17: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/17.jpg)
SolutionEstimate what would have happened to
outcomes (Y) in the absence of the
program (P).
We call this the Counterfactual.
![Page 18: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/18.jpg)
Communicating complex concepts in 3
slides… Example: What is the Impact of…
giving Ali
(P)
(Y)?
additional money
on Ali’s consumption
![Page 19: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/19.jpg)
The Perfect CloneAli Ali’s Clone
IMPACT=6-4=2 Candies
6 candies 4 candies
![Page 20: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/20.jpg)
In reality, use statistics
Treatment Comparison
Average Y=6 candies Average Y=4 Candies
IMPACT=6-4=2 Candies
![Page 21: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/21.jpg)
Choosing the best IE design for
your project
![Page 22: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/22.jpg)
Finding good comparison groups
We want to find clones for the Alis in our
programs.
The treatment and comparison groups should
o have identical characteristics
o except for benefiting from the intervention.
In practice, use program eligibility & assignment
rules to construct valid estimates of the
counterfactuals
![Page 23: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/23.jpg)
Before
vs AfterCompare: Same individuals
Before and After they
receive P.
Problem: Other things may
have happened over time.
Enrolled
vs Not EnrolledCompare: Group of
individuals Enrolled in a
program with group that
chooses not to enroll.
Problem: Selection Bias.
We don’t know why they
are not enrolled.
Two false counterfactuals to avoid
Both counterfactuals lead
to biased estimates of the
counterfactual and the
impact.
!
![Page 24: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/24.jpg)
The conversation needs to start early
Retrospective Evaluation is necessary when we have to work with a program that has already been roll-out and existing data. Rarely feasible: baseline data? Information on targeting?
In Prospective Evaluation, the evaluation is designed in parallel with the program (and targeting decisions).
The way to go: ensure baseline data is collected, and comparison group exists.
![Page 25: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/25.jpg)
Where do good Comparison Groups come
from?
The rules of program operation
determine the evaluation strategy.We can almost always find a valid
comparison group if: the operational rules for selecting
beneficiaries are equitable, transparent and
accountable;
the evaluation is designed prospectively.
Evaluation design and program design
go hand-in-hand.
![Page 26: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/26.jpg)
5 methods in IE Toolbox
1 Randomized Assignment
3 Regression Discontinuity Design
DD
2 Randomized Promotion
4 Difference-in-Differences
5 Matching
RDD
5 methods in IE toolbox take different
approaches to generate comparison
groups and estimate the counterfactual:
![Page 27: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/27.jpg)
Choosing an IE design for your program
Design IE prospectively to generate good
comparison groups and collect baseline data
3 operational questions to determine which
method is appropriate for a given program
Money: Does the program have sufficient resources to
achieve scale and reach full coverage of all eligible
beneficiaries?
Targeting Rules: Who is eligible for program benefits? Is the
program targeted based on an eligibility cut-off or is it
available to everyone?
Timing: How are potential beneficiaries enrolled in the
program – all at once or in phases over time?
![Page 28: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/28.jpg)
Choosing your IE method(s)
Money Excess demand No Excess demand
Targeting
Timing
Targeted Universal Targeted Universal
Phased
Roll-out
1 Randomized
assignment
4 RDD
1 Randomized
assignment
2 Randomized
promotion
3 DD with
5 Matching
1 Randomized
Assignment
4 RDD
1 Randomized
assignment to
phases
2 Randomized
Promotion to
early take-up
3 DD with
5 matching
Immediate
Roll-out
1 Randomized
Assignment
4 RDD
1 Randomized
Assignment
2 Randomized
Promotion
3 DD with
5 Matching
4 RDD
If less than full
Take-up:
2 Randomized
Promotion
3 DD with
5 Matching
![Page 29: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/29.jpg)
Choosing the IE method in Niger
Money Excess demand No Excess demand
Targeting
Timing
Targeted Universal Targeted Universal
Phased
Roll-out
1 Randomized
assignment
4 RDD
1 Randomized
assignment
2 Randomized
promotion
3 DD with
5 Matching
1 Randomized
Assignment
4 RDD
1 Randomized
assignment to
phases
2 Randomized
Promotion to
early take-up
3 DD with
5 matching
Immediate
Roll-out
1 Randomized
Assignment
4 RDD
1 Randomized
Assignment
2 Randomized
Promotion
3 DD with
5 Matching
4 RDD
If less than full
Take-up:
2 Randomized
Promotion
3 DD with
5 Matching
![Page 30: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/30.jpg)
Randomization as an
operational tool
![Page 31: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/31.jpg)
Many projects use randomization as a fair,
transparent assignment rule
Example 1: Public Works in Cote d’Ivoire
o Originally introduced in post-conflict setting, including to target ex-
combattants
o Public Lottery was considered the most transparent and acceptable selection
process
Example 2: Niger Safety Nets project
o Geographical targeting only works with higher administrative units (Regions,
departments, communes…)
o How to chose between long list of criteria to select villages within units?
o Public lottery was deemed the most transparent, and least controversial
approach.
o Project decided to keep using the randomization approach for purely
operational reasons, including when not needed for the IE.
![Page 32: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/32.jpg)
= Ineligible
Randomization to answer basic IE
questions
= Eligible
1. Population
External Validity
2. Evaluation sample
3. Randomize
treatment
Internal Validity
Comparison
![Page 33: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/33.jpg)
= Not eligible
Randomized to answer IE “design” questions
= Eligible
1. Population2. Evaluation Sample
3. Randomize
treatment.
Comparison
![Page 34: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/34.jpg)
Randomized Assignment
In Randomized Assignment,
large enough samples,
produces 2 statistically
equivalent groups.
We have identified the
perfect clone.
Randomized
beneficiary
Randomized
comparison
Feasible for prospective
evaluations with over-
subscription/excess demand.
Most pilots and new
programs fall into this
category.
!
Consider evaluating relative
effectiveness of alternative
program design options.
![Page 35: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/35.jpg)
Ethical Principles
![Page 36: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/36.jpg)
Ethical Principles
o Basic Considerations for Ethics of Impact Evaluation:o Evaluations should not dictate how programs are assigned;
instead, evaluations should be fitted program assignment rules to the extent that those are clear and fair.
o Groups should not be excluded from an intervention that is known to be beneficial solely for the purpose of an evaluation.
o There can also be an ethical concern about not pursing an evaluation when programs invest substantial public resources in interventions whose effectiveness is unknown
o More Specific Considerations when conducting impact evaluationo Human Subjects Research
o Open Science
![Page 37: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/37.jpg)
Human Subjects Research
Protecting Human Subjects
The World Health Organization (WHO) recommends the following basic criteria for assessing the research projects involving human subjects:
o The rights and welfare of the subjects involved should be adequately protected.
o The researchers should obtain freely given, informed consent from the participants.
o The balance between risk and potential benefits involved should be assessed and deemed acceptable by a panel of independent experts.
o Any special national requirements should be met.
An IRB is a committee that has been formally designated to review, approve, and monitor research involving humans
![Page 38: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/38.jpg)
Open ScienceP
rob
lem Publication bias
Data Mining
Lack of Replication
Co
nse
qu
en
ce
No information on
what doesn’t work
Only significant
impacts are reported
Mistakes undetected;
results between
studies not
comparable
So
luti
on Trial registries
Pre-analysis plans
Data
documentation,
changes in journal
policies
![Page 39: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/39.jpg)
Reference also:
available in Spanish
French and
Portuguese
www.worldbank.org/ieinpractice
Thank you!
![Page 40: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/40.jpg)
Evidence frontiers in SSN?
Spotlights on:
- Links between SSNs and ECD/Nutrition
- Links between SSNs and productive
interventions
![Page 41: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/41.jpg)
Spotlight on links between SSNs and
ECD/Nutrition
![Page 42: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/42.jpg)
SSNs in Early Childhood can Improve some Development
Outcomes— Tend to improve cognitive and socio-emotional development outcomes
— Tends not to affect physical development outcomes
Some advantages change over time— Language advantage may diminish with age
— School enrolment benefits seem to increase with ageSources, IEG: Outcomes age 0-6: http://wbsupport2ecd
Outcomes age 6+: https://ieg.worldbankgroup.org/evaluations/later-impacts-early-childhood-interventions
![Page 43: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/43.jpg)
Madagascar’s social safety net
project: testing behavioral
interventions for parenting
![Page 44: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/44.jpg)
1) Traditional approaches of “training” and “information” have not always been successful in changing behaviors.
2) Extreme poor tend to have a reduced “bandwidth”, focusing on elemental needs, but not making the best (long term) decisions (e.g. “buy more rice”, pulling kids out of school).
3) Behavioral interventions can be cost-effective: light touch reminders rather than heavy-handed “conditionality”.
4) In Africa, social services are rare and thus alternative ways to reach the poor need to be explored.
5) Increasing recognition of the role of parenting in ECD and of behavioral change as a complement to income support in SP
Why focus on behavioral
interventions for parenting?
![Page 45: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/45.jpg)
The Mother Leaders will be trained in five topics:
• Leadership and Self-confidence
• Citizenship
• Essential Family Practices (best practices for families in hygiene and health)
• Early childhood development (cognitive stimulation)
• Reproductive health
Fun, interactive and reflective, drawing on cultural traditions
Mother Leader Model
![Page 46: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/46.jpg)
Can nudges be used to support mothers’ behavioral changes to invest more in their children’s health, nutrition and cognitive stimulation?
Hypotheses about the nudges:
will build intrinsic motivation
serve as complements to Mother Leader sessions
help orient use of cash
Role of Nudges in the Human
Development Cash Transfer
![Page 47: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/47.jpg)
Self-Affirmation An internal attitude and belief
that one has inherent value. The
ability to act in response to one’s
needs and environment.
Plan-MakingThe ability to set goals and
determine the actions necessary
to realize those goals.
Human Development Cash
Transfer “Nudges”
![Page 48: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/48.jpg)
HDCT Randomized Control
Evaluation DesignEligible
51 communes
~51,000 HHs
Cash
38 communes
No ML
77 fkt
Mother Leaders
77 fkt
ML + Nudge 1
(Plan-making)
77 fkt
ML + Nudge 2
(Self-Affirmation)
78 fkt
No Cash
13 communes
District
Communes
(Cash Transfer)
Fokontany
(ML /
Nudge)
![Page 49: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/49.jpg)
Spotlight on linkages between SSNs and
productive interventions
![Page 50: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/50.jpg)
The BRAC graduation model (“Targeting the Ultra Poor”)
Started for households too poor to participate in microcredit
Multifaceted strategy to unlock multiple constraints to
productivity
Run in Bangladesh since 2002
The CGAP multi-country experiment
(Science, 2015) Impact measured at 12 and 24 months
12 mth
CT
Livestock
(goats,
chicken)
3-day
training +
house
visits over
24 mth
2) International evidence: the BRAC model and CGAP evaluation
![Page 51: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/51.jpg)
-0.50
0.00
0.50
1.00
1.50
2.00
Sta
nd
ard
devia
tio
n t
reatm
en
t
eff
ects
Endline 1
Endline 2
Monthly livestock revenues
Pooled endline 1: $85.81 (control mean $73.52): +117%
Pooled endline 2: $55.50 (control mean $80.62): +33%
Income and revenues
![Page 52: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/52.jpg)
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
0.50
0.60
Pooled Ethiopia Ghana Honduras India Pakistan Peru
Sta
nd
ard
devia
tio
n t
reatm
en
t
eff
ects
Endline 1
Endline 2
Per capita consumption, month
Pooled endline 1: $4.55 (control mean $78.80): +6%
Pooled endline 2: $3.36 (control mean $68.80): +5%
Per capita consumption
![Page 53: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/53.jpg)
-300%
-200%
-100%
0%
100%
200%
300%
400%
500%
Ethiopia Ghana Honduras India Pakistan Peru
Total Benefits / Total Costs by Country
Cost-effectiveness
![Page 54: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/54.jpg)
13 16.6
48.5
25.2 3
7.7
59.7
57.3
5.2 8.7
4
61.2
78.3
207.5
27.8
7 52.1
107.9
64.4
4
9.4
5
18
E L 1 E L 2 E L 3 E L 1 E L 2 E L 3 E L 1 E L 2 E L 3
H H L I V E S T O C K R E V E N U E ,
M O N T H
H H N O N - F A R M M I C R O -
E N T E R P R I S E I N C O M E , M O N T H
H H I N C O M E F R O M P A I D
L A B O R , M O N T H
USD
2014 P
PP
Control TUP
2.7
7
3.3
7
4.7
4
2.9
8
3.6
6
6.3
3
E L 1 E L 2 E L 3
S E L F - R E P O R T ED E C O N O M I C S T A T U S
1 / 1 0
Long-term income and
revenues (West Bengal)
![Page 55: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/55.jpg)
How to Build Household Resilience?
o Can Safety Nets Fulfill twin objectives
oOffer short-term protection
o Improve risk-management against climatic risks in the
medium/long term
o Nicaragua “Atencion a Crisis pilots”
Randomized assignment of
HH to 3 interventions
T1
Basic CCT
T2
Basic CCT +
Vocational training
T3
Basic CCT +
Productive investment
![Page 56: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/56.jpg)
2 years after the end of the intervention, are
beneficiaries still better protected against
shocks?
o CCT + Productive investment granto Leads to diversification out of agriculture (non-ag self-employment)
o Fosters Resilience: both income and consumption are protected against the negative welfare effects of drought shocks
o Offers promotion: there is also an average impacts on consumption and income
o CCT + Trainingo Fosters Resilience: both income and consumption are protected
against the negative welfare effects of drought shocks
o Driven by diversification at times of shocks (non-ag self-employment)
o no average impacts on consumption and income
o Traditional CCTo Short-term impacts for consumption/income disappear for those with
CCT
o After the end of the program, no protection against shock71
![Page 57: Introduction to Impact Evaluation - World Bankpubdocs.worldbank.org/en/.../SPLCC-2016-SNCC-D6S1... · The Main Concepts of Impact Evaluation ... Reports progress to managers and alerts](https://reader031.vdocuments.site/reader031/viewer/2022022506/5ac258377f8b9aca388e2872/html5/thumbnails/57.jpg)
THANKS!
Where do you see the frontier?