aadapt workshop for impact evaluation in agriculture and rural development goa, india 2009 with...
TRANSCRIPT
![Page 1: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/1.jpg)
AADAPT Workshop for Impact Evaluation in Agriculture and Rural DevelopmentGoa, India 2009With generous support from Gender Action Plan
![Page 2: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/2.jpg)
AADAPT Workshop for Impact Evaluation in Agriculture and Rural DevelopmentGoa, India 2009With generous support from Gender Action Plan
Arianna LegoviniHead, Development Impact Evaluation Initiative (DIME)World Bank
Impact Evaluation for Real Time Decision-Making
![Page 3: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/3.jpg)
Do we know…
What information and services will improve market conditions for farmers? –India soybeans
What payment system will secure the financial sustainability of irrigation schemes? –Ethiopia irrigation
What is the best way to select local projects? –Indonesia direct voting versus representatives’ decisions
Will local workforce participation improve construction and maintenance of local investments? –Afghanistan road construction
![Page 4: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/4.jpg)
Trial and error
These are difficult questions…We turn to our best judgment for
guidance and pick a subsidy level, a voting scheme, a package of services…
Is there any other subsidy, scheme or package that will do better?
![Page 5: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/5.jpg)
The decision process is complex
A few big decisions are taken during design but many more decisions are taken during roll out & implementation
DesignEarly roll outImplementation
![Page 6: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/6.jpg)
Developing a decision tree for an irrigation scheme…
Irrigation schemeBuild and operate to large private operator
Subschemes organized around farmer associationsWater payments independently
collectedWater payments subtracted from crop
sales
New user associations establishedWater payments independently
collectedWater payments subtracted from crop
salesBuilt by private constructions co. and operated by user
consortiumSubschemes organized around farmer associationsWater payments independently
collectedWater payments subtracted from crop
sales
New user associations establishedWater payments independently
collectedWater payments subtracted from crop
sales
![Page 7: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/7.jpg)
How to select between plausible alternatives?
Establish which decisions will be taken upfront and which will be tested during roll-out
Scientifically test critical nodes: measure the impact of one option relative to another or to no intervention
Pick better and discard worse during implementation
Cannot learn everything at onceSelect carefully what you want to test
by involving all relevant partners
![Page 8: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/8.jpg)
Walk along the decision tree for your irrigation scheme to get more results
Irrigation schemeBuild and operate to large private operator
Subschemes organized around farmer associationsWater payments independently
collectedWater payments subtracted from crop
sales
New user associations establishedWater payments independently
collectedWater payments subtracted from crop
salesBuilt by private constructions co. and operated by user
consortiumSubschemes organized around farmer associationsWater payments independently
collectedWater payments subtracted from crop
sales
New user associations establishedWater payments independently
collectedWater payments subtracted from crop
sales
![Page 9: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/9.jpg)
What is Impact Evaluation? Impact evaluation measures the
effect of an intervention on outcomes of interest relative to a counterfactual (what would have happened in the absence of)
It identifies the causal effect of an
intervention on an outcome separately from the effect of other time-varying conditions
![Page 10: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/10.jpg)
Impact evaluation
Application of the scientific method to understand and measure human behavior Hypothesis
▪ If we subsidize fertilizer then farmers will use more fertilizer and increase production
Testing ▪ Provide small discount with deadline after harvest or large
subsidy before planting. Compare fertilizer use and productivity
Observations▪ Fertilizer use increases more with small discount with deadline▪ Production increases and then declines with fertilizer
quantities Conclusion
▪ Timing the subsidy when farmers have financial resources is most effective
![Page 11: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/11.jpg)
What is counterfactual analysis?
Counterfactual analysis isolates the causal effect of an intervention on an outcome Effect of subsidy on fertilizer use Effect of information on market prices
Compare same individual with & without subsidy, information etc. at the same point in time to measure the effect This is impossible
Impact evaluation uses large numbers (farmers, communities) to estimate the effect
![Page 12: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/12.jpg)
What is a good counterfactual?
Treated & counterfactual groups have identical observed and unobserved characteristics
The only reason for the difference in outcomes is due to the intervention
![Page 13: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/13.jpg)
How to define a counterfactual?
Design impact evaluation before the intervention is rolled out
Define eligibility Assign interventions to some and not
some other eligible populations on a random basis or on the basis of clear and measurable criteria
Obtain a treatment and a control groups Measure and compare outcomes in those
groups over time
![Page 14: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/14.jpg)
Nudging Farmers to Use Fertilizer: Evidence from Kenya (Duflo, Kremer, Robinson, 2009)
Farmers randomly selected into groups: Free delivery offered for planting or top
dressing fertilizer just after harvest No subsidy 14.3 percentage point increase in
fertilizer use relative to controls Free delivery and 50% subsidy later
during top dressing (1-2 months after planting) 13.2 percentage point increase in
fertilizer use relative to controls Control group with none of the above
![Page 15: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/15.jpg)
Nudging Farmers to Use Fertilizer Policy conclusions
Small, well-timed discounts can induce some farmers to purchase productive inputs
Time dimensions and farmer “impatience” may be important for technology adoption
Large, costly subsidies might not be appropriate policy response
![Page 16: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/16.jpg)
How is this done?
Select one group to receive treatment (subsidy, information…)
Find a comparison group to serve as counterfactual
Use these counterfactual criteria: Treated & comparison groups have identical
initial average characteristics (observed and unobserved)
The only difference is the treatment Therefore the only reason for the difference in
outcomes is due to the treatment
![Page 17: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/17.jpg)
Methods (tomorrow)
Experimental or random assignment Equal chance of being in the treatment or
comparison group By design treatment and comparison have the
same characteristics (observed and unobserved), on average
Simple analysis (means comparison) and unbiased impact estimates
Non-experimental (Regression discontinuity, IV and encouragement designs, Difference in difference) Require more assumptions or might only estimate
local treatment effects May suffer from non-observed variable bias Use more than one method to check robustness of
results
![Page 18: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/18.jpg)
How is monitoring different from impact evaluation?Monitoring is trend
analysis Change over time Compare results before
and after on the “treated” group
Y
AfterBefore
A
B
t0 t1
A
Intervention
Intervention
Change
Change
B’ImpactImpactImpact evaluation
Change over time and relative to comparison
Compare results before and after in the “treated” group and relative to the “untreated” group
![Page 19: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/19.jpg)
Monitoring & Impact Evaluation monitoring to track
implementation efficiency (input-output)
INPUTS OUTCOMESOUTPUTS
MONITOR EFFICIENCY
EVALUATE EFFECTIVENESS
$$$
BEHAVIORBEHAVIOR
impact evaluation to measure effectiveness (output-outcome)
![Page 20: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/20.jpg)
Question types and methods
Descriptive analysis
Descriptive analysis
Causal analysis
Causal analysis
Monitoring and process evaluation
Is program being implemented efficiently?
Is program targeting the right population?
Are outcomes moving in the right direction?
Impact Evaluation What was the effect of the program on
outcomes? How would outcomes change under
alternative program designs? Is the program cost-effective?
![Page 21: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/21.jpg)
When would you use M&E and when IE?
Are grants to communities being delivered as planned?
Does participation reduce elite capture?
What are the trends in agricultural productivity?
Does agricultural extension increase technology adoption?
M&E
IE
M&E
IE
![Page 22: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/22.jpg)
Uganda Community-Based Nutrition
Failed project Project ran into financial difficulties Parliament negative reaction Intervention stopped
…but…
Strong impact evaluation results Children in treatment scored half a standard
deviation better than children in the control
Recently, Presidency asked to take a second look at the evaluation: saving the baby?
Separate performance from quality of intervention: babies & bath water
![Page 23: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/23.jpg)
Why Evaluate?
Improve quality of programs Separate institutional performance from quality
of intervention Test alternatives and inform design in real time Increase program effectiveness Answer the “so what” questions
Build government institutions for evidence-based policy-making Plan for implementation of options not solutions Find out what alternatives work best Adopt better way of doing business and
taking decisions
![Page 24: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/24.jpg)
Institutional frameworkPM/Presidency:
Communicate to constituencies
Treasury/Finance:
Allocate budget
Line ministries:
Deliver programs and negotiate
budget
Cost-effectiveness of different programs
Effects of government
program
BUDGET
SERVICE DELIVERY
CAMPAIGNPROMISES
Accountability
Cost-effectiveness of alternatives and effect of
sector programs
![Page 25: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/25.jpg)
From: Program is a set of activities designed
to deliver expected results
Program will either deliver or not
To: Program is menu of alternatives with a
learning strategy to find out which work best
Change programs overtime to deliver more results
Shifting Program Paradigm
![Page 26: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/26.jpg)
Shifting Evaluation Paradigm From retrospective, external, independent
evaluation Top down Determine whether program worked or not
To prospective, internal, and operationally driven impact evaluation /externally validated Set program learning agenda bottom up Consider plausible implementation alternatives Test scientifically and adopt best Just-in-time advice to improve effectiveness
of program over time
![Page 27: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/27.jpg)
Retrospective impact evaluation: Collecting data after the event you don’t know
how participants and nonparticipants compared before the program started
Have to try and disentangle why the project was implemented where and when it was, after the event
Prospective evaluation: design the evaluation to answer the question
you need to answer collect the data you will need
27
Retrospective (designed & evaluated
ex-post) vs. Prospective (designed ex-ante and evaluated ex-post)
![Page 28: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/28.jpg)
Is this a one shot analytical product?
This is a new model to change the way decisions are taken
It is about building a relationship between operations and research
Adds results-based decision tools to complement existing sector skills
The relationship delivers not one but a series of analytical products
Must provide useful (actionable) information at each step of the impact evaluation
![Page 29: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/29.jpg)
Ethical considerations
It is not ethical to deny benefits to something that is available and we know works HIV medicine proven to prolong life
It is ethical to test interventions before scale up if we don’t know if it works and whether it has unforeseen consequences Food aid may impair local markets and create
perverse incentives Most times we use opportunities created by
roll out and budget constraints to evaluate so as to minimize ethical considerations
![Page 30: AADAPT Workshop for Impact Evaluation in Agriculture and Rural Development Goa, India 2009 With generous support from Gender Action Plan](https://reader035.vdocuments.site/reader035/viewer/2022081514/56649f535503460f94c77497/html5/thumbnails/30.jpg)
Thank you
Financial support from
Is gratefully acknowledged