using evidence to inform policy: impact evaluation of human development programs overseas...
TRANSCRIPT
![Page 1: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/1.jpg)
USING EVIDENCE TO INFORM POLICY:IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS
Overseas Development Institute
May 4, 2010
ARIEL FISZBEINCHIEF ECONOMIST, HUMAN DEVELOPMENTWORLD BANK
![Page 2: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/2.jpg)
Motivation
‘Since it is difficult to distinguish the good from the bad prophet, we must be suspicious of all prophets: it is better to avoid revealed truths, even if we feel exalted by their simplicity and splendor, even if we find them comfortable because they come at no cost. It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’ Primo Levi
2
![Page 3: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/3.jpg)
How can we learn from experience?
Traditional approach: identify best practice cases and inductively derive conclusions.
But the strength of those conclusions is directly related to the internal and external validity of the underlying evaluation.
We could do better if we plan well and take a more systematic approach to learning from experience.
3
![Page 4: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/4.jpg)
The evaluation problem
Impact: difference in outcome with and without program. Cannot observe simultaneously. with and without comparisons deceptive if
participation depends on individual attributes that influence outcomes
before/after comparisons can be misleading if other things happened during the period
Need to estimate counterfactual (control or comparison) Treated & control have same characteristics on
average, Only reason for difference in outcomes is the
intervention
4
![Page 5: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/5.jpg)
Multiple design options
Experimental (often using phase in) Quasi-experiments/non-experimental
Regression Discontinuity (RD) Difference in difference – panel data Other (Instrumental Variables,
matching, etc) In all cases, these will involve
knowing the rule for assigning treatment
Advantages of prospective design
5
![Page 6: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/6.jpg)
Evolution of IE in the World Bank6
...Completed before 2004
…Completed from 2004
…Ongoing in 2010
0
25
50
75
100
125
150
175
200
18
58
175
Number of Ongoing IEs at the World Bank…
![Page 7: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/7.jpg)
Strong focus on HD areas7
Social Protec-tion20%
Health17%
Education13%ECD & Nutr.
7%
Other9%
Gov .4%
Urban Upgr.6%
Agric.6%
Fin. & Priv. Sec.8%
Local Dev.10%
The World Bank Impact Evaluation Program: Thematic Areas
HD thematic ar-eas: 58% of total World Bank IEs
![Page 8: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/8.jpg)
Strategic themes8
Education, Other
Pay For Performance in Health
Malaria
HIV
Early Childhood Development
Active Labor Market Programs
School Accountability
Conditional Cash Transfers
0 5 10 15 20 25
6
9
9
13
14
16
19
20
The World Bank Impact Evaluation program on Human Development
Number of OngoingImpact Evaluations
Thematic areas
![Page 9: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/9.jpg)
… and IEs of non-HD projects often measure impact on HD outcomes:
Agricultural Development Indicators: anthropometrics
Housing Indicators: morbidity, anthropometrics
Rural Electrification Indicators: study time, respiratory illnesses
Rural Roads Indicators: school attendance
Urban Upgrading Indicators: employment
Water Supply and Sanitation Indicators: morbidity, anthropometrics
9
![Page 10: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/10.jpg)
Building a global evidence base on the impact of HD programs
10
Europe & Central Asia
Middle East & North Africa
East Asia & the Pacific
Latin America & Caribbean
South Asia
Africa
0 10 20 30 40 50 60
1
3
10
17
21
54
The World Bank Impact Evaluation program on Human Development
Number of OngoingImpact Evaluations
Regions
![Page 11: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/11.jpg)
Priority thematic areas
1. Conditional Cash Transfers2. Paying for Performance in Health (P4P)3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS Prevention7. Early Childhood Development
11
![Page 12: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/12.jpg)
12
CCTs have become very popular….
![Page 13: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/13.jpg)
13
CCTs have become very popular….
![Page 14: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/14.jpg)
First generation evaluation questions14
Impact on consumption and poverty Side effects?
Impacts on service utilization
Impacts on human development outcomes
![Page 15: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/15.jpg)
Impacts on consumption and poverty
Mexico(1999)
Nicaragua(2002)
Colombia(2006)
Honduras(2002)
Average transfer (% of per capita consumption)
20% 30% 13% 11%
Impact on per capita consumption (%)
8.3%** 20.6%** 10%** 7%*
Impact: headcount index (% points)
1.3** 5.3** 2.9* --
Impact: poverty gap (% points)
3.0** 9.0** 7.0** 2.0*
Impact: sqd. poverty gap (% points)
3.4** 8.6** 2.2** 2.0*
15
![Page 16: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/16.jpg)
CCT impacts on food consumption16
![Page 17: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/17.jpg)
Impacts on school enrollment
Age range Baseline enrollment
Impact (% points)
Size of transfer
Colombia 8-1314-17
91.7%63.2%
2.1**5.6***
17%
Chile 6-15 60.7% 7.5*** 3-7%
Ecuador 6-17 75.2% 10.3*** 10%
Mexico Grade 0-5Grade 6
Grade 7-9
94.0%45.0%42.5%
1.98.7***
0.620%
Nicaragua 7-13 72.0% 12.8*** 30%
Cambodia Grade 7-9 65.0% 31.3*** 2%
Bangladesh 11-18 44.1% 12.0** 1%
Pakistan 10-14 29.0% 11.1*** 3%
17
![Page 18: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/18.jpg)
Impacts on health service utilization
Age range
Baseline level
Impact (% points)
Size of transfer
Colombia <24 months24-48
months
n.a.
n.a.
22.8**
33.2***
17%
Chile 0-6 years 17.6% 2.4 7%
Ecuador 3-7 years n.a. 2.7 10%
Honduras 0-3 years 44.0% 20.2*** 9%
Mexico 0-3 years 69.8% 8.4 20%
Nicaragua 0-3 55.4% 13.1* 27%
18
![Page 19: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/19.jpg)
Impacts on education and health outcomes
Only mixed success in terms of improving final outcomes in education and health:
Education: Increases in school enrollment and years of completed schooling have not come hand-in-hand with improved learning outcomes Health: Some programs, but my no means all, have improved child nutrition (as measured by height-for-age, hemoglobin status)
19
![Page 20: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/20.jpg)
20
Second generation evaluation questions
Role of conditions: Testing CCTs vs. UCTs (Burkina Faso, Malawi, Morocco and Yemen)
Does it matter who receives the cash? Mothers vs. fathers (Burkina Faso, Morocco and Yemen); Girls vs. parents in Malawi
Design of payments: Cambodia; Colombia
![Page 21: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/21.jpg)
Priority thematic areas
1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS Prevention7. Early Childhood Development
21
![Page 22: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/22.jpg)
Results-based financing (RBF) ≈ Pay-for-performance (P4P)
Provision of payment for the
attainment of well-defined results
Transfer of money or material goods conditional on taking a measureable action or achieving a predetermined
performance target (CGD, 2009)
DonorCentral governmentLocal governmentPrivate insurer
$Service providersFacilities / NGOsCentral governmentLocal governments
RBF takes many forms…
Payers Payees
22
![Page 23: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/23.jpg)
Impact evaluation of P4P in Health
Effects on non-contracted services provided?
Impact on equity of services?
Impact on out-of-pocket expenditures?
Impact Evaluations:
Will the introduction of pay for performance for patient services impact the quality and performance of hospitals/health centers?
23
![Page 24: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/24.jpg)
24
P4P in Rwanda
National P4P scheme to supplement input-based budgets with bonus payments based on the quantity and quality of maternal and child healthcare (14 output indicators)
![Page 25: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/25.jpg)
25
“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)
Impact on proportion of institutional deliveries
36.3
49.7
34.9
55.6
30.0
40.0
50.0
60.0
Baseline (2006) Follow up (2008)
Prop
ortio
n of o
f ins
tituti
onal
deliv
erie
s
Control facilities Treatment (PBF facilities)
7.3 % increasedue to PBF
![Page 26: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/26.jpg)
26
“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)
Impact on quality of prenatal care
-0.10
0
-0.13
0.15
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20
Baseline (2006) Follow up (2008)
Stan
dard
ized
Pren
atal
effo
rt sc
ore
Control facilities Treatment (PBF facilities)
15 % Standard deviation increase due to PBF
![Page 27: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/27.jpg)
27
P4P in Rwanda: Lessons
No effect on the number of prenatal care visits or on immunization rates. Need demand side incentives?
Greatest effect on services that had the highest payment rates and are under the provider control.
Financial performance incentives can improve both use of and quality of health services.
An equal amount of financial resources without the incentives would not have achieved the same gain in outcomes.
![Page 28: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/28.jpg)
28
P4P experiment in UgandaLundberg, Marek and Pariyo (2007) Contracting for Primary Health Care in Uganda
Three arms: (1) performance bonus, (2) freedom to allocate base grant, (3) control.
Performance bonus based on meeting targets (pre-natal care, attended births, immunizations, etc.)
Results after 18 months: bonus facilities perform the same as
or worse than others; freedom-to-allocate facilities perform
significantly better than others.
![Page 29: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/29.jpg)
29
P4P experiment in Uganda
This performance bonus didn’t work.Amounts not large enough? Scheme too complicated? Not enough time? (some evidence of
learning curve)
Facilities may allocate budgets more effectively than the Ministry of Health
![Page 30: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/30.jpg)
Priority thematic areas
1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS Prevention7. Early Childhood Development
30
![Page 31: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/31.jpg)
Strategies for improving learning outcomes
Supply side policies (classrooms, textbooks, extra tutors, teacher training)
Demand side policies (CCTs?) Child endowments/readiness to
learn -- ECD, de-worming Accountability reforms: Change the
environment in which decisions are made
31
![Page 32: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/32.jpg)
WDR 2004 framework of accountability
32
![Page 33: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/33.jpg)
33
Rationale for Education Accountability Schemes
How do we transform this teacher….
…into this teacher?
![Page 34: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/34.jpg)
School Accountability34
![Page 35: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/35.jpg)
35
3 Modalities on School Accountability
1. School-based Management (SBM)
2. Information for accountability projects
3. Teacher contracting and pay for performance projects
![Page 36: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/36.jpg)
36
How can SBM improve outcomes?
Those at the local level have more or better information and different objectivesKey decisions about school personnelKey decisions about spendingChanges in the educational processResource mobilizationChanges in the school climate
![Page 37: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/37.jpg)
37
School Based Management in Mexico
(Apoyo a la Gestión Escolar, AGE)
Grants + training to Parent Associations (PA).
AGEs increase school autonomy through improved mechanisms for participation of directors, teachers and PA in the management of the schools.
Effects on: School failure, repetition, drop out
Use phasing in of program for evaluation
![Page 38: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/38.jpg)
38
School Base Management in Mexico
Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”
![Page 39: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/39.jpg)
39
School Base Management in Mexico
Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”
![Page 40: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/40.jpg)
40
School Base Management in Mexico
Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”
![Page 41: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/41.jpg)
41
3 Modalities on School Accountability
1. School-based Management
2. Information for accountability (IFA)projects
3. Teacher contracting and pay for performance projects
![Page 42: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/42.jpg)
How could IFA improve outcomes?42
Empower parents and communities vis a vis local & school authorities – Increased importance of education in the political
sphere– Increased attention to results valued by parents
Enable better technocratic decision-making• Help guide allocation decisions
Under “strong accountability” systems:– Threat of sanction will spur increased teaching
effort When there is school choice:
– “Market-pressure” for performance
![Page 43: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/43.jpg)
Newspaper Campaign in Uganda
– In 1995 only 13 % of non-wage recurrent spending reached schools.
– From 1996-2001, amounts and dates of grant to local governments published in local and national newspapers.
43
![Page 44: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/44.jpg)
Impact of information campaign
Source: Reinikka and Svensson (2001), Reinikka and Svensson (2003a)
IMPACT ON LEAKAGE:A school close to a newspaper outlet experienced 25 percentage points less leakage compared to a school one standard-deviation (30 km) further away from a newspaper outlet.
IMPACT ON LEARNING:A one standard deviation reduction (30 km) in distance lead to a 0.09 standard deviation increase in average test scores.
44
![Page 45: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/45.jpg)
45
Information Campaigns in India
Village-level education committees (VECs), representing parents and village leaders, created by state education policy
Presumption that information campaigns are needed to empower parents/communities
Two sets of experiments PRATHAM --Banerjee et al. (2008) “Pitfalls of
participatory programs: Evidence from a Randomized Evaluation in Education in India”
Three states in India –WB + Nike Foundation --Pandey et al. (2008) Community Participation in Public Schools: Impact of Information Campaigns in three Indian States.
![Page 46: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/46.jpg)
46
Impact of Information Campaign (Banerjee et al.)
Program: Advocacy and information about VECs =(1); (1)+teaching how “learning reports cards” are done=(2); (2)+ Teaching: Capacity Building to Directly Improve
Learning (reading classes)=( 3)
Information campaigns ((1) and (2)) had no effect on VEC activism or on learning outcomes
Trained teaching volunteers (3) did have a significant effect on “reading” learning outcomes
![Page 47: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/47.jpg)
47
Impact of Information Campaign (Pandey et al.)
REGION Teacher Attendan
ce
Teacher
Activity
% Receiv
ed Unifor
m
Participation rate of parents in school inspections
Scholarships
Students received
Improvement in
Reading
UP 11 % Unchanged
18 % (girls)
.11 33 %(general caste)
27 %(Grade 3)
MP Unchanged 30% 14 % (general caste)
.02 26 %(SC/ SC)
14 %(Grade 3)
Karnataka
Unchanged Unchanged
N/A Unchanged N/A 15 %(Grade 4)
Pandey, Priyanka et al. Community Participation in Public Schools: Impact of Information Campaigns in three Indian States. South Asia Human Development. Presentation. World Bank. 2008.
47
![Page 48: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/48.jpg)
48
3 Modalities on School Accountability
1. School-based Management
2. Information for accountability projects
3. Teacher contracting and pay for performance projects
![Page 49: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/49.jpg)
49
How could TI improve outcomes?
Accreditation and merit-based incentives lead to better teaching professionals joining the system.
Decentralized hiring and monitoring of teacher performance leads to higher attendance rates.
Career advancement incentives stimulate sustained effort, improving student performance.
Pay-for-performance incentivizes improved quality and quantity of teaching improving student test scores.
![Page 50: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/50.jpg)
Teacher Pay Incentives in Andra Pradesh (Muralidharan and Sundararaman, 2008)
50
Three policies Group and individual pay incentives based on
average improvement in test scores of all students Block grants to schools (extra teachers, more inputs) Assessments conducted by an independent NGO
Results (two rounds) Incentives increase student performance by 0.22 SD 1st year both incentives work well. 2nd year individual
incentives work better Additional inputs improve scores only by 0.08 SD Teachers liked the program
![Page 51: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/51.jpg)
Priority thematic areas
1. Conditional Cash Transfers2. Paying For Performance in Health3. School Accountability4. Malaria Control5. Active Labor Market Programs / Youth
Employment6. HIV/AIDS7. Early Childhood Development
51
![Page 52: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/52.jpg)
52
Look for more information….
www.worldbank.org/hdchiefeconomist
![Page 53: USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS Overseas Development Institute May 4, 2010 ARIEL FISZBEIN CHIEF ECONOMIST,](https://reader031.vdocuments.site/reader031/viewer/2022013012/56649eb45503460f94bbcc2f/html5/thumbnails/53.jpg)
..and remember Primo Levi
‘It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’
53