OVE’s Experience with Impact (Treatment) EvaluationsPresentation prepared for DAC, 15th November 2006

Policy

The general evaluative questions proposed by the IDB’s ex post policy, approved in 2003, are (i) “…the extent to which the development objectives of IDB-financed projects have been attained.” and (ii) “… the efficiency with which those objectives have been attained” (para1.1 )

Policy left for practice: sampling, methodology, organizational framework, and the forum for the presentation of results.

Note the task is to evaluate already approved and/or closed projects ( average project time is about six years).

Implementation decisions

Project Sampling Strategy: Option: random or meta-evaluation. Decision :meta-evaluation.

Method and Project types: (i) process cum naïve or treatment (impact) evaluations. Decision Treatment effect evaluations; and (ii) projects with partial or national coverage. Decision partial coverage models

Organizational. Decisions: (I) separate activity within the office; (ii) evaluations to be carried out both in-house and outsourced.Therefore: (I) hired staff with appropriate expertise; and (ii) created EVALNET, a register of evaluators;

Forum for presenting results. Decision: overall report (sent to the Board) with background-working papers (discussed in ad hoc seminars).

Evaluative questions

what were the problems that the program was designed to tackle?

what was the policy response, i.e. the design features of the program? (theory based evaluation)

was the program of a sufficient size given the size of the problem(s)?

were the program’s deliverables provided in a cost efficient (and cost effective manner)?

What was the incidence and was the program well targeted?

what was the impact on welfare outcomes of the program?;and

what were the benefits relative to the cost of the program ?

What was the impact on welfare outcomes of the program?

To answer the question OVE normally use three approaches in the same evaluation:

(I) Naïve evaluation

(II) Regression based (cross-section and panel)

(III) Treatment effects

Social Investment Fund (naïve evaluations can be misleading

Profile: – Social Investment Fund. Panama – Basic Infrastructure to poor communities

Data: – Distribution of benefits by municipalities

from administrative data– Baseline and results of outcome

indicators from households surveys 1994-2001

Technique:– Treatment and comparison group using

PSM in double difference. The sample included 75 municipalities.

– Potential to work with a sample of more than 250 smaller geographic units but household survey was not representative at that level

Results:– Naïve evaluation: the program failed.

Impact evaluation: the program succeeded

Social Investment Fund - Panama

(4)

(3)

(2)

(1)

0

1

2

3

4

Naïve Impact

Pove

rty

Chan

ge (%

)

Labor Training Project (positive effects) Profile:

– Labor Training program – Dominican Republic

Data:– Simple randomization including a follow-

up survey done at 10-14 months after graduation from training

– 786 treated and 563 controls– Baseline has universe, follow up was a

stratified random sample (size determined by standard formulas)

Technique:– Estimated average Intention-to-treat on

treated by simple diff of means, verified with weighted diff and regression analysis (no DD b/c faulty baseline)

Results:– Employability, income and health

insurance access increased. Program succeeded

Labour Training - Dominican Republic

(4)

(2)

0

2

4

6

8

10

12

14

Employmentrate

Monthlyincome

Hours workedper week

Hourly wage Healthinsurance inprimary job

Gro

wth

(%)

Public Housing Program

Progressive Housing Phase I - Chile

(5) 0 5 10 15 20 25 30

Access to potable water

Sewerage connection

Electricity access

Overcrowding

Quality of the dwelling

Household Completeness

Health: Child undernourishment

Education: School attendance

Ocupation ratio

Indigence incidence

Poverty incidence

Materi

ality

Living

standa

rd

Absolute change (%)

Profile:– Progressive Housing Phase I – Chile– Provision of low cost basic dwellings to

poor families

Data: – Household Surveys identified

beneficiaries and applicants to the specific housing program

Technique:– Treatment from beneficiaries and

comparison from applicants using PSM. Single difference from a sample of 508 Beneficiaries and 476 applicants

Results:– Quality of dwellings improved– Little or not change in other welfare

outcome indicators. – Difference between naïve versus impact

Costs, benefits, and internal rate of return

Quality of dwellings by household income and Progressive Housing benefits - Chile

0

200

400

600

800

1,000

1,200

0.0 0.2 0.3 0.5 0.7 0.8 1.0

Quality of the dwellings (Composite Index)

Hous

ehold

Inco

me (A

vera

ge, M

onthl

y US$

)

All non-beneficiaries

PHP-I BeneficiariesImproving dwelling

quality

Be n

e fit

s

Profile:– Progressive Housing Phase I – Chile– Provision of low cost basic dwellings to

poor families

Data: – Household Surveys identified beneficiaries

and applicants to the specific housing program

Technique:– The benefits of the program are the

additional (necessary) household income required to obtain equivalent dwelling

Results:– IRR: greater than 18%– Benefits: Net present value per solution

~1150 US$

Rural Roads (decay of benefits over time) Profile:

– Rural Road – Peru– Construction and upgrade of roads in

rural areas

Data:– Specific survey of beneficiaries.

Baseline collected after program started. Follow-up survey 3 years after program closed

Technique: – single difference and double difference

Results:– Positive impact on income and assets’

values of rural households.– Decreasing impact for motorized roads

not for non-motorized roads.

Rural Roads - Peru

(20)

0

20

40

60

80

100

120

140

Short-term Medium-term Short-term Medium-term

Motorized Roads Non-motorized Roads

Abs

olut

e ch

ange

(US$

)

Per capita income per year

Per capita consumption per year

National Transfer Fund (dosage and multi-treatment effects)

Profile: – National Fund for Regional Development– Decentralized investment to finance

infrastructure and productive projects

Data:– Administrative data for distribution of benefits by

municipalities– Baseline and results of outcome indicators from

households surveys 1994-2001. The sample included 343 municipalities.

Technique:– Impact evaluation using PSM in double difference.

The municipalities grouped by per capita investment using cluster analysis.

Results:– Positive and increasing impact on poverty

incidence (reduction) on per capita investment – Not impact on poverty if investment is intensive

in education– Greater impact on welfare composite index in

municipalities with diversified investment

Accumulated impact by level of per capita investment - Chile

-20

-18

-16

-14

-12

-10

-8

-6

-4

-2

0

2 3 4 6 9 11 17 25

Per capita investment ratio respect control group

Ext

rem

e po

vert

y in

cide

nce

chan

ge (%

)

Impact by treatment type - FNDR Chile

(5)

(4)

(3)

(2)

(1)

0

1

2

3

4

Abs

olut

e ch

ange

Composite welfare indicator (Index)

Poverty Incidence (%)

Education intensive investment

Diversified investment

SCIENCE AND TECHNOLOGY: Research

Profile:– Science and Technology – Chile– Financing for R&D projects

Data:– All projects that between 1988 and 2004 received the financial

support of the program and a stratified sample of projects submitted to the program, which were not financed because they ranked below the threshold defined for being admitted to the financing.

– 2,936 different research projects (932 financed by the FONDECYT and 1704 not financed) 4,959 publications recoded in the ISI – SCI (1873 by financed researchers and 3806 by not financed researchers).

Technique:– Discontinuity regression design. The

selection process drawn by a “threshold” quality value that separates beneficiaries from non-beneficiaries

Results:– Unsuccessful. FONDECYT has no

significant positive impact on the scientific production of the financed projects.

01

02

03

0

0 500 1000 1500ranking

Rechazados AprobadosPredicion_rechazados Prediccion_aprobados

Technology Development Funds

Profile:Public grants-credits to firms for innovation

Data:

Administrative data on firms and firm level surveys (OSLO design)

Technique:

Double difference with propensity score matching

Results

Generally positive and significant effects on employment, and sales, but little evidence of effects on patents and total productivity .

Chile Argentina Brazil Employment +, * +, +, * Sales +, +, +, * Exports +, * -, na Productivity (TF) -, * -, -, Patents +, na +,* Crowding out 0,* 0, * +, *

EXPERIENCE: Findings Potable Water

Positive effect on health outcome (treatment less than naïve effect)

heterogeneity of results important. a regressive relationship between treatment effect and income, where more educated (and wealthier) households did better than less educated (and poorer) households

Ramification for project design: projects should include or be coordinated with, as a hypothesis to be tested, a health education component together with potable water expansion.

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

Bottom 25% 25%-50% 50%-75% Top 25%

Expenditure level

prop

ortio

nal c

hang

e

All Sample At least Primary

Impact on infant mortality

Balance

Since 2004 have produced about 23 evaluations

Cost per evaluation was about $60,000

But

Problem of obtaining effective counterparts (in Bank and country) to accompany the evaluation from beginning to end. Started outreach program to obtain formal counterparts in the country, and form ad hoc interested specialist for each thematic study.

Mainstream impact evaluations into other evaluations of the Office

Problem of communicating the findings. Started producing different reports for different audiences for the same evaluations.

Still far from the million words of a good picture

Before After

Regression Approach

Panel data

Cross section data

Where y is the outcome of interest, D is the dummy for participation in the program, V control variables

ij

m

j

jtiij

m

j

jtiijittiiiit LnVbLnVaDLnyaLny

,,,,1,,0

ij

m

j

jtiijitiit LnVaDaLny

,,,0