Impact Evaluations and Social Innovation in Europe

Bratislava, 15 December, 2011Joost de Laat (PhD)Human Development EconomicsEurope and Central Asia The World BankComments: jdelaat@worldbank.org

PROGRESS 15 December 2011 Deadline

Inputs Activities Outputs Impacts on Outcomes

Finance

State budget

European Social Fund  

Human resources

Min. of Labor

Public employment offices

Private/public training providers

Project preparation activities (4 months)

Identify 2000 long-term (at least 2 years) unemployed people interested in training program

Identify 10 private / public training providers

Design monitoring database

Poject implementation activities (1 year).

Offer training vouchers to 2000 unemployed

Training institutes provide 3 months skills trainings to participating unemployed

Enter data on training participation and job placement into database

Improved labor market skills of current long-term unemployed

Improved employment rates of current long-term unemployed

Improved wage rates of current long-term unemployed

Reduced poverty of current long-term unemployed

Greater education enrolment among children of current long-term enrolment

Improved health outcomes among children of current long-term unemployed.

Project preparation outputs

2000 long term unemployed identified

Contracts signed with 10 private/public training providers

Monitoring database in place

Project implementation

2000 vouchers allocated

Est. 1400 long term unemployed accept voucher and enlist in training

Est. 1000 long term unemployed complete training

Est. 600 long term unemployed find jobs following training

Database on participants

Outline

What?

Who?

How?

Ethics?

Why?

Impact Evaluations

?

What?

Isolates causal impact on beneficiary outcomes

Globally hundreds of randomized impact evaluations• Canadian self-sufficiency welfare program• Danish employment programs• Turkish employment program• India remedial education program• Kenya school deworming program• Mexican conditional cash transfer program

(PROGRESA)• United States youth development programs

Different from:e.g. evaluation measuring whether social assistance is reaching the poorest households

Who?

Often coalitions of:• Governments • International organizations• Academics• NGOs• Private sector companies

Examples:• Poverty Action Lab (academic)• Mathematica Policy Research (private)• Development Innovations Ventures (USAID)• International Initiative for Impact Eval. (3ie)• WB Development IMpact Evaluation (DIME)

7Ex 1: Development Innovation Venture (USAID)

Ex 2: International Initiative for Impact Evaluation (3ie)

9Ex 3: MIT’s Poverty Action Lab (JPAL)

10Ex 4: WB Development IMpact Evaluation (DIME)

11Make publicly available training materials in partnership with other WB groups (e.g. Spanish Impact Evaluation Fund)

12

Location Date Countries Attending Participants Project Teams

El Cairo, Egypt January 13-17, 2008 12 164 17

Managua, Nicaragua March 3-7, 2008 11 104 15

Madrid, Spain June 23-27, 2008 1 184 9

Manila, Philippines December 1-5, 2008 6 137 16

Lima, Peru January 26-30, 2009 9 184 18

Amman, Jordan March 8-12, 2009 9 206 17

Beijing, China July 20-24, 2009 1 212 12

Sarajevo, Bosnia September 21-25, 2009 17 115 12Cape Town, South Africa December 7-11, 2009 14 106 12

Kathmandu, Nepal February 22-26, 2010 6 118 15

Total 86 1,530 143Organize trainings on impact evaluations in partnership with others (e.g. Spanish Impact Evaluation Fund)

Impact Evaluation Clusters

• Conditional Cash Transfers• Early Childhood Development• Education Service Delivery• HIV/AIDS Treatment and Prevention• Local Development• Malaria Control• Pay-for-Performance in Health• Rural Roads• Rural Electrification• Urban Upgrading• ALMP and Youth Employment

Help coordinate impact evaluations portfolio

How to carry one out?

Basic Elements

• Comparison group that is identical at start of program

• Prospective: evaluation needs to be built into design from start

• Randomized evaluations generally most rigorous• Example: randomize phase-in (who goes

first?)• Qualitative information – helps program

design and understanding of the 'why'

Ethics?

Implementation considerations• Most programs cannot reach all:

randomization provides each potential beneficiary fair chance of receiving program (early)

• Review by ethical review boards

Broader considerations• Important welfare implications of not

spending resources effectively

• Is the program very beneficial? If we know the answer, there is no need for an IE

Why?

EU2020 Targets (selected)• 75% of the 20-64 year-olds to be employed • Reducing school drop-out rates below 10% • At least 20 million fewer people in or at risk of

poverty and social exclusion

Policy Options Are Many• Different ALMPs, trainings, pension rules,

incentives for men taking on more home care etc. etc.

• For each policy options, also different intensities, ways of delivery…

Why?

Selective Use of Impact evaluations• Help provide answers to program

effectiveness and design in EU2020 areas facing some of the greatest and most difficult social challenges

But impact evaluations can also• Build public support for proven programs• Encourage program designers (govts, ngos,

etc.) to focus more on program results• Provide incentive to academia to focus

energies on most pressing social issues like Roma inclusion!

Why? Help Encourage Social Innovation

+

Thank you for your attention!