dime workshop - world banksiteresources.worldbank.org/intdevimpevaini/resources/3998199... · why...

OCTOBER 13-17, 2014 | LISBON, PORTUGAL

DIME WORKSHOP

1

tinyurl.com/dime-lisbon

#IEKnowEE

http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment











Operational Lessons Learned from

Implementing IEs Chair: Guadalupe Bedoya, DIME Speakers: Jed Friedman, World Bank Caroline Cogueto, Sao Paulo State Environment Secretariat Susumu Yoshida, DIME Maria Jones, DIME


DIME WORKSHOP

3


#IEKnowEE












Ensuring Buy-in: Jed Friedman

World Bank

Take-up: Carolina Cogueto

Sao Paulo State Environment

Secretariat

Susumu Yoshida

DIME

Data Quality: Maria Jones

DIME

Operational Lessons from

Implementing IEs

Why do we care about Implementation?

More often than not you will face challenges in the implementation of your IE

More importantly… if implementation is different

than planned you may be not answering your IE question(s)

Case 1: Take-up

• Intended IE Question: What is the impact of connecting households to the electrical grid on income of rural households without electricity?

• …But only 30% of your treatment group actually connected to the electrical grid

Impact Estimate= 5%

(ITT)

Income in the Treatment

Group

Income in the Control Group

• Answered IE Question? What is the impact of offering access to the electrical grid on income of rural households without electricity?

70% w/o treatment

Case 2: Quality of treatment

• Intended IE Question: What is the impact of a quality financial literacy (FL) program on FL knowledge?

• …But teachers deliver only 50% of the curriculum (which may not really have an effect…)

Impact Estimate

= 0

FL knowledge in the

Treatment Group

FL knowledge in the

Control Group

• Answered IE Question? What is the impact of a low-quality financial literacy program on FL knowledge ?

So, changes in the implementation and bad measures of the outcomes

may lead to answering questions that…

– Are not the intended questions – May be not policy relevant or even interesting – May be misleading if not interpreted correctly – May be incorrect (bad data)

Very important to know what to monitor and plan for it…

Activities

Sessions with

children and/or

parents on

early education

(# of sessions,

# of activities)

Meals and

nutrition

supplements are

distributed to

children

(# meals per day

and supplements

distributed,

nutritional value)

Outputs

Children

/parents

attending

sessions

(# of days/hours

attended, # of

activities

performed)

Children receive

and take meals

and supplements

(# meals /

supplements

received, taken)

Short-term and

Intermediate outcomes

Parents behavior

and attitudes

toward early

childhood

development

change

(measures of

behavior, awareness,

attitudes…)

Children are healthy and

developmentally ready

for school (cognitive,

physical and

psychosocial tests)

Children perform

better in life

(level of

education, test

scores, rates of

criminality,

wages, etc…)

Long term

outcomes/Goal Components

Early

Education

1

Nutrition

2

Results Chain (Early Childhood Intervention)

Does this happen as planned? • Buy-in • Implementation monitoring

Does your data “says” what it’s supposed to?

Implementation/Process Evaluation

• Monitors program’s delivery and quality over time. • Usually includes measures of outputs and some qualified

outcomes. 1. Helps understand and interpret impact results 2. Helps improve program performance during program

operations.

• Who is served? Is the program serving the intended group?

• What is the program delivering? Is this what the program intended?


DIME WORKSHOP

12


#IEKnowEE












Operational Lessons Learned: Ensuring Buy-In

Jed Friedman, Development Research Group

The World Bank

October 15, 2014

Why do we evaluate?

• The first objective: – “What is the causal effect of intervention X on priority outcomes Y”

• Who wouldn’t want to know this? – Helps allocate resources in an efficient manner

• But…there can be many obstacles to stakeholder buy-in to evaluation – consequent hesitancy to evaluate even though many people may ultimately

benefit

• A great deal of experience in room on how to improve buy-in, but I’ll discuss three factors that I have seen help – At times IE designs need to be modified to accommodate systemic or

political realities

– Maximize external validity – a good practice anyway, also helps buy-in

– Continual need to stress that IE is a learning exercise, i.e. a tool used by prudent system managers to benefit the principal stakeholders

IE design and buy-in

• The RCT is considered a gold-standard approach in evaluative research, and for good reasons – Minimal (and largely verifiable) assumptions necessary for causal inference

– Research design (and results) easily communicated to policy makers and public

• BUT in some contexts: – Disquiet among implementers to differentially treat units under their purview

– Geographic proximity of treated and control units can create unanticipated spillovers: disgruntled or transferred service providers, innovation imitation, population movement, etc.

– Spotlight effects: Perhaps system leadership provides experimental facilities with extra attention and resources

• So: where treatment units are not subject to spill-overs or spotlight effects then by all means advocate for RCT as first-best option

• But if we suspect confounders, we need another strategy…

The challenge of clustered studies when the number of clusters is low

• Typically the alternative strategy to an RCT will involve implementation at a more aggregate level, i.e. a district

• Outcome assessed at clinic or household level while randomization at district level still constitutes a viable IE design

• The problem is that there may be relatively few districts in the study, so risk of underpowered design

• Solutions

1. Leverage additional assumptions of quasi-experimental estimator such as district matching to increase power

2. Consider Fisher-exact standard errors with one-sided hypothesis testing to increase power

External validity I

• The first objective:

– “What is the causal effect of intervention X on priority outcomes Y”

• But in reality, any single impact evaluation answers the

modified question:

– “What is the causal effect of intervention X on priority outcomes Y in

study setting Z”

• We want to abstract away from specific setting Z to inform

broader policy “at scale”

Often program evaluation lessons are difficult to generalize

• Evaluation of contract teachers in Kenya

– Bold et al (2012) find positive effect on test scores in schools randomly assigned to NGO implementation, but not in schools assigned to government

• Evaluation of anti-malaria mobilization programs in India

– Das, Friedman, and Kandpal (2014) find gains in malaria net usage and prompt care-seeking in regions assigned to some NGOs but not others

• Evaluation of energy experiments in US

– Conducted by one company in 14 cities find that impacts vary by location and characteristics of local partner (Alcott and Mullainathan, 2012)

External validity II

• Policy makers and practitioners have an implicit understanding

of the challenges of external validity

– “How can I believe the study results apply to the country as a whole?”

• Need to design the IE to maximize generalizability

– And then reassure stakeholders that this is the case

Broad geographic coverage facilitates external validity

• In Zimbabwe, MoH with strong desire to pilot new health program in every

province. For policy learning, selected two districts “representative” for

that province.

The challenge of differential access to treatment

• Resistance to the inclusion of a leave-out group

– We can all think of examples where practitioners are hesitant to

introduce a program in only part of a population

• This can be a tricky issue – we need to make sure there is a

real need for learning to justify a leave-out group

• But, if there is a real need, then cannot stress enough the

importance of learning for stakeholder benefit

– For-profit companies spend substantial resources to assess their

operations and improve efficiency

– Responsible systems stewardship also involves assessment of

effectiveness

The frame of learning: The case of health care quality in country W

• Government of W wanted to incentivize quality measures in primary health clinics

– Substantial money allocated for program ($20 million in first year) …

– … but many questions over what measures to incentivize, and how much to pay

• At the same time, much resistance to an experimental approach to evaluation

– “We do not experiment on our people”

– Recognized need to learn but aversion to the notion of evaluation

• Solution: there would be no experimentation, or even IE, but there would be selected “learning” clinics to receive the first phase of program

– There would also be no “control” facilities, but we can collect information in non-learning facilities to supplement the learning

Increasing buy-in though transparency: Public randomization

• Kyrgyzstan is implementing a hospital reform program on a pilot basis

– There are 60 maternal hospitals in the country, but pilot implementation funds sufficient for only 20 hospitals

• Every hospital wants to participate (participants receive extra resources)

– In principle, hospital directors accept randomization process as fair

– But don’t trust ministry officials with the process

• Solution: the investigators conducted a public randomization ceremony where (a) process is explained and (b) hospitals selected

– Witnessed by hospital representatives and media

– Results accepted by all

Five “guiding principles” to enhance buy-in

• Generalizable to scale – typically need to replicate successful interventions on a wider scale than study context

• Effectiveness studies with operational orientation – evaluations of programs implemented within existing system capabilities oriented towards learning

• Design regular feed-back to implementers – process monitoring validates the program model and identifies areas that need improvement

• Include a cost focus – to inform policy choices and trade-offs

• Local ownership – core involvement of government and local

investigators in key decisions critical both for effective implementation as well as policy adoption

Supplementing design with process evaluation aids buy-in

• What is process evaluation?

– Documents and describes how the program operates, the services it

delivers, and assesses whether it functions as implemented

– High frequency data gives rapid feedback to serve as input for mid-

course correction and program learning

– Will help explain why we observe success or failure

– Can be either quantitative methods, qualitative, or mixed methods

• Process evaluation contributes to buy-in because it offers

implementers useful operational knowledge


DIME WORKSHOP

26


#IEKnowEE












Lições operacionais: Adesão ao projeto

Caroline Cogueto, Secretaria de Meio Ambiente de São Paulo

October 15, 2014

DIME Workshop Lisbon, Portugal

PSA - Mina d'Água

• Projeto de Pagamentos por Serviços Ambientais "Mina d'Água"

• Pagamento em dinheiro a proprietários rurais de bacias prioritárias que se comprometam a preservar ou restaurar nascentes de água

• Pagamento varia de acordo com a importância das nascentes no manancial e seu estado de conservação

Interesse

Porque achamos que haveria bastante interesse:

• Eram áreas que já deveriam ser mantidas sem atividade agropecuária por lei

• O custo de oportunidade da terra é pequeno

– Por exemplo, pasto rende R$300 ano/ha

Avaliação de Impacto 1.0

• O PSA faz com que as nascentes sejam conservadas ou recuperadas?

– Imagens de satélite/fotos aéreas ou vistorias, comparação entre a linha de base (2010) e cinco anos depois (2015)

• O PSA muda a percepção ambiental e o comportamento do proprietário rural?

– Questionários aplicados antes e depois do projeto.

Projeto piloto: 21 municípios

Avaliação de Impacto • Quatro municípios

– No máximo 150 nascentes por município

• Plano A >150 interessados: Sorteio para escolher participantes (aleatorização)

• Plano B <150 interessados:

Regression Discontinuity

Cadastramento de elegíveis e interessados

Guapiara Ibiúna

Adesão

• Edital lançado: início 2012

• Linha de base: início 2013

• Primeiros contratos assinados: Meio 2014

MUNICÍPIO

NÚMERO DE

PRODUTORES

PRODUTORES

INTERESSADOS

CONTRATOS

ASSINADOS

Guapiara 125 53 12

Ibiuna 130 76 8

7 municípios COM EDITAL LANÇADO 14

12 municípios SEM EDITAL LANÇADO

Gargalos • Arranjo institucional muito caro e trabalhoso

• Desconhecimento e mudança da Lei florestal

Gargalos

• Pré-requisitos para participação: – Possuir matrícula (documento de comprovação da

propriedade);

– Não inadimplentes com o Estado;

– Comprometer-se a realizar a adequação ambiental de toda a propriedade (sem saber muito bem o que isso significava…);

• Desconfiança em relação à ação do governo

Lições aprendidas

• Simplificar arranjo institucional

• Simplificar pré-requisitos de participação

• Melhorar informação sobre as responsabilidades assumidas em aderir ao projeto

Here we are again – IE 2.0: testar como aumentar adesão no projeto para entender se são essas realmente as barreiras de participação

More on Friday…


DIME WORKSHOP

38


#IEKnowEE












Take up Challenge

Susumu Yoshida

Oct. 15, 2014

DIME Workshop, Lisbon

Karaoke intervention

• PDO: Have fun with workshop participants

• Component: karaoke session

• Assumption: Everybody likes “Karaoke”

• But…. it failed. Why?

did not have enough participants. Low Take Up

What could I do?

Incentives

1. Non-monetary: Autograph + photo

2. Recognition: Name/Photo on DIME website

3. Monetary: $100? $200?

Best singer DIME Workshop Lisbon, 2014

Water and Sanitation Project in Kenya

Project: $427 million Water/Sanitation infra. upgrading

Target: Kayole Soweto, Nairobi: 85,000 pp (2200 compounds)

No Hygiene Campaign

Hygiene Campaign

Low Subsidy (Standard subsidy)

C (600)

---

Medium Subsidy T1 (400)

T3 (400)

High Subsidy T2 (400)

T4 (400)


DIME WORKSHOP

44


#IEKnowEE












IE Data: Quality Matters!

Maria Jones

14 October 2014

DIME Workshop, Lisbon, Portugal

IE Theory of Change?

Rigorous IE Design

Actionable evidence

IE Success

• Success of IE depends on buy-in and take-up

• But even an IE with a rigorous design, full support from government, and 100% compliance in the field, IE can be worthless

• Need sufficient amount of high-quality data to accurately and precisely determine impact

– Sufficient amount: tomorrow

– High quality: today

IE Design

Buy- in

Base line

survey

Mid line

survey

End line

survey

IE analysis & report

Policy decision

IE Timeline

Project implementation

True or false?

“We signed a contract with a firm for a baseline survey. Finally we’ll have some time for other

work, no need to think about the baseline again until we receive the firm’s deliverable: the final

baseline report.”

what could go wrong?

What could go wrong?

• Wrong HHs interviewed

• Survey instrument missing key indicators

• Massive attrition in panel surveys

• No identification information collected, making follow-up surveys impossible

• High rates of non-response on key questions

• Inconsistent or out-of-range responses

What to do?

• Plan sufficient time for survey preparation

• Develop careful, detailed, TOR for firm

• Monitor field work

• Independent audit

• Collect data electronically

Plan sufficiently

Doing a HH survey well takes a lot of time!

Design Questionnaire (1 - 2 mo)

Procure survey firm (3 – 5 mo)

Develop electronic quest (2 mo)

Pilot survey & confirm data (1 mo)

Collect data (2 – 4 months)

Enter data (2 – 4 months)

Clean data (2 – 3 mo) Analyze data (2 – 3 mo)

Develop careful TORs

• Rigorous testing of survey instrument (especially if electronic survey) and submission of pilot data

• Regular submission of raw data (daily or weekly); test for data quality

• Regular submission of logbooks; set minimum response rate with penalties / rewards

• Submission of complete raw dataset at the end of the survey

• Sufficient information (geo-data) at baseline to track HH in future survey rounds

• Beware: ‘too cheap to be true’

Monitor, monitor, monitor!

• IE Field Coordinator works closely with survey firm every step of the process – Participates in enumerator training

– Observes interviewers

– Monitors fieldwork (unplanned visits)

– Checks data quality & provides feedback

• Best practice: independent audits – small sub-sample

– Confirm interview took place

– Check key data points (especially triggering skips)

Collect data electronically

• Potential to increase data quality a lot – Range & logical checks

– Pre-loading of respondent information

– Skip patterns and interview flow enforced

• Speeds up process – data available as soon as

survey is completed

How much is this going to cost?

• Varies a lot

• Depends on:

– Country (wages, fuel, cost of living)

– Quality and competitiveness of survey firms

– Sampling unit & sample size

– Geographic dispersion of sample

• Per-interview cost for multi-modal agricultural HH survey range from $20 - $150

Integrate IE and M&E data

• Data collection time-consuming and costly

• Do not duplicate efforts!

• Think ahead of time how to use IE surveys to meet M&E needs

• Insure questionnaire covers key results framework indicators

• Develop MIS to link all monitoring, survey, and administrative data

Take-aways

• Do not let bad data be the downfall of your IE!

• Collecting high-quality data is not easy

• But research team will help

– Questionnaire design, data cleaning and analysis

– Field coordinator supports in country, closely involved in survey supervision, builds capacity in your team as needed


DIME WORKSHOP

60


#IEKnowEE












Voting Session

61

2 questions, followed by a 15 minute discussion

1. Which presentation is most directly relevant to your project?

2. Which, if any, of the presentations

introduced a new idea that you think is worth exploring in the context of your project?

Now, let´s vote!

Which presentation is most directly relevant to your project?

1. Ensuring buy-in

2. Take up

3. Data quality

Which, if any, of the presentations introduced a new idea that you think is worth exploring in the context of your project?

1. Ensuring buy-in

2. Take up

3. Data quality

Results

100

100

100

100

100

100

Ensuring bu...

Take up

Data quality

Results

Most directly relevant

Ideas to be explored