design of an impact study to evaluate the scaling up of ... of an impact study to evaluate the...

Design of an Impact Study to Evaluate the Scaling up of the WFP Voucher Scheme

Dr. Helen Guyatt, Head of Research

[email protected]

www.kimetrica.com

KIMETRICA presentation to WFP

Develop a set of analytical tools:

To inform on the scale up of vouchers, and help

WFP determine the most effective mix between

GFD and vouchers, given available resourcing.


Research objective

To investigate how the scaling up of the

voucher scheme will affect the net distribution

of costs and benefits for:

• refugees

• traders

• host community

With particular emphasis on:

• vulnerable groups

• effects on markets (esp. prices)

• influence of exogenous factors (e.g.

food cuts, security restrictions,

changes in camp population)


Roadmap


• Stakeholder interviews

• Secondary data analysis

• Theory of Change (ToC) definition

• Model development

• Definition of key research questions

• Key indicator and variable identification

• Evaluation of WFP’s proposed M&E plan

• Design of data collection instruments

• Data analysis plan


Research Design Process

Primary data analysis:

Stakeholder interviews


Stakeholder interviews

In Nairobi, Kakuma and Dadaab.

• Donors (Dfid, ECHO, USAID, Germany)

• FAO

• WFP officials

• Implementing Partners (World Vision,

CARE, NRC)

• UNHCR

• DRA

• County Government

• Traders in the camps


Key findings

• Benefits and risks may be unevenly distributed; voucher scheme may

marginalize vulnerable groups (e.g. HH size 1, new arrivals, HHs with no

phones, traders not in the camps or not in the scheme).

• Cereal component of the ration will affect resale behavior, demand and

prices for different cereal types in the market.

• Exogenous factors may influence costs, benefits and risks and should

be monitored (eg. ration cuts, security restrictions).

• Conflicts, particularly related to intra-HH decision-making, may increase.

• Commodity prices can change with season.

Kakuma and Dadaab are very different and should be considered separately


Need to

monitor

Main differences between Kakuma and Dadaab

Kakuma Dadaab

Culturally Multi-cultural Mostly Somali

Nutritional status Low Moderate

Markets Rely heavily on Kitale More developed

Transport and access Good road from Kitale Poor road from Garissa

Host community tensions High (resources) Low (business)


Secondary data analysis


Secondary data analysis

• Refugee demographic data

• Nutrition and health surveys

• Market monitoring

• Voucher documentation

• Cost data

• Other cash/voucher programmes

• Other activities in the camps

• Estimated elasticities of food items


UNHCR demographic data

Kakuma Dadaab

Total population 184,183 351,538

Number of HHs 54,836 83,277

HH size 1 31% na

Originating from Somalia 31% 95%

Arrivals (2014) 45,627 na

Departures (2014) 3,048 na


By sub-camp: new arrivals, Country of Origin, HH size 1, female headed HH?

• Food insecurity varies by camp and sub-

camp e.g. Kakuma >Dadaab, Ifo2 and

Kambioos.

• Food ration lasts < 11 days

• Phone access < in HHs with lower FCS

(38% phones in Kambioos)

• Price of minimum healthy food basket

(2013) higher in Kakuma


Annual nutrition surveys, FSOM and HIS

Market monitoring : retail prices

Missing: Fish, potatoes, pasta, sukuma, eggs? Wholesale prices? Suppliers?

May 2014 Sep 2014 Dec 2014

Rice 93 (78-120) [7] 95 (79-199) [5] 109 (100-119) [3]

Dry beans 81 (70-95) [7] 76 (38-98) [5] 78 (70-84) [3]

Goat meat 361 (220-400) [6] 398 (390-400) [4] 484 (480-488) [2]

Goat milk 67 (59-80) [4] 65 (60-70) [2] 75 (-) [1]

Tomato 110 (96-123) [6] 104 (48-138) [4] 99 (78,120) [2]

Bananas 102 (60-170) [5] 67 (20-120) [4] 88 (46,130) [2]


Vary by camp, sub-camp and season for meat and milk

Voucher programme documentation

Market assessments, FFV evaluation and FGDs

• 30% phone ownership in Kakuma 4

• Higher prices charged to FFV customers

• Host community shop in the camps

• Prices and availability seasonal

• Can traders meet increased cereal demand?


Cost data

• WFP costs are aggregated into composite unit

prices which include purchase, transport and

overheads

• Prices for each commodity in each camp is

assumed to be the same

• Unit prices are all expressed per metric tonne

distributed: this is a problem for some

resources (e.g. staff at FDP)


Other cash/voucher experiences in refugee camps

Somalia, Sudan, Burundi and Rwanda

• Traders need to be regularly monitored

• Markets suffer if a break in the voucher pipeline occurs

• Explore reductions in HH debt

• Reduce mobile phone loss by insisting on police reports

• Relative impacts on nutrition depend on baseline values


Other activities/surveys in the camps

Activity Details Will affect

WFP supplementary feeding MCHN, food4assets HH food security

UNHCR NFI targeting Female headed HH HH SE

HSNP2 cash targeting 40,000 HHs Turkana HH SE host community

FAO-Mastercard charcoal 8,000 refugees Kakuma HH income

NDMA Livelihood surveys Supplementary data

World Bank SE survey Supplementary data


Estimating elasticities for food supply and demand

To calibrate the model : farm gate, retail and wholesale levels

For retail demand elasticities, budget

share can be specified as:


Si =ai + bi lnM

P*

æ

èç

ö

ø÷+ g ij

j=1

n

å lnPj +ei

Where:

Si is the i th budget share estimated as

Si = PjXj/M

Pj are nominal retail prices

Xj are quantities

γij are price coefficients

M is the total expenditure on all goods

P* is an aggregate price index

FSOM HH expenditure not disaggregated

into quantities and unit price

Implications for the study design

HH survey : Expenditure on all relevant

commodities, disaggregated into quantity and unit

cost alongside HH demographic data

Market/trader survey : Prices on all relevant

commodities from all markets and sub-camps

Producer/farmer survey or secondary data : farm

prices, acreage, output and input prices (e.g.

labour)


Theory of change

Achieving impacts through outcomes and outputs


Assumptions

Refugees benefit if:

• Refugees can access and use vouchers

• Traders supply food demanded at acceptable quality and price

• Refugees purchase some non-GFD commodities with voucher


Traders benefit if:

• Refugees use markets as above (a risk that some traders may lose business)

• Traders can access MPESA agents and are not targeted by thieves

Major risk: Prices may increase

• As a result of limited supply or exogenous

factors

• Resulting in negative effects on HHs and

markets

• Potentially causing increased tension and

conflicts


• For porters and temporary staff at the FDPs

• Adversely affecting HH incomes for these

specific groups

Minor risk: Diminished employment opp.

Exogenous factors

• Food rations: Cuts and composition may affect HH spending behavior

• Insecurity: Curfews affect trading hours and transport costs, the closing

of the Somali border has increased prices, and the discontinuation of

money transferring modalities has affected remittances

• Other programmes: Receipt of other food, NFIs or cash aid should be

recorded

• Other: Weather conditions, wider market conditions


Model development


Cost and benefit trade-offs

Reduced GFD Increased Vouchers Net Benefit

Refugees GFD volume Unit price

Income Prices for all items

Probable net gain

Host consumers GFD commodity prices

Non-GFD commodity prices

Net loss

Traders GFD resale volume Non-GFD commodity sales

Probable net gain

Local producers Improved market for GFD commodities

Improved market for non-GFD commodities

Net gain



Anticipated changes

for GFD commodities

Increase in prices (P2)

Net shift in supply curve (S2)


Anticipated changes for

non - GFD commodities

Increased income shifts

demand (D2)

ST: supply fixed, and prices

increase (P2)

LT: quantity increases (Q2),

and prices stabilize (P3)


General equilibrium model

Markets (c): all sub-camps, “host community” rep,

“outside” rep.

Commodities (i): set of agricultural commodities

(e.g. maize, meat, fish etc.), transport service and

“other good”

Household types (h): in sub-camps differentiated

by eg. income, labor type, ethnic group

Production factors (f): e.g. labor, capital, land

Static model : calibrated to monthly or seasonal

data

One model for each camp: calibrated with camp-

specific data

Model objective

Compute price responses to

voucher injection and compare

with in-kind GFD


Consumption and prices

• Households: maximize their welfare subject to budget constraints

• Vouchers: modeled as imperfect substitutes to food commodities

• In-kind: modeled as additional local production, monetized using market

prices


Production and transport

• Goods: assumed to be purchased from cheapest source

• Agricultural commodities: produced by combining labour with capital

and/or land

• Transport services: provided by traders using labour and capital

Key research question

How will the voucher scheme

affect the net distribution of cost

and benefits as it scales up?


Distribution of costs and benefits: across all

groups (refugees, traders, host community)

and across income and voucher and food

market access strata

Impact on market growth: including up the

value chain to imported goods

Influence of exogenous factors

Risk of price increases: negatively affecting

refugees and host population


Components of net distribution

Gap Analysis

What are the key indicators?



Research Design Process

Indicator Measurement

HH food security Standard metrics (e.g. FCS, DD, daily food energy avail.)

HH consumption Patterns in markets and HHs

HH income Direct and indirect from HH expenditure

Trader income Direct and indirect from increased employment

Tensions & conflicts Police reports and intra-HH

Access to voucher Phone ownership, pipeline etc.

Access to food Prices & quality, resale of ration

Access to income No thefts, adequate supply chain


Key indicator identification

Variable/outcome Measurement

Exogenous factors Ration cuts, income losses, supply constraints

Costs Cash and opportunity costs, provider costs

Prices Commodities, price indexes and inflation

Supply Into camps and outlying host communities

Real income HH income deflated by price index

Welfare % change or monetized equivalent


Key variable identification

Can proposed WFP M&E provide this information?

Mystery shopping, trader feedback, FGDs ?

Tools Limitations

Beneficiary contact monitoring 20 per FDP

FSOM HH survey 139 HHs (Kakuma 1-3), HH size 1 = 2%

Voice monitoring telephone 150/month, but need a phone

FSOM market monitoring Missing items, wholesale prices and sub-camps

MVAM-SMS 500/week, but need a phone


Data collection and analysis


1. HH survey

2. Trader survey

3. MM survey

HH Survey

Section Details

Interview and HH details Sub-camp, HHH, CoO, arrival, relatives

Wealth indicators Housing, cooking, electricity, assets

HH roster HH profile, age, sex, schooling, other nutrition prog.

Food consumption Quantities and changes in demand with price

Expenditure Food, NFIs, capital

GFD collection Resale, duration, opp. costs

Voucher Purchasing behavior, opp. costs

Coping strategies/income CSI, income (work, gifts), debts


Proposed HH survey sampling

400 refugee HHs to be randomly sampled in

each sub-camp

1. UNHCR list of all HHs with location

2. Sampling grid over each sub-camp and

the start at center and move outwards


Shortened HH survey: wealth and consumption in

a random sample of “host community” in the main

town (100) and in the next nearest town down the

supply chain (100).

Trader Survey

Section Details

Trader details CoO, location, type of shop

Voucher experience Use and opportunity costs

Main food traded Demand and prices, suppliers

Employees Number, wages

Assets Transport and stores


Trader survey

400 traders randomly selected across the camps – from list?

50 traders from the main town outside the camps.

Concurrent with HH survey

Builds on FFV


Market monitoring

Every market in every sub-camp, main town,

and next market town. Conducted monthly.

• Retail and wholesale prices

• Comprehensive list of commodities incl.

eggs, fish, pasta and potatoes.

• Demand, availability and main suppliers


Cost data

Provider costs (WFP) : voucher versus in-kind

• Unit costs and quantities explicitly

• Itemized frameworks

• Set-up versus recurrent costs

• Fixed versus variable costs

• By activity: procurement, sensitization,

training, monitoring, distribution

• By resources: staff, capital, rental,

consumables etc.


Supporting data

• Triangulation with WFP M&E framework

• Triangulation with other secondary data sources

– e.g. KHIBS, the World Bank SE survey in Kakuma and the NDMA

• Systematic monitoring of conflicts using police reports

• KI with local producers: prices, inputs and outputs

• Qualitative data

– Regular FGDs with host community, refugees and traders every 6 months

– Security restrictions, adverse conditions affecting supply of commodities,

changes in the camp population and GFD composition and ration cuts


Data analysis plan

Combine programme cost data with model outcomes on percentage

change in welfare (or its monetized equivalent) for different HH types.

• Food security: FCS, DD, daily food energy availability per capita

• Welfare estimates : based on commodities consumed (quantities and values)

and HH income.

• Model benchmark values: consumption, production and transport

• Model response parameters eg. Elasticiity of substitution between goods


Develop a set of analytical tools:

To inform on the scale up of vouchers, and help

WFP determine the most effective mix between

GFD and vouchers, given available resourcing.


Under what conditions (e.g. critical price level) should vouchers be scaled back?

Thank you

[email protected]

www.Kimetrica.com


design of an impact study to evaluate the scaling up of ... of an impact study to evaluate the...

Documents