Lecture 3

RANDOMISED EVALUATIONS

Agriculture Sector Dialogue Phase II

Overview

• What is randomization and how does it solve the causality problem?

• Choosing the level of randomization

• Ways to randomize

• When to do a randomized evaluation

• Case Study: Nudging farmers to use fertilizer

Africa Infrastructure Projects 2

What is randomization and how does it solve the causality problem?

Randomization and causation

• The challenge for impact evaluation is to “construct” a comparison group that mimics the group receiving the intervention

• In randomized evaluations: Participants and nonparticipants are chosen at random

• There is no reason, other than chance, that they are selected into the program

• On average, participants and nonparticipants have the same characteristics– they would, on average, have the same outcomes

• Any difference at the end is due to the program

Randomization creates groups with similar characteristics

Study sample

Treatment group 1

Comparison group

Treatment group 2

Random assignment vs. random sampling

• Random assignment– Units (people, schools etc.) are randomly assigned to different groups

(e.g. treatment and comparison)

– Creates two or more comparable groups

– Basis of randomized evaluation

• Random sampling– Want to measure the characteristics of a group (e.g. average height)

– Measure a random sample of the group

– Often used during randomized evaluations, especially group level randomization

Randomly samplefrom area to get study sample

Random sampling

Randomly samplefrom area to get study area

Randomly assignCommunities to treatmentand comparison

Random sampling and Random Assignment

Randomly sampleIndividuals to survey from both treatment and comparison

Steps in random assignment

• Define those eligible for a program

• Randomly assign which units are in the treatment and the comparison group

• Implement the policy or program for the treatment group

• Compare outcomes for those in treatment and comparison groups

Steps to randomization: 1

Steps to randomization: 2

Steps to randomization: 3

Steps to randomization: 4

Non-random assignment

HQMonthly income, per capita

1000

500

0Treatment Control

1457

947

Random assignment

Monthly income, per capita

1000

500

0Treatment Control

1257 1242

HQ

Key advantage of random assignment

Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment,

any difference that subsequently arises between them can be attributed to the program rather than to other factors.

16

Choosing the level of randomization

Unit of Randomization: Individual?

Unit of Randomization: Individual?

Unit of Randomization: Clusters?

Unit of Randomization: Class?

Unit of Randomization: Class?

Unit of Randomization: School?

Unit of Randomization: School?

Considerations for level of randomization

• Unit of measurement– Generally, best to randomize at the level at which the treatment is

administered

• Feasibility and fairness

• Spillovers

• Attrition

• Compliance

• Statistical power

• Clustering on the ground

Ways to randomise

What can be randomized?

• Different ways to randomize come from three basic elements of a program which can be randomized

• Access: we can choose which people are offered access to a program

• Timing: we can choose when people are offered access

• Encouragement: we can choose which people are given encouragement to participate

Simple lottery

• Most simple design

• Existing pool of potential participants

• Given number of slots

• Randomly assign potential participant to a treatment group or a control group

• Very useful and often the fairest way to allocate a program when resources are limited

Monthly income, per capita

1000

500

0Treatment Control

1257 1242

HQ

Simple Lottery

Randomization around a cut off (partial lottery)

• Sometimes a partner may not be willing to randomize among eligible people.

• Partner might be willing to randomize around a cut off or “at the margin”

• People “around the cut off” are people who are borderline in terms of eligibility– Just above the threshold not eligible, but almost

• What treatment effect do we measure? • The effect of the program for those around the cut off

Partial Lottery

Around the

cut off,

compare

treatment to

control

ParticipantsNon-participants

Treatment

Control

Phase-in design

• Everyone gets program eventually

• Natural approach when expanding program faces resource constraints

• What usually determines which schools, branches, etc. will be covered in which year?

Phase-in design

Round 3Treatment: 3/3

Control: 0 1

1

11

1

1

1

1

1

11

1

1

1

2

2

22

2

2

22

2

2

2

22

2

2

2

3

333

3

3

3

33

3

33

3

3

3 3

3

Round 1Treatment: 1/3

Control: 2/3

Round 2Treatment: 2/3

Control: 1/3

Randomized

evaluation ends

Encouragement design

• Sometimes a program is open to all and it’s impossible to randomize program access

• If there is low take up, there is still an opportunity to evaluate

• A treatment group can be created through randomly assigning individuals or groups to receive encouragement to take up the program

• Encouragement – e.g. phone call, text message reminder etc

• The idea is to increase the probability of take-up by the encouraged

• Outcomes you compare: encouraged group to the not encouraged group

When to do a randomized evaluation?

• When there is an important question you want/need to know the answer to that will affect many:– To establish impact and cost effectiveness (value for money)

• Timing – not too early and not too late

• Program is representative – not gold plated– Or tests a basic concept you need tested

• Time, expertise, and money to do it right

When NOT to do an RE

• When evaluating macro policies

• When it is unethical or politically unfeasible to deny a program to a control group.

• If the program is changing during the course of the experiment

• If the program under experimental conditions differs significantly from how it will be under normal conditions.

• If a RCT is too time-consuming or costly and therefore not cost-effective.

• If threats such as attrition and spill-over are too difficult to control for and hurt the integrity of the experiment

• If sample size is too small

Case Study: A Well-Timed Nudge

37

Duflo, Esther, Michael Kremer and Jonathan Robinson, 2011. "Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya" American Economic Review

Needs Assessment: Low Fertilizer Adoption

38

• Studies in Western Kenya showed that limited fertilizer use increased farmers’ annualized returns by 52-85 percent.

• But less than a third farmers reported using fertilizer in the previous two growing seasons.

• Question: If using fertilizer is profitable to farmers, why aren’t they investing in it?

Reasons for low fertilizer adoption?

39

• Limited access to cash or credit – Response: fertilizer subsidies

• Issues of timing and impatience– Farmers may have the money immediately after harvest

– But they do not need the fertilizer then so may not be motivated to buy it or pre-purchasing may be inconvenient

– Response: What happens if we make it easier for farmers to buy fertilizer after harvest when they have cash?

Program Evaluated

Savings and Fertilizer Initiative (SAFI) in Western Kenya

• In season 1 - Basic SAFI:

• Farmers visited immediately after harvest, and offered fertilizer voucher at market price, but with free delivery at a date of their choice.

• Farmers had to make a decision and purchase immediately

• In season 2, different variations of the program were tested including how a subsidy influences fertilizer adoption

40

Treatments in Season 2

41

Randomisation in season 2

• Sample Frame: built from parents of 5th and 6th grade children at 16 schools in Kenya’s Busia district

• Stratified sample (by school, class, participation in prior agricultural programmes/ treatments)

• Random assignment: Individual level - farmers

42

Balance

43

SAFI Season 2: Baseline Characteristics: Means

Comparison Basic SAFISAFI with

timing choice

Full price & free delivery, late

season50% subsidy late season

(0) (1) (2) (3) (4)

Income (1,000 Kenyan Shillings) 2.19 2.82 2.76 2.94 2.29

Years of Education of household head 7.47 6.96 6.84 7.09 7.12

Household had used fertilizer prior to season 1 0.53 0.41 * 0.40 * 0.45 0.37 *

Home has mud walls 0.88 0.88 0.88 0.90 0.86

Home has mud floor 0.87 0.82 0.87 0.89 0.85

Home has thatch roof 0.50 0.54 0.53 0.54 0.51

Observations 121 213 213 145 147

* Differences in means from comparison group significant at the 5% level

Source: Adapted from Table 3, Duflo, Esther, Michael Kremer and Jonathan Robinson, 2011.

"Nudging Farmers to Use Fertilizer: Theory and Experimental Evidence from Kenya" American Economic Review 101:2350-2390

Results

44

Results

45

• SAFI increased fertilizer adoption, but only for the duration of the program

• Free delivery during the growing season has, at most, a much smaller effect on fertilizer adoption

• The impact of SAFI was comparable to that of a 50% discount at fertilizer application time