DIME – FRAGILE STATESDUBAI, MAY 31 – JUNE 4

Making the Most out of Discontinuities

Presented byMalte Lierl (Yale University)

How do we measure program impact when random assignment is not possible ? e.g. universal take-up non-excludable intervention treatment already assigned

Solutions Make assumptions about what constitutes a plausible

control group (matching on observables, diff-in-diff) Exploit quasi-random aspects of program

implementation Quasi-experiments

Example: Regression Discontinuity Design (RDD)

Introduction

Discontinuity = Arbitrarily placed cutoff for program eligibility

Regression Discontinuity Design (RDD)

vulnerability index

income

Around the cutoff, beneficiary (‘treated’) and non-beneficiary (‘untreated’) populations are very similar.

For the population around the cutoff, RDD can be as credible as a randomized experiment.

vulnerability index

income PROGRAM

IMPACT

CUTOFF

RDD: Some examples

Example 1: Evaluate reintegration assistance for former child soldiers aged 16 and below.

An ex-combatant aged 16 years and one day would not benefit from the program.

RDD would compare individuals just above and just below 16 years of age.

RDD: Some examplesExample 2:

If you are elected into parliament, will this make you wealthier?

Can’t randomize who gets into parliament. In majoritarian systems such as in the UK, you

get into parliament if you have the majority of votes in a district.

Some districts have very close election results. Between two candidates with 49.5% and 50.5%

of votes it is as good as random who gets into parliament.

RDD: compares winners and losers in very close runoffs.

Another RDD exampleExample 3:

Minimum legal drinking age in the United States is 21

It is illegal to sell alcohol to people younger than 21

People aged 21 and people aged 20, 11 months, 29 days are treated very differently under the drinking age policy

But they are not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc.)

What is the effect of alcohol on mortality rates? In effect, the minimum drinking age

assigns people into ‘treatment’ and ‘comparison groups’ Treatment group: People between ages

20 years and 11 months and 20 years 11 months and 29 days cannot drink alcohol.

Comparison group: People just above 21 can drink.

Both groups should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates).

If we use the drinking age cutoff as RDD, we can estimate the causal impact of alcohol consumption on mortality rates among young adults.

What is the effect of alcohol on mortality rates?

Source: Carpenter & Dubkin, 2009

RDD

Proportion of days drinking, by age

Increased alcohol consumption causes higher mortality rates around the age of 21

All deaths

All deaths associated with injuries, alcohol or drug use

All other deaths

RDD

What is the effect of alcohol on mortality rates?

Death rates, by age

Source: Carpenter & Dubkin, 2009

Internal Validity If the cutoff is arbitrary:

Individuals directly above and below the cutoff should be very similar in expectation

Systematic differences in outcomes are caused by the policy

Major assumptions: Individuals have no precise control over

assignment variable Nothing else is happening. In absence of the

policy, we would not observe a discontinuity around the cutoff.

Might not be the case if: ▪ Drinking age is 18, and driving also becomes legal at

age 18 ▪ Another program provides reintegration assistance for

ex-combatants over 16 years.

RDD Requirements

Transparency and precise knowledge of the selection process

‘Treatment’ is discontinuous with respect to an assignment variable

Individuals cannot precisely manipulate the assignment variable

All other factors are continuous with respect to the assignment variable (“nothing else is happening”)

Enough data points around the cutoff

Sharp discontinuity Discontinuity precisely determines treatment status

▪ All people 21 and older drink alcohol and no one else does

▪ All ex-combatants younger than 16 receive assistance, nobody else does

Fuzzy discontinuity Percentage of participants changes discontinuously at cut-

off, but not from 0% to 100% (or from 100% to 0%)▪ Some people younger than 21 end up consuming

alcohol and/or some older than 21 don’t consume at all▪ Some youth ex-combatants under 16 don’t participate,

and their slots are given to others who are just over 16.

Sharp and Fuzzy RDDs

Sharp and Fuzzy RDDs

1

0

1

0assignment variable

assignment variable

Probability of being treated

Probability of being treated

SHARP DISCONTINUITY

FUZZY DISCONTINUITY

External validity

Are RDD estimates of program impact generalizable?

Counterfactual/control group in RDD: Individuals marginally excluded from benefits Examples: Ex-combatants over 16, candidates

with 49.5% of votes Causal interpretation is limited to

individuals/households/villages near the cutoff Extrapolation beyond this group needs

additional (often unwarranted assumptions) Or multiple cutoffs!

RDD Implementation

Data collection: Make sure to have enough observations around the cutoff

Analysis: Observations away from the cutoff should have less weight

Why? Only near the cutoff can we assume that people find themselves to the left and to the right of the cut-off by chance.

weight

assignment variable

outcome

RDD Implementation

Carefully justify study design Baseline data will be useful to verify

assumptions

assignment variable

outcome

assignment variable

outcome

BEFORE PROGRAM

AFTER PROGRAM

RDD Implementation

Carefully justify study design Graphical analysis is an important tool

assignment variable

outcome

Summary

Advantages of RDDs: RDD can be applied even when

randomization is not feasible ▪ e.g. to programs with means tests for

eligibility For the population around the cutoff,

RDD is as credible as a randomized experiment▪ Requires fewer assumptions than other non-

experimental methods RDD can be used like a ‘natural

experiment’ to evaluate a program ex-post

Summary

Drawbacks of RDDs: Limited external validity: The estimates

of program effects are informative only for the population around the cutoff.

RDD requires a lot of data around the cutoff

Knowledge about the cutoff may induce behavioral change that can bias your evaluation▪ e.g. ex-combatants misreport their age▪ e.g. candidates become frustrated because

they were ‘so close’ to getting elected

شكرا 20

Thank you!

Further reading:

Lee, David and Thomas Lemieux (2009): Regression Discontinuity Designs in Economics, NBER Working Paper No. 14723. http://www.nber.org/papers/w14723