Impact Evaluation MethodsRegression Discontinuity Design and

Difference in Differences

Slides by Paul J. Gertler & Sebastian Martinez

2

Measuring Impact

• Experimental design/randomization

• Quasi-experiments

– Regression Discontinuity

– Double differences (diff in diff)

– Other options

3

Case 4: Regression Discontinuity

• Assignment to treatment is based on a clearly defined index or parameter with a known cutoff for eligibility

• RD is possible when units can be ordered along a quantifiable dimension which is systematically related to the assignment of treatment

• The effect is measured at the discontinuity – estimated impact around the cutoff may not generalize to entire population

4

• Anti-poverty programs targeted to households below a given poverty index

• Pension programs targeted to population above a certain age

• Scholarships targeted to students with high scores on standardized test

• CDD Programs awarded to NGOs that achieve highest scores

Indexes are common in targeting of social programs

5

• Target transfer to poorest households

• Construct poverty index from 1 to 100 with pre-intervention characteristics

• Households with a score <=50 are poor

• Households with a score >50 are non-poor

• Cash transfer to poor households

• Measure outcomes (i.e. consumption) before and after transfer

Example: Effect of Cash Transfer on Consumption

6

6065

7075

80O

utco

me

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Baseline

7

6065

7075

80O

utco

me

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Baseline

Non-Poor

Poor

8

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

9

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

Treatment Effect

10

• Oportunidades assigned benefits based on a poverty index

• Where

• Treatment = 1 if score <=750

• Treatment = 0 if score >750

Case 4: Regression Discontinuity

11

Case 4: Regression Discontinuity

Fitt

ed v

alu

es

puntaje estimado en focalizacion276 1294

153.578

379.224

2

Baseline – No treatment

0 1 ( )i i iy Treatment score

12

Estimated Impact on CPC

** Significant at 1% level

Case 4 - Regression DiscontinuityMultivariate Linear Regression

30.58**(5.93)

Fitt

ed v

alu

es

puntaje estimado en focalizacion276 1294

183.647

399.51 Treatment Period

Case 4: Regression Discontinuity

13

Potential Disadvantages of RD• Local average treatment effects – not always

generalizable

• Power: effect is estimated at the discontinuity, so we generally have fewer observations than in a randomized experiment with the same sample size

• Specification can be sensitive to functional form: make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: – Nonlinear Relationships– Interactions

14

Advantages of RD for Evaluation

• RD yields an unbiased estimate of treatment effect at the discontinuity

• Can many times take advantage of a known rule for assigning the benefit that are common in the designs of social policy

– No need to “exclude” a group of eligible households/individuals from treatment

15

Measuring Impact

• Experimental design/randomization

• Quasi-experiments

– Regression Discontinuity

– Double differences (Diff in diff)

– Other options

16

Case 5: Diff in diff

• Compare change in outcomes between treatments and non-treatment

– Impact is the difference in the change in outcomes

• Impact = (Yt1-Yt0) - (Yc1-Yc0)

17

TimeTreatment

Outcome

Treatment Group

Control Group

Average Treatment Effect

18

TimeTreatment

Outcome

Treatment Group

Control Group

Estimated Average Treatment Effect

Average Treatment Effect

19

Diff in Diff

• Fundamental assumption that trends (slopes) are the same in treatments and controls

• Need a minimum of three points in time to verify this and estimate treatment (two pre-intervention)

20

Not Enrolled Enrolled t-statMean ΔCPC 8.26 35.92 10.31

Case 5 - Diff in Diff

Linear Regression Multivariate Linear Regression

Estimated Impact on CPC 27.66** 25.53**(2.68) (2.77)

** Significant at 1% level

Case 5 - Diff in Diff

Case 5: Diff in Diff

21

Case 1 - Before and After

Case 2 - Enrolled/Not

Enrolled

Case 3 - Randomization

Case 4 - Regression

Discontinuity

Case 5 - Diff in Diff

Multivariate Linear

RegressionMultivariate Linear

Regression

Multivariate Linear

Regression

Multivariate Linear

Regression

Multivariate Linear

Regression

Estimated Impact on CPC 34.28** -4.15 29.79** 30.58** 25.53**

(2.11) (4.05) (3.00) (5.93) (2.77)** Significant at 1% level

Impact Evaluation Example – Summary of Results