publ0050:advancedquantitativemethods lecture3 ... · courseoutline 1....

PUBL0050: Advanced Quantitative Methods

Lecture 3: Selection on Observables (I) – Matching andSubclassification

Jack Blumenau

1 / 57

Course outline

1. Potential Outcomes and Causal Inference2. Randomized Experiments3. Selection on Observables (I)4. Selection on Observables (II)5. Panel Data and Difference-in-Differences6. Synthetic Control7. Instrumental Variables (I)8. Instrumental Variables (II)9. Regression Discontinuity Designs10. Overview and Review

2 / 57

Motivation

The planner of an observational study should always ask thequestion, “How would the study be conducted if it were possibleto do it by controlled experimentation?”

Cochran, 1965

• Randomization forms the gold standard for causal inference,because in expectation it balances observed and unobservedconfounders

• When we can’t randomize we can design studies to captures thecentral strength of randomized experiments:

• having treatment and control groups that are as comparable aspossible

• i.e. we can try to control for observed covariates3 / 57

Lecture Outline

Identification under selection on observables

Confounding and post-treatment bias

Subclassification

Matching

Conclusion

4 / 57

Identification under selection onobservables

Intuition

Last week: randomization means that treatment assignment isindependent of potential outcomes.

This week: assume treatment is not randomized, but that it isindependent of potential outcomes so long as other factors, 𝑋, are heldfixed.

IntutionWe are assuming that amongst units with the same values for somecovariate X (i.e. conditional on 𝑋), the treatment is “as good asrandomly” assigned.

5 / 57

Identification under Selection on Observables

Identification Assumption1. Potential outcomes independent of 𝐷𝑖 given 𝑋𝑖: (𝑌1𝑖, 𝑌0𝑖)⊥⊥𝐷𝑖|𝑋𝑖

(”selection on observables” or ”conditional independence assumption”)

2. 0 < Pr(𝐷 = 1|𝑋) < 1 for all 𝑋 (common support)

Identification ResultGiven selection on observables we have

𝐸[𝑌1𝑖 − 𝑌0𝑖|𝑋𝑖] = 𝐸[𝑌1𝑖 − 𝑌0𝑖|𝑋𝑖, 𝐷𝑖 = 1] (CIA)

= 𝐸[𝑌1𝑖|𝑋𝑖, 𝐷𝑖 = 1] − 𝐸[𝑌0𝑖|𝑋𝑖, 𝐷𝑖 = 1]= 𝐸[𝑌1𝑖|𝑋𝑖, 𝐷𝑖 = 1] − 𝐸[𝑌0𝑖|𝑋𝑖, 𝐷𝑖 = 0] (CIA)

= 𝐸[𝑌𝑖|𝑋𝑖, 𝐷𝑖 = 1] − 𝐸[𝑌𝑖|𝑋𝑖, 𝐷𝑖 = 0]

Implies that for any specific value for 𝑋𝑖, i.e. 𝑥𝑖, we can define the conditional averagetreatment effect (𝛿𝑥)

𝛿𝑥 ≡ 𝐸[𝑌𝑖|𝑋𝑖 = 𝑥, 𝐷𝑖 = 1] − 𝐸[𝑌𝑖|𝑋𝑖 = 𝑥, 𝐷𝑖 = 0]

6 / 57


Identification Assumption1. Potential outcomes independent of 𝐷𝑖 given 𝑋𝑖: (𝑌1𝑖, 𝑌0𝑖)⊥⊥𝐷𝑖|𝑋𝑖

(”selection on observables” or ”conditional independence assumption”)

2. 0 < Pr(𝐷 = 1|𝑋) < 1 for all 𝑋 (common support)

Identification ResultTherefore, under the common support condition and with a discrete 𝑋𝑖, we cancalculate average effects of 𝐷𝑖 on 𝑌𝑖 by taking weighted averages of 𝛿𝑥:

𝜏ATE = ∑𝑥

𝛿𝑥𝑃(𝑋𝑖 = 𝑥)

𝜏ATT = ∑𝑥

𝛿𝑥𝑃(𝑋𝑖 = 𝑥|𝐷𝑖 = 1)

𝜏ATC = ∑𝑥


i.e. where the weights are the distribution of 𝑋𝑖 in the population (𝜏ATE),treatment group (𝜏ATT), and control group (𝜏ATC).

6 / 57


This indentification assumption and result is common to all the methodswe will study this week and next week.

• Subclassification (today)• Matching (today)• Regression (next week)• Propensity scores (next week)

These differ in a) how we condition on 𝑋𝑖 and b) how we weight 𝛿𝑥.

7 / 57

Illustrative example

Does teacher training improve university applications?Imagine that some school teachers take specialist training in how toprepare their students for university applications. Teachers select intothe training program (i.e. they are not randomly assigned). You believe,however, that conditional on the type of school in which a teacherteachers, training is as good as random.

• 𝑌𝑖: Number of students applying for top universities• 𝐷𝑖: 1 if the teacher did the training, 0 otherwise• 𝑋𝑖: Whether the teacher is at a state, private, or public school

8 / 57


You collect some data and noticethat teacher training is associatedwith teachers’ school-types:

Table 1: 𝑋𝑖, 𝐷𝑖 joint distribution

𝐷𝑖 = 0 𝐷𝑖 = 1𝑋𝑖 =State 0.30 0.05

𝑋𝑖 =Private 0.15 0.15𝑋𝑖 =Public 0.05 0.30

You also notice that average studentapplications are associated withschool-type and teacher training:

Table 2: Mean outcomes

𝐷𝑖 = 0 𝐷𝑖 = 1𝑋𝑖 =State 0 2

𝑋𝑖 =Private 3 4𝑋𝑖 =Public 5 5

9 / 57


Given this information, you calculate the difference in group meansbetween teachers who did the extra training and those who did not:


𝐷𝑖 = 0 𝐷𝑖 = 1𝑋𝑖 =State 0.30 0.05



𝐷𝑖 = 0 𝐷𝑖 = 1𝑋𝑖 =State 0 2

𝑋𝑖 =Private 3 4𝑋𝑖 =Public 5 5

DIGM ≡ 𝐸[𝑌𝑖|𝐷𝑖 = 1] − 𝐸[𝑌𝑖|𝐷𝑖 = 0]= (0.1 ∗ 2 + 0.3 ∗ 4 + 0.6 ∗ 5) − (0.6 ∗ 0 + 0.3 ∗ 3 + 0.1 ∗ 5)= 3

Is the DIGM an unbiased estimator of the ATE? No, we are assuming thatthe treatment is independent of potential outcomes conditional on X

10 / 57


Selection on observables implies that the DIGM is an unbiased estimatorfor the ATE within levels of𝑋𝑖. So let’s calculate those:


𝐷𝑖 = 0 𝐷𝑖 = 1𝑋𝑖 =State 0.30 0.05



𝐷𝑖 = 0 𝐷𝑖 = 1 𝛿𝑥𝑋𝑖 =State 0 2 2

𝑋𝑖 =Private 3 4 1𝑋𝑖 =Public 5 5 0

Then summarize the effect of 𝐷𝑖 on 𝑌𝑖 by taking weighted averages of𝛿𝑥, where the weights are determined by our estimand of interest.

11 / 57


• ATE → weights 𝛿𝑥 by the distribution of 𝑋𝑖 in the population• ATT → weights 𝛿𝑥 by the distribution of 𝑋𝑖 in the treatment group• ATC → weights 𝛿𝑥 by the distribution of 𝑋𝑖 in the control group• Common support → no weight on cells where there is 0 or 1probability of treatment (because the 𝛿𝑥 is undefined)

12 / 57


Treatment Status (D)

Exp

ecte

d ou

tcom

e

0 1

12

34

5

●

●

E[Yi|Di=0,Xi=1]

E[Yi|Di=1,Xi=1]

δ1

●

●

E[Yi|Di=0,Xi=2]

E[Yi|Di=1,Xi=2]

δ2

● ●E[Yi|Di=0,Xi=3] E[Yi|Di=1,Xi=3]δ3

●

● ATE = 1

●

● ATT = 0.5

●

● ATC = 1.5

●

●

DIGM = 3

• Why are ATE, ATT, andATC the same in arandomizedexperiment?

• Why is the DIGM biggerthan all the 𝜏 here?

14 / 57

Confounding and post-treatment bias

What should we select on?

Restate the CIA: Potential outcomes for control units are the same as fortreated units, when those units have the same covariate values (𝑋𝑖).

Question: which covariates make this assumption true?

Answer: we do not know! This is an untestable assumption.

But, there are two ways in which it might fail:

1. Selection bias (confounding variable we have not included)2. Post-treatment bias (included control is actually an outcome)

15 / 57

Selection bias/Confounding

• Selection bias is just another name for confounding

• Confounding is the bias caused by common causes of the treatmentand the outcome

• If we fail to account for any confounding variable 𝑍𝑖 that is relatedto both𝐷𝑖 and 𝑌𝑖, then our identification assumption may be wrong

• Though if our controls correlate with unobserved confounders, wemight be OK

• In general, this is an untestable assumption though it is sometimespossible to provide indirect evidence

• More on this next week.

16 / 57


We must remember that selection on observables is a large con-cession, which should not be made lightly. It is of far greaterrelevance than the technical discussion on the best way to con-dition on covariates…The identification assumption for both OLSand matching is the same: selection on observables.

Sekhon, 2009

The research design therefore depends entirely on the plausiblity oftreatment being conditionally independent of potential outcomes.

17 / 57

Do not control for post-treatment variables

Does civic education increase voter turnout?You are studying the effects of participating in a civic educationprogramme on voter turnout. You also collect data on whetherparticipants have high or low levels of political interest, where politicalinterest is measured after the education programme has been run.

• 𝑌𝑖 is the outcome (voted = 1, not voted = 0)• 𝐷𝑖 is the treatment (participated = 1, did not participate = 0)• 𝑍𝑖 is a post-treatment covariate (high interest = 1, low interest = 0)

We may wish to know the effects of education independent of politicalinterest, so we might be tempted to control for political interest.

WE. SHOULD. NOT. DO. THIS.

Why?18 / 57


Notice first that every respondent has 2 potential 𝑌𝑖 outcomes, and 2potential 𝑍𝑖 outcomes:

𝑌𝑖 = { 𝑌1𝑖 if 𝐷𝑖 = 1𝑌0𝑖 if 𝐷𝑖 = 0 𝑍𝑖 = { 𝑍1𝑖 if 𝐷𝑖 = 1

𝑍0𝑖 if 𝐷𝑖 = 0Consider the 𝑌 difference in means for those with high political interest:

DIGM𝑍𝑖=1 = 𝐸[𝑌𝑖|𝑍𝑖 = 1, 𝐷𝑖 = 1] − 𝐸[𝑌𝑖|𝑍𝑖 = 1, 𝐷𝑖 = 0]= 𝐸[𝑌1𝑖|𝑍1𝑖 = 1, 𝐷𝑖 = 1] − 𝐸[𝑌0𝑖|𝑍0𝑖 = 1, 𝐷𝑖 = 0]= 𝐸[𝑌1𝑖 − 𝑌0𝑖|𝑍1𝑖 = 1]⏟⏟⏟⏟⏟⏟⏟⏟⏟

Causal effect

+ (𝐸[𝑌0𝑖|𝑍1𝑖 = 1] − 𝐸[𝑌0𝑖|𝑍0𝑖 = 1])⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟Selection bias

These are not the same!

• 𝑍1𝑖 = 1 → High political interest with civic education

• 𝑍0𝑖 = 1 → High political interest without civic education

19 / 57


• Intuition: introducing post-treatment variables means that you are,by design, not comparing similar units

• Post-treatment bias is a problem even when the treatment is fullyrandomized (i.e. experiments will not save you)

• Post-treatment bias is only not a problem if the treatment does notaffect 𝑍𝑖 (very difficult to establish in most settings)

• Post-treatment bias can occur if

• You control for a post-treatment variable• You control for a proxy variable that is measured after the treatment• You drop or select observations based on a post-treatment criterion

Overall lesson?

DO. NOT. CONDITION. ON. POST-TREATMENT. VARIABLES.20 / 57

Subclassification

Running example

Do UN interventions Cause Peace?Gilligan and Sergenti (2008) investigate whether UN peacekeepingoperations have a causal effect on building sustainable peace after civilwars. They study 87 post-Cold-War conflicts, and evaluate whetherpeace lasts longer after conflict in 19 situations in which the UN had apeacekeeping mission compared to 68 situations where it did not.

Details:

• 𝑌𝑖: Peace duration (measured in months)• 𝐷𝑖: 1 if the UN intervened post-conflict, 0 otherwise• 𝑋1𝑖: Region of conflict (categorical)• 𝑋2𝑖: Democracy in the past (binary, based on polity)• 𝑋3𝑖: Ethnic Fractionalization (continuous)

21 / 57

Naive estimate

naive_diff <- mean(peace$dur[peace$UN == 1]) -mean(peace$dur[peace$UN == 0])

naive_diff

## [1] 74.4

Naive difference: peace lasted about 6 years longer in situations wherethe UN intervened.

22 / 57

Subclassification

Subclassification is an estimation approach suitable for instances wherewe have categorical 𝑋𝑖 variables (or where we make our 𝑋𝑖 discrete.)

We already covered subclassification in the teachers example, but let’s fixideas here with our UN data.

Procedure:

• Define subclasses• Calculate difference in mean outcome for treatment and controlwithin each subclass

• Calculate average treatment effects by taking weighted averages

23 / 57

Subclassification

Our subclasses here are the ten groups defined by the region anddemocracy variables:

Table 7: Number of observations

E. Europe L. America SS Africa Asia N Africa/Middle EastNon-democracy 10 3 21 7 8Democracy 7 9 14 6 2

For instance, in this sample there are:

• 10 post-conflict instances in Eastern European countries that wereformerly non-democracies

• 14 post-conflict instances in Sub-saharan African countries thatwere formerly democracies

• etc

24 / 57

Treated and control observations per subclass

How many treatment and control units do we have per subclass?

Table 8: (Treated N, Control N)

E. Europe L. America SS Africa Asia N Africa/Middle EastNon-democracy (5, 5) (2, 1) (4, 17) (0, 7) (1, 7)Democracy (2, 5) (2, 7) (2, 12) (0, 6) (1, 1)

• Is the common support assumption violated for any of these cells?

25 / 57

Difference in means per subclass

What is the conditional average treatment effect within each subclass?

Table 9: 𝛿𝑥 (i.e CATE)

E. Europe L. America SS Africa Asia N Africa/Middle EastNon-democracy -29.2 144.0 27.9 NA 123.7Democracy 66.8 5.1 49.0 NA 132.0

Note that these are simply the difference in means estimates betweentreated and non-treated groups, within each subclass.

26 / 57

Average treatment effects





27 / 57






𝐴𝑇 𝑇 = ∑𝑥


= −29.2(5/19) + 66.8(2/19) + 144(2/19) + 5.1(2/19)+27.9(4/19) + 49(2/19) + 123.7(1/19) + 132(1/19)

= 39.54

(i.e. weight by the proportion of treated observations in each cell)

27 / 57






𝐴𝑇 𝐸 = ∑𝑥


= −29.2(10/74) + 66.8(7/74) + 144(3/74) + 5.1(9/74)+27.9(21/74) + 49(14/74) + 123.7(8/74) + 132(2/74)

= 42.97

(i.e. weight by the proportion of all observations in each cell)

27 / 57






𝐴𝑇 𝐶 = ∑𝑥


= −29.2(5/55) + 66.8(5/55) + 144(1/55) + 5.1(7/55)+27.9(17/55) + 49(12/55) + 123.7(7/55) + 132(1/55)

= 44.15

(i.e. weight by the proportion of control observations in each cell)

27 / 57


• ATT → 39.54• ATE → 42.97• ATC → 44.15

These are all somewhat smaller than the raw difference in means (74.4).

Note that we are not really calculating ATC here:

• We cannot identify either of the 𝛿𝑥 for the Asian countries.• ATC ends up being a somewhat odd quantity: the average treatmenteffect for the control observations that have overlap with treatedobservations.

28 / 57

Are 𝑌1𝑖, 𝑌0𝑖⊥⊥𝐷𝑖|𝑋𝑖?

Subclassification is helpful for clarifying the CIA → we are assuming thattreatment is “as good as random” within subclass.

Are we convinved by this assumption?

• Region and democratic history are probably not sufficient

• What other 𝑋𝑖 variables would it be important to condition upon?

• Subclassification is restricted to categorical 𝑋𝑖.

• Not be appropriate if key conditioning factors are continuous• e.g. Here we had to ‘discrete-ize’ the polity score

Is there a better way?

29 / 57

Break

30 / 57

Matching

Matching

Recall the fundamental problem of causal inference:

𝑌𝑖 = 𝐷𝑖 ⋅ 𝑌1𝑖 + (1 − 𝐷𝑖) ⋅ 𝑌0𝑖

so

𝑌𝑖 = { 𝑌1𝑖 if 𝐷𝑖 = 1𝑌0𝑖 if 𝐷𝑖 = 0

One way of viewing this is as a missing data problem. i.e We observe halfthe potential outcomes for each unit, but not the other half.

One solution: impute the missing outcomes → This is what matchingdoes.

31 / 57

Matching

For each unit 𝑖: find the “closest” unit 𝑗 with opposite treatment status.Impute 𝑗’s outcome as the unobserved potential outcome for 𝑖.

𝜏ATT = 1𝑁1

∑𝐷𝑖=1

(𝑌𝑖 − 𝑌𝑗(𝑖))

where 𝑌𝑗(𝑖) is the observed outcome for (untreated) unit 𝑗, the closestmatch to 𝑖. i.e. 𝑋𝑗(𝑖) is closest to 𝑋𝑖 among the untreated observations.

It is also possible to use the average for the 𝑀 closest matches:

𝜏ATT = 1𝑁1

∑𝐷𝑖=1

{𝑌𝑖 − ( 1𝑀

𝑀∑𝑚=1

𝑌𝑗𝑚(𝑖))}

We could impute potential outcomes for control units and define theATE/ATC equivalently.

32 / 57

Nearest neighbour matching, single X,𝑀 = 1, with replacement

Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa ? 51Sierra Leone 1 77 SS Africa ? 35Zaire 1 90 SS Africa ? 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1𝑁1

∑𝐷𝑖=1


= 1/3 ⋅ (51 − 3) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 3)= 30.7

33 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa 3 51Sierra Leone 1 77 SS Africa ? 35Zaire 1 90 SS Africa ? 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1𝑁1

∑𝐷𝑖=1


= 1/3 ⋅ (51 − 3) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 3)= 30.7

33 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa 3 51Sierra Leone 1 77 SS Africa 11 35Zaire 1 90 SS Africa ? 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1𝑁1

∑𝐷𝑖=1


= 1/3 ⋅ (51 − 3) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 3)= 30.7

33 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa 3 51Sierra Leone 1 77 SS Africa 11 35Zaire 1 90 SS Africa 3 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1𝑁1

∑𝐷𝑖=1


= 1/3 ⋅ (51 − 3) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 3)= 30.7

33 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa ? 51Sierra Leone 1 77 SS Africa ? 35Zaire 1 90 SS Africa ? 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1/3 ⋅ (51 − 7) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 7)= 28

34 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa 7 51Sierra Leone 1 77 SS Africa ? 35Zaire 1 90 SS Africa ? 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1/3 ⋅ (51 − 7) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 7)= 28

34 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa 7 51Sierra Leone 1 77 SS Africa 11 35Zaire 1 90 SS Africa ? 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1/3 ⋅ (51 − 7) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 7)= 28

34 / 57


Country D EthFrac Region 𝑌0𝑖 𝑌1𝑖Liberia 1 83 SS Africa 7 51Sierra Leone 1 77 SS Africa 11 35Zaire 1 90 SS Africa 7 23Chad 0 83 SS Africa 3Senegal 0 72 SS Africa 11Niger 0 73 SS Africa 11

What is 𝜏ATT?

𝜏ATT = 1/3 ⋅ (51 − 7) + 1/3 ⋅ (35 − 11) + 1/3 ⋅ (23 − 7)= 28

34 / 57

More𝑋 variables

Commonly we will want to match on many 𝑋 variables, not just one ortwo.

In our UN example, for instance, we might also include:

• Number of deaths in last war• Duration of last war• Ethnic fractionalization• Military personnel• Population size• Mountains

Is this enough? What else? Are any of these post-treatment?

35 / 57

Matching distances

Adding more covariates creates a problem, however. We have to definehow we measure whether two units are “close” to one another.

Which is “closer”?

• Treated case:

• Haiti, with polity = -6, region = Latin America, and ethfrac = 1

• Control cases:

• Panama, with polity = 8, region = Latin America, ethfrac = 3• Eygypt, with polity = -7, region = N Africa, ethfrac = 4• El Salvador, with polity = -6, region = Latin America, ethfrac = 26

→ We need a metric that takes 2 vectors of covariate values and projectsthem to a unidimensional scale

36 / 57

Matching distances

• Exact matching

• Match 𝑗 with 𝑖 if 𝑗 has identical covariates to 𝑖• Rapidly breaks down with dimensionality of X, or with continuous 𝑋𝑖

• Normalized Euclidian distance

• Scale distances on each varaible by the inverse of sample variance• Good with normally distributed 𝑋, not great with binary data

• Mahalanobis distance

• Scale distances on each 𝑋 by the inverse of the covariance matrix• Good with normally distributed 𝑋, not great with binary data

• Genetic matching

• Genetic matching aims directly to find the set of matches thatminimize covariate imbalance across all variables

It is also common to use exact matching for some covariates (i.e. region)and then distance matching for others.

37 / 57

Mahalonobis distances

−3 −2 −1 0 1 2 3

−10

−5

05

10

X1

X2 (3,0)

Distance = 8.38

(0,−3) Distance = 0.56

(1,−2) Distance = 1.22

38 / 57

Other matching considerations

• What size for 𝑀? 1-to-1? Many-to-1?

• Small M: decreased bias (2nd match always further than first)• Large M: decreased variance (larger matched sample)

• Matching with or without replacement?

• With replacement: decreased bias because some controls will begood matches for multiple treatment units

• But: replacement makes inference more complicated as matchedcontrols are no longer independent (larger standard errors)

• Which treatment effect? (ATE/ATC/ATT)

• Depends on substantive interest, also on available matches

39 / 57

𝑀 and matching with/out replacement

Full sample

X1

X2

C

C

C

C

CC

CC

C

CC

C

C

CC

CC

C

C

C

TTTT T

T

TT

T

T

40 / 57


M = 1, with replacement

X1

X2

C

C

C

C

CC

CC

C

CC

C

C

CC

CC

C

C

C

TTTT T

T

TT

T

TCCCCC

CC

C

C

C

CCCCC

CC

C

C

C

TTTT T

T

TT

T

T

TTTT T

T

TT

T

T

41 / 57


M = 1, without replacement

X1

X2

C

C

C

C

CC

CC

C

CC

C

C

CC

CC

C

C

C

TTTT T

T

TT

T

TC

C

CC C

CCC

C

C

C

C

CC C

CCC

C

C

TTTT T

T

TT

T

T

TTTT T

T

TT

T

T

42 / 57



X1

X2

C

C

C

C

CC

CC

C

CC

C

C

CC

CC

C

C

C

TTTT T

T

TT

T

TC

C

C

C

C

C

C

CC

C

CC CC

C

CCC

CC

TTTTTTTT TT

TT

TTTT

TT

TT

43 / 57



X1

X2

C

C

C

C

CC

CC

C

CC

C

C

CC

CC

C

C

C

TTTT T

T

TT

T

TC

C

C

C

CCC

CC

C

C C

C

C C

CC

C

C

C

TTTTTTTT TT

TT

TTTT

TT

TT

44 / 57

Matching in practice

Y <- peace$durD <- peace$UNX <- peace[,c(”lwdeaths”, ”lwdurat”, ”ethfrac”,

”pop”, ”milper”, ”bwplty2”,”lmtnest”, ”ssafrica”, ”asia”,”lamerica”,”eeurop”)]

45 / 57


matched_att_estimate <- Match(Y = Y,Tr = D,X = X,Weight = 1, # 1 = Euclidean,

# 2 = MahalanobisM = 1, # How many matches?replace = TRUE, # Defaultestimand = ”ATT”)

matched_att_estimate$est

## [,1]## [1,] 12.47368

46 / 57


summary(matched_att_estimate)

#### Estimate... 12.474## AI SE...... 16.801## T-stat..... 0.74244## p.val...... 0.45782#### Original number of observations.............. 87## Original number of treated obs............... 19## Matched number of observations............... 19## Matched number of observations (unweighted). 19

47 / 57

Assessing balance

After matching, the distribution of X should be the same for treatmentand control groups:

• Many papers will present tables of covariate means and p-valuesbefore and after matching as evidence of comparability

• Strictly speaking, p-value are not very informative, as they aresensitive to changes in the sample size

• Instead, it is useful to measure the standardized bais of a covariatebefore and after matching and compare:

bias𝑋𝑖= �̄�𝑡 − �̄�𝑐

𝜎𝑡

where 𝜎𝑡 is the standard deviation of 𝑋 in the full treated group.

48 / 57


−1.0 −0.5 0.0 0.5 1.0

Standardized bias

eeurop

lamerica

ssafrica

lmtnest

bwplty2

milper

pop

ethfrac

lwdurat

lwdeaths RawMatched

49 / 57

Genetic matching in practice

genout <- GenMatch(Tr=D, X=X,estimand=”ATT”,M=1)

gen_matched_att_estimate <- Match(Y = Y, Tr = D,X = X,estimand = ”ATT”,M = 1,Weight.matrix = genout)

gen_matched_att_estimate$est

## [,1]## [1,] 42.47368

50 / 57

Genetic matching in practice

−1.0 −0.5 0.0 0.5 1.0

Standardized bias

eeurop

lamerica

ssafrica

lmtnest

bwplty2

milper

pop

ethfrac

lwdurat

lwdeaths RawMatched

51 / 57

Consequences of matching decisions

Table 12: ATT from different matches

M Replacement Distance ATT1 Yes Euclidean 12.472 Yes Euclidean 25.611 Yes Mahalanobis 8.952 Yes Mahalanobis 24.131 No Euclidean 25.952 No Euclidean 35.681 No Mahalanobis 23.052 No Mahalanobis 37.761 Yes Genetic 42.47

Implication: Even using the same covariates, different matching criteriacan lead to different outcomes! Particularly when 𝑁 is small.

52 / 57

Consequences of matching decisions

One approach: pick the matching procedure that results in the smalleststandardized difference in means across all covariates:

Table 13: ATT from different matches

M Replacement Distance ATT Mean standardized bias1 Yes Genetic 42.47 0.152 Yes Euclidean 25.61 0.221 Yes Euclidean 12.47 0.241 No Euclidean 25.95 0.242 Yes Mahalanobis 24.13 0.271 No Mahalanobis 23.05 0.312 No Euclidean 35.68 0.331 Yes Mahalanobis 8.95 0.332 No Mahalanobis 37.76 0.38

53 / 57

Matching advice

While extensive time and effort is put into the careful design ofrandomized experiments, relatively little effort is put into the cor-responding ‘design’ of non-experimental studies.

Stuart, 2010

Best practice is to design without access to outcome variables

1. Look at data without outcome variables; design matching strategy

• 1 to 1; many to 1; with/without replacement, etc

2. Test covariate balance; if unbalanced, go back to 1

3. Compare outcomes only after matching is completed.

54 / 57

Conclusion

Data requirements for matching

• Matching only requires cross-sectional data (one observation perunit)

• Normally requires a rich set of covariates to convince the readerthat the conditional independence assumption is satisfied

• Does not generalise easily (unlike regression) to continuoustreatment variables

For your projects:

• “Selection on observables” strategies using observational data arerelatively easy to put together

• Whether or not these strategies convincingly establish causaleffects depends entirely on how good the set of control/matchingvariables you use is.

55 / 57

Selection on observables

• By assuming treatments are “as good as random” conditional on 𝑋,we can make causal claims from non-experimental data

• How convincing our causal claims are is entirely determined by howplausible this assumption seems in a given context

• We should condition on all potentially confounding variables

• We should not condition on any post-treatment variables

• Matching and subclassification are two approaches to conditioning

56 / 57

Next week

Next week: Regression and propensity scores

57 / 57

publ0050:advancedquantitativemethods lecture3 ... · courseoutline 1....

Documents