Natural Experiments

as Multi-Method Research

Thad DunningYale University

Prepared for a short course on Multi-Method Research, American Political Science Association meetings, Seattle, Washington, August 30, 2011

Articles published in major political science and economics journals with “natural experiment” in the title or abstract (as tracked in the online archive JSTOR).

The Growth of Natural Experiments (1960-2009)

A “quantitative” technique?• Natural experiments are often regarded as quantitative:

• Standard Natural Experiments• Regression-Discontinuity Designs• Instrumental-Variables Designs

• Gathering systematic data on outcomes, exposure to treatment, and pre-treatment covariates for each case—a.k.a. Data-Set Observations (DSOs)—is indeed critical

• One advantage of strong natural experiments: the quantitative analysis may be simple and transparent• Often, a simple difference of means suffices to estimate causal effects

• Another advantage: models of the data-generating process are often credible (e.g., the Neyman model)

The crucial role of qualitative methods• However, qualitative methods are invaluable and often

prerequisite for natural experiments.

• As Angrist and Krueger (2001) put it,“Our view is that progress in the application of instrumental variables

methods depends mostly on the gritty work of finding or creating plausible experiments that can be used to measure important economic relationships—what the statistician David Freedman (1991) has called “shoe-leather” research. Here the challenges are not primarily technical in the sense of requiring new theorems or estimators. Rather, progress comes from detailed institutional knowledge and the careful investigation and quantification of the forces at work in a particular setting. Of course, such endeavors are not really new. They have always been at the heart of good empirical research” [emphasis added].

What Role Do Qualitative Methods Play?

1. Assessing as-if random

• Relative to conventional observational studies, the claim that treatment assignment is as good as random is the defining feature of a natural experiment

• Definition: A natural experiment is an observational study (that is, a study without an experimental manipulation) in which assignment to treatment and control groups is done at random—or “as if” at random

• Since there is (usually) no true randomization, this assertion is, to some extent, not testable

• Yet evidence and a priori reasoning, informed by substantive knowledge, can help validate a natural experiment

Quantitative Tools for Assessing As-If Random

• Standard Natural Experiments• Balance tests: Treatment and control groups are statistically balanced on pre-

treatment covariates—just as they should be in true (randomized controlled) experiments

• Regression-Discontinuity Designs• Balance tests, plus histograms or “conditional density” tests: units do not bunch on

one side of the RD threshold (e.g., effects of anti-discrimination laws)• Placebo tests: no apparent effect away from the RD threshold

• Instrumental-Variables Designs• Balance tests; specification tests (note many applications include covariates in

specifications—not so good)

• These are all based on Data-Set Observations (DSOs)

Treatment-Assignment CPOs

• These are nuggets of information about the process by which cases ended up in treatment or control groups• Analyst must explain why this process plausibly leads to an as-if random

allocation of units to treatment and control. • This is quite different from saying that one cannot think of any potential

confounders, and thus assignment must be as good as random (or worse, that confounders have been “controlled”)

• Knowledge of this process does not typically come by knowing the values of independent and dependent variables for each case (i.e., DSOs)

• Treatment assignment CPOs provide crucial leverage for causal inference

Example: Impact of Land Titling

• Squatters with land titles seem to do better than those without them (De Soto 2000). Do titles have a causal effect?

• Maybe—but squatters who obtain titles are different from those who don’t.

• They may have, say, different family backgrounds

• Differences in outcomes could be due to the effect of titles, due to the confounders, or both.

• We might try to control for confounders we can measure—e.g., compare titled and non-titled squatters with similar family backgrounds.

• But within these strata, more determined or motivated squatters may obtain titles—and determination is hard to measure

A Natural Experiment on Property Rights

• In 1981, squatters organized by the Catholic church occupied an urban wasteland in the province of Buenos Aires, dividing the land into similar parcels

• After the return to democracy, a 1984 law expropriated the land, with the intention of transferring title to the squatters

• Some of the original owners challenged the expropriation in court, leading to long delays, while other titles were ceded and transferred to squatters

• The legal action created a “treatment” group – squatters to whom titles were ceded – and a control group – squatters whose titles were not ceded

A Natural Experiment? Quantitative Evidence

Qualitative evidence

• Yet, just as important is qualitative evidence – for example, on the process by which the squatting took place• Squatters and Catholic Church organizers did not appear to know

the land was privately owned at the time they occupied it• They did not anticipate the later expropriation of land by the state• They had no basis for predicting which particular plots of land

would be expropriated, and thus could not assign plots to particular squatters with this in mind

• These are Treatment-Assignment CPOs: • They do not come in the form of systematic data on the values of

independent and dependent variables (DSOs)• They help the analysts understand the process by which units

were eventually allocated to treatment and control groups—and to validate the claim that this assignment was as good as random

The effects of property rights

• Galiani and Schargrodsky (2006, 2007) find significant differences in housing investment, household structure, and educational attainment of children (but not in access to credit markets, contradicting De Soto)

• Squatters who received titles also believed more in individual efficacy (!)—e.g. “people get ahead in life through hard work, rather than luck” (Di Tella et al.)

• Without the natural experiment, such a finding could be due to confounding: people who acquired land titles might have had stronger ex-ante faith in their efficacy than those who didn’t

Snow on Cholera• Nineteenth-century

London: Against the predominant theory—miasma—John Snow hypothesized that cholera was a water- or waste-born disease

• Snow’s natural experiment• In 1852, the Lambeth water company moved its intake pipe upstream on the Thames, to a purer water source• The Southwark & Vauxhall company left its intake pipe in place.

The predominant theory

Death rate from cholera in London, by source of water supply

Rate per 10,000 houses

Southwark & Vauxhall

315

Lambeth 37

Rest of London 59

(adapted from Snow 1855, Table IX)

Snow on Cholera: DSOs

“The pipes of each Company go down all the streets…A few houses are supplied by one Company and a few by the other… In many cases a single house has a supply different from that on either side. Each company supplies both rich and poor, both large houses and small; there is no difference either in the condition or occupation of the persons receiving the water of either company…

“It is obvious no experiment could have been designed which would more thoroughly test the effect of water supply on the progress of cholera than this.”

-- John Snow (1885: 74-75)

• The quote speaks to external validity, but also to the “pre-treatment equivalence” of treatment and control groups

Snow on cholera: CPOs

Exercise: Treatment-Assignment CPOsSnow on cholera. The impurity of water in the Thames was a source of concern to public authorities before the cholera outbreak, and the Metropolis Water Act of 1852 made it unlawful for any water company to supply houses with water from the tidal reaches of the Thames after August 31, 1855. Yet, while the Lambeth’s move of its intake pipe upstream was planned in the late 1840s and completed in 1852—before the cholera outbreak of 1853-54—the Southwark and Vauxhall company did not move its pipe until 1855.

In other words, the Lambeth Waterworks Company chose to move its pipe upstream before it was legally required to do so, while Southwark & Vauxhall left its intake pipe in place.

Questions: Could this fact pose a threat to as-if random? How might Snow’s discussion of the process by which water was purchased and supplied counter some potential threats to the validity of the natural experiment? Are these Treatment-Assignment CPOs?

Validating As-If Random, In Sum• Some key questions to ask of any study:

• Are treatment and control groups unbalanced with respect to variables other than the treatment?

• Do units “self-select” into groups?• Are policy interventions applied in a way that anticipates the behavioral

response of units?

• “No” to all questions may be necessary, if not sufficient, for a plausible natural experiment

• Thus, both CPOs and DSOs play a critical role

• Similar points may be made about regression-discontinuity and instrumental-variables designs

2. Independent-Variable CPOs

• These nuggets of information provide information about the presence or values of an independent variable (a treatment); they can contribute both to natural experiments and in exploratory or confirmatory research undertaken in conjunction with a natural experiment. • They can also sometimes be useful for investigating what aspect or

component of a treatment is plausibly responsible for an estimated causal effect.

• Snow on cholera:• Early in Snow’s research, he found infected water or waste was

present during cases of cholera transmission • The Broad Street pump: brewers get water from private pumps• Verifying source of water supply in his natural experiment—a case of

“shoe leather” epidemiology

3. Mechanism CPOs

• In many natural-experimental contexts, using selected units in treatment and control groups to “extract… ideas at close range” (Collier 1999) can be useful for explaining effects

• There may be rich opportunities for “experimental ethnography” (Sherman and Strang 2004), in which qualitative data shapes the interpretation of causal effects

• Mechanisms may be understood as invariant processes, not necessarily mediating variables:• E.g., “deterrence,” “empowerment,” or “solidarity” • Close range fieldwork may be quite important for identifying such

processes that might account for an effect (e.g. the Argentina land titling study)

4. Auxiliary-Outcome CPOs

• These provide “information about particular occurrences that should occur alongside (or perhaps as a result of) the main outcome of interest if in fact that outcome were caused in the way stipulated by the theory under investigation” (Mahoney 2010: 129). • They may thus be closely linked to theory testing; the metaphor of a

criminal detective searching for clues is useful here (Collier, Brady and Seawright 2010).

• Using a natural experiment, Dunning and Nilekani (2011) find that caste-based quotas in India have very weak distributive effects• They attribute this to patterns of party competition engendered by the

rotation of quotas across electoral jurisdictions (village councils)

5. Model-Validation CPOs• These are nuggets of information about causal process that can support or

invalidate assumptions of causal models• Causal inference depends on maintained assumptions—but these can sometimes be

validated, at least partially

• In principle, experiments and natural experiments are “design-based” rather than “model-based” methods• Inferential problems (such as confounding) are confronted through research design, not ex-

post statistical adjustment• Causal leverage comes (or should come) from the design and not from modeling (e.g.,

multivariate regression analysis, matching• Thus, a simple difference-of-means test can suffice

• Yet, the assumptions of models—e.g., the Neyman model—matter here as well

The Effects of Land Titles on Children’s Health

Source: Galiani and Schargrodsky (2004). Notice that this is intention-to-treat analysis.In the first two rows, data for children ages 0-11 are shown; in the third row, data for teenage girls aged 14-17 are shown. The number of observations is in brackets.

Is the Model Credible?• The Neyman causal model—which defines the estimands

that difference-of-means tests estimate—is very general

• However, there are assumptions that may or may not be credible, depending on the setting

• One unit’s potential response is assumed not to depend on the treatment assignment of other units

• Rubin referred to this “no interference between units” as the Stable Unit Treatment Value (SUTVA) assumption

• Thus the model may not be an adequate description of happens in certain natural experiments

• E.g., in Argentina, the fertility decisions of squatters in the control group may be influenced by the fertility decisions of their neighbors in the treatment group

• Model-validation CPOs can be helpful here

• E.g., how do squatters interact with each other?

Regression-Based Inference

Model-Validation CPOs: Regression

• Question: How does affluence affect attitudes towards economic policy?

• Doherty, Green, and Gerber (2005, 2006) take a survey of lottery winners and ask questions about political attitudes, e.g., towards redistributive taxation or social spending

• An innovative natural experiment: lottery winners are randomized to the level of winnings

Doherty et al. as a natural experiment

• Findings: lottery winnings influence attitudes about narrow redistributive issues (the estate tax) but not broader issues (role of government)

• A key merit: there really is a random mechanism determining assignment to treatment status

• However, our interest may lie in the effects of overall income, not lottery winnings

• Instrumental-variables regression may help—if the assumptions of IV model are correct

Lottery Winnings as an Instrument

The impact of income on political attitudes? Regress the latter on the former:

The problem: Income and are not independent, so OLS estimates of will be biased.

The solution: Use lottery winnings as an instrument for income.

Notice that INCOME = LOTTERY_WINNINGS + EARNED_INCOME

ATTITUDES = INCOME i ii

i

i i i

Thus, the model says:

ATTITUDES = (LOTTERY_WINNINGS + EARNED_INCOME ) i

• According to the model, a marginal increment in either lottery winnings or earned income is associated with the same expected marginal increment in attitudes

• This could be called the assumption of “homogenous partial effects” (Dunning 2008)

• Yet, windfall earnings associated with the lottery may have different effects than money earned through work

i i i

ATTITUDES = LOTTERY_WINNINGS + EARNED_INCOME i1 2

Suppose:

• To estimate consistently, we would need another instrument for earned income—and getting rid of the confounding of earned income is supposed to be the point

• The point is that as-if random is not enough; the model of the data-generating process must be credible as well

• In principle, model-validation CPOs could help here

• Do lottery winners say they treat windfall income differently?

i i i

2

Exercise: Model-Validation CPOs• Horiuchi and Saito (2009), in their study of the effect of turnout on fiscal

disbursements to municipalities in Japan, use election-day rainfall as an instrumental variable for municipal turnout. They argue that in this context, the assumption of “homogenous partial effects” (Dunning 2008)—that is, the assumption that variation in turnout related to rainfall has the same effect as variation unrelated to rainfall—is likely to be valid.

• Why do these authors use instrumental variables here?

• What does the assumption of “homogenous partial effects” imply in this substantive context?

• What sorts of model-validation CPOs might validate the assumption?

• What model-validation CPOs would undermine it?

Other Model-Validation CPOs

• In instrumental-variables analysis, model-validation CPOs can help assess whether the instrument affects the outcome through channels other than the treatment variable• Does rainfall affect conflict in Africa through channels other than economic

growth (Miguel et al. 2004)?• Does draft lottery number affect later earnings through channels other than

service in Vietnam (Angrist 1990)?

• Validation of the model is crucial in any study, and qualitative methods can play a central role

• However, there are fewer examples of successful Model-Validation CPOs than the other types of CPOs

• This may be because model validation is too-infrequently discussed…

Conclusions: Natural Experiments as Multi-Method Research

• Quantitative analysis is critical and has the potential advantage of simplicity and transparency

• Reliance mainly on differences-of-means tests gives analysts more space to discuss the substantive context, research design and results

• Qualitative methods can bolster the credibility of quantitative analysis

• Qualitative research is critical in other ways

• Substantive knowledge and close attention to context are crucial for recognizing the potential existence of a natural experiment

• Thus, natural experiments are pre-eminently a form of multi-method research

• Yet, the quantitative analysis of natural experiments has received much more methodological attention

Conclusions: Next Directions

• What distinguishes a good (convincing, compelling) CPO from a bad (misleading) one?

• Knowledge of context is critical, and only a few experts may possess this

• A possible tendency towards confirmation bias—ignore disconfirming CPOs while highlighting helpful ones?

• Does the adversarial tradition (e.g. debates among case experts) help?

• How can qualitative methods best bolster the strength of natural experiments along many dimensions?

Typology of Natural Experiments

Plausibility of As-If Random

Least Most

Releva

nce o

f the

Inte

rven

tion

Least

Mos

t

Cre

dib

ilit

y of

th

e M

odel

s

Lea

stM

ost

Weak Research Design

Strong Research Design