multiverse analysis of natural experiments - systematic

Multiverse analysis of natural experimentsSystematic execution, presentation and interpretation of robustness analyses

Dave Balzer and Nico Sonntag

November 17, 2020

Multiverse analysis of natural experiments November 17, 2020 1 / 21

Forking paths and researcher degrees of freedom

Reported

Analyzed

a) Traditional analysis b) Robustness analysis c) Multiverse analysis


Multiverse analysis

It is possible to calculate all plausible model combinations automatically. We build onapproaches that have been around for some time. For example:

• “I Just Ran Four Million Regressions” (Sala-i-Martin 1997)• “Multimodel analysis” und Stata module mrobust (Young & Holsteen 2017)• “Multiverse analysis” (Steegen et al. 2016)• “Specification curves (Simonsohn et al. 2015)• “Coefficient stability plots” (Rao 2020)

Our own approach emphasizes

• Relevance for sociology and natural experiments• Includes further degrees of freedom• The aim is to assess which decisions are particularly critical for results


Harmless?


Unexpected Event during Surveys Design

Terrorist attack during ESS field work as a natural experiment (Legewie 2013):

• Randomisation of whole periods (before/after attack)

• Estimate: ATE (?)



“In many ways, this identification strategy resembles a regression discontinuity design.”

δ

-500

0

500

1000

1500

2000

outc

om

e

-100 -50 0 50 100

forcing variable

• Randomisation only around the threshold

• Correctly specified functional form

• Estimate: LATE (local average treatment effect)



If longitudinal data are available, a difference-in-differences model could also beestimated:

Differenz 1

Differenz 2 }Treatment-Effekt (ATT)

Zeitpunkt 1 Zeitpunkt2

Ou

tco

me

Kontrollgruppe

Treatmentgruppe

• (As-if) randomisation with regard to time trends (common trends assumption)

• Estimate: ATT


Our research question



Ele

ctio

n

ControlGroup Treatment

Group(N = 2660)(N = 1817)

010

020

030

0

Num

ber

of R

espo

nden

ts/D

ay

29.03.2005 19.04.2005 10.05.2005

Interview Date

Benedict XVI

Ele

ctio

n

ControlGroup Treatment

Group(N = 5750)

(N = 3580)

010

020

030

040

050

0

Num

ber

of R

espo

nden

ts/D

ay

20.02.2013 13.03.2013 03.04.2013

Interview Date

Francis



To what extent did the 2005 and 2013 papal elections influence reported religious activity?

Features of our research project

• GSOEP allows the analysis of longitudinal data

• We are able to compare two similar events with the same data set

• Flexible study design: illustrative application of multiverse analysis


Religious and national identity

Mechanisms:

• Interplay between religious and national identity

• Do publicly visible religious leaders increase the salience of religion in their countryof origin?

Hypotheses:

H1: Only the 2005 papal election, but not the 2013 papal election, should haveincreased reported religious activity.

H2: The 2005 papal election primarily influences respondents with low religiousactivity.

H3: The 2005 papal election primarily affects respondents without establishedreligious identity.


Data summary

Election Benedict XVI (2005):

• GSOEP v33: years 2001, 2005, 2007

• Total N: 64 342

• N ± 3 weeks: 20 296

Election Francis (2013):

• SGOEP v33: years 2011, 2013, 2015

• Total N: 86 872

• N ± 3 weeks: 44 078

• Outcome 2013 scaled differently!

0 10 20 30 40 50 60percent

every week

every month

less frequently

never

Rel

igio

us p

artici

pation

(se

rvic

e vi

sit)

Years 2001, 2005, 2007. N = 63,940.


Multiverse analysis: Election Benedict XVI (2005)

An exemplary multiverse analysis includes several variants of a regression discontinuitydesign:

• Time trend: linear, quadratic or cubic

• Constant or changing slope after treatment (interaction time trend × treatment)

• Day 0: treatment or control group

• Sub-sample: Catholics vs. non-Catholics

• Bandwidth: [7; 42] days before and after the election

3× 2× 2× 2× 36 = 864 models

We focus on 144 models with bandwidths at weekly intervals (7, 14, 21 ... 42 days).



7

28

141421

7777

21

71421

42

7

35

2114

28

77

2828

1421

42

14

4235

7

3535

14

4235

7

42

14

2828

14

42

77

21

35

7

35

14

28

77

42

1414211421

35

14

28212821142121

2835

14

42

147

28

42

28

4235

14

42

28353535

28

4242

21

42

2121

42

2121

4242352821

42

7

3528

424242353535

283535

2828283528

14

423528

42

21

35

14212121

7

21141414

77

28

7

3528

42

21

714

7b-widthcatholic = 0catholic = 1

day 0 controlday 0 treatment

cubicquadratic

linearinteract

no interact

Specifications

-1.5

-.75

β 0

.75

1.5

Specification curve: RDD



Another exemplary multiverse analysis covers variants of the panel models:

• Fixed- vs. random effects

• Year 2007 in analysis sample yes/no

• Day 0: treatment or control group or exclusion

• Sub-sample: Catholics vs. non-Catholics

• Bandwidth: [7; 42] days before and after the election

2× 2× 3× 2× 36 = 864 models

Again, we focus on 144 models with bandwidths at weekly intervals (7, 14, 21 ... 42days).



777714

7771414

7

211414

2128

142114212828

2114

35

14

2828

77

35

21

4235

7

28

7

21

7

28

7

212114

42

14

35

7

28

7

423528282828

35

714

423535

4242

2828

42

7

35

21

7

35423528352835353535

42423535

4242

21

7

42352835

7

3542

7

4242

2828282828

7

2128

7

28

424235424242

2121

35

21211421

42

211414

4235

141414

42

212121212121141414141414

b-widthcatholic = 1catholic = 0

day 0 missing

day 0 controlday 0 treatment

w 2007w/o 2007

FERE-.3

-.15

β 0

.15

.3

Specification curve: Panel/DiD


Multiverse analysis: Crucial decisions

RD specifications:

• Overall, the results do not confirm the hypothesis

• Little systematic relationship between decisions and effect size• Non-Catholic respondents• Day of election = treatment

Panel specifications:

• Some evidence of a positive treatment effect

• Three patterns• Non-Catholic respondents• Bandwidth of 14 or 21 days• Inclusion of the year 2007



0

20

40

60

80

freq

ency

-.8 -.4 0 .4 .8β

catholic = 1

0

20

40

60

80

freq

ency

-.8 -.4 0 .4 .8β

catholic = 0

0

10

20

30

freq

ency

0 .2 .4 .6 .8 1p value

0

10

20

30

freq

ency

0 .2 .4 .6 .8 1p value

0/1 1/2 2/3

cutpoint


Conclusion and discussion

Multiverse analyses help to increase transparency

• Very few natural experiments are “harmless” in the sense that they unambiguouslycall for a particular research design

• Thus, there are many researcher degress of freedom in the analysis of naturalexperiments

Open questions

• Shifting the problem to another level?

• Overburdening the readers?

• Best possible (graphic) presentation of results?

• Difficulty of model comparisons

• Too pessimistic about theory-driven model selection?


Thank you for your attention! Pax et bonum!


References

Angrist, J. D. and J.-S. Pischke (2008). Mostly harmless econometrics: An empiricist’s companion. Princeton universitypress.

Legewie, J. (2013). Terrorist events and attitudes toward immigrants: A natural experiment. American journal ofsociology 118(5), 1199–1245.

Rao, A. (2020). Coefficient Stability Plots. https://github.com/AakaashRao/starbilit.

Sala-i-Martin, X. X. (1997). I just ran four million regressions. American Economic Review 87.

Simonsohn, U., J. P. Simmons, and L. D. Nelson (2020). Specification curve: Descriptive and inferential statistics on allreasonable specifications. Available at SSRN: https://ssrn.com/abstract=2694998.

Steegen, S., F. Tuerlinckx, A. Gelman, and W. Vanpaemel (2016). Increasing transparency through a multiverse analysis.Perspectives on Psychological Science 11(5), 702–712.

Young, C. and K. Holsteen (2017). Model uncertainty and robustness: A computational framework for multimodelanalysis. Sociological Methods & Research 46(1), 3–40.


https://github.com/AakaashRao/starbilit

https://ssrn.com/abstract=2694998

multiverse analysis of natural experiments - systematic

Documents