“Dirty Pool” Revisited: When Less is More

Robert S. Erikson, Pablo M. Pinto, Kelly T. Rader

Department of Political Science

Download paper at: http://ssrn.com/abstract=1450061

Do Democracies Trade more with other Democracies?

How the literature approaches this question:• Convention:

– Measure units as dyad-year• e.g.: Sierra Leone-Ecuador 1997

– Measure democracy as democracy score for the least democratic dyad member

– Relate democracy to trade levels while controlling for other stuff

• Main issues: – Fixed effects?– Lagged dependent variable on RHS?– Dyads as unit not an issue.

Do Democracies Trade more with other Democracies?

• Methodological issues discussed in 2001 symposium in IO

• 90,000+ dyad-years, but only up to 115 countries over 30 years

• We present a cautionary tale about using dyads as units of analysis

• Large data set with a massive N can create overconfidence in hypothesis testing

Symposium on Research Design and Methods in IR:

– Editors' Introduction (pp. 439-440)

Peter Gourevitch, David A. Lake– Dirty Pool (pp. 441-468)

Donald P. Green, Soo Yeon Kim, David H. Yoon– Clear and Clean: The Fixed Effects of the Liberal Peace  (pp. 469-

485)

John R. Oneal, Bruce Russett– Throwing out the Baby with the Bath Water: A Comment on Green,

Kim, and Yoon (pp. 487-495)

Nathaniel Beck, Jonathan N. Katz– Proper Nouns and Methodological Propriety: Pooling Dyads in Inte

rnational Relations Data (pp. 497-507)

Gary King

Our contribution

• Randomization tests on dyads to infer the correct p-values

• Switch to nation-years as units of analysis: proportion of trade with democracies as a function of regime

• Difference-in-difference analysis of change in trading partners after a regime shock

• Cross-sectional analysis of trade by regime type

GKY (current dollars)

GKY (current dollars)

GKY sample (constant dollars)

GKY sample (constant dollars)

Expanded Trade DataTable 2: Regression Analysis of Bilateral Trade, 1950-2000

Pooled Dyad fixed effectsPooled with dynamics

Dyad fixed effects with dynamics

Dyad and year fixed effects

Year fixed effects with dynamics

Dyad and year fixed effects with

dynamics

GDP 0.911 0.896 0.077 0.199 0.918 0.087 0.224(0.003) (0.005) (0.002) (0.004) (0.007) (0.002) (0.005)p=.000 p=.000 p=.000 p=.000 p=.000 p=.000 p=.000

Population 0.592 0.190 0.052 0.040 0.252 0.057 0.103(0.002) (0.007) (0.001) (0.004) (0.012) (0.001) (0.007)p=.000 p=.000 p=.000 p=.000 p=.000 p=.000 p=.006

Distance -0.751 -0.063 -0.065(0.006) (0.002) (0.002)p=.000 p=.000 p=.000

Alliance 0.351 0.378 0.028 0.065 0.384 0.024 0.064(0.014) (0.024) (0.005) (0.015) (0.024) (0.005) (0.015)p=.000 p=.000 p=4e-7 p=9e-6 p=.000 p=1e-5 p=1e-5

Democracy 0.039 0.016 0.003 0.003 0.013 0.004 0.003(0.001) (0.001) (0.000) (0.000) (0.001) (0.000) (0.000)p=.000 p=.000 p=.000 p=.000 p=.000 p=.000 p=2e-12

rand p=.001 rand p=0.011 rand p=.014 rand p=.013 rand p=.036 rand p=.001 rand p=0.022

Lagged bilateral trade 0.923 0.791 0.920 0.791(0.001) (0.002) (0.001) (0.002)p=.000 p=.000 p=.000 p=.000

Constant -17.580 -15.953 -1.498 -3.509 -17.312 -1.655 -5.161(0.090) (0.097) (0.039) (0.067) (0.259) (0.043) (0.178)p=.000 p=.000 p=.000 p=.000 p=.000 p=.000 p=.000

N=119,640 N=2,346 N=117,294 N=2,346 N=2,346 N=2,346 N=2,346T=51 T=50 T=51 T=50 T=50

R2 0.66 0.87 0.95 0.95 0.87 0.95 0.95

Democracy is the lower value within the dyad.GDP and bilateral trade are in real 1996 dollars and are natural log transformed.Population and distance are natural log transformed.A p-value equal to zero indicates a value smaller than 2.2e-16.

Expanded Trade Data

Table 2: Regression Analysis of Bilateral Trade, 1950-2000

PooledDyad fixed

effectsPooled with dynamics

Dyad fixed effects with dynamics

Dyad and year fixed effects

Year fixed effects with dynamics

Dyad and year fixed effects with

dynamics

Democracy 0.039 0.016 0.003 0.003 0.013 0.004 0.003(0.001) (0.001) (0.000) (0.000) (0.001) (0.000) (0.000)p=.000 p=.000 p=.000 p=.000 p=.000 p=.000 p=2e-12

rand p=.001 rand p=0.011 rand p=.014 rand p=.013 rand p=.036 rand p=.001 rand p=0.022

N=119,640 N=2,346 N=117,294 N=2,346 N=2,346 N=2,346 N=2,346T=51 T=50 T=51 T=50 T=50

R2 0.66 0.87 0.95 0.95 0.87 0.95 0.95DV: bilateral trade in constant 1996 dollarsDemocracy: lower value within the dyadControls omitted: GDP per capita, population, alliance, distance, lagged DV (vary by model)

Randomization Tests!

• Simulate the state of the world if the null hypothesis were true

• Scramble the key independent variable to break the relationship between it and the outcome

• Estimate coefficients and significance tests using preferred model, knowing the “answer” is zero

• Repeat at least 1000 times

• See if the actual estimated effect is rare compared to distribution of spurious effects

• Sample estimate of Fisher’s exact test (1935)

Randomization Tests!

• Drop the country labels on the floor and scramble them. Pick them up randomly and reinsert.

• Albania might become “Bolivia” for example, in all dyads involving true Albania.

• Rescramble 1000 times.

• With random country labels and the equation rerun, the data present a distribution around the null outcome (presumably zero, but with complications). T

• Where in the distribution of observations does the observed value from the actual equation fall? That position in the cumulative distribution yields the p-value with the randomization test.

Advantages of Randomization Tests

• Unlike parametric tests…

– Don’t rely on distributional assumptions about ij

– Don’t rely on distributional assumptions about the test statistic

– Standard errors and significance tests determined empirically

– Especially useful when theoretical standard errors are hard to derive

• Like the democratic trade models!

• But, do assume exchangeability of errors

Figure 1: Randomization tests results

Figure 1 (cont): Randomization tests results

Figure 1 (cont): Randomization tests results

Figure 1 (cont): Randomization tests results

Country-year Models

Figure 2: Dem. trade and regime (residuals)

Residual proportion of trade with other democracies by residual regime Data: nation-year; controls: nation and year dummies.

Y = .03 + .38 X + εstd. err = .044p = .000N= 6459Adj. r2=0.011

Figure 3: Democratic shocks

Democratic trade at tn relative to democratic shock at t0

Figure 4: Democratic shocks

Residualized democratic trade at tn relative to democratic shock at t0

Figure 5: Anti-democratic shocks

Democratic trade at tn relative to anti-democratic shock at t0

Figure 6: Anti-democratic shock

Residualized democratic trade at tn relative to anti-democratic shock at t0

Figure 7: Effect of regime shocks

Change in democratic trade as a function of democratic shocks

2.6 = mean change in democratic trade per year x 6 years

Figure 8: Country analysis

Residual democratic trade over mean regime score (year adjusted) Data: countries

Y = 1.10 X + εst. err = .185p = .000N= 154adj. r2=0.183

Conclusions• Dyads are inappropriate units of analysis for

democratic trade hypothesis:– Randomization tests show hypothesis testing in traditional

regression results are too optimistic(p-values up to 41 trillion times larger!)

• Randomization tests are useful for testing claims, especially when the appropriate parametric test is difficult to figure out

• Move to nation-year as unit of analysis– We find no regime effect

• Less is more:– Event and difference in differences analyses of shocks

seem to confirm democratic trade hypothesis