Multiple Randomization Designs

for Interference

Pat Bajari, Brian Burdick, Guido Imbens#

James McQueen,Thomas Richardson&, & Ido Rosen

( Amazon, # Stanford University, & Washington University)

NABE, ASSA

San Diego, January 3th, 2020

Outline

1. Introduction & Motivation

2. Conventional Randomized Experiments (A/B Tests)

3. Multiple Randomization Designs (MRDs)

4. Analyzing a Simple Double Randomization Design

5. Extensions

6. Conclusion

1

1. Introduction & Motivation

In conventional experiments (A/B tests) units from well-defined population are randomly assigned to treatment.

Average outcomes are compared by treatment status.

Here: Two or more populations, with outcomes indexed byboth (movies/viewers, drivers/riders, properties/renters).

Could choose to randomize units from one of the two popu-lations

But: Could assign at the level of pair movie/viewer.

a Double randomization designs can detect spillovers/interference

a Double randomization designs can adjust for richer patternsof spillovers/interference

2

Example:

Population 1: Movies

Population 2: Viewers

Treatment: provide more information about movie to viewer(e.g., pictures instead of written description, or more detailabout ratings of previous viewers)

Decision: should we implement the treatment for all moviesand viewers or for no one?

Statistical Question to Inform Decision: By how muchwould exposing all movies/viewers to the treatment improveaverage viewer satisfaction?

Key: In experiment we can vary the treatment and measurethe outcome at the level of the movie/viewer pair.

3

2. Conventional Randomized Experiments (A/B Tests)

(goes back to Fisher (1935), Neyman (1928))

a Wi " rC,Tx binary treatment, randomly assigneda Population of N units (e.g., viewers or movies).

a Potential Outcomes defined: Yi�C�,Yi�T�, interest in averageeffect

τ �1N

N

9i�1

�Yi�T� � Yi�C�

a Estimator for average treatment effect:

τ̂ � Y T � Y C

4

3. Double Randomization Designs

Now suppose we have two populations:

a I Movies, i � 1, . . . , I.

a J Viewers, j � 1, . . . , J

Outcomes and treatments are measured for pair movie/viewer:

a Yij is outcome for movie i and viewer j

a Wij " rC,Tx is binary treatment for movie i and viewer jY and W are I � J matrices of outcomes and treatments.

Question: Design of Experiment, choice of distrib p�w�.5

Assignments:

W �

������������

C C C C C CC C C C C CT T T T T TC C C C C CT T T T T T

�����������Movie ExperimentRandomize Movies

W �

������������

C T C C T TC T C C T TC T C C T TC T C C T TC T C C T T

�����������Viewer ExperimentRandomize Viewers

6

Multiple Randomization Design

W �

������������������������

viewers� 1 2 3 4 5 6 7 8movies�

1 C C C C C C C C2 C C C C C C C C3 C C C C T T T T4 C C C C T T T T5 C C C C T T T T

�����������������������Randomize Both Movies and Viewers.

7

Assignment Types defined by consistency of treatment expe-rience:

T �

������������

c c c c c cc c c c c ct t t t t tc c c c c ct t t t t t

�����������

Movie ExperimentRandomize Movies

2 Groups to Compare1 Control Group

T �

������������

c t c c t tc t c c t tc t c c t tc t c c t tc t c c t t

�����������

Viewer ExperimentRandomize viewers

2 Groups to Compare1 Control Group

T �

������������

c c c iv iv ivc c c iv iv iv

im im im t t tim im im t t tim im im t t t

�����������

Double Random. DesignRandomize Both

4 Groups to Compare3 Control Groups to Compare

8

Problems with Movie Experiments

a Viewers do not have a consistent experience, treated for

some movies, not for other movies. They may shift engage-

ment from control movies to treated movies.

Formally: Violations of SUTVA (stable-unit-treatment-value-

assumption, Rubin) because there are spillovers from treating

movie/viewer �i, j� to different movie / same viewer �i¬, j�.If wij � w

¬

ij, but wi¬j j w¬

i¬j (for other movie i¬ but same viewer

j) then it may be that

Yij�w� j Yij�w¬�

9

Main Issues

a Both within movie and withing viewer spillovers/interference

may be present: Viewer or movie experiments cannot de-

tect the presence of such interference.

a Double randomization designs can detect spillovers: More

ex ante comparable groups that may differ systematically

ex post.

a Double randomization designs can adjust for spillovers

that viewer or movie experiments cannot adjust for: but

need to put structure on interference (role for models).

10

Double Randomization Designs

a Randomize movies into NM ' 2 groups with indicator WMi

a Randomize viewers into NV ' 2 groups with indicator WVj

Assignment is

Wij � f �WMi ,WVj For example NM � NV � 2 (Simple Double RandomizationDesign):

W � f �WM,WV ���������

C C C C C CC C C C C CC C C T T TC C C T T T

�������.

a Creates NM�NV (� 4 here) ex ante comparable groups thathave systematically different experiences ex post.

11

4. Analyzing a Simple Double Randomization Design

WMi " r0,1x, WVj " r0,1xWij � f �WMi ,WVj � w T if WMi �WVj � 1,C otherwise.

For example:

W � f �WM,WV ���������

C C C C C CC C C C C CC C C T T TC C C T T T

�������.

12

The pair �WMi ,WVj � defines four types of movie/viewer pairs:

Tij �

~

c �consistent control� if WM � 0,WVj � 0,im �inconsistent movie control� if WM � 1,WVj � 0,iv �inconsistent viewer control� if WM � 0,WVj � 1,t �treated� if WM � 1,WVj � 1.

The assignment and type matrices are

W �

��������C C C C C CC C C C C CC C C T T TC C C T T T

�������

T �

��������c c c c iv iv iv ivc c c c iv iv iv iv

im im im im t t t tim im im im t t t t

�������13

Limits on Spillovers: main assumption

The potential outcomes satisfy the local interference as-

sumption if

Yij�w� � Yij�w¬�,for any pair �i, j�, any w, and any w¬, with wij¬ � w¬ij¬ for allj¬

" J, and wi¬j � w¬

i¬j for all i¬

" I.

To change outcome for movie i and viewer j �Yij�w�� youneed to change treatment either for the same movie,

or for the same viewer (or both).

Could use richer models for spillovers.

14

Assignments with the same outcome for movie 3 and viewer

4 under local interference:

W �

��������C C T T C CC T C C C TT C C T T TT C C C C C

�������

W¬

�

��������T C T T C CT C T C C CT C C T T TT C C C T T

�������

W¬¬

�

��������C C T T T TC C C C T CT C C T T TT T C C T T

�������Implication � Y34�W� � Y34�W¬� � Y34�W¬¬�

Define:

wMi �

1J

J

9j�1

wij, wVj �

1I

I

9i�1

wij.

Assume: There are functions hij � r0,1x ( R, hMij � �0,1� ( Rand hVij � �0,1� ( R such that for all i " I, all j " J, and allw " W,

Yij�w� � hij�wij� � hMij �wMi � hVij �wVj . (1)h

Mij ��� �hVij���� captures movie (viewer) spillovers.

Define class of average effects

τ �pM, pV � 1IJ9i,j

�hij�1��hij�0�� 1IJ9i,j

hMij �pM� 1IJ9

i,j

hVij �pV16

Define the average outcome by movie/viewer pair type, for

s " rc, im, iv, tx:Y s �

1Ns

I

9i�1

J

9j�1

Yij1Tij�s

Estimators:

τ̂ � Y t � Y c estimates ave treatment effect τ �pM, pV�τ̂direct � Y t � Y im � Y iv � Y c estimates τ �0,0�Y im � Y c estimates τ �pM,0� � τ �0,0� �movie spillovers�Y iv � Y c estimates τ �0, pV� � τ �0,0� �viewer spillovers�

Exact variances of estimators: see paper.

17

5. Extensions

A. Multiple Randomization Designs

B. Equilibrium Designs

C. Clustered Double Randomization Designs

D. Synergistic double randomization designs

18

5.A Multiple Randomization Designs

W �

����������������

C C C C C C T TC C C C C C T TC C C C T T T TC C C C T T T TC C T T T T T TC C T T T T T T

���������������.

More ex ante comparable groups exposed to control treat-

ment (six in this example).

¼ More comparisons to shed light on interactions:

more variation in pM and pV in τ �pM, pV�.20

5.B Equilbrium Designs

Suppose a decision maker makes a promotion decision sep-

arately for each movie, taking into account the demand for

that movie given the treatment assignment for all movie/viewer

pairs. Hence the promotion for each movie may depend on

the fraction of viewers in the treatment group Pi � Pi�WMi �.Of particular interest are PTi � Pi�1�, the promotion if (al-most) all the viewers for movie i are exposed to the treat-

ment, and PCi � Pi�0�, the promotion if no or few viewers formovie i are exposed to the treatment.

Viewers make movie renting decisions that depend both on

the promotion and on the treatment status. Denote the

renting decision by viewer j for movie i, given price p and

given the treatment w " rC,Tx as Yij�w, p�.21

Experimental Design

First we split the viewers randomly into two groups, with

fractions pVA and pVB � 1 � p

VA. Let T

Vj " rA,Bx denote the

assignment for viewers to these two groups.

For the first group of viewers (viewers with assignment TVj �

A) we do a movie experiment. Let WMi " rC,Tx be theassignment for the movies in this experiment.

For the second group of viewers (viewers with assignment

TCj � B) we do a viewer experiment, with assignment W

Vj "rC,Tx for the viewers in this experiment.

22

For example (movies are rows, viewers are columns):

W �

�����������������������������������������������������

Movie Experiment V iewer Experimentviewers� 1 2 3 4 5 6 7 8 9 10 11 12 13 14

A A A A A A A A B B B B B Bmovies�

1 C C C C C C C C T C T C C T2 C C C C C C C C T C T C C T3 T T T T T T T T T C T C C T4 C C C C C C C C T C T C C T5 T T T T T T T T T C T C C T6 T T T T T T T T T C T C C T7 C C C C C C C C T C T C C T8 T T T T T T T T T C T C C T9 T T T T T T T T T C T C C T10 C C C C C C C C T C T C C T

����������������������������������������������������23

There are three sets of movie/viewer pairs that receive the

control treatment but have systematically different environ-

ments:

1. TCj � B, WVj � C, W

Mi � C (red C) in Viewer experiment,

assigned to control group, and the movie in the Movie

experiment is in the control group.

2. TCj � B, WVj � C, W

Mi � T (blue C) in Viewer experiment,

assigned to control group, but the movie in the Movie

experiment is in the treatment group.

3. TCj � A, WMi � C, (green C): in Movie experiment, as-

signed to control group.

24

Compare the red C and the blue C outcomes:

This comparison reflects on the (equilibrium) spillover effect

of other viewers for the same movie being in the treatment

(versus control) group.

The comparison between the average outcomes for these two

sets of pairs of viewers/movies is informative about the indi-

rect effect of the treatment on purchases through the effect

of average treatments on prices, that is, it is estimating an

average of Yij�C, PTi � � Yij�C, PCi �.

25

5.C Clustered Double Randomization Designs

a Suppose movies are clustered in groups.

a A regular movie experiment would be:

w �

���������������������������������

cluster�

1 C C C C C C C C1 T T T T T T T T2 T T T T T T T T2 T T T T T T T T3 C C C C C C C C3 T T T T T T T T4 C C C C C C C C4 T T T T T T T T

��������������������������������where the first column denotes the movie-cluster.

Such an experiment may suffer from biases if there are treat-ment effects that affect all movies within a cluster.

26

A cluster-randomized movie experiment would be

w �

���������������������������������

cluster�

1 C C C C C C C C1 C C C C C C C C2 T T T T T T T T2 T T T T T T T T3 C C C C C C C C3 C C C C C C C C4 T T T T T T T T4 T T T T T T T T

��������������������������������a Such a cluster-randomized movie experiment would be ro-

bust to interference within clusters of movies.

a It will not inform us about the presence of such interference.

27

We could combine this with double randomization in a num-

ber of ways.

5.C.1 We could do separate cluster-randomized movie ex-

periments for two sets of viewers:

w �

���������������������������������

A A A A B B B B

1 C C C C T T T T1 C C C C T T T T2 T T T T T T T T2 T T T T T T T T3 C C C C C C C C3 C C C C C C C C4 T T T T C C C C4 T T T T C C C C

��������������������������������a This would be informative about within-movie spillovers.

28

5.C.2 We could combine a cluster-randomized movie exper-

iments for one sets of viewers with an movie experiment for

a second set of viewers:

w �

���������������������������������

A A A A B B B B

1 C C C C T T T T1 C C C C C C C C2 T T T T T T T T2 T T T T T T T T3 C C C C T T T T3 C C C C C C C C4 T T T T C C C C4 T T T T C C C C

��������������������������������a This would be informative about within-movie spillovers.

29

5.D Synergistic Double Randomization Designs

W �

����������������

C C C C C C CC C C C C C CC C C C C T TC C C C C T TC C T T T C CC C T T T C C

���������������T �

����������������

c c iv iv iv iv ivc c iv iv iv iv iv

im im imv imv imv t tim im imv imv imv t tim im t t t imv imvim im t t t imv imv

���������������Here there are four sets of movie/viewer pairs that are ex-

posed to the control treatment:

1. consistent controls (c): movie/viewers where the movies

are always exposed to the control treatment, and viewers

who are always exposed to the control treatment.

30

2. inconsistent movie controls (im): movie/viewers where

the movies are exposed to the new treatment for other

viewers, but the viewers are always exposed to the control

treatment

3. inconsistent viewer controls (iv): movie/viewers where

the movies are always exposed to the control treatment,

but the viewers are exposed to the new treatment for

some other movies.

4. inconsistent viewer and movie controls (imv): movie/viewers

exposed to the control treatment where both the movies

and viewers have an inconsistent experience.

6. Conclusion

a In settings with multiple populations more complex experi-

mental designs are possible.

a Such designs (e.g. multiple randomization designs) can

answer more questions about interference/spillovers than con-

ventional designs by creating multiple comparison groups.

a Opens up lots of design questions.

a Opens up lots of inference questions.

a Important role for modeling (limits on) spillovers.

31