2014-10-22 eugm | proschan | blinded adaptations permutations tests and t-tests

Blinded Adaptations, Permutation Tests & T‐Tests

Michael Proschan (NIAID)Michael Proschan (NIAID)

IntroductionIntroduction

• Joint work with Ekkehard Glimm and MartinJoint work with Ekkehard Glimm and Martin Posch 2014, Stat. in Med. online

• See also Posch & Proschan 2012, Stat. in Med. 31 4146 415331, 4146‐4153


• Clinical trials are pre‐meditated!Clinical trials are pre meditated!• We pre‐specify everything

S i it / i f i it– Superiority/noninferiority– Population (inclusion/exclusion criteria)– Primary endpoint– Secondary endpoints– Analysis methods– Sample size/power


• Changes made after seeing data are rightlyChanges made after seeing data are rightly questioned: are investigators trying to get an unfair advantage?– Changing primary endpoint because another endpoint has a bigger treatment effect

– Increasing sample size because the p‐value is close– Changing primary analysis because “assumptions are violated”are violated

– Changing population because of promising subgroup resultssubgroup results


• What’s the harm? 0 05 is arbitrary anywayWhat s the harm? 0.05 is arbitrary anyway• Problem: if unlimited freedom to change anything the real error rate could be hugeanything, the real error rate could be huge

• Reminiscent of Bible code controversy– Clairvoyant messages such as “Bin Laden” and “twin towers” by skipping letters in Old Testament

– Similar messages can be found by skipping letters in any large book (Brendan McKay)


• But changes made before unblinding areBut changes made before unblinding are different

• Under strong null hypothesis that treatment• Under strong null hypothesis that treatment has NO effect, blinded data give no info about treatment effecttreatment effect– Impossible to cheat even if it seems like cheating

E if bli d d d t h bi d l di t ib ti it• E.g., even if blinded data show bimodal distribution, it is not caused by treatment if strong null is true

Permutation TestsPermutation Tests

• Permutation tests condition on all data otherPermutation tests condition on all data other than treatment labels

• Under strong null (D Z ) are independent• Under strong null, (D,Z ) are independent, where Z are ±1 treatment indicators & D are datadata – Observed data D would have been observed regardless of the treatment givenregardless of the treatment given

– It is as if we observed D FIRST, then made the treatment assignments Ztreatment assignments Z


• Peaking at data changes nothing becausePeaking at data changes nothing because permutation tests already condition on D

• Conditional distribution of test statistic T(Z Y)• Conditional distribution of test statistic T(Z,Y) given D is that of T(Z,y) where y is fixedDi ib i f Z d d d i i• Distribution of Z depends on randomization method – Simple– Permuted block, etc.


T T C C C T C T C C T T C T T C

4 8 4 0 1 3 0 4 4 0 2 5 0 2 1 0

T-C T-C T-C T-C

O ll T C

4.0 3.0 1.5 1.5

Overall T-C2.5


T C C T C T C T T T C C C T C T

4 8 4 0 1 3 0 4 4 0 2 5 0 2 1 0

T-C T-C T-C T-C

O ll T C

-4.0 3.0 -1.5 0.5

Overall T-C-0.5

Rerandomization DistributionPermutation Distribution10

0

y

80

Freq

uenc

y

060

040

02

11T-C Mean

-3 -2 -1 0 1 2 3

Blinded 2‐Stage ProceduresBlinded 2 Stage Procedures

• Blinded 2‐stage adaptive procedures use 1stBlinded 2 stage adaptive procedures use 1st stage to make design changes– Sample size (Gould, 1992, Stat. in Med. 11, 55‐66; p ( , , , ;Gould & Shih, 1992 Commun. in Stat. 21, 2833‐2853) P i d i ( di li li– Primary endpoint (e.g., diastolic versus systolic blood pressure)

• Previous argument shows that if adaptation is• Previous argument shows that if adaptation is made before unblinding, a permutation test on 1st stage data is still validon 1st stage data is still valid


• Careful! Subtle errors are possibleCareful! Subtle errors are possible• E.g., in adaptive regression, which of the following is (are) valid?following is (are) valid?1. From ANCOVAs Y=β01+βz+βixi, i=1,…,k, pick xi

that minimizes MSE; do permutation test onthat minimizes MSE; do permutation test on winner

2 From ANCOVAs Y=β 1+β x i=1 k pick x that2. From ANCOVAs Y=β01+βixi, i=1,…,k, pick xi that minimizes MSE; do permutation test on Y=β01+βz+β*x*, where x* is winnerβ0 β β ,


• Unblinding and apparent α‐inflation also possible U b d g a d appa e t α at o a so poss b eif strong null is false

• E.g., change primary endpoint based on “blinded” g g p y pdata (X,Y1,Y2), Y1 and Y2 are potential primaries and X=level of study drug in blood– X completely unblinds– Can then pick Y1 or Y2 with biggest z‐scoreClearly inflates α– Clearly inflates α

– Problem: strong null requires no effect on ANYvariable examined (including X=level of study drug)


• Claim: the following procedure is validClaim: the following procedure is valid– After viewing 1st stage data D1, choose test statistic T1(Y1 Z1) and second stage data to collectstatistic T1(Y1,Z1) and second stage data to collect

– After observing D2, choose T2(Y2,Z2) and method of combining T1 and T2, f(T1,T2)of combining T1 and T2, f(T1,T2)

– Conditional distribution of f(T1,T2) given (D1,D2) is its stratified permutation distributionp

– Stratified permutation test controls conditional, & therefore unconditional type I error rate

Focus of Rest of TalkFocus of Rest of Talk

• Permutation tests are asymptoticallyPermutation tests are asymptotically equivalent to t‐tests

• Suggests that adaptive t tests might be valid if• Suggests that adaptive t‐tests might be valid if adaptive permutation tests areW id i b• We consider connections between permutation and t‐tests, and validity of d i f d i iadaptive t‐tests from adaptive permutation tests

One‐Sample CaseOne Sample Case

• Community randomized trials sometimes pair Co u ty a do ed t a s so et es pamatch & randomize within pairs

• E.g., COMMIT trial used community intervention g yto help people quit smoking—11 matched pairs

• D=difference in quit rates between treatment (T) & control (C)

T C D=T‐CPair i 0.30 0.25 +0.05


• Community randomized trials sometimes pair Co u ty a do ed t a s so et es pamatch & randomize within pairs

• E.g., COMMIT trial used community intervention g yto help people quit smoking—11 matched pairs

• D=difference in quit rates between treatment (T) & control (C)

C T D=T‐CPair i 0.30 0.25 ‐0.05


• Permuting labels changes only sign of DPermuting labels changes only sign of D• Permutation test conditions on |Di|= di+;

d + d d + ll lik l‐di+ and di+ are equally likely

• The permutation distribution of Di is dist. ofThe permutation distribution of Di is dist. of

21w p1where /ZdZ

21 w.p.1

21 w.p.1where,

/

/ZdZ iii


• In 1st stage, adapt based on |D1|,…,|Dn| (blinded)g , p | 1|, ,| n| ( )– E.g., increase stage 2 sample size because |Di| is very large

• What is conditional distribution of 1st stage sum• What is conditional distribution of 1st stage sum ΣDi given |D1|=d1+,…,|Dn|= dn+ and the adaptation?adaptation?– The adaptation is a function of |D1|,…,|Dn|

– The null distribution of ΣDi given |D1|=d1+,…,|Dn|= dn+i g | 1| 1 , ,| n| nIS its permutation distribution

– Conclusion: permutation test on stage 1 data still valid


• Mean and variance of permutationMean and variance of permutation distribution are

0)(E iiii ZEddZ

222 )(var

)(

iiiii

iiii

dZEddZ


• Asymptotically, permutation distribution is sy ptot ca y, pe utat o d st but o snormal with this mean and variance (Lindeberg‐Feller CLT)

• I.e., conditional distribution of Di given , i g|D1|=d1+,…,|Dn|= dn+ is asymptotically N(0,di2)

• Depends on |D1|=d1+,…,|Dn|= dn+ only through L2=di2L di


• Asymptotically, permutation distribution ofAsymptotically, permutation distribution of

Nd

dN

D

DT ii

2

2

2)1,0(

,0'

LD

dD ii

2

nLDns

ns

DT i

i2

2202

0

)/1( ;'

• Like t‐test with variance estimate s02 instead of usual sample variance s2


• Recap: Permutation distribution of T’ is dist ofRecap: Permutation distribution of T is dist of

12

|||,...,| given ' ni DD

D

DT

2i'

i

DT

D

22

2

d dtd ')10(

given ' i

DLN

DT

• Conclusion: T’ is asymptotically indep of L2 22ondependt doesn' )1,0( iDLN


• Begs question, is this true for all sample sizesBegs question, is this true for all sample sizes under normality assumption?

• if Di are iid N(0,2), then canif Di are iid N(0, ), then can

?fti d db' 2 i DD

T ?oft independenbe ' 22

i

i

i DD

T

• Seems crazy, but it’s true!


• One way to see that T’ is independent of Di2One way to see that T is independent of Di

uses Basu’s theorem: • Recall S is sufficient for θ if F(y|s) does not d d θ i i l if { ( )} f ll θdepend on θ; it is complete if E{g(S)}=0 for all θimplies g(S)≡0 with probability 1

• A is ancillary if its distribution does not depend• A is ancillary if its distribution does not depend on θ

• Basu, 1955, Sankhya 15, 377‐380:If S is a complete, sufficient statistic and A is ancillary, then S and A are independent


• Consider Di iid N(0 2) with 2 unknownConsider Di iid N(0, ) with unknown

–Di2 is complete and sufficient

– T’= Di/(Di2)1/2 is ancillary because it is scale‐

invariant

– By Basu’s theorem, T’ and Di2 are independent


• Same argument shows that the usual t‐Same argument shows that the usual tstatistic is independent of Di

2

2 2• Under Di iid N(0,2) with 2 unknown–Di

2 is complete and sufficient

– Usual t‐statistic T= Di/(ns2)1/2 is ancillary

– By Basu’s theorem T and D 2 are independent– By Basu s theorem, T and Di are independent ( Shao (2003): Mathematical Statistics, Springer)


• This result is important for adaptive sample sizeThis result is important for adaptive sample size calculations– Stage 1 with n1= half of original sample size: changeStage 1 with n1 half of original sample size: change second stage sample size to n2=n2(ΣDi

2)

– Conditioned on ΣD 2:– Conditioned on ΣDi : • Test statistic T1 has exact t‐distribution with n1‐1 d.f.• Test statistic T2 has exact t‐distribution with n2‐1 d.f. and is 2 2independent of T1

• P‐values P1 and P2 are independent U(0,1)• Y={n 1/2Φ‐1(P )+n 1/2Φ‐1(P )}/(n +n )1/2 is N(0 1) under H• Y={n11/2Φ 1(P1)+n21/2Φ 1(P2)}/(n1+n2)1/2 is N(0,1) under H0


• Reject if Y>zReject if Y>zα• Conditioned on ΣDi

2, type I error rate is α• Unconditional type I error rate is α as well• Most other two‐stage procedures are onlyMost other two stage procedures are only approximate


• Could even make other adaptations like changing p g gprimary endpoint

• Look at ΣDi2 for each endpoint and determine

which one is primary 2– E.g., pick endpoint with smallest Di

2

• Slight generalization of our result shows that• Slight generalization of our result shows that conditional distribution of T given adaptation is still exact t


• Shows that conditional type I error rate givenShows that conditional type I error rate given adaptation is controlled at level α

• Unconditional type I error rate must also be• Unconditional type I error rate must also be controlled at level αD i i l i i li• Derivation assumes multivariate normality with variance/covariance not depending on mean

Two‐Sample CaseTwo Sample Case

• Can use same reasoning in 2‐sample setting Ca use sa e easo g sa p e sett g• With equal sample sizes, the numerator is

YZYY

• Permutation distribution is distribution of

iiC

iT

i YZYY

Permutation distribution is distribution of

0 ,1 each , iiii ZZyZ

• Let sL2 be “lumped” variance of all data (treatment and control)(treatment and control)


• Mean and variance of permutation distribution pare

0)(EE iiii ZyyZ 22)(

11var Lii syy

nyZ

• Basu’s theorem shows usual 2‐sample T is independent of sL2 under null hypothesis ofindependent of sL under null hypothesis of common mean

• Conditional distribution of T given sL2 is still t


• Two‐stage procedureTwo stage procedure– Stage 1: look at lumped variance and change stage 2 sample size

– Conditioned on 1st stage lumped variance & H0• T1 has t‐distribution with n1‐2 d.f.• T2 has t‐distribution with n2‐2 d.f. & independent of T1• P‐values P1 and P2 are independent uniforms• {n11/2Φ‐1(P1)+n21/2Φ‐1(P2)}/(n1+n2)1/2 is N(0 1) under H0{n1 Φ (P1)+n2 Φ (P2)}/(n1+n2) is N(0,1) under H0

– Controls type I error rate conditionally and unconditionally

SummarySummary

• Permutation tests are often valid even inPermutation tests are often valid even in adaptive settings if blind is maintained

• There is a close connection between• There is a close connection between permutation tests and t‐testsC d d lidi f d i f• Can deduce validity of adaptive t‐tests from validity of adaptive permutation tests

2014-10-22 eugm | proschan | blinded adaptations permutations tests and t-tests

Health & Medicine

tc tc tc tc o ll t c

overall tc

tc mean

t ib ti

bli d d d t h bi d

f z d d d

strong null d z

data changes