probabilities bayesian predictive

38
RCTdesign Introduction Bayesian operating characteristics Coarsened Bayes approach Implementation in RCTdesign Posterior probability scale Bayesian contour plots in RCTdesign Bayesian Predictive Probabilities UW Grp Seq - Sec 4 - pg 1 Sequential Monitoring of Clinical Trials Session 4 - Bayesian Evaluation of Group Sequential Designs Presented August 8-10, 2012 Daniel L. Gillen Department of Statistics University of California, Irvine John M. Kittelson Department of Biostatistics & Informatics University of Colorado Denver c 2012 Daniel L. Gillen, PhD and John M. Kittelson, PhD

Upload: others

Post on 19-Dec-2021

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 1

Sequential Monitoring of ClinicalTrialsSession 4 - Bayesian Evaluation of Group Sequential Designs

Presented August 8-10, 2012

Daniel L. GillenDepartment of Statistics

University of California, Irvine

John M. KittelsonDepartment of Biostatistics & Informatics

University of Colorado Denver

c©2012 Daniel L. Gillen, PhD and John M. Kittelson, PhD

Page 2: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 2

Bayesian paradigm

Bayesian Operating Characteristics

I Thus far, we have primarily focused on the evaluation of aclinical trial design with respect to frequentist operatingcharacteristics

I type I errorI statistical powerI sample size requirementsI estimates of treatment effect that correspond to early

terminationI precision of confidence intervals

I However, there has been much interest in the design andanalysis of clinical trials under a Bayesian paradigm

Page 3: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 3

Bayesian paradigm

Bayesian Operating Characteristics

I In the Bayesian paradigm, we consider a joint distributionp(θ, X ) for the treatment effect parameter θ and the clinicaltrial data X .

I We most often specify the prior distribution pθ(θ) and thelikelihood function pX |θ(X |θ), rather than specifying thejoint distribution p(θ, X )

I The prior distribution pθ(θ) represents the information aboutθ prior to (in the absence of) any knowledge of the value ofX

I From a clinical trial, we observe data X = x and baseinference on the posterior distribution pθ|X (θ|X = x),where by Bayes rule

pθ|X (θ|X = x) =pX |θ(X |θ) pθ(θ)∫pX |θ(X |θ) pθ(θ) dθ

Page 4: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 4

Bayesian paradigm

Bayesian Operating Characteristics

I As with frequentist inference, we are interested in:

I point and interval estimates of a treatment effect,

I a measure of strength of evidence for or against particularhypotheses, and perhaps

I a binary decision for or against some hypothesis.

Page 5: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 5

Bayesian paradigm

Bayesian Operating Characteristics

I Bayesian inferential quantities include:

1. Point estimates of treatment effect that are summarymeasures of the posterior distribution such as:

1.1 the posterior mean : E(θ|X = x),

1.2 the posterior median : θ0.5 such that

Pr(θ ≤ θ0.5|X = x) ≥ 0.5 and Pr(θ ≥ θ0.5|X = x) ≥ 0.5,

1.3 the posterior mode : θm such that

pθ|X (θm|X = x) ≥ pθ|X (θ|X = x) for all θ

Page 6: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 6

Bayesian paradigm

Bayesian Operating Characteristics

2. Interval estimates of treatment effect that are computed byfinding two values (θL, θU) such that

Pr(θL ≤ θ ≤ θU |X = x) = 100(1 − α)%

2.1 the central 100(1− α)% of the posterior distribution of θ isdefined by finding some ∆ such that θL = θ̂ −∆ andθU = θ̂ + ∆ provides the desired coverage probability, whereθ̂ is one of the Bayesian point estimates of θ

2.2 the interquantile interval is defined by defining θL = θα/2 andθU = θ1−α/2, where θp is the p-th quantile of the posteriordistribution, i.e., Pr(θ ≤ θp|X = x) = p

2.3 the highest posterior density (HPD) interval is defined byfinding some threshold cα such that the choices

θL = min{θ : pθ|X (θ|X = x) > cα} and

θU = max{θ : pθ|X (θ|X = x) > cα}

provide the desired coverage probability

Page 7: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 7

Bayesian paradigm

Bayesian Operating Characteristics

3. Posterior probabilities or Bayes factors associated withspecific hypotheses that might be used to make a decisionfor or against a particular hypothesis.

3.1 For instance, in the sepsis trial example, we might beinterested in computing the posterior probability of the nullhypothesis Pr(θ ≥ 0|X = x) or the posterior probability of thedesign alternative Pr(θ ≤ −0.07|X = x).

Page 8: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 8

Bayesian paradigm

Bayesian Operating Characteristics

I A note on Bayesian inference under group sequentialtesting

I The Bayesian inference presented on the previous slides isunaffected by the choice of stopping rule, so long as there isno need to consider the joint distribution of estimates acrossthe multiple analyses of the accruing data

I So long as one is content to regard inference at each analysismarginally, then the stopping rule used to collect the data isimmaterial

I However, the expected cost of a clinical trial does dependvery much on the stopping rule used, even when Bayesianinference is used as the basis for a decision

Page 9: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 9

Bayesian paradigm

Bayesian posterior probability scale in RCTdesign

I In considering the Bayesian approach, we maintain thestance that the derivation of the stopping rule is relativelyunimportant.

I There is a 1:1 correspondence between stopping rulesdefined for frequentist statistics (on a variety of scales)and Bayesian statistics for a specified prior

I Specifically, in RCTdesign we consider a robust approachto Bayesian inference based on a coarsening of the databy using the asymptotic distribution of a nonparametricestimate of treatment effect

I For example, in the case of the sepsis trial, we use theapproximate normal distribution for the difference in 28 daymortality rates

θ̂ ≡ p̂1 − p̂0∼̇N(

θ,p1(1− p1) + p0(1− p0)

N

)

Page 10: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 10

Bayesian paradigm

Bayesian posterior probability scale in RCTdesign

I Reasoning behind the “coarsened” Bayesian approach

1. Robustness of inference to misspecification of the likelihood

2. No true conflict between frequentist and Bayesian inference.Instead, they merely answer different questions.

2.1 Role of statistics is to help quantify the strength of evidenceused to convince the scientific community

2.2 Reasonable people might demand evidence:

(1) demonstrating results that would not typically be obtainedunder any other hypothesis (frequentist), or

(2) that overpowers his/her prior beliefs

2.3 To address these different standards of proof within the samesetting, it would seem most appropriate to use the sameprobability model in each approach

Page 11: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 11

Bayesian paradigm

Bayesian posterior probability scale in RCTdesign

I Reliance on the asymptotic distribution of the estimatorimplies that a normal prior is conjugate andcomputationally convenient

θ ∼ N(ζ, τ2)

I Thus we can define a Bayesian posterior probabilitystatistic by computing the approximate posteriorprobability that the null hypothesis H0 : θ ≥ θ0 is false

Bj(ζ, τ2, θ0) = Pr(θ ≤ θ0 |Sj = sj)

= Φ

(θ0[Njτ

2 + V ]− τ2sj − V ζ√V τ√

Njτ2 + V

),

where V = p0(1− p0) + p1(1− p1) and Φ(z) is thecumulative distribution function for a standard normalrandom variable

Page 12: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 12

Bayesian paradigm

Bayesian posterior probability scale in RCTdesign

I Reliance on the asymptotic distribution of the estimatorimplies that a normal prior is conjugate andcomputationally convenient

θ ∼ N(ζ, τ2)

I Thus we can define a Bayesian posterior probabilitystatistic by computing the approximate posteriorprobability that the null hypothesis H0 : θ ≥ θ0 is false

Bj(ζ, τ2, θ0) = Pr(θ ≤ θ0 |Sj = sj)

= Φ

(θ0[Njτ

2 + V ]− τ2sj − V ζ√V τ√

Njτ2 + V

),

where V = p0(1− p0) + p1(1− p1) and Φ(z) is thecumulative distribution function for a standard normalrandom variable

Page 13: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 13

Case Study : Sepsis Trial

Boundaries on various design scales

I Bayesian posterior probabilities for observed estimates onthe Futility.8 boundary

I Computed using a optimistic (ζ = −0.09) dogmatic prior(τ2 = 0.0152) prior

> changeSeqScale(Futility.8,seqScale("B",priorTheta=-0.09, priorVariation=0.015^2))

STOPPING BOUNDARIES: Bayesian scale(Posterior probability of hypotheses based on priordistribution having median -0.09and variation parameter 0.000225)

a dTime 1 (N= 425) 1 0.7954Time 2 (N= 850) 1 0.8239Time 3 (N= 1275) 1 0.8361Time 4 (N= 1700) 1 0.8423

Page 14: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 14

Case Study : Sepsis Trial

Boundaries on various design scales

I Example interpretation (1st analysis, efficacy bound (a) ):Under the optimistic dogmatic prior, after sampling 425subjects if the observed rate in the treatment arm were16.97% lower than the control arm, the posteriorprobability of the event θ < 0 is computed to be approx 1

I Example interpretation (1st analysis, futility bound (d) ):Under the optimistic dogmatic prior, after sampling 425subjects if the observed rate in the treatment arm were4.73% lower than the control arm, the posterior probabilityof the event θ < −0.0866 is computed to be 0.7954

Page 15: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 15

Case Study : Sepsis Trial

Boundaries on various design scales

I Bayesian posterior probabilities for observed estimates onthe Futility.8 boundary

I Computed using a pessimistic (ζ = 0.02) dogmatic prior(τ2 = 0.0152) prior

> changeSeqScale(Futility.8,seqScale("B",priorTheta= 0.02, priorVariation=0.015^2))

STOPPING BOUNDARIES: Bayesian scale(Posterior probability of hypotheses based on priordistribution having median 0.02and variation parameter 0.000225)

a dTime 1 (N= 425) 0.5240 1Time 2 (N= 850) 0.5228 1Time 3 (N= 1275) 0.5218 1Time 4 (N= 1700) 0.5208 1

Page 16: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 16

Case Study : Sepsis Trial

Boundaries on various design scales

I Bayesian posterior probabilities for observed estimates onthe Futility.8 boundary

I Computed using a consensus (ζ = −0.04) dogmatic prior(τ2 = 0.0152) prior

> changeSeqScale(Futility.8,seqScale("B",priorTheta=-0.04,priorVariation=0.015^2))

STOPPING BOUNDARIES: Bayesian scale(Posterior probability of hypotheses based on priordistribution having median -0.04and variation parameter 0.000225)

a dTime 1 (N= 425) 0.9999 1.0000Time 2 (N= 850) 0.9999 1.0000Time 3 (N= 1275) 0.9997 0.9999Time 4 (N= 1700) 0.9996 0.9999

Page 17: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 17

Case Study : Sepsis Trial

Boundaries on various design scales

I Note 1: By default, seqScale() will compute theposterior probability of the hypothesis being tested foreach boundary

I seqDesign() defaults to a symmetric test so that thefutility boundary tests the (1−α) power point for with size α

I For the Futility.8 design, this is the 97.5% power pointcorresponding to θ = −0.0866

I To see this, one can view the Futility.8$hypothesis

> Futility.8$hypothesis

HYPOTHESES:Theta is difference in probabilities (Treatment - Comparison)One-sided hypothesis test of a lesser alternative:

Null hypothesis : Theta >= 0.00 (size = 0.0250)Alternative hypothesis : Theta <= -0.07 (power = 0.8888)

Boundary hypotheses:Boundary a rejects : Theta >= 0.0000 (lower size = 0.025)Boundary b rejects : Theta <= -0.0866 (lower power = 0.975)Boundary c rejects : Theta >= 0.0000 (upper power = 0.975)Boundary d rejects : Theta <= -0.0866 (upper size = 0.025)

Page 18: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 18

Case Study : Sepsis Trial

Boundaries on various design scales

I Note 2: An asymmetric boundary can be obtained bymodifying the alpha option in seqDesign()

I As this changes the hypotheses being tested at eachanalysis, it will change the boundaries and operatingcharacteristics of the stopping rule

I Note 3: To simply compute a different posterior probability,you can use the threshold option in seqScale()

changeSeqScale(Futility.8,seqScale("B", threshold=-0.06,

priorTheta=-0.09,priorVariation=0.015^2))

STOPPING BOUNDARIES: Bayesian scale(Posterior probability that treatment effect exceeds -0.06based on prior distribution having median -0.09and variation parameter 0.000225)

a dTime 1 (N= 425) 0.0031 0.1461Time 2 (N= 850) 0.0155 0.1471Time 3 (N= 1275) 0.0509 0.1365Time 4 (N= 1700) 0.1225 0.1225

Page 19: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 19

Case Study : Sepsis Trial

Boundaries on various design scales

I Bayesian posterior probabilities for observed estimates onthe Futility.8 boundary

I Computed using a non-informative prior (τ =∞)

> changeSeqScale(Futility.8,seqScale("B",priorTheta= 0.02,priorVariation=Inf))

STOPPING BOUNDARIES: Bayesian scale(Posterior probability of hypotheses based on priordistribution having median 0.02and variation parameter Inf)

a dTime 1 (N= 425) 1.0000 0.9991Time 2 (N= 850) 0.9975 0.9946Time 3 (N= 1275) 0.9891 0.9880Time 4 (N= 1700) 0.9766 0.9808

Page 20: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 20

Case Study : Sepsis Trial

Sensitivity of posterior probabilities to prior

I Bayesian inference can depend heavily on the choice ofprior distribution pθ(θ) for the treatment effect parameter

I For that reason, Bayesian inferential procedures havesometimes been criticized because it is not clear how theprior distribution should be selected for any particularproblem

Page 21: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 21

Case Study : Sepsis Trial

Sensitivity of posterior probabilities to prior

I The Bayesian evaluation of stopping rules proceeds muchthe same as for the evaluation of stopping rules withrespect to frequentist inference

I The major difference relates to the magnitude of theresults that need to be presented

I When evaluating a stopping rule with respect to frequentistinference, we present the estimate, confidence interval, andP value for clinical trial results corresponding to the stoppingboundaries

I For Bayesian inference, we must consider how thatinference is affected by the choice of prior

Page 22: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 22

Case Study : Sepsis Trial

Sensitivity of posterior probabilities to prior

I In RCTdesign, the approach we take is to use a spectrumof normal prior distributions specified by their mean andstandard deviation

I Normal prior is specified entirely by two parameters, greatlyreducing the dimension of the space of prior distributions tobe considered

I Means and standard deviations are commonly used andunderstood by many researchers

I Normal distribution is known to maximize entropy over theclass of all priors having the same first two moments

I Potentially deleterious if an individual’s well-based prior ismore informative than the normal prior with the same meanand standard deviation

I In this case, an analysis with a normal prior may suggest thatthe data has overwhelmed an investigator’s initial belief, whenit in fact has not

Page 23: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 23

Case Study : Sepsis Trial

Sensitivity of posterior probabilities to prior

Bayesian Evaluation of Group Sequential Designs - 05/01/2006, 6

Table 1: Posterior probabilities of hypotheses for trial results corresponding to stopping boundaries of Futility.8 stopping rulewith four equally spaced analyses after 425, 850, 1275, and 1700 subjects have been accrued to the study. Posterior probabilitiesare computed based on optimistic, the sponsor’s consensus, and pessimistic centering of the priors using three levels of assumedinformation in the prior. The variability of the likelihood of the data corresponds to the alternative hypothesis: event rates of

0.30 in the control group and 0.23 in the treatment group.

Efficacy (lower) Boundary Futility (upper) Boundary

Posterior Probability of Posterior Probability ofBeneficial Treatment Effect Insufficient Benefit

Pr(θ < 0|X) Pr(θ ≥ −0.087|X)

Crude Est Sponsor’s Crude Est Sponsor’sAnalysis of Trt Optimistic Consensus Pessimistic of Trt Optimistic Consensus Pessimistic

Time Effect ζ = −.09 ζ = −.04 ζ = .02 Effect ζ = −.09 ζ = −.04 ζ = .02

Dogmatic Prior: τ = 0.0151:N=425 -0.170 1.000 1.000 0.524 0.047 0.795 1.000 1.0002:N=850 -0.085 1.000 1.000 0.523 -0.010 0.824 1.000 1.0003:N=1275 -0.057 1.000 1.000 0.522 -0.031 0.836 1.000 1.0004:N=1700 -0.042 1.000 1.000 0.521 -0.042 0.842 1.000 1.000

Consensus Prior: τ = 0.0401:N= 425 -0.170 1.000 1.000 0.991 0.047 0.981 0.999 1.0002:N= 850 -0.085 1.000 0.998 0.974 -0.010 0.976 0.997 1.0003:N=1275 -0.057 0.999 0.993 0.955 -0.031 0.970 0.994 1.0004:N=1700 -0.042 0.998 0.987 0.936 -0.042 0.963 0.991 0.999

Noninformative Prior: τ = ∞1:N= 425 -0.170 1.000 1.000 1.000 0.047 0.999 0.999 0.9992:N= 850 -0.085 0.998 0.998 0.998 -0.010 0.995 0.995 0.9953:N=1275 -0.057 0.989 0.989 0.989 -0.031 0.988 0.988 0.9884:N=1700 -0.042 0.977 0.977 0.977 -0.042 0.981 0.981 0.981

interval estimates of a treatment effect, a measure of strength of evidence for or against particular

hypotheses, and perhaps a binary decision for or against some hypothesis. Commonly used Bayesian

inferential quantities include:

1. Point estimates of treatment effect that are summary measures of the posterior distri-

bution such as the posterior mean (E(θ|X = x)), the posterior median (θ0.5 such that

Pr(θ < θ0.5|X = x) ≥ 0.5 and Pr(θ ≥ θ0.5|X = x) ≥ 0.5), or the posterior mode (θm such

that pθ|X(θm|X = x) ≥ pθ|X(θ|X = x) for all θ).

2. Interval estimates of treatment effect that are computed by finding two values (θL, θU ) such

that Pr(θL < θ < θU |X = x) = 100(1 − α)%. Various criteria can be used to define such

“credible intervals”:

(a) the central 100(1 − α)% of the posterior distribution of θ is defined by finding some ∆

such that θL = θ̂ − ∆ and θU = θ̂ + ∆ provides the desired coverage probability, where

θ̂ is one of the Bayesian point estimates of θ.

Page 24: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 24

Case Study : Sepsis Trial

Sensitivity of posterior probabilities to prior

I As is often the case, the optimistic and pessimistic priorspresented were chosen rather arbitrarily, and thus may notbe relevant to some of the intended audience for thepublished results of a clinical trial

I As such, it may be beneficial to present contour plots ofBayesian point estimates (posterior means), lower andupper bounds of 95% credible intervals, and posteriorprobabilities of the null and alternative hypotheses for aspectrum of prior distributions

I In RCTdesign, such plots can be produced withseqBayesContour()

Page 25: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 25

Case Study : Sepsis Trial

Contour plot of posterior median conditional upon stopping atanalysis 2 for futility (observed statistic on boundary)

−0.10 −0.08 −0.06 −0.04 −0.02 0.00 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r S

D fo

r θ

0.990 0.900 0.500 0.200 0.025Power (lower) to detect θ

2.00

0.50

0.20

0.10

0.05

Prio

r In

form

atio

n A

bout

θ a

s P

ropo

rtio

n o

f Inf

orm

atio

n fr

om M

axim

al S

ampl

e S

ize

XO

Mdn(θ l M = 2, T = −0.01)

−0.082

−0.067−0.052

−0.036

−0.021−0.006

0.01

Page 26: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 26

Case Study : Sepsis Trial

Contour plot of posterior median conditional upon stopping atanalysis 2 for futility (user-defined contours)

−0.10 −0.08 −0.06 −0.04 −0.02 0.00 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r S

D fo

r θ

0.990 0.900 0.500 0.200 0.025Power (lower) to detect θ

2.00

0.50

0.20

0.10

0.05

Prio

r In

form

atio

n A

bout

θ a

s P

ropo

rtio

n o

f Inf

orm

atio

n fr

om M

axim

al S

ampl

e S

ize

XO

Mdn(θ l M = 2, T = −0.01)

−0.09−0.08

−0.07−0.06

−0.05−0.04

−0.03

−0.02−0.01

0

0.01

Page 27: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 27

Case Study : Sepsis Trial

Contour plot of credible interval bounds at analysis 2 futilitybound (user-defined contours)

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r S

D fo

r θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

XO

Q0.025(θ l M = 2, T = −0.01)

−0.1

−0.09

−0.08

−0.07

−0.06

−0.05

−0.04−0.03

−0.02−0.01

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

2.00

0.50

0.20

0.10

0.05

Prio

r In

form

atio

n A

bout

θ a

s P

ropo

rtio

n o

f Inf

orm

atio

n fr

om M

axim

al S

ampl

e S

ize

XO

Q0.975(θ l M = 2, T = −0.01)

−0.06−0.05

−0.04−0.03−0.02−0.01

00.01

0.02

Page 28: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 28

Case Study : Sepsis Trial

Contour plot of posterior probabilities for a 2 analysis design

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r S

D fo

r θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

XO

Pr(θ > −0.0849) (prior)

0.9990.99

0.9750.95

0.90.5

0.1

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

XO

Pr(θ > −0.0849 l M = 1, T = −0.01)

0.999

0.99

0.9750.95

0.9

0.5

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

2.00

0.20

0.10

0.05

Prio

r In

form

atio

n A

bout

θ a

s P

ropo

rtio

n o

f Inf

orm

atio

n fr

om M

axim

al S

ampl

e S

ize

XO

Pr(θ > −0.0849 l M = 2, T = −0.042)

0.999

0.99

0.975

0.95

0.9

0.5

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r S

D fo

r θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

XO

Pr(θ < 0) (prior)

0.1

0.50.9

0.9750.99

0.999

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

XO

Pr(θ < 0 l M = 1, T = −0.084)

0.5

0.9

0.975

0.99

0.999

−0.10 −0.06 −0.02 0.02

Prior Median for θ

0.990 0.800 0.200 0.010Power (lower) to detect θ

2.00

0.20

0.10

0.05

Prio

r In

form

atio

n A

bout

θ a

s P

ropo

rtio

n o

f Inf

orm

atio

n fr

om M

axim

al S

ampl

e S

ize

XO

Pr(θ < 0 l M = 2, T = −0.042)

0.5

0.9

0.975

0.99

0.999

Page 29: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 29

Case Study : Sepsis Trial

Contour plot of ASN for Futility.8 design

−0.10 −0.08 −0.06 −0.04 −0.02 0.00 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r SD

for θ

0.990 0.900 0.500 0.200 0.025Power (lower) to detect θ

2.00

0.50

0.20

0.10

0.05

Prio

r Inf

orm

atio

n Ab

out θ

as

Prop

ortio

n o

f Inf

orm

atio

n fro

m M

axim

al S

ampl

e Si

ze

XO

Expected Sample Size Under Prior

850900

950

9501000

10001050

10501100

11001150

1200

1250

1300

Contours of Sample Size Distribution

Page 30: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 30

Case Study : Sepsis Trial

The seqBayesContour() function

I Important arguments to seqBayesContour() are:

dsn : the seqDesign() object used for computingBayesian operating characteristics

analysis.index : the analysis time to condition on; if missing,operating characteristics will be computedfor all analyses

observed : the observed statistic to condition on; ifmissing, boundary values are used

priorTheta : mean of prior distribution

priorVariation : variance of prior distribution

thetaThreshold : threshold for computing posterior probabilities;if missing, boundary hypotheses are used

Page 31: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 31

Case Study : Sepsis Trial

The seqBayesContour() function

I Important arguments to seqBayesContour() are:

posteriorProbContours : specifies contour lines for whichposterior probabilities are to be drawn;defaults to c(0.001, 0.01, 0.025, 0.05,0.1, 0.5, 0.9, 0.975, 0.99, 0.999)

posteriorMeanContours : if missing, plot with posterior meancontours will be drawn; if NULL, noplot will be drawn

lowerCredibleBoundContours : if missing, plot with contoursrepresenting lower limits of specifiedcredible interval will be drawn; ifNULL, no plot will be drawn

upperCredibleBoundContours : same as above

PlotASN : ASN contours; defaults to TRUE

SampSizeQuantiles : quantiles of sample size distributionfor which contours are to be drawn;defaults to 0.75

Page 32: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 32

Bayesian paradigm

Predictive probability scale in RCTdesign

I A Bayesian approach similar to the stochastic curtailmentprocedures would consider the Bayesian predictiveprobability that the test statistic would exceed somespecified threshold at the final analysis

I This statistic uses a prior distribution and the observeddata to compute a posterior distribution for the treatmenteffect parameter at the j-th analysis

I Using the sampling distribution for the as yet unobserveddata and integrating over the posterior distribution, thepredictive distribution of the test statistic at the finalanalysis can be computed

Page 33: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 33

Bayesian paradigm

Predictive probability scale in RCTdesign

I Again relying on the asymptotic distribution of theestimator and utilizing a normal prior, θ ∼ N(ζ, τ2), we cancompute a predictive probability statistic analogous to aconditional power statistic as

Hj (c, ζ, τ2) =

ZPr(θ̂J < c |Sj = sj , θ) p(θ |Sj = sj ) dθ

= Φ

0B@ [Πjτ2 + σ2/NJ ][c − θ̂j ] + [1 − Πj ][θ̂j − ζ]σ2/NJq[1 − Πj ][τ2 + σ2/NJ ][Πjτ2 + σ2/Nj ]σ2/NJ

1CA .

I Note that taking the limit as τ2 → 0 (yielding a point-massprior at ζ), the predictive probability statistic converges tothe conditional power statistic discussed previously

Page 34: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 34

Bayesian paradigm

Probability of observing an economically important estimates oftreatment effect

I Science vs. Marketing

I Just as we separate statistical significance from clinicalsignificance, marketing tends to have it’s own threshold forsignficance

I What will sell?

I In this case, Bayesian predictive probabilities may be ofuse in judging whether it is useful to continue a clinical trialin the hopes of obtaining an economically attractiveestimate of treatment effect

I For example, suppose that in the case of the sepsis trialan observed reduction of 6% (difference) in mortalitywould be deemed highly marketable...

Page 35: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 35

Case Study : Sepsis Trial

Boundaries on various design scales

I Bayesian predictive probabilities for observed estimateson the Futility.8 boundary

I Computed using a consensus prior (ζ = −0.04,τ2 = 0.042)

> 1-changeSeqScale(Futility.8,seqScale("H",priorTheta=-0.04,priorVariation=.04^2,threshold=-.06))

STOPPING BOUNDARIES: Predictive Probability scale(Predictive probability that estimated treatment effectat the last analysis exceeds -0.06based on prior distribution having median -0.04and variation parameter 0.0016)

a dTime 1 (N= 425) 0.9784 0.0057Time 2 (N= 850) 0.8066 0.0101Time 3 (N= 1275) 0.3501 0.0085Time 4 (N= 1700) 0.0000 0.0000

Page 36: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 36

Case Study : Sepsis Trial

Boundaries on various design scales

I Bayesian predictive probabilities for observed estimateson the Futility.8 boundary

I Computed using a non-informative prior (τ =∞)

> 1-changeSeqScale(Futility.8,seqScale("H",priorTheta=-0.04,priorVariation=Inf,threshold=-.06))

STOPPING BOUNDARIES: Predictive Probability scale(Predictive probability that estimated treatment effectat the last analysis exceeds -0.06based on prior distribution having median -0.04and variation parameter Inf)

a dTime 1 (N= 425) 0.9985 0.0018Time 2 (N= 850) 0.8778 0.0092Time 3 (N= 1275) 0.3901 0.0093Time 4 (N= 1700) 0.0000 0.0000

Page 37: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 37

Case Study : Sepsis Trial

Contour plot of predictive probabilities at 3rd efficacy boundary

−0.10 −0.08 −0.06 −0.04 −0.02 0.00 0.02

Prior Median for θ

0.02

0.04

0.06

0.08

Prio

r S

D fo

r θ

0.990 0.900 0.500 0.200 0.025Power (lower) to detect θ

2.00

0.50

0.20

0.10

0.05

Prio

r In

form

atio

n A

bout

θ a

s P

ropo

rtio

n o

f Inf

orm

atio

n fr

om M

axim

al S

ampl

e S

ize

XO

Pr(θ̂J < −0.06 l M = 3, T = −0.057)

0.10.15

0.2

0.25

0.3

0.350.4

0.45

0.5

0.550.6

0.65

0.7

Page 38: Probabilities Bayesian Predictive

RCTdesign

IntroductionBayesian operatingcharacteristics

Coarsened Bayes approach

Implementation inRCTdesignPosterior probability scale

Bayesian contour plots inRCTdesign

Bayesian PredictiveProbabilities

UW Grp Seq - Sec 4 - pg 38

Case Study : Sepsis Trial

Probability of observing an economically important estimates oftreatment effect

I For a result corresponding to a crude estimate oftreatment effect of -0.0566 at the third analysis, thepredictive probability of obtaining a crude estimate oftreatment effect less than -0.06 at the final analysis is35.0% under the sponsor’s consensus prior and 39.0%under a noninformative prior

I In either case, such high probabilities of obtaining a moreeconomically viable estimate of treatment effect may beenough to warrant modifying the stopping rule to avoidearly termination at the third analysis with a crudeestimate between -0.0566 and -0.06.