controversy? what controversy? an attempt to structure the debate on adaptive designs marc...

Controversy? What controversy?An attempt to structure the debate on adaptive designs

Marc Vandemeulebroecke

MCP, Vienna, 11.7.2007

Vandemeulebroecke Controversy? What controversy?1


A. Wald (1945): „The National Defense Research Committee considered these developments sufficiently useful for the war effort to make it desirable to keep the results out of the reach of the enemy.“

Examples (1): Sequential analysis

F. J. Anscombe (1963): „‘Sequential analysis‘ is a hoax.“

P. Armitage (1989): „If [...] ‘sequential analysis is a hoax‘, then its proponents are amongst the most persuasive swindlers, and their clients amongst the most gullible consumers, in the history of scientific endeavour.“


Example (2): Group sequential designs

such that for all,1, KkIk 21)( kIP k

Repeated confidence intervals (C. Jennison, B. Turnbull 1989)

P. Armitage: „ingenious derivation“

S. J. Pocock: „a minor change of presentation style“

Example (3): Adaptive/flexible designs


C.-F. Burman, C. Sonesson (2006): Are flexible designs sound?

M. Proschan: „Such methods, if extremely abused, produce illogical conclusions, but that is no more a condemnation of adaptive methods than Jack the Ripper is a condemnation of cutlery.“

P. Bauer: „Is wine sound? It is the way to drink it that matters.“

„The weighted test violates inference principles [...] and may lead to unreasonable results.“

03.010000 100 XNTest H0: with ; interim:

005.0,28.25.2,1 1011012 XZXnChange to observe

Example (4): Terminology


A. J. Phillips, O. N. Keene (2006): „The terms ‘flexible design’ and ‘adaptive design’ are used interchangeably.”

C. Jennison, B. Turnbull (2006): „The adjectives ‘flexible’ and ‘adaptive’ are not synonymous. Indeed adaptive designs can range from being totally inflexible to very flexible.”


Issues

Imagine a study protocol:• Double-blind multicenter trial in Crohn‘s Disease• Active vs placebo 1:1• Endpoint: CDAI after 4 weeks• IA at half of the sample size, switch to 2:1 if encouraging

Trial conduct:• IA disappointing, but new budget available Continue 1:1, ad hoc increase of sample size• Final analysis correctly adjusted and significant• In stage 2, patients are more severe, and placebo is doing much worse

Issue 1: Feasibility


Practical, logistical or operational issues regarding the conduct of the study

• Speed of recruitment vs. speed of response

• Availability of specialized software

• „Supply chain management“: Data capture/cleaning/ processing; real-time randomization and drug supply

• Training/SOPs/“charter“

• Investigate behavior of procedure by simulation, play through scenarios with decision makers / IDMC


Statistical/mathematical property.

• Correct type I error

• Point estimates available and sensible (e.g., mean/median unbiased)

• Confidence intervals available, sensible and correct

Issue 2: Validity


Hard to quantify. Concern: „operational bias“

Consistency/homogeneity across trial stages (A. Koch: identification problem) MCP: T. Friede, A. Faldum...

Jeopardizing integrity means jeopardizing credibility.

Measures to maintain trial integrity:

• avoid unblinding if possible • control dissemination of results• control scope and amount of adaptations

Issue 3: Integrity


V. Dragalin (2006, PhRMA):

„To maintain study validity means providing correct statistical inference [...], ensuring consistency between different stages of the study, and minimizing operational bias.

To maintain study integrity means providing convincing results to a broader scientific community; preplanning, as much as possible, based on intended adaptations; and maintaining the blind of interim analysis results.“

Issues 2/3: Validity/Integrity

H. Schäfer (2006): internal validity - external validity


Only interpretable results are communicable.

Intregrity and interpretability have the main impact of the persuasiveness of the study results.

• Endpoints meaningful

• Point estimates and confidence intervals available and sensible

Jeopardizing integrity (e.g., across-stage consistency or amount of adaptations) will also reduce interpretability.

Issue 4: Interpretability


Issue 5: Efficiency

Often, adaptation comes at the cost of efficiency.

• ...in terms of power / expected information

• Reason: non-sufficient statistics

Any adaptive design can be outperformed by a nonadaptive design - and vice versa!

„Statistical efficiency is not always the priority“ (Koch 2006)

„Response-dependent choice of group sizes can itself be a source of improved efficiency“ (Schmitz 1993, Jennison/Turnbull 2006)


Two main questions:

• Should there be an a priori selection of study features that may be adapted? If so, now many and which?

• Should the design changes follow prespecified rules or may they be done ad hoc? Should they depend on information internal or external to the trial?

Issue 6: Flexibility

stopping

+ stage sizes

+ other

Ada

ptab

le

feat

ures

Prespeficied rule Not dep. on trial-internal data Completely freeNature of change

Group sequential

Spending functions

p-value combinations

„Schmitz designs“


Issues• Feasibility: „Only“ operational issues

• Validity: „Only“ mathematical issues

• Integrity: Requirements?

„Operational bias“? Homogeneity across stages?

• Interpret‘y: Requirements?

• Efficiency: Weigh against flexibility (sponsor‘s choice)

• Flexibility:

Summary/conclusions

A priori selection of adaptable features? How many/which?

Adaptations ad hoc or following prespecified rules? Depending on internal or external information ?

Biom J 48 (2006, special issue), in particular: O‘Neil, Hung et al., Koch, and discussants: Section on „Regulatory perspectives“, pp. 559-622

Burmann, Sonesson (2006): Are flexible designs sound? Biometrics 62, pp. 664-683 (with discussion)

EMEA (2006): Reflection paper on methodological issues in confirmatory clinical trials with flexible design and analysis plan. CHMP/EWP/2459/02

Fleming, DeMets (1993): Monitoring of clinical trials: Issues and Recommendations. Contr Clin Trials 14, pp. 183-197

Gallo et al. (PhRMA, 2006): Adaptive designs in clinical drug development - an executive summary of the PhRMA working group. J Biopharm Stat 16, pp. 275-283 (with discussion)

Jennison, Turnbull (2006): Adaptive and nonadaptive group sequential tests. Biometrika 93, pp. 1-21

Phillips, Keene (PSI, 2006): Adaptive designs for pivotal trials: discussion points from the PSI adaptive design expert group. Pharm Stat 5, pp. 61-66

PhRMA (2006): Full white paper. DIJ 40, pp. 421-484

Posch, Bauer, Brannath (2003): Issues in designing flexible trials. Stat in Med 22, pp. 953-969

Stat in Med 25 (2006, special issue), pp. 3229-3408

Some references


...and many thanks to Norbert Benda!

Appendix: Terminology

Vandemeulebroecke Controversy? What controversy?A1

PhRMA: Definition of “adaptive design”

P. Gallo et al (2006): “a clinical study design that uses accumu-lating data to decide how to modify aspects of the study as it continues, without undermining the validity and integrity of the trial. [...] In such trials, changes are made ‘by design’, and not on an ad hoc basis.”

V. Dragalin (2006): “a multistage study design that uses accumulating data to decide how to modify aspects of the study without undermining the validity and integrity of the trial. [...] preplanning, as much as possible, based on intended adaptations.”



Other definitions of “adaptive design”

EMEA (2006): “A study is called ‘adaptive’ if statistical methodology allows the modification of a design element [...] at an interim analysis with full control of the type I error.”

Novartis slide deck: “Adaptive designs allow for initial uncertainties in trial design to be confirmed/adapted during the trial. The integrity of the trial is maintained and the evidence for the same hypothesis before and after the adaptation is combined.”



Other terms

• flexible• adaptive combination• flexible group sequential• flexible adaptive• adaptive group sequential• internal pilot• self designing• seamless adaptive• design automates• ....

(List stolen from J. Röhmel‘s talk at BfArM Discussion on Adaptive Designs, 2006)


Jennison, Turnbull (2006): „The strength of adaptive redesign lies in coping with the unexpected, in particular responding to external information that could not have been anticipated at the start of a study.“

Appendix: Flexibility

P. Gallo et al. (2006): „Should or should it not be viewed that adaptive designs might remedy inadequate planning?“

M. Frisén (2006): „One should use planned adaptive designs when one expects that it will be necessary to adapt the design. If one unexpectedly has strong reasons to change the plans, one should be very careful.“

controversy? what controversy? an attempt to structure the debate on adaptive designs marc...

Documents

validity slide

worse slide

sequential analysis

flexible designs sound

group sequential designs

decision makers idmc

adjectives flexible

terms flexible design