![Page 1: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/1.jpg)
Effectiveness: Overview of Current Approaches and Emerging Trial Designs
Doug Taylor, PhDDirector of Biostatistics
Family Health International
![Page 2: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/2.jpg)
Fleming and Richardson (2004). Some Design Issues in Trials of Microbicides for the Prevention of HIV Infection. JID.
Trussell and Dominik (2005). Will Microbicide Trials Yield Unbiased Estimates of Microbicide Efficacy? Contraception.
Pocock and Abdalla (1998). The Hope and the Hazards of Using Compliance Data in Randomized Controlled Trials. Statistics in Medicine, 17(3): 303-317.
Extra References
![Page 3: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/3.jpg)
• Efficacy vs. Effectiveness
• Basic Design of Effectiveness Trials
• Choice of Control Group
• Study Populations
• Strength of Evidence
• Power, Study Size and Adherence
• Future Directions
OUTLINE
![Page 4: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/4.jpg)
Efficacy vs. Effectiveness
Efficacy: reduction in risk of acquiring HIV when the microbicide is used as intended for
• Every act during the study (coitally dependent)
• Every day during the study (daily product)
• Following any other specified regimen (twice-daily, etc.)
![Page 5: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/5.jpg)
Efficacy vs. Effectiveness
Effectiveness: reduction in risk, recognizing that
• Participants don’t use microbicide for all acts/days
• Microbicide may be withdrawn for AE or pregnancy
• Condom use not independent of microbicide use
• Possibility of infection due to other exposure routes
![Page 6: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/6.jpg)
Basic Design
Phase 2b or 3 randomized, controlled trial powered to make a conclusive statement about effectiveness
Chance of concluding effectiveness depends on….
• What we mean by “conclusive statement”
• The efficacy of the microbicide
• Adherence to product use
• How much information (events) observed
![Page 7: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/7.jpg)
Basic Design
• HIV-negative participants randomized to either the microbicide or control
• Study staff and participants blind to treatment assignment (where possible)
• Intensive counseling on HIV risk-reduction and use of product; condom promotion; treatment of STIs
• Monitoring of AEs
• Independent Data Monitoring Committee (IDMC)
![Page 8: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/8.jpg)
• Testing for HIV at regular intervals for a year or more of follow-up
• Primary Outcome: estimated time to HIV infection
• Primary Analysis: compare the rate of infection between treatment groups using Intent-to-Treat principle
note on ITT: all participants/outcomes included, regardless of whether they continue to use product (e.g., even if withdrawn due to pregnancy)
Basic Design
![Page 9: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/9.jpg)
![Page 10: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/10.jpg)
Choice of Control Group
Placebo
• Ideally an ‘inert’ microbicide
• Blinding possible: risk-taking behaviors may be comparable between groups
Arguments against relying on a placebo control
• Placebo may not be inert
• Fail to capture effect of condom migration or other behavior changes that occur in real world
![Page 11: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/11.jpg)
Choice of Control Group
Condom-only
• Attempts to evaluate the effect of adding a microbicide to real world settings
• Risk-taking behaviors (including condom use) will almost certainly differ between groups
Arguments against relying on a condom-only arm
• Differences in behaviors may overwhelm any effect of microbicide on HIV
• Clinical trials are not the real world
![Page 12: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/12.jpg)
Study Populations
Minimal Requirements
• Risk of HIV exposure
• Minimal exposure to HIV via routes not protected by microbicide
• Willingness to use product
![Page 13: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/13.jpg)
Study Populations
Participants recruited from
• General, sexually active populations
• STI clinics
• Discordant couples
• Sex workers
![Page 14: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/14.jpg)
Refers to how confident we are that an apparent effect observed in a clinical trial is real and not due to chance
Related to concepts of
• Type I error (‘α level’) of a statistical hypothesis test when planning a trial, and
• Observed p-value or confidence interval for relative risk when interpreting trial results
Strength of Evidence
![Page 15: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/15.jpg)
Typical trials are designed to have no more than a 2.5% chance of concluding treatment is effective, if in reality there is no effect (type I error)
e.g. “the one-sided p-value for test of effect must be less than 0.025,” or “the upper 95% confidence bound for Relative Risk of HIV must be less than 1.0”
A study designed to meet this requirement can be thought of as targeting the strength of evidence of one trial
Strength of Evidence - One Trial
![Page 16: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/16.jpg)
RR2.01.00.7 3.00.3
0.40 0.99 95% CI
Estimated RR
p-value < 0.025 (one-sided)
Strength of Evidence of Single Trial
![Page 17: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/17.jpg)
• FDA has traditionally required two trials, each demonstrating that there is a protective effective at the one-sided α=0.025 level
• Trials could be conducted sequentially or in parallel
• Protects against a spurious result
• Helps to ensure that product will be effective in different settings/populations
Strength of Evidence - Two Trials
![Page 18: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/18.jpg)
RR2.01.00.7 3.00.3
0.40 0.94 95% CI (p-value < .025)
RR2
RR1
0.20 0.80 95% CI (p-value < .025)
Strength of evidence - Two Trials
![Page 19: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/19.jpg)
Could be unethical to conduct second trial. A single microbicide study could suffice if
• Well conducted
• Multi-site
• Large strength of evidence (p-value < 0.005, equivalent to 1.5-2 trials)
• Results consistent across sites and within important subgroups (e.g. age)
• Strong safety data
Strength of Evidence
![Page 20: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/20.jpg)
RR2.01.00.7 3.00.3
EstimatedRR
0.60 0.99 99.9% CI
p-value < 0.005 (one-sided)
Single Study with Larger Strength of Evidence
![Page 21: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/21.jpg)
• Achieving the strength of evidence of more than one trial could require 5000-10000+ participants, 50+ million dollars
• Difficult to commit those types of resources without plans for stopping trial early if product appears unlikely to achieve desired effect
→ futility analysis
Resource Management
![Page 22: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/22.jpg)
Phase 3 study designed to have 90% chance of detecting a 40% reduction in risk with α=0.005
Need to observe ~240 events
Half-way through the trial (120 events), the IDMC finds that the estimated treatment effect is zero
Even if the product is truly 40% effective (and the interim result is simply really bad luck), the chance of observing final p-value < 0.005 is only 20%
(but was interim result due to poor adherence?!)
Futility Analysis - example
![Page 23: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/23.jpg)
Alternatively, could consider a “2b screening trial” (HPTN/MTN 035; MTN 003 VOICE):
• One-third the size of the single Phase 3 trial
• Clear rules for when/how to proceed to second trial
Resource Management
![Page 24: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/24.jpg)
Power
No. ofEvents
Effectiveness
EfficacyAdherence Follow-upIncidence
Strength of evidence (alpha level)
Number enrolled
![Page 25: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/25.jpg)
• In treatment trials, every participant may have an outcome measure (e.g. viral load, CD4 count) and contribute to power to detect an effect of treatment
• In a prevention trial, very few participants have the outcome (HIV) and so very few contribute directly to the precision of the estimate of effectiveness
• Prevention trials are “event driven”
Number of Events and Power
![Page 26: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/26.jpg)
Events Required to Achieve 90% Power(Strength of Evidence of One Trial, one-sided α =0.025)
![Page 27: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/27.jpg)
Estimated Study Size for 90% Power(one-sided α=.025)
![Page 28: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/28.jpg)
When designing a study we state an effectiveness level that would be important to detect
For example, we might design a study to have 90% power to detect a 40% effectiveness level, assuming participants use it for 80% of acts/days
Such a study would require 160 events to achieve strength of evidence of one trial
Adherence and Power
![Page 29: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/29.jpg)
Effect of Non-Adherence on Power (160 Event trial)
![Page 30: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/30.jpg)
Challenges
• Poor adherence will doom an otherwise efficacious microbicide
• Condom use and risk-reduction counseling may significantly reduce incidence of HIV in control arm
• Do not know the infection status of partners
• Do not know whether microbicide was used for acts with exposure to HIV, or the route of exposure for individual acts leading to infection
• N9: Heterogeneity of effect across sites and studies; the same is possible for other products
![Page 31: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/31.jpg)
Future Directions - Adherence
• Target something closer to efficacy by monitoring of adherence (IPM)
• Better identify participants who will use the product
• Develop products or delivery systems (e.g. rings) that people will use
• Develop products that take adherence largely out of the hands of participants (e.g. injectables) or that participants are less likely to forget (e.g. rings)
![Page 32: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/32.jpg)
Future Directions – Design & Analysis
• Still have to perform the study
• Adaptive designs for screening out products, re-estimating power, futility and safety analyses
• Bayesian methods for combining phase 2 and phase 3 evidence
• Non-inferiority trials (when we have an effective microbicide)
![Page 33: Effectiveness: Overview of Current Approaches and Emerging Trial Designs Doug Taylor, PhD Director of Biostatistics Family Health International](https://reader035.vdocuments.site/reader035/viewer/2022062719/56649eea5503460f94bfbe82/html5/thumbnails/33.jpg)
Discussion
Questions?