Dealing with heterogeneity of treatment effects: is the literature up to the challenge?

Download Dealing with heterogeneity of treatment effects: is the literature up to the challenge?

Post on 06-Jul-2016

213 views

Category:

Documents

1 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>ral</p><p>ssBioMed CentTrials</p><p>Open AcceResearchDealing with heterogeneity of treatment effects: is the literature up to the challenge?Nicole B Gabler*1, Naihua Duan2, Diana Liao3, Joann G Elmore4,5, Theodore G Ganiats6 and Richard L Kravitz1,7</p><p>Address: 1Center for Healthcare Policy and Research, University of California, Davis, California, USA, 2Columbia University, New York State Psychiatric Institute, New York, USA, 3University of California, Los Angeles Neuropsychiatric Institute, California, USA, 4University of Washington School of Medicine, Seattle, Washington, USA, 5Department of Medicine, Harborview Medical Center, Seattle, Washington, USA, 6Department of Family and Preventive Medicine, University of California, San Diego, California, USA and 7Department of Internal Medicine, University of California, Davis, California, USA</p><p>Email: Nicole B Gabler* - nrbloser@ucdavis.edu; Naihua Duan - Naihua.Duan@Columbia.edu; Diana Liao - DLiao@mednet.ucla.edu; Joann G Elmore - jelmore@u.washington.edu; Theodore G Ganiats - tganiats@ucsd.edu; Richard L Kravitz - rlkravitz@ucdavis.edu</p><p>* Corresponding author </p><p>AbstractBackground: Some patients will experience more or less benefit from treatment than theaverages reported from clinical trials; such variation in therapeutic outcome is termedheterogeneity of treatment effects (HTE). Identifying HTE is necessary to individualize treatment.The degree to which heterogeneity is sought and analyzed correctly in the general medicalliterature is unknown. We undertook this literature sample to track the use of HTE analyses overtime, examine the appropriateness of the statistical methods used, and explore the predictors ofsuch analyses.</p><p>Methods: Articles were selected through a probability sample of randomized controlled trials(RCTs) published in Annals of Internal Medicine, BMJ, JAMA, The Lancet, and NEJM during oddnumbered months of 1994, 1999, and 2004. RCTs were independently reviewed and coded by twoabstractors, with adjudication by a third. Studies were classified as reporting: (1) HTE analysis,utilizing a formal test for heterogeneity or treatment-by-covariate interaction, (2) subgroup analysisonly, involving no formal test for heterogeneity or interaction; or (3) neither. Chi-square tests andmultiple logistic regression were used to identify variables associated with HTE reporting.</p><p>Results: 319 studies were included. Ninety-two (29%) reported HTE analysis; another 88 (28%)reported subgroup analysis only, without examining HTE formally. Major covariates examinedincluded individual risk factors associated with prognosis, responsiveness to treatment, orvulnerability to adverse effects of treatment (56%); gender (30%); age (29%); study site or center(29%); and race/ethnicity (7%). Journal of publication and sample size were significant independentpredictors of HTE analysis (p &lt; 0.05 and p &lt; 0.001, respectively).</p><p>Conclusion: HTE is frequently ignored or incorrectly analyzed. An iterative process ofexploratory analysis followed by confirmatory HTE analysis will generate the data needed tofacilitate an individualized approach to evidence-based medicine.</p><p>Published: 19 June 2009</p><p>Trials 2009, 10:43 doi:10.1186/1745-6215-10-43</p><p>Received: 3 October 2008Accepted: 19 June 2009</p><p>This article is available from: http://www.trialsjournal.com/content/10/1/43</p><p> 2009 Gabler et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Page 1 of 12(page number not for citation purposes)</p></li><li><p>Trials 2009, 10:43 http://www.trialsjournal.com/content/10/1/43</p><p>BackgroundRandomized controlled trials (RCTs) are the cornerstoneof evidence-based medicine. Such trials rely on randomassignment to alternative treatment groups to control forbaseline patient factors that could affect outcomes. Theresulting estimate of the average treatment effect is anaverage of the individual treatment effects (ITEs) for partic-ipants in the study. While estimates of the average treat-ment effect are generally useful, some treated individuals,both within and outside of clinical trials, will experiencemore or less benefit than the reported average. Such vari-ation in treatment effect is termed heterogeneity of treat-ment effects (HTE) [1,2].</p><p>HTE may be quantitative (subgroup effects in the same direc-tion as the average effect but varying in magnitude) or qual-itative (treatment effects in different directions in differentsubgroups, where treatment is beneficial in some subgroupsand harmful in others). The prevalence of HTE is unknownand perhaps unknowable, but highly variable treatmentresponse rates for many common conditions suggest it issubstantial. [3,4] For example, Allen Roses, the vice presidentof genetics at GlaxoSmithKline, has stated, "Our drugs don'twork on most patients" [5]. Several empirical demonstra-tions of HTE have recently been published, including studiesof ischemic stroke [6], risk reduction by carotid endarterec-tomy [7,8], and diabetes [9]. While qualitative HTE may beuncommon, quantitative HTE should not be dismissed,because even modest variations in the magnitude of nettreatment benefits may have important implications forpatient care and cost-effectiveness.</p><p>HTE can be assessed in several ways. The most directapproach is the n-of-1 clinical trial, which assigns individ-ual patients to receive alternative treatment in a randomlypredetermined sequence [10,11]. Results from a series ofsuch trials can be aggregated to assess heterogeneity in thepopulation. However, n-of-1 trials are applicable to a rel-atively small subset of conditions and treatments [10,11]and are subject to random within-patient variability (thusrequiring a careful design and repeated crossovers)[12,13]. A second approach is to stratify patients accord-ing to risk of disease-related adverse events [14,15]. Athird approach, typically performed for purposes ofhypothesis generation rather than testing, entails a carefulexamination of subgroups within RCTs.</p><p>Subgroup analysis can be perilous. Real effects can bemissed because of inadequate statistical power [16,17],and reported effects may be spurious because of the per-formance of multiple statistical tests (1316) and/or dueto random intra-individual variability [12,13]. Randomintra-individual variability is especially problematic</p><p>design. In parallel group trials, participants are only rand-omized to one treatment and do not crossover to alterna-tive treatments. As such, it is not possible to estimate anyvariation that occurs within a participant. In recognition ofthe drawbacks of subgroup analysis, the ConsolidatedStandards of Reporting Trials (CONSORT) statementwarns that subgroup analyses, especially post hoc sub-group comparisons, "do not have great credibility" [18].</p><p>On the other hand, it has been claimed that nearly every-thing we have learned from epidemiology resulted fromsubgroup analysis [19]. While this conclusion applies mostobviously to observational studies, careful scrutiny of sub-group-specific effects in randomized trials has generatedimportant new hypotheses and sometimes directly influ-enced practice. Nevertheless, subgroup analyses are notalways performed correctly. Some studies (e.g. [20-23])report results by subgroups, without any statistical testing orinterval estimation for the difference across subgroups;these studies do not provide quantitative information onHTE per se. Other studies report p-values corresponding toeach subgroup, subsequently claiming that the treatmenteffect differs across subgroups because it is statistically sig-nificant in one subgroup and not in another [24]. However,both treatment effect and sample size influence the p-value,such that similar effect sizes within each subgroup mightgenerate markedly different p-values. Instead of comparingthe p-values across subgroups, the appropriate way to iden-tify significant HTE is to make statistical comparisons fortreatment effects across subgroups, using a test for heteroge-neity or interaction [7,17,18,25-27].</p><p>The tension between needing to understand HTE andlacking the statistical power to properly examine itpresents difficulties for researchers, clinicians, andpatients. While some experts have offered general encour-agement to perform more HTE analyses [28,29], the liter-ature is relatively silent on how to manage the risks ofover- and under-testing. Kraemer et al. [25,30,31] havesuggested a sequential approach that could shed light onpossible HTE. Defining treatment modifiers as factors thatinfluence the treatment effect size across subgroups, theypropose that all RCTs use exploratory interaction analyses asa method to generate hypotheses regarding moderators oftreatment effects. The presence of strong moderator effectswould encourage future researchers to perform ade-quately powered confirmatory studies stratified prospec-tively on these moderators. While the proposal ofKraemer et al. makes sense, several small reviews, mostpublished well before the revised CONSORT statement,suggest that testing for HTE is reported in only 25% to50% of RCTs [18,27,32-35].Page 2 of 12(page number not for citation purposes)</p><p>because it is not possible to estimate this variability in par-allel group trials, the most common type of clinical trial</p><p>We undertook the current review of the prevalence of HTEanalyses in a comparatively larger sample of articles pub-</p></li><li><p>Trials 2009, 10:43 http://www.trialsjournal.com/content/10/1/43</p><p>Page 3 of 12(page number not for citation purposes)</p><p>Article selection for systemic review of heterogeneity of treatment effect in RCTs published in five general medical journalsFigure 1Article selection for systemic review of heterogeneity of treatment effect in RCTs published in five general medical journals.</p></li><li><p>Trials 2009, 10:43 http://www.trialsjournal.com/content/10/1/43</p><p>Page 4 of 12(page number not for citation purposes)</p><p>Table 1: Characteristics of included articles (n = 303) and RCTs described in these articles (n = 319)*</p><p>Characteristic Articles Represented # RCTs Included</p><p>Journal of publicationAnnals 30 (10) 30 (9)</p><p>BMJ 43 (14) 47 (15)</p><p>JAMA 45 (15) 50 (16)</p><p>Lancet 97 (32) 101 (32)</p><p>NEJM 88 (29) 91 (29)</p><p>Year of publication1994 86 (28) 91 (29)</p><p>1999 102 (34) 106 (33)</p><p>2004 115 (38) 122 (38)</p><p>Medical condition under studyCardiovascular 69 (23) 74 (23)</p><p>Infectious Disease 62 (20) 70 (22)</p><p>Cancer 41 (14) 42 (13)</p><p>Psychiatry/Neurology 25 (8) 25 (8)</p><p>Other 106 (35) 108 (34)</p><p>First author's study regionNorth America 115 (38) 121 (38)</p><p>Other 188 (62) 198 (62)</p><p>Study designParallel - 304 (95)</p><p>Crossover - 15 (5)</p><p>Analysis reportedHTE analysis - 92 (29)</p><p>Subgroup without statistical comparison - 88 (28)</p><p>None - 139 (44)</p><p>Sample size 262 (101 708)</p><p>- [641,000]</p><p>Abbreviations: RCTs, randomized controlled trials; BMJ, British Medical Journal; JAMA, Journal of the American Medical Association; NEJM, New England Journal of Medicine; HTE, heterogeneity of treatment effects.*Discrete data are expressed as No. (%); continuous values are presented as median (inter-quartile range) [range].</p></li><li><p>Trials 2009, 10:43 http://www.trialsjournal.com/content/10/1/43</p><p>Page 5 of 12(page number not for citation purposes)</p><p>Table 2: HTE reporting by study characteristics (n = 319 RCTs in 303 articles)</p><p>Characteristic N No. (%) Reporting HTE No. (%) Reporting Either subgroup or HTE</p><p>Journal of publication Annals 30 11 (37) 16 (53)</p><p>BMJ 47 9 (19) 18 (38)</p><p>JAMA 50 21 (42) 33 (66)</p><p>Lancet 101 21 (21) 53 (53)</p><p>NEJM 91 30 (33) 60 (66)</p><p>Year of publication1994 91 20 (22) 47 (52)</p><p>1999 106 30 (28) 62 (58)</p><p>2004 122 42 (34) 71 (58)</p><p>Medical condition under study Cardiovascular 74 21 (28) 43 (58)</p><p>Infectious disease 70 25 (36) 48 (69)</p><p>Cancer 42 15 (36) 30 (71)</p><p>Psychiatry/Neurology 25 7 (28) 12 (48)</p><p>Other 108 24 (22) 47 (44)</p><p>First author's study region North America 121 47 (39) 80 (66)</p><p>Other 198 45 (23) 100 (51)</p><p>Study design ||Parallel 304 89 (29) 177 (58)</p><p>Crossover 15 3 (20) 3 (20)</p><p>Sample size ** **Quintile 1 (median = 37) 64 9 (14) 18 (28)</p><p>Quintile 2 (median = 124) 64 7 (11) 29 (45)</p><p>Quintile 3 (median = 263) 64 22 (34) 45 (70)</p><p>Quintile 4 (median = 549) 64 21 (33) 40 (63)</p><p>Quintile 5 (median = 1560) 63 33 (52) 48 (76)</p><p>Abbreviations: HTE, heterogeneity of treatment effects; RCTs, randomized controlled trials; BMJ, British Medical Journal; JAMA, Journal of the American Medical Association; NEJM, New England Journal of Medicine.p &lt; 0.05 by Pearson Chi-square testp &lt; 0.01 by Pearson Chi-square testp &lt; 0.05 by Fisher's exact test||p &lt; 0.01 by Fisher's exact testp &lt; 0.05 by Mantel-Hantzel test for trend**p &lt; 0.0001 by Mantel-Hantzel test for trend</p></li><li><p>Trials 2009, 10:43 http://www.trialsjournal.com/content/10/1/43</p><p>tistical methods used, and explore the predictors of suchanalyses. A persistently low rate of appropriate HTE orinteraction analysis would suggest missed opportunitiesfor identifying HTE.</p><p>MethodsOverviewWe conducted a literature sample of RCTs published infive prominent general medical journals during 1994,1999, and 2004. The search strategy and abstraction formsincorporated input from a Project Advisory Committee.Human Subjects committee approval was not required.The study was funded by Pfizer, Inc., under a contract tothe academic institutions involved. However, the investi-gators were solely responsible for all aspects of studydesign, data collection and analysis, and result reporting.</p><p>Data Sources and SearchesUsing PubMed, we searched for RCTs published in theAnnals of Internal Medicine, British Medical Journal, Journalof the American Medical Association, Lancet, and New Eng-land Journal of Medicine during odd numbered months in1994, 1999, and 2004 (Figure 1). These prominent medi-cal journals were selected because they have a broad read-ership, are the publication venue for many landmarkstudies, and have disproportionate influence on medicalresearch and practice [36,37]. During the period of inter-est, 4,863 articles appeared in these journals. Of these,907 were excluded for having publication types of "biog-raphy", "case report", "editorial", "news", "letters", "com-ments", and "patient education handout". The remaining3,956 articles were then restricted to include only journalarticles and reviews, thereby eliminating all articles with-out a research abstract (n = 1,815 articles excluded). Toassess the validity of these exclusions, one of the investi-gators reviewed a 5% random sample of the 2,722excluded articles, resulting in no questionable exclusions.Finally, the remaining 2,141 articles were subdivided intotwo groups: those with "clinical trial" as a publicationtype (n = 541) and those without (n = 1,600). The 1,600articles without clinical trial as publication type wereexcluded after they were examined by one investigator,and a 10% random sample was additionally examined bya second investigator, without finding any clinical trials.The 54...</p></li></ul>

Recommended

View more >