Meta-analysis in Risk-Benefit Evaluation

Demissie Alemayehu, Ph.D.April 2012

2

Outline• Part I: Standard Meta-

Analysis

– What is Meta-

Analysis?

– Benefits of Meta-

Analysis

– Procedures for

Meta-Analysis

– Criteria for

Causality

– Case Study

• Part II: Network Meta-

analysis

• Introduction

• Indirect and Mixed

Treatment Comparisons

• Exchangeability

– Issues and

Approaches

• Heterogeneity

—Heterogeneity in CER

—Issues and

Approaches

• Recent Developments

• Concluding Remarks

3

Workshop GoalsNon-practitioners of Meta-analysis

– Understand meaning of meta-analysis

– Understand limitations and strengths of MA

– Critically review MA literature

Practitioners of Meta-analysis

• Review standard approaches

• Review recent evelopments

• Explore further opportunities for research

“Many of the groups…are far too small to allow of any definite opinion being formed at all, having regard to the size of the probable error involved.”

Karl Pearson, 1904

5

6

Definition: Meta-Analysis

• Meta-Analysis is a statistical approach for

combining information from independent

studies to address a pre-specified

hypothesis of interest.

– Based on systematic literature review

– Provides precise estimate of treatment effect,

giving due weight to studies included

77

Benefits of Meta-analysis

• Precision / Power

– Individual clinical trials may lack power to provide conclusive results

– Particularly useful for rare events

• To assess consistency (generalizability) of results

– To settle controversies arising from conflicting studies

• Answer questions not posed by the individual studies

– Generate new hypotheses

– Subgroup risk/benefit

Unsystematic clinical observations

Physiologic Studies

Single Observational Study

Systematic Review of Observational Studies

Single Randomized Trial (RCT)

Systematic review of RCT

Harbour R, Miller J. A new system for grading recommendations in

evidence based guidelines. BMJ 2001;323:334–6.

Hierarchy of Evidence

When to Do Meta-Analysis

• When more than one study has estimated an effect

• When there are no differences in the study characteristics that are likely to substantially affect outcome

• When the outcome has been measured in similar ways

• When all the data are available

9

1010

When not to do a Meta-Analysis

• „Garbage in - garbage out‟

– A meta-analysis is only as good as the studies in it

• Beware of „mixing apples with oranges‟

– Compare like with like

• Beware of reporting biases

11

Questions to Ask

1. Detailed written protocol developed?

2. Comprehensive / systematic search for eligible studies performed?

3. Presentation of results appropriate?

4. Sensitivity analysis / test for heterogeneity performed?

5. Limitations of analysis discussed?

6. Results reported in light of available body of knowledge?

12

• Formulation of problem to be addressed

• Eligibility criteria for studies to be included

defined a priori

• Statistical methods for combining the data

Protocol Development

13

Selection of Studies

• Minimize Sources of Bias

– Publication Bias

• Positive studies more likely to be published than

negative studies

– Reviewer Bias

• Tendency to include only studies that favor the

hypothesis of interest

• Define universe of studies to be considered

• Define explicit & objective criteria for study

inclusion/ rejection based on quality grounds

14

Selection of Studies, cont’d

• Popular Databases

– PubMed / MedLine

– Cochrane Review

– Trial result registries

• Other Sources

– Hand Search: FDA, library

– Personal references, emails

– Web: Google, etc.

15

Funnel Plot ▪ Precision: increasing function of study size

▪ In absence of bias, results from small studies scatter widely at bottom of graph

▪ Publication bias: asymmetrical funnel plots

Publication Bias

16

• Effect Measures– Mean Difference

– Relative Risk

– Odds Ratio

– Risk Difference

• Peto OR:

Approximation to the

Odds Ratio.

– Used in rare events

Statistical Methods

• Choice depends

– Ease of

communication

of effect

– Study design

– Analytical

convenience

Total Exposed

EventObserved

Risk of Event

Active 1000 10 10/1000=0.01

Control 1000 1 1/1000 =0.001

Experiment 1

Risk Difference: 0.01-0.001 = 0.009

Relative Risk: 0.01/0.001 = 10

Odds Ratio:

Odds for Active: 0.01/0.99 = 0.010101

Odds for Control: 0.001/0.999 =0.001001

OR : 0.010101/0.001001 =10.09

Total Exposed

EventObserved

Risk of Event

Active 1000 410 410/1000=0.410

Control 1000 401 401/1000 =0.401

Experiment 2

Risk Difference: 0.410-0.401 = 0.009

Relative Risk: 0.410/0.401 = 1.02

Odds Ratio:

Odds for Active: 0.410/(1-0.410) = 0.70

Odds for Control: 0.401/(1-0.401) = 0.67

OR : 0.70/0.67 = 1.05

Statistical Methods (cont.)

18

Inference: • Different approaches exist,

but there is no single

"correct" method

• Two general approaches

–Fixed Effect Models

–Random Effects Models

•Bayesian procedure

–Increasingly used,

especially in network

meta-analysis

Statistical Methods (cont.)

19

• Fixed Effect Models

− If it is reasonable to assume underlying effect is the SAME for all studies

− Only one source of sampling error: within study

− Methods: inverse variance, Mantel-Haenszel

Statistical Methods, cont’d

20

• Random Effects Models

− True effect could vary from study to study

− E.g., effect size higher in older subjects

− Sources of sampling error: within study & between studies

− Common method: DerSimonian-Laird

Statistical Methods, cont’d

21

• Bayesian Methods– Use prior information in

statistical inference

Statistical Methods, cont’d

Thomas Bayes

(1702-1761)

21

22

• Bayesian Formulations

Statistical Methods, cont’d

Thomas Byes

(1702-1761)

23

• Large-sample procedures may not be

appropriate for pooling rare events

− Comparison of Procedures

• Bradburn et al., Stat Med 2007;26:53-77

• Event Rates < 1%

– Peto one-step odds ratio

• Reasonable bias, power

• Bias substantial, unbalanced case

Meta-Analytical Procedures:

Rare Events

24

• Event Rates 2%-5%

– Logistic Regression, Exact Method, MH OR better performance (in terms of bias) than Peto

• Risk Difference

– Conservative confidence intervals

– No need to exclude studies with 0 events in both groups

• For sparse data, incorporation of heterogeneity into estimation of treatment effects has minimal impact on performance

Meta-Analytical Procedures:

Rare Events, cont’d

25

Graphical display of results from individual

studies on a common scale: Blobbograms

http://www.childrens-mercy.org/stats/model/metaanalysis.asp

Data Presentation

26

• Heterogeneity is variation between the studies‟ results

• Causes:

– Differences in patient groups studied

– Differences in interventions studied

– Differences in primary outcome studies

– Studies carried out in distinct settings

e.g., different countries

Heterogeneity

27

• Cochran‟s Q Test– Low power, esp.

when # of studies is small

– Liberal if number of studies is large

Heterogeneity (cont.)

Under random effects el

Q w

pooled relative risk fixed effects el

study specific estimated relative risk

w Var

Q under H

s ss

S

s

s s

S B

mod ,

( ) ,

log , mod

log

[ ( )]

~ :

2

1

1

1

2

0

2 0

28

• I2

– proportion of variation that is due to

heterogeneity rather than chance

– Large values of I2 suggest heterogeneity

– Roughly, I2 values of 25%, 50%, and 75% could be

interpreted as indicating low, moderate, and high

heterogeneity

– Higgins JPT et al. Measuring inconsistency in meta-

analyses. BMJ 2003;327:557-60.

Heterogeneity (cont.)

29

Handling Heterogeneity

HETEROGENEOUS

TREATMENT EFFECTS

IGNORE ACCOUNT

FOR

EXPLAIN

FIXED

EFFECTS

MODEL

RANDOM

EFFECTS

MODEL

SUBGROUP

ANALYSES

META-

REGRESSION

(control rate,

covariates)

Post-hoc natureMultiplicities

Apples and oranges phenom

Interpretation?

Meta-Regression

Assess heterogeneity of effects by covariates that are constant within

study

• E.g., gender, smoking status

Model:

β s = β 0 + β1 GENDERs + β2 CURRENTs

+ β2 PASTs + bs + εs

GENDERs = 1 if studys is male; 0 if female

CURRENTs = 1 if studys has current smokers only, 0 otherwise

PASTs = 1 if studys has past smokers only, 0 otherwise

H0: β1 = 0 no effect-modification by gender

Mixed effects models can be used to test hypotheses and estimate

parameters (Stram, Biometrics, 1996)

• Limitations– Ecological fallacy

• Association present at patient level may not be necessarily true at the study level

– A model that includes a covariate that is an aggregate of a person-level characteristic rather than a study characteristic can produce biased results.

– Post-hoc specification of prognostic factors may lead to spurious results

– Cannot handle factors that vary by patient within study

Meta-Regression (cont.)

32

• Examine Robustness of Findings

– Examine effects in certain studies, certain

groups of patients, or certain interventions

– Robustness to departures from model

assumptions

– Sensitivity to departure from study selection

criteria

Sensitivity Analysis

33

• When interpreting results, don‟t just emphasise the

positive results.

• Discuss Limitations of Analysis

– Bias: Publication

– Use of aggregate data: Confounding factors

– Model assumptions

– Heterogeneity

• Summarized in light of available body of knowledge

Reporting of Results

Alemayehu D. (2011) Perspectives on Pooled Data Analysis: the Case for an Integrated

Approach. Journal of Data Science, 9(3): 399-426

34

"Correlation does not imply

causation"

35

• Strength of Association

– Validity of measure: OR, RR, etc.

– Clinical / statistical significance

• Biological Plausibility

– Known potential biologic basis to suggest a

causal link

– Reference pharmacogenomic evidence

Assessment of Causality

• Assessment of Consistency

– Review similar reports for drug

– Review similar reports for class

36

• Epidemiology

– Consider the expected rate in the general

population for comparable demographic groups

• False Positive Findings

– Address issue of multiplicity

– FDR

• Imbalance (wrt Relevant Confounding Factors)

– Identify relevant risk factors

• Pharmacovigilance Databases

– Recent advances in analysis of such data

– Measures of disproportionality

Assessment of Causality, cont’d

37

• Specificity

– Cause and effect relationship already established for a

similar exposure-disease combination

• Temporality

– Cause precedes effect problem

• Dose-Response

– Larger exposures associated with higher rates of

disease

Assessment of Causality, cont’d

• Reversibility

– Reversing the exposure is associated with lower

rates of disease

38

“Talking about unsettled science is a muchmore complicated communications challengethan simply disseminating hard conclusions.”

“The goal should be to effectively communicatefindings to patients, healthcare providers, andregulatory agencies to enable informeddecision-making.”

Gottlieb, 2006

Communications of Findings

• PRISMA: – Preferred Reporting Items for Systematic Reviews and Meta-

Analyses.

– An evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.

– A 27-item checklist, and a four-phase flow diagram.

– An update and expansion of the now-out dated QUOROM Statement

• Aim : – Help authors improve the reporting of systematic reviews and meta-

analyses.

– Focus on randomized trials, can also be used as a basis for reporting systematic reviews of other types of research

Source: http://www.prisma-statement.org/statement.htm

PRISMA 2009 Flow Diagram

Records identified through database searching

(n = )

Scre

en

ing

Incl

ud

ed

Elig

ibili

tyId

en

tifi

cati

on

Additional records identified through other sources

(n = )

Records after duplicates removed(n = )

Records screened(n = )

Records excluded(n = )

Full-text articles assessed for eligibility

(n = )

Full-text articles excluded, with reasons

(n = )

Studies included in qualitative synthesis

(n = )

Studies included in quantitative synthesis

(meta-analysis)(n = )

Source: http://www.prisma-

statement.org/statement.htm

Section/topic # Checklist item Reported

on page #

TITLE

Title 1 Identify the report as a systematic review, meta-analysis, or both.

ABSTRACT

Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study

eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results;

limitations; conclusions and implications of key findings; systematic review registration number.

INTRODUCTION

Rationale 3 Describe the rationale for the review in the context of what is already known.

Objectives 4 Provide an explicit statement of questions being addressed with reference to participants,

interventions, comparisons, outcomes, and study design (PICOS).

METHODS

Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if

available, provide registration information including registration number.

Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years

considered, language, publication status) used as criteria for eligibility, giving rationale.

Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to

identify additional studies) in the search and date last searched.

Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it

could be repeated.

Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if

applicable, included in the meta-analysis).

Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and

any processes for obtaining and confirming data from investigators.

Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any

assumptions and simplifications made.

Risk of bias in individual

studies

12 Describe methods used for assessing risk of bias of individual studies (including specification of

whether this was done at the study or outcome level), and how this information is to be used in any

data synthesis.

Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means).

Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of

consistency (e.g., I2) for each meta-analysis.

Section/topic # Checklist item Reported

on page #

Risk of bias across studies

15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).

Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.

RESULTS

Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with

reasons for exclusions at each stage, ideally with a flow diagram.

Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.

Risk of bias within studies

19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).

Results of individual studies

20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.

Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency.

Risk of bias across studies

22 Present results of any assessment of risk of bias across studies (see Item 15).

Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]).

DISCUSSION

Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).

Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias).

Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research.

FUNDING

Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.

QUOROM vs. PRISMA checklists

45

45

Cumulative Meta-Analysis

• Commonly executed with chronologically ordered RCTs

– Perform a new statistical pooling every time a new RCT becomes available

– Impact of each study on the pooled estimate may be assessed

– Reveals (temporal) trend towards superiority of the treatment or the control, or towards indifference

– Performed retrospectively, the year when a treatment could have been found to be effective could be identified

– Performed prospectively, an effective treatment may be identified at the earliest possible moment

46

Example of Cumulative Meta-Analysis

Lau et al (1995) J Clin Epi (48) 45-57

47

4747

Basic Method of Cumulative Meta-Analysis

Studies ordered

chronologically ofby covariates Study 1

Study 2

Study 3

Study 4

Cumulative M-A 1

Cumulative M-A 2

Cumulative M-A 3

Pool Studies 1 to 2

Pool Studies 1 to 4

Pool Studies 1 to 3

Study n-1 Cumulative M-A n-2

Study n Cumulative M-A n-1Pool Studies 1 to n

Pool Studies 1 to n-1

• Arrangement of trials– Reporting time

– Effect size

– Study size

– Study quality

– Event rate of control

– Other covariates of interest

• Benefits– Identify signal early

– Design of future trials

– Need for further trials

– Identify subgroups of interest

Cumulative Meta-Analysis (cont.)

• Issues– Multiplicity: Use of p-

values without correction– Use of Bayesian approaches

– Contributions of large studies minimized

– Experience shows that large studies reflect meta-analysis results of small studies

– Evolution of treatment effect

over time

– Changes in patient

demographics

– Changes in healthcare delivery

– Evolving medical practices

Software

• “rmeta” package in R for statistical computing

http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_meta_analysis.pdf

• EpiMeta: from Epi Info

• Revman: from Cochrane Collaboration

• meta module in STATA

• MIX for Excel 2000 or 2003

http://www.mix-for-meta-analysis.info

Index:

meta.DSL Random effects (DerSimonian-Laird) meta-analysismeta.MH Fixed effects (Mantel-Haenszel) meta-analysis

meta.summaries Meta-analysis based on effect estimatesmetaplot Confidence interval (forest) plotfunnelplot Funnel plot for publication bias

meta.colors Control colours in meta-analysis plot

cummeta Cumulative meta-analysiscummeta.summaries Cumulative meta-analysis

Package: rmetaAuthor: Thomas Lumley <tlumley@u.washington.edu>

ntrt Number of subjects in treated/exposed group

nctrl Number of subjects in control group

ptrt Number of events in treated/exposed group

pctrl Number of events in control group

conf.level Coverage for confidence intervals

names names or labels for studies

statistic "OR" for odds ratio, "RR" for relative risk

x,object a meta.DSL object

summary Plot the summary odds ratio?

meta.DSL(ntrt, nctrl, ptrt, pctrl, conf.level=0.95,. . . ,statistic="OR")

summary(object, conf.level=NULL, ...)

plot(x, conf.level=NULL, colors=meta.colors(), xlab=NULL,...)

Arguments:

x Treatment difference

se Standard error of x

size Variable for the vertical axis

summ summary treatment difference

funnelplot(x, se, size=1/se,

summ=NULL, xlab="Effect",

ylab="Size", colors=meta.colors(),

conf.level=0.95, plot.conf=FALSE, zero=NULL, mirror=FALSE, ...)

Arguments

Graphics:

forestplot ()

metaplot

plot.meta.DSL,

plot.meta.MH,

plot.meta.summaries

Part II: Network Meta-Analysis

• Introduction

• Indirect and Mixed Treatment Comparisons

• Exchangeability

– Issues and Approaches

• Heterogeneity

—Heterogeneity in CER

—Issues and Approaches

• Recent Developments

• Concluding Remarks

Introduction

• Comparative Effectiveness Research

– Institute of Medicine, IOM:• Generation and synthesis of evidence that compares the benefits

and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care

– The purpose is to assist consumers, clinicians, purchasers, and policy makers to make the informed decisions that will improve health care at both the individual and population levels. http://books.nap.edu/openbook.php?record_id=12648&page=29

– Congressional Budget Office: • A rigorous evaluation of the impact of different options that are

available for treating a given medical condition for a particular set of patients.

– Such a study may compare similar treatments, such as competing drugs, or it may analyze very different approaches

Introduction (cont.)RCTs vis-à-vis CER• Randomized controlled trials (RCTs) designed to provide

data on a treatment efficacy on average for a given population.

• Applicability of results of RCTs to real-world situations limited by factors inherent to the design– Stringent entry criteria exclude subgroups of interest– For operational reasons, certain subgroups may not be

adequately represented in study– Other protocol requirements impose constraints that are

inconsistent with real-world practice and adherence

• CER: Emphasis on generalizability of RCT results to different subgroups or settings― Knowledge of heterogeneity of treatment effects is critical in

decision making about treatment options for individual patients and subpopulations

• Limitations of traditional meta-analysis also apply to CER

– Quality • Publication bias

– Heterogeneity• “Apples and oranges”

phenomenon

– Methodological issues• Lack of uniform

approaches

• Handling heterogeneity

• Assessing bias

• Covariate issues

Issues specific to CER

• Lack of head-to-head RCTs for comprehensive assessment

− Need to use non-standard procedures

− Stringent model assumptions!

• Assessment of heterogeneity more complex

Traditional Meta-Analysis vs. CER Meta-Analysis

– Several glaucoma drugs

– Rank available glaucoma drugs according to their intraocular pressure (IOP)-reducing effect.

– Several RCTs available, but not all treatments have been compared directly.

Source: R. van der Valk et al. / Journal of Clinical Epidemiology 62 (2009) 1279-1283

NB: n: Number

of direct

comparisons

between two

drugs.

Example

Standard Meta-Analytic Framework

Goal:

Synthesize information and obtain a pooled estimate of treatment

effect.

Standard Meta-analytic Framework (cont.)

Key assumption:

Homogeneity across trials

Standard Meta-analytic Framework (cont.)

Key Issue:

Problem of interpretation, when trials are heterogeneous

Standard Meta-analytic Framework (cont.)

Key Issue:

Choice of priors

Indirect Comparison

A

BA vs. C?

C

Scenario:

i) Direct head-to-head

comparative evidence available

for A vs. B, and for B vs. C

• Could be single RTCs or

pooled data in meta-

analyses

ii) Interested in A vs. C

iii) Limitations of using direct effect

measures for A and B

• Preservation of

randomization (at least

partially

Indirect Comparison

Ref: Bucher HC, et al. Journal of Clinical Epidemiology 1997;50:683-691.

Song F, et al;. BMJ 2003;326:472-476.

Partial preservation of randomization

Indirect Comparison (cont.)

• Cannot handle multi-arm trials or situations with direct and indirect evidence.

• Optimality of procedure

– Generally conservative

Indirect Treatment Comparisons: 2-step Approach

Jansen JP, et al. (2008)

Network Meta-analysisA

C

DB

Lumley (2002). Statist. Med; 21:2313-23A measure of inconsistency

Limitations :

• Indirect comparison follows through a closed loop design: Necessary for inference

• Cannot handle correlations in multi-arm studies

• Measure of inconsistency requires a sizeable number of studies

Mixed Treatment Comparisons

• Allow combining

evidence from direct and

indirect estimates

– Assumption of

consistency: Lu and

Ades (2006, JASA)

and Salanati et al.

(2008)

• MTC also permits

handling of multi-arm

studies: Lu and Ades

(2004, Stat Med

• Most approaches based

on Bayesian techniques

A vs C comparison via direct and

indirect means

Li et al. BMC Medicine 2011 9:79 doi:10.1186/1741-7015-9-79

Mixed Treatment Comparisons (cont.)

Non-informative priors

a) Normal for “treatment effect”

b) inverse gamma or uniform prior for dispersion.

WinBUGS Codes: www.bris.ac.uk/cobm/docs/intro%20to%20mtc.doc

Statistical Issues

Exchangeability Assumption:

• Relative efficacy of a treatment is the same in all trials included in the indirect comparison

• Same synthesized comparative treatment effect would result if direct comparisons were performed with trials mimicking the other (indirect) experimental conditions:

– Comparability of distribution of effect modifiers (treatment x covariate interactions) among trials

– Methodological similarity (e.g., quality, definition of outcomes) also required

Statistical Issues (cont.)

• Reliable statistical procedures unavailable to validate assumptions– Exchangeability/consistency

• Available procedures do not readily adjust for study heterogeneity– Adjusted indirect comparison, misleading term

– Meta-regression in the context of MTC not well studies

• Rigorous study of the operating characteristics of commonly used techniques not available– Low power, as a result of use of two separate variances

• Impacts reliability of non-inferiority conclusions

• Often leads to indeterminate results, due to inflated Type II error

• Most methods do not handle correlated data

Approaches for Assessing Exchangeability Assumptions

In general, exchangeability assumption difficult to verify

Approaches for Assessing Exchangeability Assumptions (cont.)

Qualitative approaches

• Differences in quality and design

– Blinding

– Duration of follow-up

– Outcome measures

– Treatment: Dose, timing

• External factors

– Healthcare systems

– Geography

– Setting: Hospital vs. ambulatory care

– Temporal

• Patient characteristics

− Demography

− Disease severity, etc.

Quantitative approaches

• Consistency: Constant event rates of reference group across trials

– Inference to rule out play of chance

• Absence of heterogeneity when meta-analysis is involved

– Usual issues with tests of homogeneity

• Formal inference in a (closed) network meta analysis

Approaches for Assessing Exchangeability Assumptions (cont.)

Approaches for Assessing Exchangeability Assumptions (cont.)

Approaches for Assessing Exchangeability Assumptions (cont.)

• Mahalanobis distance matching:

– ML estimators of location and covariance

• Modified propensity score analysis

– Perform usual case, with study assignment as dependent variable, rather than treatment

– Use propensity scores to identify subgroups of patients with a high probability of being in a given study

Alemayehu D. (2011) Assessing exchangeability in indirect and mixed treatment comparisons.

Comparative Effectiveness Research, 1: 51 - 55

76

Suppose we have treatment pairs AB, AC, BC with direct evidence

Indirect estimate:ˆ ˆ ˆindirect direct direct

BC AC ABd d d

Measure of

inconsistency: ˆ ˆˆ indirect direct

BC BC BCd d

Approximate test ˆ

ˆ

BCBC

BC

zV

ˆ direct direct direct

BC BC AC ABV V d V d V d

Measuring Consistency

Heterogeneity

• Heterogeneity and exchangeability different, but related concepts

• Heterogeneity

– Consistency of overall results across subgroups

• CER guidelines based on average results may not be optimal, and may even be substantially harmful

– Consistency of results across studies

• Heterogeneity issues associated with traditional meta-analysis apply to CER

• Objectives of heterogeneity analysis:

– Identify sources of heterogeneity

• Tailor treatments accordingly

– Adjust for heterogeneity

• Traditional regression approach for measured confounders

• Instrumental variables (Basu et al http://www.ssc.wisc.edu/~snavarro/Research/Limitations_of_IV_v28.pdf)

• Need to distinguish between clinical vs. statistical heterogeneity

Clinical vs. Statistical Heterogeneity

• Clinical heterogeneity (CH):

– Variation in study population characteristics, coexisting conditions, cointerventions, and outcomes evaluated across studies included in an SR or CER that may influence or modify the magnitude of the intervention measure of effect

• Statistical heterogeneity (SH):

– Variability in the observed treatment effects beyond what would be expected by play of chance

• CH may lead to SH. However SH may result from:

– CH, methodological heterogeneity or play of chance

http://www.effectivehealthcare.ahrq.gov/ehc/products/93/533/Clinical_Heterogeneity_Revised_Report_

FINAL%2010-5-10.pdf

Statistical Issues with Heterogeneity Analyses

• Post hoc vs. Pre-specified Analyses– Bias introduced by timing of specification of hypotheses

being tested relative to examination of data

• Multiplicity

– With multiple subgroup analyses, probability of a false positive finding substantial.

• Power– Chance of missing significant effect a function of size

• Inappropriate Heterogeneity Assessment– Claiming heterogeneity on basis of separate tests of

treatment effects within each subgroup– Claiming heterogeneity on basis of observed treatment-

effect sizes within each subgroup

Best Practices: HeterogeneityPre-specification • Clear Definition of Clinical

Heterogeneity– Demographic Characteristics– Clinical variables: Risk factors for

the principal condition under study– Expert consensus vs. Statistical

criteria

• Pre-Specification of factors in Protocol– Essential for confirmatory evidence

of efficacy in a subgroup

• Sample Size– A subgroup should be large enough

to ensure reliability of inferences

• Handling of Multiplicity– Specify adjustment strategy for

multiplicity

• Reporting of Results– Transparency/balance

Analytical Strategy

• No universally accepted method even in standard meta-analysis:

– Low power for small number of studies or rare events

• Meta-regression

– Limitations: Ecological fallacy

• Sensitivity Analysis

– Sensitivity to departures from definition of clinical variables

– Sensitivity to inclusion/exclusion of studies

– Sensitivity to departures from model assumptions

Alemayehu D. (2011) Evaluation of Heterogeneity of Treatment Effects in Comparative Effectiveness Research.

J Biomet Biostat 2:125. doi:10.4172/2155- 6180.1000125

Recent Developments

• AHRQ– Comparative Effectiveness Review Methods: Clinical Heterogeneity

– Best practices for addressing clinical heterogeneity in systematic reviews and CER

http://www.effectivehealthcare.ahrq.gov/ehc/products/93/533/Clinical_Heterogeneity_Revised_Report.pdf

• FDA CER Initiative– Framework for analyzing heterogeneity of treatment effects in CER

– Emphasis on value of pre-specification, exploratory vs. confirmatory subgroup analyses, testing for interactions, displaying graphically results, validating subgroup results.

http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/ScienceBoardtotheFoodandDrugAdministration

Recent Developments (cont.)

• Role of simulation in MTC

– Recent progress in simulated treatment comparisons (Caro & Itshak, 2010)

– Simulated experiments can serve several purposes:

• Perform head-to-head treatment comparisons, in the absence of such data

• Assess differences among trials

• Assess heterogeneity and exchangeability assumptions

Concluding Remarks

• Meta-analysis and Indirect comparison an integral tool of medical research– Considerable practical and methodological challenges

• Opportunity to engage in methodological work for statisticians– Assess exchangeability

– Assess/Adjust for heterogeneity, confounders

• Effective assessment of heterogeneity important to diverse stakeholders: – Clinicians, patients, policymakers and other healthcare providers

• Without sound methodological foundation, potential for adverse impacts on public health.

84

84

Selected References• Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-

analysis of randomized controlled trials. J Clin Epidemiol 1997;50(6):683-91.

• Eddy DM, Hasselblad V, Shachter R. An introduction to a Bayesian method for meta-analysis: The confidence profile method. Med Decis Making. 1990 Jan-Mar;10(1):15-23.

• Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ. 2003;326:472.

• Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004;23:3105–24.

• Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005;331:897–900.

• Jansen JP, Crawford B, Bergman G, Stam W. Bayesian meta-analysis of multiple treatment comparisons: an introduction to mixed treatment comparisons. Value Health. 2008; 11:956-64.

• Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: Application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Stat Med. 2009;28:1861-81.

• Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ. 2009 Apr 3;338

• Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med 2002;21(16):2313-24.

• Ades AE. A chain of evidence with mixed comparisons: models for multi-parameter synthesis and consistency of evidence. Stat Med 2003;22(19):2995-3016.

• Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison and indirect evidence. BMJ 2005;331(7521):897-900.

• Vandermeer BW, Buscemi N, Liang Y, Witmans M. Comparison of meta-analytic results of indirect, direct, and combined comparisons of drugs for chronic insomnia in adults: a case study. Med Care 2007;45(10 Supl 2):S166-S172.

• Song F, Harvey I, Lilford R. Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. J Clin Epidemiol 2008;61(5):455-63. 40.

• Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D'Amico R, et al. Indirect comparisons ofcompeting interventions. Health Technol Assess 2005;9(26):1-134.