Quantitative Synthesis IIPrepared for:

The Agency for Healthcare Research and Quality (AHRQ)Training Modules for Systematic Reviews Methods Guide

www.ahrq.gov

Systematic Review Process Overview

To understand how to explore between-study heterogeneity in a meta-analysis

To understand the pros and cons of subgroup analyses

To understand what meta-regression is and why it is useful

To understand the usefulness of control-rate meta-regression

Learning Objectives

Statistical Homogeneity

Barrington KJ. Cochrane Database Syst Rev 2000;(2):CD000505.

Statistical Heterogeneity

Reprinted from Pakos E, et al. J Bone Joint Surg Am 2005;87:1438-45, with permission from Rockwater, Inc..

Patellar Resurfacing in Total Knee Arthroplasty for Pain

RE = random effects model

In meta-analysis, heterogeneity refers to between-study diversity.

This term is often used to refer to: differences in study characteristics, and variability of study results.

What Is Heterogeneity?

Methodological diversity pertains to specifics of study design and analysis: Type of study Length of followup Proportion and handling of dropouts

Clinical diversity pertains to differences in: Populations Interventions and cointerventions Outcomes

Methodological and Clinical Diversity

Statistical heterogeneity exists when the results of individual studies are not “consistent” among themselves.

Statistical Heterogeneity

Clinical diversity

Methodological diversity

Biases

Chance

Statistical heterogeneity

Clinical and methodological diversity is abundant: Our aim in a meta-analysis is to explore that diversity and use our observations to formulate interesting hypotheses.

Often clinical and methodological heterogeneity results in a statistically significant test.

Chance, technical issues, or biases may result in statistically significant results in heterogeneity tests.

Clinical and Methodological DiversityVersus Statistical Heterogeneity

One of the most important roles of meta-analytic methodologies is to quantify statistical heterogeneity and to explore whether and to what extent it is explained by clinical and methodological diversity.

Exploration of Heterogeneity IsCentral to Evidence Synthesis

HETEROGENEOUS TREATMENT EFFECTS

IGNORE INCORPORATEESTIMATE(insensitive) EXPLAIN

FIXED EFFECTS MODEL

DO NOT COMBINE WHEN

HETEROGENEITY IS PRESENT

RANDOM EFFECTS MODEL

SUBGROUP ANALYSES

META-REGRESSION(control rate, covariates)

Dealing With Heterogeneity (I)

OVERALL ESTIMATEOVERALL ESTIMATE Combining Summary DataCombining Summary Data

Trea

tmen

t effe

ct

Trea

tmen

t effe

ct

variable of interest

META-REGRESSIONMETA-REGRESSION Modeling Summary DataModeling Summary Data

RESPONSE SURFACERESPONSE SURFACE Modeling Individual Patient DataModeling Individual Patient Data

Trea

tmen

t effe

ct

SUBGROUP ANALYSESSUBGROUP ANALYSES Differentiating Effects in SubgroupsDifferentiating Effects in Subgroups

Dealing With Heterogeneity (II)

Subgroup analyses can help: identify modifiers of the treatment effect, recognize biologically interesting phenomena, or formulate hypotheses.

We discuss two illustrative examples: Time-to-thrombolysis affects the treatment effect of

thrombolytic drugs in patients with acute myocardial infarction.

Effects of vitamin E supplementation on mortality may differ by vitamin E dose.

Promises of Subgroup Analyses

Risk Ratio (95% Confidence Interval)

Effects of Thrombolytic Therapy on Mortality in Patients With Acute Myocardial Infarction (Mean Time-to-Treatment)

Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I)

Largest Effect in the Subgroup of Trials With Mean Time-to-Treatment of 0 to 3 Hours

Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II)

Treatment Effect Diminishes When Mean Time-to-Treatment Is Between 3.1 and 5 Hours

Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III)

Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV)

Treatment Effect Diminishes Further WhenMean Time-to-Treatment Is Between 5.1 and 10 Hours

No Evidence of a Treatment Effect WhenMean Time to Treatment Longer than 10 Hours

Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V)

Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality

Miller ER 3rd, et al. Ann Intern Med. 2005;142:37-46. Reprinted with permission from the American College of Physicians.

Subgroup analyses are a form of multiple testing.

When uncontrolled, multiple testing can yield spurious findings.

Most meta-analyses do not perform statistical adjustments for multiple testing.

Hazards of Subgroup Analyses:Multiple Testing (I)

In the example that follows, we discuss subgroup analyses from the Second International Study of Infarct Survival (ISIS-2), a 2x2 factorial study of streptokinase versus placebo and aspirin versus placebo in more than 17,000 patients with myocardial infarction. For streptokinase versus placebo, the treatment effect does

not differ across subgroups by history of prior myocardial infarction.

For aspirin versus placebo, the treatment effect does differ.

Hazards of Subgroup Analyses:Multiple Testing (II)

Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.

Subgroup Analysis:Second International Study of Infarct Survival (ISIS-2) (I)

Subgroup Analysis:Second International Study of Infarct Survival (ISIS-2) (II)

Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.

When analyzing data, it is important to distinguish subgroup analyses that are specified a priori (without knowing what the data are) versus those that are specified post hoc (after the researcher has been exposed to the data).

This distinction is very clear when analyzing a prospective study.

In most meta-analyses, the distinction is not as clear.

Can You Avoid the Hazardsof Subgroup Analyses? (I)

Most meta-analyses use data that are published (and potentially known).

When researchers adequately prepare before embarking on a meta-analysis, they inevitably become acquainted with the data they will analyze.

This makes it difficult for researchers to claim that they specified subgroups without knowing anything about their data.

Can You Avoid the Hazardsof Subgroup Analyses? (II)

Meta-analysts should do their best to define subgroups that make methodological and biological sense.

Treat the results of subgroup analyses with a healthy dose of skepticism, especially when adjustments for multiple testing are not performed.

Can You Avoid the Hazards ofSubgroup Analyses? (III)

Meta-regression can help examine how the treatment effect changes across the levels of a variable.

All subgroup analyses can be formulated in a meta-regression framework, but meta-regression goes well beyond subgroup analyses.

Beyond Subgroup Analyses:Meta-Regression

Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality

Miller ER 3rd, et al. Ann Intern Med. 2005;142:37-46. Reprinted with permission from the American College of Physicians.

Corresponding Univariate Meta-Regression:A Meta-analysis of Vitamin E Doses and Mortality

Miller ER 3rd, et al. Ann Intern Med. 2005;142:37-46. Reprinted with permission from the American College of Physicians.

Ioannidis JPA. In: Publication bias in meta-analysis: prevention, assessment and adjustments. 2005.

Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo

Study level Examples: presence/absence of blinding, intervention dose

(in experimental studies) Participant level

Examples: mean age, proportion of diabetic patients, mean intake of vitamin E (in observational studies)

Two Types of Covariates in Meta-Regressions

Aggregate-data meta-regressions on participant-level covariates can mislead, because they are susceptible to ecological fallacy.

The observed relationship between the study-level treatment effect and the mean of a patient-level factor does not necessarily reflect the corresponding true relationship at the individual patient level.

Spurious Associations in Meta-Regressions or Subgroup Analyses (I)

Therefore, associations of treatment effect and participant-level covariates should be interpreted with caution.

Such associations can be biologically plausible and informative, or can mislead.

Unfortunately, there is no universal way to distinguish true from fallacious findings.

Spurious Associations inMeta-Regressions or Subgroup Analyses (II)

In control-rate meta-regression, we examine whether the treatment effect changes across studies with different event rates in the controls.

The control rate is a surrogate for the baseline risk for the event of interest (severity of disease).

This approach examines whether the underlying risk of an event explains differences in the treatment effect across studies.

Control-Rate Meta-Regression

Intravenous Streptokinase Therapy inPatients With Acute Myocardial Infarction (I)

Intravenous Streptokinase Therapy inPatients With Acute Myocardial Infarction (II)

Control Rate Meta-Regression in the Preceding Streptokinase Example

Schmid CH, et al. Stat Med 1998;17:1923-42.

Subgroup analyses, meta-regressions, and control-rate meta-regressions are tools to explore between-study heterogeneity. Use them to understand data.

They are mostly hypothesis-forming tools. Especially for meta-regressions on patient-level covariates,

ecological fallacy may mislead. Beware when interpreting their results.

Key Messages

Barrington KJ. Umbilical artery catheters in the newborn: effects of position of the catheter tip. Cochrane Database Syst Rev 2000;(2):CD000505.

DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177-88.

ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:349-60.

Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta-analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142:37-46.

References (I)

Pakos E, Ntzani EE, Trikalinos TA. Patellar resurfacing in total knee arthroplasty. A meta-analysis. J Bone Joint Surg Am 2005;87:1438-45.

Ioannidis JPA. Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester, England: Wiley; 2005. p. 287-302.

Schmid CH, Lau J, McIntosh MW, et al. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998;17:1923-42.

References (II)

This presentation was prepared by Joseph Lau, M.D., and Thomas Trikalinos, M.D., Ph.D., members of the Tufts Medical Center Evidence-based Practice Center.

The information in this module is based on Chapter 9 in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf).

Authors