systematic review: analytical methods of meta-analysis stephen bent, m.d. assistant professor of...

Systematic Review:Analytical Methods of

Meta-analysis

Stephen Bent, M.D.Assistant Professor of Medicine, Epidemiology

and BiostatisticsUCSF

8 Steps to Systematic Review

1. Research Question 2. Protocol 3. Search 4. Study selection (inclusion/exclusion) 5. Quality assessment 6. Data abstraction 7. Analysis

– A) Create summary measure– B) Assess for heterogeneity– C) Assess for publication bias– D) Conduct sensitivity/subgroup analyses– E) Advanced issues/techniques

8. Interpretation

Juni et al, Hazards of scoring the quality of

clinical trials. JAMA. 1999;282:1054-60.

Why conduct a systematic review?

The best way to summarize evidence on a scientific topic

Concisely communicates findings to others in the field

Identifies author(s) as expertsIdentifies areas for future studyPerfect for background of grantsDon’t need to do primary data collection (so

can be done while waiting for data from other projects)

You have to do the work anyway, so might as well get a publication!

You can effect change in clinical management

Cumulative Meta-analysis

Antman EM et al: JAMA. 1992;268:240-248

Systematic Review: Clinical Implications (Antiarrhythmic Drugs for

Acute MI)

Teo KK et al. JAMA. 1993;270:1589-1595

Sample Systematic Reviews Kangelaris KN, Bent S, Nussbaum RL, Garcia DA, Tice JA.

Genetic testing before anticoagulation? A systematic review of the safety and efficacy of pharmacogenetic dosing of warfarin. Journal of General Internal Medicine (in press).

Nguyen SP, Bent S, Chen Y, Terdiman JP. Gender as a Risk Factor for Advanced Neoplasia and Colorectal Cancer: A Systematic Review and Meta-analysis. Clinical Gastroenterology. 2009;7:676-81.

Simon J, Chen Y, Bent S. The relation of alpha-linoleic acid to the risk of prostate cancer: a systematic review. Am J Clin Nutr. 2009;89:1-7S.

Li J, Winston LG, Moore DH, Bent S. Efficacy of short-course antibiotic regimens for community-acquired pneumonia: a meta-analysis. American Journal of Medicine. 2007;120(9):783-90.

Margaretten M, Kohlwes J, Moore D, Bent S. The rational clinical examination: does this patient have septic arthritis. JAMA. 2007;297:1478-1488.

Sample Systematic Reviews

Hsu J, Kohlwes J, Bent S. Efficacy of antifungal therapy in chronic rhinosinusitis: A systematic review. J Allergy Clin

Immunol. 2010 125:2

Guarnieri M, Bent S. Death from coronary artery disease in patients with systemic lupus erythematosus: a systematic

review and meta-analysis of mortality cohort studies. (submitted to Arthritis Care and Research 1/2012).

Lee JK, Liles EG, Bent S, Levin TR, Corley DA. Diagnostic Accuracy of Fecal Immunochemical Tests for Colorectal

Cancer: Systematic Review and Meta-analysis (submitted to JAMA 4/2013).

8 Steps of Systematic Review



8. Interpretation

Create a Summary Measure

Before we get to the mechanics of a summary measure….

Be sure to provide your audience with a concise summary table

A “visual meta-analysis”Readers should be able to examine

Table 1 and reach their own conclusions about the data

Example

Antibiotics for acute bronchitis.After search and application of

inclusion/exclusion criteria, 8 studies were included.

RCTs in Acute Bronchitis

Study, yr N Abx Outcome Result*Stott, 76 207 Doxy Days of Yellow Spit 0.6 (-0.2 to 1.4)

Franks, 84 54 TMP/S Cough Amount Score 0.2 (-0.2 to 0.6)

Williamson, 84 69 Doxy Days of Purulent Sputum -0.2 (-1.2 to 0.8)

Dunlay, 87 45 Erythro Sputum production score 0.5 (0.1 to 0.9)

Scherl, 87 31 Doxy Days of sputum 1.9 (-0.2 to 4.0)

Verheij, 94 140 Doxy Days of productive cough 0.5 (-0.4 to 1.4)

Hueston, 94 23 Erythro Days of productive cough -0.4 (-2.4 to 1.6)

King, 96 91 Erythro Days of sputum production 0.7 (-1.3 to 2.7)

* Positive numbers indicate antibiotics are superior to placebo

How do you create a summary measure?

Clinical example: 5 year old girl presents with ear pain and is found to have an acute otitis media.

Should she get antibiotics?

Research Questions:1.In children with OM, are antibiotics

effective for pain relief?2.In children with OM, do antibiotics

reduce the rate of complications (mastoiditis, hearing problems)?

3 studies are identified (examining effect of Abx on Pain)

Study 1: N = 100RR=1.41Study 2: N=200 RR=0.98Study 3: N=300 RR=1.01

You could take the average effect: (1.41 + 0.98 + 1.01) / 3 = 1.13

Is this a good summary measure?

Summary measure weighted by sample size

Provide “weight” for studies based on their sample size

600Total1.013003

0.9820021.411001RRNStudy

summary effect estimate= Σ (Ni x effect estimatei) = 640 =1.07 Σ(Ni) 600

More refined: Provide “weight” by using inverse of

variance

Summary = Σ (weighti x effect estimatei) = 30.5 = 1.00effect estimate Σ(weighti) 30.3

Study N RR Var RR Weight

1 100 1.41 3.0 0.33

2 200 0.98 0.1 10

3 300 1.01 0.05 20

Total 700

Does the largest study always have the lowest variance and

therefore the greatest “weight”?

Dichotomous outcomes

Continuous outcomes

Confidence Intervals Around Summary Effect

Calculate variance of summary effect estimate, or the 95% CI around the summary estimate

Variance of summary estimate = 1 Σ(weightsi)

Variance of summary estimate = _1_ = .03 30.5

95% CI = + 1.96 √0.03 = + 0.34

Summary OR and 95% CI = 1.00 (0.65 - 1.33)

Type of Model?

Variance RRs = 1/wiVariance RRs = 1/wi

Weighti = 1

variance RRi + D

Weighti = 1

variance RRi

Variance of individual studies + variance of differences between studies

Weights: variance of individual studies

Existing studies are a random sample

Existing studies are the entire population

Goal: estimate the “true” effect

Goal: weighted average of risk from existing studies

Random EffectsFixed Effects

Formulas for D

Fixed Effects

Model:

Random Effects

Model:

Summary RRb

Summary RRa

Summary RRb

Summary RRa

Random VS. Fixed Effects Model Practical Implications of the Choice

Confidence intervals: RE model produces wider confidence intervals

Statistical significance: less likely with RE model

BOTTOM LINE: If the individual study findings are similar, the model

makes little difference in estimate or statistical significance.

If the individual study findings are heterogeneous, the model can affect the statistical significance.

Mantel-Haenszel Method (Fixed Effects Model)

Diseased Not diseasedTreated (exposed) ai ci

Not treated (unexposed) bi di

ORi = ai/ ci = ai x di lnORmh = Σ (wi x lnORi )

bi/ di bix ci Σwi

variance lnORi = 1 + 1 + 1+ 1 variance ORmh = 1 ai bi ci di Σ wi

weighti = (wi) = 1 variance lnORi

95% CI = elnORmh (1.96 x √variance lnORmh)

Randomized Trials of Antibiotic Rx for acute OM to prevent TM

perforationStudy 1 Perforation No PerforationAntibiotic 1 114Placebo 3 116

Study 2 Perforation No PerforationAntibiotic 7 65Placebo 12 65

1. Calculate OR and lnOR for each study:OR1= 1 x 116 = 0.34 lnOR1 = -1.08

3 x 114

OR2 = 7 x 65 = 0.58 lnOR2 = -0.54 12 x 65


perforation

2. Calculate variance lnORi for each study:

Var ln OR1 = 1 + 1 + 1 + 1 = 1.35 1 3 114 116

Var ln OR2 = 1 + 1 + 1 + 1 = 0.26 7 12 65 65

3. Calculate wi for each study:

w1 = 1 = 0.74 1.35

w2 = 1 = 3.85

0.26



4.Calculate the wi x ln ORi for each study: w1 x lnOR1 = 0.74 x -1.08 = -0.80

w2 x lnOR2= 3.85 x -0.54 = -2.08


perforation

5. Calculate the sum of the wi

w1 + w2 = 0.74 + 3.85 = 4.59

6. Summary lnORmh = Σ (wi x lnORi) = -0.80 + -2.08 = -0.63

Σ wi 4.59= ORmh = 0.53

7. Calculate variance ORmh = 1 = 1 = 0.22

Σ wi 4.59

8. Calculate 95% CI = elnORmh + (1.96 x √ variance lnORmh)

= e-.63 + (1.96 x √ 0.22) = 0.21 - 1.34

Summary OR = 0.53 (95% CI 0.21 – 1.34)


perforation

Dersimonian and Laird Method (Random Effects Model)

Similar formula to Mantel-Haenszel:ln ORdl = Σ (wi x ln ORi) wi = 1

Σwi variancei + D

Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

But…All you need to know is:

When combined, individual study effect estimates are weighted by their inverse variance

Variance is related to sample size AND # of events (dichotomous) and precision (continuous)

Fixed effects just combines all weighted estimates, while random effects “penalizes” estimates for variation between studies




8. Interpretation

HeterogeneityAre you comparing apples and oranges?Clinical heterogeneity: are studies asking same

question?Statistical heterogeneity: is the variation likely

to have occurred by chance?

Measures how far each individual OR/RR is from the summary OR/RR.

Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

Heterogeneity

Refers to the degree that the study results differ

Visual ApproachStatistical Approach

Q = sum [weighti x (ESs – ESi)]

p < 0.05 indicates heterogeneity

Summary RR = 0.93 (0.87-0.99)

Problem of Heterogeneity

Study findings are different – should they be combined?

Study OR1 0.012 1.03 10.0

Study OR1 0.352 0.563 0.974 1.155 1.756 1.95

Statistical tests of Heterogeneity

Is the variation in the individual study findings likely due to chance?

Ho: Effect estimate in each study is the same (or homogeneous)

Ha: Effect estimate in each study is not the same (or heterogeneous)

Q = Σ(wi x (ln ORmh – ln ORi )2) df = (N studies -1)

p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous

Heterogeneity – Interpret Findings (Example: RR of Colon CA, Men vs.

Women)

8 Steps to Sytematic Review



8. Interpretation

Assessing for Publication Bias

Publication Bias – the publication or “non-publication” of research findings, depending on the nature and direction of the results.

Rosenthal, 1979 – published an article describing the “file-drawer problem” that journals publish only 5% of all negative studies, while the file drawers in the back of the lab contain the other 95%.

Methods for Assessing Publication Bias

Funnel plots – simple scatter plots of treatment effects (horizontal axis) vs. some measure of study size (vertical axis).

Choice of axes– Log scale for treatment effects (to ensure that

treatment effects in opposite directions are the same distance from 1.0 – e.g., 0.5 and 2.0)

– Standard error for measure of sample size• Power depends on both sample size and #

events• Standard error is consistent with the statistical

tests

Funnel Plot of Log Relative Risk vs Standard Error

Log Relative Risk

Sta

nd

ard

err

or

5

4

3

2

1

0

Example: ALA and Prostate Cancer Risk

RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

ALA – Funnel Plot

Funnel Plot with Imputed Values for Publication Bias

RR=0.94, 95% CI: 0.79-1.17

Publication bias caveatFunnel plot asymmetry does not always

indicate bias– It is possible that smaller studies enrolled

higher risk patients, for example, and therefore found a greater effect.

– Small studies are often conducted before larger studies. In the intervening years, other interventions may have improved, thus reducing the relative efficacy of the treatment.

Statistical methods to assess publication bias

Examine associations between study size and treatment effect.– Sensitivity is poor when < 20 studies

Begg’s test: an adjusted rank correlation

Egger’s test: a weighted regression of effect size vs. standard error.– Basically asks if the regression line has a non-zero

slope– More sensitive than Begg’s test, but more false

positives, especially when 1) large treatment effects, 2) few events per trial, 3) all trials of similar size. (In these cases, one may decide a priori to use Begg’s test).

Begg's funnel plot with pseudo 95% confidence limits

RR

s.e. of: RR0 .2 .4 .6 .8

-1

0

1

2

Begg's Test adj. Kendall's Score (P-Q) = -30 Std. Dev. of Score = 14.58 Number of Studies = 12 z = -2.06 Pr > |z| = 0.040 z = 1.99 (continuity corrected) Pr > |z| = 0.047 (continuity corrected)

Egger's test------------------------------------------------------------------------------ Std_Eff | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- slope | .9810716 .1103858 8.89 0.000 .7351168 1.227026 bias | -.9911295 .3236382 -3.06 0.012 -1.71224 -.2700187------------------------------------------------------------------------------




8. Interpretation

Subgroup & Sensitivity Analysis

Subgroup Analysis – MA of a subgroup of eligible studies

age

ethnicity

risk factors

treatment

Sensitivity Analysis – add or delete questionable studies

eligibility

treatment

Subgroup Analysis

OR 95% CI N Ever user

Of estrogen:

All eligible studies

Cohort studies

Case-Control studies

2.3*

1.7*

2.4*

2.1 - 2.5

1.3 - 2.1

2.2 - 2.6

29

4

25

Dose of

estrogen:

0.3 mg

0.625 mg

1.25 mg

3.9

3.4

5.8

1.6 - 9.5

2.0 - 5.6

4.5 - 7.5

3

4

9

Duration of

use:

< 1 year

1-5 years

5-10 years

10 years

1.4

2.8

5.9

9.5*

1.0 - 1.8

2.3 - 3.5

4.7 - 7.5

7.4 - 12.3

9

12

10

10

Regimen: Cyclic

Daily

3.0*

2.9*

2.4 - 3.8

2.2 - 3.8

8

8

* p for heterogeneity < 0.05

Analytical Methods: Summary Points

Always start the meta-analysis with a “visual meta-analysis” (i.e., a great table 1). – A clinician should be able to interpret the results

Step 1: Calculate a summary measure = “weighted mean effect estimate”– You can combine anything, but use judgment

Step 2: Assess for heterogeneity– Heterogeneity is not always a problem

Step 3: Assess for publication bias– Both visual and statistical methods

Step 4: Perform subgroup/sensitivity analyses– Ideally specify these a priori




8. Interpretation

Can you conduct a systematic review when there are only a few studies?

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of

randomised controlled trials Objectives To determine whether parachutes are effective in preventing

major trauma related to gravitational challenge. Design Systematic review of randomised controlled trials. Data sources: Medline,Web of Science, Embase, and the Cochrane

Library databases; appropriate internet sites and citation lists. Study selection: Studies showing the effects of using a parachute during

free fall. Main outcome measure Death or major trauma. Results We were unable to identify any randomised controlled trials of

parachute intervention. Conclusions As with many interventions intended to prevent ill health,

the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.

Smith GCS and Pill JP. BMJ 2003;327:1459–61

Advanced Topics

Individual participant dataMissing dataDifferent types of dataObservational studiesGeneralized synthesis of evidenceMeta-regressionCritique of a systematic review

Different types of data

Different scales (example)Ordinal dataBinary dataContinuous outcomesDiagnostic tests (example)

RCTs in Acute Bronchitis: Different Scales

Study, Study, yryr

NN AbxAbx OutcomeOutcome ResultResult

Stott, 76Stott, 76 202077

DoxyDoxy Days of Yellow SpitDays of Yellow Spit 0.6 (-0.2 to 0.6 (-0.2 to 1.4)1.4)

Franks, 84Franks, 84 5454 TMP/TMP/SS

Cough Amount ScoreCough Amount Score 0.2 (-0.2 to 0.2 (-0.2 to 0.6)0.6)

Williamson, Williamson, 8484

6969 DoxyDoxy Days of Purulent Days of Purulent SputumSputum

-0.2 (-1.2 to -0.2 (-1.2 to 0.8)0.8)

Dunlay, 87Dunlay, 87 4545 ErythErythroro

Sputum production Sputum production scorescore

0.5 (0.1 to 0.5 (0.1 to 0.9)0.9)

Scherl, 87Scherl, 87 3131 DoxyDoxy Days of sputumDays of sputum 1.9 (-0.2 to 1.9 (-0.2 to 4.0)4.0)

Verheij, 94Verheij, 94 141400

DoxyDoxy Days of productive Days of productive coughcough

0.5 (-0.4 to 0.5 (-0.4 to 1.4)1.4)

Hueston, 94Hueston, 94 2323 ErythErythroro

Days of productive Days of productive coughcough

-0.4 (-2.4 to -0.4 (-2.4 to 1.6)1.6)

King, 96King, 96 9191 ErythErythroro

Days of sputum Days of sputum productionproduction

0.7 (-1.3 to 0.7 (-1.3 to 2.7)2.7)

Problem

How do you combine studies with slightly different outcomes?

Option 1: - don’t do itOption 2: Transform all outcomes

to an effect size

What is an Effect Size?

Effect size – a way of expressing results in a common metric

Units – standard deviation

Effect Size

ES = X1 – X2

SDpooled

1. ES increases as difference between means increases

2. ES increases as SD decreases

3. ES is expressed in units of SD

4. Summary ES combines the weighted ES from each study.

Effect Size

Effect Size

Rough Estimates– SMALL 0.2

– MEDIUM 0.5

– LARGE >0.7

Context– Mean Duration of Cough = 8 days

– Standard Deviation = 3 days

Main Result

Summary ES = 0.21 (95% CI 0.05 to 0.36)

Summary Mean Differences

Outcome MeasureOutcome Measure Summary Mean Summary Mean Difference (95% CI)Difference (95% CI)

Days of Productive Days of Productive Cough (6 studies)Cough (6 studies)

0.4 days (-0.1 to 0.8)0.4 days (-0.1 to 0.8)

Days of cough (4 Days of cough (4 studies)studies)

0.5 days (-0.1 to 1.1)0.5 days (-0.1 to 1.1)

Time off work (6 Time off work (6 studies)studies)

0.3 days (-0.6 to 1.1)0.3 days (-0.6 to 1.1)

Different Types of Data: Diagnostic Tests

Sensitivity and Specificity

Sensitivity TP/(TP + FN)Positive in Disease

SpecificityTN/(TN + FP)Negative in Health

TNFNTest

-

FPTPTest

+

Disease

-

Disease

+

(+) Likelihood Ratio = Sensitivity

1-Specificity

(-) Likelihood Ratio = 1-Sensitivity1-Sensitivity Specificity Specificity

Does this patient have a specific disease?

What we thought before (pre-test probability)

+ Clinical information (diagnostic test, LR)

= What we think after (post-test probability)

Diagnostic OR = +LR/-LR

= TP x TN / FP x FNSensitivity Specificity Pos LR Neg LR Diag OR

0.5 0.5 1 1 10.6 0.6 1.5 0.67 2.30.7 0.7 2.3 0.43 5.40.8 0.8 4 0.25 160.9 0.9 9 0.11 810.95 0.95 19 0.05 3610.99 0.99 99 0.01 9801

Example: US and CT for Appendicitis

Goal: to determine whether US or CT is a “better” test for the evaluation of suspected appendicitis.

Diagnostic tests are complicated because there are 5 potential outcomes to summarize– LR+, LR-– Sensitivity, Specificity– Diagnostic OR– Assess heterogeneity, publication bias for

EACH outcome

Advanced Topics

Individual participant dataMissing dataDifferent types of dataObservational studiesGeneralized synthesis of evidenceMeta-regressionCritique of a systematic review

Meta-regression

Examines whether the study effects (outcomes) are related to one or more of the study characteristics.

Can be used to understand/explain heterogeneity.

Can be thought of as an epidemiological study of the trials or studies.

Clinical Questions: Meta-Regression

Are there certain situations in which BCG may be more effective for preventing TB?

Meta-regression: example

StudyStudy OROR 95% CI95% CI

11 0.3910.391 0.121, 1.2620.121, 1.262

22 0.1890.189 0.077, 0.4620.077, 0.462

33 0.2500.250 0.069, 0.9090.069, 0.909

44 0.2330.233 0.176, 0.3080.176, 0.308

55 0.8030.803 0.514, 1.2560.514, 1.256

66 0.3840.384 0.316, 0.4660.316, 0.466

77 0.1950.195 0.077, 0.4970.077, 0.497

88 1.0121.012 0.894, 1.1460.894, 1.146

99 0.6240.624 0.391, 0.9960.391, 0.996

1010 0.2460.246 0.144, 0.4220.144, 0.422

1111 0.7110.711 0.571, 0.8860.571, 0.886

1212 1.5631.563 0.373, 6.5480.373, 6.548

1313 0.9830.983 0.582, 1.6610.582, 1.661

BCG vaccine: used to prevent tuberculosis

Odds ratio estimates from 13 trials (right)

Scientists have suggested that effects may be related to geographic latitude

Funnel Plot

Funnel Plot – Organized by Latitude

Meta-regression: example, continued

Log odds ratio versus absolute latitude:

Meta-regression: example, cont

Same plot, showing precision:


Same plot, with fitted (meta-)regression line:


Is the slope of the line significantly different from 0?

If yes, we conclude that the study effects are in fact related to latitude

Meta-regression: details

In a regression model for the data: each study represents one observation

Weights equal to the study precision

Random effects model (recommended)

Built-in function in Stata: ‘metareg’

Critique of a Systematic Review

1. Research Question2. Protocol3. Search4. Study selection (inclusion/exclusion)5. Quality assessment6. Data abstraction7. Analysis


8. Interpretation

Reviewing Journal Articles

Very little formal teaching“Because reviews are often highly

negative, the new researcher implicitly learns from the negative reviews received on his or her own submitted papers that reviews are supposed to be negative. It is as if the implicit message is: A reviewer’s job is to criticize the manuscript.”

12 Tips on Reviewing Articles

1. Know your mission 2. Be speedy 3. Read carefully 4. Say positive things in your review 5. Don’t exhibit hostility 6. Keep it brief 7. Don’t nitpick 8. Develop your own style 9. Be careful in recommending further experiments 10. Watch for egocentrism 11. Make a recommendation 12. Sign your review

http://www.psychologicalscience.org/observer/getArticle.cfm?id=2157

ConclusionsYou can combine almost anything

Use clinical judgment to guide you in deciding how and whether to combine studies.

Remember the main mission of a systematic review: to summarize a body of literature in a concise and clear way.

Get statistical input as needed.

systematic review: analytical methods of meta-analysis stephen bent, m.d. assistant professor of...

Documents

steps of systematic

meta analysis

bent s

interpretation slide

clinical management

kohlwes j

simon j

biostatistics ucsf slide