Download - Ispor workshop 08 april2016

How can policy-makers and clinicians trust the results of a network meta-analysis? A workshop on how to apply the International Society For Pharmacoeconomics and Outcomes Research (ISPOR) tool

Prepared for : 2016 CADTH Symposium

April 10, 2016

Knowledge Translation Program Li Ka Shing Knowledge InstituteSt. Michael's Hospital Toronto, Canada

Areti Angeliki Veroniki, MSc, PhDSharon E. Straus, MD, FRCPC, MSc

Andrea C. Tricco, MSc, PhD (contributor)

[email protected]@[email protected]

Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Canada

We have no actual or potential conflict of interest in relation to this presentation


of the presentation

• To gain knowledge about what a network meta-analysis (NMA) is • What is the definition of NMA and how is it related to

pairwise MA?

• To gain knowledge and skills in assessing the validity of an NMA • How do we know whether the NMA is credible?

• To work in small groups to establish the credibility of a NMA • Do we trust the results of this NMA?• Is there anything missing?• Could the authors have done things differently?

3


Network Meta-analysis (NMA)

NMA is an extension

of pairwise meta-

analysis that

simultaneously

compares multiple

treatments for a

medical condition

from two or more

studies that have one

treatment in common


14 serotonin (5-HT3) receptor antagonists combinations for vomiting for patients undergoing surgery

? Placebo

Ondansetron

Granisetron

DolasetronTropisetron

Ondansetron+Dexamethasone

Palonosetron

Ramosetron

Ondansetron+DroperidolIV

Ondan+Metoclopr IV

Granisetron+Dexamethasone

Palonosetron+DexamethasoneDolasetron+Dexamethasone

Dolasetron+DroperidolIV

Granisetron+DroperidolIV

5

238 RCTs and 33 meta-analyses Tricco et al BMC Medicine 2015


Terminology• Network meta-analysis (or multiple treatment meta-analysis or

mixed treatment comparison)o Combines direct and indirect data across a network of studies to

infer the relative effectiveness/safety of ≥3 interventions in a single model

• Indirect comparisono Allows the estimation of the relative effectiveness/safety of ≥2

treatments in the absence of head-to-head evidence

• Mixed comparisono Combines both direct and indirect evidence for a specific

comparison to obtain a weighted average of the estimates for a single treatment comparison at the time

Salanti Res Syn Meth 2012


7


Nikolakopoulou et al PLoS One 2013 Number of publications

1997

2000

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

0 10 20 30 40 50

The number of published systematic reviews that employ NMA are increasing over time.

Given the relevance of NMA to inform healthcare decision-making, it is of great interest to improve

understanding of these studies by decision-makers and policy-makers.

Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaExample – NMA published in BMJ

OpenPlease read the following paper

Tricco et al BMJ Open 2015


Jansen et al 2014

International Society for Pharmacoeconomics and Outcomes

Research (ISPOR) checklist


Questionnaire Items – 2 main domains

Relevance (4 questions)

Credibility (22 questions)

Analysis (7 questions)

Reporting quality & transparency (6 questions)

Interpretation (1 question)

Conflict of interest (2 questions)

Evidence Base (6 questions)


Scoring

• Credibility and Relevance are scored separately

• Questions are grouped by domain and each question is answered as ‘yes’/’no’/’unclear’

• Each domain can be rated as ‘strong’/‘weak’/’neutral’


Evidence base1. Did the researchers attempt to identify and include all

relevant RCTs?• Did the search strategy include terms for RCTs of all interventions

of interest?• Were multiple databases searched (e.g., MEDLINE, EMBASE, and

Cochrane Central Registry of Trials)?• Do the inclusion criteria include all RCTs of interest (if identified

by the literature search)?– Language limitations, search for unpublished material

A ‘yes’ to the above implies an adequate attempt to include all available RCTs


Systematic Review Conduct– The same considerations as in a traditional systematic review

and meta-analysis still apply!• A well-conducted systematic review follows the Cochrane Collaboration

methodology

– Importance of clinical input on the systematic review conduct • Need to ensure the team includes a range of expertise in systematic

review methods, clinical, and statistical domains

– Arguably more important to have a well-conducted underlying systematic review for NMA

• NMA used to base policy decisions on the population level = affect many more people


A Systematic Review usually has….

o Comprehensive (≥2 databases) and PRESS’ed literature search

o Pre-defined inclusion and exclusion criteria (i.e., study eligibility criteria)

o Risk of bias appraisal (Cochrane tool for randomized clinical trials)

o Pre-defined data abstraction form

o Synthesis based on the totality of evidence and reported using PRISMA

o Discussion, providing limitations of included studies and review process

o Protocol using PRISMA-P, PROSPERO registry, published in an open access journal (e.g., Sys Rev journal, BMJ Open)

o Each step conducted by 2 reviewers, independently

Cochrane Handbook (editors: Higgins and Green) 2011, Shamseer BMJ 2015, Sampson JCE 2009, Moher BMJ 2009


Evidence base1. TASK: Did the researchers attempt to identify and include

all relevant RCTs?


Evidence base2. Do the trials for the interventions of interest form one

connected network of RCTs?• Network geometry and connectedness

No_treat

ZIDO,AZT,ZDV

ZDV+3TC

Any_ARTZDV+3TC+ABC

ZDV+3TC+NEVI

HAART

Any_treat

NEVI

EFAV

LAMIV,3TCTENOF

d4T+3TC

ZDV+ddI+NEVI

No_treat

ZIDO,AZT,ZDV

ZDV+3TC

Any_ART

ZDV+3TC+ABC

ZDV+3TC+NEVI

HAART

TENOF d4T+3TC

ZDV+ddI+NEVI


Evidence base2. TASK: Do the trials for the interventions of interest form

one connected network of RCTs?


Evidence base3. Is it apparent that poor quality studies were included,

thereby leading to bias?• Cochrane ROB (Sensitivity analysis by components of ROB)

7

6

5

4

3

2

1

9%

39%

74%

98%

98%

16%

37%

39%

55%

14%

2%

1%

84%

63%

52%

5%

13%

1%

0%

0%

Low Unclear High

1 2 3 4 5 6 7Random Sequence

generationAllocation

concealmentBlinding of participants

and personnelBlinding of outcome

assessmentIncomplete

outcome dataSelective reporting

Other bias


Evidence base3. Is it apparent that poor quality studies were included,

thereby leading to bias?• Cochrane ROB (Sensitivity analysis by components of ROB)

Placebo

BUDE

FORMINDAC

SAML

ACLI

GLYC

TIOTFORM/BUDE VILA/FLUT

SALM/FLUT

INDA/GLYC

Randomization

Placebo

BUDE

FORMINDAC

SAML

ACLI

GLYC

TIOTFORM/BUDE VILA/FLUT

SALM/FLUT

INDA/GLYC

Allocation Concealment


Evidence base3. TASK: Is it apparent that poor quality studies were included,

thereby leading to bias?


Evidence base4. Is it likely that bias was induced by selective reporting of

outcomes in the studies?• Use the Cochrane ROB tool to help ascertain this

0.5

Sta

ndar

d er

ror o

f log

odd

s ra

tio

-2 -1 0 1 2Log-odds ratio centred at comparison-specific pooled effect

Comparison adjusted funnel plot

Tricco et al BMJ 2014; Chaimani et al Plos One 2013

Asymmetry


Evidence base4. TASK: Is it likely that bias was induced by selective reporting

of outcomes in the studies?


Evidence base5. Are there systematic differences in treatment effect

modifiers across the different treatment comparisons in the network?• Relates to baseline patient or study characteristics that have an impact on

the treatment effects (transitivity assumption)

20 25 30

20 25 3020 25 30

20 25 30

A

B

C

A

B

CInvalid Indirect Comparison × Valid Indirect

Comparison

Age effect modifier



modifiers across the different treatment comparisons in the network?• Relates to baseline patient or study characteristics that have an impact on

the treatment effects (transitivity assumption)

A

B

C

C

T

O

P

P

Ondasetron

P-InjectionP-Pill

Plac

ebo Might be an

inappropriate common comparator

Tropisetron



modifiers across the different treatment comparisons in the network?

Placebo

Ondan-4mgOndan-8mg

Ondan-16mgOndan-12mg

Ondan-1mgOndan-3mg



Ondan-2mgOndan-10mgOndan-6mg

Ondan-5mgOndan-0.2mgOndan-9mg

Grani-0.1mgGrani-0.2mg

Grani-0.3mgGrani-1mg

Grani-2mgGrani-3mg



Grani-0.4mgGrani-1.1mgGrani-0.7mgGrani-2.2mgGrani-0.8mg

Dola-12.5mgDola-25mgDola-37.5mg

Dola-7.0mgDola-54.0mg

Tropi-2mgTropi-5mgTropi-0.5mg

Tropi-0.1mgTropi-1mg

Tropi-1.5mgTropi-7.3mg

Tropi-4.3mgPalono-0.025mgPalono-0.05mgPalono-0.075mg

Palono-0.008mgPalono-0.021mg

Palono-0.074mgPalono-0.219mg

Palono-2.130mgPalono-0.25mgRamo-0.3mgRamo-0.6mgRamo-0.1mgRamo-0.9mgRamo-0.2mgPlacebo

OndasetronGranisetro

n

Dolasetron

Tropisetron

Palonosetron Ramosetron

Lumping or splitting nodes?



modifiers across the different treatment comparisons in the network?• ‘Missing’ arms are missing at random: Observed and unobserved

estimates do not differ beyond what can be explained by heterogeneity

Study Observed

AC

AB

Study If arm were included…

Observed and Unobserved

AC B

AB C

Lu and Ades 2006B

C

A

C

B

B might have been used in trials in the 1990s , whereas C in 2000s



modifiers across the different treatment comparisons in the network?• Interpretation of transitivity:

1. Treatment A is similar when it appears in AB and AC trials

2. The two sets of trials AB and AC do not differ with respect to the distribution of effect modifiers.

3. Participants included in the network could in principle be randomized to any of the 3 treatments A, B, C

4. ‘Missing’ treatment in each trial is missing at random

5. There are no differences between observed and unobserved relative effects of AB and AC beyond what can be explained by heterogeneity

Salanti Res Synth Methods 2012

B

C

A


Evidence base5. TASK: Are there systematic differences in treatment effect

modifiers across the different treatment comparisons in the network?

Hint: The answer to question 5 is a “yes” if there are substantial (or systematic) differences in effect modifiers, which can be judged by comparing study-specific inclusion and exclusion criteria, baseline patient characteristics, and study characteristics that are expected to be effect modifiers.


Evidence base6. If yes, were these imbalances in effect modifiers across the

different treatment comparisons identified before comparing individual study results?

It is recommended:a) Before undertaking a NMA, generate a list of potential treatment

effect modifiersb) Next, compare the suggested treatment effect modifiers across

studies to identify any imbalances between the different types of direct comparisons in the network.




Orange: no pregnant women

NPH[od/bid]

NPH[od]

D[od/bid]

D[qid]

D [od]G[bid] G[od]

Placebo

O

G

DTO+D

P

R

O+D_IV

O+M_IV

G+DexP+Dex

D+Dex

D+D_IV

G+D_IV

1: children (orange), 2:adults (purple), 3:all (red)




Treatment Comparison RoB - Allocation concealment RoB - Incomplete outcome data Age AD severity Comorbidities

DONE vs PLAC High High >75 Moderate-Severe 0RIVA vs PLAC Low Low <75 Mild-Moderate 0GALA vs DONE Unclear Unclear <75 Mild-Moderate 0RIVA vs DONE Low Low <75 Mild-Moderate 0

DONE vs PLAC Unclear High <75 Moderate 0GALA vs PLAC Low High <75 Moderate 0RIVA_O vs PLAC Low High <75 Mild-Moderate 0RIVA_P vs PLAC Low High <75 Moderate 0MEMA vs PLAC High High <75 Moderate-Severe 0GALA vs DONE Unclear High <75 Mild-Moderate 0RIVA_O vs DONE Unclear High <75 Mild-Moderate 0RIVA_O vs GALA Unclear Unclear <75 Mild-Moderate 0RIVA_P vs RIVA_O Unclear High >75 Mild-Severe 0

Bradycardia outcome

Diarrhea outcome


Evidence base6. TASK: If yes, were these imbalances in effect modifiers

across the different treatment comparisons identified before comparing individual study results?


Validity [or Credibility as per ISPOR]


Analysis7. Were statistical methods used that preserve within-study

randomization?• No naïve indirect comparisons

A

B

C

Although NMA is based on RCTs, randomization does not hold across the set of trials used for the analysis because patients are not randomized to different trials.


Analysis7. Were statistical methods used that preserve within-study

randomization?

Treatment comparison Studies

NPH[od/bid] Detemir[od/bid] 6

NPH[od/bid] Glargine[od] 2

NPH[od] Detemir[od] 4

Detemir[od/bid] Glargine[od] 1

Detemir[qid] Glargine[od] 1

Detemir[od] Glargine[od] 1

Glargine[bid] Glargine[od] 1

NPH[od] Glargine[od] -

Tricco et al BMJ 2014 NPH: Neutral Protamine Hagedorn, od: once daily, bid:twice daily

Interventions for type 1 diabetes and severe hypoglycemia: series of pairwise meta-analyses


Neutral Protamine Hagedorn

[once / twice daily]

Neutral Protamine Hagedorn [once daily]

Detemir [once daily / twice daily]

Detemir [four times daily]

Detemir [once daily]

Glargine [twice daily]

Glargine [once daily]

6

4

2

1

1

1

1

Tricco et al BMJ 2014 36


How to compare NPH[od] to Glargine[od]?


Detemir [once daily] Glargine [once daily]

2

1

?

Tricco et al BMJ 2014 37


Indirect Comparison

NPH [od]

Glargine [od]

Detemir [od]

Comparison Odds ratio (OR) 95% Confidence Interval (CI)

NPH[od] vs Detemir[od] 1.72 (1.20, 2.50)Glargine[od] vs

Detemir[od] 2.21 (0.63, 7.77)

How to compare NPH[od] to Glargine[od]?Estimate indirect OR and 95% CI

38Bucher et al J Clin Epidemiol 1997


1. Indirect treatment effect:

2. Variance=((High CI – Low CI)/3.92)2

3. 95% CI for the indirect estimate:

=[0.28, 0.92] =[-0.46, 2.05]

=

Indirect Comparison

39NG:NPH [od] vs Glargine [od] , ND:NPH [od] vs Detemir [od] , GD: Glargine [od] vs Detemir [od]

=0.04 =0.41

==0.45

=


Unadjusted/Naïve Indirect ComparisonTreatment A Treatment B

Treatment A Treatment C

𝑅 𝑅𝐵𝑣𝑠𝐶=

1333212431

23674

=2.27

Naïve Indirect Comparison

Adjusted Indirect Comparison

𝑅 𝑅𝐵𝑣𝑠𝐶=0.30

Should be avoided!

40


Indirect OR for NPH[od] vs. Glargine[od]: 0.78 [0.40, 1.52]









6

4

2

1

1

1

13

41


Indirect and mixed effects

B

C

A

B

C

Indirect effectMixed effect

Direct effectOdds ra

tio

0.01 0.11

10

Odds ratio

0.01 0.11

10

Odds ratio0.01 0.1 1 10

42

B

C


𝑉𝑎𝑟 (𝑀𝑖𝑥𝑒𝑑 𝐿𝑜𝑔𝑂𝑅)=1

1𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐷𝑖𝑟𝑒𝑐𝑡 )

+1

𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡)

𝑀𝑖𝑥𝑒𝑑 𝐿𝑜𝑔𝑂𝑅=

𝐿𝑜𝑔𝑂 𝑅𝐷𝑖𝑟𝑒𝑐𝑡

𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂𝑅𝐷𝑖𝑟𝑒𝑐𝑡 )+

𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡

𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡)1

𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂𝑅𝐷𝑖𝑟𝑒𝑐𝑡 )+ 1

𝑉𝑎𝑟 (𝐿𝑜𝑔𝑂 𝑅𝐼𝑛𝑑𝑖𝑟𝑒𝑐𝑡)

Mixed comparison

Meta-analysis Pooled Effect

Effect estimate and

variance

43Salanti Res Synth Methods 2012

Mixed Comparison

Direct Comparison

Indirect ComparisonLog odds ratio

0

-1.560

1.56

Summarize direct and indirect effect size into a single

mixed effect


Mixed comparisonNPH [od]

Detemir [od] Glargine [od]

6

4

21

11

13

We gain precision

NPH[od] Vs. Glargine[od] LogOR [Variance]

Direct Comparison 0.10 [0.20]

Indirect Comparison -0.25 [0.45]

Mixed Comparison -0.008 [0.14]

44


Extend the idea of mixed effect sizes in the entire network









6

4

2

1

1

1

13

45


Extend the idea of mixed effect sizes in the entire network

Placebo

Ondasetron

Granisetron

Dolasetron

TropisetronOnd+De

x

Palonosetron

Ramosetron

Ond+Drop IV

Ond+Metoclop IV

Gran+Dex Palon+De

x

Dolas+Dex

Dolas+Drop IV

Gran+Drop IV

You need to use more sophisticated techniques to

simultaneously analyze evidence coming from the

entire network


Comprehensive use of all available data

Avoids selective use of indirect evidence

Comparison of interventions which haven’t been directly compared in any experiment

Network meta-analysis

Plac

Ond

Gran

DolasTrop

Ond+DexPalon

Ramos

Ond+Drop IV

Ond+Metoclop IV

Gran+DexPalon+Dex

Dolas+Dex

Dolas+Drop IV

Gran+Drop IV

47





It can increase precision in the estimated treatment effects


48



Ondansetron vs Placebo

Granisetron vs Placebo

Dolasetron vs Placebo

Tropisetron vs Placebo

Ondansetron+DEX vs Placebo

Palonosetron vs Placebo

Ramosetron vs Placebo

Ondansetron+DROP vs Placebo

Ondansetron+METO vs Placebo

Granisetron+DEX vs Placebo

Dolasetron+DEX vs Placebo

Dolasetron+DROP vs Placebo

Granisetron+DROP vs Placebo

Treatment Comparison

0.35 (0.32, 0.39)0.36 (0.33, 0.40)0.24 (0.16, 0.34)0.26 (0.21, 0.34)0.42 (0.21, 0.83)0.44 (0.30, 0.63)0.32 (0.22, 0.48)0.32 (0.23, 0.43)0.16 (0.09, 0.27)0.16 (0.12, 0.23)0.53 (0.38, 0.73)0.38 (0.24, 0.60)0.42 (0.26, 0.68)0.28 (0.18, 0.43)0.15 (0.07, 0.31)0.14 (0.08, 0.26)0.16 (0.06, 0.43)0.15 (0.06, 0.42)0.16 (0.08, 0.31)0.15 (0.09, 0.24)0.06 (0.01, 0.30)0.18 (0.06, 0.49)0.16 (0.07, 0.35)0.19 (0.07, 0.52)0.30 (0.05, 1.66)0.31 (0.11, 0.82)

0.35 (0.32, 0.39)0.36 (0.33, 0.40)0.24 (0.16, 0.34)0.26 (0.21, 0.34)0.42 (0.21, 0.83)0.44 (0.30, 0.63)0.32 (0.22, 0.48)0.32 (0.23, 0.43)0.16 (0.09, 0.27)0.16 (0.12, 0.23)0.53 (0.38, 0.73)0.38 (0.24, 0.60)0.42 (0.26, 0.68)0.28 (0.18, 0.43)0.15 (0.07, 0.31)0.14 (0.08, 0.26)0.16 (0.06, 0.43)0.15 (0.06, 0.42)0.16 (0.08, 0.31)0.15 (0.09, 0.24)0.06 (0.01, 0.30)0.18 (0.06, 0.49)0.16 (0.07, 0.35)0.19 (0.07, 0.52)0.30 (0.05, 1.66)0.31 (0.11, 0.82)

Odds Ratio (95% CI)

1.01 1 100

Tricco et al 2015 BMC Medicine

Treatments for 5ht3 surgery data

Network Estimates

Direct Estimates

We gain precision

49





It can increase precision in the estimated treatment effects

It can rank all competing treatments for the same condition


50


0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

Dolasetron Dolasetron+Dexamethasone Dolasetron+DroperidolIV Granisetron

Granisetron+Dexamethasone Granisetron+DroperidolIV Ondansetron Ondansetron+Dexamethasone

Ondansetron+DroperidolIV Ondansetron+MetoclopramideIV Palonosetron Palonosetron+Dexamethasone

Placebo Ramosetron Tropisetron

Pro

babi

litie

s of

eac

h ra

nk

Rank


Tricco et al 2015 BMC Medicine

Treatment SUCRA Mean Rank

Ond+Drop IV 85.4 3Gran+Dex 84 3

Ond+Dex 79.5 4Ond+Met IV 78.6 4

Dol+Dex 71.8 5Dola+Drop IV 68.2 6

Gran 54.4 7Ramos 49.2 8Tropis 41.6 9

Gran+Drop IV 44.6 9Ond 30.8 11

Palon 29.1 11 Dolas 20.8 12

Placebo 4.6 14Palon+Dex 7.5 14

51


Analysis7. TASK: Were statistical methods used that preserve within-

study randomization?


Analysis8. If both direct and indirect comparisons are available, was

agreement in treatment effects evaluated or discussed?• Relates to consistency

Indirect evidence

Direct evidence

Are the results valid?Network

Meta-analysis Results

B

C

A

B

C

If all three A, B and C are transitive then the loop is consistent



Nikolakopoulou et al PLoSOne 2013

01020304050Number of publications

Bayesian hierarchical model

Adjusted Indirect ComparisonNot reported

Meta-regression

0 10 20 30 40 50

1997

2000

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

Full networkStar network

Systematic reviews that employ NMA are undertaken and published with increasing frequency.

BUT! The validity of the results from NMA rests on the assumption of transitivity!

54


B

C

A

Consistency = transitivity across a loopThe TRANSITIVITY assumption states that:‘the benefit of A over B’ =

‘the benefit of A over C’ +

‘the benefit of C over B’

The CONSISTENCY assumption states that:‘the overall treatment effect in AB studies’ =

‘overall treatment effect in AC studies’ +

‘overall treatment effect in CB studies’

Untestable assumption

Testable assumption

55

When the common comparator is transitive, it allows a valid comparison of the treatments to which it is linked

Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaAssumption underlying indirect comparison

and NMA (in addition to considering homogeneity)

Assumption for indirect and mixed comparison

Conceptual definition

(Transitivity)

Clinical Methodological

Property of parameters and

data (Consistency)

Statistical

Cipriani et al Ann of Int Medicine 2013 56


B

C

A

B

C

Assumption of consistency

Direct and indirect

evidence are in agreement

Consistency is a property of a ‘closed loop’ - a path that starts and ends at the same node.

57


010

2030

40N

umbe

r of p

ublic

atio

ns

1997

2000

2002

2003

2005

2006

2007

2008

2009

2010

2011

2012

Appropriate statistical methods

Inappropriate methodsNone reported

A database of 186 NMAs showed that… In 24% of the networks the authors used inappropriate methods to evaluate consistency

- Comparison of direct with NMA estimates- Comparison of previous meta-analyses with NMA

results In 44% of the networks the authors did not report a method to evaluate consistency


Nikolakopoulou et al PLoS One 2013 58


Forms of InconsistencyLoop Inconsistency

AC

AB

Direct AB

Indirect ABB

C

A

BC

Lu and Ades JASA 2006

If they statistically differ :Inconsistency!

59


Forms of InconsistencyDesign Inconsistency

AB

ABDesign AB

Design ABC

If they statistically differ :

B

C

A

Inconsistency!White et al RSM 2012Higgins et al RSM 2012 60


Approaches for evaluating…LOCAL INCONSISTENCY Loop-Specific (LS) Node-splitting / Separating Indirect and Direct Evidence (SIDE) Separating One Design from the Rest (SODR)

GLOBAL INCONSISTENCY Composite test for inconsistency Lu and Ades (LA) Design by treatment interaction (DBT)

Note: There is also Comparison of model fit and parsimony between consistency and inconsistency models approach

- Requires Bayesian framework – uses the measures of model fit & parsimony (e.g. DIC)

- Does not provide inconsistency estimates- Infers on global inconsistency 61


Properties of the inconsistency approaches

Loop-Specific

SIDE/Node splitting SODR Composite

test LA DBT

Simple to compute Insensitive to parameterization of multi-arm studies

Indirect estimate derived from the entire network

Does not suffer from multiple testing Power ? ? ?

62 Song et al BMC Med Res Methodol 2012, Veroniki et al BMC Med Res Methodol 2014


• Lower statistical heterogeneity is associated with greater chance to detect inconsistency but the estimated magnitude of inconsistency is lower

• Low power: ‘Absence of evidence is not evidence of absence’• The methodological and clinical plausibility of the consistency assumption

should be further considered

• The lack of direct evidence (‘open’ loops) makes the statistical evaluation of consistency impossible • But the transitivity assumption is still needed to derive the indirect estimate!

Issues with statistical estimation of consistency

Song et al BMC Med Res Methodol 2012, Veroniki et al IJE 2013, Veroniki et al BMC Med Res Methodol 2014

1

2

3

4

5

6

7

8

910

11

12

Consistent?


• Lower statistical heterogeneity is associated with greater chance to detect inconsistency but the estimated magnitude of inconsistency is lower.

• Low power: ‘Absence of evidence is not evidence of absence’• The methodological and clinical plausibility of the consistency assumption

should be further considered

• The lack of direct evidence (‘open’ loops) makes the statistical evaluation of consistency impossible • But the transitivity assumption is still needed to derive the indirect estimate!

• Results and inferences on the prevalence of inconsistency are sensitive to the estimation method of heterogeneity

o Magnitude of heterogeneityo Random/Fixed effects to derive treatment effect estimateso Estimation method for heterogeneity (DL, REML, SJ etc)o Same or different heterogeneity across comparisons in the network

Issues with statistical estimation of consistency

Song et al BMC Med Res Methodol 2012, Veroniki et al IJE 2013, Veroniki et al BMC Med Res Methodol 2014


Inconsistency - Heterogeneity

Direct AB

Indirect AB

AC

BC

Important heterogeneity might decrease the prevalence of inconsistency!

Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaWhat should readers look for when

statistically significant inconsistency is found in an NMA?

2. Potentially no NMA: Investigators may decide not to conduct a NMA in the presence

of excessive inconsistency. Direct and indirect results may be presented separately.

• The magnitude of the estimated inconsistency factor and its confidence interval may be

interpreted

3. Exploration of inconsistency: The network may be split into subgroups or may use

network meta-regression to account for differences across studies and comparisons.

4. Encompass inconsistency in the results: May apply DBT or LA models that relax

the consistency assumption

1. Assessment for data abstraction errors: Inconsistency in loops where a

comparison is informed by a single study is particularly suspicious for

data errors.

66


Analysis8. TASK: If both direct and indirect comparisons are available,

was agreement in treatment effects evaluated or discussed?

• If authors report that all but one comparison was not consistent, is the NMA still invalid? Would 1 inconsistent loop affect the whole network?

“According to the two-level random-effects model, no important incoherence between comparisons was detected (incoherence = .001). Only one particular comparison loop showed incoherence.”


Analysis9. TASK: In the presence of consistency between direct and

indirect comparisons, were both direct and indirect evidence included in the network meta-analysis?

Tricco et al BMC Medicine 2015

Network Estimates using the random effects model and common within-network heterogeneity

Ondansetron+DroperidolIVGranisetron+Dexamethasone

Ondansetron+MetoclopramideIVOndansetron+DexamethasoneDolasetron+Dexamethasone

Dolasetron+DroperidolIVGranisetronRamosetron

Granisetron+DroperidolIVTropisetron

OndansetronPalonosetron

DolasetronPalonosetron+Dexamethasone

0.14 (0.08,0.26) (0.06,0.36)0.15 (0.09,0.24) (0.06,0.35)

0.15 (0.06,0.42) (0.04,0.53)0.16 (0.12,0.23) (0.08,0.36)0.18 (0.06,0.49) (0.05,0.62)

0.19 (0.07,0.52) (0.06,0.65)0.26 (0.21,0.34) (0.12,0.56)0.28 (0.18,0.43) (0.12,0.64)

0.31 (0.11,0.82) (0.09,1.03)0.32 (0.23,0.43) (0.15,0.69)

0.36 (0.33,0.40) (0.18,0.74)0.38 (0.24,0.60) (0.16,0.89)

0.44 (0.30,0.63) (0.20,0.97)1.43 (0.20,10.14) (0.18,11.60)

OR 95%CI 95%PrITreatment effect

0 .2 1 3 12

Reference treatment: Placebo


Analysis10. With inconsistency or an imbalance in the distribution of

treatment effect modifiers across the different types of comparisons in the network of trials, did the researchers attempt to minimize this bias with the analysis?• Subgroup analyses, meta-regression, inconsistency models

Covariate (Treatment effect modifier, e.g., age)

Trea

tmen

t Eff

ect E

stim

ate

(e.g

., Lo

g-O

R)

Children Elderly


Analysis10. With inconsistency or an imbalance in the distribution of

treatment effect modifiers across the different types of comparisons in the network of trials, did the researchers attempt to minimize this bias with the analysis?• Subgroup analyses, meta-regression, inconsistency models• Apply an individual patient data NMA (IPD NMA)

Apply network meta-regression, assuming that the estimated relationship between effect modifier and treatment effect is not affected by other biases (e.g., aggregation bias ) – otherwise apply an IPD-NMA!

• IPD-NMA with patient-level covariates allows modeling within-study variation of effect modifiers. Hence, bias due to an imbalance of patient-level characteristics across comparisons can be minimized.


Analysis10. TASK: With inconsistency or an imbalance in the distribution of

treatment effect modifiers across the different types of comparisons in the network of trials, did the researchers attempt to minimize this bias with the analysis?• Subgroup analyses, meta-regression, inconsistency models• Apply an individual patient data NMA (IPD NMA)

Hint: This question should be answered with a ‘yes’: In the absence of inconsistency and absence of

differences in effect modifiers across treatment comparisons.

If meta-regression (or subgroup analyses) have been used to explore inconsistency or bias.o If inconsistency is identified and the authors did

not attempt to adjust, the answer should be ‘no’.


Analysis11. Was a valid rationale provided for the use of random-effects (RE)

or fixed-effect (FE) models?• Same considerations as in pairwise meta-analysis!

FE: no clinical or methodological heterogeneity is expected (Tricco, Open Med, 2012)

RE: clinical and/or methodological heterogeneity is expected (Tricco, CMAJ, 2014)


Analysis11. Was a valid rationale provided for the use of RE or FE models?

• FE = Fixed (common) effect, no effect modifiers

Random-effects model assumption:The observed study-specific effects estimate different true effects, which are related

and come from the same distribution

Fixed-effect model Assumption:Studies are sufficiently similar in aspects that could modify the treatment effect

There is a single true effect for all studies in the same treatment comparison.

Each study has each own true effect – all study-specific true effects are exchangeable

within the same treatment comparison.


Analysis11. Was a valid rationale provided for the use of RE or FE models?

FE and RE NMAs may be identical when the between-study variance is zero (if common within-network heterogeneity is assumed)

Fixed-effect model is often unrealistic “Since systematic reviews bring together studies that are diverse both clinically and

methodologically, heterogeneity in their results is to be expected.”

Random-effects meta-analysis suitable for unexplained small to moderate heterogeneity

When all treatment comparisons in a network include a single study, the RE model is not feasible

“We decided to apply a RE model, as we expected methodological and clinical heterogeneity across the included studies that compared the same

pairs of interventions.”

Tricco et al., BMC Med 2015

Higgins et al., BMJ 2003


Analysis11. TASK: Was a valid rationale provided for the use of RE or FE

models?Examples for rationale selecting between fixed-effect and random-effects model

“A random-effects network meta-analysis was conducted because we anticipated that the treatment effects were heterogeneous across the included RCTs. We assumed common heterogeneity across treatment comparisons, as the included treatments are of the same nature, hence, clinically reasonable to share a common heterogeneity parameter.”

“According to prior analyses in the field we anticipated a small number of included trials, which limits the applicability of random effect models because vague and weak informative prior distributions of the between-study variance have been shown to exert an unintentionally large degree of influence on any inference. Fixed effect models using vague prior distributions for all means were performed”

“Random effect models were consistently presented to account for the between study heterogeneity. However, in the case of the IPD analyses, fixed effect models were used given the limited number of studies to estimate the between study heterogeneity”


Analysis12. If a RE model was used, were assumptions about heterogeneity

explored or discussed?

• The heterogeneity variance can be estimated using several approaches– E.g., DerSimonian and Laird (DL), Maximum Likelihood (ML), Restricted Maximum Likelihood

(REML)

• In a Bayesian framework several priors can be assigned– E.g., Informative, Minimally informative, Vague

• Several heterogeneity assumptions can be considered– Common-within network heterogeneity: All treatment comparisons in the network are associated

with the same magnitude of heterogeneity

– Comparison-specific heterogeneity: Each treatment comparison in the network has its own amount of heterogeneity

– Common-within loop heterogeneity: The treatment comparisons included in each closed loop in the network share the same amount of heterogeneity.


12. TASK: If a RE model was used, were assumptions about heterogeneity explored or discussed?

Analysis

Examples for the heterogeneity assumption

“This model assumes that the degree of between-study within-comparison heterogeneity is constant across all intervention comparisons in the network.”

“We assume a common heterogeneity parameter across all comparisons”

“The between study variability of treatment effects was assigned a uniform (0, 2) prior density and assumed to be the same for all pairwise comparisons”

“The model was fitted into a Bayesian context with hierarchical models. The model used the assumption that the between-trial heterogeneity was equal across all comparisons.”

“We assumed common heterogeneity across treatment comparisons, as the included treatments are of the same nature and it was clinically reasonable to share a common heterogeneity parameter”


Analysis13. If there are indications of heterogeneity, were subgroup analyses or

meta-regression analyses with pre-specified covariates performed?• Pre-specified covariates based on patterns in the data

Placebo

BUDE

FLUT

MOMEAZD3199

FORMINDACSAML

VILA

ACLIGLYC

TIOT

UMEC

FORM/BECLO

FORM/BUDE

VILA/FLUT

SALM/FLUT

FORM/MOME

TIOT/FORMTIOT/SALMIND/TIOT

INDA/GLYCVILA/UMEC

GSK961081

TIOT/FLUT/SALM

TIOT/BUDE/FORM

Full network Purple: No exacerbations in previous year; Red: Had an exacerbation in previous year; Green: Not Reported

Placebo

BUDE

FLUT

MOME

AZD3199FORM

INDACSAMLVILA

ACLIGLYC

TIOT

UMEC

FORM/BECLO

FORM/BUDE

VILA/FLUT

SALM/FLUT

FORM/MOMETIOT/FORM

TIOT/SALM IND/TIOTINDA/GLYC

VILA/UMEC

GSK961081

TIOT/FLUT/SALM

TIOT/BUDE/FORM


Analysis13. If there are indications of heterogeneity, were subgroup analyses or

meta-regression analyses with pre-specified covariates performed?• Pre-specified covariates based on patterns in the data

Sub-network based on patients with exacerbations in past year or more

Placebo

BUDE

FLUT

FORM

INDACSAML

VILA

GLYC

TIOT

FORM/BECLO

FORM/BUDE

VILA/FLUT

SALM/FLUT TIOT/SALMINDA/GLYC

TIOT/FLUT/SALM

TIOT/BUDE/FORM


Analysis13. TASK: If there are indications of heterogeneity, were subgroup analyses

or meta-regression analyses with pre-specified covariates performed?


Reporting quality14. TASK: Is a graphical or tabular representation of the

evidence network provided with information on the number of RCTs per direct comparison?• Network diagram, table with studies in the rows and interventions and

results in the columns

NPH [od / td]

NPH [od]

D[od/ td]

D[fd]

D[od]

G [td]

G[od]

6

4

2

1

11

13

Comparison # Studies # Patients # EventsA vs B 2 185 94A vs C 4 708 428A vs D 3 586 283B vs C 1 102 70B cs D 2 1087 592C vs D 1 300 28


Reporting quality15. TASK: Are the individual study results reported?

• Increases reproducibility and face validity

Treatment A B C D E

Study# of

Events# of

Patients# of

Events# of

Patients# of

Events# of

Patients# of

Events# of

Patients# of

Events# of

Patients1 4 50 37 50 2 1 35 20 45 3 1 17 18 34 52 68 4 6 158 141 158 5 8 267 437 504 20 3306 88 112 67 120 7 0 57 15 70 8 3 27 4 30 9 13 150 15 140

10 0 49 20 70 11 15 71 2 80 12 0 87 128 181 13 0 89 136 178


Reporting quality16. TASK: Are results of direct comparisons reported separately

from results of the indirect comparisons or NMA?• Consistency assumption

Tricco et al BMJ 2014


Reporting quality17. TASK: Are all pairwise contrasts between interventions as

obtained with the NMA reported along with measures of uncertainty?• For example, ORs and 95% CrIs

A

3.71(0.65 – 6.00) B

1.83(0.13 – 3.00)

1.66(0.19 – 4.76) C

2.70(0.45 –5.00)

2.54(0.18 – 5.32)

1.52(0.09 – 7.72) D

League Table



Reporting quality18. Is a ranking of interventions provided given the reported

treatment effects and its uncertainty by outcome?• Ranking probabilities


Placebo

O

G

DTO+D

P

R

O+D_IV

O+M_IV

G+Dex

P+Dex

D+Dex

D+D_IV

G+D_IV



treatment effects and its uncertainty by outcome?• Ranking probabilities

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

0.2

.4.6

.81

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

Dolasetron Dolasetron+Dexamethasone Dolasetron+DroperidolIV Granisetron

Granisetron+Dexamethasone Granisetron+DroperidolIV Ondansetron Ondansetron+Dexamethasone

Ondansetron+DroperidolIV Ondansetron+MetoclopramideIV Palonosetron Palonosetron+Dexamethasone

Placebo Ramosetron Tropisetron

Pro

babi

litie

s of

eac

h ra

nk

Rank

Treatment SUCRA Mean Rank

Ond+Drop IV 85.4 3Gran+Dex 84 3

Ond+Dex 79.5 4Ond+Met IV 78.6 4

Dol+Dex 71.8 5Dola+Drop IV 68.2 6

Gran 54.4 7Ramos 49.2 8Tropis 41.6 9

Gran+Drop IV 44.6 9Ond 30.8 11

Palon 29.1 11 Dolas 20.8 12

Placebo 4.6 14Palon+Dex 7.5 14




treatment effects and its uncertainty by outcome?• Rank-heat plot

Veroniki et al JCE 2016


Reporting quality18. TASK: Is a ranking of interventions provided given the

reported treatment effects and its uncertainty by outcome?


Reporting quality19.Is the effect of important patient characteristics on

treatment effects reported?• Relative treatment effects for different levels of the effect modifier

Author, year

Sample size

Age Category

% female ASA statusSurgery

type

H/o motion

sickness H/o PONV

Comorbidities

Randomized clinical trials (n=429)

Browning, 2013

118 Adults NR I or II

Obstetrics &

Gynaecological

NR NRMental health

Aleyasin, 2012

120 Adults 100 NR Breast NR NR NR

Blitz, 2012 118 Adults 84 I or IIMiscellaneo

usNR NR NR

Choi, 2012 120 Adults 100 I or IIOrthopaedi

cYES YES NR

de Orange, 2012 129 Children 24 I or II

Miscellaneous

NR YES NR

Eidi, 2012 219 Adults 72 I or IIEar, nose

and larynxNR NR NR



Reporting quality19. TASK: Is the effect of important patient characteristics on

treatment effects reported?• Other important items for reporting can be found in the PRISMA for

NMA

Hutton et al Annals of Int Med 2015


Interpretation20. Are the conclusions fair and balanced?

• Consistent with NMA results and available evidence base, credible analysis methods, no bias concerns

NPH [od / td]

NPH [od]

D[od/ td]

D[fd]

D[od]

G [td]

G[od]

6

4

2

1

11

1

Network of hypoglycemia


Inconsistency (DBT model): χ2=7, d.f.=1, P-value=0.0082

Should we trust these results?


Interpretation20. Are the conclusions fair and balanced?

• Consistent with NMA results and available evidence base, credible analysis methods, no bias concerns

NPH [od / td]

D[od/ td]

D[fd]

D[od]

G [td]

G[od]

6

31

11

1

Network of hypoglycemia


Inconsistency (DBT model): χ2=2.56, d.f.=1, P-value=0.1093

High risk of bias in 3 studies caused inconsistency!


Interpretation20. TASK: Are the conclusions fair and balanced?


Conflict of interest21. Were there any potential conflicts of interest?

• Financial or personal relationships or affiliations that could bias the results


Conflict of interest21. TASK: Were there any potential conflicts of interest?


Conflict of interest22. TASK: If yes, were steps taken to address these?

• All COI should be noted, publication should be peer-reviewed, contribution of each author reported


Knowledge Translation, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, CanadaExample – NMA published in BMJ

OpenHow would you explain the results to a decision maker?

Tricco et al BMJ Open 2015


1. Baldwin D, Woods R, Lawson R, Taylor D. Efficacy of drug treatments for generalised anxiety disorder: systematic review and meta-analysis. BMJ. 2011 Mar 11;342:d1199.

2. Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997;50(6):683-91.

3. Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in STATA. PLoS One. 2013 Oct 3;8(10):e76654

4. Cipriani A, Higgins JP, Geddes JR, Salanti G. Conceptual and Technical Challenges in Network Meta-analysis. Ann Intern Med 2013;159:130-137.

5. Donegan S, Williamson P, D'Alessandro U, Tudur-Smith C. Assessing key assumptions of network meta-analysis: a review of methods Res Synth Meth. 2013; 4(4), 291-323.

6. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.

7. Jansen J, Trikalinos TA, Cappelleri JP, et al, Indirect treatment comparison/network meta-analysis study questionnaire to assess study relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health 2014;17, 157–173

8. Lu G, Ades A. Modeling between-trial variance structure in mixed treatment comparisons. Biostatistics. 2009; 10(4) :792-805.

9. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009 Jul 21;339:b2535.

10. Nikolakopoulou A, Chaimani A, Veroniki AA, Vasiliadis HS, Schmid CH, Salanti G, Characteristics of networks of interventions: a description of a database of 186 published networks. PLoS One. 2014;9(1):e86754.

11. Higgins JPT, Jackson D, Barrett JK, Lu G, Ades AE, White IR. Consistency and inconstency in network meta-analysis: concepts and models for multi-arm studies. Research Synth. Meth. 2012;3, 98-110.

12. Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res. Synth. Meth. 2012;3(2), 80-97.

13. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, et al; PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015 Jan 2;349:g7647.

References (1)

http://www.cochrane-handbook.org/


14. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009 Sep;62(9):944-52.

15. Song F, Xiong T, Parekh-Bhurke S, Loke YK, Sutton AJ, Eastwood AJ, et al. Inconsistency between direct and indirect comparisons of competing interventions: Meta-epidemiological study. BMJ. 2011;343:d4909.

16. Tricco AC, Ashoor HM, Antony J, Beyene J, Veroniki AA, Isaranuwatchai W, et al. Safety, effectiveness, and cost effectiveness of long acting versus intermediate acting insulin for patients with type 1 diabetes: systematic review and network meta-analysis, BMJ. 2014;349:g5459

17. Tricco AC, Soobiah C, Blondal E, Veroniki AA, Khan PA, Vafaei A, et al. Comparative efficacy of serotonin (5-HT3) receptor antagonists in patients undergoing surgery: A systematic review and network meta-analysis, BMC Medicine. 2015; 13:136

18. Tricco AC, Alateeq A, Tashkandi M, Mamdani M, Al-Omran M, Straus SE. Histamine H2 receptor antagonists for decreasing gastrointestinal harms in adults using acetylsalicylic acid: systematic review and meta-analysis. Open Med. 2012 Aug 21;6(3):e109-17.

19. Tricco AC, Antony J, Ivers NM, Ashoor HM, Khan PA, Blondal E, Ghassemi M, MacDonald H, Chen MH, Ezer LK, Straus SE. Effectiveness of quality improvement strategies for coordination of care to reduce use of health care services: a systematic review and meta-analysis. CMAJ. 2014 Oct 21;186(15):E568-78

20. Tricco AC, Soobiah C, Ho JM, Berliner S, Ng CH, Hamid JS, Qi Y, Perrier L, Hemmelgarn B, Majumdar SR, Straus SE. Comparative safety and effectiveness of cognitive enhancers for Alzheimer's dementia: A systematic review and network meta-analysis. Submitted to Annals of Internal Medicine in 2015.

21. Veroniki AA, Vasiliadis HS, Higgins JP, Salanti G. Evaluation of inconsistency in networks of interventions. Int J Epidemiol 2013;42, 332-345.

22. Veroniki AA, Mavridis D, Higgins JP, Salanti G. Statistical evaluation of inconsistency in a loop of evidence: a simulation study informed by empirical evidence. BMC Med. Res. Methodol. 2014; 14, 106.

23. Veroniki AA, Straus SE, Fyraridis A, Tricco AC. The rank-heat plot is a novel way to present the results from a network meta-analysis including multiple outcomes. J. Clin. Epidemiol. 2016; pii: S0895-4356(16)00153-0.

24. White IR, Barret JK, Jackson D, Higgins JPT. Consistency and inconsistency in multiple treatments meta-analysis: model estimation using multivariate meta-regression. Research Synthesis Methods 2012;3(2):111-25

References (2)


[email protected]@[email protected]

Download - Ispor workshop 08 april2016

Top Related