Generalised Evidence Synthesis

Keith Abrams, Cosetta Minelli, Nicola Cooper &

Alex SuttonMedical Statistics Group

Department of Health Sciences,

University of Leicester, UK

CHEBS Seminar

‘Focusing on the Key Challenges’

Nov 7, 2003

Outline

• Why Generalised Evidence Synthesis?

• Bias in observational evidence

• Example: Hormone Replacement Therapy (HRT) & Breast Cancer

• Discussion

Why Generalised Evidence Synthesis?• RCT evidence ‘gold standard’ for assessing efficacy (internal

validity)

• Generalisability of RCT evidence may be difficult (external validity), e.g. CHD & women

• Paucity of RCT evidence, e.g. adverse events

• Difficult to conduct RCTs in some situations, e.g. policy changes

• RCTs have yet to be conducted, but health policy decisions have to be made

• Consider totality of evidence-base – (G)ES beyond MA of RCTs

Assessment of Bias in Observational Studies - 1

• Empirical evidence relating to potential extent of bias in observational evidence (Deeks et al. 2003)

• Primary studies: – Sacks et al. (1982) & Benson et al. (2000)

• Primary & Secondary studies (meta-analyses): – Britton et al. (1998) & MacLehose et al. (2000)

• Secondary studies (meta-analyses):– Kunz et al. (1998,2000), Concato et al. (2000) &

Ioannidis et al. (2001)

Assessment of Bias in Observational Studies - 2

• Using a random effects meta-epidemiology model (Sterne et al. 2002)

– Sacks et al. (1982) & Schultz et al. (1995) ~ 30%– Ioannidis et al. (2001) ~ 50%– MacLehose et al. (2000) ~ 100%

• Deeks et al. (2003) simulation study: comparison of RCTs and historical/concurrent observational studies – Empirical assessment of bias – results similar to

previous meta-epidemiological studies– Methods of case-mix adjustment, regression &

propensity scores fail to properly account for bias

Approaches to Evidence Synthesis

• Treat sources separately, possibly ignoring/downweighting some implicitly

• Bayesian approach & treat observational evidence as prior for RCTs & explicit consideration of bias:– Power Transform Prior– Bias Allowance Model

• Generalised Evidence Synthesis

Example – HRT

• HRT used for relief of menopausal symptoms

• Prevention of fractures, especially in women with osteoporosis & low bone mineral density

• BUT concerns have been raised over possible increased risk of Breast Cancer

Odds Ratio (log scale)

<

<

<

<

<

<

<

>

>

>

Viscoli 2001 5 / 337 5 / 327

<

<

<

<

<

<

<

>

>

>

Mosekilde 2001 2 / 502 5 / 504

<

<

<

<

<

<

<

>

>

>

Gallagher 2001 0 / 243 4 / 246 <

<

<

<

<

<

<

>

>

>

Herrington 2000 1 / 204 0 / 105

<

<

<

<

<

<

<

>

>

>

Angerer 2000 0 / 215 1 / 106

<

<

<

<

<

<

<

>

>

>

Alexandersen 2000 1 / 150 1 / 50

<

<

<

<

<

<

<

>

>

>

Komulainen 1999 2 / 232 1 / 232

<

<

<

<

<

<

<

>

>

>

Hulley (HERS) 1998 32 / 1380 25 / 1383

<

<

<

<

<

<

<

>

>

>

Steele 1997 1 / 37 0 / 37

<

<

<

<

<

<

<

>

>

>

Speroff 1996 6 / 1128 0 / 137

<

<

<

<

<

<

<

>

>

>

PEPI 1995 7 / 701 1 / 174

<

<

<

<

<

<

<

>

>

>

Munk-Jensen 1994 2 / 100 0 / 51

<

<

<

<

<

<

<

>

>

>

Maheux 1994 0 / 30 1 / 30

<

<

<

<

<

<

<

>

>

>

Aloia 1994 0 / 31 1 / 70

<

<

<

<

<

<

<

>

>

>

Lufkin 1992 1 / 39 1 / 36

<

<

<

<

<

<

<

>

>

>

Gallagher 1991 0 / 62 1 / 20

<

<

<

<

<

<

<

>

>

>

Genant 1990 1 / 116 1 / 40

<

<

<

<

<

<

<

>

>

>Christiansen 1980 2 / 56 1 / 259

<

<

<

<

<

<

<

>

>

>

Nachtigall 1979 0 / 84 4 / 84

RCT Year HRT Control

Pooled

0.05 0.1 1 5 10 20

HRT & Breast Cancer – RCT Evidence before July 2002

Source: Torgerson et al. (2002)

OR 0.97 95% CI 0.67 to 1.39

Odds Ratio (log scale)

Iowa Womens Health 1991 355 / 1338 178 / 702Netherlands Cohort 1988 30 / 125 306 / 1076Nurses Health 1986 618 / 2442 714 / 3084Schairer 1985 341 / 1418 370 / 1422Canadian NBSS 1985 205 / 954 243 / 976

Cohort

Pooled

Stanford 1989 117 / 134 149 / 161Yang/Gallagher 1989 132 / 148 269 / 2774 State Study 1988 604 / 720 1892 / 2297Long Island 1984 157 / 122 519 / 547Ewertz 1983 136 / 109 400 / 414Bain 1983 39 / 86 226 / 458Hislop 1981 86 / 84 275 / 282CASH 1981 437 / 542 335 / 420Brinton 1976 808 / 932 714 / 869

Case-Control (Population)

Pooled

Franceschi 1992 151 / 132 1265 / 1379Katsouyanni 1990 42 / 70 404 / 770La Vecchia 1987 119 / 64 1496 / 1386Vessey 1982 47 / 51 369 / 411Morabia 1974 80 / 144 104 / 178

Case-Control (Hospital)

Pooled

HRT No HRT

0.5 1 2

ALL Observational

RCTs

HRT & Breast Cancer – Observational Evidence*

Source: Lancet (1997)

* Adjusted for possible confounders

All Observational OR 1.18 95% CI 1.10 to 1.26

RCTs OR 0.97 95% CI 0.67 to 1.39

Use of Observational Evidence in Prior Distribution

Case-ControlCase-ControlQuasi RCTsQuasi RCTs

RCTsRCTs

CohortCohort

Prior

Synthesis

Empirical EvidenceBias

Empirical EvidenceBias

Following Ibrahim & Chen (2000)

• 0 1 is degree of downweighting = 0 total discounting = 1 accept at ‘face value’• Evaluate for a range of values of

Power Transform Prior

)()|()|()|( PObsLRCTsLDataP

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

0.50

0.75

1.00

1.25

1.50

Alpha

Odd

s R

atio

(lo

g sc

ale)

Power Transform Prior – Results 1

* is unbiased true effect in observational studies is bias associated with observational evidence 2 represents a priori beliefs regarding the possible

extent of the bias

),0(~ &

for prior form obs. i.e. |~

),(~ & )(~ : ObsMA

),(~ & ),(~ :RCTsMA

2*

22,

22

N

z

NvNz

NsNy

jjjj

iiii

Bias Allowance ModelFollowing Spiegelhalter et al. 2003

Bias Allowance Model - Results

Belief/Source Bias 2 OR 95% CrI P(OR>1)

‘Face Value’ 0% 0 1.14 1.07 to 1.20 1.00

Total Discounting % 0.87 0.30 to 1.60 0.31

Sacks & Schultz 30% 0.02 1.08 0.85 to 1.37 0.72

Ioannidis 50% 0.08 1.00 0.68 to 1.45 0.50

MacLehose 100% 0.24 0.94 0.56 to 1.49 0.40

HRT & Breast Cancer: Evidence – July 2002

HERS II (JAMA July 3) [Follow-up of HERS]– n = 2321 & 29 Breast Cancers – OR 1.08 (95% CI: 0.52 to 2.25)

• WHI (JAMA July 17) [Stopped early]– n= 16,608 & 290 Breast Cancers– OR 1.28 (95% CI: 1.01 to 1.62)

• HERS II & WHI– OR 1.26 (95% CI: 1.01 to 1.58)

• Revised Meta-Analysis of RCTs– WHI 68% weight – OR 1.20 (95% CI: 0.99 to 1.45)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

0.50

0.75

1.00

1.25

1.50

Alpha

Odd

s R

atio

(log

sca

le)

HERS II & WHI

Power Prior Posterior

30%

Power Transform Prior – Results

Generalised Evidence Synthesis

• Modelling RCT & observational (3 types) evidence directly;

– Hierarchical Models (Prevost et al, 2000;Sutton & Abrams, 2001)

– Confidence Profiling (Eddy et al, 1990)

• Overcomes whether RCTs should form likelihood & observational studies prior

Generalised Evidence Synthesis

Quasi RCTsQuasi RCTs Case-ControlCase-Control RoutineRoutineRCTsRCTs CohortCohort

Beliefs

Synthesis

Decision Model

Utilities Costs

Hierarchical Model

],[~ & ],[~ ],,[~

],[~

],[~

,,1 & ,,1 ],[~

22

2

2

2

j

j

jjij

jijijij

N

N

JjnisNy

HRT: Hierarchical Model - Results

Independent Hierarchical

OR 95% CrI OR 95% CrI

RCT 0.89 0.39 to 1.52 1.02 0.76 to 1.27

Cohort 0.98 0.84 to 1.12 1.01 0.89 to 1.13

CC-P 1.06 0.96 to 1.06 1.05 0.97 to 1.15

CC-H 1.23 0.94 to 1.55 1.12 0.93 to 1.36

Overall 1.05* 0.98 to 1.13 1.05 0.87 to 1.24

* Ignores study-type

Hierarchical Model - Extensions

• Inclusion of empirical assessment of (differential) bias with uncertainty, i.e. distribution

• Bias Constraint, e.g. HRT

CCHCCPCohRCT

Discussion – 1 • Direct vs Indirect use of non-RCT evidence

– Direct: intervention effect, e.g. RR– Indirect: other model parameters, e.g. correlation

between time points

• Allowing for bias/adjusting at study-level– IPD if aggregate patient-level covariates are

important, e.g. age, prognostic score– Quality – better instruments for non-RCTs &

sensitivity of results to instruments

Discussion – 2

• Subjective prior beliefs regarding relative credibility (bias or relevance) of sources of evidence– Elicitation

• Bayesian methods provide …– A flexible framework to consider inclusion of all

evidence, & …– which is explicit & transparent, BUT …– Require careful & critical application

ReferencesDeeks JJ et al. Evaluating non-randomised intervention studies. HTA 2003;7(27).

Eddy DM et al. A Bayesian method for synthesizing evidence. The Confidence Profile Method. IJTAHC 1990;6(1):31-55.

Ibrahim JG & Chen MH. Power prior distributions for regression models. Stat. Sci. 2000 15(1):46-60.

Prevost TC et al. Hierarchical models in generalised synthesis of evidence: an example based on studies of breast cancer. Stat Med 2000;19:3359-76.

Sterne JAC et al. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat. Med. 2002;21:1513-1524.

Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials & Health-care Evaluation. London: Wiley, 2003.

Sutton AJ & Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. SMMR 2001;10(4):277-303.