Cochrane Methodology Reviews

Lisa BeroSan Francisco Branch, US Cochrane Center

UCSF

CER Symposium, January 2012

Methodology Reviews

• A "methodology study" is a study of the methods used in randomized trials, other healthcare evaluations or systematic reviews.– Consent procedures– Recruitment methods– Association of allocation concealment with

estimates of treatment effect• A "methodology review" is a systematic

review of methodology studies

4

Study Designs for CER/ PCOR•head to head randomized trials,•cluster randomized trials, •adaptive designs, •practice / pragmatic trials,• PBE-CPI “practice based evidence for clinical practice improvement,” •natural experiments, •observational or cross-sectional studies of registries and databases including electronic medical records, •meta-analysis, •network meta-analysis, •modelling and simulation

• observational study analysis approaches employing so-called causal inference techniques, which can include instrumental variables, marginal structural models, propensity scores, among others.

• “NEW” design terms such as “observational randomization study”

Source: IOM-National Priorities Committee 2009

•A number of reviews comparing the effect sizes and biases in randomized and non-randomized studies have been conducted.

• Most compared randomized to non-randomized trials. • Most often limited the comparison with observational

studies to cohort studies, or the types of observational designs included were not specified.

• Most published between 1982 and 2003

•Compared to RCTs, observational designs have been found to overestimate treatment effects ; underestimate treatment effects; or show no difference.

“Best “ study design for CER?

Health care outcomes assessed with non-experimental designs compared with those assessed in randomized trials

Lisa BeroAndrew AnglemyerTara Horvath

San Francisco Branch of the US Cochrane CenterHIV / AIDS Cochrane Review GroupUCSF

Funding: Clinical and Translational Sciences Institute (CTSI), University of California, San Francisco (UCSF), USA

Protocol: in press, Cochrane Library

Objectives

• To assess the impact of study design--to include RCTs vs observational study designs, different types of observational studies, and/or choice of analytic techniques -- on the effect measures estimated in observational and randomized studies

• To explore methodological variables that might explain any differences identified

• To identify gaps in the existing research comparing study designs

Inclusion criteria

• Systematic or non-systematic reviews designed as methodological studies to compare study designs

• Clinical outcomes: efficacy or harms of alternative interventions to prevent or treat a clinical condition or improve the delivery of care

A priori subgroup analyses

• Comparisons of drug interventions• Clinical topic• Heterogeneity of included methodological

studies

Included studies – RCT vs. observationalPRELIMINARY DATA:

9 studies in meta-analysis

• Included 19 – 276 studies• Evaluated a mix of interventions

– Lower back pain– Digestive surgery– Various interventions

• One focused on drug –drug comparisons (Naudet)

• One focused on adverse events from (mostly) pharmacological treatements (Golder)

PRELIMINARY DATA:

Study or Subgroup1.1.1 RCT vs All Observational

Concato 2000Benson 2000Bhandari 2004Shikata 2006Furlan 2008Beynon 2008Mueller 2010Golder 2011Naudet 2011Subtotal (95% CI)

Heterogeneity: Tau² = 0.05; Chi² = 44.79, df = 8 (P < 0.00001); I² = 82%Test for overall effect: Z = 1.19 (P = 0.23)

1.1.2 RCT vs Cohort

Benson 2000Concato 2000Bhandari 2004Furlan 2008Golder 2011Subtotal (95% CI)

Heterogeneity: Tau² = 0.03; Chi² = 10.95, df = 4 (P = 0.03); I² = 63%Test for overall effect: Z = 0.30 (P = 0.76)

1.1.3 RCT vs Case Control

Concato 2000Golder 2011Subtotal (95% CI)

Heterogeneity: Tau² = 0.04; Chi² = 2.34, df = 1 (P = 0.13); I² = 57%Test for overall effect: Z = 0.20 (P = 0.84)

Test for subgroup differences: Chi² = 0.49, df = 2 (P = 0.78), I² = 0%

Weight

15.3%7.2%

11.1%12.9%4.3%

14.9%13.7%15.1%5.6%

100.0%

11.1%32.9%21.9%7.2%

26.9%100.0%

58.9%41.1%

100.0%

IV, Random, 95% CI

1.07 [0.95, 1.21]0.95 [0.58, 1.56]0.70 [0.52, 0.95]0.97 [0.77, 1.22]1.94 [0.93, 4.05]0.87 [0.75, 1.00]1.48 [1.22, 1.80]1.08 [0.95, 1.23]3.55 [1.94, 6.50]1.11 [0.93, 1.33]

1.52 [0.87, 2.64]1.04 [0.91, 1.18]0.70 [0.52, 0.95]1.94 [0.93, 4.05]1.02 [0.82, 1.27]1.03 [0.83, 1.29]

1.20 [0.94, 1.54]0.84 [0.57, 1.23]1.04 [0.73, 1.46]

Year

200020002004200620082008201020112011

20002000200420082011

20002011

Risk Ratio Risk RatioIV, Random, 95% CI

0.2 0.5 1 2 5Obs Reflect Greater Risk RCTs Reflect Greater Risk

PRELIMINARY DATA:

Preliminary Findings

• Differences in effect measures not associated with study design – explore other reasons

• Conduct subgroup analyses• Need methodological studies comparing trials

with other observational designs (not just cohorts, case control) and different analytic methods for observational data