The Second David Sackett Symposium

THE RANDOMIZED TRIAL AS ROYALTY

John Concato, MD, MS, MPH

Yale University School of Medicine

VA Clinical Epidemiology Research Center

27 September 2013

 John Concato has no potential for conflict of interest

with this presentation

 

DISCLOSURE

OBJECTIVES

• Understand specific strengths and limitations of observational studies and randomized trials • Appreciate need for methodological and clinical judgment when evaluating patient-oriented research

DEFINITIONS OF ‘ROYAL’ Oxford Dictionary: • people of royal blood or status; member of a royal family • the status or power of a king or queen • the most successful, famous, or highly regarded members of a particular group

RCT AS ‘KING’ OF STUDY DESIGN Grades of evidence for study design:

I: At least one properly randomized, controlled trial (RCT)

II-1: Well-designed trials without randomization

II-2: Well-designed cohort or case-control analytic studies, preferably multi-site

II-3: Multiple time series with or without the intervention, or dramatic results in uncontrolled experiments (e.g., penicillin or insulin)

III: Opinions of respected authorities, based on clinical experience; descriptive studies, case reports, expert committee reports

(US Preventive Services Task Force 1996)

STUDY DESIGN IN BROADER CONTEXT

Validity • study architecture: randomize, adjust for confounding, etc. • source of data: primary, secondary, pooled, etc. Note: observational studies don’t always use “2o data” Generalizability • intended spectrum based on inclusion/exclusion criteria • potential impact of participation itself (e.g., Hawthorne effect) Note: observational studies have inherent advantage

EMPHASIS RE: STUDY DESIGN

Architecture Strengths Limitations

Randomized trials a a

Observational studies a a

LIMITATIONS OF RCTs Randomized trials on same topic often contradictory • e.g., Horwitz Am J Med 1987;82:498 • to be expected, but discordance is often overlooked Meta-analyses and large/simple RCTs often don’t agree • e.g., LeLorier New Engl J Med 1997;337:536 • under-appreciated; attributed partly to heterogeneity RCTs can have restricted generalizability • e.g., Coca JAMA 2006;296:1377 • acknowledged, but implications are often minimized

RCT GENERALIZABILITY (cont’d) Coca, et al., JAMA 2006;296:1377 (cont’d) “[Cardiovascular RCTs] do not provide adequate information on the effect of interventions on patients with renal disease” Juurlink, et al., New Engl J Med 2004;351:543 “Publication of [spironolactone RCT] was associated with abrupt increases in hyperkalemia-associated morbidity and mortality.” Luce, et al., Ann Intern Med 2009;151:206 “many RCTs are ill suited to meet the evidentiary needs implicit in the [Inst of Med] definition of [comp effectiveness research]”

SPECTRUM OF OBSERVATIONAL STUDIES Indirect cause-and-effect evidence:

• ecological study (unit of observation is >1 patient/person) • case series (no comparison group)

• historical control trial (non-concurrent comparison group)

Direct cause-and-effect evidence: • observational cohort study (with concurrent comparison)

• case-control study (including “nested” within cohort)

STRENGTHS OF OBSERVATIONAL STUDIES Strategies can promote validity of observational methods • e.g., Horwitz Am J Med 1990;89:630 (including RCT-like restricted cohort, specific zero-time for intervention) ‘Non-randomized’ treatments are effective/safe • e.g., Tsimberidou J Clin Oncol 2009;27:6243 (including bleomycin, cisplatin, etoposide for testicular cancer) Observational studies and RCTs often agree head-to-head • e.g., Concato New Engl J Med 2000;342:1887 (“well- designed observational studies do not systematically overestimate [results when compared with RCTs]”)

OBSERV STUDIES & RCTs ‘AGREE’ (cont’d) McKee, et al., BMJ 1999;319:312 “…one method does not give a consistently greater effect”

Benson and Hartz, New Engl J Med 2000;342:1887 “[Observational results are] neither consistently larger than nor qualitatively different from those obtained in RCTs”

Ligthelm, et al., Clin Ther 2007;29:1284 “…observational studies can be conducted using the same

exacting and rigorous standards as are used for RCTs”

PREDICTION RE: RANDOMIZED TRIALS

Alvan R. Feinstein

PROBLEM OF CONFOUNDING

• Confounding factors threaten validity of observ. study, whereas randomization tends to balance such factors

Several available options:

• conduct only randomized, controlled trials (per EBM)

• use methods that “substitute” for randomization (e.g., propensity scores, instrumental variables)

• use traditional observational methods rigorously

Current emphasis is problematic:

• expectation of requiring RCT or complex model

• observational studies held in low regard

HOW TO DEAL WITH CONFOUNDING

VARIABLE IMPACT OF CONFOUNDING

Status of confounding factors → implications for study:

• known, not accounted for → major methodological weakness

• known & accounted for → sound research practice

• unknown → clinicians would not treat preferentially, reducing potential for bias

Note: extent of problem varies (as with limitations of RCTs)

EVIDENCE-BASED MEDICINE (EBM) APPROACH

Validity: whether results are “true” for patients enrolled in the trial or observational study Generalizability: whether the results “generalize” to a broader population of patients with condition ► Validity is considered to be of primary importance; generalizability is considered subsequently

‘BEYOND RANDOMISED VS. OBSERVATIONAL STUDIES’ Accuracy: whether results are “true” for patients who would receive exposure, intervention, etc.;

► Consider validity and generalizability together (Concato & Horwitz, Lancet 2004;363:1660)

MEDICINE-BASED EVIDENCE (MBE)

In a medicine-based evidence approach, the primary emphasis is on clinically relevant issues of who and where were the patients, what and why were the treatments, & when and how were the outcomes measured—as well as an assessment of validity and generalizability considered together and denoted as accuracy.

(Concato JAMA 2012; 307:1641)

EXAMPLE #1: HORMONE REPLACEMENT THERAPY Research question: Does hormone replacement therapy (HRT) for post-menopausal women improve health outcomes? Observational studies: showed benefit of HRT regarding cardiovascular outcomes RCTs: showed “early” cardiovascular harm from HRT Conventional wisdom: RCTs are correct, observational studies are flawed; problem is inherent to “type” of research design

Relative risks or odds ratios from:

Outcome RCTs Observational

Colorectal ca 0.63 (0.43-0.92) 0.66 (0.59-0.74)

Hip fracture 0.66 (0.45-0.98) 0.75 (0.68-0.84)

Stroke 1.41 (1.07-1.85) 1.45 (1.10-1.92)

Pulm emb 2.13 (1.39-3.25) 2.1 (1.2-3.8)

(Grodstein New Engl J Med 2003;348:645)

EXAMPLE #1: RCT/OBS STUDIES FOR HRT

Relative risks or odds ratios from:

Outcome RCTs Observational

Colorectal ca 0.63 (0.43-0.92) 0.66 (0.59-0.74)

Hip fracture 0.66 (0.45-0.98) 0.75 (0.68-0.84)

Stroke 1.41 (1.07-1.85) 1.45 (1.10-1.92)

Pulm emb 2.13 (1.39-3.25) 2.1 (1.2-3.8)

Coronary dz 1.29 (1.02-1.63) 0.61 (0.45-0.82)

(Grodstein New Engl J Med 2003;348:645)

EXAMPLE #1: RCT/OBS STUDIES FOR HRT

EXAMPLE #1: MEDICINE-BASED EVIDENCE

Hormone replacement & coronary disease:

Patients:

Exposure:

Outcome:

• time since menopause • socioeconomic status

• dose of drug(s) • duration of treatment • timing of treatment

• duration of follow-up • capture of “early events”? • equal detection of MIs?

Not adjusted for socioeconomic status:

Adjusted for socioeconomic status:

(Von Elm BMJ 2004;329:869)

HRT & CV OUTCOMES: OBSERVATIONAL STUDIES

For randomised trials, [the start of HRT] is the natural analysis because therapy starts at randomisation…the first years of hormone replacement by combined oestrogen-progestin did increase coronary heart disease, which then waned.

Most current users [in observational studies] were past the window wherein coronary heart disease risk was increased… when data from the observational part of the [WHI trial] were re-analysed according to time since start of therapy, the same pattern emerged of an initial increase in risk, followed by a decrease.

(Vandenbroucke Lancet 2009;373:1233)

TIMING OF HRT VS. ONSET OF MENOPAUSE

Laine Ann Intern Med 2002;137:290 “Thus, rather than HRT keeping women healthy, healthy women were taking HRT.” Vandenbroucke Lancet 2009;373:1233 “Thus nothing was intrinsically wrong with the observational data; what went wrong was an analysis that had not taken into account that the effect of HRT might be different over time…Neither design [RCT nor observational] help superior truth.”

EXAMPLE #1: IMPLICATIONS

EXAMPLE #2: SCREENING FOR PROSTATE CANCER Observational studies: did not confirm benefit of screening with prostate-specific antigen (PSA), but were dismissed as inferior evidence RCTs: Two major RCTs were launched 20 years ago Superficial assessment: await “pronouncement” from RCTs, because observational studies are flawed Recent history: results from the trials eagerly anticipated as of 2009, yet RCTs on same topic often don’t agree

EXAMPLE #2: ‘RCT EXCLUSIVITY’ United States Preventive Services Task Force (USPSTF) 2008 recommendations on screening with PSA: “Although the 2002 USPSTF review considered case-control studies and ecological data related to this key question, we excluded these study types from this part of the evidence update to avoid potential sources of confounding that are inherent in nonrandomized studies.”

(USPSTF Ann Intern Med 2008;149:192)

Perspective when writing: ongoing RCTs may not be definitive. Statement (in Discussion): “we should recognize that the [King’s pronouncement] (i.e., results of the RCTs) may not provide a definite answer to the question of whether screening with PSA decreases mortality.

(Cancer J 2009;15:7-12)

[King]

EXAMPLE #2: RCTs & OBSERVATIONAL STUDIES

PSA screening and prostate-cancer mortality:

• RCT (Schroder 2009); HR = 0.80 (0.65-0.98)

• RCT (Andriole 2009); HR = 1.13 (0.75-1.70)

• Case-control (Weinmann 2004); OR = 0.70 (0.46-1.1)

• Case-control (Concato 2006); OR = 1.13 (0.63-2.06)

Note: results of the two RCTs are “contradictory”; RCT & observational study results overlap

EXAMPLE #2: ‘ROYAL DECREE’

Amer Urol Assoc (AUA) 2009 Best-Practice update:

Because there is now evidence from a RCT regarding a mortality

decrease associated with PSA screening, the AUA is recommending

PSA screening, as proposed in this document, for well-informed men

who wish to pursue early diagnosis.

EXAMPLE #2: MEDICINE-BASED EVIDENCE

Issues re: screening for prostate cancer & mortality:

Patients:

Exposure:

Outcome:

• age; comorbidity • indolent vs. aggressive cancer

• PSA alone (vs. velocity, density, etc.) • with or without digital rectal exam? • compared to “usual care”?

• overall vs. cause-specific mortality • consider morbidity? • duration of follow-up

SPECTRUM OF VIEWS ON RESEARCH DESIGN ‘Randomized trials or observational tribulations?’ “Only randomized treatment assignment can provide a reliably unbiased estimate of treatment effects”…perhaps we have not tried hard enough to convert the skeptics.” (Pocock, New Engl J Med 2000;342:1907) ‘Why there’s no cause to randomize’ “None of [the arguments in favor of randomization] supplies any practical reason for thinking of randomization as having unique epistemic power”…It is in fact very difficult to see any cogent reason for thinking as highly of RCTs as the medical community does” (Worrall, Brit J Phil Sci 2007;58:451)

CRITICISM OF ‘ROYAL STATUS’ GIVEN TO RCTs Decisions should be based on best available evidence, to avoid:

• RCT-myopia → action only justified by RCTs

• Evidence-based paralysis → no action without RCTs

(Ziemer Arch Intern Med 2006;166:1672)

…“remain skeptical until a randomized trial is done.”

Do We Really Know What Makes Us Healthy?

Sep 16, 2007

DISCUSSION IN ‘LAY’ PRESS

RESPONSE AS LETTER-TO-EDITOR “The take-home message for readers should be not to expect certainty from any individual study, observational or randomized. Progress in science is iterative and incremental—in the long run, we get closer to ‘truth’ as overall evidence accumulates on a given topic. Putting what we know into practice would prevent more disease than [bowing before the throne] of randomized trials.” John Concato, M.D., The New York Times, Sep. 30, 2007

SUMMARY ‘Science as experiment; science as observation’ “The importance lies not in arguing about which methodology is better than the other, but what can be learned about disease activity and therapy from each type of study.” (Chakravarty, Nat Clin Pract Rheumatol 2006;6:286) ‘On the evidence for decisions [re:] therapeutic interventions’ “Decision makers need to assess and appraise all the available evidence irrespective of whether it has been derived from randomised controlled trials or observational studies, and the strengths and weaknesses of each need to be understood. [There is] no shame in accepting that judgments are required.” (Rawlins, Lancet 2008;372:2152)

TAKE-HOME MESSAGES

• Carefully-designed observational studies can address confounding adequately; limitations of RCTs are often overlooked; results from both designs are informative • Patient-oriented research needs more Medicine-Based Evidence, i.e., more methodological and clinical judgment