american heart association - assessing the quality of ......american heart association overview •...
TRANSCRIPT
Assessing the Quality of
Observational studies in
ILCOR
Eddy Lang
Associate Professor
University of Calgary
Michael Sayre
Professor, Emergency Medicine
University of Washington
Peter Morley
Associate Professor
University of Melbourne
Russell Griffin
Science and Medicine Advisor
American Heart Association
Overview
• Observational studies (non-RCTs) in
evidence hierarchy
• Assessing risk of bias in observational
studies
• GRADE risk of bias tool
• Pooling data from observational studies
4
Quality assessment criteria Lower if…
Quality of
evidence
High
Moderate
Low
Very low
Study limitations
(design and execution)
Inconsistency
Indirectness
Imprecision
Publication bias
Higher if…
Study
design
RCTs
Observational
studies
Large effect (e.g., RR 0.5)
Very large effect (e.g., RR 0.2)
Evidence of dose-response
gradient
All plausible confounding…
…would reduce a
demonstrated effect
…would suggest a spurious
effect when results show
no effect
It’s not fair!
• Observational studies may be best
available
• RCTs not feasible ? ethical
• Large observational studies > RCTs
6
“Categories” of quality (1)
Further research is very unlikely to change our
confidence in the estimate of effect High
Low
Further research is very likely to have an important
impact on our confidence in the estimate of effect and is
likely to change the estimate
Moderate
Further research is likely to have an important impact on
our confidence in the estimate of effect and may change
the estimate
Very low Any estimate of effect is very uncertain
7
Conceptualizing quality (2)
We are very confident that the true effect lies close to
that of the estimate of the effect. High
Low
Our confidence in the effect is limited: The true effect
may be substantially different from the estimate of the
effect.
Moderate
We are moderately confident in the estimate of effect:
The true effect is likely to be close to the estimate of
effect , but possibility to be substantially different.
Very low
We have very little confidence in the effect estimate:
The true effect is likely to be substantially different from
the estimate of effect.
Systematic review
Guideline development
P I C O
Outcome
Outcome
Outcome
Outcome
Critical
Important
Critical
Not
Summary of findings & estimate of effect for each outcome
Rate overall quality of evidence across outcomes based on
lowest quality of critical outcomes
RCT start high, obs. data start low
1. Risk of bias 2. Inconsistency 3. Indirectness 4. Imprecision 5. Publication
bias
Gra
de
do
wn
G
rad
e u
p
1. Large effect 2. Dose
response 3. Confounders
Very low
Low
Moderate
High
Formulate recommendations: • For or against (direction) • Strong or weak (strength)
By considering: Quality of evidence Balance benefits/harms Values and preferences
Revise if necessary by considering: Resource use (cost)
• “We recommend using…” • “We suggest using…” • “We recommend against using…” • “We suggest against using…”
Newcastle-Ottawa Quality Assessment Scale:
Cohort Studies
• Selection (4)
• Comparability (1)
• Outcome (3)
– A study can be awarded a maximum of one star for each
numbered item within the Selection and outcome categories.
A maximum of two stars can be given for Comparability
Selection 1. Representativeness of the exposed cohort
a) truly representative of the average ___________ (describe) in the community
b) somewhat representative of the average ___________ in the community
c) selected group of users eg nurses, volunteers
d) no description of the derivation of the cohort
2. Selection of the non exposed cohort
a) drawn from the same community as the exposed cohort
b) drawn from a different source
c) no description of the derivation of the non exposed cohort
3. Ascertainment of exposure
a) secure record (eg surgical records)
b) structured interview
c) written self report
d) no description
4. Demonstration that outcome of interest was not present at start of study
a) yes
b) no
In the case of mortality
studies, outcome of
interest is still the presence
of a disease/ incident,
rather than death; that is a
statement of no history of
disease or incident earns a
star
Comparability
1. Comparability of cohorts on the basis of the
design or analysis
a) study controls for ___________ (select
the most important factor)
b) study controls for any additional factor (This
criteria could be modified to indicate specific
control for a second important factor.)
Outcome
1. Assessment of outcome
a) independent blind assessment
b) record linkage
c) self report
d) no description
2. Was follow up long enough for outcomes to occur
a) yes (select an adequate follow up period for outcome of interest)
b) no
3. Adequacy of follow up of cohorts
a) complete follow up - all subjects accounted for
b) subjects lost to follow up unlikely to introduce bias - small number
lost - > ___ % (select an adequate %) follow up, or description of those
lost)
c) follow up rate < ___% (select an adequate %) and no description of
those lost
d) no statement
Adjusted Effect Estimates for Coronary Heart Disease
(All Events) (HRT: Estrogen Ever Use)
Cohort Studies
Selection Comparability Outcome
Lauritzen / 83
Wilson / 85
Petitti / 87
Henderson / 91
Lafferty / 94
Folsom / 95
Ettinger / 96
Wolf / 96
0.01 0.1 1 10
Key Questions for Body of Obs
Studies
• Do I preserve the LoE at low for this
outcome?
• Do I downgrade to very low?
• Are upgrade criteria present?
RoB in Observational Studies
• 1. Failure to develop and apply
appropriate eligibility criteria (inclusion of
control population)
• Under- or overmatching in case-control
studies
• Selection of exposed and unexposed in
cohort studies from different populations
RoB in Observational
• 2. Flawed measurement of both exposure
and outcome
• Differences in measurement of exposure
(e.g., recall bias in case-control studies)
• Differential surveillance for outcome in
exposed and unexposed in cohort studies
RoB in observational
• 3. Failure to adequately control
confounding
• Failure of accurate measurement of all
known prognostic factors
• Failure to match for prognostic factors
and/or lack of adjustment instatistical
analysis
• 4. Incomplete follow-up
Reasons to upgrade?
• Large effect size
– OR > 2.0 or < 0.5 = increase to moderate
– OR > 4.0 or < 0.2 = increase to high
• Dose – response ? Time factor
• All possible confounding supports
conclusions
