approaches to synthesis of heterogeneous evidence
DESCRIPTION
The following slides were presented at a meeting of potential editors and methods advisors for the proposed Cochrane review group in February 2008. The slides were designed to promote discussion rather than represent the views and directions of this group. Approaches to Synthesis of - PowerPoint PPT PresentationTRANSCRIPT
The following slides were presented at a meeting of potential editors and
methods advisors for the proposed Cochrane review group in February 2008. The slides were designed to promote discussion rather than
represent the views and directions of this group.
Approaches to Synthesis of
Heterogeneous Evidence
Randy Elder, PhD, MEdScientific Director for Systematic ReviewsGuide to Community Preventive Services
National Center for Health MarketingCenters for Disease Control and Prevention (CDC)
It is the mark of an educated man to look for precision in each class of things just so far as the nature of the subject admits.
Aristotle, c. 350 BC
Goals
Review of tools for addressing heterogeneity Discuss utility of descriptive statistical
approaches when MA is not feasible Raise conceptual and practical issues
related to heterogeneity Substantive sources of variance Methodological sources of variance
Methods for SynthesizingHeterogeneous Data
Inferential Statistical Approaches: Meta-analysis
Requires sufficient homogeneity for estimate of central tendency to be useful Likely to be relatively uncommon in HPPH reviews Less complex interventions are most likely candidates
E.g., safety belt laws Subgroup analysis can be used to account for some heterogeneity
Inferential Statistical Approaches: Meta-regression
Able to account for sources of heterogeneity in more complex interventions
Partially addresses colinearity issues that bedevil univariate subgroup analyses
Potentially useful for selected interventions with large evidence base
E.g., some school-based interventions Pitfalls include
Poor reporting/measurement of effect modifiers Underpowered analyses of effect modification Potential for false positives with multiple comparisons Susceptibility to ecological fallacy
Descriptive Approaches:Narrative Synthesis
Likely to be the most common approach for complex HPPH reviews
ESRS guidance on narrative synthesis is a valuable tool for editors and authors
Pro Can be applied to any data Often only option given heterogeneous interventions,
populations, and outcomes Allows thoughtful synthesis of small bodies of evidence
Con Challenging for larger bodies of evidence
• Tabular and graphical techniques can be helpful additions More prone to biased interpretation
• E.g., temptation to engage in vote-counting More difficult to evaluate effect modification
Use of Descriptive Statisticswith Narrative Synthesis
Descriptive summary statistics can provide a useful supplement to tabular and graphical methods
Facilitate simple, concise text summaries of distribution of study results What is the central tendency? (e.g., median) How much variation in results can be expected?
(e.g., range, interquartile interval)
Conceptual and Practical Issuesin Addressing Heterogeneity
Accounting for Heterogeneity: Effect Modification and Subgroup Analysis
HPPH and ECRS guidance on subgroup analysis* Do it (within reason and with theoretical justification) Report results Interpret them cautiously
This has the practical benefits of providing end-users with information they need
Decisions re: when, where, how, and with whom to implement interventions need to be made
Any information is preferable to none An a priori assumption of homogeneity is a far less
conservative approach Analyses done from a hypothesis-testing perspective face issues
of confounding and tend to be underpowered (substantial risk of Type II error)
Not doing analyses effectively guarantees Type II error (of uncertain magnitude)
*As I understand it
Incorporating Non-randomized Studies: Cochrane NRS Guidance
Cochrane NRS Group guidance Don’t use NRSs to supplement RCT data on
effectiveness Few RCTs provide imprecise, unbiased estimate Including NRSs increases precision, but at the
unacceptable cost of accuracy
This position has some merit, but ignores some important characteristics of HPPH interventions and reviews
Sources of Variance in HPPH Reviews
Meta-synthesis of psychological, behavioral and educational interventions (Wilson & Lipsey , 2001) Reasonable generalizability to HP interventions
Substantive variance (25% of total) Methodological Variance (21% of total)
Study design (4%) Operationalization of outcome (8%)
EPPI meta-synthesis on policy studies will be useful
Rationale for Including NRSs in Complex Population-level
Interventions (1)
Bias needs to be considered at two levels The study (internal validity) The systematic review (generalizability)
Distinction between systematic and non-systematic biases is also important
Rationale for Including NRSs in Complex Population-level
Interventions (2) Non-systematic sources of bias appear to contribute
variance within an acceptable range of “noise” Selection bias is the major systematic threat in NRSs
Self-selection Researcher-selection
Threats of self-selection bias are not identical across interventions
Most likely with individual-level interventions Less likely with population-level interventions
• Complicated causal pathway to implementation reduces risk of confounding
• Availability of data on comparability of groups pre-intervention
Registry of PH interventions would help address researcher-driven selection biases
Rationale for Including NRSs in Complex Population-level
Interventions (3) Limiting reviews to RCTs may introduce more bias
than it prevents Bias=systematic error in the population effect estimate
RCTs may provide biased effect estimates for complex interventions due to
ITT analysis (difference between the effectiveness of the intervention and of randomization to the intervention condition)
Resources Population selection Adherence to protocol
Benefits of including NRSs Power
Rationale for Including NRSs in Complex Population-level
Interventions (4)
Generalizability Power
Increases potential to provide useful guidance on “lumped” effects
Dramatically increases potential to provide useful guidance on effect modification issues (but only when there is no “firewall” between RCTs and other studies)
A Judgment Call
Is study design such a unique and important source of variance that it should be singled out from among all other potential sources of bias and effect modification?
Or do the harms of treating study design as qualitatively different from all other potential modifiers of effect estimates outweigh the benefits?
If the Latter..
Guidance re: addressing “quantitative” differences in study quality should apply: Consider limiting review to studies above a
threshold “design quality”• Considering plausible systematic sources of variance
for given subject matter Use sensitivity analysis to evaluate robustness
of findings (giving up on the quest for precision) Avoid or cautiously apply “quality weighting” by
design alone
Beware of the “Outlier” Randomized Trial
Shatterproof glassware Students Against Drunk Driving Any multi-million dollar trial that can’t
feasibly be brought to scale
