an introduction to systematic reviews...2/20/2012 3 outline 1. introduction: need for reviews 2....
Post on 24-Feb-2021
5 Views
Preview:
TRANSCRIPT
2/20/2012
1
An Introduction to
Systematic Reviews
Susan L. Norris, MD, MPH, MSAssociate Professor
Oregon Health & Science University Portland, OR
norriss@ohsu.edu
Biography, SL Norris• MD, MS, University of Alberta• MPH, University of Washington• Board Certified: general surgery (Canada), family medicine (US)
• Clinical practice: Group Health Cooperative, Washington state, 1990‐1999
2
g ,• CDC: directed systematic review group with focus on diabetes, 1999‐2004
• Oregon Health & Science University (current): – clinical practice guideline development– systematic review methodology (nonrandomized studies, reporting bias), comparative effectiveness reviews
– effects of physician‐industry relationships on practice guidelines and systematic reviews
Disclosures• Financial: none
• Intellectual
– Member, GRADE Working Group
– Methods work: nonrandomized studies, sources of bias in reviews
3
reviews
– Impact of physician‐industry relationships on primary research and evidence synthesis
• Professional
– Investigator, Evidence‐based Practice Center (AHRQ)
– Funders: CDC, NIH, AHRQ, American College of Chest Physicians, American Urological Association
2/20/2012
2
Ben-Adhem picked up a stone from
beside the road. “It had written on it,
“Turn me over and read.’ So he
4
picked it up and looked at the other
side.
And there was written, “Why do
you seek more knowledge when
you pay no heed to what you know
5
y p y y
already?’”Shah (1968)
Learning Objectives
1. To define the main characteristics of systematic reviews and to differentiate them from narrative reviews.
2. To outline the steps involved in producing a systematic review
6
systematic review.3. To understand potential sources of bias in reviews.4. To outline five characteristics of a high‐quality
systematic review and to apply those to a review published in the peer reviewed literature.
5. To learn how to find systematic reviews in Medline or in repositories of systematic reviews.
2/20/2012
3
Outline1. Introduction: need for reviews
2. Typology and terminology‐ Steps in a systematic review
3. Formulating an answerable question
4. Searching for evidence
5 Ri k f bi i i di
7
5. Risk of bias in primary studies
6. Evidence synthesis‐ Qualitative
‐ Quantitative (meta‐analysis)
7. Assessing strength of evidence across studies‐ GRADE
8. Sources of high quality reviews and resources for performreviews
Part 1. Introduction
8
Introduction
Knowing What Works in Health Care:IOM Report 2008
“ …decisions about the care of individual
patients should be based on the
9
conscientious, explicit, and judicious use
of the current best evidence on the
effectiveness of clinical services.”
2/20/2012
4
500,000.00
600,000.00
700,000.00
800,000.00
900,000.00
Records added to MEDLINE each year
10
0.00
100,000.00
200,000.00
300,000.00
400,000.00
,
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
MEDLINE
11PLoS Med 7(9): e1000326. doi:10.1371/journal.pmed.1000326
Part 2. Systematic Reviews:
12
Systematic Reviews: Typology and Terminology
2/20/2012
5
Typology
• Integrative publication
• Systematic review
• Narrative review
• Qualitative synthesis
• Quantitative synthesis
• Meta‐analysis
• Practice guidelines
13
• Nonsystematic review
• Review of comparative effectiveness
• Practice guidelines
• Economic evaluation
• Decision analysis
Integrative Publications
Combine the results of multiple studies
• Reviews
• Narrative, quantitative
• Systematic, nonsystematic
14
• Practice guidelines
• Economic evaluations
• Clinical decision analysis
Types of Reviews
• Nonsystematic (narrative or traditional) – Widespread (eg, NEJM)
– Typically, 1 or 2 experts write the review based on literature the authors are familiar with
Conclusions related more to the experiences and
15
– Conclusions related more to the experiences and views of the authors than to the evidence
– Different authors might reach different conclusions on the same topic or even on the same evidence
– Small but important effects may be missed
• Systematic
2/20/2012
6
Systematic Review
A concise summary of the best available
evidence that addresses a sharply
defined clinical question (Mulrow 1998)
16
defined clinical question (Mulrow 1998)
• Qualitative synthesis = narrative summary
• Quantitative synthesis = meta‐analysis
Systematic Review
“In its ideal form, is a review that includes an explicit and detailed
17
description of how it was conducted so that any interested reader would be able to replicate it” (Jadad 1998)
• Define the extent and limits of the evidence
• Facilitate decision making– Do tell you what to do
– Other factors to consider • Equity, judgment, patient preferences, resource constraints
Cl if h t ti / d ti i b d
Systematic Reviews: Aims
18
• Clarify what practice/recommendation is based on evidence and what is based on other opinion
• Help to understand inconsistencies
• Help to understand potential applicability to specific situations or settings
• Identify research gaps
2/20/2012
7
Attributes of Systematic Reviews
T Transparent
R Reproducible
b d
19
U Unbiased
E Exhaustive
E Explicit methods
Reviews of Comparative Effectiveness
Comparative effectiveness
= the comparison of effective interventions among patients in typical patient care settings
20
among patients in typical patient care settings, with decisions tailored to individual patient needs
Sox and Greenfield, 2009
Meta‐Analysis
A systematic review that employs statistical methods to combine and summarize the results of several studies (Cook 1995)
21
Other terms:• Quantitative synthesis
• Pooling, pooled analysis
2/20/2012
8
When is a Systematic Review Needed?1. When there is a substantive clinical or public health
question
2. Uncertainty exists• Effects of an intervention, specific population or setting
• Balance of benefits and harms
• Small effects in individual studies
22
• Small effects in individual studies
• Need to explain inconsistency in results across studies
3. Need to define knowledge gaps, develop a research agenda
4. Several primary studies exist– Often with disparate findings
– There may be situations where there is a need to establish the lack of evidence
Steps in a Systematic Review
• Develop the review question
• Develop inclusion/exclusion criteria
• Search for literature
23
• Quality assess individual studies
• Data abstraction and analysis
• Synthesis of findings
• Grading the strength of evidence
Part 3. Formulating an Answerable
24
Systematic Review Question
2/20/2012
9
Formulating an Answerable Question
• Most important step in the review• Depends on perspective• Results have impact on patient care• Not dependent on available data
25
• Considerations:– Common conditions– Areas of uncertainty– Resource-intensive– New treatments– Heterogeneity of effects
26
Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials
GCS Smith, J PellBMJ 2003
27
2/20/2012
10
Levels of questions
• Can it work?
• Does it work?
• When/in whom does it work?• When/in whom does it work?
• Is it worth it?
• What research is needed?
Brian Haynes, EBM
What is the Purpose of Different Questions?
• Can it work?– Useful for: Coverage decisions
• How well does it work overall? – Useful for: Quality measurement and improvement
• When/in whom does it work?– Useful for: Patient/clinician decisions; health systems
• Is it worth it? – Useful for: Informing patient choice
• What kind of research is needed?– Researchers, funding agencies
D Atkins
What is Needed to Answer Those Questions?
• Can it work?– Studies of high internal validity + quantitative synthesis
• How well does it work overall? – Studies with external & internal validity
30
y
• When/in whom does it work?– External validity & internal validity
• Is it worth it (benefits vs. harms)? – Quantitative synthesis + decision models
• What kind of research would be most useful?– Qualitative synthesis, broad inclusion criteria
D Atkins
2/20/2012
11
Formulating a Review Question
P Person, population
I Intervention, exposure
C Comparator
O Outcome
(T) Timing
(S) Setting, study design
Example: The Well-Formulated Question
Does anticoagulation therapy improve outcomes in early treatment of patients with ischemic stroke?
Intervention Outcome Population Setting
Condition of Interest
32
Heparin
Dicumarol
CY 222sc
Phenindione
Kabi 2165sc
Recurrent Stroke
Extra/Intracranealhaemorrhage
Pulmonary Embolism
Venous Thrombosis
Death
>18 yrs, Hospitalized
≤48h from onset
Abrupt impairment of brain function caused by a variety of pathologic changes involving cranial blood vessels
���������� ������
33
2/20/2012
12
Setting inclusion and Exclusion CriteriaExclusion Criteria
What Evidence is Admissible?
• Consider topic by topic
• One size does not fit all
• Importance of protocol
• Educated decisions a priori
• Be explicit
• Document and defend changes
• Starts with a clear question
What Evidence is Admissible?
P Person, populationI Intervention, exposureC ComparatorO OutcomeT TimingS Setting, study design
2/20/2012
13
Inclusion Criteria for a Inclusion Criteria for a Drug Class Review (TZDs)Drug Class Review (TZDs)
Populations: Adults with type 2 diabetes:
ADA or WHO definitionAdults with prediabetes:
Defined as …
Adults with metabolic syndrome:ATP definition..
↑ TG level↓ HDL‐cholesterol↑ BP↑ FPG
Part 4.Searching the Literature
38
Image from cartoonbank.com removed.
Image description: Picture of sheriffs in forest with 3 bl dh d d D i l t th
39
bloodhound dogs; Dogs are using a laptop on the ground. Caption reads: First, they do an online search.
2/20/2012
14
Literature Searches for a Systematic Review
• Purpose:– To locate all relevant studies
– Comprehensive, unbiased collection of studies:• Failure to search and identify all relevant sources can lead to a biased reviewa biased review
• Challenges– Locating literature indexed in electronic databases
– Locating literature not‐indexed
– Identifying, locating unpublished studies
– Balance resource allocation and return
Methods of Searching for Relevant Studies
• Electronic Databases– Medline, Cochrane, PsychINFO, Embase, etc.
• Trial registries
• Hand searching
• Grey literature searches
• Experts
• Reference lists of reviews and included studies
Rationale for Comprehensive Searching
• Limitations of the search methods
– Only a small proportion of citations that are identified by comprehensive electronic searching (eg, MEDLINE) are relevant
– The typical MEDLINE search retrieves 50%‐ 80% of relevant studies
– Ability to search more precisely depends on database
2/20/2012
15
Rationale for Comprehensive Searching
• Non‐English language articles – not easily accessible on MEDLINE
– Non‐English language articles are of similar quality to English language articles and results areto English language articles and results are negative more often
• Publication Bias– 25‐50% of studies are never published, even as abstracts
– Studies with negative findings less likely to be published
Rationale for Comprehensive Searching
Failure to search and identify all relevantFailure to search and identify all relevant sources can lead to a biased review
Part 5.Assessing the Risk of Bias of
45
Individual Studies
2/20/2012
16
Bias
“ a systematic error, or deviation from the
truth, in results or inferences”
46
Cochrane Manual, 9/08
Quality
“extent to which the authors conducted their
h t th hi h t ibl t d d
47
research to the highest possible standards
Cochrane Manual, 9/08
Internal validity = The extent to which the results of a study can
be reliably attributed to the intervention under evaluation
48
vs. external validity = applicability of results to other populations and settings
vs. precision = measure of likelihood of random errors
2/20/2012
17
Why Assessment of Risk of Bias is Important
– Assess and minimize bias in the review
– Guide interpretation of findings
49
Hierarchy of Evidence
STUDY DESIGN
Randomized Controlled Trials
↓ BIAS
Cohort Studies and Case Control Studies
Case Reports and Case Series
Expert opinion
↑ BIAS
51
2/20/2012
18
Classification Scheme for Bias
• Selection bias
• Performance bias
• Attrition bias
52
• Attrition bias
• Detection bias
• Reporting bias
53
54
2/20/2012
19
Risk of Bias Tool
• For assessing RCTs only
• Developed by international group (largely Cochrane based)
D ib d i d t il i C h l
55
• Described in detail in Cochrane manual (9/08) www.cochrane.org
Components of RoB Tool
• Sequence generation
• Allocation concealment
• Blinding (by outcome)
56
• Incomplete outcome data (by outcome)
• Selective outcome reporting
• Other
Domain Description (example) Review authors’ judgment
Sequence generation “patients were randomly allocated”
Was the allocation sequence adequately generated?
YES / NO / UNCLEAR
Allocation concealment No information Was allocation adequately concealed? YES / NO / UNCLEAR
Blinding of participants, personnel and outcome assessors
Outcome:
“Assessors were blinded to participant’s treatment group”
Was knowledge of the allocated intervention adequately prevented during the study?
YES / NO / UNCLEAR
Blinding of participants, personnel and outcome assessors
Outcome:
Was knowledge of the allocated intervention adequately prevented during the study?
YES / NO / UNCLEAR
Cochrane Risk of Bias Tool
57
/ /
Incomplete outcome data Outcome:
Were incomplete outcome data adequately addressed?
YES / NO / UNCLEAR
Incomplete outcome data Outcome:
“20% of participants in the intervention group and 35% in the control group withdrew prior to the last follow‐up measurement”
Were incomplete outcome data adequately addressed?
YES / NO / UNCLEAR
Selective outcome reporting Some components of the composite outcome were reported and those presented were significant .
Are reports of the study free of suggestion of selective outcome reporting?
YES / NO / UNCLEAR
Other sources of bias Study funded by the makers of the study drug; three coauthors were industry employees.
Was the study apparently free of other problems that could put it at a high risk of bias?
YES / NO / UNCLEAR
2/20/2012
20
What do You do with the RoB Assessment?
• Stratify studies by RoB
• Formal comparisons of effects by RoB with meta‐regression or test for differences across subgroups
ll d d d d
58
• Present all studies and provide a narrative discussion
• Restrict primary analysis of all studies and then of low, or low and unclear RoB
• Exclude high RoB studies in synthesis
• Sensitivity analysis: how are conclusions affected if high RoB studies were included?
Part 6. Evidence Synthesis
59
y
Synthesis (noun)
1 a : the composition or combination of parts or elements so as to form a whole; c : the combining of often diverse conceptions into a coherent whole;a coherent whole;
Merriam Webster Online Dictionary, 2009
2/20/2012
21
Synthesis in Systematic Reviews
• Explore relationships in the data• Describe patterns across included studies
• Direction of effects
• Size of effects
• Draw conclusions about the likely size and direction of effect
• Assess robustness of the data and conlusions• Strength of the evidence
• Applicability of conclusions on effect size to different populations and/or contexts
Popay J, Roberts H, Sowden A, et al. Guidance on the Conduct of narrative Synthesis in Systematic Reviews (draft). ESRC Methods Program, UK.
Additional Elements of Synthesis in Systematic Reviews
• Identify gaps in the evidence– Little or no direct evidence– Poor quality evidence– Variation that is not explainable– Future research recommendation
• Based on findings• Specific enough for funders
Additional Elements of Synthesis in Systematic Reviews
• Limitations of the review– Scope
• Any important elements not addressed
– Search strategies• Databases searched• Language restrictions
– Study selection and abstraction• Two reviewers vs one• Indirect evidence
– Limitations of analyses• Low power due to few studies
2/20/2012
22
Quantitative Synthesis: Meta‐analysis
• Pool results of a specific outcome across multiple studies
• Studies are weighted by a measure of dispersion from each study
• Provides an overall point estimate of effect
64
• Narrower confidence interval
• May be appropriate when: • limited sample sizes prevent finding a difference where one may exist
• studies have conflicting results: assess heterogeneity
• Designed to address direct evidence for a singleoutcome of interest
• Data obtained (ideally) from a systematic review
Meta‐analysis: Enoxaparin versus Unfractionated Heparin
65Gould et al. Ann Intern Med 1999 May 18;130(10):800Gould et al. Ann Intern Med 1999 May 18;130(10):800‐‐9.9.
Part 7.
Grading the Strength of Evidence
66
Across Studies
2/20/2012
23
Grading a Body of Evidence• Subjective summary
• Assessor applies a descriptive word to evidence• Eg, good, fair, poor
• Imprecise and subjective
• Formal systems for grading a body of evidence
67
• Many systems exist; most flawed
• Gives greater weight to better designs
• Difficult to apply when contradictory evidence exists
• Different systems consider different study features: quality, design, consistency, number of studies, etc.
Grades of Recommendation Assessment, Development, and
Evaluation (GRADE)
Since 2000
Researchers/guideline developers with
68
interest in methodology
Aim: to achieve a common, transparent and sensible system for grading the quality of evidence and the strength of recommendations
GRADE UptakeWorld Health OrganizationNational Institute Clinical Excellence (NICE)Agency for Healthcare Research and Quality (AHRQ)Canadian Agency for Drugs and Technology in Health Allergic Rhinitis in Asthma Guidelines (ARIA)American Thoracic Society American College of Chest Physicians
69
g yUpToDate British Medical Journal American College of PhysiciansCochrane Collaboration European Society of Thoracic SurgeonsClinical Evidence American Urological Association Many other organizations
2/20/2012
24
Grading Body of Evidence:GRADE method
• Directness of evidence– Health outcomes vs intermediate outcomes, efficacy outcomes
– Direct comparisons vs indirect comparisons
• Precision of estimates– Degree of certainty surrounding an effect estimate with respect to a
71
given outcome
• Consistency of results– Effect sizes have the same sign
– The range of effect sizes is narrow
• Risk of bias– Study design
– Aggregate risk of bias of the studies under consideration
*GRADE Working Group: www.GradeWorking‐Group.org
Rate Quality of the Evidence
= Confidence in an estimate of effect• High: considerable confidence in estimate of effect
• Moderate: further research may change
72
estimate
• Low: further research is very likely to impact the estimate
• Very low: estimate of effect is very uncertain
2/20/2012
25
Summary of Findings Tables
74
Part 8.Resources for Identifying or
Performing High Quality Systematic g g y yReviews
2/20/2012
26
76 23
Archie Cochrane
Image of Archive Cochrane removed
77
Image of Archive Cochrane removed.
Image of Cochrane logo removed
78
Image of Cochrane logo removed.
2/20/2012
27
Recommendations on reporting of systematic reviews
• QUORUM, 1999
– Quality of Reporting of Meta‐analyses
• MOOSE, 2000
l i f b i l di i
79
– Meta‐analysis of observational studies in epidemiology (Stroup et al. JAMA April 2000)
• PRISMA, 2009
– Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (Moher et al. PLoS Med July 2009)
Quality Assessment of Systematic Reviews: AMSTAR
11‐Items answered as:• � Yes• � No• � Can’t answer
80
• � Not Applicable
‐ Addresses review planning, searching, quality assessment, synthesis, conflicts of interest
Shea et al. BMC Medical Research Methodology 2007
Locating Systematic Reviews
• Search Strategies in Medline
– Study type: systematic review
– EBM reviews
“systematic review” or “Medline” as text term in
81
– systematic review or Medline as text term in title or abstract
• Key things to look for:
– Year of last search
– Inclusion criteria: PICOTS
– Quality of the review
2/20/2012
28
Organizations that Produce High Quality Systematic Reviews
• Agency for Healthcare Research and Quality– Effective Healthcare Program (effectivehealthcare.ahrq.gov)
– Evidence‐based Practice Center program (www.ahrq.gov)
• US Preventive Services Task Force– Supported by EPC program (www.ahrq.gov/clinic/uspstfix.htm)
82
Supported by EPC program (www.ahrq.gov/clinic/uspstfix.htm)
• Cochrane Collaboration reviews (www.cochrane.org)
• Health Technology Assessment programs– E.g. UK NCCHTA (www.ncchta.org)
• National Institute for Clinical Excellence (www.nice.org.uk)
• Drug Effectiveness Review Project (www.ohsu.edy/drugeffectiveness)
Resources for Performing Systematic Reviews
• Organizations
– Cochrane Collaboration: www.cochrane.org
– NICE, SIGN
– AHRQ: Methods Guide for Effectiveness and Comparative Effectiveness Reviews 2009
83
Comparative Effectiveness Reviews 2009, http://effectivehealthcare.ahrq.gov
• Books
– Systematic Reviews in Health Care: Meta‐Analysis in Context: Eggers et al. BMJ Books
– Cochrane Handbook: 2008, Wiley & Sons
– IOM report 2011: Finding What Works in Healthcare
Caveat Emptor: Systematic Reviews
• SRs should give the best estimate of any true effect, but– MUST be well done
• Findings must be taken in concert with other evidence and factors – Evidence outside the scope of the review may be relevant– Cost and implementation implications– Generalizability of findings
84
y g
• Readers sometimes have unrealistic expectations– Limitations of the available evidence
• Lack of evidence:– No large and/or good quality trials– No trials at all– Interventions or populations of interest not specifically studied
– Limitations of a review process• The review does not make decisions for you• An individual review may not answer all questions related to a given decision• While a systematic approach reduces bias, it does not eliminate it
• Reviews become outdated
2/20/2012
29
Suggested ReadingsMoher D, et al. Preferred Reporting Items for Systematic Reviews
and Meta‐Analyses: The PRISMA Statement; PLoS Medicine 6 (2009)
Juni P, et al. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001;323:42–6
85
Guyatt G, et al. GRADE guidelines: 1. Introduction ‐ GRADE evidence profiles and summary of findings tables; J Clin Epi. 64 (2011) 383‐394
IOM 2008. Systematic Review: The Central Link between Evidence and Clinical Decision making. Report Brief, January, 2008
References & Resources1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2
[updated September 2009]. The Cochrane Collaboration, 2009. Available from www.cochrane‐handbook.org.
2. Merriam Webster Online Dictionary. Synthesis. Retrieved 2009 from http://www.merriam‐webster.com/dictionary/synthesis.
3. Popay J, Roberts H, Sowden A, et al. Guidance on the Conduct of narrative Synthesis in Systematic Reviews (draft). ESRC Methods Program, UK.
4. Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group. (2005‐2011), from http://www.gradeworkinggroup.org/index.htm
5. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy‐five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med, 7(9), e1000326. doi: 10.1371/journal.pmed.1000326
6. Gould, M. K., Dembitzer, A. D., Sanders, G. D., & Garber, A. M. (1999). Low‐molecular‐weight heparins compared with unfractionated heparin for treatment of acute deep venous thrombosis. A cost‐effectiveness analysis. [Comparative Study Meta‐Analysis Research Support, U.S. Gov't, P.H.S.]. Ann Intern Med, 130(10), 789‐799.
7. Guyatt, G. H., Oxman, A. D., Schunemann, H. J., Tugwell, P., & Knottnerus, A. (2011). GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol, 64(4), 380‐382. doi: 10.1016/j.jclinepi.2010.09.011
8. Norris, S. L., Kansagara, D., Bougatsos, C., & Fu, R. (2008). Screening adults for type 2 diabetes: a review of the evidence for the U.S. Preventive Services Task Force. [Research Support, U.S. Gov't, P.H.S.Review]. Ann Intern Med, 148(11), 855‐868.
References & Resources9. Sackett, D. L. (1979). Bias in analytic research. J Chronic Dis, 32(1‐2), 51‐63.
10. Moher D, et al. Preferred Reporting Items for Systematic Reviews and Meta‐Analyses: The PRISMA Statement; PLoS Medicine 6 (2009)
11. Juni P, et al. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001;323:42–6
12. Guyatt G, et al. GRADE guidelines: 1. Introduction ‐ GRADE evidence profiles and summary of findings tables; J Clin Epi. 64 (2011) 383‐394
13. IOM 2008. Systematic Review: The Central Link between Evidence and Clinical Decision making. Report Brief, January, 2008
14. Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: Introduction. In: Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane‐handbook.org.
15. Cook, D. J., Sackett, D. L., & Spitzer, W. O. (1995). Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta‐Analysis. [Consensus Development Conference Guideline Review]. J Clin Epidemiol, 48(1), 167‐171.
16. Egger, M., Smith, George Davey, Altman, Douglas. (2001). Systematic Reviews in Health Care: Meta‐Analysis in Context (2 ed.): BMJ Books.
17. Haynes, B. (1999). Can it work? Does it work? Is it worth it? BMJ, 319(7211), 652‐653. doi: 10.1136/bmj.319.7211.652
2/20/2012
30
References & Resources18. Jadad, A. (1998). Randomised Controlled Trials: A User's Guide. BMJ, 317(7167), 1258. doi:
10.1136/bmj.317.7167.1258
19. Mulrow, C. D., Cook, D. J., & Davidoff, F. (1997). Systematic reviews: critical links in the great chain of evidence. [Comment Editorial]. Ann Intern Med, 126(5), 389‐391.
20. Research, C. o. S. f. S. R. o. C. E., & Medicine, I. o. (2011). Finding What Works in Health Care: Standards for Systematic Reviews: The National Academies Press.
21. Shea, B., Grimshaw, J., Wells, G., Boers, M., Andersson, N., Hamel, C., . . . Bouter, L. (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol, 7(1), 10.
22. Smith, G. C., & Pell, J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. [Review]. BMJ, 327(7429), 1459‐1461. doi: 10.1136/bmj.327.7429.1459
23. Sox, H. C., & Greenfield, S. (2009). Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med, 151(3), 203‐205.
24. **Rockwood, K., Hogan, D. B., & Patterson, C. J. (2004). Incidence of and risk factors for nodding off at scientific sessions CMAJ (2004/12/08 ed., Vol. 171, pp. 1443‐1445).
top related