updated - what makes systematic review systematic - anna sidorchuk

What makes a systematic review systematic? Epidemiologic perspectives

ANNA SIDORCHUK, MD, PHD, RESEARCH COORDINATOR

KAROLINSKA INSTITUTET DEPARTMENT OF PUBLIC HEALTH SCIENCES

[email protected]

THE EPIDEMIOLOGY PHD PROGRAM SEMINAR

APRIL 29, 2013

mailto:[email protected]








Outline of the speech

Non-systematic vs. systematic review

Why do systematic reviews? What is the added value?

PRISMA/MOOSE/AMSTAR guidelines and checklists

Randomized vs. observational data

Process (criteria, optimize search, determine data to be

abstracted, quality score)

Anna Sidorchuk 2

30 april 2013 Anna Sidorchuk

Primary data versus secondary data

Traditionally, researchers collect and analyze their own data

(referred to as primary data)

Secondary data analysis is based on data collected by someone

else (or reanalysis of your own published data)

Systematic review is one of the ways to analyze secondary data

Systematic review - systematic, qualitative review of published research in a particular field

Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford


What The Lancet thinks about systematic reviews

In 2005, the editors wrote:

“. . . we will require authors of clinical trials submitted to The

Lancet to include a clear summary of previous research findings,

and to explain how their trial’s findings affect this summary. The

relation between existing and new evidence should be illustrated

by direct reference to an existing systematic review and meta-

analysis…”

Stages of waste in the production and reporting of research evidence relevant to clinicians and patients by Chalmers and Glasziou (Lancet, vol. 374, 2009)

Anna Sidorchuk 5


Why do we do systematic reviews?

To help busy clinicians to summarize current knowledge in

relation to the area of interest

To provide high-quality research evidence to guide clinical

practice

To support clinical decision-making

To support research proposals


How systematic review can help busy clinicians?

By systematically:

identifying,

appraising,

synthesizing,

and, if appropriate, statistically combining studies on a

specific topic


In classifications of levels of evidence,

systematic reviews are included in the highest level of

evidence

Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions

Journal of Clinical Nursing Volume 12, Issue 1, pages 77-84, 20 DEC 2002 DOI: 10.1046/j.1365-2702.2003.00662.x http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2702.2003.00662.x/full#f1

http://onlinelibrary.wiley.com/doi/10.1111/jcn.2003.12.issue-1/issuetoc

http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2702.2003.00662.x/full





Meta-analysis

A statistical method to combine findings across studies

Should be considered within the framework of systematic reviews

review needs to use a systematic approach to minimize bias, address

the issues of the completeness of the evidence, quality of studies

and combinability of studies

Meta-analysis - the statistical pooling of the results of

studies that are part of a systematic review

When systematic review is required?

Before undertaking a systematic review it is necessary to

check whether there are already existing or ongoing reviews,

and whether a new review is justified

Search:

the Database of Abstracts of Reviews of Effects (DARE)

the Cochrane Database of Systematic Reviews (CDSR)


Adapted from: Systematic Reviews. CRD’s guidance for undertaking reviews in health care (2009). Centre for Reviews and Dissemination, University of York. ISBN 978-1-900640-47-3.

When is meta-analysis appropriate?

There exists a critical mass of comparable studies designed to

address a common research question

Data are presented in a form that allows the meta-analyst to

compute an effect size for each study

Characteristics of each study are described in sufficient detail

to allow meta-analysts to compare characteristics of different

studies and to judge the quality of each study 30 april 2013 Anna Sidorchuk

Systematic reviews vs. non-systematic

Compared to traditional literature reviews:

there is a definite methodology employed in the research analysis

(more like that used in primary research); and

the results of the included studies are quantified to a standard metric

thus allowing for statistical techniques for further analysis.

Therefore process of reviewing research literature is more objective, transparent, and replicable; less biased and idiosyncratic to the whims of a particular researcher


Systematic reviews vs. non-systematic Feature Systematic review Narrative review

Question Often a focused one question Often broad in scope

Sources and search

Comprehensive sources and

explicit search strategy

Not usually specified,

potentially biased

Selection

Criterion-based selection,

uniformly applied

Not usually specified,

potentially biased

Appraisal Rigorous critical appraisal Variable

Synthesis

Qualitative summary that

includes statistical synthesis

(meta-analysis)

Often a qualitative summary

Inferences Usually evidence-based Sometimes evidence-based

Anna Sidorchuk 14

Adapted from: Y Yuan and R H Hunt, Am J Gastroenterol 2009; 104:1086–1092; doi:10.1038/ajg.2009.118

18 September, 2009 Anna Sidorchuk 15

Increase in the number of published meta-analyses (PubMed references)

Publication year

• The essence of good science is replicable and generalisable results

Do we get the same answer to research questions when we run the study again?

• The primary aims of meta-analysis is to test the generalisability of results

across a set of studies designed to answer the same research question

Are the results consistent? If not, what are the differences in the studies that explain

the lack of consistency?

Why are systematic review and meta-analysis important?


When there is systematic variation in outcomes from different studies, meta-analysis tries to explain these differences in terms of study characteristics:

measures used

study design

participant characteristics

controls for potential bias

etc.

If the results of individual studies are inconclusive or controversial…


“… doing a meta-analysis is easy,

doing one well is hard.”

Ingram Olkin, Stanford University

Benefits of systematic reviews, incl. meta-analysis

Increased power: by combining information from many

individual studies, the meta-analyst is able to detect systematic

trends not obvious in the individual studies

Conclusions based on the set of studies are likely to be more

accurate than any one study



Improved precision: based on information from many studies,

the meta-analyst can provide a more precise estimate of the

population effect size (and a confidence interval)

Provides potential corrections for potential biases,

measurement error and other possible artefacts

Identifies directions for further primary studies to address unresolved issues



Typically there is study-to-study variation in results. When

this is the case, the meta-analyst can explore what

characteristics of the studies explain these differences (e.g.,

study design) in ways not easy to do in individual studies

Easy to interpret summary statistics (useful if communicating

findings to a non-academic audience)


Limitations of systematic reviews, incl. meta-analysis

Comparing apples and oranges

Quality of the studies included in the meta-analysis

What to do when studies don’t report sufficient information

(e.g., “non-significant” findings)?

Including multiple outcomes in the analysis (e.g., different

achievement scores)

Publication bias

22

Limitations of systematic reviews, incl. meta-analysis

Meta-analysis conclusions may still differ if different studies are sampled or excluded for different reasons

Need to be explicit

Quality standards

Inappropriate handling of data can lead to wrong conclusions

Sample size consideration

23

Publication bias

Studies that are published are more likely to report

statistically significant findings. This is a source of potential

bias


Publication bias The debate about using only published studies:

peer-reviewed studies are presumably of a higher quality

VERSUS

significant findings are more likely to be published than non-

significant findings

There is no agreed upon solution. However, one should retrieve all studies

that meet the eligibility criteria, and be explicit with how they dealt with

publication bias. Some methods for dealing with publication bias have

been developed (e.g., Fail-safe N, Trim and Fill method)


RCT vs. observational studies

Both are fine as long as all the methodological issues are

addressed

If your focus is effect of therapy or prevention – use RCTs

If you are interested in prognosis, etiology, adverse

effects, association between risk factors and outcomes –

use observational studies

Fine to use both types of studies in the same review

Anna Sidorchuk 26

Steps in a meta-analysis Establish research question

Define relevant studies

Develop code

materials

Locate and collate studies

Pilot coding; coding

Data entry and effect

size calculation

Main analyses

Supplementary analyses

30 april 2013 Anna Sidorchuk Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford

Practical tips

Define all key elements (PICO model) a priory

Formulate the research question as clear as possible

Make an agreement with all your co-authors on the key elements

If needed ask the experts in the area of outcome or/and exposure for

advice

Keep everything in the protocol

When writing an article include the elements in the aim


Practical tips

Define all inclusion/exclusion criteria and components of search a priory

Make criteria and the components as clear as possible, but feel free to

make some changes during the actual literature search (usually due to

availability of data, e.g. if only retrospective studies are available on the

issue, or no info on ethnicity, etc)

Make an agreement with all co-authors on abovementioned

Again, keep everything in the review protocol


Practical tips

Define coding and form for data extraction a priory

Make an agreement with all co-authors on abovementioned

If use Excel for data extraction and, therefore, for coding, keep the legend

for each coding element within the same file on the separate sheet

As an alternative – use Access

Keep everything in the review protocol!


Practical tips

Define as detailed as possible by reading other meta-analysis and

systematic reviews on the similar outcome(s) and exposure(s):

All key words

MESH terms

Define the electronic sources to be used for your search

Ask librarians for advice on search strategy and technic

Define if grey literature will be included

Keep everything in the protocol!!!


Critical issue

Quality, quality, quality!


Quality Standards in Systematic reviews and Meta-Analysis Several organizations work on this

Cochrane collaboration

http://www.cochrane.org

Campbell collaboration

http://www.campbellcollaboration.org

Consort: Consolidated Standards of Reporting Trials, includes Quorum: Quality of Reporting of Meta-analyses group

http://www.consort-statement.org/mod_product/uploads/QUOROM Statement 1999.pdf

PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses), AMSTAR (A measurement tool to assess systematic reviews)


http://www.cochrane.org/



http://www.campbellcollaboration.org/











PRISMA: Checklist of items to include when reporting a systematic review or meta-analysis.

Section/Topic # Checklist Item Reported

on Page #

TITLE

Title 1 Identify the report as a systematic review, meta-analysis, or both.

ABSTRACT

Structured

summary 2

Provide a structured summary including, as applicable:

background; objectives; data sources; study eligibility criteria,

participants, and interventions; study appraisal and synthesis

methods; results; limitations; conclusions and implications of key

findings; systematic review registration number.

INTRODUCTION

Rationale 3 Describe the rationale for the review in the context of what is

already known.

Objectives 4

Provide an explicit statement of questions being addressed with

reference to participants, interventions, comparisons, outcomes,

and study design (PICOS).




on Page #

METHODS

Protocol and

registration 5

Indicate if a review protocol exists, if and where it can be

accessed (e.g., Web address), and, if available, provide

registration information including registration number.

Eligibility

criteria 6

Specify study characteristics (e.g., PICOS, length of follow-

up) and report characteristics (e.g., years considered,

language, publication status) used as criteria for eligibility,

giving rationale.

Information

sources 7

Describe all information sources (e.g., databases with dates of

coverage, contact with study authors to identify additional

studies) in the search and date last searched.

Search 8

Present full electronic search strategy for at least one

database, including any limits used, such that it could be

repeated.

Study selection 9

State the process for selecting studies (i.e., screening,

eligibility, included in systematic review, and, if applicable,

included in the meta-analysis).

Data collection

process 10

Describe method of data extraction from reports (e.g., piloted

forms, independently, in duplicate) and any processes for

obtaining and confirming data from investigators.

Data items 11

List and define all variables for which data were sought (e.g.,

PICOS, funding sources) and any assumptions and

simplifications made.




on Page #

METHODS

Risk of bias in

individual studies 12

Describe methods used for assessing risk of bias of

individual studies (including specification of whether this

was done at the study or outcome level), and how this

information is to be used in any data synthesis.

Summary

measures 13

State the principal summary measures (e.g., risk ratio,

difference in means).

Synthesis of

results 14

Describe the methods of handling data and combining

results of studies, if done, including measures of consistency

(e.g., I2) for each meta-analysis.

Risk of bias

across studies 15

Specify any assessment of risk of bias that may affect the

cumulative evidence (e.g., publication bias, selective

reporting within studies).

Additional

analyses 16

Describe methods of additional analyses (e.g., sensitivity or

subgroup analyses, meta-regression), if done, indicating

which were pre-specified.




on Page #

RESULTS

Study selection 17

Give numbers of studies screened, assessed for eligibility,

and included in the review, with reasons for exclusions at

each stage, ideally with a flow diagram.

Study

characteristics 18

For each study, present characteristics for which data were

extracted (e.g., study size, PICOS, follow-up period) and

provide the citations.

Risk of bias

within studies 19

Present data on risk of bias of each study and, if available,

any outcome-level assessment (see Item 12).

Results of

individual studies 20

For all outcomes considered (benefits or harms), present, for

each study: (a) simple summary data for each intervention

group and (b) effect estimates and confidence intervals,

ideally with a forest plot.

Synthesis of

results 21

Present results of each meta-analysis done, including

confidence intervals and measures of consistency.

Risk of bias

across studies 22

Present results of any assessment of risk of bias across

studies (see Item 15).

Additional

analysis 23

Give results of additional analyses, if done (e.g., sensitivity

or subgroup analyses, meta-regression [see Item 16]).




on Page #

DISCUSSION

Summary of

evidence 24

Summarize the main findings including the strength of

evidence for each main outcome; consider their relevance to

key groups (e.g., health care providers, users, and policy

makers).

Limitations 25

Discuss limitations at study and outcome level (e.g., risk of

bias), and at review level (e.g., incomplete retrieval of

identified research, reporting bias).

Conclusions 26 Provide a general interpretation of the results in the context of

other evidence, and implications for future research.

FUNDING

Funding 27

Describe sources of funding for the systematic review and

other support (e.g., supply of data); role of funders for the

systematic review.

Quality Assessment of Primary Studies

A common practice is to assign a quality score to primary studies

Preferably based on an explicit checklist usually

The judgment to include or exclude a study must be stated

and justified explicitly

This judgment is mostly based on quantitative criteria, but

involves some subjectivity


NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE CASE CONTROL STUDIES

Note: A study can be awarded a maximum of one star for each numbered item within the Selection and Exposure categories. A maximum of two stars can be given for Comparability.

Selection

1) Is the case definition adequate? a) yes, with independent validation ¯ b) yes, eg record linkage or based on self reports

c) no description

2) Representativeness of the cases a) consecutive or obviously representative series of cases ¯

b) potential for selection biases or not stated

3) Selection of Controls a) community controls ¯ b) hospital controls

c) no description

4) Definition of Controls

a) no history of disease (endpoint)

b) no description of source


NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE CASE CONTROL STUDIES

Comparability

1) Comparability of cases and controls on the basis of the design or analysis

a) study controls for _______________ (Select the most important factor.) ¯

b) study controls for any additional factor ¯ (This criteria could be modified to indicate specific control for a second important factor.)

Exposure

1) Ascertainment of exposure

a) secure record (eg surgical records) ¯

b) structured interview where blind to case/control status ¯

c) interview not blinded to case/control status

d) written self report or medical record only

e) no description

2) Same method of ascertainment for cases and controls

a) yes ¯

b) no

3) Non-Response rate

a) same rate for both groups ¯

b) non respondents described

c) rate different and no designation


“Ugly face” of systematic reviews and meta-analyses

Can be misleading

Can be misused


updated - what makes systematic review systematic - anna sidorchuk

Health & Medicine

researcher

practical

significant

publication

systematic

data extraction

esrc workshop

systematic