Systematic Review: Interpreting Results

and Identifying Gaps

October 17, 2012

Stages of Systematic Review

1. Define the Problem2. Literature Search3. Data Evaluation4. Data Analysis5. Interpretation of Results6. Presentation of Results

Finishing a systematic review is not an endpoint!

Outcomes of Systematic Review

• Answer a question with current evidence

• Describe status of present research literature on a subject

• Summarize and reflect

• Highlight what is known and unknown

Looking for an answer?

• Often cannot give a definitive answer, but…

• Leads to new theories• Provides basis for future research• Ideas for new directions in the field• Provide information for insightful decisions on policy

Interpreting Systematic Review

Results o Quality of included studies

• Study design• Bias• Missing data• Heterogeneity of effect sizes• Generalizability

o Certainty vs uncertainty

o Known vs unknown

Certainty in Systematic Review

Certainty – confidence with which scientific community accepts associations between variables and theories for associations

• Need to determine how certain conclusions are• High certainty – additional research of the kind

evaluated is not necessary for further substantiation• Low certainty – more research is needed

Certainty in Systematic Review

o False certainty• Conclusions portrayed as certain when they are not• Encouraged by publication expectations?• Failure to assess bias, heterogeneity, validity,

mechanisms, etc• Can discourage needed research

o Need to acknowledge areas of uncertainty in systematic review• Critical evaluation of goals of review: assessing

relationships or assessing moderators of relationships?

Using the Results of Review

• Highly certain reviews provide a foundation of accepted knowledgeo Concise base of theory to direct future worko Direct policy

• Uncertainty also directs the futureo Synthesis can reveal what is lackingo Identify what studies need to be doneo Identify the unknowno Uncertain vs negative or neutral results

Interpreting Systematic Reviews: Sometimes, More is Better

Improving the Next Generation of

Research:

Where are the gaps?

• Systematic review reveals studies that are completely lacking in the evidence base – important in Conservation Medicine

• Consider systematic review as a stepping stone

• Not a final phase for a question or hypothesis

• Systematic review is a method to generate new theories

• Use uncertainty to prioritize research

At what stage of a systematic review can you identify gaps?

Identifying Gaps• May become apparent during literature

searcho Complete lack of studies on certain topicso Lack of certain study designso Lack of proper methods

• Evaluation of certainty of conclusionso Quality, bias, validity

• Generalizable resultso What research is needed to fill gaps?

Systematic Review and Public Policy

• Review question may aim to influence policy

• Provides alternative to single research study, expert consensus, personal or public opinions

• Answer questions about programs, policies, practices, or products

• Interest in evidence based practices

Influencing Public Policy

• Positive and negative influence• Bias and utility of the review• Who requested or initiated the review?• Who funded the review?• Influence of business• Influence of politics• Transparency of results

Public Policy and Conservation Medicine

• Often identify gaps in research!

• Systematic review as a tool to influence policy to fund research to fill gaps

• Prioritize research to fill gaps• Institute policies that protect health• Communication of review results to policy makers

and public

Organic Foods Review

What is the conclusion?

How certain is the conclusion?

What factors influence the certainty?

Organic Foods Review

What specific new research is called for in the conclusion?

What policy considerations are influenced by the conclusion?

Finishing Review Paper