Evidence-Based Research

Karen A. Robinson

Johns Hopkins University

Evidence-Based Research Network

Bergen Norway

3 December 2014

Scientific knowledge is cumulative,and evidence should be cumulated

scientifically.

Research Synthesis: Formal methods for summarizing evidence for a research question by collecting studies addressing similar questions and evaluating the consistencies and variability in these studies

Cornerstone of evidence-based medicine

Evidence-based Medicine

“The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.”

- Sackett et al., 1996

Sackett et al. Evidence based medicine: what it is and what it isn’t. BMJ 1996;312(7023):71-2.

Robinson, Karen A. Use of prior research in the justification and interpretation of clinical trials. The Johns Hopkins University, ProQuest, UMI Dissertations Publishing, 2009. 3392375.

“While the use of research synthesis to make evidence-informed decisions is now expected in health care, there is also a need for clinical trials to be conducted in a way that is evidence-based.

Evidence-based research is one way to reduce waste in the production and reporting of trials, through the initiation of trials that are needed to address outstanding questions and through the design of new trials in a way that maximizes the information gained.”

Evidence-Based Research

Using evidence to inform research so that it is addressing questions that matter in a valid, efficient and accessible manner.

The scientific and ethical justification for studies requires the use of prior research

“Medical research involving human subjects must conform to generally accepted scientific principles, [and] be based on a thorough knowledge of the scientific literature,...” - Paragraph 11, Declaration of Helsinki

Emanuel EJ, Wendler D, Grady C. What makes clinical research ethical? JAMA. 2000 May 24-31;283(20):2701-11.

Justification and interpretation

In embarking on a trial- Requirements for ethical clinical research:

• Social or scientific value• Favorable risk-benefit ratio

At completion of trial – Integration of new results with existing evidence:

• Place results within context of existing knowledge• Prior research is needed to interpret the results of a trial and

what they mean for our understanding research question

Do researchers use prior research in providing the justification and interpretation of the results of trials?

52 additional trials

> 5,600 patients

Fergusson, D; Glass, K C; Hutton, B, and Shapiro, S. Randomized controlled trials of aprotinin in cardiac surgery using clinical equipoise to stop the bleeding. Clinical Trials. 2005; 2(3)218-232.

“No trial is an island, entire of itself; every trial is a piece of the continent, a part of the main.”

- Clarke and Chalmers, 1998

Classification of discussion sections of reports of RCTs in 5 major medical journals

Date of Publication(Number of reports)

1997(n=26)

First trial 1

Contained an updated systematic review integrating the new results

2

Discussed a previous review but did not attempt to integrate new results

4

No apparent systematic attempt to set new results in context of other trials

19

Classification of discussion sections of reports of RCTs in 5 major medical journals

Date of Publication(Number of reports)

1997(n=26

)

2001(n=33

)

2005(n=18

)

2009(n=29

)

2012(n=35

)

First trial 1 3 3 5 2

Contained an updated systematic review integrating the new results

2 0 0 1 2

Discussed a previous review but did not attempt to integrate new results

4 3 5 10 11

No apparent systematic attempt to set new results in context of other trials

19 27 10 13 20

Citation of Similar TrialsCitation of Similar Trials

Objective

To assess the extent of the use of prior research by measuring the citation of prior related trials in reports of randomised controlled trials (RCTs).

Specifically, do reports of RCTs cite prior similar trials?

Methods

Briefly:1. Created cohorts of RCTs addressing the

same question2. Determined if RCTs in same cohort cite each

other

Methods

For each RCT, we calculated:

Prior Research Citation Index (PRCI)Number of RCTs cited Total number of potential RCTs to cite

Sample Size Citation Index (SSCI)Total number of participants in cited RCTsTotal number of participants in potential RCTs to

cite

Prior Research Citation Index

Median PRCI 0.21(95% CI 0.2 to 0.25, Mean =0.29, n=1,523 RCTs)

PRCI = Number of RCTs cited / Total number of potential RCTs to cite

Publication Date

Number of RCTs

Median PRCI

2 years or earlier

1,314 0.21

5 years or earlier

747 0.11

10 years or earlier

311 0.04

Number of potential prior trials to cite

Number of reports

Number of reports that cited 0 or 1 prior trials (%)

3 or more 1,523 741 (49%)5 or more 1,101 511 (46%)10 or more 508 261 (51%)15 or more 282 148 (52%)

Of 1,101 RCTs with 5 or more prior trials to cite 511 (46%) cited no prior trial or only one prior trial

Citation of zero or one prior RCT

Citation of prior trials by sample size

• Sample Size Citation Index (SSCI) calculated for 1,261 RCTs

• Median SSCI 0.24 (95% CI 0.21 to 0.27)

SSCI =Number of participants in RCTs cited / Total number of participants in potential RCTs to cite

Conclusions

• About 80% of prior studies about the same question were NOT cited in reports of RCTs– About half of the RCTs cited zero or one prior trial

• Information from 76% of participants enrolled in prior trials not acknowledged

Why happening?

This is better now.

Rich get Richer

If trial cited, 60% probability it will be cited again

If trial NOT cited – more than 60% probability that it will remain uncited

What are causes of lack of citation?

• Cite systematic reviews instead• Studies assessing factors that predict citation:

– In 8 of 10 studies, factors other than methodological features of the study predicted subsequent citation

– Other factors were significant predictors of citation, such as:

• geographical location of author• results seen as supportive

Why and better now?

• Combined new search results (2011) with existing database

• Citation patterns in pharmacological CVD RCTs

• 86 meta-analyses with 580 trials

Veronica Ivey Sawin. Recognition of prior research: Citation patterns in reports of clinical

trials in cardiovascular research. ScM Thesis. Johns Hopkins University Bloomberg School of

Public Health. 2013

Results – Not better

Results in CVD:•Median PRCI 0.25 (95% CI 0.23 to 0.27)•Median SSCI 0.31 (95% CI 0.25 to 0.36)

No difference between 2004 and 2011 datasets

Approximately 75% of prior trials ignored and only 30% of trial participants represented

Difference over time?

Black = PRCI Orange = SSCI

Supportive vs not?

Trials were about 45% more likely to be cited by subsequent trials if results supportive.RR 1.45 (95% CI 1.303 to 1.612)

Inadequate, and biased, consideration of prior research remains – implications include ethically unjustifiable trials, wasted research and unnecessary risk for trial participants.

54% were not a single connected network:

– 39% two islands– 4% 10 or more separate

islands

Robinson KA, Dunn AG, Tsafnat G, Glasziou P. Citation networks of trials: Feasibility of iterative bidirectional citation searching. Journal of Clinical Epidemiology 2014;67(7):793-9.

“No trial is an island, entire of itself; every trial is a piece of the continent, a part

of the main.”- Clarke and Chalmers, 1998

What should happen?

Use evidence to:– Identify worthwhile questions– Design valid and informative studies– Report results within context of what is known

Identifying QuestionsFor systematic reviews:

http://capsmg.cochrane.org/

Identifying Questions

For studies, use:– guidelines– systematic reviews

Identified gaps from evidence-basedguidelines as:

– Insufficient evidence, no recommendations made– Insufficient evidence, consensus

recommendations made– “Needs further research”

Identifying research gaps from guidelines:

•62 gaps identified, of which only one was fully specified (PICO) •Only about 20% of the gaps were called out by guideline committees as research gaps.

Framework for Determining Research Gaps During Systematic Reviews

Robinson KA, Saldanha IJ, Mckoy NA. Frameworks for determining research gaps during systematic reviews. Methods Future Research Needs Report No. 2 AHRQ Publication No. 11-EHC043-EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2011. PMID: 21977524Robinson KA, Saldanha IJ, Mckoy NA. Development of a framework to identify research gaps from systematic reviews. Journal of Clinical Epidemiology 2011 Dec;64(12):1325-30. PMID: 21937195Robinson KA, Akinyede O, Dutta T, Sawin VI, Li T, Spencer MR, Turkelson CM, Weston C. Framework for determining research gaps during systematic review: Evaluation. Rockville (MD): Agency for Healthcare Research and Quality (US) 2013 Feb. Report No. 13-EHC019-EF. PMID: 23487868.

• Identification of gaps from systematic reviews in a systematic way

• Framework to facilitate the identification and characterization of gaps:Where the evidence falls short, as well as how and why

We applied the framework to 50 systematic reviews

Framework- Results• 144 review questions in the 19 EPC reports (average 5.5

questions) and 31 Cochrane reviews (average 1.3 questions).• A total of approximately 600 unique research gaps were

identified and characterized.• Number of gaps per question: average 12.75 per question for EPC

reports (95% CI 9.31 to 16.19) and 8.5 per question for Cochrane reviews (95% CI 6.23 to 10.32).

• Insufficient information (Reason A) was most frequent reason for the gaps, followed by inconsistency (C), not the right information (D), and biased information (B).

Challenges in Practicing EBR

• Ever increasing number of studies• Methodologically poor studies• Reporting bias• Resources – time and money• Skills and competencies

What do we need?

• Ongoing research– Empirical research to improve research

prioritization, including to systematically identify gaps from systematic reviews and guidelines

– Research synthesis methods– Implications of not practicing EBR

What do we need?

• Adequate support for research synthesis– Training– Support– Academic credit

• Expectations at funding agencies, ethics boards and journals– Criteria for ethics panels and funding agencies to

determine if proposals follow EBR practices

What should happen?

Use evidence to inform research so that it is answering questions that matter in a valid,

efficient and accessible manner.

Evidence-Based Research

Worthwhile questions; Valid and informative studies; Results within context of what is known

All scientific work is incomplete…that does not confer upon us a freedom to ignore the knowledge we already have…

-Sir Austin Bradford Hill

Hill AB. The reasons for writing. BMJ. 1965;4:870

Evidence-Based Research Network

www.ebrnetwork.org