Documenting and reportingimplementation research:

New expectations, new opportunities

Brian S. Mittman, PhDVA Center for Implementation Practice

and Research Support (CIPRS)

January 6, 2009

Quality Enhancement Research Initiative

1. Background, motivation and goals for improved reporting

2. Ongoing initiatives to strengthen reporting

3. Recommendations and future possibilities

Outline

Research documentation and reporting:Current practices and problems

• Increasing focus on shortcomings in health research documentation and reporting– Variable reporting: different measures, design elements,

analysis standards– Incomplete reporting: publication bias, selective reporting

of outcomes (to favor evidence of efficacy)• Challenges for meta-analyses, replication, cumulative

knowledge

• Concerns over return on societal (public) investment in research, validity of conclusions (policy/practice implications)

Reporting problems in implementation science

• Each of these problems exists in implementation research, and many are more severe than in mainstream clinical research– Selective reporting and publication bias– Variable, inconsistent reporting– Emphasis on impact reporting only– Limited “return on investment” and value of research

1. Background, motivation and goals for improved reporting

2. Ongoing initiatives to strengthen reporting

3. Recommendations and future possibilities

Outline

Major initiatives to strengthen reporting

• CONSORT Statement, extensions and similar statements (e.g., SQUIRE guidelines; EQUATOR Network)

• ClinicalTrials.gov and other registries

• Ottawa statements and protocol publication

• External validity reporting

• Intervention reporting

• Trial bank initiative

CONSORT; EQUATOR Network

• Standardized format, structure, contents of reports of specific types of studies (experimental, observational, reviews, economic)

• Consistent reporting of core features (design, methods, results)

• “The EQUATOR Network is an international initiative that seeks to improve the quality of scientific publications by promoting transparent and accurate reporting of health research.” (www.equator-network.org)

• Updated SQUIRE guidelines are a recent addition

Trial registries

• Advance registration of clinical trials in a public registry reduces the likelihood of (un-detected) selective publication

• The International Committee of Medical Journal Editors (ICMJE) (and most journals) require a priori trial registration as a condition for publication

• Public Law 110-85, Title VIII expands the types of clinical trials that must be registered in ClinicalTrials.gov, increases the number of data elements that must be submitted, and requires submission of results data

Protocol publication

• The Ottawa Statement on trial registration is a consensus document that aims to guide the implementation of global trial registration (http://ottawagroup.ohri.ca/)

• The Statement addressed fundamental principles for trial registration (Part 1) and operationalization of protocol registration (Part 2) and results reporting (Part 3)

External validity; intervention details

• Several recent articles and editorials promote and guide enhanced reporting of external validity to increase the “relevance, generalizability, and applicability” (i.e., value in policy and practice) of health research (Green, Glasgow)

• Other initiatives include ongoing efforts to stimulate more complete, consistent reporting of implementation programs and interventions; contextual factors; etc.

The Trial Bank Project

• “The prose reporting of trial information could be powerfully augmented by a computable repository of trial information --a global trial bank. Unlike prose, computable information is structured and coded for computation and allows the use of advanced information technologies for knowledge management.”

• “The Global Trial Bank seeks to make available open-access and computable peer-reviewed results from all clinical trials conducted worldwide, regardless of whether the results are favorable or not, to provide an unbiased knowledge base for improving human health. “

1. Background, motivation and goals for improved reporting

2. Ongoing initiatives to strengthen reporting

3. Recommendations and future possibilities

Outline

Comprehensive documentation plan

• Systematic, comprehensive documentation plan– Protocol– Baseline/interim results– Main findings– Supporting findings– Secondary analyses

• Required elements of each component– Protocol includes current systematic review(s), power

calculations for contribution to existing evidence base– Main findings linked to protocol; report analyses for all (vs.

selected) outcomes– Supporting analyses explaining findings (mechanisms of action)

Rationalizing publication

• Distinguish types of evidence/knowledge– Policy/practice relevant (systematic reviews)– Primary evidence (main findings)– Explanatory evidence (process evaluations)

• Classify types of evidence, knowledge– Clinical domains– Health services domains

• Allocate types of evidence and knowledge to distinct publication outlets (review journals, researcher journals/on-line resources, trial banks, etc.)

• Expand registries to improve access to evidence, research output

• Link funding programs to registries, trial banks

Successful Implementation Manuscripts: Beyond the Basics

Jeroan J. Allison, MD, MS

Catarina I. Kiefe, PhD, MD

Co-Editors-in-Chief, Medical Care

January 6, 2008

VA Implementation Research Seminar

Conceptual Model for Implementation Research

Salanitro A. et al. In Essentials of Clinical Research. New York: Springer, 2008.

After carefully studying your manuscript, I am sorry to report that we will not be able to

publish it in Medical Care.

Medical Care currently receives over 800 original submissions each year, and we

eventually accept less than 20% for publication.

Because of the increasing competition for journal space, we are not able to assign a

high priority to all worthy manuscripts.

Current Publication Practices May Distort Science

• Problems– Preferential publication of extreme results– Artificial scarcity of publication outlets

• Potential Solutions– Increase publication preference for negative studies– Base acceptance on methodological quality– Enhance post-publication review– Dissociate academic advancement from journal

branding

Young N., et al. PLoS Medicine. 2008; 5(10): e201

“Why, sometimes I’ve believed as many as six impossible things before breakfast.”

L. Carol, Alice in Wonderland

Fundamental Assumption

• Interesting question• High-quality data• Appropriate analysis• Crisp presentation• Important conclusions

Necessary but not sufficient!

Potential Publication Barriers for Implementation Research

• Small sample size

• Local population

• Lack of “strong” study design

• Failure to distinguish care improvement from generalizable knowledge

• Competition with other types of research

Beyond the Basics

• Establish dialogue with readership

• Frame manuscript to show substance

• Demonstrate impact and relevance

• Be persistent

Dialogue with Readership

• Understand the target journal• Establish continuity with readership• Review relevant articles from target journal

– Journal self-citation– Inappropriate impact factor padding

• World Association of Medical Editors rich discussion

www.wame.org

Medical Care Vision Statement

A Message from the New Medical Care Co-Editors-in-Chief: Building Upon a Legacy of Scholarship and Impact. Medical Care. 2007; 45(1): 1-2.

Health Care Delivery*•Safe•Effective•Patient-centered •Timely•Efficient •Equitable

Health Care Populations

Methods

*From the Institute of Medicine in Crossing the Quality Chasm: A New Health System for the 21st Century, 2001.

Substance: Avoid the Least Publishable Unit

Exceptions• Same message multiple audiences• Repeated methods from large studies

OriginalSubmission

Plagiarism

Duplicate Publication

www.famousplagiarists.com

Substance: Avoid the Least Publishable Unit

Recently submitted manuscript• Concluded intervention equally effective

by race/ethnicity• Reviewer comment: focus on “broader

perspective of the impact of the intervention”

• MedLine search by editor: request fulfilled in another journal

Impact: Advance the Paradigm

• Address the “so what” question– Varies from journal to journal– Policy relevance for Medical Care

• Understand the cutting edge– Health Disparities: from understanding to

change– Risk Adjustment: statistical vs. tangible

meaningT. Kuhn. The Structure of Scientific Revolutions. Chicago: University of Chicago Press, 1962.

“So-What” Issues for Implementation Research

• Linking “basic” human/social sciences with process, structure, and outcomes

• Detangling the confusing picture of practice-change interventions

• Exploring unexpected consequences of emerging paradigms (e.g., P4P)

• Demonstrating value by mixed-methods

“So-What” Issues for Implementation Research

• Measuring the right outcomes

• Testing interventions at interface of patient, doctor, system, and community

• Promoting bi-directional information flow along implementation spectrum

Impact: Advance the Paradigm

• Use bounded speculation

• Focus on the core message

Watson, J. D. and Crick, F. H. 1953. Molecular structure of nucleic acids. Nature 171: 737738. Quoted in Sand-Jensen, 2005. Oikos 116: 723-727

It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material *

Why Some Ideas Survive and Others Die

• Simple: focus on core messagePersonal lesson learned repeatedly

• Unexpected: unfold with mystery• Credible• ...

C. Heath, D. Heath. Made to Stick: Why some ideas survive and others die. New York: Random House, 2007.

Why Some Papers are Published and Others Die

• Simple: focus on core message– Personal lesson learned repeatedly

• Unexpected: unfold with mystery• Credible• ...

C. Heath, D. Heath. Made to Stick: Why some ideas survive and others die. New York: Random House, 2007.

The Value of Persistence

• Rejected manuscript– Poor quality– Poor fit– Bad luck

• Peer-review system– Best available– Not Perfect– Notable shortcomings

• Moniz won Nobel Prize in 1949• Respected peer-review publications touted lobotomy for headaches,

anxiety, delusions, confusion

• 0.20 x 5 = 1• No. journals in Medline, March 2007: 5,197

Take Home Points

• Increasing competition for top journal space

• More than science and traditional aspects of quality– Establish dialogue with readership– Address “cutting-edge” questions– Frame manuscript to show substance– Demonstrate impact and relevance– Be persistent

The real trouble with this world of ours is not that it is an unreasonable world, nor even that it is a reasonable one. The commonest kind of trouble is that is it nearly reasonable, but not quite.

G. Chesterton

I am pleased to inform you that your work has now been accepted for

publication in Medical Care.  

All manuscript materials will be forwarded immediately to the

production staff for placement in an upcoming issue.