Understanding the New NIH Rigor & Reproducibility Standards

Jill Jividen, PhD Manager of Research Development

UMMS Office of Research jjgoff@umich.edu

April 26, 2016

Agenda • Background/Context • Guiding Principles of Rigor & Transparency • Summary of changes o Application instructions o Review criteria

• Impact • Resources

Background Dr. Oswald Steward Sr. Assoc. Dean for Research, UC-Irvine School of Medicine Director, Reeve-Irvine Research Center http://mediasite.health.uci.edu/Mediasite/Play/4de917befc7b4974b3889d2c270ae1d21d

2003: NINDS funded 3 contracts to replicate promising studies in spinal cord injury (UCI, Miami, Ohio State); over 10 years, 20 promising, high-profile studies were repeated (18 published to date)

Outcome: Only about 10% of the published findings could be replicated; methods sections incomplete or misleading

Steward et al (2012) Replication and reproducibility in spinal cord research. Exp. Neurol. Special Issue, 233, 597-605.

Background Increasing reports of failure to replicate:

Inconsistencies found in

2/3 of studies

Only 6/53 “landmark”

papers could be replicated

Background This “Reproducibility Challenge” noted in research

community and beyond

• Problem across research areas

• Especially in

preclinical research

Background

Why is it so hard to ensure rigor & transparency in reporting science?

Science often viewed as “self-correcting”; immune from reproducibility problems Principle of self-correction true over long term

Factors that “short circuit” self-correction in short term: • Poor training→inadequate exp design, “p-hacking” • Hyper-competitive environment (funding, “high

profile” publishing, P&T) • Publication practices contribute

o Difficulty publishing negative findings o Overemphasis on “exciting, big picture” findings

Collins, FS, Tabak, LA. Policy: NIH Plans to enhance reproducibility. Nature. 27 January 2014.

Background

2012: NINDS convened a workshop for stakeholders. Outcome: Recommendations regarding minimal

requirements to improve rigor involving pre-experiment sample size estimation, randomization, blinding, data handling (data inclusion/exclusion) stopping rules and thorough and transparent reporting.

Nature, 490, 187-191, 2012.

Background

Begley, CG. Nature. 2013; 497 (745): 433-4. PMID 23698428 • Were experiments performed

blinded? • Were basic experiments repeated? • Were all the results presented? • Were there positive and negative

controls? • Were reagents validated? • Were statistical tests appropriate?

Background

Back to Basics

Nature 526, 182–185 (08 October 2015).

Back to Basics

Nature 526, 182–185 (08 October 2015).

Back to Basics

Nature 526, 182–185 (08 October 2015).

Back to Basics

5 requirements for “good” experimental design • Be unbiased • Have high precision • Have a wide range of applicability • Be simple • Have the ability to calculate uncertainty

Good experimental design (and reporting) underlies Rigor & Reproducibility of findings

Back to Basics

• Clarify NIH’s longstanding expectations regarding rigor and transparency and how they would like to see this described in applications

• Raise awareness and begin culture shifts in the scientific community

• Prompt applicants to consider issues that they may have previously downplayed or ignored, which may have a detrimental effect on the quality of the science they produce

NIH Guiding Principles for Rigor & Transparency “…to enhance reproducibility of research findings through increased scientific rigor and transparency.”

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics,” Oct 2015 NIH Regional Seminar.

Back to Basics

• Improve the way that applicants describe their work; provide sufficient information for reviewers

• Demonstrate to public stakeholders that NIH is seriously considering their concerns

• Ensure that NIH is investing in the best science and minimizing unnecessary burden

NIH Guiding Principles for Rigor & Transparency

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics,” Oct 2015 NIH Regional Seminar.

NIH Changes for 2016

NOT-OD-16-011 NOT-OD-16-012

NIH Changes for 2016

Three new required sections to be incorporated in Research Strategy: • Scientific Premise = Significance • Scientific Rigor = Approach • Biological Variables = Approach Incorporated into the 12-page limit = less room for other things START EARLY!

NIH Changes for 2016

A 4th new required section: Authentication of Key Biological and/or Chemical Resources Pre-May 25: Uploaded into “Other Attachments” May 25 on: FORMS-D field for attaching

“Authentication of Key Resources”

NIH Changes for 2016

Key Area Application Instructions

Review Instructions Contribute to Overall Impact Score?

Scientific Premise Research Strategy: Significance

Scored Review Criterion: Significance

Yes

Scientific Rigor Research Strategy: Approach

Scored Review Criterion: Approach

Yes

Consideration of Relevant Biological Variables, such as sex

Research Strategy: Approach

Scored Review Criterion: Approach

Yes

Authentication of Key Biological and/or Chemical Resources

Other Research Plan Attachment

Additional Review Consideration: Acceptable or unacceptable

No

Scientific Premise

All research builds upon prior research.

• The scientific premise for an application is the research that is used to form the basis for the proposal research questions, whether observations, preliminary data, or published literature.

• Describe the general strengths & weaknesses of the prior

research being cited as crucial to support the application. This could include attention to the rigor of previous experimental designs, as well as the incorporation of relevant biological variables and/or authentication of key resources.

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics,” Oct 2015 NIH Regional Seminar.

Scientific Premise

• Explain the importance of the problem or critical barrier to progress

in the field that the proposed project addresses. • Describe the scientific premise for the proposed project, including

consideration of the strengths and weaknesses of published research or preliminary data crucial to the support of your application.

• Explain how the proposed project will improve scientific knowledge, technical capability, and/or clinical practice in one or more broad fields.

• Describe how the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field will be changed if the proposed aims are achieved.

Research Plan Instructions—Significance

Scientific Premise

Significance—Reviewer Instructions • Does the project address an important problem or a critical barrier

to progress in the field? • Is there a strong scientific premise for the project? • If the aims of the project are achieved, how will scientific

knowledge, technical capability, and/or clinical practice be improved?

• How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

Scientific Rigor

• Scientific rigor is the strict application of the scientific

method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results.

• Describe the experimental design and methods proposed and how they will achieve robust and unbiased results. Robust and unbiased results are obtained using methods designed to avoid bias and these results can be reproduced under well-controlled and reported experimental conditions.

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics,” Oct 2015 NIH Regional Seminar.

Scientific Rigor

Research Plan Instructions—Approach

• Describe the overall strategy, methodology, and analyses to be used to accomplish the specific aims of the project.

• Describe the experimental design and methods proposed and how they will achieve robust and unbiased results. Unless addressed separately in the Resource Sharing Plan attachment below, include how the data will be collected, analyzed, and interpreted.

• Discuss potential problems, alternative strategies, and benchmarks for success anticipated to achieve the aims.

• If the project is in the early stages of development, describe any strategy to establish feasibility, and address the management of any high risk aspects of the proposed work.

Scientific Rigor

Approach—Reviewer Instructions • Are the overall strategy, methodology, and analyses well-

reasoned and appropriate to accomplish the specific aims of the project?

• Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?

• Are potential problems, alternative strategies, and benchmarks for success presented?

• If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

Consideration of Relevant Biological Variables

• Biological variables, such as sex, age, weight, and

underlying health conditions, are often critical factors affecting health or disease.

• Biological variables should be factored into research designs, analyses, and reporting in vertebrate animal and human studies. Strong justification from scientific literature, preliminary data or other relevant considerations must be provided for applications proposing to study only one sex.

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics,” Oct 2015 NIH Regional Seminar.

Consideration of Relevant Biological Variables

Research Plan Instructions—Approach

• Explain how relevant biological variables, such as sex, are factored into research designs and analyses for studies in vertebrate animals and humans. – For example, strong justification from the scientific

literature, preliminary data, or other relevant considerations, must be provided for applications proposing to study only one sex.

– Please refer to NOT-OD-15-102 for further consideration of NIH expectations about sex as a biological variable.

Consideration of Relevant Biological Variables

Approach—Reviewer Instructions

• Have the investigators presented adequate plans to address relevant biological variables, such as sex, for studies in vertebrate animals or human subjects?

Authentication of Key Biological and/or Chemical Resources

Since the 1960s, more than 400 widely used cell lines worldwide have been shown to have been misidentified.

A 2011 study of 122 different head and neck cancer cell lines revealed that 37 (30%) were misidentified.

Studies using just two misidentified cell lines were included in 3 grants funded by the NIH, two clinical trials, 11 patents and >100 papers.

Science, 19 Dec 2014.

Authentication of Key Biological and/or Chemical Resources

• The quality of resources used to conduct research is critical to the

ability to reproduce the results. NIH expects that key biological and/or chemical resources will be regularly authenticated to ensure their identity and validity for use in the proposed studies.

• Key biological and/or chemical resources are those that 1. may differ from lab to lab or over time; 2. may have qualities or qualifications that could influence research data; and 3. are integral to the proposed research and may or may not be generated with NIH funds. These include, but are not limited to, cell lines, specialty chemicals, antibodies and other biologics.

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics,” Oct 2015 NIH Regional Seminar.

Authentication of Key Biological and/or Chemical Resources

• Lack of consensus guidelines for many crucial

resources NIH hopes they will be developed by user

community • Transparently report how you will authenticate,

so a consensus can emerge about best practices • Continued discussion and deliberation

Authentication of Key Biological and/or Chemical Resources

UMMS Office of Research website: https://medicine.umich.edu/medschool/sites/medicine.umich.edu.medschool/files/res_oor.NIH_.Authentication%20of%20Key%20Biological%20and%20Chemical%20Resources.pdf

Authentication of Key Biological and/or Chemical Resources

Other Research Plan Sections—1 page

Briefly describe methods to ensure the identity and validity of key biological and/or chemical resources used in the proposed studies. • Key biological and/or chemical resources may or may not be generated with NIH

funds and: – 1) may differ from laboratory to laboratory or over time; – 2) may have qualities and/or qualifications that could influence the research

data; and – 3) are integral to the proposed research. These include, but are not limited to,

cell lines, specialty chemicals, antibodies, and other biologics. • Standard laboratory reagents that are not expected to vary do not need to be

included in the plan. Examples are buffers and other common biologicals or chemicals.

• Reviewers will assess the information provided in this Section. Any reviewer questions associated with key biological and/or chemical resource authentication will need to be addressed prior to award.

Authentication of Key Biological and/or Chemical Resources

Additional Review Consideration • For projects involving key biological and/or chemical

resources, reviewers will comment on the brief plans proposed for identifying and ensuring the validity of those resources.

Reviewer Guidelines

https://grants.nih.gov/grants/peer/guidelines_general/Reviewer_Guidance_on_Rigor_and_Transparency.pdf

Reporting

Research Performance Progress Reports (RPPR) submitted after January 25, 2016, will be expected to emphasize rigorous approaches taken to ensure robust and unbiased results. Rigor should be addressed in the RPPR for any grant that funds research; grants that support other activities do not need to address rigor. This includes non-competing continuation reports (Type 5) for grants reviewed and awarded before the implementation policy. Instructions updated January 25, 2016.

Impact

I. Research Career Development Centers People-based Program Projects Small Business

II. Resource & Related III. Training Fellowships Institutional Training

• Administrative supplements • Conferences • Construction • Instrumentation • Publication support

The review criteria in the Funding Opportunity Announcement (FOA) always prevail!

Policy Applies to: Does Not Apply to:

Adapted from Judy Hewitt, PhD, “Rigor & Reproducibility: Back to Basics, “ Oct 2015 NIH Regional Seminar.

Impact

Challenges for researchers: • Proposal page limit • Cost (no increased budget) • Scope • Loss of innovation?

Resources

http://grants.nih.gov/reproducibility/index.htm

Resources

https://grants.nih.gov/reproducibility/module_1/presentation.html

Resources

https://www.nigms.nih.gov/training/pages/clearinghouse-for-training-modules-to-enhance-data-reproducibility.aspx

Resources

UMMS Office of Research> Research Development Support https://medicine.umich.edu/medschool/research/office-research/research-development-support/rigor-reproducibility PIBS 503: Research Responsibility and Ethics Contact: Dr. Kate Barald (kfbarald@umich.edu) or Kierstin Fiscus (kfiscus@umich.edu) Updated Application Instructions to Enhance Rigor & Reproducibility http://www.nih.gov/research-training/rigor-reproducibility/updated-application-instructions-enhance-rigor-reproducibility NIH Examples of Rigor http://grants.nih.gov/reproducibility/index.htm#resources

Resources

Journal Articles • Becker, JB, et al. “Strategies and methods for research on sex differences in

brain and behavior.” Endocrinology, 2005 Apr; 146(4): 1650-73. • Begley, C. Glenn, and Lee M. Ellis. “Drug development: Raise standards for

preclinical cancer research.” Nature, 483, 531-533 (29 March 2012). • Cahill, Larry, and Dana Aswad. “Sex Influences on the Brain: An Issue Whose

Time Has Come.” Neuron 88, December 16, 2015. • Clayton, JA. “Studying both sexes: a guiding principle for biomedicine.” FASEB

J, 2016 Feb; 30(2): 519-24. • Prinze, Florian, et al. “Believe it or not: how much can we rely on published

data on potential drug targets?” Nature Reviews Drug Discovery, 10, 712 (September 2011).

• Ritz, SA, et al. “First steps for integrating sex and gender considerations into basic experimental biomedical Research.” FASEB J, 2014 Jan; 28(1): 4-13.

• Steward, Oswald, and Rita Balice-Gordon. “Rigor or Mortis: Best Practices for Preclinical Research in Neuroscience.” Neuron, 2014 Nov 5; 84(3): 572-81.

Thank you! Jill Jividen, PhD Sr. Manager of Research Development UMMS Office of Research jjgoff@umich.edu