How Do We Improve Research Methods and Reporting?

Shai D. Silberberg NINDS / NIH

DisclaimerOpinions I will voice are not official opinions of NIH!

What are the problems? What happens if we don’t improve research? How can we fix the problems?

Prinz, Schlange and Asadullah

Nature Reviews Drug Discovery 2011; 10: 712-713

43 / 67

Begley and Ellis. Nature 2012, 483:531-533

“scientific findings were confirmed in only 6 (11%) cases.”

Confounding variables

Clever Hans

Problems withresources

Science at the cutting edge

What Are The Causes For Poor Reproducibility?

Lack of transparency in reporting

Deficient experimental procedures

Publication bias

Human nature

Poor reproducibility× × =×

“Human action can be modified to some extent, but human nature cannot be changed.”

Abraham Lincoln

Cooper Union AddressNew York, New York

February 27, 1860

Human Nature

“The moment one has offered an original explanation for a phenomenon which seems satisfactory, that moment affection for his intellectual child springs into existence...

...So soon as this parental affection takes possession of the mind, there is a rapid passage to the adoption of the theory. There is an unconscious selection and magnifying of the phenomena that fall into harmony with the theory and support it, and an unconscious neglect of those that fail of coincidence….

….There springs up, also, an unconscious pressing of the theory to make it fit the facts, and a pressing of the facts to make them fit the theory.”

Journal of Geology, 1897

Thomas Chrowder Chamberlin

The Method of Multiple Working Hypotheses

Chalmers et al., N Engl J Med 1983; 309: 1358-1361

Schulz et al., PLOS Medicine 2010; 7: 1–7

“Randomized trials can yield biased results if they lack methodological rigour.

To assess a trial accurately, readers of a published report need complete, clear, and transparent information on its methodology and findings.”

The CONSORT statement provides guidelines for reporting clinical trials

Among the 35 items included in the CONSORT guidelines are:

How sample size was determinedMethod used to generate the random allocation sequenceMechanism used to implement the random allocation sequenceWho was blinded after assignment to interventions and howLosses and exclusions after randomisation, together with reasons

Ignoring any one of these items can lead to bias

Chavalarias and Ioannidis, J Clin Epid 2010; 63: 1205-1215

“The reliability of a study is determined by the investigator’s choices about critical details of research design and conduct”

(David F. Ransohoff, 2010. J Clin Oncol 28: 698-704)

“Bias is unintentional and unconscious. It is defined broadly as the systematic erroneous association of some characteristic with a group in a way that distorts a comparison with another group…..”

“…..The process of addressing bias involves making everything equal during the design, conduct and interpretation of a study, and reporting those steps in an explicit and transparent way.”

The definition of bias

What about pre-clinical studies?

Malcom Macleod Emily Sena David Howells

Sena et al., JCBFM. 2014; 34: 737-742

Insufficient reporting of methodological approaches is evident for pre-clinical studies

Effect size for studies of FK506 (Tacrolimus) in experimental stroke.

Sena et al., Trends Neurosci 2007; 30: 433-439

The fewer methodological parameters are reported, the greater the apparent efficacy!

Hackam and Redelmeier, JAMA 2006; 14: 1731-1732

Journals:• Cell• Nature• Science• Nature Medicine• Nature Genetics• Nature Immunology• Nature Biotechnology

>500 citations

Inadequate reporting is widespread

“Peer review is the evaluation of work by one or more people of similar competence to the producers of the work.”

Peer Review

Wikipedia

Year

Publications(x106)

The Escalation in Scientific Reporting(Annual PubMed Publications in English)

Publish or perish! Grant support

Impact factor Innovation

Significance Novelty

Hackam and Redelmeier, JAMA 2006; 14: 1731-1732

Journals:• Cell• Nature• Science• Nature Medicine• Nature Genetics• Nature Immunology• Nature Biotechnology

>500 citations

Inadequate reporting is widespread

Amyotrophic lateral sclerosis (ALS)

Death within 5 years of diagnosis Central pathological finding is motor neuron death

3% of cases from gain of function mutations in SOD1 Rodents over-expressing SOD1 recapitulate ALS

2002: Minocycline reported by a number of groups to extend survival of SOD1 mice

2003: Randomized placebo controlled trial (412 patients treated for 9 months)

2007: Results of the trial are published - minocycline found to have a harmful effect on patients with ALS

“In the past five years we have screened more than 70 drugs in 18000 mice across 221 studies, using rigorous and appropriate statistical methodologies. While we were able to measure a significant difference in survival between males and females with great sensitivity, we observed no statistically significant positive (or negative) effects for any of the 70 compounds tested, including several previously reported as efficacious. “

Scott et al., Amyotroph Lateral Scler 2008; 9: 4-15

ALS Therapy Development Institute (ALS TDI)

“…the majority of published effects are most likely measurements of noise in the distribution of survival means as opposed to actual drug effect.“

Scott et al., Amyotroph Lateral Scler 2008; 9: 4-15

2241 SOD1G93A control mice

The probability of seeing an effect by chance alone is significant even with 10 animals per group

Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa

THEODORED . STERLINGUniversity of Cincinnati

“There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs - an "error of the first kind“ - and is published.”

Journal of the American Statistical Association, 1959; 54:30-34

Psychological Bulletin, 1979; 86: 638-641

“For any given research area, one cannot tell how many studies have been conducted but never reported.

The extreme view of the "file drawer problem" is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show non-significant results.”

The “File Drawer Problem” and Tolerance for Null Results

ROBERT ROSENTHALHarvard University

“We evaluated 340 articles included in prognostic marker meta-analyses (Database 1) and 1575 articles on cancer prognostic markers published in 2005 (Database 2).

……..Only five articles in Database 1 (1.5%) and 21 in Database 2 (1.3%) were fully ‘negative’ for all presented results in the abstract and without efforts to expand on non-significant trends or to defend the importance of the marker with other arguments.”

European Journal of Cancer, 2007; 43: 2559 - 2579

SOD1G93A transgenic mice Started at 10 weeks of age i.p. 25 and 50 mg/kg/day 7 animals / group (females) Not randomized “The experimenter was blinded to

the treatment protocol.”

SOD1G93A transgenic mice Started at 5 weeks of age i.p. 10mg/kg/day 10 animals / group (sex?) Not randomized Not blinded

The survival benefit of minocycline in the SOD1G93A mouse model of ALS might be due to small sample size and/or Bias

What happens if we don’t improve research?

We will: Stifle advances in science and medicine Abuse valuable resources Fail current and future generations Lose credibility / public support

“Most people say that it is the intellect which makes a great scientist. They are wrong: it is character.”

Albert Einstein

Lack of transparency in reporting

Deficient experimental procedures

Publication bias

Human nature

Poor reproducibility× × =×

Education

Attentiveness to bias;Good experimental

design

How do we improve research methods and reporting?

RFA-GM-15-006

Training Modules to Enhance Data Reproducibility (R25)

“This FOA will support creative educational activities with a primary focus on developing courses for skills development, specifically, training modules for graduate students, postdoctoral fellows, and beginning investigators designed to enhance data reproducibility.”

Google: NIGMS clearinghouse

Lack of transparency in reporting

Deficient experimental procedures

Publication bias

Human nature

Poor reproducibility× × =×

Review

transparency in

reporting

Education

Attentiveness to bias;Good experimental

design

How do we improve research methods and reporting?

Evaluation of the strength of the data used in support of grant applications

June 20 – 21, 2012Washington Plaza Hotel

Washington DC

R

Editors

ReviewersInvestigators

Funders

“To ease the interpretation and improve the reliability of published results we will more systematically ensure that key methodological details are reported, and we will give more space to methods sections. We will examine statistics more closely and encourage authors to be transparent, for example by including their raw data.”

1. Rigorous statistical analysis

2. Transparency in reporting Journals should have no limit or generous limits on the length of methods sections Journals should use a checklist during editorial processing to ensure the reporting

of key methodological and analytical information to reviewers and readers.

3. Data and material sharing

4. Consideration of refutations

5. Consider establishing best practice guidelines

http://www.nih.gov/about/reporting-preclinical-research.htm

Lack of transparency in reporting

Deficient experimental procedures

Publication bias

Human nature

Poor reproducibility× × =×

Review

transparency in

reporting

Culture

Focus on rigor not

glitter

Education

Attentiveness to bias;Good experimental

design

How do we improve research methods and reporting?

New Biographical Sketch Format Required for NIH and AHRQ Grant Applications Submitted for Due Dates on or After May 25, 2015

The new format extends the page limit for the biosketch

from four to five pages.

Allows researchers to describe up to five of their most

significant contributions to science, along with the

historical background that framed their research.

Investigators can outline the central findings of prior work

and the influence of those findings on the investigator’s

field.

NOT-OD-15-032

“Experiments framed by hypotheses establish the idea of “positive” data versus “negative” data. Positive data are consistent with the hypothesis and negative data falsify the hypothesis. This creates a potential bias to amass positive data, owing to the desire to avoid falsification.”

Lack of transparency in reporting

Deficient experimental procedures

Publication bias

Human nature

Poor reproducibility× × =×

Review

transparency in

reporting

Culture

Focus on rigor not

glitter

Betterreproducibility× × =

Education

Attentiveness to bias;Good experimental

design

How do we improve research methods and reporting?

“It is not the slowness with which conclusions are arrived at that should give satisfaction to the moral sense, but the thoroughness, the completeness, the all-sidedness, the impartiality, of the investigation.”

Thomas Chrowder Chamberlin 1897