challenges for investigators generating reproducible research results · 2018. 3. 12. ·...
TRANSCRIPT
Challenges for investigators –
generating reproducible research results
University of Minnesota, 3/2018
John P.A. Ioannidis, MD, DSc
C.F. Rehnborg Chair in Disease Prevention
Professor of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics
Stanford University
Co-Director, Meta-Research Innovation Center at Stanford (METRICS)
Goodman, Fanelli, Ioannidis. Science Translational Medicine 2016
Different types of reproducibility
• Reproducibility of methods: the ability to understand or repeat as
exactly as possible the experimental and computational procedures.
• Reproducibility of results: the ability to produce corroborating
results in a new study, having followed the same experimental
methods.
• Reproducibility of inferences: the making of knowledge claims of
similar strength from a study replication.
Scientific discovery has become a boring nuisance: 96% of
the biomedical literature claims significant results (and the
vast majority of them say they are novel)
Chavalarias, Wallach, Li, Ioannidis, JAMA 2016
Typical recipe of research practices
• Solo, siloed investigator
• Small sample size studies
• Cherry-picking of one/best hypothesis
• Post-hoc
• P<0.05 is enough
• No registration
• No data sharing
• No replication
fMRI studies: only very small
studies discover a lot of foci
David et al, PLoS ONE 2013, Ioannidis et al Trends in Cognitive Sciences 2014
Szucs and Ioannidis, PLoS Biology 2017
Empirical studies on fields where
replication practices are common suggest
that most of the initially claimed
statistically significant effects in
observational associations are false
positives or substantially exaggerated
Candidate genes replicated through GWAS:
replication rate = 1.2%
Ioannidis, Tarone, McLaughlin, Epidemiology 2011
Prinz et al., Nature Reviews Drug Discovery 2011
Failed replication in preclinical
research
Replicated: only 6 of 53 landmark studies for
Amgen oncology drug target projects
• “The failure to win “the war on cancer”
has been blamed on many factors, … But
recently a new culprit has emerged: too
many basic scientific discoveries… are
wrong.”
Begley et al. Nature 2012
Begley and Ioannidis, Circulation Research 2015
Begley and Ioannidis, Circulation Research 2015
Potential solutions to improve
reproducibility
• Some solutions have already worked in specific fields and may need to be considered in other fields as well
• Other solutions are more speculative
• Empirical evidence as to their efficacy is needed
• Seemingly effective solutions may also have collateral damages
• Do no harm
Ioannidis, PLoS Medicine 2014
Large-scale collaboration and adoption of
replication culture
Replication – by whom?
• Same investigators
• Different investigators of the same school
• Different investigators of competing theories/hypotheses
• Combinations of the above
• Open to the wide public
Levels of registration
• Level 0: no registration
• Level 1: registration of dataset
• Level 2: registration of protocol
• Level 3: registration of analysis plan
• Level 4: registration of analysis plan and raw data
• Level 5: open live streaming
Sharing
data –
who,
when,
and how?
Doshi, Goodman,
Ioannidis, TiPS
2013
Science, December 2, 2016
Begley and Ioannidis, Circulation Research 2015
Modeling and modeling plus
experimentation
Grimes
et al,
JRSOS
2018
Re-engineering the reward system
Ioannidis and
Khoury, JAMA 2014
Understand and align interests of stakeholders
Concluding comments
• Reproducibility has different meanings across
different scientific disciplines but eventually all of
them aim at enhancing our trust in scientific findings
• There are many possible interventions that may
improve the efficiency of research practices and the
reproducibility of the scientific literature
• Empirical meta-research would be useful not only to
assess the prevalence of problems of reproducibility,
but also to assess the effectiveness and potential
harms of interventions that try to make research
more reproducible.
Daniele Fanelli
Steve Goodman
Shanil Ebrahim
Despina
Contopoulos-
Ioannidis
Georgia Salanti
Chirag Patel
Lars Hemkens
Ann Hsing
Lamberto Manzoli
Maria Elena Flacco
George Siontis
Denes Szucs
Kostas Siontis
Vangelis Evangelou
Kristin Sainani
Muin Khoury
Orestis Panagiotou
Florence Bourgeois
Special thanks
Joseph LauMalcolm MacLeodMarcus MunafoDavid AllisonJosh WallachFotini KarassaAthina TatsioniEvi NtzaniIoanna TzoulakiDemos KatritsisNikos PatsopoulosFainia KavvouraBrian NosekVictoria StoddenΕle ZegginiBelinda BurfordKostas TsilidisJodi Prochaska
Special thanks
Charitini StavropoulouEvropi Theodoratou Nikos PandisHuseyin NaciVanesa BellouΑntony DoufasLazaros BelbasisChris DoucouliagosStelios SerghiouAnna ChaimaniFotini ChatzinasiouStephania PapatheodorouFlorian NaudetTom HardwickePerrine JaniaudIoana-Alina CristeaShannon BrownleeVikas SainiMatthias EggerPatrick BossuytAndre UitterlindenDoug AltmanDeb ZarinKatherine Flegal
Special thanks
Shanthi Kapaggoda
Ewoud Schuit
Stefania Boccia
David Chavalarias
Jennifer Ware
Viswam Nair
Stephan Bruns
Dorothy Bishop
Tom Trikalinos
Kristina Sundquist
Johanna Int’hout
Kevin Boyack
Brett Thombs
Raj Manrai
Nazmus Saquib
Elizabeth Iorns
Abraham Verghese
Euan Ashley
Special thanks