Download - Being Reproducible: SSBSS Summer School 2017
Being Reproducible: Models, Research Objects and R* Brouhaha
Professor Carole Goble, [email protected]
The University of Manchester, UKThe FAIRDOM Association CoordinatorELIXIR-UK Head of NodeCo-lead ELIXIR Interoperability Platform
SSBSS 2017, July 17 2017, Cambridge, UK4th International Synthetic & Systems Biology Summer School
Reproducibility Rampancy
47/53 “landmark” publications
could not be replicated
[Begley, Ellis Nature, 483, 2012]
Retraction
http://www.nature.com/news/misconduct-is-the-main-cause-of-life-sciences-retractions-1.11507
Misconduct is the main cause of life-sciences retractionsZoë Corbyn01 October 2012
Vahan Simonyan,
Center for Biologics Evaluation and Research
Food and Drug Administration
USA
NIH Rigor and Reproducibility
https://www.nih.gov/research-training/rigor-reproducibility
cos.io/top
http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research/
John P. A. Ioannidis How to Make More Published Research True, October 21, 2014 DOI: 10.1371/journal.pmed.1001747
Reproducibility of biological experiments is hard
for in vivo/vitro and for in silico analysis• OS version• Revision of scripts• Data analysis software versions• Version of data files• Command line parameters written on
a napkin• “Black magic” only a grad student
knows
Fix with latest technologies, best practices and willingness
[Keiichiro Ono, Scripps Institute]
The first step is to be FAIRSee the whole of the previous talk…
Record AllAutomate AllContain AllAccess All
Findable (Citable)Accessible (Trackable)Interoperable (Intelligible)Reusable (Reproducible)
designcherry picking data, random seed reporting, non-independent bias, poor positive and negative controls, dodgy normalisation, arbitrary cut-offs, premature data triage, un-validated materials, improper statistical analysis, poor statistical power, stop when “get to the right answer”, software misconfigurations misapplied black box software
reportingincomplete reporting of software configurations, parameters & resource versions, missed steps, missing data, vague methods, missing software
Empirical Statistical ComputationalV. Stodden, IMS Bulletin (2013)
Reproducibility and reliability of biomedical research: improving research practice
https://www.sciencenews.org/article/12-reasons-research-goes-wrong
“When I use a word," Humpty Dumpty said in rather a scornful tone, "it means just what I choose it to mean - neither more nor less.”
Carroll, Through the Looking Glass
re-compute
replicate
rerunrepeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regeneraterevise
recycle
redo
robustness tolerance
verification compliance validation assurance
remix
Scientific publications goals: (i) announce a result(ii) convince readers its correct.
Papers in experimental science should describe the results and provide a clear enough protocol to allow successful repetition and extension.
Papers in computational science should describe the results and provide the complete software development environment, data and set of instructions which generated the figures.
Virtual Witnessing*
*Leviathan and the Air-Pump: Hobbes, Boyle, and the
Experimental Life (1985) Shapin and Schaffer.
Jill Mesirov
David Donoho
“Micro” Reproducibility
“Macro” Reproducibility
Fixivity
Validate
Verify
Trust
Repeatability:“Sameness”Same result1 Lab1 experiment
Reproducibility:“Similarity”Similar result> 1 Lab> 1 experimentwhy the differences?
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
Validate
Verify
Method Reproducibility
the provision of enough detail about study procedures and data so, in theory or in actuality, the same procedures could be exactly repeated.
Result Reproducibility (aka replicability)
obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible
Goodman, et al Science Translational Medicine 8 (341) 2016
Validate
Verify
What are you reproducing?Algorithm vs its script conflation
Methodstechniques, algorithms, spec. of the steps, models
Materialsdatasets, parameters, algorithm seeds
Instrumentscodes, services, scripts, underlying libraries, workflows, ref datasets
Laboratorysw and hw infrastructure, systems software, integrative platformscomputational environment
ProductivityTrack differences
Validate
Verify
Validate
Verify
Recompute By Degrees
Fixivity - Liveness• New/updated/deprecated methods,
datasets, services, codes, h/w• Snapshots
Dependency – Containment• Streams, non-portable data/software, • 3rd party services, supercomputer access,
licensing restrictions….• Locally contained and maintained• External dependencies
Transparency• Blackboxes, proprietary software,
manual steps
Robustness• Bounds of use• Stochastics, non-deterministics,
contexts
https://xkcd.com/797/
Components and Dependencies
Software are typically compound works.
Libraries. Plug-ins. Code fragments.
We are encouraged to reuse and not reinvent
Combining licenses.
License compatibilities
Black boxes
• closed codes
• closed external or cloud services
• method obscurity
• manual steps
[Thanks to Jason Scott]
The Reproducibility Window
all experiments become less reproducible over time….
• Can’t contain everything– Pesky Internet in a Box
• Can’t automate everything– Pesky people intervening
• Can’t fix and fossils everything– Pesky science keeps changing
Results may vary
Bonus slide
At SSBSS Theodor Gescher came up with REALSCI
Robust -many runs
Environment -describe the equipment/OS
Another -done by not your lab
Limits -parameters
Standards -well understood/comprehensible methods
Complete -not cherry picking
Immortal -community supported commodity systems
Mixed Central and Distributed stores: Containment and Dependencies. Upload vs Referencing
In House Stores
External Databases
Publishing services
Model Resources
Mixed Central and Distributed stores: Containment and Dependencies. Upload vs Referencing
In House Stores
External Databases
Publishing services
Model Resources
Migrations into FAIRDOMHubFor long term reproducibility
Shades of ReproducibilityRunning an active instrumentReading an archived record
Are you using hard-wired localhost ids?
WorkflowsSOPs
Containers, cloud services, common services
Markup languages, reporting guidelines and checklists, ontologies, catalogues
Sounds hard….what can I do?
Catalogue
Protocol specs and sharing…
A language for specifying experimental protocols for biological research in way that is precise, unambiguous, and understandable by both humans and computers.
Validation Data
https://fairdomhub.org/sops/203https://fairdomhub.org/investigations/56
Standard Operating Procedures
Quality Control
in situ reproducible models in FAIRDOMmetadata annotation against standardsvalidation, comparison and simulation
SBML Model simulation
Model comparison
Model versioning
Reproducing simulations
[Jacky Snoep, Dagmar Waltemath, Martin Peters, Martin Scharm]
JWS Online
Tracking versi0ns
Tracking model versions smartly
Scharm, M., Wolkenhauer, O., & Waltemath, D. (2015). An algorithm to detect and communicate the differences in computational models describing biological systems. Bioinformatics, btv484
Model simulation in FAIRDOMHubusing JWS Online
A simulation database allows a one-click, live figure reproduction in a FAIRDOM-SEEK
JWS model Excel data file
Dagmar Waltemath, Uni RostockJacky Snoep, Uni Stellenbosch
Simulation Experiment Description Markup Language: XML-based format for encoding simulation setups, to ensure exchangeability and reproducibility of simulation experiments• which models to use in an experiment,• modifications to apply on the models before using them,• which simulation procedures to run on each model,• what analysis results to output,• and how the results should be presented.
FAIRDOMHub Journal ProgrammeMolecular Systems Biology
Model Technical curation forJournals
[Jacky Snoep (Stellenbosch), Dagmar Waltemath, Martin Peters, Martin Scharm (Rostock)]
* store DOI citable supplementary files on FAIRDOMHub** model and data curation*** reproducible clickable figures in papers using SED-ML
CataloguingPackaging
Penkler, G., du Toit, F., Adams, W., Rautenbach, M., Palm, D. C., van Niekerk, D. D. and Snoep, J. L. (2015), Construction and validation of a detailed kinetic model of glycolysis in Plasmodium falciparum. FEBS J, 282: 1481–1511. doi:10.1111/febs.13237
https://fairdomhub.org/investigations/56
DOI: 10.15490/seek.1.investigation.56
Snapshotpreservation
active
18/07/2017 39
An “evolving manuscript” would begin with a pre-publication, pre-peer review “beta 0.9” version of an article, followed by the approved published article itself, [ … ] “version 1.0”.
Subsequently, scientists would update this paper with details of further work as the area of research develops. Versions 2.0 and 3.0 might allow for the “accretion of confirmation [and] reputation”.
Ottoline Leyser […] assessment criteria in science revolve around the individual. “People have stopped thinking about the scientific enterprise”.
http://www.timeshighereducation.co.uk/news/evolving-manuscripts-the-future-of-scientific-communication/2020200.article
Packaging: CombineArchivehttps://sems.uni-rostock.de/projects/combinearchive/
Scharm M, Wendland F, Peters M, Wolfien M, Theile T, Waltemath DSEMS, University of Rostock
zip-like file with a manifest & metadata- Bundling files - Keeping provenance- Exchanging data - Shipping results
Bergmann, F. T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., ... & Olivier, B. G. (2014). COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project. BMC bioinformatics,15(1), 1.
Standards-based metadata framework for bundling (scattered) resources with context and citation
Packaging: Research Objects
http://researchobject.org
Packaging: Research Objects
Publishing Archive
InstitutionalArchive
1.Export2.Exchange
http://researchobject.org
Manifest Construction
Container
Manifest Description
Packaging Platforms: Zip files, BagIt,
Docker, Conda, Singularity
Repositories FAIRDOMHub
Packaging:Research Objects in a nutshell
Different manifest description profiles for different kinds of objects
From Virtual Machines to Executable Containersfor portable execution
• Containers everything required to make a piece of software run is packaged into isolated containers.
• Unlike VMs, containers do not bundle a full operating system - only libraries and settings required to make the software work.
• Efficient, lightweight, self-contained systems• Guarantees that software will always run the same,
regardless of where it’s deployed.
https://www.software.ac.uk/c4rr/ https://biocontainers.pro/
Biocontainers
Use commodity and community systems
Sustained platformsCommunities to drive themTooling and training
Spreadsheets are the Cockroaches of Science
EU FAIR Data Expert Group Consultation
https://github.com/FAIR-Data-EG/consultation/issues
What to know more?Go on a Software or Data Carpentry Course
https://tess.elixir-europe.org
Make software open and reusable
Software Sustainability Institute , http://www.software.ac.uk
Goble, Better Software Better Research IEEE Internet Computing 18(5), (2014 ) DOI: 10.1109/MIC.2014.88
Jiménez RC, Kuzak M, Alhamdoosh M et al.Four simple recommendations to encourage best practices in research software [version 1; referees: 3 approved]. F1000Research 2017, 6:876 (doi: 10.12688/f1000research.11407.1)
Use Common PlatformsGet the licencing right…MATLABMathematica….
Proprietary software
Cloud Centralised Service insitu reproducibility….
Galaxy
FAIRDOMHub + JWS Online
Blackbox vs Whitebox
https://view.commonwl.org/workflows/github.com/ProteinsWebTeam/ebi-metagenomics-cwl/tree/fa86fce/workflows/rna-selector.cwl
Use and document workflowspreferrably a workflow management system, Living Research Objects!
http://commonwl.org/
Workflow repository
Use a workflow – the vision!preferrably a workflow management systempreferrably described using Common Workflow Language
Experimentalworkflows
Event BUS Business Process Management
Taverna Knime Galaxy
WorkflowBPM layer
WorkflowComputationApplicationlayer
Computing resources DatabasesEffector layer
Front-endWeb interface / Monitoring interface
PipelinePilot
FAIRDOM SEEKWorkflow repository
Workflow portal
repository
launch, results
FAIRDOM
[Jean Loup Fallon, Carole Goble]
https://hive.biochemistry.gwu.edu/htscsrs/workshop_2017
Reproducible Pipelines for Robust Regulation
BioCompute Objects
Emphasis on fixing the pipeline so it can be replicated, and on reporting the parameter space
Use an Electronic Lab Notebook
What can you do?
• Follow the 10 RACA Principles
• Take action, be imperfect
• Demand reproducibility in reviews.
• Educate your PIs and supervisors.
[Norman Morrison]
Technological Debt: Appropriate EffortRetrospective Reusability
What are the incentives?
[Garza] [Malone] [Resnik]
Acknowledgements
• David De Roure• Tim Clark• Sean Bechhofer• Robert Stevens• Christine Borgman • Victoria Stodden• Marco Roos• Jose Enrique Ruiz del Mazo• Oscar Corcho• Ian Cottam• Steve Pettifer• Magnus Rattray• Chris Evelo• Katy Wolstencroft• Robin Williams• Pinar Alper• C. Titus Brown• Greg Wilson• Kristian Garza
• Juliana Freire• Jill Mesirov• Simon Cockell• Paolo Missier• Paul Watson• Gerhard Klimeck• Matthias Obst• Jun Zhao• Pinar Alper• Daniel Garijo• Yolanda Gil• James Taylor• Alex Pico• Sean Eddy• Cameron Neylon• Barend Mons• Kristina Hettne• Stian Soiland-Reyes• Rebecca Lawrence• Michael Crusoe
Jon Olav Vik, Norwegian University of Life Science
Maksim ZakhartsevUniversity Hohenheim, Stuttgart, Germany
Alexey KolodkinSiberian BranchRussian Academy of Sciences
Tomasz Zieliński,SynthSys CentreUniversity Edinburgh, UK
Martin Peters, Martin Scharm Systems Biology BioinformaticsUniversity of Rostock, Germany
Web sites
• Force11 http://www.force11.org
• TeSS https://tess.elixir-europe.org
• FAIRDOM http://www.fair-dom.org
• FAIRDOMHub http://www.fairdomhub.org
• Software Carpentry http://software-carpentry.org
• Data Carpentry http://datacarpentry.org
• Software Sustainability Institute http://www.software.ac.uk
• Rightfield http://www.rightfield.org.uk
• FAIRSharing http://www.fairsharing.org
• Common Workflow Language http://commonwl.org/
Reading List (refs also throughout)
• John P. A. Ioannidis How to Make More Published Research True, October 21, 2014 DOI: 10.1371/journal.pmed.1001747
• Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124
• Steven N. Goodman*, Daniele Fanelli and John P. A. Ioannidis, What does research reproducibility mean? Science Translational Medicine 01 Jun 2016: Vol. 8, Issue 341, pp. 341ps12 DOI: 10.1126/scitranslmed.aaf5027
• Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285
• Massimiliano Assante, Leonardo Candela, Donatella Castelli, Paolo Manghi and Pasquale Pagano, Science 2.0 Repositories: Time for a Change in Scholarly Communication, D-Lib Magazine January/February 2015, Volume 21, Number 1/2 , DOI: 10.1045/january2015-assante
• Waltemath, D., Henkel, R., Hälke, R., Scharm, M., & Wolkenhauer, O. (2013). Improving the reuse of computational models through version control.Bioinformatics, 29(6), 742-748.
• Bergmann, F. T., Adams, R., Moodie, S., Cooper, J., Glont, M., Golebiewski, M., ... & Olivier, B. G. (2014). COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project. BMC bioinformatics,15(1), 1.
• Scharm, M., Wolkenhauer, O., & Waltemath, D. (2015). An algorithm to detect and communicate the differences in computational models describing biological systems. Bioinformatics, btv484
• http://www.reuters.com/article/2012/03/28/us-science-cancer-idUSBRE82R12P20120328
• http://www.acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research/