975!rrr4mor .96!replicability, reproducibility and ... · replicability, reproducibility, and...
TRANSCRIPT
RRR 4 MORReplicability, Reproducibility and Reusability for Model Order Reduction
Jorg Fehr, Jan Heiland, Christian Himpe, Jens Saak
2017-01-12
3rd Workshop on Model Reduction of Complex Dynamical SystemsMODRED 2017
Best Practices for RRR in MOR
An open-access review article, doi.org/bsb2
Jens Saak, [email protected] RRR 4 MOR 2/17
Introduction: MotivationObservation (of an inadequacy):
Numerical experiments gain importance as part of science:
1971 Nitsche: Uber ein Variationsprinzip zur Losung vonDirichlet-Problemen bei Verwendung von Teilraumen, die keinenRandbedingungen unterworfen sind— no numerical experiments reported
1986 Saad & Schultz: GMRES: A Generalized Minimal ResidualAlgorithm for Solving Nonsymmetric Linear Systems— 2/14 pages are devoted to numerical experiments
2010 Chataranbutat & Sorensen: Nonlinear Model Reduction viaDiscrete Empirical Interpolation— consists to more than 30% of numerical examples or reasoningsbased on numerical experiments
Still, there is no standard defining how to present, document andpublish numerical experiments.
Jens Saak, [email protected] RRR 4 MOR 3/17
Introduction: Motivation
Observation (of a chance):
Science builds on previous findings
in theory – using e.g. theoremsand in practice – using established methods
Often, the first step of a new scientific endeavor is the reproduction ofprevious results
Building upon numerical investigations can be incredibly easy
software is easy to share and to modifyhardware is easy to replace
Jens Saak, [email protected] RRR 4 MOR 4/17
Introduction: Motivation
Observation (of a chance):
Science builds on previous findings
in theory – using e.g. theoremsand in practice – using established methods
Often, the first step of a new scientific endeavor is the reproduction ofprevious results
Building upon numerical investigations could be incredibly easy
software is easy to share and to modifyhardware is easy to replace
Jens Saak, [email protected] RRR 4 MOR 4/17
Introduction: Our Aim
Improve Computer-Based Experiments (CBEx):
Create problem-awareness and ensure scientificity
Define terminology
Establish best-practices
Formulate discipline-agnostic practical guidelines
Jens Saak, [email protected] RRR 4 MOR 5/17
Introduction: Our Aim
Improve Computer-Based Experiments (CBEx):
Create problem-awareness and ensure scientificity
Define terminology
Establish best-practices
Formulate discipline-agnostic practical guidelines
Jens Saak, [email protected] RRR 4 MOR 5/17
Outline
1. Best Practices for RRR in MOR
2. Introduction
3. RRRReplicabilityReproducibilityReusabilityThe Road to Reusability
4. Proposed Best Practices
Jens Saak, [email protected] RRR 4 MOR 6/17
RRR
We argue that Good Scientific Practice is all about
Replicability,
Reproducibility, and
Reusability
We discuss
what RRR looks like for CBEx,
how RRR forms a hierarchy, and
how RRR can be implemented up to a desired or feasible level.
Jens Saak, [email protected] RRR 4 MOR 7/17
RRR
We argue that Good Scientific Practice is all about
Replicability,
Reproducibility, and
Reusability
We discuss
what RRR looks like for CBEx,
how RRR forms a hierarchy, and
how RRR can be implemented up to a desired or feasible level.
Jens Saak, [email protected] RRR 4 MOR 7/17
RRR
We argue that Good Scientific Practice is all about
Replicability,
Reproducibility, and
Reusability
We discuss
what RRR looks like for CBEx,
how RRR forms a hierarchy, and
how RRR can be implemented up to a desired or feasible level.
Jens Saak, [email protected] RRR 4 MOR 7/17
RRR : Replicability
Definition
The attribute Replicability describes the ability to repeat a CBEx and tocome to the same (in a numerical sense) results. Sometimes the equivalentterm Repeatability is used for this experimental property.
Replicability is a basic requirement of reliable software as well as of itsresult as it shows a certain robustness of the procedure against
statistical influences
and bias of the observer.
Also, only replicable CBEx can serve as a benchmark to which newmethods can be compared, cf. [Vitek & Kalibera ’11].
Jens Saak, [email protected] RRR 4 MOR 8/17
RRR : Reproducibility
Definition
Reproducibility of a CBEx means that it can be repeated by a differentresearcher in a different computer environment.
This is an adaption of the general concept of Reproducibility
that is key in any science that relies on experiments,
that is a subject in the theory of science, and
which absence in a significant fraction of publications in manyresearch areas has shaped the term Reproducibility crisis in recentyears [Marcus ’13]; cf. also [Collberg, Proebsting, & Warren ’04] onReproducibility in computer science.
Jens Saak, [email protected] RRR 4 MOR 9/17
RRR : Reusability
Definition
In the sphere of CBEx, Reusability refers to the possibility to reuse thesoftware or parts thereof for different purposes, in different environments,and by researchers other than the original authors.
In particular, Reusability enables the utilization of the test setup orparts of it for other experiments or related applications.
Although theoretically, any bit of a software can be reused for differentpurposes, here, Reusability applies only for reproducible parts.
Jens Saak, [email protected] RRR 4 MOR 10/17
The Road to Reusability
Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
• ReplicabilityRequired: Basic Documentation
Recommended: Automation & Testing
• ReproducibilityRequired: Extensive Documentation
Recommended: Availability
• ReusabilityRequired: Accessibility
Recommended: Software Management,Modularity & Licensing
Jens Saak, [email protected] RRR 4 MOR 11/17
The Road to Reusability
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Replicability
← This is a sanity check
Reproducibility
← This makes it science
Reusability
← This is a competitive advantage
Replicability Reproducibility Reusability
Jens Saak, [email protected] RRR 4 MOR 12/17
The Road to Reusability
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Replicability ← This is a sanity check
Reproducibility
← This makes it science
Reusability
← This is a competitive advantage
Replicability Reproducibility Reusability
Jens Saak, [email protected] RRR 4 MOR 12/17
The Road to Reusability
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Replicability ← This is a sanity check
Reproducibility
← This makes it science
Reusability
← This is a competitive advantage
Replicability Reproducibility Reusability
Jens Saak, [email protected] RRR 4 MOR 12/17
The Road to Reusability
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Replicability ← This is a sanity check
Reproducibility ← This makes it science
Reusability
← This is a competitive advantage
Replicability Reproducibility Reusability
Jens Saak, [email protected] RRR 4 MOR 12/17
The Road to Reusability
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Replicability ← This is a sanity check
Reproducibility ← This makes it science
Reusability
← This is a competitive advantage
Replicability Reproducibility Reusability
Jens Saak, [email protected] RRR 4 MOR 12/17
The Road to Reusability
Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Replicability ← This is a sanity check
Reproducibility ← This makes it science
Reusability ← This is a competitive advantage
Replicability Reproducibility Reusability
Jens Saak, [email protected] RRR 4 MOR 12/17
Outline
1. Best Practices for RRR in MOR
2. Introduction
3. RRR
4. Proposed Best PracticesCode Availability SectionBasic DocumentationCode Meta Data
Jens Saak, [email protected] RRR 4 MOR 13/17
Proposed Best Practices
Code Availability Section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Introduced by Nature (doi:10.1038/514536a, doi:10.1038/sdata.2015.4).
An additional section stating the availability of source code.
Code should be shared [LeVeque’13]
Obvious at a glance.
Code Availability SectionThe source code of the implementations used to compute thepresented results can be obtained from:
doi:???????/???????? and is authored by: X Y, A B.
Please contact X Y for licensing information.
Jens Saak, [email protected] RRR 4 MOR 14/17
Proposed Best Practices
Basic Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
README - Every code should have a README file
Title, Version, Release-Date, Summary, Table-of-Contents, ...
RUNME - Every scientific code should have a script reproducing its results
LICENSE - Licensing contents
AUTHORS - List of authors and contributors
CITATION - How to properly cite the code
DEPENDENCIES - required hardware & software
CODE - Code meta-data (see next slide)
More: CHANGELOG, FAQ, INSTALL, TODO
Source file headers:
Project, Authors, Summary, ...
All in plain text!possibly using, e.g. markdown
Jens Saak, [email protected] RRR 4 MOR 15/17
Proposed Best Practices
Basic Documentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
README - Every code should have a README file
Title, Version, Release-Date, Summary, Table-of-Contents, ...
RUNME - Every scientific code should have a script reproducing its results
LICENSE - Licensing contents
AUTHORS - List of authors and contributors
CITATION - How to properly cite the code
DEPENDENCIES - required hardware & software
CODE - Code meta-data (see next slide)
More: CHANGELOG, FAQ, INSTALL, TODO
Source file headers:
Project, Authors, Summary, ...
All in plain text!possibly using, e.g. markdown
Jens Saak, [email protected] RRR 4 MOR 15/17
Proposed Best Practices
Code Meta Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Proposed by: [Katz & Smith’15] We suggest: .ini format
Sample keys:name
shortname
version
release-date
id
id-type
authors
orcids
associated
topic
type
license
license-type
repository
repository-type
languages
dependencies
systems
website
keywords
Jens Saak, [email protected] RRR 4 MOR 16/17
Open For Discussions
Suggestions? / Remarks? Talk to us, e.g. over dinner tonight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Jorg Fehr
Jan Heiland
Christian Himpe
Jens Saak
Thank very much for your attention!
Questions?
Jens Saak, [email protected] RRR 4 MOR 17/17
Open For Discussions
Suggestions? / Remarks? Talk to us, e.g. over dinner tonight. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Jorg Fehr
Jan Heiland
Christian Himpe
Jens Saak
Thank very much for your attention!
Questions?
Jens Saak, [email protected] RRR 4 MOR 17/17
Our Questions
Public code is a major recommendation of our guide.
Can you think of situations in which code publicationis not possible or desired?
Do you have additions to the list of suggesteddocumentation files?
Do you have additions to the code meta data tags?
Do you miss any information?
Jens Saak, [email protected] RRR 4 MOR 1/4
References IS. Chaturantabut and D. C. Sorensen.Nonlinear model reduction via discrete empirical interpolation.SIAM J. Sci. Comput., 32(5):2737–2764, 2010.
C. Collberg, T. Proebsten, and A. M. Warren.Repeatability and benefaction in computer systems research.Technical report, University of Arizona, 2014.Accessed: 2016-09-22.
J. Fehr, J. Heiland, C. Himpe, and J. Saak.Best practices for replicability, reproducibility and reusability of computer-basedexperiments exemplified by model reduction software.AIMS Mathematics, 1(3):261–281, 2016.
D. S. Katz and A. M. Smith.Transitive credit and JSON-LD.Journal of Open Research Software, 3(1), 2015.
R. J. LeVeque.Top ten reasons to not share your code (and why you should anyway).SIAM News, 46(3), 2013.
Jens Saak, [email protected] RRR 4 MOR 2/4
References IIG. Marcus.The crisis in social psychology that isn’t.www.newyorker.com/tech/elements/the-crisis-in-social-psychology-that-isnt,2013.
Code share.Nature, 514:536, 2014.
Ctrl alt share.Scientific Data, 2, 2015.
J. Nitsche.Uber ein Variationsprinzip zur Losung von Dirichlet-Problemen bei Verwendung vonTeilraumen, die keinen Randbedingungen unterworfen sind.Abhandlungen aus dem Mathematischen Seminar der Universitat Hamburg, 36(1):9–15,1971.
Y. Saad and M. H. Schultz.GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems.SIAM Journal on Scientific and Statistical Computing, 7(3):856–869, 1986.
Jens Saak, [email protected] RRR 4 MOR 3/4
References IIIJ. Vitek and T. Kalibera.Repeatability, reproducibility, and rigor in systems research.In Proceedings of the 9th ACM International Conference on Embedded Software, pages33–38, 2011.
Jens Saak, [email protected] RRR 4 MOR 4/4