1Lawrence Livermore National Laboratory

Publication Policy Breakout

Peter Olver Matthias Troyer Ron Boisvert Carol Woodward Neil Calkin Judy Borwein Nicolas Limare Ian Mitchell Randy LeVeque

2Lawrence Livermore National Laboratory

Recommend that we put forth a set of best practices for what authors should do for reproducibility

Set of procedures for authors, referees, and editors Put forth a rubric for rating papers that all can use• Individual journals could adapt this as appropriate

Could pull examples from sources where reproducibility is encouraged currently• SIGMOD• IPOL• Ian’s conference (he has a draft set of

recommendations)

3Lawrence Livermore National Laboratory

What would best practices include?

VM (full supplied by authos, reference VMs and partial VMs)• Pros: Can execute anywhere• Cons: Big (IPOL does not allow VMs because of this,

reference and partial will make these submissions smaller), proprietary software is an issue

All source codes• Pros: All source present• Cons: Hard to install and run in general (can specify

compilers and make procedures to help with this) Code excerpts with implementations of relevant

algorithms (ETH requires this, Science requires this – has retracted papers for this)• Pros: Protects development investment, lower barrier

to submit• Cons: Hard to run for testing

Documentation (in and out of code) and instructions for running code

Test suites for submitted code and / or algorithms

4Lawrence Livermore National Laboratory

How to introduce this expectation into the published literature

Invite accepted papers to submit to a reproducibility review• If not reproducible as submitted, ask for more

information to bring them up to the standard Develop a special issue where papers undergo a

reproducibility review (like an editors’ choice issue) Overlays for arXiv, other archives or journals (like a

certification webpage) with links to pointers to “certified” papers

Supplementary journals like SIAM Imaging Science and IPOL

Certifying journals on sites like Romeo or ISI Thompson

5Lawrence Livermore National Laboratory

Other items discussed

Award: we generally all thought awards for excellent papers meeting reproducibility criteria would be good. However, we thought it a bit premature now. Let’s introduce criteria and standards of best practices then introduce awards in 3-5 years.

Who would do reproducibility reviews? Will senior researchers respect the review?• For SIGMOD it has been students and postdocs. • Accepted papers have been asked to go through the

process so generally senior researchers have been OK with process.

How to bring in standards for reproducibility?• Optional but bring in a certification system for those

that meet it• Positive encouragement from referees and editors –

“This is a good paper but it would be better if…”• Key is to ensure editors make referes aware of the

expectation and opportunities

6Lawrence Livermore National Laboratory

More points

Refereeing this will take time. How do you handle that?• A certification level could be one that has a level that

certifies that the paper has the information to be reproducible but this has not been checked.

Results on super computers could be problematic. Here details but not executables could be provided. SIGMOD tested used donated computer time.

Remaining questions:• How to deal with release restrictions form industry and

labs?• What about papers from numerical analysts with small

problems for numerical results?

7Lawrence Livermore National Laboratory

Title

Points