andrew davison unic, cnrs facets codejam #2 gif sur...
TRANSCRIPT
![Page 1: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/1.jpg)
Thoughts on simulationproject management
Andrew DavisonUNIC, CNRS
FACETS CodeJam #2Gif sur Yvette, 5th-8th May 2008
![Page 2: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/2.jpg)
Outline
1 Reproducible research, drowning in data and other problems
2 Solutions
3 The real problem: I’m lazy and my brain is too small
4 Sumatra
5 Sumatra++
![Page 3: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/3.jpg)
Reproducible research
”I thought I used the same parameters but I’m gettingdi!erent results”
I can’t remember which version of the code I used to generatefigure 6
Ted Carnevale wants to put the code for that model Ipublished 3 years ago into ModelDB but he can’t reproducethe figures
Why did I do that?
![Page 4: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/4.jpg)
Drowning in data
$ tree bfstdp data | tail -133 directories, 7018 files
photo of lab notebook?
photo of big stack of printouts? laid out on floor?
physically take a file of printouts from my thesis in a honkingbig binder
![Page 5: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/5.jpg)
Outline
1 Reproducible research, drowning in data and other problems
2 Solutions
3 The real problem: I’m lazy and my brain is too small
4 Sumatra
5 Sumatra++
![Page 6: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/6.jpg)
Solutions?
Stage 1
Version control by filename, parameter values in filenames
Lab notebook with printouts stuck in
Stage 2
Parameters, model-definition code, model control code inseparate files
Excel spreadsheet to record parameters used, reasons fordoing each simulation, summary of results of each simulations
Stage 3
Eureka! Version control (versioning of entire tree, not justindividual files)
Keep using spreadsheet, now record svn revision for eachsimulation
But still...
A lot of manual work, easy to forget to check-in changes
![Page 7: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/7.jpg)
Solutions?
Stage 1
Version control by filename, parameter values in filenames
Lab notebook with printouts stuck in
Stage 2
Parameters, model-definition code, model control code inseparate files
Excel spreadsheet to record parameters used, reasons fordoing each simulation, summary of results of each simulations
Stage 3
Eureka! Version control (versioning of entire tree, not justindividual files)
Keep using spreadsheet, now record svn revision for eachsimulation
But still...
A lot of manual work, easy to forget to check-in changes
![Page 8: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/8.jpg)
Solutions?
Stage 1
Version control by filename, parameter values in filenames
Lab notebook with printouts stuck in
Stage 2
Parameters, model-definition code, model control code inseparate files
Excel spreadsheet to record parameters used, reasons fordoing each simulation, summary of results of each simulations
Stage 3
Eureka! Version control (versioning of entire tree, not justindividual files)
Keep using spreadsheet, now record svn revision for eachsimulation
But still...
A lot of manual work, easy to forget to check-in changes
![Page 9: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/9.jpg)
Solutions?
Stage 1
Version control by filename, parameter values in filenames
Lab notebook with printouts stuck in
Stage 2
Parameters, model-definition code, model control code inseparate files
Excel spreadsheet to record parameters used, reasons fordoing each simulation, summary of results of each simulations
Stage 3
Eureka! Version control (versioning of entire tree, not justindividual files)
Keep using spreadsheet, now record svn revision for eachsimulation
But still...
A lot of manual work, easy to forget to check-in changes
![Page 10: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/10.jpg)
Existing tools
Project management tools in NEURON: RCS control ofsimulation projects in a single directory using hoc commandsivdialog, prjnrncmp, prjnrninit, prjnrnci, prjnrnco,prjnrnpr
NeuroConstruct Simulation Browser
![Page 11: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/11.jpg)
Outline
1 Reproducible research, drowning in data and other problems
2 Solutions
3 The real problem: I’m lazy and my brain is too small
4 Sumatra
5 Sumatra++
![Page 12: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/12.jpg)
Automating record-keeping
Core functionality:
Make it easy to record code versions, parameter sets, datafiles.Automate as much as possible, prompt me for the rest
Make it easy to review the history of the project
Make it very easy to repeat a previous simulation and checkthe results haven’t changed
Make it easy to run distributed simulations
Make it easy to run batch simulations (e.g. repeat n timeswith di!erent random seeds, systematic stepping through n-Dparameter space)
Support any command-line driven simulator/arbitraryexecutable
![Page 13: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/13.jpg)
Automating record keeping
Desirable, but non-core functionality:
Help me manage output datafiles, easily preview file contents,visualise as graphs, archive, compare between simulations
Analysis workflow management...
More di"cult - interactive sessions, GUI sessions
![Page 14: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/14.jpg)
Outline
1 Reproducible research, drowning in data and other problems
2 Solutions
3 The real problem: I’m lazy and my brain is too small
4 Sumatra
5 Sumatra++
![Page 15: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/15.jpg)
Sumatraa command-line tool for simulation management/record-keeping
Written in Python (big surprise)
Supports any simulator that allows simulations to be run fromthe command-line, although o!ers extra support for NEURON(e.g. finds the executables automatically, will ensure .modfiles are recompiled if the code has changed)
Requirements:pysvnsqlitedjango
Still alpha software, but I use it and anyone is welcome to tryit (GSL licence?).
![Page 16: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/16.jpg)
smt help
$ smt helpUsage: smt <subcommand> [options] [args]Simulation management tool, version 0.1
Available subcommands:runbatchsetupinfolistcommentrepeatpackagedeleterunserverdebug
![Page 17: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/17.jpg)
smt setup
smt setup [options] NAME REPOS MAINFILE
NAME is the project name.
REPOS is the URL of a Subversion repository with the pathof the project.
MAINFILE is the name of the simulator script that would besupplied on the command line if running thesimulator normally, e.g. init.hoc.
Options:-d [--datapath] PATH : set the path to the directory in which
smt will search for datafiles generatedby the simulation. Defaults to ./Data
-s [--simpath] PATH : set the path to the simulatorexecutable. If this is not set, smtwill assume the simulator is NEURON,and will search for the executables.
![Page 18: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/18.jpg)
smt setup
$ smt setup Test1 https://svn.example.com/repos/myproject smttest.hocCreating table simulation_database_booleanparameterCreating table simulation_database_simrecordCreating table simulation_database_floatparameterCreating table simulation_database_stringparameterCreating table simulation_database_integerparameterCreating table simulation_database_listparameterCreating table simulation_database_tagCreating table simulation_database_parametergroupInstalling index for simulation_database.BooleanParameter modelInstalling index for simulation_database.FloatParameter modelInstalling index for simulation_database.StringParameter modelInstalling index for simulation_database.IntegerParameter modelInstalling index for simulation_database.ListParameter modelInstalling index for simulation_database.ParameterGroup modelSimulation project successfully set up
![Page 19: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/19.jpg)
smt info
$ smt infoName: Test1Repository: https://svn.example.com/repos/myprojectData root: DataMain file: smttest.hocSimulator: /usr/local/nrn6.1/i686/bin/nrniv
![Page 20: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/20.jpg)
smt run
$ smt run smttest1.paramLabel: smttest1.paramTime stamp: 20080502-155932Subversion: No version number provided. Using working copy (revision 136)Writing simulation parameters to smttest1.param_20080502-155932.paramCommand: i686/special smttest1.param_20080502-155932.param smttest.hocloading membrane mechanisms from /home/andrew/tmp/smt_test/i686/.libs/libnrnmech.so>>> Created cell>>> Inserted mechanisms>>> Inserted electrode>>> Set parameters
1>>> Running...
10
Archiving data to file /home/andrew/tmp/smt_test/smttest1.param_20080502-155932.tar.gzData [] [’smttest1.param_20080502-155759.log’,’smttest1.param_20080502-155911.log’, ’smttest1.param_20080502-155932.log’]Deleting [’Data/smttest1.param_20080502-155932.log’]
![Page 21: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/21.jpg)
smt run
$ smt run smttest1.param i_stim=100.0
$ smt run --label=Figure3 --reason=’Test for CodeJam’ smttest1.param
$ smt run smttest1.paramLabel: smttest1.paramTime stamp: 20080502-161150There are local changes to the simulation code.Do you want to commit them (y/n)? [default=’y’]:Please enter a log message: Fixed bug
$ smt run --version=136 smttest1.paramLabel: smttest1.paramTime stamp: 20080502-161508Subversion: Version requested is not the same as the working copy.
Checked out code version 136
![Page 22: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/22.jpg)
smt comment
$ smt comment ’Wow! Nature here we come!’
$ smt comment Figure3_20080502-160909 ’Veni, vidi, vici’
![Page 23: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/23.jpg)
smt list
$ smt listsmttest1.param_20080502-155932smttest1.param_20080502-160650Figure3_20080502-160909smttest1.param_20080502-161150smttest1.param_20080502-161508
$ smt list Figure3Figure3_20080502-160909
![Page 24: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/24.jpg)
smt list
$ smt list --mode=long Figure3-----------------------------------------------------------------Id : Figure3_20080502-160909Reason : Test for CodeJamLabel : Figure3Time_Taken : 0.0612869262695Code_Version : 136Sim_Version : {’date’: ’2007-11-24’, ’version’: ’6.1.1’,
’revision’: ’1894’}Outcome : Veni, vidi, viciTimestamp : 2008-05-02 16:09:09.795710
![Page 25: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/25.jpg)
smt delete
$ smt delete smttest1.param_20080502-1559321 record deleted
![Page 26: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/26.jpg)
smt batch
![Page 27: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/27.jpg)
smt runserver
![Page 28: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/28.jpg)
Browser interface
![Page 29: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/29.jpg)
Outline
1 Reproducible research, drowning in data and other problems
2 Solutions
3 The real problem: I’m lazy and my brain is too small
4 Sumatra
5 Sumatra++
![Page 30: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/30.jpg)
Limitations of the current version
Subversion only
No GUI. Browser interface is read-only (would be nice to beable to lauch simulations via web interface as well)
No support for multi-user, distributed projects
MPI support could be better
No support for post-simulation data analysis
Built with my own preferred workflow in mind - I have no ideaif other people work in the same or a similar way
![Page 31: Andrew Davison UNIC, CNRS FACETS CodeJam #2 Gif sur …neuralensemble.org/media/slides/CodeJam2_Davison_Sumatra.pdfOutline 1 Reproducible research, drowning in data and other problems](https://reader031.vdocuments.site/reader031/viewer/2022022805/5ca61c1c88c9938b538d5cf0/html5/thumbnails/31.jpg)
Proposed redesign
A more modular, loosely-coupled structure...
...to give flexibility and support many di!erent workflows
Support multiple interfaces (command-line, GUI, web)
Support di!erent version control tools (Subversion, Bazaar,...)
Plug-in based analysis workflow
Support multi-user, distributed projects