madagascar software package technology for reproducible computational experiments sergey fomel...
TRANSCRIPT
Madagascar Software Package Technology for
Reproducible Computational Experiments
Sergey FomelBureau of Economic GeologyJackson School of GeosciencesThe University of Texas at Austin
http://rsf.sf.net/
Scientific Software Day 2007
Outline
Past Seismic Imaging
MIT, Industry, Stanford, …
Present Test-driven development
Reproducible computational experiments
Future Asking for help
Automatic testing, parallel processing, graphics, …
Seismic Imaging Computations 45 clusters on the top 500 list
many missing Europe’s most powerful supercomputer
(CGG/IBM) Data size
Gb (2-D; 1980x) Tb (3-D; 1990x) ?? (4-D, 4-C, multi-azimuth, …; 200x)
History of Seismic Imaging
Norbert Wiener (© RLE/MIT) Enders Robinson
Jon Claerbout
SEPlib Software Innovations Universal data file format (RSF)
Data separated from text headers Conceptual regularly gridded N-dimensional
hypercubes
If you feel an urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes. -- Eric S. Raymond, TAUP
Universal graphics format (Vplot) Reproducible research 30+ years legacy
R. Clayton, J. Claerbout, D. Hale, S. Levin, J. Dellinger, S. Cole, D. Nichols, M. Karrenbach, B. Biondi, R. Clapp, …
Data
n1=1000 in=“/path/data.rsf@”n2=500 n3=100 d1=0.001 d2=0.1 o2=1
“Madagascar” Innovations
Open source (GPL) Open community (SourceForge) Open science
Command-line modules Data processing scripts Documentation
Monday Get an idea
Tuesday Implement it
Wednesday Test it
ThursdayCommunicate it
Friday Apply it
Technology Transfer
I
2
3
Three Levels of Technology
Implement it (Tuesday) Command-line modules
C (C++, Fortran-77, Fortran-90, Matlab, Python)
400 modules Test it (Wednesday)
Data processing flows Python/SCons 4,000 tests
Communicate it (Thursday) Papers and reports
LaTeX + Python/SCons 60 papers
Reproducible Computations
Test Driven Development Document computational experiments Old experiments become regression
tests
Within the world of science, computation is now rightly seen as a third vertex of a triangle complementing experiment and theory. However, as it is now often practiced, one can make a good case that computing is the last refuge of the scientific scoundrel. –R. LeVeque
Asking for Help
Open problems Automatic reproducibility testing Parallelization Interfaces Graphics …
Open opportunities Summer students Consultants
Information
http://rsf.sourceforge.net/ Wiki, blog, e-mail lists, forums
RSF/Madagascar School in Austin April 20, 2007 Bureau of Economic Geology
http://rsf.sourceforge.net/RSF_Austin_School_2007