madagascar software package technology for reproducible computational experiments sergey fomel...

Madagascar Software Package Technology for

Reproducible Computational Experiments

Sergey FomelBureau of Economic GeologyJackson School of GeosciencesThe University of Texas at Austin

http://rsf.sf.net/

Scientific Software Day 2007

Outline

Past Seismic Imaging

MIT, Industry, Stanford, …

Present Test-driven development

Reproducible computational experiments

Future Asking for help

Automatic testing, parallel processing, graphics, …

Seismic Imaging Computations 45 clusters on the top 500 list

many missing Europe’s most powerful supercomputer

(CGG/IBM) Data size

Gb (2-D; 1980x) Tb (3-D; 1990x) ?? (4-D, 4-C, multi-azimuth, …; 200x)

History of Seismic Imaging

Norbert Wiener (© RLE/MIT) Enders Robinson

Jon Claerbout

SEPlib Software Innovations Universal data file format (RSF)

Data separated from text headers Conceptual regularly gridded N-dimensional

hypercubes

If you feel an urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes. -- Eric S. Raymond, TAUP

Universal graphics format (Vplot) Reproducible research 30+ years legacy

R. Clayton, J. Claerbout, D. Hale, S. Levin, J. Dellinger, S. Cole, D. Nichols, M. Karrenbach, B. Biondi, R. Clapp, …

Data

n1=1000 in=“/path/data.rsf@”n2=500 n3=100 d1=0.001 d2=0.1 o2=1

“Madagascar” Innovations

Open source (GPL) Open community (SourceForge) Open science

Command-line modules Data processing scripts Documentation

Monday Get an idea

Tuesday Implement it

Wednesday Test it

ThursdayCommunicate it

Friday Apply it

Technology Transfer

I

2

3

Three Levels of Technology

Implement it (Tuesday) Command-line modules

C (C++, Fortran-77, Fortran-90, Matlab, Python)

400 modules Test it (Wednesday)

Data processing flows Python/SCons 4,000 tests

Communicate it (Thursday) Papers and reports

LaTeX + Python/SCons 60 papers

Reproducible Computations

Test Driven Development Document computational experiments Old experiments become regression

tests

Within the world of science, computation is now rightly seen as a third vertex of a triangle complementing experiment and theory. However, as it is now often practiced, one can make a good case that computing is the last refuge of the scientific scoundrel. –R. LeVeque

Asking for Help

Open problems Automatic reproducibility testing Parallelization Interfaces Graphics …

Open opportunities Summer students Consultants

Information

http://rsf.sourceforge.net/ Wiki, blog, e-mail lists, forums

RSF/Madagascar School in Austin April 20, 2007 Bureau of Economic Geology

http://rsf.sourceforge.net/RSF_Austin_School_2007