virtual geophysics laboratory (vgl) vgl v1.1 launch ryan fraser, terry rankine, joshua vote, lesley...
TRANSCRIPT
Virtual Geophysics Laboratory (VGL)
VGL v1.1 LaunchRyan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock
February 2013
CSIRO | MINERALS DOWN UNDER FLAGSHIP
Gather data, process it, publish resultsSimple, isn’t it?
bedrock
surficial
mineral
geochemical
geochronologic
hyrdrogeological
Geo-information
geophysical
knowledgedata
Virtual Geophysics Laboratory v1.1 | Ryan Fraser2 |
Hardcopy of dataHardcopy of data
SSH ClientSSH Client
Maths PackageMaths PackageProprietary PackageProprietary Package
Let’s map it out for Geophysics …
Transform to a regular grid
Transform to a regular grid
Crop data to area of interest
Crop data to area of interest
Reformat data for processingReformat data for processing
Upload data to HPC
resource
Upload data to HPC
resource
Configure job and start
processing
Configure job and start
processing
Download results
Download results
Get handed field data
Get handed field data Visualise dataVisualise data
Virtual Geophysics Laboratory v1.1 | Ryan Fraser3 |
The Past• Compile raw data using proprietary FORTRAN
• Also use other proprietary software • Transform to a regular grid using more software
• MATLAB, Intrepid, ER Mapper, ESRI ArcGIS, QGIS, etc• Crop data spatially
• eg: everything in Victoria• Transform data into a file format that can be read by proprietary
scientific code• This is usually done with some handwritten python or c• There is no version control, code is often rewritten / redone
• Upload data to HPC• Manually enter input parameters/start job• Where did my results go, how do I visualise?
Virtual Geophysics Laboratory v1.1 | Ryan Fraser4 |
There seems to be a problem…
• Reproducibility – there is none• What was the input of your model?• What transformations occurred?
• It’s a manual process• Time consuming• Error prone
• Expensive • Licensing costs• Effort/time
Virtual Geophysics Laboratory v1.1 | Ryan Fraser5 |
Scientific workflow – Virtual Geophysics Laboratory (VGL)• Automated - Simplify access and integration• Reproducible process• Open access (less expense)• Allow scientists to focus on the science and not the tech• Exploiting the Cloud and HPC• Operational or education tool
• Automates and massively expands geophysics computational capacity via the Cloud
Virtual Geophysics Laboratory v1.1 | Ryan Fraser6 |
V(what)GL
• VEGL – Virtual Exploration Geophysics Laboratory• One primary science collaboration• One primary workflow• One collection of geophysical data sets
• VGL - Virtual Geophysics Laboratory• NeCTAR funded activity• Collaboration with multiple partners (CSIRO, NCI, GA, UQ, Monash, ANU)• Supporting multiple workflows• New data types and collections• New Use – Not just exploration.
DoneDone
Virtual Geophysics Laboratory v1.1 | Ryan Fraser7 |
Hardcopy of dataHardcopy of data
SSH ClientSSH Client
Maths PackageMaths PackageProprietary PackageProprietary Package
Let’s map it out…
Transform to a regular grid
Transform to a regular grid
Crop data to area of interest
Crop data to area of interest
Reformat data for processingReformat data for processing
Upload data to HPC resource
Upload data to HPC resource
Configure job and start
processing
Configure job and start
processing
Download results
Download results
Get handed field data
Get handed field data Visualise dataVisualise data
Virtual Geophysics Laboratory v1.1 | Ryan Fraser8 |
…to thisVirtual Geophysics LaboratoryVirtual Geophysics Laboratory
Build “science” from existing
libraries
Build “science” from existing
libraries
Run jobRun jobCollect and
publish resultsCollect and
publish results
Discover raw data
Discover raw data
Select spatial bounds
Select spatial bounds
Virtual Geophysics Laboratory v1.1 | Ryan Fraser9 |
Virtual Geophysics Laboratory v1.1 | Ryan Fraser10 |
Introducing The Virtual Geophysics Laboratory
Powered by the Spatial Information Services StackCommon Platform
MarineEnvironment, Water
Groundwater GeologyGeophysics
Virtual Geophysics Laboratory v1.1 | Ryan Fraser11 |
VGL: Opportunities
• Free for research use• Supporting multiple workflows• Model Registry (3D) – researcher contributed models• New Scientific Codes – Underworld, eScript, UBC, Airborne EM inversion
codes + others (relatively “easy” to add)• Data sets from GA: National Airborne Geophysical DB including
– Gravity, Radiometric, AEM, Magnetics– Data sets from State Surveys
• Broad application and generic implementation• Scalable Infrastructure (Cloud)
Virtual Geophysics Laboratory v1.1 | Ryan Fraser13 |
What’s left?
• More data, more codes, more workflows!• BYO Resource (cloud allocation)• Confidential Data• Pawsey Centre, NCI, commercial clouds
14 | Virtual Geophysics Laboratory v1.1 | Ryan Fraser
Thank you
Ryan FraserResearch Manager
t +61 8 6436 8760e [email protected] siss.auscope.org
VGL is Power by:
VGL acknowledges funding from the NeCTAR project http://www.nectar.org.au NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund.
VGL is a collaboration between:
VGL is supported by: University of Queensland, Monash University and the Australian National University
VGL Launch Event is sponsored by:
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote16 |
Introducing The Virtual Geophysics Laboratory
No Installation Necessary!
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote17 |
Don’t Compute on your Laptop
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote18 |
Flexibility
What I want to do ≠
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote19 |
Where I want to do itHow I want to do it ≠
Provenance
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote20 |
Don’t be this guy...
To Quickly Recap
• VGL is a different kind of workflow engine...
• No installation necessary• Doesn’t process on your machine• Very flexible• Records results and the process
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote21 |
Time for a live demo...
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote22 |
VGL is Power by:
VGL acknowledges funding from the NeCTAR project http://www.nectar.org.au NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund.
VGL is a collaboration between:
VGL is supported by: University of Queensland, Monash University and the Australian National University
VGL Launch Event is sponsored by:
http://vgl.auscope.org
Data discovery
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote24 |
Layers discovered via remote registries
Layers consist of numerous remote data services
Data discovery
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote25 |
Some data services support subsetting
Some data services support reformattinge.g. CSV, NetCDF, GeoTIFF
Data discovery
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote26 |
Some data is only registered with flat files
Data processing
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote27 |
A variety of different scientific codes are already available in the form of “Toolboxes”
Data processing
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote28 |
Further input files can be uploaded.
Input files are passed directly into the cloud
Data processing
Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote29 |
The steps so far have been building an environment to run a processing script
...or build from existing templates
Either write your own...
Managing results - provenance
Presentation title | Presenter name30 |
All of a job’s outputs are also accessible Each job has a lifecycle that
can be managed
Successful jobs can have their entire process captured in a ISO 19115 ‘provenance record’
VGL is Power by:
VGL acknowledges funding from the NeCTAR project http://www.nectar.org.au NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund.
VGL is a collaboration between:
VGL is supported by: University of Queensland, Monash University and the Australian National University
VGL Launch Event is sponsored by:
http://vgl.auscope.org