automating real-time seismic analysis through streaming and

Automating Real-time Seismic AnalysisThroughStreamingandHighThroughputWorkflows

RafaelFerreiradaSilva,Ph.D.

http://pegasus.isi.edu

Pegasus http://pegasus.isi.edu 2

Do we need seismic analysis?


USArrayA continental-scale Seismic Observatory

US Array TA (IRIS Service)836 stations

394 stations have online data available

http://ds.iris.edu/ds/nodes/dmc/earthscope/usarray/_US-TA-operational/


The development of reliable risk assessment methods for thesehazards requires real-time analysis of seismic data


So, how to efficiently process these data?


ScientificProblem

AnalyticalSolution

Computational Scripts

Automation

ExperimentTimeline

MonitoringandDebug

ScientificResultModels,QualityControl,

ImageAnalysis,etc.

Fault-tolerance,Provenance,etc.

Shellscripts,Python,Matlab,etc.

EarthScience, Astronomy,Neuroinformatics,Bioinformatics,etc.

DistributedComputing

Workflows,MapReduce,etc.

Clusters,HPC,Cloud,Grid,etc.


What is involved in an experiment execution?


Automate

Recover

Debug

Why Scientific Workflows?

Automates complex, multi-stage processing pipelines

Enables parallel, distributed computations

Automatically executes data transfers

Reusable, aids reproducibility

Records how data was produced (provenance)

Handles failures with to provide reliability

Keeps track of data and files


Taking a closer look into a workflow…

job

dependencyUsuallydatadependencies

split

merge

pipeline

Command-lineprograms

DAGdirected-acyclic graphs

abstract workflow

executable workflow

storage constraints

optimizations


From the abstraction to execution!

stage-in job

stage-out job

registration job

Transferstheworkflowinputdata

Transferstheworkflowoutputdata

Registerstheworkflowoutputdata

abstract workflow

executable workflow

storage constraints

optimizations


Optimizing storage usage…

cleanup jobRemovesunuseddata

abstract workflow

executable workflow

storage constraints

optimizations


Workflow systems provide tools togenerate the abstract workflow

dax = ADAG("test_dax")firstJob = Job(name="first_job")firstInputFile = File("input.txt")firstOutputFile = File("tmp.txt")firstJob.addArgument("input=input.txt", "output=tmp.txt")firstJob.uses(firstInputFile, link=Link.INPUT)firstJob.uses(firstOutputFile, link=Link.OUTPUT)dax.addJob(firstJob)for i in range(0, 5):

simulJob = Job(id="%s" % (i+1), name="simul_job”)simulInputFile = File("tmp.txt”)simulOutputFile = File("output.%d.dat" % i)simulJob.addArgument("parameter=%d" % i, "input=tmp.txt”,

output=%s" % simulOutputFile.getName())simulJob.uses(simulInputFile, link=Link.INPUT)simulJob.uses(simulOutputFile, line=Link.OUTPUT)

dax.addJob(simulJob)dax.depends(parent=firstJob, child=simulJob)fp = open("test.dax", "w”)dax.writeXML(fp)fp.close()

abstract workflow

W h i c h W o r k f l o wManagement System?


PegasusNimrod


…and which model to use?

task-oriented stream-based

files

parallelismnoconcurrency

datatransfersviafiles

heterogeneous executiontaskscanruninheterogeneousresources

streams

concurrencytasksrunconcurrently

datatransfersviamemoryormessage

homogeneous executiontasksshouldruninhomogeneousresources


What does Pegasus provide?

AutomationAutomatespipelineexecutions

Parallel,distributedcomputationsAutomaticallyexecutesdatatransfers

Heterogeneous resourcesTask-orientedmodel

Applicationisseenasablackbox

DebugWorkflowexecutionandjobperformancemetricsSetofdebuggingtoolstounveilissuesReal-timemonitoring,graphs,provenance

OptimizationJobclusteringDatacleanup

RecoveryJobfailuredetection

CheckpointFilesJobRetry

RescueDAGs


…and dispel4py?

AutomationAutomatespipelineexecutions

Concurrent,distributedcomputationsStream-basedmodel

WorkflowCompositionPythonLibraryGrouping(all-to-all,all-to-one,one-to-all)

OptimizationMultiple streams(in/out)AvoidsI/O(sharedmemoryormessagepassing)

MappingSequential

Multiprocessing (sharedmemory)Distributedmemory,messagepassing(MPI)

DistributedReal-time (ApacheStorm)ApacheSpark(Prototype)


Asterism greatlysimplifiestheeffortrequiredtodevelop

data-intensive applicationsthatrunacrossmultiple

heterogeneous resourcesdistributedinthewidearea

Pegasus

sub-workflowdispel4pyworkflow

ASTERISM


Where to run scientific workflows?

High Performance Computing

http://pegasus.isi.edu 19Pegasus

sharedfilesystem

submit host(e.g.,user’slaptop)

There are several possible configurations…

typically most HPC sites

WorkflowEngine

Cloud Computing


objectstorage


High-scalable object storages

Typical cloud computing deployment (Amazon S3,

Google Storage)

WorkflowEngine



local data management

Typical OSG sitesOpenScience Grid

WorkflowEngine

Grid Computing


sharedfilesystem


And yes… you can mix everything!

ComputesiteB

ComputesiteA

object storage

WorkflowEngine


How do we use Asterism to automate seismic analysis?

WORKFLOW


Phase1(pre-process)

Phase2(cross-correlation)

Seismic Ambient Noise Cross-Correlation

Preprocesses and cross-correlates traces (sequences of measurements of acceleration in three dimensions) from multiple seismic stations (IRIS database)

Phase 1: data preparation using statistics for extracting information from the noise

Phase 2: compute correlation, identifying the timefor signals to travel between stations. Infers properties of the intervening rock

Seismic Ambient Noise Cross-Correlation


Distributed computation framework for event stream processing

Designed for massive scalability, supports fault-tolerance with a “fail fast, auto restart” approach to processes

Rich array of available spouts specialized for receiving data from all types of sources

Hadoop of real-time processing, very scalablespout

bolt

http://pegasus.isi.edu 26

Seismic workflow execution

Pegasus

IRISdatabase(stations)

datatransfersbetweensitesperformedbyPegasus

ComputesiteB(ApacheStorm)

ComputesiteA(MPI-based)

input data (~150MB)


output data (~40GB)

Phase1

Phase2

WORKFLOW


Southern California Earthquake Center’s CyberShake

Builders ask seismologists: What will the peak ground motion be at my new building in the next 50 years?

Seismologists answer this question using Probabilistic Seismic Hazard Analysis (PSHA)

286 sites, 4 modelseach workflow has 420,000 tasks


A few more workflow features…


Performance, why not improve it?

clustered jobGroupssmall jobstogethertoimproveperformance

tasksmallgranularity

workflow restructuring

workflow reduction

pegasus-mpi-cluster

hierarchical workflows


What about data reuse?

data alreadyavailable

JobswhichoutputdataisalreadyavailableareprunedfromtheDAG

data reuse


workflow reduction

pegasus-mpi-cluster


workflowreduction

data alsoavailable

data reuse

http://pegasus.isi.edu 31

Handling large-scaleworkflows

pegasus-mpi-cluster

recursion endswhen workflow withonly compute jobsis encountered

sub-workflow

sub-workflow


workflow reduction



Running fine-grainedworkflows on HPC systems…

pegasus-mpi-cluster

HPCSystemsubmit host(e.g.,user’slaptop)

workflow wrapped as an MPI jobAllowssub-graphs ofaPegasusworkflowtobe

submittedasmonolithic jobstoremoteresources


workflow reduction


PegasusAutomate, recover,anddebug scientificcomputations.

Get Started

PegasusWebsitehttp://pegasus.isi.edu

[email protected]

[email protected]

HipChat

Thank You

Questions?

RafaelFerreiradaSilva,[email protected] KaranVahi

RafaelFerreiradaSilva

RajivMayani

MatsRynge

Ewa Deelman

Automating Real-time Seismic AnalysisThroughStreamingandHighThroughputWorkflows

automating real-time seismic analysis through streaming and

Documents