scec community modeling environment (scec/cme): cyberinfrastructure for earthquake science philip...

SCEC Community Modeling Environment SCEC Community Modeling Environment (SCEC/CME):(SCEC/CME):

Cyberinfrastructure for Earthquake ScienceCyberinfrastructure for Earthquake Science

Philip MaechlingPhilip MaechlingSouthern California Earthquake CenterSouthern California Earthquake Center

University of Southern CaliforniaUniversity of Southern California

SCEC/UseIT Intern ProgramSCEC/UseIT Intern Program

June 6, 2005June 6, 2005

02/13/04 2

People on the SCEC/CME ProjectPeople on the SCEC/CME Project

02/13/04 3

SCEC/CME Researchers• Principal Investigators:

– Tom Jordan (USC)– Bernard Minster (Scripps Institution of Oceanography)– Carl Kesselman (USC/ISI)– Reagan Moore (San Diego Supercomputer Center)

• Research Leads:– Ned Field (USGS) -- Jacobo Bielak (CMU) – Kim Olsen (SDSU) -- Dave O’Hallaron (CMU) – Steve Day (SDSU) -- Ralph Archuleta (UCSB) – Tim Ahern (IRIS) -- Hans Chalupsky (ISI)– Yolanda Gil (ISI)

• Project Manager: – Phil Maechling (USC)

02/13/04 4

SCEC/CME ProjectSCEC/CME ProjectGoal:Goal: To develop a cyberinfrastructure that can support system-level To develop a cyberinfrastructure that can support system-level earthquake science – earthquake science – the SCEC Community Modeling Environment (CME)the SCEC Community Modeling Environment (CME)

Support:Support: 5-yr project funded by the NSF/ITR program under the CISE 5-yr project funded by the NSF/ITR program under the CISE and Geoscience Directoratesand Geoscience Directorates

Start date:Start date: Oct 1, 2001 Oct 1, 2001

SCEC/ITRProject

NSFCISE GEO

SCECInstitutions

IRIS

USGSISI

SDSCInformationInformation

ScienceScienceEarthEarth

ScienceScience

www.scec.org/cme

02/13/04 5

Seismic Hazard Analysis

• Intensity measure: Intensity measure: peak ground peak ground acceleration (PGA)acceleration (PGA)

• Interval: 50 yearsInterval: 50 years

• Probability of Probability of exceedance: 2%exceedance: 2%

Definition:Definition: Specification of the maximum intensity of shaking Specification of the maximum intensity of shaking expected at a site during a fixed time intervalexpected at a site during a fixed time interval

Example:Example: National seismic hazard mapsNational seismic hazard maps (http://geohazards.cr.usgs.gov/eq/)(http://geohazards.cr.usgs.gov/eq/)

02/13/04 6

Seismic Hazard Analysis

• Intensity measure: Intensity measure: peak ground peak ground acceleration (PGA)acceleration (PGA)

• Interval: 50 yearsInterval: 50 years

• Probability of Probability of exceedance: 2%exceedance: 2%

Definition:Definition: Specification of the maximum intensity of shaking Specification of the maximum intensity of shaking expected at a site during a fixed time intervalexpected at a site during a fixed time interval

Example:Example: National seismic hazard mapsNational seismic hazard maps (http://geohazards.cr.usgs.gov/eq/)(http://geohazards.cr.usgs.gov/eq/)

02/13/04 7

Structural fragilityStructural fragility

Risk = Probable Loss (lives & dollars) = Risk = Probable Loss (lives & dollars) =

Hazard Hazard Exposure Exposure Fragility Fragility

Extent & density of built Extent & density of built environmentenvironment

Faulting, shaking, Faulting, shaking, landsliding, liquifactionlandsliding, liquifaction

Risk Analysis: A System-Level ProblemRisk Analysis: A System-Level Problem

02/13/04 8

The FEMA 366 Report “HAZUS’99 Estimates of Annual Earthquake Losses for the

United States”, September, 2000

• U.S. annualized earthquake loss (AEL) is about $4.4 billion/yr.

• For 25 states, AEL > $10 million/yr

• 74% of the total is concentrated in California

• 25% is in Los Angeles County alone

02/13/04 9

Pathway 1: Puente Hills M 7.1 Scenario

PeakGroundAcceleration(% g)

0 - 1212 - 2424 - 3636 - 4848 - 6060 - 72

Los AngelesCounty

DowntownDowntownLALA

02/13/04 10

MantleConvection

ClimateSystemThree Global

GeosystemsAtmosphere

Hydrosphere

Cryosphere

Lithosphere

Asthenosphere

Deep Mantle

Outer Core

Biosphere

Inner Core

Core Dynamo

02/13/04 11

SHA Computational SHA Computational PathwaysPathways

IntensityMeasures

Earthquake Forecast Model

AttenuationRelationship

1

Standardized Seismic Hazard AnalysisStandardized Seismic Hazard Analysis

Ground motion simulationGround motion simulation

Physics-based earthquake forecastingPhysics-based earthquake forecasting

Ground-motion inverse problemGround-motion inverse problem

AWMGroundMotionsSRM

Unified Structural RepresentationFaults Motions Stresses Anelastic model

2

AWP = Anelastic Wave PropagationAWP = Anelastic Wave Propagation

SRM = Site Response ModelSRM = Site Response Model

RDM

FSM

3

FSM = Fault System ModelFSM = Fault System Model

RDM = Rupture Dynamics ModelRDM = Rupture Dynamics Model

Invert

Other DataGeologyGeodesy

4

Physics-basedPhysics-basedsimulationssimulations

EmpiricalEmpiricalrelationshipsrelationships

Improvement Improvement of modelsof models

2

3

1

4

02/13/04 12

Pathway Instantiations

SCEC Community Modeling EnvironmentSCEC Community Modeling EnvironmentA collaboratory for system-level earthquake scienceA collaboratory for system-level earthquake science

Knowledge Base

OntologiesCurated taxonomies,

Relations & constraints

Pathway ModelsPathway templates,

Models of simulation codes

Code Repositories

Data & SimulationProductsData Collections

FSM

RDM

AWM

SRM

Storage

GRIDPathway Execution

Policy, Data ingest, Repository access

Grid ServicesCompute & storage management, Security

DIGITALLIBRARIES

Navigation &Queries

Versioning,Replication

MediatedCollectionsFederated

access

KNOWLEDGEACQUISITION

Acquisition InterfacesDialog planning,

Pathway constructionstrategies

Pathway AssemblyTemplate instantiation,

Resource selection,Constraint checking

KNOWLEDGE REPRESENTATION & REASONINGKnowledge Server

Knowledge base access, InferenceTranslation Services

Syntactic & semantic translation

Computing

Users

02/13/04 13

SCEC/CME Computational Pathway ConstructionSCEC/CME Computational Pathway Construction

A major SCEC/CME objective is the ability to construct and run A major SCEC/CME objective is the ability to construct and run complex computational pathways for SHAcomplex computational pathways for SHA

9000 Hazard Curve files (9000 x 0.5 Mb = 4.5Gb)

Extract IMR

Value

Plot Hazard

Map

Lat/Long/Amp (xyz file) with 3000 datapoints (100Kb)

Calculate Hazard Curves

Gridded Region Definition

IMR Definition

ERF Definition

Probability of Exceedence

and IMRDefinition

GMT MapConfigurationParameters

Define Scenario

Earthquake

Pathway 1 example

02/13/04 14

Example Application of Pathway 1:Scenarios for M 7.4 Southern San Andreas Rupture

Courtesy of Ned Field, USGS, PasadenaCourtesy of Ned Field, USGS, Pasadena

Without soil & basin effects With soil & basin effects

02/13/04 15

SCEC Collaboratory for system-level earthquake sciencefor system-level earthquake science

02/13/04 16

Pathway Comparisons

SCEC/CME computational testbed was used to generate PGV Hazard Maps utilizing Pathway 1 and Pathway 2 data sets and SCSN observed data.

Pathway 1 (All Firm Soil) Pathway 1 (SCEC CVM 3.0) Pathway 2 (Olsen AWM)

PGV data for Northridge from SCSN System(Pathway 0)

02/13/04 17

SCEC IT Challenges

Many geophysical models, computational programs, and data sets and data types.

Large scale simulations, high performance computing and large-scale data management are required in a physics-based approach to earthquake modeling at high spatial-temporal resolution requires.

Communication tools, distributed model development, and computer resource sharing are required by the distributed SCEC collaboration.

02/13/04 18

SCEC/CME Research Areas• Geoscience Research Areas:

– Probabilistic Seismic Hazard Analysis– Anelastic Wave Propagation Modeling– Rupture Dynamics Modeling– Data Inversion

• IT Research Areas:– High Performance Computing– Grid– Digital Library– Knowledge Representation and Reasoning– 4D Data Visualization– Creation of Computational Pathways– Web Services– Data Integration– Data Standards– Community Computational Models

• Outreach and Education:– Undergraduate and Graduate Research Opportunities– Access to non-scientific Users (Emergency Management, Public)

02/13/04 19

Composition Analysis Tool (CAT) Interface

User building a pathway specification from library of components

Errors and fixes generated by ErrorScan algorithm

Earthquake Simulation

SCEC ITR Collaboration

The SCEC

02/13/04 21

Major Earthquakes on the San Andreas Fault, 1680-present

19061906M 7.8M 7.8

18571857M 7.9M 7.9 ~1680~1680

M 7.7M 7.7

02/13/04 22

TeraShake Simulation Area

02/13/04 23

33 researchers, 8 InstitutionsSouthern California Earthquake Center

San Diego Supercomputer CenterInformation Sciences Institute

Institute of Geophysics and Planetary Physics (UC) University of Southern California

San Diego State UniversityUniversity of California, Santa Barbara

Carnegie-Mellon UniversityEXonMobil

02/13/04 24

TeraShake Peak Ground Velocity Maps

NW to SE rupture

SE to NW rupture

02/13/04 25

SCEC/CME Grid InfrastructureSCEC/CME has established grid-based connectivity, job-scheduling, and user and host authentication between SCEC, USC, ISI, SDSC, and PSC.

horizon.sdsc.edu- IBM Blue Horizon- 1152 CPUs- 576GB RAM

SDSC

hpc.usc.edu- IBM Linux cluster- 640 CPUs- 320GB RAM

USC

almaak.usc.edu- SUN Sunfire 15K- 64 CPUs- 256GB RAM

condor.usc.edu- Condor pool- A collection of 320 SUN workstations

SCEC Grid Testbed

epi.usc.edu- SUN E3800- 8CPUs- 8GB RAM

gravity.usc.edu- Linux- 4 CPUs- 4GB RAM

SCEC

sidecar.psc.edu- Linux- 1CPU- 1GB RAM

PSC

pinto.isi.edu- Linux- 2CPUs,500MHz- 380MB RAM

ISI

giis.scec.org/scec-giis.isi.edu- Linux- 2CPUs, 1GHz- 1GB RAM

Current SCEC Grid configuration

02/13/04 26

SCEC Community Library

• Data grid architecture using SDSC Storage Resource Broker– Supports user customizable portals

– Maintains associations between data and metadata

• Current collections contain 1.6 million files (10 terabytes)– 3D ground motion for LA Basin (36

scenarios)

– Rupture Dynamics 4D Wavefield

• http://www.sdsc.edu/SCEC

02/13/04 27

SCEC UseIT Undergraduate Intern Program

02/13/04 28

LA3D GeoWall Software

02/13/04 29

Group Interaction and Collaboration Tools

• Applying communication tools to help with this distributed collaboration.

– Web Sites

– Software Configuration Management Tools

– Document Management Tools

– Bug Tracking Tools

– Data File and Metadata Tools

– Email Lists

02/13/04 30

Hosting of SCEC Community Models• Provide access to SCEC Community Models, possibly

alternative models, and utilities for working with the models.

Community Fault Model (CFM-A)

Community Velocity Model (CVM.3.0)

Community Crustal Motion Map (CMM.3.0.1)

02/13/04 31

Comparisons for 09/03/02 Yorba Linda Earthquake

Data in black, SCEC CVM (FD) in blue, Harvard model (SEM) in red

Validation Exercises for Simulation Codes

Comparison of Dynamic Rupture Models Rupture Test Case Contours

02/13/04 32

Hosting of SCEC Community Codes

• Provide access to SCEC geophysical codes

Pathway 1:

• OpenSHA

Pathway 2:

• Olsen AWM

• CMU Hercules AWM

Pathway 4:

• Synthetic Seismograms

• Fréchet Kernels

02/13/04 33

Supporting and Running Large Scale Simulations

8 Processors (in 2002) 240 Processors (in 2004)

02/13/04 34

Establishment of SCEC Grid InfrastructureSCEC/CME has established grid-based connectivity, job-scheduling, and user and host authentication between SCEC, USC, ISI, SDSC, PSC, and

TeraGrid sites.

horizon.sdsc.edu- IBM DataStar- 1152 CPUs- 576GB RAM

SDSC

hpc.usc.edu- IBM Linux cluster- 640 CPUs- 320GB RAM

USC

almaak.usc.edu- SUN Sunfire 15K- 64 CPUs- 256GB RAM

condor.usc.edu- Condor pool- A collection of 320 SUN workstations

SCEC Grid Testbed

epi.usc.edu- SUN E3800- 8CPUs- 8GB RAM

gravity.usc.edu- Linux- 4 CPUs- 4GB RAM

SCEC

sidecar.psc.edu- Linux- 1CPU- 1GB RAM

PSC

pinto.isi.edu- Linux- 2CPUs,500MHz- 380MB RAM

ISI

giis.scec.org/scec-giis.isi.edu- Linux- 2CPUs, 1GHz- 1GB RAM

horizon.sdsc.edu- IBM Blue Horizon- 1152 CPUs- 576GB RAM

TeraGrid

02/13/04 35

SCEC Digital Library - Providing Data Management Capabilities

• Storage Resource Broker based Digital Library Collection now includes SCEC/PEER Scenario Ground Motion data collection, USC Green Tensors data collection (40TB+ Storage), TeraShake Simulations (40 TB+), and Puente Hills Simulation.

SCEC Community

Library

Select Receiver (Lat/Lon)

OutputTime HistorySeismograms

Select ScenarioFault Model

Source Model

02/13/04 36

SDSC Data Visualization

02/13/04 37

ISI Data Visualization

02/13/04 38

Example SCEC and SCEC/CME IT-oriented ActivitiesExample SCEC and SCEC/CME IT-oriented Activities

• UseIT Intern Program

•Unified Structural Representation

• Ground Motion Prediction

• Communication, Education, Outreach

•OpenSHA Seismic Hazard Analysis

• Earthquake Forecasting

• Unified Structural Representation

• Ground Motion Prediction

• Communication, Education, Outreach

02/13/04 39

Example SCEC and SCEC/CME IT-oriented ActivitiesExample SCEC and SCEC/CME IT-oriented Activities• CyberShake Waveform-based Seismic Hazard Analysis

• Unified Structural Representation

• Earthquake Rupture Dynamics

• Ground Motion Prediction

• Communication, Education, Outreach

•TeraShake 2 Dynamic Rupture-based Simulations

• Unified Structural Representation

• Earthquake Rupture Dynamics

• Ground Motion Prediction

• Communication, Education, Outreach

02/13/04 40

CyberShake Project

Using 3D Synthetic Seismic Waveforms In Seismic Hazard Analysis

Intensity-Measure RelationshipIntensity-Measure Relationship

List of Supported IMTs

List of Site-Related Ind. Params

IMT, IMT, IML(s)IML(s) Site(s)Site(s) RuptureRupture

Prob(IMT IML | Site,Rup)

Attenuation Relationships

Simulation IMRs

exceed. prob. computed using a suite of synthetic seismograms

Vector IMRs

compute joint prob. of exceeding multiple IMTs

(Bazzurro & Cornell, 2002)

Multi-Site IMRs

compute joint prob. of exceeding IML(s) at multiple sites

(e.g., Wesson & Perkins, 2002)

Various IMR types (subclasses)

Gaussian dist. is assumed; mean and std. from various parameters

02/13/04 42

Ruptures in ERF within 200KM of USC

02/13/04 43

CyberShake Computational Elements

• Large (TeraShake Scale) forward calculations for each site.

• Requires calculation of 100,000+ seismogram for each site.

• SCEC/CME Grid-based scientific workflow system required to work at this scale.– Access to distributed computing resources

– Large scale file management

– High performance and high throughput computing.

• TeraGrid allocation awarded for effort– 145K SU (TG-BCS050001N)

02/13/04 44

End