Anatomy of a Climate Science-centric Workflow
Harinarayan Krishnan, CAlibrated and Systematic Characterization, Attribution,
and Detection of Extremes (CASCADE Team)
Kevin Bensema, Surendra Byna, Soyoung Jeon, Karthik Kashinath, Burlen Loring, Pardeep Pall, Prabhat, Alexandru Romosan, Oliver Ruebel, Daithi Stone, Travis
O'Brien, Christopher Paciorek, Michael Wehner, Wes Bethel, William Collins
Challenges
• Scale of data already at TBs and will only grow larger.
• Processing Three to Six hours of intervals frequently.
• Foci now is on High resolution 1/4th to 1/8th degree. Extensible to higher.
• High resolution and high frequency analysis add several orders of magnitude.
Proposed Strategy
• Identification of use cases, extraction of common computational algorithms, scaling & optimization of current work.
• Template workflow configurations of common use cases.
• Abstraction of services to HPC environments.
• Easy to use archiving, distribution, and verification strategies.
• Standardization of parallel work environment.
What it is/What it is not
• What it is not— Not a general workflow— Not a general infrastructure – Balancing between
performance & exploratory science.• What it is
—…
For Example:
t = cascade.Teca()
t['filename'] = ‘myfile’
writer = cascade.Writer(cascade.ESGF)
writer[‘input’] = t[‘out’]
n = workflow.NERSC(<resources>, writer)
n.execute()
Note: Active Work in progress & ongoing…
Start (Stage Data)
Schedule Job (Time|
Log)
Load & Run Teca task
(Time|Log)
Verify & Validate
Load & Run Teca task
(Time|Log)
Verify & Validate
Load & Run Teca task
(Time|Log)
Verify & Validate
Publish using ESGF node
Write & Distribute to
ESGF
Record Workflow
Load & Run Teca task
(Time|Log)
Verify & Validate
Load & Run Teca task
(Time|Log)
Verify & Validate
What it is/What it is not
• What it is not— Not a general workflow— Not a general infrastructure – Balancing
between performance & exploratory science.• What it is
— A highly customized climate-centric API (Zonal Mean Averages, GEV, etc…)
— Workflow – Verification/Validations, Job scheduling, Staging, Deployment, etc…
—Modules – Performance & Timing Support, Calendar Support, etc…
— Template workflows
Climate Science-centric Workflow
• Workspace – A collaboration environment to share, track documents, visualize status, update issues.
• One-on-one – Identify use cases that require implementing new features or scaling & performance optimization of existing ones.
• Software tools – Development and Deployment of algorithms & software packages as well as building & maintaining packages for target environments.
• Workflow components – Connecting it all together.
Communication Infrastructure
Quick Note: Software Environment
• Infrastructure - cascade.lbl.gov/esg02.nersc.gov
• Confluence – Portal to publish and collaborate with team members
• Jira – Bug & Issue tracking portal.
• CDash/Jenkins – Infrastructure to report status of software build & regression tests.
• BitBucket – Main software repository.
• ESGF service – Service for distribution of data generated by CASCADE.
CASCADE Team
• Detection & Attribution Team – Characterization, detection, and attribution of simulated and observed extremes in a variety of different contexts -- Analysis Algorithms
• Model Fidelity – . Evaluation and improvement of model fidelity in simulating extremes
• Statistics – Development of statistical frameworks for extremes analysis, uncertainty quantification, and model evaluation
• Formulation of highly parallel software for analysis and uncertainty quantification of extremes
Analysis Infrastructure Tasks
• Development of new climate-centric algorithms and evaluation of current ones. Implement scalable, parallel versions as needed.
• Performance analysis and data management.
• Deployment and Maintenance on HPC environments.
• Creating a standardized environment – Provide same execution environment on all deployed platforms, and seamless bridges different technologies (Python <-> R).
• User Support.
Detection & Attribution
• Single Program Multiple Data SPMD scripts – refactoring current algorithms to work in parallel.
• Distribution/Staging – Functionality to distribute data generated through ESGF also stage data at NERSC.
• TECA – Active development of Parallel Toolkit for Extreme Climate Analysis.
• Teleconnections – Ensemble analysis & software solutions to investigate of frequency of teleconnection events.
Model Fidelity
Model Fidelity
• ILIAD workflow—The parallelization of the generation of initial
conditions.—Dynamic Building, Compilation & Execution of
CESM.—Module verification – monitor execution status
& successful completion.—Module for automation of archiving of output
(initial conditions, namelist files, CESM output).
• DepCache – External tool for speeding up execution of Python libraries.
Statistics
• Integration of Statistical Algorithms – Working to deploy relevant statistical algorithms within CASCADE framework.
• Parallelization – Scaling statistics scripts to work in a parallel environment.
• llex Installation – Generalized Extreme Value Analysis & Peaks Over Threshold statistical analysis algorithms (Developed by Stats team members)
Software Suite
• Python environment— IPython, mpi4py, numpy, …—CDAT-Core (cdms2, cdtime,…)—Rpy2 (Python-R bridge)
• R environment—extRemes, ismev—Llex – GEV & POT (Dr. Chris Paciorek’s package)—pbdR - pbdMPI, pbdSLAP, pbdPROF, pbdNCDF (ORNL)
• TECA – parallel toolkit developed at LBNL (TC, ETC, AR)
- Prototype deployment at NERSC (module load cascade)
- Transitioning maintenance of NERSC ESGF Node to CASCADE analysis group.
Workflow Infrastructure
• Unified Workflow Service— Load balanced services that
handle job Scheduling, Validation & Verification, Fault Tolerance
• Core Modules— Calendar support— Data Reduction Operations
(Sum, Max, Min, Average, etc…)
— I/O services (Parallel Read/Write)
— Threading/MPI wrapping (Map|Foreach)
Additional Services
• MPO – A tool for recording scientific workflows, Developed by General Atomics & LBNL.
• Tigres – Template Interfaces for Agile Parallel Data-Intensive Science, Developed by Advanced Computing for Science Group at LBNL.
• ESGF – Support for automated distribution through ESGF installation.
Modules & API
• CoreModule —Timing, Logging—Standard definition of parameter inputs &
outputs—All modules are inherently Workflows of one.— implicit connectivity of workflow
• BaseAPI (Pythonic)—__getitem__,__setitem__: param[“input”] = val—cascade_static_{param|output}_spec:
{name, value, type, user_defined}—cascade_execute – core execution function
Example Workflow
• Example use case: Running a single module
^^^^^^^^^^^^^^
t = Teca() # Where teca is a derived class of CascadeBase
filename = 'myfile’
t['filename'] = filename
t.execute()
^^^^^^^^^^^^^^^^^
t1 = Teca() # Where Teca is a derived class of CascadeBase
t2 = TecaAnalysis() # Where TecaAnalysis is a derived class of CascadeBase
t2['inputdata'] = t1['outputdata'] # Note, this establishes a link
t2.execute()
Proposed Workflow
t1 = Teca()
t2 = TecaAnalysis()
t3 = TecaAnalysis()
s = Diff()
t2['inputdata'] = t1['outputdata’]
t3[‘inputdata’] = t1[‘outputdata’]
s[‘inputdata1’] = t2[‘outputdata’]
s[‘inputdata2’] = t3[‘outputdata’]
s.write(‘prefix’, ‘file’)
s.execute()
t1
t2 t3
S
output
Schedule Job
Stage Data
Perform MPI-based task
Validate/Verify
Recap: Anatomy of Climate Science-centric Workflow
• Software Environment – Development, Deployment, and Maintenance
• Custom Use Case Support for D&A, Model Fidelity, and Statistics team needs.
• Software Suite – Scaling, Parallelism, Performance Management, Software Services (Python, R, TECA)
• Workflow Development – Thin Client & Workflow service, Module development, Optimization (Data Movement, Workflow execution), Provenance.
Thanks
• Questions?