Computational Challenges inBig Data Assimilation withExtreme-scale Simulations

Takemasa MiyoshiRIKEN Advanced Institute for Computational Science

Takemasa.Miyoshi@riken.jp

May 1, 2013, BDEC workshop, Charleston, SC

With many thanks toY. Sato (JMA),

UMD Weather-Chaos group,Data Assimilation Research Team

Data Assimilation (DA)

Numerical modelsObservations

©Vaisala

Data assimilation best combines observations and a model, and

brings synergy.

Data Assimilation

DA has an impact.JMA operational system under development

Using the same NWP model and observations.DA matters!

OBSOBS

FCSTFCST

Miyoshi and Sato (2007)

SV w/ 4D-Var LETKF

Expanding collaborations

WRF

ROMS

AFES

CFES

Mars GCM

MRI Chem

U.Tokyo Aerosol

WRF-ROMS

JMA MSM

JMA GSM

GFSOFES

CPTEC Brazil

:Existing

:Possible future expansion

JAMSTEC Chem

SPEEDY CO2

CAMMOM

LETKFLocal Ensemble Transform Kalman Filter

Atmosphere

Ocean

Chemistry

Numerical Weather Prediction (NWP)

time

Forecast

Observation

Analysis

Observation

True atmosphere (Unknown)

Analysis

Forecast

Analysis

Model Simulation

Global Observing System

Surface station

Weather balloon

AircraftSatellite

Ship

Buoy

Radar

Collecting the data

World’s effort! (no border in the atmosphere)

Collecting the data

We consider the evolution of PDFObs.Analysis ensemble mean

T=t0 T=t1 T=t2

Analysis w/ errors FCST ensemble mean1

)( 111 −≈

m

Tft

ftf

tXXP δδ

R

An approximation to KF with ensemble representations

Flow chart of DA

Simulation

DA

Initial State

Simulated State

Observations

(Best estimate)

PDF represented by an ensemble

Sim-to-Obsconversion

Sim-minus-ObsDA

Flow chart of DA

Simulation

DA

Initial State

Simulated State

Observations

(Best estimate)

PDF represented by an ensemble

Sim-to-Obsconversion

Sim-minus-Obs

Broad-sense DA

Data size in NWP

0

50

100

150

200

250

300

350

Size[GB/Month]

Size[GB/Month]

ObservationsSimulated State

DA~2GB/6h(~1TB/6h)

Courtesy of JMA

28-km global mesh ~3GB100 members for PDF ~300GB7 time slots ~2TB

Extreme-scale Simulation:3.5-km global mesh ~400GB100 members ~40TB7 time slots ~300TB

~2TB/6h(~300TB/6h)

Future: new satellites,world’s radar data, etc.

Flow chart with current data size

Simulation

DA

Initial State

Simulated State

Observations

(Best estimate)

Sim-to-Obsconversion

Sim-minus-Obs

~300GB (100 members, 1 time level)

~2TB (100 members, 7 time levels)~300GB(100 members, 1 time level)

~2GB

~2TB (100 members, 7 time levels)

~200GB(100 members) ~200GB (100 members)

~300GB(100 members, 1 time level)

Flow chart with exa-scale data size

Simulation

DA

Initial State

Simulated State

Observations

(Best estimate)

Sim-to-Obsconversion

Sim-minus-Obs

~40TB (100 members, 1 time level)

~300TB (100 members, 7 time levels)~40TB(100 members, 1 time level)

~1TB

Challenge in global data sharing among

weather services

~300TB (100 members, 7 time levels)

~100TB(100 members) ~100TB (100 members)

I/O intensive!Repetitions of I/O between separate programs

~40TB(100 members, 1 time level)

Strategy for fast I/O

It would be ideal to write files to RAM or fast-access memory device (~1PB required)

I/O intensive!Repetitions of I/O between separate programs

Computational challenge:

A strategy:

0

20

40

60

80

100

120

140

Wall clock time (min.)

Timing of SPEEDY-model experimentsShared drive RAM

Using a Linux cluster (4 nodes, 32 cores)

2-month DA cycles Experiments with an

intermediate atmospheric model (SPEEDY model)

Accelerationdue to RAM access

An experiment:

Almost pure computational time

File access

How about parallel processing?

Sim-to-Obsconversion

Simulation

Member 1 Member 100…

Simulation

DA

Sim-to-Obsconversion

~1.4TB ~1.4TB

~140TB

~0.4TB (1 time level) ~0.4TB

~3TB (7 time levels) ~3TB

LETKF is parallel efficient, requiring all-to-all comm. only twice.

Observations~1TB

Ensemble simulations have ideal parallel efficiency.

An efficient architectural design

Sim-to-Obsconversion

Simulation

Member 1 Member 100…

Simulation

Sim-to-Obsconversion

~0.4TB ~0.4TB

~3TB ~3TB

Cluster 1 Cluster 100

Fast communication is important within each cluster;slower inter-cluster communication is acceptable.

DA

~140TB

LETKF requires inter-cluster communications only TWICE.

Other challenges of Big DA• Transferring Big Data

– To assimilate “Big Data” into extreme-scale simulations, we need to collect them in an HPC.

– Can we apply a “cloud” approach?

• Exploring useful data– e.g., live camera images may be useful for weather

forecasting, but it is hard to collect, qc, and use them…

• Archiving– Extreme-scale DA produces at least ~1PB per day.