“big data assimilation”€¦ · may 8, 2020, egu, session as1.1, live presentation “big data...
TRANSCRIPT
![Page 1: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/1.jpg)
Takemasa Miyoshi*T. Honda, S. Otsuka, A. Amemiya, Y. Maejima,
Yo. Ishikawa, H. Seko, Y. Yoshizaki, N. Ueda, H. Tomita,Yu. Ishikawa, S. Satoh, T. Ushio, K. Koike, Y. Nakada
*PI and presenting, RIKEN Center for Computational [email protected]
May 8, 2020, EGU, Session AS1.1, Live Presentation
“Big Data Assimilation”
Real-time Workflow for 30-second-update Forecasting and Perspectives toward DA-AI Integration
![Page 2: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/2.jpg)
3
Phased Array Weather Radar (PAWR)
3-dim measurement using a parabolic antenna (150 m,
15 EL angles in 5 min)3-dim measurement using a phased array antenna
(100 m, 100 EL angles in 30 sec)
100xdata size
(Courtesy of NICT)
![Page 3: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/3.jpg)
Big Data Assimilation
SimulationsObservations
Powerful supercomputerNew sensors, IoT
Big Data Big Data100x 100x
![Page 4: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/4.jpg)
DA workflow
Simulation
DA
Initial State
Simulated State
Observations
(Best estimate)
Sim-to-Obsconversion
Sim-minus-Obs
Broad-sense DA
Data-driven
Process-driven
![Page 5: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/5.jpg)
9/11/2014, sudden local rain
![Page 6: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/6.jpg)
9/11/2014, sudden local rain
1-h-lead downpour forecastrefreshed every 30 seconds
at 100-m mesh
![Page 7: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/7.jpg)
Pushing the limitsBig Data × Big Simulations
Big ensemble (10240 ensemble members)Rapid update (30-second update)
High resolution (100-m mesh) Future Numerical Weather Prediction
![Page 8: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/8.jpg)
Toward Weather-Ready Society5.0 with
![Page 9: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/9.jpg)
Toward Weather-Ready Society5.0 with
Toward demonstration at TOKYO 2020
![Page 10: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/10.jpg)
00 UTC 06 UTC 12 UTC 18 UTC
PAWR obsfor past cases
…
…
…
Bnd&Obs Bnd&Obs Bnd&Obs
D1 (18-km mesh) DA
D1 (18-km mesh)
D2 (6-km mesh)
D3 (1-km mesh)
D4 (250-m mesh)every-30-second DA
D4 (250-m mesh)
CREST BDA achievement
Online, real-time (Lien et al. 2017, SOLA)
Offline (Miyoshi et al. 2016a,b, BAMS, PIEEE)Computational time <30-sec.
![Page 11: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/11.jpg)
00 UTC 06 UTC 12 UTC 18 UTC
PAWR obsvia JIT-DT
…
…
…
Bnd&Obs Bnd&Obs Bnd&Obs
D1 (18-km mesh) DA
D1 (18-km mesh)
D2 (6-km mesh)
D3 (1-km mesh)
D4 (250-m mesh)every-30-second DA
D4 (250-m mesh)
Goal: fully online, real-time workflow
All online, real-time
![Page 12: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/12.jpg)
12 UTC
…
PAWR obsfor a past case
23:00UTC
…
…
…
D1 (18-km mesh)
D2 (6-km mesh)
D3 (1-km mesh)
23:00 D4 (250-m mesh)every-30-second DA
D4 (250-m mesh)
Test with U.Tokyo OFP on 29 AugustLarge-scale HPC Challenge17:46 NCEP GFS (boundary)
19:18 PREPBUFR (obs)
19:25-20:07 D1 DA
20:15-21:21 D1&D2 forecast21:31-22:41 D3 forecast
![Page 13: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/13.jpg)
12 UTC
…
PAWR obsfor a past case
23:00UTC
…
…
…
D1 (18-km mesh)
D2 (6-km mesh)
D3 (1-km mesh)
23:00 D4 (250-m mesh)every-30-second DA
D4 (250-m mesh)
Test with U.Tokyo OFP on 30 Nov.Large-scale HPC Challenge17:46 NCEP GFS (boundary)
19:18 PREPBUFR (obs)
19:25-20:07 D1 DA
20:15-21:21 D1&D2 forecast21:31-22:41 D3 forecast
64 DA cycles (for 32 minutes)
50 forecasts
![Page 14: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/14.jpg)
00 UTC 06 UTC 12 UTC 18 UTC
PAWR obsvia JIT-DT
…
…
…
Bnd&Obs Bnd&Obs Bnd&Obs
D1 (18-km mesh) DA
D1 (18-km mesh)
D2 (6-km mesh)
D3 (1-km mesh)
D4 (250-m mesh)every-30-second DA
D4 (250-m mesh)
Remaining part
Use JIT-DT in real time
![Page 15: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/15.jpg)
16
100
100
dual polarization 100×100
elements array antenna
Multi-parameter phased array weather radar (MP-PAWR) was developed by SIP (Cross-ministerial Strategic Innovation Promotion Program) in 2014-2018as a research subject of “torrential rainfall and tornadoes prediction.”
Early forecasting by water vapor, cloud, and precipitation observation
generate develop mature
★ Saitama Univ.(MP-PAWR site)● Olympic and Paralympic venues
Radius 60 km
Radius 80 km
Arakawa basin
Development of MP-PAWR
MP-PAWR features
MP-PAWR observation area
MP-PAWR antenna
MP-PAWR installed at Saitama Univ. on Nov 21, 2017, and observation began in July 2018.
![Page 16: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/16.jpg)
17
PAWR Data Utilization System (in NICT cloud)Okinawa Kobe
Suita PAWR
JGN AP Kinki-2
JGN AP Kanto-3vlanA
JGN AP Kinki-3
JOSE VMpanda-VM00
M2M
Suita
Koganei
6-gou SW
pawr-dp01 ~ dp05
trans-panda-sc
Kobe PAWROkn PAWR
dfsuc-panda-kgn
ql-panda-kgn
Keihanna
panda-nc-web
Osaka Univ. LAN
NICT Res.NW LAN
DB server(hostDB)
ql-panda-kob
NICT cloud FW
Handai-AP
L3-SWpawr-data-trans
ncloud AP
vlanA
vlanBvlanB
vlanA
vlanA
vlanB
db-panda-kobdb-panda-okn
ql-panda-okn
panda-gw
https://pawr.nict.go.jp
Saitama
SaitamaMP-PAWR
RSPU
MP-PAWR FW
JGN AP Kanto-3
Saitama-KoganeiNetwork opened on 5 Sep. 2019
Saitama MP-PAWR
DS01
Univ of Tokyo
Web server
https://weather.riken.jp
Opened on17 Dec. 2019
panda-nc-storage
NICT cloud FW RIKEN Kobe
Web server
Oakforest-PACShibiki
Internet
SINET AP
![Page 17: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/17.jpg)
Phased-Array Weather Radar3D precipitation nowcasting
(NICT)
App by MTI
30-second-update10-min forecast
![Page 18: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/18.jpg)
Qualitycontrol
Data conv&
Motionvector
JIT-DT61MB
Qualitycontrol
10-min.nowcasting
Visualization& Dissemination
Data conv.&
Motionvector
42MB
210MB
JIT-DT61MB
0Time (sec.)
10 20 30 40-10
Working continuously inREAL-TIME
REAL-TIME 30-second-update nowcasting~1min.
10-min.nowcasting
Visualization& Dissemination
conv.& tionctor
42MB
210MB
TB
ation nation
![Page 19: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/19.jpg)
Real-time dissemination started on 7/27/2017 with MTI Ltd.
Making societal impact
>247,000 DLRanked #3!
RIKEN’s3D Nowcast
![Page 20: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/20.jpg)
PAWR 3D NowcastingObservation
Weather Simulation
INPUT
OUTPUT 3D Precip. Nowcasting
Improved forecast
Parallel test of real-time nowcasting w/ ConvLSTMsince June 6, 2019!!
No input of future data at this moment
![Page 21: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/21.jpg)
Conv-LSTM is effective.(Work with Mr. Viet Phi Huynh and Prof. Pierre Tandeo)
2.5-min prediction
![Page 22: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/22.jpg)
Future direction: Fusing ML+DA+SimulationObservation
Weather Simulation
INPUT
OUTPUT New 3D Precip. Nowcasting
Improved forecast
Input of future data from NWP!!
![Page 23: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/23.jpg)
DA-AI Integration
Simulation
DA
Initial State
Simulated State
Observations
(Best estimate)
Sim-to-Obsconversion
Sim-minus-Obs
Broad-sense DA
Predict high-resolutionfrom low-resolution model
Predict model error
Model-obs relationship
Quality control
DA algorithm
Surrogate model
Need to learn big computation data on HPC (cannot move)
![Page 24: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/24.jpg)
DA-AI Integration
Simulation
DA
Initial State
Simulated State
Observations
(Best estimate)
Sim-to-Obsconversion
Sim-minus-Obs
Broad-sense DA
Predict high-resolutionfrom low-resolution model
Predict model error
Model-obs relationship
Quality control
DA algorithm
Surrogate model
Need to learn big computation data on HPC (cannot move)Integrating DA and AI
Pioneering new meteorology
![Page 25: “Big Data Assimilation”€¦ · May 8, 2020, EGU, Session AS1.1, Live Presentation “Big Data Assimilation” Real-time Workflow for 30-second-update Forecasting and Perspectives](https://reader030.vdocuments.site/reader030/viewer/2022040614/5f0c59b27e708231d434f774/html5/thumbnails/25.jpg)
Toward Weather-Ready Society5.0 with
Goals1. “Big Data Assimilation” × AI2. International collaborations3. Demonstration at Tokyo 20204. Societal impacts