Modeling North Carolina Meteorology and Storm

Surge at The Renaissance Computing Institute

(www.renci.org)

Howard LanderBrad Viviano

Brian BlantonBrian Etherton

Modeling at RENCI 2

The Renaissance Computing Institutewww.renci.org

• Formed in 2004 as a collaborative institute involving the University of North Carolina at Chapel Hill, Duke University and North Carolina State University.

• RENCI develops and deploys advanced technologies to enable research discoveries and practical innovations.

• This science of cyberinfrastructure is essential to continuing scientific discovery and innovation.

Modeling at RENCI 3

RENCI Resources

• A diverse group of people including domain scientists in oceanography, meteorology, chemistry, infomatics and computer science.

• A diverse set of projects spanning the domains listed above and more.

• Several compute clusters with an aggregate peak computing power of approximately 30 Teraflops.

• Approximately 900 Tb of spinning disk.

• An ideal laboratory to develop the science of cyberinfrastructure

Modeling at RENCI 4

Why study storm related disasters?• Significant loss of life:

– Katrina (2005): about 1500 deaths– In North Carolina: Floyd (1999) 35 deaths

• Large property damage:– Katrina: $81 Billion– In North Carolina:

Fran (1996) $3.2 Billion, Floyd (1999) $4.5 Billion

• Large economic impact in North Carolina- Coastal Tourism is $2 Billion per year- Ports in Wilmington and Morehead city support almost 50000 NC jobs

Modeling at RENCI 5

Why is this happening?

• 44% of world population lives within 150 km of the coast!• US Estimate (2003): 53% of population in coastal counties• In North Carolina: 10% of population in coastal counties

– But this is on the increase…– In particular, the retiree population is increasing rapidly in our southern

counties– -Google for Retire Brunswick County NC: 198000 hits!

Modeling at RENCI 6

What do we hope to achieve?

• Emergency Management Applications:– Better prediction of storm track and surge for decision making

and recovery. “$1 million to $50 million” per storm(J. Whitehead, Ocean and Coastal Mgt, 2003.)

• Develop better policies for coastal planning– Perform what if analyses: Katrina or Floyd as Category 5

storms– Calculate FEMA flood zones using probabilistic computer

studies of the coastal flood plain

Modeling at RENCI 7

New Orleans – Katrina 2005

Floodwall on I-510: FEMA

Don McCloskey

Modeling at RENCI 8

Floyd North Carolina (1999)Tar River OVER the 500 year flood stage!

Edgecombe County: FEMA

Tar River: FEMA

Modeling at RENCI 9

Meteorology and Storm Surge at RENCI• Part of RENCI’s “Disaster Studies” Group• Primary models run include

– Weather Research and Forecasting (WRF) model: structured grid mesoscale weather prediction system

– ADCIRC (ADvanced CIRCulation): an unstructured grid, finite element storm surge and tidal model

– SWAN: a structured grid model for modeling waves

– PADCSWAN: coupled version of ADCIRC + SWAN

– WW3: a basin scale wave model

– Hope to add in hydrologic modeling soon.

Modeling at RENCI 10

Meteorology Modeling at RENCI

• RENCI is producing WRF model output in a production mode.– RAPID REFRESH– DAILY– ENSEMBLE– HURRICANE

WRF forecast of temperature at 6A.M. E.S.T. on January 9, 2010. WRF model is at 1km resolution. Features such as mountains, lakes, and urban areas visible in output.

Modeling at RENCI 11

Rapid Refresh• Model specifics

– 9km/3km/1km resolution– 12 hour forecast length– RUC IC/BC– Initialized on ‘odd’ hours

• Run time example– 1200 RUC IC (1hr fcst)– 1000 RUC BC– 1315 start– 1430 complete

• Technical Details– 256 cores– 320 CPU Hours

WRF forecast of radar reflectivity are at 9 P.M. E.D.T. on October 27, 2010. WRF model is at 1km resolution. Features such as mountains, lakes, and urban areas visible in output.

Modeling at RENCI 12

Daily

• Model specifics– 9km/3km resolution– 84 hour forecast length– NAM IC/BC– Initialized at 00Z and 12Z

• Run time example– 1200 NAM IC/BC– 1450 start– 1745 complete

• Technical Details– 128 cores– 376 CPU Hours

WRF forecast of top of the atmosphere brightness temperature at 10 A.M. E.D.T. (1400 UTC) on October 29, 2010. Forecast initialization time was 1200 UTC on October 26, 2010.

Modeling at RENCI 13

Ensemble• Model specifics

– 9km/3km resolution– 30 hour forecast length– NAM IC/BC– Initialized at 06Z and 18Z

• Run time example– 0600 NAM IC/BC– 0850 start– 1230 complete

• Technical Details– 16 Ensemble Members– 512 cores (32x16)– 2000 CPU Hours

Ensemble mean forecast of 2-meter temperature valid at Noon E.D.T. (1600 UTC) on October 20, 2010. Forecasts were initialized at 0600 UTC on October 20, 2010

Modeling at RENCI 14

Hurricane

Forecast of track/intensity of “Earl” for August 30 to September 4, initialized at 1200 UTC on August 30.

• Model specifics– 27km/9km/3km resolution– 132 hour forecast length– GFS IC/BC– Initialized at 00Z and 12Z

• Run time example– 1200 GFS IC/BC– 1620 start– 2030 complete

• Technical Details– 256 cores– 1024 CPU Hours

Modeling at RENCI 15

Earl – prediction versus reality

• RENCI/NCSU WRF forecasts provided non-standard information (such as this ‘simulated satellite’ image)

Forecast valid at: 06Z03SEP2010

Top of Atmosphere Brightness Temperature (C)

Modeling at RENCI 16

Track Forecasts from Aug 30

• RENCI/NCSU WRF forecasts predicted a track closer to the North Carolina coast than other operational forecast models.

Modeling at RENCI 17

Storm Surge Modeling at RENCI

• RENCI is producing storm surge predictions in several modes. – DAILY– HURRICANE– Federal Emergency

Management Agency (FEMA)

Isabel 2003 Maximum Water Level Simulation Displayed on Google Earth

Modeling at RENCI 18

Daily/Hurricane Run Computational Grid

Finite element method (FEM) provides much more detail near the coast.Smallest cells: 200 MetersLargest cells: 100 Km!

Modeling at RENCI 19

Grid Detail: Masonboro Inlet, NC

Modeling at RENCI 20

Daily• Model specifics

– 295000 Node Irregular Grid– 84 hour forecast length– NAM Wind Forcing– Initialized every 6 hours

• Run time example– 1200 NAM Wind Forcing– 1500 NAM Wind Arrival– 1505 start– 1750 complete

• Technical Details– 128 cores– 350 CPU Hours– PADCSWAN

Maximum inundation forecast NC coast November 4, 2010 T06 – November 11 T18. Inundation is water level – terrain height.

Modeling at RENCI 21

Detail: Brunswick County NC

The seventeenth fastest growing county in the United States:

•2009 Population: 107,000•2000 Population: 73,000•10 year growth: 46.4%•Over age 65: 20.6% (US Avg 12.7%)

Modeling at RENCI 22

Hurricane• Model specifics

– 295000 Node Irregular Grid– Variable Forecast Length– ADCIRC internal wind model– Initialized on arrival of National Hurricane

Center forecast track.

• Run time example– 0900 forecast track arrives– 0905 start– Dependent on track length

• Technical Details– 128 cores– Dependent on track length– PADCSWAN

Significant Wave Height prediction off the coast of North Carolina, September 6, 2010 Hurricane Earl

Modeling at RENCI 23

Detail: Pamlico Sound NC

Note how little wave energy makes it inside the sound despite 10 – 12 feet waves at the beach. Maybe it’s better to live on the sound!

Modeling at RENCI 24

FEMA• Flood insurance Study

– Used to define 100 and 500 year flood plains.

– Used to set insurance rates– Study is probabilistic in nature:

uses multiple synthetic storms and a complex statistical methodology to extract results.

– Requires high resolution study of bathymetry and topography

– Combines Wave and Wind effects

– Storm runs are processed statistically to produce end result

Current Flood Insurance rate map for Bald Head Island. It indicates regions expected to flood at the 1% (100 year) and .2% (500 year) flood levels as well as high wave effect regions.

Modeling at RENCI 25

Synthetic Storm Tracks

• North Carolina: Based on 20 historical storms

• 675 tracks with different parameters

• Parameters include Central Pressure, Forward Speed, Radius to Max Wind …

P. Vickery, Applied Research Associates/RISK

Modeling at RENCI 26

FEMA Flood Plain Computational Workflow• Workflow consists of 8 discrete computational steps

– Driving winds are generated from the storm tracks

– WaveWatch III and ADCIRC are run to produce data for SWAN RUNS

– SWAN is run 4 times: There are a northern and southern nested grid systems

– SWAN results and winds drive a final ADCIRC run which produces the surge prediction for that storm

– North Carolina project: 675 tracks, 10 million CPU hours, 30 Tb data.

Modeling at RENCI 27

FEMA Flood Plain Computational Workflow

Wind Model Wave Watch 3 produces oceanic wave field

SWAN for near shore wave field and radiation stress. Note that there are 2 northern and 2 Southern grids.

ADCIRC produces setup data for SWAN, as well as the final surge prediction.

Modeling at RENCI 28

Sample Results

Track for synthetic storm lf2_dp4r3b2c3h2l1

Maximum inundation for synthetic storm lf2_dp4r3b2c3h2l1

29

Blue Ridge Design considerations

• 90% of the code used is existing research code (WRF, ADCIRC, etc) not developed at RENCI. Most development work is done on Linux commodity cluster using MPI.

• Profiling tool runs on our various MPI applications showed memory transfers where important in relation to core count.

• RENCI is partnered with Microsoft Research on various projects, so this system needs to be able to run Windows HPC.

• IT support costs, specialized hardware platforms require dedicated support staff and training.

30

Memory Bandwidth comparison using pChase (pchase.org)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

1000200030004000500060007000

NEH3: 2 sockets using 1 thr/core [memory per ref.=16MB; page

size=4KB;cacheline size=64bytes] 2 threads4 threads6 threads8 threads

Concurrent misses / threadMem

ory

Band

wid

th (M

B/s)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

10000

20000

30000

40000

NEH3: 2 sockets using 1 thr/core [memory per ref.=16MB; page

size=4KB;cacheline size=64bytes] 2 threads4 threads6 threads8 threads

Concurrent misses / threadMem

ory

Band

wid

th (M

B/s)

Dell M600• 2 x Intel E5440• 8 x 4GB 667Mhz Memory

Dell M610• 2 x Intel X5560• 6 x 4GB 1333 Mhz Memory

Presentation title goes here 31

Other considerations

• M600/M605 are PCIe Generation 1. Maximum speed on mezzanine card is 20Gbit.

• M610 is PCIe Generation 2. Maximum speed on mezzanine card is 40Gbit.

• Dell DDR Infiniband Switch is 2-to-1 blocking

32

Current Hardware LayoutCompute (8 Blade Centers/128 Nodes) GPGPU (2 Nodes)

Dell PowerEdge M610 Dell PowerEdge R710

2 x Intel x5560 (4C 2.8Ghz) Processor 2 x Intel x5560 (4C 2.8Ghz) Processor

24GB (6 x 4GB) 1333Mhz Memory 96GB (12 x 8GB) 1066Mhz Memory

Mellanox QDR Infiniband HCA Mellanox QDR Infiniband HCA in x4 slot

1Gbit Ethernet 10 Gbit Ethernet

nVidia Tesla S1070-500• 960 1.4Ghz Cores• 16GB Memory• 4.14 Tflops

33

MPI Fabric

• Core Stack: 8 Mellanox 36 Port MTS3600Q Switches

• Quad Data Rate Infiniband (40 Gbit)• Non-Blocking Fabric for up to 18 Dell M1000e

Blade Centers using Dell M3601Q Switches (288 Nodes, 2304 cores)

34

File System Network

• Force10 S2410CP, 24 Port 10GbE Non-Blocking Switch• 40 Gbit LAG to core data center switch (Force10 E600i)

providing connection to BlueArc NFS Server and PVFS2 cluster.

• Dell PowerConnect M6220 blade center 1GbE switch w/10GbE Uplink

• 1.6-to-1 Blocking factor

35

Blue Ridge Software

• SDSC Rocks (Open Source) w/ CentOS 5• Mellanox OFED 1.4 stack• Moab/Torque Batch Scheduler• MVAPICH 1, MVAPICH 2, OpenMPI• Globus

36

ADCIRC Run Times500k North Carolina Coast Grid

Cores Blue Ridge (x5560)2009

Kitty Hawk (x5250)2007

Ocracoke (BlueGene/L) 2005

16 4h:16m:27s 12h:44m:42s

32 1h:50m:12s 6h:07m:42s 19h:22m:40s

64 1h:39m:58s 3h:14m:48s 10h:22m:01s

128 52m:04s 1h:53m:31s 5h:22m:40s

256 31m:24s 2h:08m:31s 2h:56m:19s

512 26m:58s 4h:28m:32s

768 21m:40s

Modeling at RENCI 37

Contact information

Howard Lander <howard@renci.org>