current emc global model dynamics and beyond...2016/05/24 · 3 june 2016 twpgfs2016, h. juang 1...

3 June 2016 TWPGFS2016, H. Juang 1

Current EMC global model dynamics and beyond

Hann-Ming Henry Juang

Environment Modeling Center, NOAA/NWS/NCEP, Washington, DC

Concerns

• Current EMC global model dynamics is a spectral model.

• Two dynamics versions in the code

– Operational hybrid coordinate SL

– Generalized coordinate enthalpy NDSL

• Beyond (in-house GSM and NGGPS)

– Nonhydrostatic system

– Deep atmosphere


Contents

• Advantages of traditional global spectral model

• Developments has been done – Improve grid structure

– Improve thermodynamics

– Improve semi-Lagrangian

– Improve spectral transform

• Under testing and further plan– enhance parallelization

– different spectral truncation

– nonhydrostatic system

– deep atmosphere system


Traditional GSM

• The problem on sphere => Spherical coordinate

• Spherical harmonic (spectral transform)

– > leads to no pole problem

• Highest accuracy in horizontal discretization

• No bias due to computing direction

• Linear computation has no phase error

• easy to do semi-implicit time scheme

• Only vertical grid discretization needed


Contemporary GSM

• Reduced Spherical Gaussian grid

• Generalized vertical coordinates

• Considering gases into thermodynamics

• Semi-Lagrangian advection with semi-implicit

• Refine Legendre coefficient for high resolution

• Expand full dimensional MPP

• Exam cubic truncation with less diffusion

• Consider deep atmospheric dynamics


6

Juang 2004

Juang 2004

Based accuracy of associated Legendrepolynomial function, higher latitudes havefew significant waves leads to smaller gridnumber for spectral transform


Table 2: Details of model configurations.

NH-GFS (Baseline) * FV3 MPAS NIM NMMB-UJ NEPTUNE IFS (RAPS13) *

Resolution 13 km (TL1534) ~12 km (C768)* 12km * 13.4 * 13 km 12.71 km * 12 5 km (Tc799)

Grid Points3072x1536 (unreduced)

3,126,128 (reduced)

6x768x768

3,538,9444,096,002 ** 3,317,762

6x768x768

3,538,944 *3,110,402 **

3,336,946

(reduced)

Vertical Layers * 128 127 ** 127 *** 128 128 127 *** 137

Time Step TBD

600s (slow phys)

150s (vertical, fast

phys)

150/10 (horiz.

acoustic)

72 s (RK3 dynamics)

12 s (acoustic)

72 s (RK3 scalar

transport)

72 s 24 s **75 s (advective),

15 s (sound) ****450

Resolution 3 km (TL6718) ~3 km (C3072) * 3km 3.3 km ** 3 km 3.13 km * 3.125 km (Tc3199)

Grid Points13440x6720 (unred.)

59,609,088 (reduced)

**

6x3072x3072

56,623,10465,536,002 53,084,162

6x3072x3072

56,623,104 *61,440,000 **

51,572,436

(reduced)

Vertical Layers * 128 127 ** 127 *** 128 128 128 137

Time Step TBD

150 s (slow phys)

37 5 s (vertical,

fast phys)

37.5/10 s (horiz.

acoustic)

18 s (RK3 dynamics)

3 s (acoustic)

18 s (RK3 scalar

transport)

18 s 6 s **15 s (slow RK3 dyn.)

2 5 s (fast dyn.)120

Notes

* Unl es s noted,

la yers refers to the

number of la yers ,

not the number of

i nterfa ces between

la yers + top +

bottom

* Baseline configuration is

tentative, pending test

evaluation.

** Rough estimate for

reduced Gaussian grid

based on reduction factor

(0.66) of 13 km grid. This will

likely be revised after

further testing of accuracy

of spectral transform at

TL6718.

* True resolution is

average over equator

and/or from south to

north pole. For 13km,

max cell size (edge of

finite volume): 14.44

km, min: 10.21 km,

global avg: 12.05 km.

For 3.25 km, divide by

4.

** Favorable OpenMP

Performance

* Resolution refers to

mean cell-center

spacing on the mesh

** Subdivision of 60 km

mesh by factor of 5.

*** Following the FV3

configuration, we will

use 127 levels where

density,

theta and horizontal

momentum are defined

(on our Lorenz-grid

vertical

discretization) and 128

levels for w (that

includes both the lower

boundary and the

model top "lid").

* Generated by 6

bisections followed

by 2 trisections.

Distances between

neighbors: 13.367

average, 12.245

min., 14.397 max..

Maximum ratio of

neighboring grid

point distances:

1.17577

** Generated by 8

bisections followed

by 2 trisections.

Distances between

neighbors: 3.3417

average, 3.060 min.,

3.601 max..

Maximum ratio of

neighboring grid

point distances:

1.1765.

* B-grid mass points

** For fast modes and

advection of basic

model variables. Time

step for tracers is

longer by 2x.

* Resolution refers to the

representative nodal

spacing in the element

measured as the

midpoint between the

minimum and mean

nodal spacing and

averaged over the globe.

** Horizontal grid points

is six faces of cube times

number of elements per

face times polynomial

order squared.

* Hydrostatic

The Tc799 cubic grid has

the same number of grid

columns as a TL1599

linear grid.

While the Tc3199 cubic

grid has the same

number of grid columns

as a TL6399 linear grid.

No

min

ally

13

kmN

om

inal

ly 3

km

13

AVEC report: NGGPS Level-1 Benchmark and software evaluation

Generalized vertical coordinate

• First version of GSM used sigma coordinate, then move to hybrid sigma-pressure coordinate

• A dynamics system can be used for sigma, sigma-pressure, and sigma-theta, sigma-theta-pressure etc (Juang 2011), instead of only sigma coordinate or hybrid sigma-p coordinates

• The same model dynamics to be easily compared for different coordinates


June 3, 2016 Hann-Ming Henry Juang 13

From the ideal-gas law for individual gas as

p = rRT

pi = riRiT

p = riRii=1

N

å T = rrirRi

i=1

N

å T = rRT we havethrough

R = qiRii=1

N

å

qi =rir

let

rdCPT

dt-dp

dt= rQ

Then put them into internal equations, we have

we let

h =CPT as a prognostic variable, enthalpy.

we have

CP = qiCP ii=1

N

åso

14Black s: operational GFS Red t: sigma-theta GFS

Case Results - AC

PROW: gc s-pPROEW: gc s-p enthalpy

Root Mean Square error

Opr GFS vs WAM

22

64 layers150 layers

3 June 2016 TWPGFS2016, H. Juang


Example of T profileof 150 layers

WAM uses generalized hybrid coordinatewith enthalpy CpT as thermodynamicsvariables , where Cp is summation ofeach gases.

R Cp

O 519.674 1299.18

O2 259.837 918.096

O3 173.225 820.239

Dry air 296.803 1039.64

H2O 461.50 1846.00

3 June 2016 H. Juang

in do_dynamics_two_loop for spdmx at kdt= 40825spdmx(001:010)= 19. 20. 21. 23. 25. 26. 27. 28. 28. 28.spdmx(011:020)= 28. 27. 27. 27. 27. 28. 28. 28. 29. 30.spdmx(021:030)= 31. 33. 35. 37. 40. 42. 44. 46. 49. 53.spdmx(031:040)= 58. 61. 63. 63. 62. 60. 55. 47. 45. 44.spdmx(041:050)= 45. 45. 47. 49. 52. 55. 59. 62. 65. 68.spdmx(051:060)= 72. 76. 80. 84. 87. 90. 93. 95. 97. 98.spdmx(061:070)= 102. 110. 118. 127. 135. 143. 149. 153. 155. 152.spdmx(071:080)= 147. 145. 142. 138. 135. 132. 130. 126. 121. 119.spdmx(081:090)= 114. 112. 110. 106. 100. 95. 94. 90. 89. 89.spdmx(091:100)= 87. 82. 91. 95. 99. 97. 104. 100. 111. 120.spdmx(101:110)= 125. 133. 148. 167. 172. 164. 159. 160. 147. 124.spdmx(111:120)= 117. 125. 133. 138. 137. 157. 183. 202. 220. 243.spdmx(121:130)= 269. 297. 319. 338. 355. 368. 378. 386. 392. 396.spdmx(131:140)= 399. 402. 404. 405. 406. 407. 408. 409. 410. 410.spdmx(141:150)= 411. 412. 412. 413. 413. 414. 414. 415. 415. 418.

Maxima wind (m/s) at NCEP GFS 150 layers WAM

24


WAM Euleriandt=180sec

T62L150


WAM NDSLdt=400sec

Non-iteration and Dimensional-split Semi-Lagrangian advection method

• No need to iteratively find departure points • Dimensional split with MPI transpose, no need of

halo, so avoid the decision of halo side • Positive defined in interpolation, so no q<0• No polar grid needed • Option to have mass conservation• Based on generalized vertical coordinate enthalpy

system

June 3, 2016 Henry Juang 27


72h fcst specific humidity

at model layer 40

control

ndsl

dt=600

dt=1800


24hr fcst cloud water

at model layer 30

control

ndslmcpd


T574 GC Eulerian

T574 Opr EulerianTL878 NDSL

TL878 NDSL

dt=120dt=600


Day 1

Day 5

Opr T574 EulerianTL878 NDSL

dt=120dt=600


Day 1

Day 5

Opr T574 EulerianNDSL TL878

dt=120dt=600

Anomaly correlation- Day 5 HGT

Leads to higher and higher resolutionfor semi-Lagrangian vs Eulerian comparison

Examine Legendre coefficient

• We found Legendre polynomial function in sp lib and model have problem of spherical harmonic transform while using high truncation, say T1000 and above

• X-number technique fixes the problem• Furthermore, increase precision only in

computing Gaussian weight helps accuracy by order of 3

• These steps provide necessities to implement TL1534 with enhanced and corrected spectral transform



0.0

underflow overflow

2-960 2960

Machine representable real number

X number takes care of under and over flow by xB**I with B=2**960, soxB**0 is machine representable real number but xB**(-1) represents underflowAnd xB**(+1) represents overflow. The details in Fukushima 2011

Implement X-number into SP lib and GSM model dynamics

xB**ix time yB**iy, should be equal to x*yB**(ix+iy), but we should take care underflow by x*y, to do so, consider normalize of any X-number into the rangeof 2**(-480) and 2**(480). If x and y are normalized, then x*y will not be over-or under-flow.

Normalized range


AVEC Report:NGGPS Level-1 Benchmarks and Software Evaluation

Full Dimensional MPP

• Current NCEP GSM has multi-threading by OpenMP and one dimensional MPI

• The limitation of OpenMP may be 12 and 1DMPI based on half of spectral truncation

• For 2D MPI, it is limited to spectral truncation and vertical layer number.

• Extend to 3DMPI with MPI-FFT and MPI-Legendre transform

• Invest halo or transpose for dimensional dependency computation


1D MPIGlobal communication

2D MPIGroup communication

June 3, 2016 13th on the use of hpc in meteorology 47

0 360

Halo Exchange for opr SL

Extra memory is required,which may be as huge

as computing grid while number of MPP cpu increases.

1D

2D

halo

June 3, 2016 13th on the use of hpc in meteorology 48

np

sp

np

0 360

np

sp

.

.

.

0 . . . . 180

<=>

Transpose for NDSL

No halo is requiredNo increasing memory with

Increasing number of PE(cpu)


Black grouped with red ones forMPI FFT without transpose

Black grouped with yellow ones forMPI Legendre w/o transpose

Black grouped with blue ones forregrouping into vertical formodel physics and vertical dependentmodel dynamics computations

So spectral transform will be no transpose, only vertical needs….One side MPI ....

3D MPI

Comparison

• Using TL1534 L64 as example

• 1DMPI+12 threads

=> We have 767 MPI 12 thread = 9,204 cpus

• 2DMPI + 12 threads

=> We have 767*64 MPI 12 thread = 589,056 cpus

• 3DMPI + 12 threads

=> 767*64*1534 MPI 12 threads= 903,611,904 cpus


Experimental cubic truncation

• Based on ECMWF IFS recent experiments, they have implemented cubic truncation octahedral reduced Gaussian grid for IFS Tco1279 ( 9 km )

• For semi-Lagrangian, we used to have linear truncation, for Eulerian, we have quadratic truncation to avoid aliasing

• Linear truncation resolves up to 2dx wave, quadratic to 3dx, and cubic to 4dx.

• Use cubic truncation with much less diffusion and without de-aliasing filter

• Octahedral grid is for future spectral/grid hybrid system and requires different FFT package (FFTW)



ECMWF Newsletter#147 spring2016

Implement Cycle 41r2Move from linear truncation reduced gridTo cubic truncation octahedral reduced grid


IFS octahedral reduced grid, more regular than spherical reduced grid, they arepreparing for next generation of hybrid spectral/grid-point configuration.


More resolving small scales

More efficient in computing time



IFS experimental results

ECMWF Newsletter #146 winter2015/2016

More small scale waves in TcBest mass conservation in Tc

Deep Atmos vs non-Hydro• From Deep atmosphere, we require r changes with time,

thus we need dw/dt equation• And we need full curvature and Coriolis force terms to

satisfy conservation• Thus, based on conservation requirement, a deep

atmospheric dynamic is a non-hydrostatic dynamic. A non-hydrostatic dynamics can be shallow or deep atmospheric dynamics.

• Both r and vertical components of curvature and Coriolisforce should be considered in deep atmosphere; and should not be considered in shallow atmosphere. (see Juang 2014 NCEP ON#477)

• Derive deep atmospheric system as the same form as a shallow atmospheric system with options for easy implementation


du*

dt+u*w

r - fsv

* + fc*w +

kh

p

1

r

¶ p

¶l-

¶ p

¶z

¶z

¶r

¶r

¶l

æ

èç

ö

ø÷ = Fu

dv*

dt +v*w

r + fsu

* +m2 s*2

rsinf +

kh

p

1

r

¶ p

¶j-

¶ p

¶z

¶z

¶r

¶r

¶j

æ

èç

ö

ø÷ = Fv

dw

dt -m2 s

*2

r -m2 fc

*u* +kh

p

¶p

¶z

¶z

¶r +g = Fw

dh

dt-

kh

p

dp

dt= Fh

¶r*

¶t+m2

¶r* u*

r¶l

+m2

¶r* v*

r¶j

+¶r* z

·

¶z= Fr

*

dqi

dt= Fqi

p = rkh

d()

dt=

¶()

¶t+ l

· ¶()

¶l+j

· ¶()

¶j+z

· ¶()

¶z=

¶()

¶t+m2u* ¶()

r¶l+m2v* ¶()

r¶j+z

· ¶()

¶z ; r* = r

r2

a2

¶r

¶z

fs = 2Wsinf ; fc* = 2Wcos2 f ; g = g(r) ; k =

R

CP ; g =

CP

CV ; s*2

= u*2

+ v*2

where


Deep Atmos Dyn

in spherical mapping

& generalized coordinates

Staniforth and Wood (2003)Juang (2014) NCEP Office Note

To derive deep-atmosphere continuity equation into the same

form as shallow atmosphere

We define a coordinate pressure as hydrostatic one based on

the coordinate density as

Put it into deep-atmosphere continuity equation

We have

¶r*

¶t+m2

¶r* u*

r¶l

+m2

¶r* v*

r¶j

+¶r* z

·

¶z= 0


where g0 is constant

In deep atmosphere, horizontal wind with Gaussian weighting

Modify continuity equation

We have


u* = rcos2 fdl

dt= ucosf

We can define, height-weighted horizontal wind as

v* = rcosfdf

dt= vcosf


Since and r* > 0

Thus is monotone with vertical coordinate

We can use it for coordinate definition

For opr compatibility, we use

and relation with height as

because r* = rr2

a2

¶r

¶z

let


w =r2

a2w

so rk2wk =

1

2r2

k+1wk+1 +1

2r2

kwk

we have wk =1

2wk+1 +

1

2wk

rk3 =

1

2r3

k+1 +1

2rk

3

From angular momentum principle, we have relationbetween model layer and model level as

so vertical momentum eq can be written as

with g =a2

r2g0

¶ p

¶1

r

= rg0a2

Then the hydrostatic means


In summary, we have following

So prognostic variables are

dh

dt-

kh

p

dp

dt= 0 and

where e =r

a

Summary• NCEP GSM may be an “old” model, but we are using

GSM2010s not GSM1980s.

• Current operational GSM is TL1534 (13km) semi-Lagrangian two-time level scheme with reduced spherical Gaussian grid and refined spectral transform.

• Current experimental GSM has options to be as Whole Atmospheric Model (WAM) with enthalpy and NDSL.

• Further speed up in computational wall time by multi-threading and full dimension MPI, maybe 2D MPI is more than enough.

• Experimental cubic and/or quadratic truncations with spherical reduced Gaussian grid instead of linear truncation for semi-Lagrangian, and with less diffusion.

• Developing GSM Deep-Atmospheric model beyond nonhydrostatic system with shallowness options for NGGPS.


current emc global model dynamics and beyond...2016/05/24 · 3 june 2016 twpgfs2016, h. juang 1...

Documents