Experience Experience in numerical forecast verification in numerical forecast verification

in the Hydrometeorological in the Hydrometeorological Centre of RussiaCentre of Russia

N. P. Shakina, E. N. Skriptunova, A. R. Ivanova

Zürich 2005

COSMO 7th General Meeting

The models:The models:• SM HMC, global spectral model (T85L31) of      Hydrometeorological Centre of Russia

• SLM HMC, global semi-Lagrangian model, resolution      0.9x0.720, 28 σ-levels to 6 hPa

• UKMO global model

The objective analyses:The objective analyses:• OA UKMO, 2.5x2.5o

• OA HMC, operative version, 2.5x2.5o and 1.25x1.25o

• OA DAS (Data Assimilation System) of SLM HMC

WMO/ICAO standards for high levelsWMO/ICAO standards for high levels::

Wind speed, error above 250 hPa 10 m/s in 90 % of points

Wind speed, error below 250 hPa 7 m/s in 90 % of points

Wind direction error 30 deg in 90 % of points

Max wind level height, error 600 m in 70 % of points

Tropopause height error 600 m in 70 % of points

Convective cloud top error 600 m in 70 % of points

RMSE Abs. error Bias Projec tion SM

HMC SLM HMC

SM HMC

SLM HMC

SM HMC

SLM HMC

24 7.60 7.05 4.28 3.87 -1.55 -1.21 48 10.18 9.70 5.60 5.06 -2.07 -1.49 72 12.77 12.24 6.80 6.20 -2.31 -1.76

Standard estimates of 250 hPa wind speed forecasts (Laboratory for Forecast Testing), August 2005, 00 UTC, Northern Hemisphere

Maximum wind and jet streamsMaximum wind and jet streams(standard approach)(standard approach)

From the cubic spline approximation of wind component profiles, maximum wind speed, MW, and level, H(MW), are determined in the gridpoints

100150

200

250

300

400

500

700

850

925

1000

P , m b

Wm ax

W(p)

SLM forecasts SM

forecasts UKMO forecasts

ОА DAS SLM Date Time

ОА HMC

ОА UKMO 2.5х2.50 0.72х0.900

OA HMC ОА

UKMO

1 00 86.9 80.3 92.5 91.6 72.4 93.8 1 12 84.3 78.9 96.0 96.1 79.5 95.0 2 00 87.5 85.7 97.9 96.4 75.5 97.0 2 12 84.2 74.3 92.7 94.3 76.3 97.6 … … … … … … … … 8 12 91.4 81.7 90.1 91.2 85.1 97.1 9 00 85.1 80.0 90.0 91.3 80.7 97.1 9 12 92.0 80.1 95.3 94.5 82.1 98.4

10 00 90.7 86.6 95.0 94.6 83.0 98.5 10 12 92.4 86.4 97.3 97.0 85.7 98.4

00 87.6 82.3 93.4 93.3 79.7 96.2 12 88.7 79.7 93.6 94.2 82.5 96.9

Ave rage Total 88.1 81.0 93.5 93.7 81.1 96.5

Percentage of ICAO standards fulfilled for maximum wind speed by the SLM, SM and UKMO models,

with different OA taken as “fact”:1-10 Oct 2004, 24-h projection

maximum wind speeds, m/s Model

<30 30 – 40

40 – 50

50 – 60

>60 >30 Total

SLM HMC -0.8 -2.0 -3.1 -3.7 -4.5 -2.8 -1.5 SM

HMC -1.6 -3.1 -4.9 -6.3 -8.9 -4.6 -2.7

UKMO 0.7 1.2 1.3 1.1 0.7 1.2 0.9

Distribution of maximum wind speed 24-h forecast biases for SLM, SM HMC and UKMO models

along the maximum wind speed spectrum, 1-10 Oct 2004

temperature

potential vorticity

0 2 4 6 8 10

200-20-40100

1000

500

700

850

300

T, C

PV, pvu

p, hPa

pf

y

u

x

vgpPV

)()(

Typical profiles of temperature and PV in the troposphere and lower stratosphere

Comparison of thermal and dynamic tropopause 24-h forecasting bias, hPa (UKMO model), with respect to the

corresponding UKMO objective analysis, 1-10 Oct 2004

P, hPa >400 400-350

350-300

300-250

250-200

200-150

150-100

<100 Total

Thermal tropopause

-47.2 -24.0 1.3 5.5 2.2 1.9 9.9 - 3.9

Number of cases

310 365 2412 8826 8414 2430 6143 0 28900

Dynamic tropopause

-16.9 -1.6 0.1 2.3 3.9 3.0 0.9 2.3 2.1

Number of cases

185 1004 3812 8345 6511 5927 2616 500 28900

Dynamic tropopause height, hPa

400-350 350-300 300-250 250-200

200-150

Average

24-h projection July, SM HMC -31.4 -14.8 -5.9 3.5 10.4 -1.4 July, UKMO -0.6 1.6 0.0 2.5 2.6 1.3 Oct, SM HMC -42.8 -19.7 -6.8 0.4 5.2 -5.0 Oct, UKMO -17.0 -0.7 0.6 2.7 3.7 1.1

36-h projection July, SM HMC -39.0 -18.3 -6.6 5.5 12.7 -1.2 July, UKMO -7.5 0.2 -0.1 3.7 4.1 2.3 Oct, SM HMC -51.6 -23.9 -7.7 1.4 6.2 -5.6 Oct, UKMO -16.8 -0.1 2.4 5.0 5.0 2.8

Distribution of the tropopause height bias as dependent on the tropopause height; 1-10 July and 1-10 October 2004

Percentage of ICAO standard fulfilled for the tropopause height predicted by SM HMC and UKMO model

24 h 36 h Model

00 12 Total 00 12 Total July 74.7 75.3 75.0 67.5 66.5 67.0

SM HMC Oct 78.1 78.2 78.2 71.5 72.4 71.9 July 90.4 89.8 90.1 78.6 78.3 78.5

UKMO Oct 89.9 90.2 90.0 82.4 83.1 82.8

Note: the UKMO thermal tropopause 24-h forecasts meet the ICAO standard in 77% of the gridpoints (100W-0-1800E-1600W, 35-750N)

Level of neutral buoyancy, LNB, km

Convective available potential energy, CAPE, J

Convective inhibition energy, CIN, J

Free convection level, FC, km

Downdraft origination level, DOL, km

Characteristics Characteristics of of convective instabilityconvective instability

Comparison of LNB>2km 24-h forecasts against different objective analyses taken as ‘facts”, 1-10 October 2004

SLM Time UKMO SM HMC

0.72х0.900 2.5х2.50 00 503 396 3511 472 Number of gridpoints

with LNB >2 km 12 480 419 3736 349 Total number of gridpoints

1445 1445 13104 1445

Fact: OA UKMO 00 83.6 61.5 - 69.0 Correct forecasts

LNB>2 km occurrence, %

12 84.3 63.2 - 66.0

00 88.5 84.4 - 77.7 Correct forecasts LNB >2 km non- occurrence, %

12 89.5 84.4 - 79.5

00 75 232 - 117 False alarms, number of gridpoints 12 68 197 - 136

00 112 132 - 235 Aim missings, number of gridpoints 12 104 142 - 212

SLM Time UKMO SM HMC

0.72х0.900 2.5х2.50 00 503 396 3511 472 Number of gridpoints

with LNB >2 km 12 480 419 3736 349 Total number of gridpoints

1445 1445 13104 1445

12 104 142 - 212 Fact: OA HMC

00 - 45.7 - 53.7 Correct forecasts LNB >2 km occurrence, %

12 - 50.6 - 53.3

00 - 85.7 - 82.1 Correct forecasts LNB >2 km non- occurrence, %

12 - 83.8 - 80.4

00 - 327 - 178 False alarms, number of gridpoints 12 - 263 - 188

00 - 121 - 189 12 - 147 - 203

Aim missings, number of gridpoints

12 - - 765 86

Comparison of LNB>2km 24-h forecasts against different objective analysis taken as ‘facts”, 1-10 October 2004 (cont.)

SLM Time UKMO SM HMC

0.72х0.900 2.5х2.50 00 503 396 3511 472 Number of gridpoints

with LNB >2 km 12 480 419 3736 349 Total number of gridpoints

1445 1445 13104 1445

Fact: OA DAS SLM

00 - - 78.2 77.3 Correct forecasts LNB >2 km occurrence, %

12 - - 65.0 65.0

00 - - 83.2 83.6 Correct forecasts LNB >2 km non- occurrence, %

12 - - 91.8 91.7

00 - - 768 87 False alarms, number of gridpoints 12 - - 1312 142

00 - - 1611 175 Aim missings, number of gridpoints 12 - - 765 86

Comparison of LNB>2km 24-h forecasts against different objective analyses taken as ‘facts”, 1-10 October 2004 (cont.)

CONCLUSIONS:

1. Apart from standard criteria of numerical forecast accuracy, we estimate regularly (10 days in central month of seasons) accuracy of predicted quantities obtained from the model output data by means of post-processing.

2. The quantities under consideration represent cumulative characteristics of

wind profile (maximum wind),

wind and temperature profiles and horizontal distributions (potential vorticity),

temperature and humidity profiles (convective instability).

CONCLUSIONS (cont.):

3. Accuracies of maximum wind, tropopause, convection (and others) are estimated for the operatively used global models (SM HMC, SLM HMC, UKMO model) and for the corresponding objective analyses.

4. The results allow us revealing and quantitatively estimating features of models and their DAS which are not clearly seen from the standard accuracy criteria, for example:

spectral dependence of wind errors;

smoothing of tropopause funnels and domes by the      models;

“diurnal cycle” of convective instability

CONCLUSIONS (cont.):

5. Analysis of these effects, in a close co-operation with the model developers and with numerical forecast users (aviation forecasters) leads to better interpretation of the numerical forecasts and to improving of the models.

Thank you!