Huiling Yuan1, Xiang Su1, Yuejian Zhu2, Yan Luo2, 3, Yuan Wang1

1. Key Laboratory of Mesoscale Severe Weather/Ministry of Education and School of Atmospheric Sciences, Nanjing University, China

2. Environmental Modeling Center/NCEP/NWS/NOAA, College Park, Maryland, USA

3. I.M. Systems Group, Inc., College Park, Maryland, USA

WWOSC 2014, Montreal, Canada, 21 August 2014

Evaluation of TIGGE ensemble predictions of Northern Hemisphere

summer precipitation during 2008-2012

Outline

Objectives The TIGGE Ensemble Prediction Systems (EPSs) data Evaluation results of TIGGE quantitative precipitation

forecasts (QPFs) and probabilistic QPF (PQPFs) Summary

the THORPEX Interactive Grand Global Ensemble (TIGGE)

Objectives

Evaluate the QPF and PQPF performance of TIGGE

EPSs

Assess the performance change before and after

major EPS upgrade

Su et al. 2014, JGR

Evaluation of TIGGE QPFs and PQPFs

Study period: 2008-2012 summer (June-August)

Spatial coverage: Northern Hemisphere (NH)

tropics (0-20°N) and midlatitude (20-49°N)

Accumulation period: 24-h precipitation (12 UTC-12 UTC)

Forecast lead time: 1-9 days

Horizontal resolution: 1° ×1° (interpolation)

4

Observation data: TRMM 3B42 V7 (gauge adjustment)

ECMWF portal http://tigge-portal.ecmwf.int/

Center Base time(UTC) members

Horizontal resolutionarchived

Fcstlength(day)

Initialperturb method

Modeluncertainty

Major EPS upgrade time

CMA(China)

00/12 14+1 0.56º×0.56º 0-10 BVs - -

CMC(Canada)

00/12 20+1 1.0º×1.0º 0-16 EnKF PTP + SKEBmulti-physics

17 Aug 2011

ECMWF 00/12 50+1 N320 (~0.28º)N160 (~0.56º)

0-1010-15

EDA-SVINI

SPPT + SPBS 9 Nov 2010

JMA 12 50+1 1.25º×1.25º 0-9 SVs SPPT 17 Dec 2010

NCEP 00/06/12/18

20+1 1.0º×1.0º 0-16 BV-ETR STTP 23 Feb 2010

UKMO 00/12 23+1 0.83º×0.56º 0-15 ETKF RP + SKEB 9 Mar 2010

1. The CMC EPS was upgraded to version 2.0.2 on 17 August 2011.2. The ECMWF EPS used a horizontal resolution of N200 (~0.45º) for 0-10 day forecasts and N128 (~0.7º) for 10-15 day forecasts before 26 January 2010. EVO-SVINI was used as the initial perturbation method before 24 Jun 2010. The SPBS method has been added on 9 November 2010.3. The JMA EPS began to use the SPPT method on 17 December 2010.4. The NCEP EPS was upgraded to version 8.0 and began to use the STTP method on 23 February 2010. In 14 February 2012, the NCEP EPS was upgraded to version 9.0.5. The UKMO EPS used a horizontal resolution of 1.25º×0.83º before 9 March 2010.

Referenced to the CMA EPS (frozen), the impacts of major model upgrades on the forecast performance are examined for other five EPSs.

Configurations of six TIGGE EPSs

Verification methods of QPFs and PQPFs

Area-weighted scores consider the latitude discrepancies Continuous scores: RMSE, Spatial correlation (SC), CRPSS, and

spread-skill relationship Dichotomous scores: Bias, ETS, POD, FAR, BSS,

attributes diagram (reliability curve and three decomposed terms of BS), ROC area and potential economic value (PEV)

6

where ：wi=cos(lat)xi, yi are forecast and observation samples, N is the number of samples

Su et al. 2014, JGR

Precipitation climatology (day +3)

7

JMA day +1 ensemble mean QPFshave large moist biases in the NH

tropics

Cause: JMA employs moist SVs over the entire tropics and perturbs the specific humidity with a large amplitude (Yamaguchi and Majumdar, 2010)

TRMM

JMA day 1

JMA

NCEPCMA

CMC

ECMWF

UKMO

mm/day

Forecast error (RMSE)

Control:dotted (…..)

Ensemble mean:solid (——)

Ensemble is better than control, especially in longer lead times

EC ensemble best

EC control is better than CMA ensemble in short lead times

JMA control in NH midlatitude best

RMSE and frequency

9

Cause: JMA underestimates heavy rain

It is not appropriate to use only RMSE to evaluate QPFs

0 10 20 30 40 50 60 70 80 90 100-6

-5

-4

-3

-2

-1

0

Precipitation (mm day-1)

Log1

0 of

fre

quen

cyPDF of Day+3

TRMM OBSCMA

CMC

ECMWF

UKMO

NCEPJMA

The control QPFs of JMA have the smallest RMSE in the NH midlatitude

Discrimination diagrams

The discrimination ability decreases with the lead time

EPSs have weak ability to discriminate heavy events

In the NH tropics, CMA shows little discrimination ability among different rain events

10

Dichotomous scores of ensemble mean QPFs

ECMWF best CMA very poor in the NH tropics

11

Spread-skill relationship

The day +1 ensemble spread of JMA is the largest in the NH tropics

CMC has the largest spread and it grows with the lead time:

NH midlatitude: level with the ensemble mean error

NH tropics: slightly overdispersive

12Spread: dash (- - -), RMSE: solid (——)

PQPF error: CRPSS

ECMWF best

In the NH tropics, the day +1 CMA is even poorer than the day +9 ECMWF

Skill of CMC rapidly drops from day +2

13

PQPF skill: BSS and ROC area

Light rain:

CMC best

Heavy rain:

CMC and ECMWF better

CMA is very poor in the NH tropics

14

Attributes diagram (reliability curve, BSS, and BS terms)

NH Midlatitute

Potential economic value

Prob. thresholds of CMC are most reliable

ECMWF has the highest PEV

Performance changes due to major EPS upgrade

Spread: grey, RMSE: black

Changes of spread, Spread/RMSE ratio are significant for CMC and ECMWFChange of RMSE is only significant for CMC (increase)

Performance changes due to major EPS upgrade

Spread: grey, RMSE: black

Changes of spread, Spread/RMSE ratio are significant for UKMO, NCEP, JMAChange of RMSE is only significant for UKMO (decrease)

Score Center Before After Change

Spread CMC-CMA 1.1 4.4 3.3

ECMWF-CMA -0.1 -0.7 -0.6

UKMO-CMA -0.4 -0.2 0.2

NCEP-CMA -1.7 -0.8 0.9

JMA-CMA -1.6 -1.3 0.3

RMSE CMC-CMA -0.4 -0.1 0.3

ECMWF-CMA -0.7 -0.8 -0.1

UKMO-CMA -0.1 -0.3 -0.3

NCEP-CMA 0 -0.2 -0.2

JMA-CMA -0.5 -0.3 0.2

Spread CMC-CMA 0.20 0.59 0.39

/RMSE ECMWF-CMA 0.05 -0.02 -0.07

ratio UKMO-CMA -0.05 0 0.05NCEP-CMA -0.23 -0.10 0.13

JMA-CMA -0.19 -0.16 0.03

Changes due to major EPS upgrade

Use the frozen CMA as the reference to eliminate the interannual variability

CMC greatly increases spread and spread/RMSE ratio

Similar performance changes for other lead times

Changes due to major EPS upgrade of day +3 spread, RMSE, and Spread/RMSE ratio in the NH midlatitude (significant change with 95% confidence interval)

20

Skill changes

After EPS major upgrade:

CMC: decreased skill

ECMWF, UKMO,

NCEP: improved

JMA: no significant

change

Wednesday February 13, 2013 Major upgrade to the Global Ensemble Prediction System (GEPS)

version 3.0.0 at the Canadian Meteorological Centre

“Changes installed uniquely into the forecast component include: adjustments to how physics tendencies are perturbed for convective precipitation; the physics tendencies perturbations are applied at every level except the very last one; addition of diffusion into the advection procedure; perturbation of the bulk drag coefficient in the orographic blocking scheme; and fine tuning of the adjustment factor alpha of the stochastic kinetic energy backscattering scheme.”

http://collaboration.cmc.ec.gc.ca/cmc/CMOI/product_guide/docs/changes_e.html#20130213_geps

21

CMC EPS upgrade

Evaluation of the QPFs and PQPFs from six TIGGE EPSs in the NH midlatitude and tropics during the boreal summers of 2008-2012:

Ensemble mean QPFs:

CMA: large systematic biases, poor performance in the NH tropics

ECMWF: less errors and best skill

JMA: unusually large moist biases of day +1 QPFs in the NH tropics PQPFs:

CMC: relatively good for light precipitation and short lead times,

increased spread and larger errors for longer lead times;

better reliability and reliable probabilistic thresholds in PEV

ECMWF: best skill, except for light precipitation;

best discrimination ability and highest potential economic benefit

NCEP and UKMO: most sharp

22

Summary

The model upgrade in EPS cannot always guarantee skill improvements

The enlarged ensemble spread of CMC forecasts after the upgrade increases the QPF errors

Uncertainties and quality of verification data

How to fairly evaluate an EPS is essential for the development and upgrade of the EPSs

Comprehensive evaluation with multiple verification metrics

Provide general guidance for the postprocessing of the EPSs

Reliability and discrimination ability

23

Summary

E-mail: yuanhl@nju.edu.cn

Thank you