Preliminary Skill Comparisons AmongIRI-Multi-Model Ensemble-,CDC-CCA- Hindcasts and

CPC-Official Seasonal Temperature Forecasts

Ed O’Lenic (CPC), Wesley Ebisuzaki (CPC), Marty Hoerling (CDC), Lisa

Goddard (IRI)28th Climate Diagnostics and Prediction Workshop

Reno, NevadaOctober 20, 2003

Outline• US-average skill of categorical and probabilistic seasonal

forecasts of US surface temperature is examined.• Hindcasts of IRI multi-model ensemble (MM) from 1998-

2003 and CDC-CCA (CCA) from 1995 to 2003 are compared with CPC-Official forecasts (OFF) from 1995 to 2003.

• This comparison serves as a sanity check on the verification system and points out advantages and disadvantages of the several techniques.

• Among the recommendations are: *CPC forecasts may be too conservative (encouraged by the skill metric we use) in areal coverage and in probabilities, *MM and CCA probabilities appear too large, *MM and CCA categorical forecasts may be better than 1/3 in cl regions, *MM and CCA are sufficiently skillful to serve as a first-guess forecast for CPC’s seasonal forecast system.

Cautionary Notes• CPC (OFF) forecasts are the original operationally- issued

forecasts, while those from IRI (MM) and CDC (CCA) are hindcasts. • We do not know for certain whether these hindcasts would be as

good as they are if they were issued operationally.

• MM occasionally predicts CL (EC), CCA does not, OFF uses CL alot.

• MM and CCA predict the near normal category, OFF almost never does.

• The sample size of these datasets is small and results are likely not statistically significant.

• CL and EC are used interchangeably.

Model Descriptions

•MM – IRI’s fully-automated multi-model ensemble of GCM forecasts. Data begins in February, 1998. Hindcasts.

•CCA – CDC’s fully-automated technique which uses CCA to predict what a combination of several models would predict as a function of sst. Forced by presisted ssts. Data begins January, 1995. Hindcasts.

•OFF – CPC’s subjective official forecasts. Based on statistical and dynamical forecast tools, including MM and CCA, along with Constructed Analogs, individual model forecasts, Trend, Soil Moisture and ENSO and ENSO-neutral composites. Data from Jan 1995. Operational Forecasts.

• These are available at:http://www.cpc.ncep.noaa.gov/products/predictions/90day/tools/briefing/index.pri.html

Skill Scores

• Non-EC stations: s=((c-e)/(t-e))*100• All Stations : s=((c+(1/3)*cl-e)/(t-

e))*100 c = # stations correct e = # stations correct by chance t = # stations in total cl= # stations predicted equal chances

• rpss = 1- rps/rpsclimate

• rps = (1/(M-1)) [(pu) – (ou)]

2

u=1

m

u=1

m

M=1

M

A sample forecast: JFM 1999

CCA

MM

OFF

OBS

PRECTEMP

MMcl

CCAcl

MMNo-cl

CCANo-cl

95 96 97 98 99 00 01 02 03

95 96 97 98 99 00 01 02 03

95 96 97 98 99 00 01 02 03

95 96 97 98 99 00 01 02 03

MM (+), better 17

OFF (+), better 13

MM (+) OFF (-) 16

OFF (+) MM (-) 6

6

6

0.06 vs 0.13

MM (+), better 8

10

OFF (+), better 22

OFF (+) MM(-) 7

3

MM (+) OFF(-) 15

0.12 vs 0.12

OFF (+), better 18

CCA (+), better 28OFF (+) CCA (-) 16

CCA (+) OFF (-) 203

15

0.08 vs 0.12

CCA (+), better 14OFF (+) CCA(-) 16

CCA (+) OFF(-) 20

OFF (+), better 33

10

7

0.17 vs 0.12

CCA and MM probabilisticforecasts appear too liberal, OFF too conservative. RPSS of all have declined over the last 2 years

RPSS=0.02 RPSS=0.00

RPSS=0.00

MM

OFF CCA

Conclusions, Recommendations• CPC uses non-Cl T skill as its metric. This encourages forecasting

small areas. This is bad only if generally increased area is justified. We did better before 1997. Is a better set of skill masks needed?

• Purely numerical MM does as well in non-Cl T forecasts as combined statistical/numerical CPC.

• CPC does better than CCA in non-Cl T forecasts. • Both CCA and MM do better than CPC in Cl T forecasts. This appears

to indicate CCA and MM score better than 1/3 in those areas.• CCA Cl T forecasts have high skill when CPC skill is low.• Should areas of EC should be reduced in CPC forecasts (CPC too

conservative)?• The decline of skill in all the tools over the last 2 years indicates

reduced predictability.• All three probabilistic forecast systems can be improved by

calibration.• Differences among forecast skills likely not statistically significant.• MM and CCA should be combined and used by CPC as a first guess.