Theoretical Review of Seasonal Predictability

In-Sik Kang

Theoretical Review of Seasonal Predictability

In-Sik Kang

Analysis of Variance of JJA Precipitation Anomalies (SNU case)

(a) Total variance

(b) Forced variance

(c) Free variance

Free variance

Intrinsic transients due to natural variability

Forced variance

Climate signals caused by external forcing

N

ii XX

N 1

2)(1

1

N

i

n

jiij XX

nN 1 1

2)()1(

1

Forced Variance Internal Variance

Signal-to-noise

Decomposition of climate variablesDecomposition of climate variables

Climate state variable (X) consists of predictable and unpredictable part.

Predictable part = signal (Xs) : forced variability

Unpredictable part = noise (Xn) : internal variability

X = Xs + Xn

The dynamical forecast (Y) also have its forced and unforced part.

forecast signal (Ys) : forced variability of model

forecast noise (Yn) : internal variability of model

Y = Ys + Yn

The internal variability (noise) is stochastic

If the forecast model is not perfect, Xs≠Ys. (there is a systematic error)

Prediction skillPrediction skill

)(

),(),(),(),(),()(

)(

))(()(),(

:

:

2

s

nnnsenesnss

nnnsenesnss

nesns

nesns

ns

xV

yxCovyxCovyxCovyxCovxxCovxV

yxyxyxyxxxx

yyxxxxyyxCov

yyxyyyForecast

xxxnObservatio

Noise and Error are not correlated with others

Alpha : regression coeff. of signal

Cor(x,y)= 2/122/1 )()()()(

)(

nes

s

yVyVxVxV

xV

The correlation coefficient is maximized by removing V(ye) and V(yn)

The most accurate forecast will be the SIGNAL of perfect model.

When the forecast is perfect signal, the correlation coefficient is

SNRV

V

V

V

Noise

Signal

Total

Signal

,

1

Maximum prediction skill : potential predictabilityMaximum prediction skill : potential predictability

Maximum prediction skill (= potential predictability of particular predictand) is a function of Signal to Noise Ratio

Cor(x,y)=

2/1

2/1

2/12/122/1

)(

)(

)()(

)(

)()()()(

)(

xV

xV

xVxV

xV

yVyVxVxV

xV

s

s

s

nes

s

Perfect model correlation & Signal to Total variance ratioPerfect model correlation & Signal to Total variance ratio

Z500 winter (C20C, 100 seasons, 4 member)

Although the 4 member is not enough to estimate Potential predictability precisely, the patterns of 2 metrics are quite similar

Strategy of predictionStrategy of prediction

1. Reduction of Noise

• Averaging large ensemble members (if number of ensemble members is infinte, Noise will be zero in the ensemble mea

n)

2. Correct signal

• Improving GCM

• Statistical post-process (MOS)

The strategy of seasonal prediction is to obtain “perfect signal” as close as possible.

(i.e. reducing variance of systematic error and variance of noise)

Ensemble averaging : reduction of noiseEnsemble averaging : reduction of noise

22111

),(

N

R

N

RyxCor Max

Necessary number of ensemble is dependent on the signal to noise ratio

Extratropical forecast needs larger ensemble members than tropics.

Correlation with perfect forecast (perfect signal) N : number of ensemble member

2/12/12/1

2/12/12/122/1

/1)(/)()(1)(/)((

)(/)(

])(

)([)]()([

)(

])(

)([)(

)(

)()()()(

)(

NxVxVxVxV

xVxV

N

xVxVxVxV

xVN

xVxVxV

xV

yVyVxVxV

xV

nsns

ns

nsns

s

ns

s

nes

s

Multi Model EnsembleMulti Model Ensemble

The averaging of large ensembles reduces noise in the forecast.

The multi model ensemble combines ensembles with different forecast signal, the cancellation of systematic bias is possible : a sort of post processing.

Thus the multi model ensemble (MME) can be more efficient ensemble technique to get “perfect signal” : it permits both of noise reduction and signal correction.

And statistically optimized MME technique (eg. superensemble) can be more beneficial in the correction of systematic error like as usual statistical post-processes.

However, there are still debates on benefit of multi model ensemble in seasonal prediction.

□ Pattern Correlation (0-360E, 40S-60N)

MME1

MME2MME3

Single model

(a) 850 hPa Temperature

(b) Precipitation

0.43, 0.44, 0.53

0.40, 0.47, 0.58

MMES (based on point-wise correction, CPPM)MMES (based on point-wise correction, CPPM)

Issues on Multi Model Ensemble predictionIssues on Multi Model Ensemble prediction

Debates on MMEP of Seasonal forecast

Is a multi model system better than a single good model?

(Graham et al. 2000; Peng et al. 2002; Doblas-Reyes et al. 2000)

Is a sophisticated technique better than a simple composite?

(Krishnamurti et al. 2000; Kharin and Zwiers 2002; Pavan and Doblas-Reyes 2000 )

Strong limitation of seasonal predictability study : small samples

MME prediction experiment in a simple climate system

(Krishnamurti et al. 2000; Palmer 1993, 1999; Qin and Robinson 1995)

Forced variability

Internal variability

Design of the simple climate model and predictability experiment

Simple Chaotic Model

Low frequency forcing

Atmospheric process

Simple Chaotic Model 1

Simple Chaotic Model 2

Simple Chaotic Model 9

:

Different parameters

)('''

)*(''

'2)(

1)(

2

1

11

)(

)(2

20

20

2

0

2221

20

2

222

iii

iii

iiij

N

jiji

N

j

N

mmjjjmmjmjmjijmi

ii

iij

N

jiji

N

j

N

mjjmmjmjmjijmii

TSa

h

a

hkkban

haacna

at

kban

haacan

t

Simple chaotic model : 2 layer Nonlinear QG spectral model, Reinhold & Pierrehumbert (1982)

Time varying forcing

(Interannual time scale)

Multimodel ensemble forecast in simple model

Equivalent to the seasonal forecast using multi-AGCMs with prescribed SST

mh

*

0

Modelparameter M1 M2 M3 M4 M5 M6 M7 M8 Obs

0.25 0.25 0.25 0.25 0.25 0.25 0.20 0.30 0.25

0.20 0.20 0.19 0.22 0.20 0.22 0.20 0.21 0.20

k’ 0.015

0.020

0.015

0.020

0.020

0.025

0.015

0.020

0.015

h’’ 0.045

0.045

0.045

0.045

0.035

0.045

0.045

0.055

0.045

0.16 0.14 0.15 0.15 0.15 0.12 0.15 0.14 0.15

Multimodel ensemble forecast in simple modelPhysical parameters of each models

Observation

Case 1 Case 2

Prediction (20 ensembles)

Ens. mean

Prediction experiments : 120cases, 20 ensembles

Interannual variability : a particular wave component

Obs model

Fcst (ensemble member)

Multimodel ensemble prediction schemes

MME1 : simple composite of individual forecast with equal weighting. (special case of MME2)

MME2 (Superensemble) : Optimally weighted composite of individual forecasts. The weighting coefficient is

defined by regression of forecasts and observation during training period.

MME3 : simple composite of individual forecasts, which was corrected by statistical post process

i

iFMP

1

i

iiFaP

i

iFMP ˆ1

Observation

Hindcast

■ Statistical correction (Kang et al. 2004)

SVD

Coupled Pattern

Coupled Pattern New Forecast

Corrected forecast

Projection coeff.

Prediction skills of single and multi model ensemblePrediction skills of single and multi model ensemble

Pattern correlation of indiv. Model and MME (60case avg.)

corrected

MME1 MME2

MME3

MME is not better than a single good model !!

It is due to the inclusion of bad model (M2, M6)

What will happen if bad model excluded in MME?

Which models? : combination of modelsWhich models? : combination of models

skill Models

0.4613 1, 2, 3, 4, 5, 6, 7, 8

0.4601 1, 2, 3, 4, 5, 6, 7, 8

0.4592 1, 2, 3, 4, 5, 6, 7, 8

skill Models

0.2112 1, 2, 3, 4, 5, 6, 7, 8

0.2835 1, 2, 3, 4, 5, 6, 7, 8

0.2842 1, 2, 3, 4, 5, 6, 7, 8

Good combinations (MME1) Bad combinations (MME1)

Combining all available models does not guarantee best skill

“Systematically” bad model needs to be excluded.

skill Models

0.4409 1, 2, 3, 4, 5, 6, 7, 8

0.4407 1, 2, 3, 4, 5, 6, 7, 8

0.4402 1, 2, 3, 4, 5, 6, 7, 8

skill Models

0.2683 1, 2, 3, 4, 5, 6, 7, 8

0.3313 1, 2, 3, 4, 5, 6, 7, 8

0.3315 1, 2, 3, 4, 5, 6, 7, 8

Good combinations (MME2) Bad combinations (MME2)

“Systematically” bad model can be useful : difficult to find good combination

Comparing 219 combinations (3 to 8 models)

Number of models

MME1

MME2

MME3

NT=30

NT=60

NT=90

Composite forecast : more models, better forecasts

Due to overfitting, regression based forecast getting worse with increasing number of models.

When the signal to noise ratio is large, superensemble is more skillful than simple composite.

30 case mean skill averaged over each number of models

Debates on MMEP of Seasonal forecast

Is a multi model system better than a single good model?

It depends on the combination of models, if there are sysetmatically bad model, MME is not better than a single good model.

Is a sophisticated technique better than a simple composite?

When the signal to noise ratio is small, superensemble tends to be unstable due to overfitting. On the other hand, superensemble can be better than a simple composite where the signal to noise ratio ( potential predictability is high) and number of model is not large.

Results of simple model experiments

Summary

Due to the unpredictable noise, the most accurate deterministic forecast is a perfect signal. (ensemble mean of perfect model)

To obtain perfect signal, we have to

- reduce noise in the forecast

- correct forecast signal (systematic error correction)

MME is a reasonable approach to the perfect signal

Composite based MME has some dependency on the combination of models : need to exclude bad models.

Complex MME (Superensemble) have a problem of overfitting in the case of low signal to noise ratio and short historical record.

Generally, Composite based MME is more feasible and skillful. The composite of calibrated forecast is more beneficial.

Probabilistic Seasonal Forecastsand the Economic value

Value of the forecast : accuracy & utility

Forecast is valuable when it is used by decision maker and has some benefits.

Forecast Information $, ¥, ₩,

Decision making

Resolution → Resolution →

UtilityAccuracy

Forecast value

Resolution →

Forecast value = Accuracy x Utility

Choice of forecast system

Decision maker

Forecast system

Forecast system

Forecast system

Forecast system

Forecast system

Choice : the most Valuable forecast system

The form and properties should be matched by a particular situation of user

Climatological probability of event : Pc

Cost-Loss ratio of user : C/L

Flexibility of interpretation by user : Pt

OBSSingle model

Multi model

Probability distribution of summer mean precipitation

□ Distribution of total ensemble members

Probability formulation

Climatological PDF

Ensemble PDF of particular year

0 Xc-Xc

A

B

C

A

B

C

Probability of ABOVE normal

Probability of NORMAL

Probability of BELOW normal

Multi model ensemble probability

Model 1 Model 2

• • •

Collecting all normalized ensemble members

(5 model , 45 samples)

MME1 Probability

MME3 ProbabilityChange the ensemble mean with MME3 in deterministic forecast

μ

μ μ *

μ : ensemble mean

μ *: corrected ensemble mean

Model 3

MME1

Value

Number of occurrence

Reliability Diagram (Above normal)

(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)

Single model

MME1MME3

Fcst probability Fcst probability

Obs

. pro

babi

lity

Reliability Diagram (Below normal)

(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)

Fcst probability Fcst probability

Obs

. pro

babi

lity

Single model

MME1MME3

Economic Value of Prediction System

CONTINGENCY TABLE OF C/L

Hit (h)

Mitigated loss (C+Lu)

Miss (m)

Loss (L=Lp+Lu)

False Alarm (f)

Cost (C)

Correct rejection (c)

No cost (N)

Forecast/action

Ob

serv

atio

n Yes No

Yes

No

C: Total cost r=C/LpLu: Unprotectable loss o: the climatetologicalLp: Protected against frequency of the event

perfectclimate

forecastclimate

EE

EEV

ECONOMIC VAULE : V

GlobalEast Asia

West US

Australia

Eco

nom

ic V

alue

Economic Value as a Function of Cost/Loss Ratio (GCPS)

)(

),(Min

)()(

perfect

climate

forecast

u

uup

upu

LCoE

oLCLLoE

LLmfCLChE

Single model

MME1MME2MME3

Economic value of deterministic forecast (Above normal)

(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)

C/L

Single model

MME1MME3

(a) Monsoon(40E-160E,20S~60N) (b) ENSO (160E-280E,20S~20N)

Economic value of Probabilistic forecast (Above normal)

Black line: MME3 in deterministic forecast

Economic value of forecast (above normal, C/L=0.1)

Deterministic Probabilistic

Generalized application of forecast

Application to the simple decision system : Y or N – 1bit problem.

Observation (real event)

Yes No

Forecast(action)

Yes Cost Cost

No Loss 0

• Probabilistic forecast is converted deterministic forecast using pt

• Benefit of probabilistic forecast cannot be maximized.

Application to the generalized decision system : Action function & Contingency map

Temperature (T)Action function : F(T)

Probability forecast p(T)

Deterministic forecast (T0)

• Action using Det. Forecast : F(T0)

• Action using Prob. Forecast : ∫p(T)F(T)dT

Forecast utilization covering wide range of problem

ForecastObservation

Net

benefit

Contingency map

Action function