uncertainty and ensembles in climate projections

Uncertainty and Ensembles in Climate Projections

llsang Ohn

Seoul National University

February 27, 2016

Uncertainty and Ensembles in Climate Projections February 27, 2016 1 / 38

1 Introduction

2 REA and its Bayesian extension

3 Other Bayesian approaches

4 Challenges


Introduction

1 Introduction



4 Challenges


Introduction

Uncertainty in climate projections

• To aid climate policy decisions, accurate quantitative descriptions of the uncertainty inclimate projections are needed.

• Uncertainty is described in a variety of ways from qualitative statements such as “likely” and“unlikely”, or “low”, “medium” and “high” confidence to quantitative representations like arange of plausible values, a standard deviation or a full probability distribution providingcnfidence bounds.


Introduction

Multi-model ensembles

• Ensemble means a group of comparable climate models.

• The ensemble can be used to gain a more accurate projection of a future climate.

• Variation of the results across the ensemble members gives an estimate of uncertainty.


Introduction

Multi-model ensembles

• Suppose there are M climate models in the ensemble.

• Let Xj be a projection of some current climate variable generated by model j , and Yj aprojection of some future climate variable generated by model j .

• We also have an observation X0 of the true current climate.

• The future climate is estimated by a weighted average of the individual projections, namely,

Y =

∑Mj=1 wjYj∑Mj=1 wj

.

• Various methods have been used to assign weights to climate models

• equal weights• regression methods (if series data are available)• based on a model performance criterion• ...


REA and its Bayesian extension

1 Introduction



4 Challenges



References

• Giorgi, F., & Mearns, L. O. (2002). Calculation of average, uncertainty range, and reliabilityof regional climate changes from AOGCM simulations via the reliability ensembleaveraging(REA) method. Journal of Climate.

• Tebaldi, C., et al. (2005). Quantifying uncertainty in projections of regional climate changeA Bayesian approach to the analysis of multimodel ensembles. Journal of Climate.

• Smith, R. L., Tebaldi, C., Nychka, D., & Mearns, L. O. (2009). Bayesian modeling ofuncertainty in ensembles of climate models. Journal of the American Statistical Association.



Reliability ensemble average (REA)

• Reliability ensemble average (REA) method (Giorgi and Mearns 2002) quantified the twocriteria of “bias” and “convergence” for multimodel evaluation, and produced estimates offuture climate through a weighted average of the individual climate model results.

• In order to estimate change ∆ ≡ Y − X , Giorgi and Mearns used the weighted ensembleaverage

∆ =

∑Mj=1 λj∆j∑Mj=1 λj

,

where REA weight of model j (called reliability) is given by

λj = (λmB,jλ

nD,j)

1/mn =

[min

(1,

ε

|Xj − X0|

)m

×min

(1,

ε

|∆j − ∆|

)n]1/mn

,

with |Xi − X0| being the bias of model j , |∆j − ∆| the convergence of model j , ε a measureof natural variability. The parameters m and n controls the relative importance given to thesetwo quantities (Giorgi and Mearns suggested m = n = 1).

• The REA weights and the weighted ensemble average are iteratively reevaluated.

• Although this procedure appears to lack formal statistical justification, Nychka and Tebaldi(2003) showed that it can be interpreted as a robust estimator, choosing ∆ to minimize∑

j

(ε(m+n)/n

|Xj−X0|

)1/m|∆j − ∆|2−1/n. In the case n = 1, this reduces to a weighted median.



Uncertainty in REA

• Giorgi and Mearns (2002) defined the uncertainty range around the REA average by 2δYwhere

δY =

(∑Mj=1 λj(∆j − ∆)2∑M

j=1 λj

)1/2

.

• Usually the REA method tends to reduce the uncertainty estimate, namely, 1

M

M∑j=1

(∆j −

M∑j=1

∆j

M

)21/2

≥

(∑Mj=1 λj(∆j − ∆)2∑M

j=1 λj

)1/2

.

This is because the contribution of model outliers and/or strongly biased models is effectivelyfiltered out.



Bayesian REA

• In general, to quantify uncertainty probabilistically, it is necessary to statistical techniques toderive PDFs for quantities of interest.

• Bayesian approaches provide the specification of a fully probabilistic statistical model torepresent the structure of the ensemble and the data it generates.

• Tebaldi et al. (2005) proposed a Bayesian statistical model to which the REA weightingscheme corresponds.

• Smith et al. (2009) modified and extended this Bayesian approach.



Bayesian REA

• Data used in Smith et al. (2009)

• mean temperature from current (1961-1990) and future (2071-2100) periods• two seasons: DJF (December, January, February) and JJA (June, July, August)• 22 regions• nine climate models• two emission scenarios, so called A2 and B2, with A2 scenario representing faster growth

and consequently higher emissions



Univariate Bayesian REA

• Gaussian distributions for X0, Xj and Yj are assumed:

X0 ∼ N (µ, λ−10 )

Xj ∼ N (µ, λ−1j )

Yj |Xj ∼ N (ν + β(Xj − µ), (θλj)−1).

• With the exception of λ0 (which is treated as a known constant), these random variablesdepend on unknown parameters whose prior densities are assumed to be as follows:

µ, ν, β ∼ U(−∞,∞)

θ ∼ Gamma(a, b)

λ1, . . . , λM |aλ, bλ ∼ Gamma(aλ, bλ)

aλ, bλ ∼ Gamma(a∗, b∗).

• The mean parameters µ and ν are assumed to be the same for all models. This is because, inthe absence of either (informative) prior knowledge about the performance of each model,bias terms would not be identifiable.



Univariate Bayesian REA

• The posterior distributions for µ and ν aer given by

µ|− ∼ N

(µ,

1

λ0 + (1 + θβ2)∑M

j=1 λj

),

ν|− ∼ N

(ν,

1

θ∑M

j=1 λj

),

respectively, where

µ =λ0X0 +

∑Mj=1 λjXj − θβ

∑Mj=1 λj(Yj − ν − βXj)

λ0 + (1 + θβ2)∑M

j=1 λj

,

ν =

∑Mj=1 λj(Yj − β(Xj − µ))∑M

j=1 λj

.

• The posterior mean of λj is

λj =a + 1

b + 12(Xj − µ)2 + θ

2(Yj − ν − β(Xj − µ))2

.

• The forms are analogous to the REA results.

• Mean temperature change is estimated by ν − µ.



Multivariate Bayesian REA

• A disadvantage of the approach so far is that each of the 22 regions is treated as an entirelyseparate data analysis.

• The data available for any one region consist solely of M climate model projections Xj andYj , plus a single observational value X0, and the analysis is open to the objection that it istrying to produce rather complicated inferences based on a very limited set of data.



Multivariate Bayesian REA

• Xij and Yij represent the current and future projections of model j for the temperatureaverage over region i .

• Gaussian distributions for Xi0, Xij and Yij are assumed:

Xi0 ∼ N (µ0 + ζi , λ−1i0 )

Xij ∼ N (µ0 + ζi + αj , (τijφiλj)−1)

Yij |Xij ∼ N (ν0 + ζ′i + α′j + βi (Xij − µ0 − ζi − αj), (τijθiλj)−1).

• With the exception of λi0 (which is again treated as a known constant), these randomvariables depend on unknown parameters whose prior densities are assumed to be as follows

λj |aλ, bλ ∼ Gamma(aλ, bλ)

τij |c ∼ Gamma(c, c)

αj |φ0 ∼ N (0, φ−10 )

α′j |αj , β0, θ0, φ0 ∼ N (β0αj , (θ0φ0)−1)

µ0, ν0, ζi , ζ′i , βi , β0 ∼ U(−∞,∞)

θi , φi , ψ0, θ0, c, aλ, bλ ∼ Gamma(a, b).

• Mean temperature change is estimated by the posterior mean of ν0 − µ0 + ζ′i − ζi .



Delta T = ν − µ or ν0 − µ0 + ζ′i − ζi



Figure: Predictive distribution for mean temperature change under the multivariate model



• The overall comparison favors the multivariate method as producing tighter predictivedistributions, but this result is not uniform over all regions, so the comparison is notcompletely clear-cut.



Cross validation in Bayesian REA

• Direct validation based on future climate is impossible.

• However, a cross-validation approach is feasible: if we think of the given climate models as arandom sample from the universe of possible climate models, we can ask ourselves how wellthe statistical approach would do in predicting the response of a new climate model.

• If someone gave us a new climate model for which the projected current and future climatemeans were X † and Y †.

• When the paramters µ, ν, β, θ, aλ and bλ are denoted by η, and the dataX0, {Xj}Mj=1, {Yj}Mj=1 by D, the poeterior predictive distribution of ∆† ≡ Y † − X † is given by

p(∆†|D) =

∫p(∆†|η)p(η|D)dη

=

∫ ∫p(∆†|λ†, η)p(λ†|η)p(η|D)dλ†dη

where p(λ†|η) = Gamma(aλ, bλ) and p(∆†|λ†, η) = N (ν − µ, ((β − 1)2 + θ−1)/λ†).



Cross validation in Bayesian REA

1 For each j ∈ {1, . . . ,M}, rerun the Bayesian REA procedure. Let ∆j = Yj − Xj .

2 Let a(n)λ , b

(n)λ , ν(n), µ(n), β(n) and θ(n) be the hyperparameter values corresponding to the nth

draw from the posterior distribution. Then draw λj,n ∼ Gamma(a(n)λ , b

(n)λ ) and calculate

U(n)j = Φ

{∆j − ν(n) + µ(n)√

((β(n) − 1)2 + 1/θ(n))/λj,n

}.

3 Let Uj be the mean value of U(n)j over all n draws from the posterior distribution. This is

therefore an estimate of the predictive distribution function, evaluated at the true ∆j . If themodel is working correctly, Uj should have U(0, 1).

4 Recompute steps 1-3 for each region, so we have a set of test statistics Uij , i = 1, . . . ,R,j = 1, . . . ,M.

5 Plot the Uij ’s to look for systematic discrepancies and apply standard tests of fit, such asKolmogorov-Smirnov, for a formal test that the predictive distribution is consistent with thedata.


Other Bayesian approaches

1 Introduction



4 Challenges



References

• Greene, A. M., Goddard, L., & Lall, U. (2006). Probabilistic multimodel regional temperaturechange projections. Journal of Climate.

• Buser, C. M., Knsch, H. R., Lthi, D., Wild, M., & Schr, C. (2009). Bayesian multi-modelprojection of climate: bias assumptions and interannual variability. Climate Dynamics.



Greene et al. (2006)

• Greene et al. (2006) projected regional temperature using a Bayesian linear model withcovariates being climate model outputs.

• Data used in Greene et al. (2006)

• temperature series data from current (1902-1998) and future (2005-2098) periods• annual, DJF and JJA• 22 regions• 14 climate models• two emission scenarios, A2 and B1




• Let Yit be the observed temperature for year t at region i and Xijt be the temperatureprojected at region i by climate model j for year t.

• Three probability model structures, designated A, B, and C, are considered:

Model A Model B Model C

Yit ∼ N (µit , σ2) Yit ∼ N (µit , σ

2i )

µit = β0i +∑

j βijXijt

β0i ∼ N (µ0, τ20 ), µ0 ∼ N (0, 0, 0001), τ 20 ∼ Gamma(0.001, 0.001)

βij ∼ N (θij , τ2ij ) βi ∼ N (θ,Σ)

• θjk and τjk are not intended to represent hyperparameters; in the estimation process thesemeans and variances are assigned fixed priors. Models A and B thus have no hierarchicalstructure.

• For model 3, a Wishart prior is assigned to Σ−1.



Figure: Projected temperature changes for 2079-98, relative to 1979-98. The two values representing thelatter are the 5th and 95th percentiles of the temperature change probability distribution.




• The main assumption governing this approach is that of stationarity of the relation betweenobserved and simulated trends, estimated in the training period of the twentieth century andapplied to future simulations.

• In fact, this strong assumption causes obvious differences between the simple averageprojections from the GCMs and the projections synthesized from the calibrated ensemble,inmany cases resulting in distributions over a range of values significantly shifted, more oftentowards lower values.



Buser et al. (2009)

• Buser et al. (2009) is a different extension of Tebaldi et al (2005) for time series data.

• Buser et al. (2009) considered climate model biases.

• Data used in Buser et al. (2009)

• temperature series data from current (1961-1990) and future (2071-2100) periods• annual, DJF and JJA• 1 region• 5 climate models• A2 emission scenarios



Buser et al. (2009)

• We denote by X0t the observations in year 1960 + t, by Xjt the current output of model j inyear 1960 + t and by Yjt the future output of model j in year 2070 + t witht = 1, . . . ,T = 30 years. Although the observations Y0t for the years 2070 + t are notavailable, they are included as unobserved data in the model.

• The distibutions of current climate are

X det0t ≡ X0t − γ(t − T0)

iid∼ N (µ, σ2)

X detjt ≡ Xjt − γ(t − T0)

iid∼ N (µ+ βj , σ2b2

j )

with T0 = (T + 1)/2. The iid assumption reflects that the different data series stem fromindependent realizations of the same climate state.

• The distibution of future climate is

Y det0t ≡ Y0t − (γ + ∆γ)(t − T0)

iid∼ N (µ+ ∆µ, σ2q2)

but for Y detjt ≡ Yjt − (γ + ∆γ)(t − T0), the two distributions are assumed:

• constant bias: Y detjt

iid∼ N (µ+ ∆µ+ βj + ∆βj , σ2q2b2

j q2bj

)

• constant relation: Y detjt

iid∼ N (µ+ bj∆µ+ βj + ∆βj , σ2q2b2

j q2bj

)



Buser et al. (2009)

• Note that µ+ βj − µ = βj and b2j σ

2/σ2 = b2j .

• Since ∆βj ≈ 0 and qbj ≈ 1 (due to the prior),

µ+ ∆µ+ βj + ∆βj − (µ+ ∆µ) ≈ βj

µ+ bj∆µ+ βj + ∆βj − (µ+ ∆µ) ≈ (bj − 1)∆µ+ βj .



Prior distributions for parameters:



Other approaches

• Furrer, R., Sain, S. R., Nychka, D., & Meehl, G. A. (2007). Multivariate Bayesian analysis ofatmosphereocean general circulation models. Environmental and ecological statistics.

• used a spatial process of sphere for grid data

• Tebaldi, C., & Sanso, B. (2009). Joint projections of temperature and precipitation changefrom multiple climate models: a hierarchical Bayesian approach. Journal of the RoyalStatistical Society: Series A.

• derived posterior distributions for the joint projections of future temperature andprecipitation trends and changes by applying a Bayesian hierachical model


Challenges

Challenges

1 PDFs for ensembles are not guaranteed to represent the full range of uncertainty, which hasled to questions about the relevance of the resulting probabilities to the downstream user.

2 The idea that the performance can be improved by combining results from multiple models isbased on the fundamental assumption that errors tend to cancel if the models areindependent, and thus uncertainty should decrease as the number of models increases. But,models are similar in many respects: grids, theoretical arguments for their parametrizations,numerical methods to solve the equations, etc.

3 There are several extensions:

• considering emission scenario uncertainty• modelling spatial-temporal data• considering the climate variables of shoter periods (e.g. monthly)


uncertainty and ensembles in climate projections

Data & Analytics