estimation de la fonction de covariance dans le …fbachoc/redice_06...statistical model (1/2)...

.

Estimation de la fonction de covariance dans lemodèle de Krigeage et applications: bilan de thèse

et perspectives

François BachocJosselin Garnier

Jean-Marc Martinez

CEA-Saclay, DEN, DM2S, STMF, LGLS, F-91191 Gif-Sur-Yvette, FranceLPMA, Université Paris 7

Juin 2013

. Estimation de la fonction de covariance dans le modèle de Krigeage et applications: bilan de thèse et perspectives 1/49 .

.The PhD

PhD started in October 2010

Two components in the PhDI Use of Kriging model for code validation and metamodelingI Work on the problem of the covariance function estimation


.Kriging for code validation and metamodeling

Statistical model and method for code validationApplication to the FLICA 4 thermal-hydraulic codeGERMINAL code meta-modeling

Maximum Likelihood and Cross Validation for hyper-parameter estimation

Finite sample analysis of ML and CV under model misspecification

Asymptotic analysis of ML and CV in the well-specified caseAsymptotic frameworkConsistency and asymptotic normalitySketch of proofAnalysis of the asymptotic variance

Conclusion


.Numerical code and physical system

A numerical code, or parametric numerical model, is represented by afunction f :

f : Rd × Rm → R(x , β) → f (x , β)

Observations can be made of a physical system Yreal

xi → Yreal → yi

I The inputs x are the experimental conditionsI The inputs β are the calibration parameters of the numerical codeI The outputs f (xi , β) and yi are the variable of interest

A numerical code modelizes (gives an approximation of) a physical system


.Statistical model (1/2)

Statistical model followed in the phd

I The physical system Yreal is a deterministic functionI There exist a correct parameter β and a model error function Z so that

Yreal (x) = f (x , β) + Z (x)

I The observations are noised Yi = Yreal (xi ) + εi , where εi are i.i.dcentered Gaussian variables

The model is different from other classical models for inverse problemswhere

I Yi = f (xi , βi ) + εi . The βi are i.i.d realizations of the random vector βI Hence the physical system is random

Fu S, An adaptive kriging method for characterizing uncertainty ininverse problems, Journée du GdR MASCOT-NUM - 23 mars 2011.

de Crécy A, A methodology to quantify the uncertainty of the physicalmodels of a code, Rapport CEA DEN/DANS/DM2S/STMF.


.Statistical model (2/2)

I Z is modeled as the realization of a centered Gaussian processI A Bayesian modelling is possible for the correct parameter β

I no prior information case. β is constant and unknownI prior information case. β is the realization of a Gaussian vectorN (βprior ,Qprior )

I Linear approximation of the code w.r.t β

f (x , β) =mX

j=1

hj (x)βj

Eventually, the statistical model is a Kriging model

Yi =mX

j=1

hj (xi )βj + Z (xi ) + εi

Hence calibration and prediction are carried out within the Kriging framework







Conclusion


.Results with the thermal-hydraulic code Flica IV (1/2)

The experiment consists in pressurized and possibly heated water passingthrough a cylinder. We measure the pressure drop between the two ends ofthe cylinder.Quantity of interest : The part of the pressure drop due to friction : ∆PfroTwo kinds of experimental conditions :

I System parameters : Hydraulic diameter Dh, Friction height Hf , Channelwidth e

I Environment variables : Output pressure Ps , Flowrate Ge, Parietal heatflux Φp , Liquid enthalpy hl

e, Thermodynamic title X eth, Input temperature

Te

We dispose of 253 experimental resultsImportant : Among the 253 experimental results, only 8 different systemparameters→ Not enough to use the Gaussian processes model forprediction for new system parameters→We predict for new environmentvariables only


.Results with the thermal-hydraulic code Flica IV (2/2)

RMSE 90% Confidence IntervalsNominal code 661Pa 234/253 ≈ 0.925

Gaussian Processes 189Pa 235/253 ≈ 0.93

- 4000

- 3000

- 2000

- 1000

0

1000

0 50 100 150 200 250

Resultats predict ion Nominal

Indice

erreur de predict ionIntervalles de confiance 90%

- 4000

- 3000

- 2000

- 1000

0

1000

0 50 100 150 200 250

Resultats predict ion Processus Gaussiens

Indice

erreur de predict ionIntervalles de confiance 90%


.Influence of the linear approximation

I The Gaussian process model of the model error is also tractable in thenon-linear case

M. J. Bayarri, J.O. Berger, R. Paulo, J. Sacks, J.A. Cafeo, J.Cavendish, C.H. Lin and J. Tu A framework for validation ofcomputer models, Technometrics, 49 (2), 138-154.

I On the FLICA 4 data, we compare the linear approximation we use withthe Bayes formula in the non-linear case for calibration and prediction.

I Integrals are evaluated on a 5× 5 grid in the calibration parameter spaceI The same grid is used for the linear-case

I We obtainI a 10% difference for calibrationI a 1% difference for prediction

I In this case, the model error takes the linearization error into account


.Conclusion on code Validation

I We can improve the prediction capability of the code by completing thephysical representation with a statistical model

I Number of experimental results needs to be sufficient. No extrapolationI More influence of the non-linearity of the code for calibration than for

prediction

For more details

Bachoc F, Bois G, Garnier J and Martinez J.M, Calibration andimproved prediction of computer models by universal Kriging, Acceptedin Nuclear Science and Engineering, http ://arxiv.org/abs/1301.4114v2







Conclusion


.GERMINAL code meta-modeling (1/3)

ContextI GERMINAL code : simulation of the thermal-mechanical impact of the

irradiation on a nuclear fuel pinI Its utilization is part of a multi-physics and multi-objective optimization

problem from reactor core designI PhD thesis of Karim Ammar (CEA, DEN)

Characteristics for KrigingI 12 inputsI 2 outputs : a simple and a complicated relationshipI Numerical instabilities −→ nugget effect to estimateI Large data bases (thousands)



Simple Kriging model, Matérn ( 32 ) covariance function with nugget effect,

estimation by Maximum Likelihood

Summary of the resultsI For estimation : the correlation lengths and the nugget effect make

senseI For prediction : good prediction results compared to neural network

methods (+ predictive variances).I Virtual Leave-One-Out standardized errors for detection of outlier

GERMINAL calculations : confirmed by a new version of GERMINALI The size of the data base is a computational problem for Kriging

estimation and prediction



From the point of view of GERMINAL usersI Kriging gives good prediction (RMSE) compared to neural networksI However, the cost of a call to the metamodel function is significantly

more costly (0(n))I We have not investigated Kriging methods dedicated to large data bases

I The predictive variances are considered a "plus" of Kriging and theLeave-One-Out for outlier detection has been appreciated

For more detailsI In the PhD manuscript (planned for September)

I Karim AMMAR, Edouard HOURCADE, Cyril PATRICOT, FrançoisBACHOC and Jean-Marc MARTINEZ Improvement ofsupercomputing based core design process with parallelestimations and statistical analysis, Submitted proceeding to theJoint International Conference on Supercomputing in NuclearApplications and Monte Carlo 2013, Paris, 27-31 October, 2013







Conclusion


.Covariance function estimation

ParameterizationCovariance function model

˘σ2Kθ, σ2 ≥ 0, θ ∈ Θ

¯for the Gaussian Process

Y .I σ2 is the variance hyper-parameterI θ is the multidimensional correlation hyper-parameter. Kθ is a stationary

correlation function.

EstimationY is observed at x1, ..., xn ∈ X , yielding the Gaussian vectory = (Y (x1), ...,Y (xn)).Estimators σ2(y) and θ(y)

"Plug-in" Kriging prediction

1 Estimate the covariance function

2 Assume that the covariance function is fixed and carry out the explicitKriging equations


.Maximum Likelihood for estimation

Explicit Gaussian likelihood function for the observation vector y

Maximum LikelihoodDefine Rθ as the correlation matrix of y = (Y (x1), ...,Y (xn)) undercorrelation function Kθ .The Maximum Likelihood estimator of (σ2, θ) is

(σ2ML, θML) ∈ argmin

σ2≥0,θ∈Θ

1n

„ln (|σ2Rθ|) +

1σ2

y t R−1θ y

«


.Cross Validation for estimation

I yθ,i,−i = Eσ2,θ(Y (xi )|y1, ..., yi−1, yi+1, ..., yn)

I σ2c2θ,i,−i = varσ2,θ(Y (xi )|y1, ..., yi−1, yi+1, ..., yn)

Leave-One-Out criteria we study

θCV ∈ argminθ∈Θ

nXi=1

(yi − yθ,i,−i )2

and

1n

nXi=1

(yi − yθCV ,i,−i )2

σ2CV c2

θCV ,i,−i

= 1⇔ σ2CV =

1n

nXi=1

(yi − yθCV ,i,−i )2

c2θCV ,i,−i


.Virtual Leave One Out formula

Let Rθ be the covariance matrix of y = (y1, ..., yn) with correlation functionKθ and σ2 = 1

Virtual Leave-One-Out

yi − yθ,i,−i =

“R−1θ y

”i“

R−1θ

”i,i

and c2i,−i =

1

(R−1θ )i,i

O. Dubrule, Cross Validation of Kriging in a Unique Neighborhood,Mathematical Geology, 1983.

Using the virtual Cross Validation formula :

θCV ∈ argminθ∈Θ

1n

y t R−1θ diag(R−1

θ )−2R−1θ y

andσ2

CV =1n

y t R−1θCV

diag(R−1θCV

)−1R−1θCV

y







Conclusion


.Objectives

We want to study the cases of model misspecification, that is to say thecases when the true covariance function K1 of Y is far fromK =

˘σ2Kθ, σ2 ≥ 0, θ ∈ Θ

¯In this context we want to compare Leave-One-Out and Maximum Likelihoodestimators from the point of view of prediction mean square error andpoint-wise estimation of the prediction mean square error

We proceed in two stepsI When K =

˘σ2K2, σ

2 ≥ 0¯

, with K2 a correlation function, and K1 thetrue unit-variance covariance function : theoretical formula andnumerical tests

I In the general case : numerical studies


.Case of variance hyper-parameter estimation

I Y (xnew ) : Kriging prediction with fixed misspecified correlation functionK2

I Eh(Y (xnew )− Y (xnew ))2|y

i: conditional mean square error of the

non-optimal predictionI One estimates σ2 by σ2.I Conditional mean square error of Y (xnew ) estimated by σ2c2

xnew withc2

xnew fixed by K2

The RiskWe study the Risk criterion for an estimator σ2 of σ2

Rσ2,xnew= E

»“Eh(Y (xnew )− Y (xnew ))2|y

i− σ2c2

xnew

”2–

−→ Explicit formula for estimators of σ2 that are quadratic forms of theobservation vector


.Summary of numerical results

For variance hyper-parameter estimation

I We make the distance between K1 and K2 vary, starting from 0I For not too regular design of experiments : CV is more robust than ML

to misspecificationI Larger variance but smaller bias for CVI The bias term becomes dominating when K1 6= K2

I For regular design of experiments, CV is less robust to modelmisspecification

For variance and correlation hyper-parameter estimation

I Numerical study on analytical functionsI Confirmation of the results of the variance estimation case

Bachoc F, Cross Validation and Maximum Likelihood estimations ofhyper-parameters of Gaussian processes with model misspecification,Computational Statistics and Data Analysis 66 (2013) 55-69,http ://dx.doi.org/10.1016/j.csda.2013.03.016.







Conclusion


.Framework and objectives

EstimationWe do not make use of the distinction σ2, θ. Hence we use the set{Kθ, θ ∈ Θ} of stationary covariance functions for the estimation.

Well-specified modelThe true covariance function K of the Gaussian Process belongs to the set{Kθ, θ ∈ Θ}. Hence

K = Kθ0 , θ0 ∈ Θ

Objectives

I Study the consistency and asymptotic distribution of the CrossValidation estimator

I Confirm that, asymptotically, Maximum Likelihood is more efficientI Study the influence of the spatial sampling on the estimation


.Spatial sampling for hyper-parameter estimation

I Spatial sampling : Initial design of experiment for KrigingI It has been shown that irregular spatial sampling is often an advantage

for hyper-parameter estimation

Stein M, Interpolation of Spatial Data : Some Theory for Kriging,Springer, New York, 1999. Ch.6.9.

Zhu Z, Zhang H, Spatial sampling design under the infillasymptotics framework, Environmetrics 17 (2006) 323-337.

I Our question : Is irregular sampling always better than regular samplingfor hyper-parameter estimation ?


.Asymptotics for hyper-parameters estimation

Asymptotics (number of observations n→ +∞) is an area of activeresearch (Maximum-Likelihood estimator)

Two main asymptotic frameworks

I fixed-domain asymptotics : The observations are dense in a boundeddomainFrom 80’-90’ and onwards. Fruitful theory

Stein, M., Interpolation of Spatial Data Some Theory for Kriging,Springer, New York, 1999.

However, when convergence in distribution is proved, the asymptoticdistribution does not depend on the spatial sampling −→ Impossible tocompare sampling techniques for estimation in this context

I increasing-domain asymptotics : A minimum spacing exists between theobservation points −→ infinite observation domain.Asymptotic normality proved for Maximum-Likelihood under generalconditions

Sweeting, T., Uniform asymptotic normality of the maximumlikelihood estimator, Annals of Statistics 8 (1980) 1375-1381.

Mardia K, Marshall R, Maximum likelihood estimation of modelsfor residual covariance in spatial regression, Biometrika 71 (1984)135-146.


.Randomly perturbed regular grid

I Our sampling model : regular square grid of step one in dimension d ,(vi )i∈N∗ . The observation points are the vi + εXi . The (Xi )i∈N∗ are iidand uniform on [−1, 1]d

I ε ∈]− 12 ,

12 [ is the regularity parameter. ε = 0 −→ regular grid. |ε| close

to 12 −→ irregularity is maximal

Illustration with ε = 0, 18 ,

38

1 2 3 4 5 6 7 8

12

34

56

78

2 4 6 8

24

68

2 4 6 8

24

68







Conclusion


.Main assumptions (1/2)

Control of the derivatives

Kθ (t) ≤C

1 + |t |d+1,

∀i,∂

∂θiKθ (t) ≤

C1 + |t |d+1

,

∀i, j,∂

∂θi

∂

∂θjKθ (t) ≤

C1 + |t |d+1

,

∀i, j, k ,∂

∂θi

∂

∂θj

∂

∂θkKθ (t) ≤

C1 + |t |d+1

,

Positive continuous Fourier transform

I Kθ has a Fourier transform KθI (θ, f )→ Kθ (f ) is strictly-positive on Θ× Rd .


.Main assumptions (2/2)

Set of interpoint spacings explored by the samplingDε := ∪v∈Zd\0

`v + [−2ε, 2ε]d

´Identifiability

I GlobalI For ε = 0, there does not exist θ 6= θ0 so that Kθ (v) = Kθ0 (v) for all

v ∈ Zd

I For ε 6= 0 there does not exist θ 6= θ0 so that Kθ = Kθ0 a.s. on Dε, andKθ (0) = Kθ0 (0)

I LocalI For ε = 0, there does not exist vλ = (λ1, ..., λp) 6= 0 so thatPp

k=1 λk∂∂θk

Kθ0 (v) = 0 for all v ∈ Zd

I For ε 6= 0 there does not exist vλ = (λ1, ..., λp) 6= 0 so thatPpk=1 λk

∂∂θk

Kθ0 = 0 a.s. on Dε, andPp

k=1 λk∂∂θk

Kθ0 (0) = 0

Correlation function family (only for Cross Validation)∀θ ∈ Θ, Kθ(0) = 1

Assumptions verified by all classical stationary covariance function families


.Consistency and asymptotic normality

For ML

I a.s convergence of the random Fisher information : The random trace1n Tr

“R−1 ∂R

∂θiR−1 ∂R

∂θj

”converges a.s to the element (IML)i,j of a p × p

strictly-positive deterministic matrix IML as n→ +∞I asymptotic normality : With VML = 2I−1

ML

√n“θML − θ0

”→ N (0,VML)

For CVSame result with more complex random traces for asymptotic covariancematrix VCV







Conclusion


.Notations

I X : the random vector (X1, ...,Xn) of the perturbationsI x : A vector of ([−1, 1]d )n, as a realization of XI y : The random vector (Y (X1), ...,Y (Xn))

I Rθ := covθ(y |X) : The random covariance matrix

I Lθ := 1n

“ln (|Rθ|) + y t R−1

θ y”

: the ML criterion (to minimize)

I CVθ := 1n y t R−1

θ diag(R−1θ )−2R−1

θ y : the CV criterion (to minimize)


.Results on random matrices (1/)

Control of the eigenvalues

I The eigenvalues of Rθ , ∂∂θi

Rθ , ∂∂θi

∂∂θj

Rθ and ∂∂θi

∂∂θj

∂∂θk

Rθ are

upper-bounded uniformly in n, x , θ.I Because, e.g,

Pj∈N∗,j 6=i Kθ

˘vi − vj + ε

`xi − xj

´¯is bounded uniformly in

x, θI The eigenvalues of Rθ are lower-bounded uniformly in n, x , θ.

I Comes from

nXi,j=1

αiαj Kθ(vi + xi , vj + xj ) =

ZRd

Kθ(f )

˛˛

nXi=1

αi e“

Jf t (vi +xi )” ˛˛

2

df



Class of matrix involved in the ML and CV criteriaLet a matrix sequence M, whose expression uses Rθ , R−1

θ , ∂∂θi

Rθ ,∂∂θi

∂∂θj

Rθ , ∂∂θi

∂∂θj

∂∂θk

Rθ , the matrix product, the diag operator and the

matrix diag(R−1θ )−1.

e.g, M = R−1θ

∂∂θi

RθR−1θ

∂∂θj

Rθ .

e.g, M = R−1θ diag(R−1

θ )−2R−1θ

Control of eigenvaluesThe matrices M above have their eigenvalues upper-bounded uniformly inn, x , θ.



Almost sure convergence of random traces1n Tr(M) converges a.s. to a deterministic limit S.

Sketch of proof

I Make the approximation that the Gaussian process Y is composed of apartition of independent Gaussian processes

I This boils down to approximating M of size n ≈ n1n2 by

M ≈ Mn1,n2 :=

0BBBBB@M(1)

n1

M(2)n1

. . .

M(n2)n1

1CCCCCAI | 1n Tr(M)− 1

n Tr(Mn1,n2 )| →n1,n2→+∞ 0

I The M(i)n1

are iid so that 1n Tr(Mn1,n2 )− E

“1n Tr(M(1)

n1)”→n2→+∞ 0

I Conclude by letting n1, n2 → +∞ and by using the Cauchy criterion



Convergence of random quadratic forms1n y t My converges in mean square to 1

n Tr(MRθ0 )

Asymptotic normality of random quadratic formsWhen Tr(M) = 0, let S be the almost sure limit of 1

n Tr(MRθ0 MRθ0 )

Then 1√n

y t My converges in law to a N (0, 2S)

Sketch of proofLet L(zi |X) =iid N (0, 1). Then

1√

ny t My =

1√

n

nXi=1

φi (MRθ0 )z2i

We then use an almost sure (with respect to X ) Lindeberg-Feller criterion.


.Asymptotic normality

I After having proved consistency and that the almost sure limit of ∂2

∂2θLθ0

is a strictly-positive matrixI Using the results of random matrices above, we directly show that

∂

∂θLθ0 →L N (0, 2IML)

and∂2

∂2θLθ0 →p IML

I We conclude using standard M-estimator techniques.I Same method for CV


.Strictly-positive second derivative

Strictly-positive second derivative (with d = 1)There exist A > 0 so that, uniformly in n,X , θ

E(∂2

∂θ2Lθ0 |X) ≥ A

Xi∈N∗|∂

∂θKθ0 (vi + Xi )|2

andP

i∈N∗ |∂∂θ

Kθ0 (vi + Xi )|2 converges in probability to

IP

v∈Zd | ∂∂θKθ0 (v)|2 if ε = 0

IR

DεfT (t)| ∂

∂θKθ0 (t)|2 + | ∂

∂θKθ0 (0)|2 if ε 6= 0

(with fT the triangular pdf on [−2ε, 2ε]d )We conclude with the identifiability assumption.Same method for CV

Generalization to d > 1Consider the covariance function familyn

K(θ0)1+δλ1,...,(θ0)p+δλp , δinf ≤ δ ≤ δsup

o







Conclusion


.Objectives

The asymptotic covariance matrix VML,CV depend only on the regularityparameter ε.−→ in the sequel, we study the functions ε→ VML,CV

Small random perturbations of the regular gridWe study

“∂2

∂ε2 VML,CV

”ε=0

Closed form expression for ML for d = 1 using Toeplitz matrix sequencetheoryOtherwise, it is calculated by exchanging limit in n and derivatives in ε

Large random perturbations of the regular gridWe study ε→ VML,CVClosed form expression for ML and CV for d = 1 and ε = 0 using Toeplitzmatrix sequence theoryOtherwise, it is calculated by taking n large enough


.Small random perturbations of the regular grid

Matèrn model. Dimension one. One estimated hyper-parameter.Levels plot of (∂2

εΣML,CV )/ΣML,CV in `0 × ν0

Top : MLBot : CVLeft : ˆ (ν0 known)Right : ν (`0 known)

0.5 1.0 1.5 2.0 2.51

23

45

l0

ν 0−130

−79

−39

−13

−0.85

2

0.5 1.0 1.5 2.0 2.5

12

34

5

l0

ν 0

−700

−400

−190

−52

−0.68

30

0.5 1.0 1.5 2.0 2.5

12

34

5

l0

ν 0

−130

−79

−39

−13

−0.85

2

0.5 1.0 1.5 2.0 2.5

12

34

5l0

ν 0

−700

−400

−190

−52

−0.68

30

There exist cases of degradation of the estimation for small perturbation forML and CV. Not easy to interpret


.Large random perturbations of the regular grid

Plot of ΣML,CV . Top : ML. Bot : CV.From left to right : (ν, `0 = 0.5, ν0 = 2.5), (ˆ, `0 = 2.7, ν0 = 1), (ν, `0 = 2.7,ν0 = 2.5)

−0.4 −0.2 0.0 0.2 0.4

1000

020

000

3000

040

000

5000

0

n=200n=400local expansion

epsilon

asym

ptot

ic v

aria

nce

−0.4 −0.2 0.0 0.2 0.47.

357.

407.

457.

507.

55


epsilon

asym

ptot

ic v

aria

nce

−0.4 −0.2 0.0 0.2 0.4

6.8

7.0

7.2

7.4

7.6


epsilon

asym

ptot

ic v

aria

nce

−0.4 −0.2 0.0 0.2 0.4

1000

020

000

3000

040

000

5000

0


epsilon

asym

ptot

ic v

aria

nce

−0.4 −0.2 0.0 0.2 0.4

4850

5254

5658


epsilon

asym

ptot

ic v

aria

nce

−0.4 −0.2 0.0 0.2 0.4

4243

4445

4647

4849


epsilonas

ympt

otic

var

ianc

e


.Conclusion on the well-specified case

I CV is consistent and has the same rate of convergence than MLI We confirm that ML is more efficientI Irregularity in the sampling is generally an advantage for the estimation,

but not necessarilyI With ML, irregular sampling is more often an advantage than with CVI Large perturbations of the regular grid are often better than small ones for

estimationI Keep in mind that hyper-parameter estimation and Kriging prediction are

strongly different criteria for a spatial sampling

For further details :

Bachoc F, Asymptotic analysis of the role of spatial sampling forhyper-parameter estimation of Gaussian processes, Submitted,available at http ://arxiv.org/abs/1301.4321.


.Conclusion on covariance function estimation

General conclusionI ML preferable to CV in the well-specified caseI In the misspecified-case, with not too regular design of experiments :

CV preferable because of its smaller biasI In both misspecified and well-specified cases : the estimation benefits

from an irregular samplingI The variance of CV is larger than that of ML in all the cases studied.


.

Thank you for your attention !


estimation de la fonction de covariance dans le …fbachoc/redice_06...statistical model (1/2)...

Documents