stochastic local interaction models and space-time ......the errors of the karhunen-loeve expansion...

Intro SSRF SSRF-1d SSRF-2/3d SSRF-Appl SSRF-GMRF SLI SLI-Appl S-T Conclu Biblio

Stochastic Local Interaction Models andSpace-Time Covariance Functions based on

Linear Response Theory

Dionissios T. Hristopulos

Geostatistics Laboratory, School of Mineral Resources EngineeringTechnical University of Crete, Chania, Greece

Stochastic Weather Generators Workshop, May 17-20, 2016University of Bretagne Sud, Vannes, France

A G

¥ ¥¥ ¥ ¥ ¥ ¥¥ ¥ ¥¥¥ ¥¥

A G

¥ ¥¥ ¥ ¥ ¥ ¥¥ ¥ ¥¥¥ ¥¥

Technical University of Crete Geostatistics Laboratory: http://www.geostatistics.tuc.gr/4908.html

Dionisis Hristopulos: [email protected] Local Interaction & Linear Response Models 1/34


Motivation for Local Random Field Models

I Spatial dependence can be encoded using local interactions inGibbs random fields

I Locality⇒ Sparse precision matrix⇒ Computational efficiency

I Locality leads to rational spectral densities

I Rational spectral density⇒ Connection with SPDEs (similar toMatern random fields)

I Linear response theory⇒ Extension to space-time SPDEs




A Brief Roadmap of the Presentation




Statistical Field Theories are Local and can be usedas Geostatistical Models

I In geostatistics, we write the joint pdf of spatial data as

fX ∝ exp

− N∑i,j=1

xi C−1x ; i,j xj

I In statistical field theories, the joint pdf of random fields is

determined from spatial interactions, i.e.,

fX ∝ exp(−∫

d~s φ[x(~s)])

I A local framework: Geostatistical models based on localinteractions: Spartan Spatial Random Fields (SSRFs)

I Motivation:1. Flexible parametrization of spatial dependence2. Efficient parameter estimation3. Interpolation and simulation of large datasets




Spartan Spatial Random Fields (SSRFs) are GibbsRFs with Local Structure

I Gibbs probability density function (PDF)

f [x(~s)] = Z−1 e−H[x(~s)], H[x(~s)] : energy functional,

I Z: partition function Z =∫Dx(~s) e−H[x(~s)]

I Hfgc[x(~s)] =∫ d~s

2η0ξd

{[x(~s)

]2+ η1 ξ

2[∇x(~s)

]2+ ξ4

[∇2x(~s)

]2}I Properties: Gaussian, zero-mean, stationary, isotropic SRF

FGC-SSRF Coefficientsη0 : scale, η1 : rigidity, ξ: characteristic length; kc : spectral cutoff




SSRF Covariance for Time Series and Drill-hole Data

G(h) =η0

4e−hβ2

[cos(hβ1)

β2+

sin(hβ1)

β1

], |η1| < 2

G(h) = η0(1 + h)

4 eh, η1 = 2

G(h) =η0

2 ∆

( e−hω1

ω1−

e−hω2

ω2

), η1 > 2

NotationI h = |r |/ξ : normalized lag

I β1,2 =(|2∓η1|

4

)1/2

I ω1,2 =(|η1∓∆|

2

)1/2

I ∆ = |η21 − 4|

12

d=1

0 1 2 3 4 5−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Distance lag

Cor

rela

tion

η

1=−1.9

η1=−1

η1=1

η1=16

Hristopulos and Elogne (2007), IEEETransactions on Information Theory,

53(12), 4667-4679




Classical Damped Harmonic Oscillator in a Heat Bathis a one-dimensional SSRF

Langevin equation: x(t) + Γ x(t) + ω20 x(t) = ε(t), G(h) = E[x(t + h) x(h)]

G(h) =η0

4e−hβ2

[cos(hβ1)

β2+

sin(hβ1)

β1

], |η1| < 2, �Underdamping

G(h) = η0(1 + h)

4 eh, η1 = 2, �Critical damping

G(h) =η0

2 ∆

( e−hω1

ω1−

e−hω2

ω2

), η1 > 2, �Overdamping

0 1 2 3 4 5 6 7 8h

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ρxx

(h;θ

)

η1=-0.5

η1=2

η1=4 ⇐ Oscillator displacement correlation function

Nørrelykke, S.F., Flyvbjerg, H.: Harmonic oscillator in heat bath:

Exact simulation of time-lapse-recorded data and exact

analytical benchmark statistics. Phys. Rev. E 83, 041103 (2011)




An Optimal Basis for the 1D SSRF model is given bythe Karhunen-Loeve Expansion

Karhunen-Loeve Expansion of SSRFs in a NutshellI Determining the K-L basis requires solving a fourth-order ODE

with a 4× 4 system of boundary conditions

I There are two eigenfunction branches: the first involves onlyharmonic functions, while the second involves a superposition ofharmonic and hyperbolic eigenfunctions

I Each branch contains eigenfunctions with even, f (s) = f (−s),and odd, f (s) = −f (−s), symmetry

I The eigenvalues are determined by solving (numerically)transcendental equations

For more details see: Tsantili and Hristopulos (2016), Probabilistic EngineeringMechanics, 43, 132-147




Convergence of the Variance of the Karhunen-LoeveExpansion for d = 1 SSRFs

0 10 20 30 40n

-5

-4

-3

-2

-1

0

1

2

3

Log(

Eig

enva

lues

)

6

6$

6

6$

0 10 20 30 40n

0

20

40

60

80

100

En=E

tot(%

)(Left) Eigenvalues λ, λ∗, λ, λ∗ obtained for SSRF with η0 = 2, η1 = −1 and ξ = 5.(Right) Cumulative energy En =

∑n′<n En′ by superposition of largest n eigenvalues of K-L basis

as percentage of σ2 =∑∞

n′=1 En′ . Open circles: First-branch, even-sector eigenvalues.

Squares: First-branch, odd-sector eigenvalues. Diamonds: Second-branch, even-sector

eigenvalues. Filled circles: Second-branch, odd-sector eigenvalues.




The Errors of the Karhunen-Loeve Expansion can beControlled

K-L Expansion - Local Approximation Errors

Local approximation errors involved for SSRF correlation function with η0 = 2, η1 = −1and ξ = 5. (Left) K-L approximation with four terms and domain length L ≈ 13.6.(Right) K-L approximation with twenty four terms and domain length L = 100.




SSRF Covariance Functions for Processes in thePlane

G(h) =η0 = [K0(h z+)]

π√

4− η21

, |η1| < 2

G(h) =

(η0 h4π

)K−1(h), η1 = 2

G(h) =η0 [K0(h z+)− K0(h z−)]

2π√η2

1 − 4, η1 > 2

NotationI =: Imaginary partI z± =

√−t∗±

I t∗± =

(−η1 ±

√η2

1 − 4)/2

I Kν(z): modified Bessel functionof the second kind and order ν

d=2

0 1 2 3 4 5 6 7 8−0.05

0

0.05

0.1

0.15

0.2

0.25

h

G(h

)

η1=−1.2

η1=−0.5

η1=0.5

η1=2

η1=4

Hristopulos (2015), Stoch. Environ. Res. RiskAssesss., 29(3), 739–754.




SSRF Covariance Functions for Spatial Processes in3D Space (in R3 or in R2 × T )

G(h) = η0e−hβ2

2π∆

[sin (hβ1)

h

], |η1| < 2

G(h) =η0

8πe−h, η1 = 2

G(h) =1

4π∆

[e−hω1 − e−hω2

h (ω2 − ω1)

], η1 > 2

NotationI h = ‖~r‖/ξ

I β1,2 =(|2∓η1|

4

)1/2

I ω1,2 =(|η1∓∆|

2

)1/2

I ∆ = |η21 − 4|

12

d=3

0 0.2 0.4 0.6 0.8 1−0.2

0

0.2

0.4

0.6

0.8

1

1.2

Distance lag

Cor

rela

tion

η1=−1

η1=2

η1=8

η1=16

Hristopulos and Elogne, IEEE Trans. Inform.Theor., 2007




Applications - Groundwater Level Estimation

Estimated map of Mires basin (Messara valley,Crete) groundwater level using regression krigingcombined with Thiem’s multiple well equation as

trend.

Varouchakis and Hristopulos, Advances in Water

Resources, (2012)

Wet versus dry period variability.

E. Varouchakis, “Geostatistical Analysis and

Space-Time Models of Aquifer Levels,” PhD

Dissertation, Technical University of Crete (2012)




Applications - Lignite Mining

−2.2 −2.1 −2 −1.9 −1.8 −1.7

x 104

2.1

2.15

2.2

2.25

2.3

2.35

2.4

2.45

2.5

2.55

2.6

2.65x 10

4

West−East (m)

So

uth

−N

ort

h (

m)

Mavropigi mine

Estimation of area density of lignite energy content (Gcal/m2) for Mavropigi mine (WesternMacedonia, Greece) using regression kriging with Spartan variogram Pavlides et al. (2015),Energy, 93(Part 2), 1906–1917.




An SSRF on a Regular Grid is a Gauss-Markov RFwhose Energy involves the Squares of theFluctuations, their Gradient and their Curvature

H[x(~s)] =λN∑

n=1

{[x(~sn)− µX

]2+ c1

d∑i=1

[x(~sn + ai ei)− x(~sn)

ai

]2

+c2

d∑i=1

[x(~sn + ai ei)− 2x(~sn) + x(~sn − ai ei)

a2i

]2}

ei , i = 1, . . . , d : unit vectors in lattice directions

ai : lattice steps

0 2 4 6 8 100

1

2

3

4

5

6

7

8

9

10

Hfgc [X (~s)] = λS0 + c1 λSG + c2 λSc

Rue and Held, Gaussian Markov Random Fields: Theory

and Applications, Chapman and Hall/CRC, 2005




Reconstruction of Walker Lake Data (260× 300 grid)Anisotropic model based on 39 000 sampling points (50%)

0

200

400

600

800

1000

1200

1400

1600

0

200

400

600

800

1000

1200

1400

1600

Left top: Full data setLeft bottom: Sample

Below: Scatter plot of samplevs SLI predictions

Right top: SLI-based mapRight bottom: SLI spatial error

SLI Prediction0 500 1000 1500

Sam

ple

0

200

400

600

800

1000

1200

1400

1600 0

200

400

600

800

1000

1200

1400

1600

-800

-600

-400

-200

0

200

400

600

800




Reconstruction of Walker Lake Data (260× 300 grid)Anisotropic model based on 7 800 sampling points (10%)

0

200

400

600

800

1000

1200

1400

1600

0

200

400

600

800

1000

1200

1400

1600

Left top: Full data setLeft bottom: Sample

Below: Scatter plot of samplevs SSRF predictions

Right top: SSRF-based mapRight bottom: SSRF spatial

error

SLI Prediction0 500 1000 1500

Sam

ple

0

200

400

600

800

1000

1200

1400

16000

200

400

600

800

1000

1200

1400

1600

-600

-400

-200

0

200

400

600

800




Stochastic Local Interaction Model for Scattered DataCombining SSRF ideas with kernel functions the following energy

functional can be used for scattered data xS ≡ (x1, . . . , xN)T

H(xS;θ) =1

2λ

[S0(xS) + α1 S1(xS;~h1) + α2 S2(xS;~h2)

],

where ~hi = (hi,1, . . . ,hi,N)T , i = 1,2, is a vector of local bandwidths.

Main IdeasI Use kernel functions to express the energy terms Si(xS)

I Adjust the kernel bandwidths adaptively according to localsampling densityhi = µDi,[k ](SN ), where Di,[k ](SN ) is the distance between ~si and its k -nearestneighbor in SN , and µ > 1 is a data-dependent parameter

I Estimate parameters using leave-one-out cross validation (faster)or maximum likelihood (slower)




Thanks to SLI Precision Matrix Formulation, Predictiondoes not Require Covariance Matrix Inversion

I Energy functional (Hristopulos (2015), Computers & Geosciences, 85, 26–37)

H(xS;θ) =12

(xS − µX)T J(θ) (xS − µX)

I Precision matrix (sparse, explicit)

J(θ) =1λ

{INN

+ α1 c1 J1(~h1) + α2

[c2,1 J2(~h2)− c2,2 J3(

√2~h2)− c2,3 J4(2~h2)

]}I Gradient and Curvature Precision sub-matrices

[Jq(~hq)]i,j = −ui,j (hq;i )− ui,j (hq;j ) + [IN ]i,j

N∑l=1

[ui,l (hq;i ) + ul,i (hq;l )

],

I Kernel Weights

ui,j (hq;i ) =K((~si −~sj )/hq,i

)∑Ni=1∑N

j=1 K((~si −~sj )/hq,i

) .Technical University of Crete Geostatistics Laboratory: http://www.geostatistics.tuc.gr/4908.html



SLI Mode Predictor has O(Nα) Numerical Complexity,where 1 ≤ α ≤ 2 for Scattered Data and α = 1 onRegular Grids

I Mode predictor: xp = arg minxp H(xS, xp;θ∗) , where xp is the

value at the prediction point

I Prediction equation: xp = µX − 1Jp,p(θ∗)

∑Ni=1 Jp,i(θ

∗) (xi − µX)

I Property 1: Unbiased estimator, i.e., E [xp] = µX

I Property 2: Not necessarily an exact interpolator

I Property 3: Interpolation “limited” within the convex hull of thedata




Study of Groundwater Level Data (Small Data Set forIllustration Purposes)

Scattered data with modest size (N = 250) are used to compare withOrdinary Kriging (based on the Spartan variogram)

Plot of sample values

5 5.05 5.1 5.15

x 105

4.54

4.545

4.55

4.555

x 106

Easting (m)

No

rth

ing

(m

)

50

100

150

200

250

300

350

400

450

Frequency histogram




SLI Interpolation Analysis: Bandwidth & PrecisionMatrixBi-triangular kernel, k = 2, Estimated parameters: α1 ≈ 5.38, α2 ≈ 0.53, µ ≈ 2.93

Relative size of localbandwidths per site

0 20 40 60 80 1000

20

40

60

80

100

X

Y

h1

5

10

15

20

25

30

35

Precision matrix






0 20 40 60 80 1000

20

40

60

80

100

X

Y

h1

5

10

15

20

25

30

35

Precision matrixPrecision Matrix

50 100 150 200 250

50

100

150

200

250 −0.5

0

0.5

1

1.5

2

2.5






0 20 40 60 80 1000

20

40

60

80

100

X

Y

h1

5

10

15

20

25

30

35

Precision matrixPrecision Matrix

50 100 150 200 250

50

100

150

200

250




SLI Interpolation Analysis - MapsBi-triangular kernel, k = 2, Estimated parameters: α1 ≈ 5.38, α2 ≈ 0.53, µ ≈ 2.93

Ordinary Kriging Map SLI Generated Map




Cross Validation Analysis - A difference is a differenceonly if it makes a difference

Table: Leave-one-out Cross Validation results of SLI interpolation(Bi-triangular kernel) and OK (with Spartan variogram).

CV measures SLI (bi-tria) OK (SSRF)ME: Mean error (bias) 0.05 −0.02MAE: Mean absolute error 1.76 1.76MARE: Mean absolute relative error 0.44 0.43RMSE: Root mean square error 2.77 2.69RP: Pearson correlation coefficient 0.68 0.70SR: Spearman correlation coefficient 0.72 0.75Min Dif: min{z} −min{z} 0.79 0.76Max Dif: max{z} −max{z} −7.31 −8.5




Fast SLI Interpolation: Campbell County Coal Data -Mapping of Coal ThicknessQuadratic kernel, k = 2, Ntrain = 8700, Estimated parameters: α1 ≈ 267.48, α2 ≈ 0.60, µ ≈ 2.23,Nval = 8700, 100× 100 grid




Linear Response of Local Interaction Gibbs Models

I Linear response theory describes the non-equilibrium responsedue to small deviation from equilibrium and leads to S-TLangevin equations Hohenberg and Halperin (1977), Reviews of ModernPhysics, 49(3), 435–479

∂x(~s, t)∂t

= −DδH[x(~s)]δx(~s)

∣∣∣∣x(~s)=x(~s,t)

+ ζ(~s, t) = V [x(~s, t)] + ζ(~s, t)

I Equilibrium-restoring velocity for the ST-SLI random field

V [x(~s, t)] = − 12ξd η0

(1− η1ξ

2∇2 + µ ξ4∇4) x(~s)∣∣∣∣x(~s)=x(~s,t)

I D is a diffusion coefficient

I ζ(~s, t) is the random velocity (e.g., Gaussian white noise)




Equations of Motion for the ST-SLI Covariance can bederived using Linear Response Theory

EOM:∂Cx(r, τ)

∂τ= − sign(τ)

τc

(1− η1ξ

2∇2 + µ ξ4∇4) Cx(r, τ),

where τ−1c = D/(2ξd η0), and the initial condition is the SSRF spectral density

C(k , τ = 0) =η0 ξ

d

1 + η1(kξ)2 + µ(kξ)4.

Zero-µ solution in d = 1 dimension

C1(h,u) =η0 λ

4

[e−λ h erfc

(√u − λh

2√

u

)+ eλ h erfc

(√u +

λh2√

u

)],

λ = 1/√η1, h = r/ξ, u = |τ |/τc , and erfc(·) is the complementary error function.

C1(h, u) is equivalent (except for different parametrization) to a covariance derived froma parabolic SPDE in 1+1 dimensions by Heine (1955). Biometrika, 42(1-2), 170–178.




Equations of Motion for the ST-SLI Covariance can bederived using Linear Response Theory

Zero-µ solution in d = 3 dimension

C3(h,u) =η0 ξ λ

2

8π r

[e−λ h erfc

(√u − λh

2√

u

)− eλ h erfc

(√u +

λh2√

u

)],




The variance divergence can be tamed by adding thecurvature term

Small µ approximation

C(~r , τ) ≈2 e−D |τ |

(4π)d/2

2M∑m=0

(−µ)m

m!

Γ(d/2 + 2m)

Γ(d/2)Rm(r , τ)

Rm(r , τ) =

∫ ∞0

dκe−κβ0 vm

u4m+d 1F1

(2m +

d2,

d2

;−r2

4 u2

)where r = ‖~r‖, and

I v = ξ4(

D|τ |+ κβ0

)I u2 = η1ξ

2(

D |τ |+ κβ0

)I 1F1(a1, a2; z) is the confluent hypergeometric

function

Hristopulos & Tsantili (2015), Int. J. Modern Phys. B, 29,1541007

0 5 10 15-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

r

CHr,

0L

Η =-1.5Η =-1Η=-0.5

0 5 10 15-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

r

CHr,

3L

Η =-1.5Η =-1Η=-0.5

µ = 1, ξ = 3

Top: τ = 0. Bottom: τ = 3.




Taming the variance divergence with space transformsI Space transforms are mathematical operations that can generate

higher-dimensional functions based on lower-dimensionalprojections, e.g., Ehrenpreis, L. (2003). The Universality of the RadonTransform, Oxford.

I For example [Mantoglou and Wilson (1982). Water Resources Research,18(5), 1379–1394]

C3(r , τ) =1r

∫ r

0dx C1(x , τ) =

1h

∫ h

0dy C1(y ,u)

I Applying the above to the d = 1 local-interaction covariancefunction we obtain

Space-transformed ST-SLI (linear response) covariance

C3(h, u) =η04h

[2 e−u erf

(λh

2√

u

)+ eλ h erfc

(√u + λh

2√

u

)− e−λ h erfc

(√u − λh

2√

u

)]Technical University of Crete Geostatistics Laboratory: http://www.geostatistics.tuc.gr/4908.html



Visualization of the ST-SLI covariance obtained bymeans of the space transform

-1.5 -1 -0.5 0 0.5 1 1.5h (space lag)

-1.5

-1

-0.5

0

0.5

1

1.5u

(tim

e la

g)




Visualization of the ST-SLI covariance obtained bymeans of the space transform




Brief Review of Linear Response Theory ResultsI S-T local-interaction Langevin equation

∂x(~s, t)∂t

= −D

2ξd η0

(1− η1ξ

2∇2 + µ ξ4∇4)

x(~s, t) + ζ(~s, t)

I 1D+T SLI covariance

C1(h, u) =η0 λ

4

[e−λ h erfc

(√u −

λ h2√

u

)+ eλ h erfc

(√u +

λ h2√

u

)]

I 3D+T space-transformed SLI covariance

C3(h, u) =η0

4h

[2e−u erf

(λh

2√

u

)+ eλh erfc

(√u +

λh2√

u

)− e−λh erfc

(√u −

λh2√

u

)]

I The above covariances do not scale in u and h like the Gneiting(2002) class C(h,u) = 1

ψ(u2)d/2φ(

h2

ψ(u2)

), φ(·) : completely monotone,

ψ(·): Bernstein (positive definite with completely monotone derivative)




Conclusions and Future Directions




Thank you for your attention!

Collaborations: Dr. S. Elogne, Dr. M. Zukovic, Dr. A. Chorti, Dr. E. Varouchakis,

Mr. I. Spiliopoulos, Dr. Ivi Tsantili, Mr. A. Pavlides, Mrs. V. Agou, and Mr. M.Petrakis

Research funded by the project SPARTA implemented under the“ARISTEIA” Action of the operational programme “Education andLifelong Learning” co-funded by the European Social Fund (ESF)and National Resources




For more information ...D. T. Hristopulos (2003). “Spartan Gibbs Random Field Models for Geostatistical Applications,”SIAM J. Sci. Comput., 24(6), 2125-2162.

D. T. Hristopulos and S. Elogne (2006). “Analytic Properties and Covariance Functions of a NewClass of Generalized Gibbs Random Fields,” IEEE Trans. Infor. Theory, 53(12), 4667 - 4679.

S. N. Elogne and D. T. Hristopulos (2008). “Geostatistical applications of Spartan spatial randomfields,” in geoENV VI - Geostatistics for Environmental Applications, pp. 477-488 (ed. by A. Soareset al.) 512p.

S. Elogne, D. T. Hristopulos, M. Varouchakis (2008). “An application of Spartan spatial randomfields in environmental mapping: focus on automatic mapping capabilities,” Stoch. Envir. Res. RiskA., 22(5), 633-646.

D. T. Hristopulos and S. N. Elogne (2009). “Computationally efficient spatial interpolators based onSpartan spatial random fields”, IEEE Trans. Signal Proc., 57(9), 3475–3487.

M. Zukovic, and D. T. Hristopulos (2009b). “Classification of missing values in spatial data usingspin models,” Phys. Rev. E, 80(1), 011116.

D. T. Hristopulos (2015). “Covariance functions motivated by spatial random field models with localinteractions,” Stoch. Envir. Res. Risk A., 29(3), 739–754.

D. T. Hristopulos (2015). “Stochastic Local Interaction (SLI) Model for Incomplete Data ind−Dimensional Metric Spaces” Computers & Geosciences, 85, 26–37.




Absence of Space-Time Dimple Effect

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2h (spatial lag)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Cov

aria

nce

00.25 0.5 11.25 1.5 2

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2u (time lag)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Cov

aria

nce

00.25 0.5 11.25 1.5 2



stochastic local interaction models and space-time ......the errors of the karhunen-loeve expansion...

Documents