Efficient Kriging via Fast Matrix-Vector Products

Nargess MemarsadeghiNASAIGSFC, Code 587Greenbelt, MD, 20771

Nargess.Memarsadeghi;nasa.gov

Vikas C. RaykarSiemens Medical Solutions

Malvern, PA 19355vikas.raykar@siemens.com

Ramani DuraiswamiUniversity of Maryland,College Park, MD 20742ramani@cs.u.md.edu

David M. MountUniversity of MarylandCollege Park, MD 20742mount@cs.umd.edu

Abstract-Interpolating scattered data points is a problem ofwide ranging interest. Ordinary kriging is an optimal scat-tered data estimator, widely used in geosciences and, remotesensing. A generalized version of this technique, called cok-riging, can be used for image fusion of remotely sensed, data.However, it is computationally very expensive for large datasets. We demonstrate the time efficiency and accuracy of ap-proximating ordinary kriging through the use of fast matrix-vector products combined with iterative methods. We usedmethods based on the fast Multipole methods and nearestneighbor searching techniques for implementations ofthe fastmatrix-vector products.

Keywords-geostatistics, image fusion, kriging, approximatealgorithms, fast multipole methods, fast Gauss transform,nearest neighbors, iterative methods.

1234

5678

TABLE OF CONTENTS

INTRODUCTION ...................................ORDINARY KRIGING VIA ITERATIVE METHODSMATRIX-VECTOR MtJLTIPLICATION .............FAST APPROXIMATE MATRIX-VECTOR MU-LTI-PLICATION ........................................

DATA SETS ........................................EXPERIMENTS ....................................RESULTS ..........................................CONCLUSIONS ....................................REFERENCES .....................................BIOGRAPHY .......................................

4445S7

1. INTRODUCTION

Scattered data interpolation is a problem of interest in numer-ous areas such as electronic imaging, smooth surface model-ing, and computational geometry [ 1,2]. Our motivation arisesfrom applications in geology and mining which often involvelarge scattered data sets and a demand, for high accuracy. Forsuch cases, the method of choice is ordinary kriging [3]. Thisis because it is a best unbiased estimator [3-5]. Also in remote sensing, imagefusion of multi-sensor data is often usedto increase either the spectral or the spatial resolution of the

U.S. Government work not protected by U.S. copyrightThis work was done when Nargess Memarsadeghi and Vikas C. Raykarwere graduate students in Computer Science Department of the Universityof Maryland at College Park.The work ofD. M. Mount has been supported in part by the National ScienceFoundation under grant CCR-0635099

images involved [6, 7]. A generalized version of the ordinarykriging, called cokriging [3, 5], can be used for image fusionof multi-sensor data [8, 9]. Unfortunately, ordinary kriginginterpolant is computationally very expensive to compute ex-actly. For n scattered, data points, computing the value ofa single interpolant involves solving a dense linear systemof order n x n, which takes O(n3). This is infeasible forlarge n. Traditionally, kriging is solved, approximately by lo-cal approaches that are based on considering only a relativelysmall number of points that lie close to the query point [3,5].There are many problems with this local approach, however.The first is that determining the proper neighborhood size istricky, and. is usually solved by ad hocmethods such as select-ing a fixed number ofnearest neighbors or all the points lyingwithin a fixed radius. Such fixed neighborhood sizes may notwork well for all query points, depending on local densityof the point distribution [5]. Local methods also suffer fromthe problem that the resulting interpolant is not continuous.Meyer showed that while kriging produces smooth continu-ous surfaces, it has zero order continuity along its borders[10]. Thus, at interface boundaries where the neighborhoodchanges, the interpolant behaves discontinuously. Therefore,it is important to consider and, solve the global system foreach interpolant. However, solving such large dense systemsfor each query point is impractical.

Recently an approximation approach to kriging has beenproposed based on a technique called covariance tapering[11, 12]. However, this approach is not efficient when covari-ance models have relatively large ranges. Also, finding a validtaper, as defined in [11L], for different covariance functions isdifficult and at times impossible (e.g. Gaussian covariancemodel).

In this paper, we address the shortcomings of the previousapproaches through an alternative based on Fast MultipoleMethods (FMM). The FMM was first introduced by Greengardand, Rokhlin for fast multiplication of a structured matrix bya vector [13, 14]. If the Gaussian function is used for gen-erating the matrix entries, the matrix-vector product is calledthe Gauss transformn We use efficient implementations oftheGauss transform based on the FMM idea (see [15,16]) in com-bination with the SYMMLQ iterative method [17] for solvinglarge ordinary kriging systems Billings et at [18] had alsosuggested the use of iterative methods in combination withthe FMM for solving such systems.

The remainder of this paper is organized as follows. Section

1

2 describes the ordinary kriging system and solving it via aniterative method. In Section 3 we introduce the matrix-vectorproducts involved in solving such systems. We mention twoexisting efficient and approximate implementations of suchproducts in Section 4. Section 5 describes our data sets. Ourexperiments and results are presented in Sections 6 and 7 re-spectively. Section 8 concludes this paper.

2. ORDINARY KRIGING VIA ITERATIVEMETHODS

Kriging is an interpolation method named after Danie Krige,a South African mining engineer [5]. Kriging and its vari-ants have been traditionally used in mining and geostatisticsapplications [3-5]. Kriging is also referred to as the Gaus-sian process predictor in the machine learning domain [19].The most commonly used variant is called ordinairy kriging,which is often referred to as a Best Linear Unbiased Esti-mator (BLUE) [3, 1 1L]. It is considered to be best because itminimizes the variance of the estimation error. It is linearbecause estimates are weighted linear combination of avail-able data, and is unbiased since it aims to have the mean errorequal to zero [3]. Minimizing the variance of the estimationerror forms the objective function of an optimization prob-lem. Ensuring unbiasedness of the error imposes a constrainton this function. Forrmalizing this objective function with itsconstraint results in the following system [3, 5, 12].

( LT(..Co..\1)

method requires one O(T 2) matrix-vector multiplication periteration. The storage is 0(n) since the matrix-vector multi-plication can use elements computed on the fly without stor-ing the matrix. Empirically the number of iterations required,k, is generally small compared to n leading to a computa-tional cost of O(kn 2).

3. MATRIX-VECTOR MULTIPLICATIONThe 0(kn2) quadratic complexity is still too high forlarge data sets. The core computational step in eachSYMMLQ iteration involves the multiplication of a matrixC with some vector, say q. For the Gaussian covari-ance model the entries of the matrix C are of the form[C]i4 = exp (-3 |xi- 2 /a2) Hence, the jth ele-ment of the matrix-vector product Cq can be written as(Cq)j = =qE exp (-3 xi 2 a2)-which is theweighted sum of n Gaussians each centered at xi and eval-uated at x.

Discrete Gauss transform (GT)

The sum of multivariate Gaussians is known as the discreteGauss transform in scientific computing. In general, for eachtargetpoint {Yi £ PRd}7, (which in our case are the same asthe source points xi) the discrete Gauss transform is definedas

G(yj) = .qiei=l(1)

where C is the matrix of points' pairwise covariances, L is acolumn vector of all ones and of size , and w is the vectorof weights wi, . . . wr, Therefore, the minimization prob-lem for a points reduces to solving a linear system of size(n + 1)2, which is impractical for very large data sets via di-rect approaches. It is also important that matrix C be positivedefinite [3, 12]. Pairwise covariances are often modeled asa function of points' separation. These functions should re-sult in a positive definite covariance matrix. Christakos [20]showed necessary and sufficient conditions for such permis-sible covariance functions. Two of these valid functions arethe Gaussian and Spherical covariance functions [3,5,20]. Inthis paper, the Gaussianu covariance function is used. For twopoints xi and xj, the Gaussian covariance function is definedas Cij = exp (- 3 lzxi- 2 /a2), where a is the range ofthe covariance function.

For large data sets it is impractical to solve the ordinary krig-ing system using direct approaches that take 0(n3) time Iterative methods generate a sequence of solutions which con-verge to the true solution in n iterations. In practice, however,we loop over k a 0, we define C to be an E-exact ap-proximation to G if the maximum absolute error relative tothe total weight Q = =qiq is upper bounded by E, i. e.,

max [G(yj)C-QG(yj)vi Q

C. (3)

The constant in 0(Tri na) depends on the desired accuracy cwhich however can be arbitrary. At machine precision thereis no difference between the direct and the fast methods. Themethod relies on retaining only the first few terms of an infinite series expansion for the Gaussian function. These meth-ods are inspired by the fast multipole methods (FMM), origi-nally developed for the fast summation of the potential fieldsgenerated by a large number of sources, such as those arisingin gravitational potential problems [13]. The fast Gauss trans-form (FGT) is a special case where FMM ideas were used for

2

the Gaussian potential [23]. The improved fast Gauss trans-form (IFGT) is a similar algorithm based on a single differentfactorization and data structure. It is suitable for higher di-mensional problems and provides comparable performancein lower dimensions [ 15].

Improvedfast Gauss transform (IFGT)

IFGT is an efficient algorithm for approximating the Gausstransform. The fast Gauss transform, first proposed byGreengard and Strain [23], is an E-exact approximation al-gorithm for the Gauss transform. This algorithm reduces theGauss transform's computational complexity from O(MTI) toO(m + n). However, this algorithm's constant factor growsexponentially with dimension d. Later improvements, includ-ing the IFGT algorithm, reduced this constant factor to asymp-totically polynomial order in terms of d. The IFGT algorithmwas first introduced by Yang et al. [15]. Their implementa-tion did not use a sufficiently tight error bound to be usefulin practice. Also, they did not adaptively select the necessaryparameters to achieve the desired error bound. Raykar et allater presented an approach that overcame these shortcom-ings [16,24]. We used the implementation due to Raykar etal We briefly describe some of the key ideas in IFGT. Pleasesee [1L6,24] for more details. For any point x£ C lRd the GaussTransform at yj can be written as,

(5)

Rearranging the terms Equation (5) can be written as

G(yj) 2E [ 2E -lx z l/lael

source points are calculated via fixed-radius nearest neighborsearch routines of the ANN library [27]. Finally, for each tar-get point, their nearest neighbor source points are calculatedin matrix-vector product calculations, involving the covari-ance matrix C in the ordinary kriging system.

5. DATA SETS

We generated three sets of sparse data sets. For the first set,the number of sampled points varied from 1000 up to 5000,while their covariance model had a small range value of 12.For the second set, we varied the number of samples in thesame manner, except that the points' covariance model had alarger range equal to 100. Finally, we sampled 5000 pointsfrom dense grids, where points' covariance model had rangesequal to a = 12, 24,100, 250, and 500. For each input dataset we use 200 query points which are drawn from the samedense grid but are not present in the sampled data set. Onehundred of these query points were sampled uniformly fromthe original grids. The other 100 query points were sampledfrom the same Gaussian distributions that were used in thegeneration of a small percentage of the sparse data.

6. EXPERIMENTS

We used the SYMMLQ iterative method as our linear solver.We set the desired solutions' relative error, or the convergencecriteria for the SYMMVLQ method, to 62 = 10-3. Thus, ifSYMMLQ is implemented exactly, we expect the relative er-ror to be less than 10-3. The exact error is likely to be higherthan that. We developed three versions of the ordinary krig-ing interpolator, each of which calculates the matrix-vectorproducts involved in the SYMMLQ method differently.

Gauss Transform (GT): Computes the matrix-vector prod-uct exactly.Improved Fast Gauss Transform (IFG T): Approximatesthe matrix-vector product to the e precision via the IFGTmethod mentioned in Section 4.Gauss transform with nearest neighbors (GTANN): Ap-proximates the matrix-vector product using the GTANNmethod mentioned in Section 4.

All experiments were run on a Sun Fire V20z running RedHat Enterprise release 3, using the g++ compiler version3.2.3. Our software is implemented in C++ and uses theGeostatistical Template Library (GsTL) [28] and Approxi-mate Nearest Neighbor library (ANN) [27] GsTL is usedfor building and solving the ordinary kriging systems, andANN is used for finding nearest neighbors when using theGTANN approach In all cases for the IFGT and GTANN approaches we required the approximate matrix-vector productsto be evaluated within e-= 10 accuracy. All experiments'results are averaged over five runs We designed three experiments to study the effect of covariance ranges and number ofdata points on performances of our different ordinary krigingversions.

Experiment 1: We varied number of scattered data pointsfrom 1000 up to 5000, with a small fixed Gaussian covariancemodel ofrange a = 12.Experiment 2: We varied number of sampled points. Thistime, points had a larger range equal to a = 100 for theircovariance model.Experiment 3: Finally, we examined the effect of differentcovariance ranges equal to a = 12, 25, 100, 250, and 500 ona fixed data set of size 5000.

7. RESULTS

In this section we compare both the quality of results and thespeed-ups achieved for solving the ordinary kriging systemvia iterative methods combined with fast approximate matrix-vector multiplication techniques.

Figure 1 presents results of the first experiment mentioned.When utilizing approximate methods IFGT and GTANN theexact residuals are comparable to those obtained from GT.The IFGT approach gave speed-ups ranging from 1.3-7.6,while GTANN resulted in speed-ups ranging roughly from 50-150. This is mainly due to the fact that the covariance func-tion's range is rather small. Since there are only a limitednumber of source points influencing each target point, collect-ing and calculating the influence of all source points for eachtarget point (the IFGT approach) is excessive. The GTANNapproach works well for such cases by considering only thenearest source points to each target point. In both cases, thespeed-ups grow with number of data points. Algorithms per-form similarly for points from the Gaussian distribution tothose from uniform distribution.

Figure 2 shows results of the second experiment. While theGTANN approach did not result in significant speed-ups, theIFGT gave constant factor speed-ups ranging roughly from 1.5to 7.5, as we increased the number of data points. The IFGTapproach results in larger residuals for small data sets, andwhen solving the ordinary kriging system for query pointsfrom the uniform distribution. In particular, for n = 1000andd n = 2000, the performance ofIFGT is not acceptable withrespect to the exact residuals calculated. This poor overall re-sult for these cases is because the SYMM[LQ method did notmeet its convergence criteria and reached maximum numberof iterations. Increasing the required accuracy for the IFGTalgorithm, by changing the e = 10-4 to 10-6 resolved thisissue and gave residuals comparable to those obtained fromthe GT method. As the number of points increases, the qual-ity of results approaches those of the exact methods For thequery points from the Gaussian distribution the quality ofresults are comparable to the exact method, when using theIFGT approach The GTANN approach also results in comparable residuals to the exact methods in all cases.

Figure 3 presents results of the last set of experiments In allcases, the quality of results is comparable to those obtainedfrom exact methods. The IFGT approach resulted in speed-ups of 7-15 in all cases The GTANN approach is best when

4

used for covariance functions with small range values of 12,and 25. While the GTANN approach is slower than the exactmethods for range values larger than 100, it results in speed-up factors of 151 -53, and 47-49 for range values 12 and 25respectively. Thus, the GTANN approach is efficient for smallrange values, and so is the IFGT approach for large ranges.

8. CONCLUSIONSWe integrated efficient implementations of the Gauss Trans-form for solving ordinary kriging systems. We examinedthe effect of number of points and the covariance functionsranges on the running time for solving the system and thequality of results. The IFGT is more effective as number ofdata points increases. Our experiments using the IFGT ap-proach for solving the ordinary kriging system demonstratedspeed-up factors ranging from 7-15 when using 5000 points.Based on our tests on varying number of data points, weexpect even higher speed-up factors compared to the exactmethod when using larger data sets. In almost all cases, thequality of IFGT results are comparable to the exact meth-ods. The GTANN approach outperfornned the IFGT methodfor small covariance range values, resulting in speed-up fac-tors as high as 153 and 49 respectively. The GTANN approachis slower than the exact methods for large ranges (100 andover in our experiments), and thus is not recommended forsuch cases. The quality of results for GTANN was compara-ble to the exact methods in all cases. Please see [26, 29] fordetails ofmethods used in this work.

Future work involves efficient kriging via fast approximatematrix-vector products for other covariance functions, wherea factorization exists. We also plan on using precondition-ers [2 1 ] for our iterative solver, so that the approximate ver-sions converge faster, and we obtain even higher speed-ups.

REFERENCES

[1] I. Amidror, "Scattered data interpolation methods forelectronic imaging systems: A survey," Journal ofElec-tronic Imaging, vol. I 1, no. 2, pp. 157-176, April 2002.

[2] P. Alfeld, "Scattered data interpolation in three or morevariables," Mathematical methods in computer aidedgeometric design, pp. 1-33, 1989.

[3] E. H. Isaaks and R. M. Srivastava, An Introduction toApplied Geostatistics. New York Oxford. Oxford Uni-versity Press 1989

[4] A. G. Journel and C. J. Huijbregts Mining Geostatistics.New York: Academic Press Inc, 1978.

[5] P. Goovaerts, Geostatisticsfor Natural Resources Eval-uation. New York, Oxford: Oxford University Press,1997.

[6] D. L. Hall Mathematical techniques in multisensor datafusion. Norwood: Artech House Inc, 1992.

[7] Y. Zhang, 'Understanding image fusion," Photogram-metric Engineering and Remote Sensing, pp. 657-661,

June 2004.

[8] N. Memarsadeghi, J. L. Moigne, D. M. Mount, andJ. Morisette, "A new approach to image fusion basedon cokriging," in the Eighth International Conferenceon Information Fusion, vol. 1, July 2005, pp. 622-629.

[9] N. Memarsadeghi, J. L. Moigne, and D. M. Mount, "Im-age fusion using cokriging," in IEEE International Geo-science and Remote Sensing Symposium (IGARSS'06),July 2006, pp. 2518 - 2521.

[10] T. H. Meyer, "The discontinuous nature of kriging in-terpolation for digital terrain modeling," Cartographyand Geographic Information Science,, vol. 3 1, no. 4, pp.209-216,2004.

[ 1] R. Furrer, M. G. Genton, and D. Nychka, "Covari-ance taperincg for interpolation of large spatial datasets,"Journal of Computational and Graphical Statistics,vol. 15, no. 3, pp. 502-523, September 2006.

[12] N. Memarsadeghi and D. M. Mount, "Efficient imple-mentation of an optimal interpolator for large spatialdata sets," in Proceedings of the International Confer-ence on Computational Science (ICCS'07), ser. Lec-ture Notes in Computer Science, vol. 4488. Springer-Verlag, May 2007, pp. 503-510.

[13] L. Greengard and V. Rokhlin, "A fast algorithm forparticle simulation," Journal ofComputational Physics,vol. 73, no. 2, pp. 325-348, 1987.

[14] L. Greengard, "The rapid evaluation of potential fieldsin particle systems," Ph.D. dissertation, Yale, NYU,1987.

[15] C. Yang, R. Duraiswami, and L. Davis, "Efficient ker-nel machines using the improved fast Gauss transform."in Advances in Neural Information Processing Systems,L. K. Saul, Y. Weiss, and L. Bottou, Eds., vol. 17. MITPress, 2005, pp. 1561-1568.

[16] V. C. Raykar, C. Yang, R. Duraiswami, andN. Gumerov, "Fast computation of sums of Gaussiansin high dimensions," Department of Computer Science,University of Maryland, College Park, MD, 20742,Tech. Rep., 2005, CS-TR-4767.

[17] C. C. Paige and M. A. Saunders, "Solution of sparseindefinite systems of linear equations," SIAM Journalon Numerical Analysis, vol. 12, no. 4, pp. 617-629,September 1975.

[18] S D Billings, R K Beatson and G N Newsamm "In-terpolation of geophysical data using continuous globalsurfaces," Geophysics, vol. 67, no. 6, pp. 1810-1822,November-December 2002.

[19] C. E. Rasmussen and C. K. I. Williams Gaussian Pro-cessesfor Machine Learning. MIT Press, 2006.

[20] G. Christakos '"On the problem of permissible co-variance and variogram models," Water Resources Re-search vol 20 no 2,pp 251-265, February 1984

5

Average Exact Residuals Average CPU Time for Solving the SystemOver 200 Query Points Over 200 Query Points15E-4 200

n~~~~~~~~~~~~~~~~~~~~~0 -

> 5E-4

ct~~~¢

E-1O0°° 2000 3000 4000 5000 °1000 2000 3000 4000 5000Number of Scattered Data Points (n) Number of Scattered Data Points (n)

Figure 1L Expelri:ment 1, Left: Average albsolute erro:rs. Right: Avelrage CPU ti:mes

Average Exact Residuals Average CPU Time for Solving the SystemOver 200 Query Points Over 200 Query Points

0.04 270

,120

~~~~~< 0.02 ~~~~ ~ ~ ~ ~ ~ ~ ~ ~ ~~~~~.

WI E >~~~~~~I 100[< 0.01 4 1

IFGT >~~ 5 IFGTI"GTAN-N "~~~~~~GTAN-

IE-1$00 2000 3000 4000 50001s00 2000 3000 4000 5000Number of Scattered Data Points (n) Number of Scattered Data Points (n)

Figure 2. Experiment 2 Left: Average absolute errors. Right: Average CPU times

Average Exact Residuals Average CPU Time for Solving the SystemOver 200 Query Points Over 200 Query Points

0.02 20

2500-

2E-3 ¢IF G T

IFGT G

1E-3 GlA _ L lIFGT

2E- G4A12 100 200 300 400 5( 0 0 200 300 400 5000Range of the Gaussian Covariance Function Range of the Gaussian Covariance Function

Figure 3. Experiment 3, Left: Average absolute errors. Right: Average CPU times

6

[21] Y. Saad, Iterative methods for sparse linear systems.SIAM, 2003.

[22] S. G. Nash and A. Sofer, Linear and Nonlinear Pro-gramming. McGraw-Hill Companies, 1996.

[23] L. Greengard and J. Strain, "The fast Gauss transform,"SIAM Journal of Scientific and Statistical Computing,,vol. 12, no. 1, pp. 79-94, 19911.

[24] V. C. Raykar and R. Duraiswami, Large Scale KernelMachines. MIT Press, 2007, ch. The Improved FastGauss Transform with applications to machine learning.

[25] T. Feder and D. H. Greene, "Optimal algorithms for ap-proximate clustering," in Proc. 20th AnnualACM Sym-posium on Theory ofComputing, 1988, pp. 434-444.

[26] V. C. Raykar, 'Scalable machine learning for massivedatasets: Fast summation algorithms," Ph.D. disserta-tion, University of Maryland, College Park, MD,20742,March 2007.

[27] D. M. Mount and S. Arya, "ANN: A library for approx-imate nearest neighbor searching," http //www.cs.umd.edu/mount/ANN/, May 2005.

[28] N. Remy, "GsTL: The Geostatistical Template Libraryin C++," Master's thesis, Department of Petroleum En-gineering of Stanford University, March 2001.

[29] N. Memarsadeghi, "Efficient algorithms for clusteringand interpolation of large spatial data sets," Ph.D. dis-sertation, University of Maryland, College Park, MD,20742, April 2007.

BIOGRAP1H[Y

Nargess Memarsadeghi is a computerengineer at the Information Systems Di-vision (ISD) of the Applied Engineer-ing and Technology Directorate (AETD)at NASA Goddard Space Flight Center(GSFC) since July 2001. She receivedher Ph.D. from the University of Mary-land at College Park in Computer Sci-

ence in May 2007. She is interested in design and devel-opment of efficient algorithms for large data sets with ap-plications in image processing, remote sensing, and opticsvia computational geometry and scientific computation tech-niques.

Vikas C. Raykar received the B E. de-M gree in electronics and, communication

engineering from the National Instituteof Technology, Trichy, India, in 2001,the M.S. degree in electrical engineeringfrom the University of Maryland, Col-lege Park, in 2003, and. the Ph.D. degreein computer science in 2007 from the

same university. He currently works as a scientist in SiemensMedical Solutions, USA. His current research interests in-clude developing scalable algorithms for machine learning.

Ramani Duraiswami is an associateprofessor in the department of computerscience and UMIACS at the Univer-sity of Maryland, College Park. He di-rects the Perceptual Interfaces and Re-ality Lab. and has broad research interests in scientific computing, computa-tional acoustics and audio computer vi-

sion and machine learning.

David Mount is a professor in the De-partment of Computer Science at theUniversity of Maryland with a joint ap-pointment in the University's Institutefor Advanced. Computer Studies (UMI-ACS). He received his Ph.D. from Pur-due University in Computer Science in1983, and since then has been at the

University of Maryland. He has written over 100 researchpublications on algorithms for geometric problems, partic-ularly problems with applications in image processing, pat-tern recognition, information retrieval, and computer graph-ics. Hle currently serves as an associate editor for the jour-nal ACM Trans. on Mathematical Software and served onthe editorial board of Pattern Recognition from 1999 to 2006.He served as the program committee co-chair for the 19thACM Symposium on Computational Geometry in 2003 andthe Fourth Workshop on Algorithm Engineering and Experi-ments in 2002.

7