Reconstruction of missing data in satellite and in situ data sets with DINEOF

A. Alvera-Azcárate, A. Barth, J.-M. Beckers

GHER – University de Liège

FNRS – National Fund for the Scientific Research, Belgium

• Reconstruction method for gappy data based on an EOF decomposition

• Parameter-free, no need of a priori information

• resolves time and space

• much faster than objective analysis

• Determines optimal number of EOFs by cross-validation. Error estimation

• Lanczos method for EOF decomposition (Toumazou and Crétaux, 2001): large matrices

Characteristics of DINEOF

DINEOF: Data Interpolating Empirical Orthogonal Functions

N EOFs

●●

●● ● ● ●

k

pj

Tpippji vuX

1, )()(

2nd: EOF decomposition with N=1 EOF

Calculate missing values:

Improved guess for missing values

Convergence: best value for missing data with 1 EOF

cross validation: error

EOF decomposition with N=2 EOFs

Calculate missing values

Improved guess for missing values

Then we repeat with N= 3 EOFs

and so on…

1st: Demeaned matrix: missing data flagged and set to zero

Some data are set aside for cross-validation

err

or

How it works

SST

Chlorophyll

Wind-induced filaments in Adriatic Sea

Mississippi river plume advected by Loop Current

Some Examples

Alvera-Azcárate et al, 2005

Multivariate DINEOF: examples

Radial currents: positive values: current towards the antenna negative values: current away from the antenna

˚C m/s

SST + HF-Radar currents

Time series reconstructionHourly air temperature at Izas catchment (Pyrenees)

˚C

hours

hours

hours

months months

Ori

gin

al d

ata

, ˚C

Reconstruction, ˚C

Time series reconstruction:original vs. reconstruction

-30 -20 -10 0 10 20 30-30

-20

-10

0

10

20

30

f(x) = 0.91x + 0.37R² = 0.92

Error maps are calculated for SST reconstructions using:i) the EOF basis from DINEOF as background covarianceii) the location of valid data

Outliers (pixels with value larger than the statistically expected misfit calculated during the analysis) will be objectively identified and removed from initial data

Error maps and outlier detection

Spike removal in temporal EOFs

When too few data are present: temporal EOFs poorly constrained: unrealistic discontinuities

Sharp transition

Spike removal in temporal EOFs

We apply a filter to the temporal covariance matrix

Alvera-Azcárate et al, 2009

CFF=C T~

F is a Laplacian filter Filter on C instead of X: C is much smaller and less sensitive to missing data Filter applied iteratively: more iterations, further reach of the filter

days days

days

days

Spike removal in temporal EOFs

Conclusions

• DINEOF is a reliable, easy to use technique to reconstruct missing data

Widely tested and results validated with in situ data

• A variety of data and platforms can be used

SST, chlorophyll, TSM, HF-Radar….

• Univariate and Multivariate approaches

• Satellite data and time series

• Error maps and outliers

Generation of error maps to assess the confidence of the reconstruction

Outliers (cloud edges, haze...) can be removed from original data

• Time filter to remove spikes

• Code is freely available; installation instructions; mailing list

http://modb.oce.ulg.ac.be/mediawiki/index.php/DINEOF

References (chronologic)

• J.-M. Beckers and M Rixen. EOF calculations and data filling from incomplete oceanographic data sets. Journal of Atmospheric and Oceanic Technology, 20(12):1839-1856, 2003.

• A. Alvera-Azcárate, A. Barth, M. Rixen, and J. M. Beckers. Reconstruction of incomplete oceanographic data sets using Empirical Orthogonal Functions. Application to the Adriatic Sea. Ocean Modelling, 9:325-346, 2005.

• J.-M. Beckers, A. Barth, and A. Alvera-Azcárate. DINEOF reconstruction of clouded images including error maps. Application to the Sea Surface Temperature around Corsican Island. Ocean Science, 2(2):183-199, 2006.

• A. Alvera-Azcarate, A. Barth, J. M. Beckers, and R. H. Weisberg. Multivariate reconstruction of missing data in sea surface temperature, chlorophyll and wind satellite fields. Journal of Geophysical Research, 112:C03008, 2007.

• A. Alvera-Azcárate, A. Barth, D. Sirjacobs, J.-M. Beckers. Enhancing temporal correlations in EOF expansions for the reconstruction of missing data using DINEOF. Ocean Science, 5, 475-485, 2009.

• A. Alvera-Azcárate, A. Barth, D. Sirjacobs, F.Lenartz, J.-M. Beckers. Data Interpolating Empirical Orthogonal Functions (DINEOF): a tool for geophysical data analyses. Submitted to Mediterranean Marine Science. 2009.

• U. Ganzedo, A. Alvera-Azcarate, G.Esnaola, A. Ezcurra and J. Saenz. 2009. Reconstruction of sea surface temperature by means of DINEOF. A case study during the fishing season in the Bay of Byscay. Accepted in International Journal of Remote Sensing.

• D. Sirjacobs, A. Alvera-Azcárate, A. Barth, G. Lacroix, Y. Park, B. Nechad, K. Ruddick, J.-M. Beckers. Cloud filling of ocean color and sea surface temperature remote sensing products over the Southern North Sea by the Data Interpolating Empirical Orthogonal Functions methodology. Accepted in Journal of Sea Research, 2010.

GHER

References

• Y.Z. Ding, Z.H. Wei, Z.H. Mao, X.F. Wang, D.L. Pan. Reconstruction of incomplete satellite SST data sets based on EOF method. Acta Oceanologia Sinica, 28 (2): 36-44 2009.

•E. Mauri, P.M. Poulain, Južnič-Zonta, Z. MODIS chlorophyll variability in the northern Adriatic Sea and relationship with forcing parameters. Journal of Geophysical Research, vol 112, C03S11, 2007.

• E. Mauri, P.M. Poulain, G. Notarstefano. Spatial and temporal variability of the sea surface temperature in the Gulf of Trieste between January 2000 and December 2006. Journal of Geophysical Research. Vol. 113, No. C10, C10012. 2008.

• T.N. Miles, R. He, M. Li. Characterizing the South Atlantic Bight seasonal variability and cold-water event in 2003 using a daily cloud-free SST and chlorophyll analysis. Geophysical Research Letters, 36: Art. No. L02604 JAN 22 2009.

• N.P. Nezlin, K. Kamer, J. Hyde, E.D. Stein, ED. Dissolved oxygen dynamics in a eutrophic estuary, Upper Newport Bay, California. Estuarine, Coastal and Shelf Science, 82 (1): 139-151 MAR 20 2009.

• R. F. Sánchez, P. Relvas, M. Delgado. Coupled ocean wind and sea surface temperature patterns off the western Iberian Peninsula. Journal of Marine Systems, 68, 103-127. 2007.

… and other applications: