Reconstruction of Missing Meteorological Data Using

Wavelet Transform

N. Tuğbagül Altan, B.Berk Ustundag

Istanbul Technical University

Center of Agricultural & Environmental Informatics,

Uydu Yer İstasyonu Binası, TR 34469, Maslak

Istanbul, Turkey

Abstract – Land surface heterogeneity is effective on required

sensor population of the regional or national telemetry networks

used for agricultural information services. Temporary or

permanently missing data sometime may occur in such large

scale sensor networks due to type of failure. The agro-informatics

measurement network (www.tarit.org) in Turkey is planned to

have more than 30.000 sensors at 1200 stations. Agricultural

risks, irrigation management and many other online services

require continuity of the reliable data in real time. It is necessary

to complete the missing data as close to its actual value until

recovery in a fault tolerant acquisition system. A method is

developed for recovery of missing data by using the correlation of

wavelet coefficients of neighboring measurement stations. As

different from similar solutions we reconstruct missing data

through inverse wavelet transformation accompanied with

regression model together. Mean square error (MSE) and mean

absolute error (MAE) between the measured and reconstructed

temperature and humidity patterns are used as performance

measure. Mean square error (MSE) and mean absolute error

(MAE) are seen to be reduced more than 26% for temperature

data reconstruction with respect to pure linear regression case.

Keywords: Reconstruction of missing meteorological data,

Temperature, Wavelet Transform

I. INTRODUCTION

Earlier studies on missing meteorological data reconstruction were based on data estimation via the use of wavelet transform method. One of the most significant studies on the use of the wavelet transform method in relation to meteorological data was carried out in 1990s by Kumar and F. Georgiou. In their study, precipitation data was analyzed and decomposed into its components using multi resolution method [1].

In another study by M. Kucuk, N. Agiralioglu, 2006, the modeling of the hydraulic current sequences was intended using wavelet transform method. Stream-flow sequences of two independent measuring stations were separated into their components applying discrete wavelet transform. Modeling was based on regression type model developed by these components. As a result, it was found out that the models established by means of using the components selected in line with the climatic characteristics of each different regions included in this framework were more successful with regard to a number of error criteria [2].

The agro-informatics measurement network in Turkey is planned to have more than 30.000 sensors at 1200 stations. This study mainly intends temporary data reconstruction for uninterrupted real time spatial and temporal data services in a parcel based country wide system.

Measurement data of the two stations performing best correlation values to their neighboring station having the failure are used to predict time series data in the given case studies. Proposed method finds out the missing data by using linear regression model and wavelet transform techniques together. The reliability of the model is evaluated with respect to Mean Square Error (MSE) and Mean Absolute Error (MAE) calculations between the predicted and the measured data here in section III. Discrete wavelet transform using Daubechies wavelet function on training data where the signal is fragmented into an approximation is seen as an efficient way in this study.

Reconstructed data pattern of the target station that is virtually assumed to have data acquisition failure is named as training set when its own wavelet coefficients are used in inverse wavelet transformation. The difference between the target station measurement data pattern and the training set for a given time interval is said training error. When the data is reconstructed through the correlating data of the external stations then it is named as test data set here. The difference between the test data set and the real measurement pattern of the target station is named as test set error. MAE and MSE of the test sets are used for performance assessment here. As a result of comparative analysis, it is shown that using the developed proposed wavelet transformation method model reduces the mean absolute error of the test set.

II. DISCRETE WAVELET TRANSFORMATION

The coefficients between the signal and wavelet functions are computed for a certain time interval depending on the selection of the wavelet transform type. These coefficients show the correlation, i.e. the similarity between the wavelet function and the signal itself. Unlike continuous wavelet transform, discrete wavelet transform (DWT) calculates the transform coefficients based on a predefined scale and location set [3]. Consequently, in continuous wavelet transform, a large number of wavelet coefficients are computed, whereas in discrete wavelet transform the wavelet coefficients number is based on desired output [4]. Scaling and predefining the time

intervals in discrete wavelet transform facilitates the analysis of the signals and shortens the computation time.

In this study, discrete wavelet transform is preferred due to its scalability upon predefined coefficients and simplified framing features. Daubechies wavelet has orthonormal characteristic. Since fidelity of the reconstructed signal pattern is directly related to performance measure here Daubechies type wavelet has been preferred. Daubechies 6 wavelet (db6) was used for the first case study, Daubechies 3 wavelet (db3) was used for the second and the fourth case studies and Daubechies 5 wavelet (db5) was used for the third case study here, depending on results of the tested model at various orders. Transformed and reconstructed test signal is intended to have minimum possible loss in its characteristic features. Narrow framing functions where the sign changes rapidly and wide framing functions where the sign changes slowly is used for this reason. In other words, the signal is analyzed within a large time frame for low frequencies and within a small time frame for high frequencies by means of wavelet transform. Optimal time-frequency resolution of the reconstructed data is achieved with least loss of the characteristics of the original signal’s frequency components by means of this frame switching in wavelet transform [5, 6]. It is imposed that the scale s varies

only along the dyadic sequence 2�єᴢ�

by Mallat and Zhong

tolerate fast numerical implementations [7]. The equation used for the discrete wavelet transform is given below.

��,� (�

�) = �

�/���,������

�

��� (1)

In this equation, mrepresents the translation parameter of

the wavelet on the scale axis, n represents the translation

parameter of the wavelet on the time axis, �� represents the

value of the translation interval on the time axis, � represents a

fixed translation step (s�>1). The frequently used values for �

and �� are 1 and 2, respectively.

These two values are also used in this study and the

equation of the wavelet function has been rearranged as

indicated below.

ψ�,� (t) =2�/�ψ (2�t � n) (2)

Discrete wavelet transform is shown as specified below:

W�,�=2�/�� x!ψ"2�k � n$%&

!'& (3)

(�,� are the coefficients of wavelet transform having the

values with 2� scale(s), 2�)time "�$.

N is a number equal to the one of the multiples of 2. It is well known that a signal can be perfectly reconstructed from its wavelet-decomposed components: an approximation component and a set of details components [8]. The signal is divided into a certain number of scales by means of multi resolution analysis (MRA). The multi-resolution approach to wavelets enables us to characterize the class of functions ψ"x$є*�(R) that generate an orthonormal basis [9]. With this method, the signal divided into its approximation and details

components using DWT may be subsequently divided into its approximation and details components. For MRA, scales and wavelet functions with low and high filtration are used. By means of discrete wavelet transform and multi resolution analysis, the number of samples decreases by half at the end of each division process [10]. Yet, whereas the resolution decreases in time, it increases in frequency [11]. As the frequency bandwidth at each low level decreases in half, as a consequence uncertainty in the frequency also decreases. Fig. 1 indicates subsequent decomposition steps.

Figure 1. Discrete Wavelet Transform and Multi-Resolution Decomposition.

III. LINEAR REGRESSION WITH WAVELET MODEL

A. Regression Model

Our first study is conducted at Beyazkule station which is nearby Sanliurfa [12]. Distance r is between 95km and 105km for this station. For Beyazkule station temperature time series recordings were taken every ten minutes and cover 516 data points in total. This data set is divided into two parts. The first half consists of 258 data points and is defined for the training data set. The second half comprises the remaining 258 points and is defined for the testing data set. The first half of the data used is dated as 09.06.2012, 02:40 o’clock and the second half is dated as 12.06.2012, 16:30 o’clock. Before dividing Rumi, Aydogdu stations temperature time series into two parts as the training and test data sets, the correlation values between these stations with Beyazkule station are determined. These correlation values are given in Table I, expressing the relationship of temperature time series of Beyazkule with the other two stations.

TABLE I. WAVELET COEFFICENTS’ CORRELATION OF YAKACIK

STATION TO THAT OF NEAREST NEIGHBORING STATIONS AT 95KM TO

105KM DISTANCE

Station

Number

Station

Name

Correlation Ratio between Beyazkule

Station

18 Rumi 0,9576

19 Aydogdu 0,974

Two independent multivariable linear regression model is used for estimation of Beyazkule station. The data in training set from Beyazkule station is defined as dependent variable, while the data from Rumi and Aydogdu stations is used as

independent variables and so the coefficients of regression equation are calculated:

y = Temperature data of Beyazkule Station

+& = Temperature data of Rumi Station

+� = Temperature data of Aydogdu Station

, = Index of data

And the equation for computing the data for Beyazkule station is given below.

-.= α+β1.+.&+β2.+.� (4)

The coefficients of the linear regression equation are calculated as: a= 0.7871, b1 = 0.7582 and b2 = 0.2200.

Therefore, the linear regression equation acquired is as following:

-.= 0.7871+ 0.7582 +.& + 0.2200 +.� (5)

By using this equation on both the training and test inputs, predicted outputs are calculated and Mean Square Error (MSE) is calculated between the real outputs and predicted ones. The equation for MSE is given below:

MSE=&

�∑ "y1"234564573$

�1'& � y1"82391:;3964573$$

� (6)

MSE gives the square of the error so, peak values in the figure or rapid temperature drops lead to high error rates. For this reason in this study Mean Absolute Error (MAE) is also used. MAE gives the mean of the absolute difference of real and measured value [13]. Equation of MAE can be given as it follows:

ΜΑΕ=&

� ∑ |"y1"234564573$

�1'& � y1"82391:;3964573$| (7)

Real values and predicted values which are the results of the regression model on the training set are shown in Fig. 2 and on the test set are shown in Fig. 3.

Figure 2. Training temperature data set real values and the values found as a

result of the linear regression modeling with samples(x10 min.).

Figure 3. Test temperature data set real values and the values found as a result

of the linear regression modeling with samples(x10 min.).

B. Regression and Wavelet Transform Model

In the second model used, DWT has been applied to temperature-time series acquired from the stations by using Daubechies wavelet which was found by Ingrid Daubechies and enables to apply discrete wavelet transform to all of the stations involved in the model.

When compared to other separate level decomposition models, the db6 led to the best result for predicting temperature time series, which are decomposed by the fact that the number data is downsized in half at each level [14]. In this comparison, the correlation analysis between temperature-time series of Beyazkule station with the other stations again for each level has been used. By using the dependent variables (the data of Beyazkule station) and the independent variables (the data of Rumi and Aydogdu stations), 7 different linear regression equations have been formed for the approximation (A6) and details components (D1, D2, D3, D4, D5, D6) of Beyazkule, Rumi and Aydogdu stations and the parameters of regression equation has been calculated. The generated model equations are listed below:

Temperature value (T) of the Beyazkule Station (BS) is

=>?=@6>?+B6>?CB5>? C B4>? + B3>?+B2>?+B1>? (8)

Temperature value (T) of the Rumi Station (RM) is

=>?= @6H?+ B6H?CB5H? C B4H?+ B3H?+ B2H?+ B1H? (9)

Temperature value (T) of the Aydogdu Station (AS) is

=I?= @6I?+ B6I?CB5I? C B4I?JB3I?+ B2I?+ B1I? (10)

i(from 1 to 6), the regression models shown below between approximations and details components have been composed to represent the degree of detail component.

@6>? (calculated) =a + b1. @6H?+ b2. @6I? (11)

B>? (calculated) =a + b1. B,H?+ b2. B,I? (12)

”a” regression coefficient, “b1”, “b2” parameters at each level have been found. The approximation and detail

components found as a result of the linear regression have been collected by applying inverse wavelet transform. The inverse wavelet transform are described in [7].

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 4. The graphics about regression model of approximate, detail and

original signals found and tested the training set, (a) A6 Approximation, (b)

D1 , (c) D2, (d) D3, (e) D4 ,(f) D5, (g) D6, detail components, (h) Observed

and Calculated Temperature Series.

The predicted temperature time series have thus been found. The training and test sets are separated according to the approximation (A6) and detail components (D1, D2, D3, D4, D5, D6) by using db6 wavelet to compare the result. The graphical representations of the comparison of the components occurring as result of modeling with the approximation and details components of the training and test sets are given above. The graphical representation of the modeling with the test set is given bellow.

Figure 5. The graphics about regression+ wavelet transform model, original

signals found on the training set and tested on the test set, Observed and

Calculated Temperature Series with samples(x10min.).

TABLE II. TIME SERIES TEMPERATURE DATA PREDICTION

PERFORMANCE WITH RESPECT TO TEMPERATURE DATA SETS OF TWO

NEIGHBORING STATIONS (THE FIRST CASE STUDY)

The Linear Regression Model MSE MAE

On the Training Set 0,962 0,7523

On the test Set 2,0984 1,0923

The Linear Regression+Wavelet Model MSE MAE

On the Training Set 0,7291 0,6995

On the Test Set 1,5323 0,9526

Test set of linear regression+wavelet model yields better MSE

and MAE results as shown in Table II. The second study is conducted on Yakacik Station within

less distance radius distance r: 15<r<20 km. Station positions with respect to Yakacik station are shown in fig.6. The correlation values between these stations with Yakacik Station are determined. These correlation values are given in Table III.

Figure 6: Distances between three stations used in second study.

TABLE III. STATIONS WITHIN A RADIUS OF 15 KM BETWEEN 20KM

(THE SECOND CASE STUDY)

Station

Number

Station

Name

Correlation Ratio between Yakacik

Station

20 Akçatat 0,9834

25 Poyralı 0,9619

Observations made for the Yakacik Station contain 1432

data points. This data set was divided into two parts, first 716 of these data points are defined as the training data set and the last 716 points are defined as the test data set. The first of the data used is dated on 01.05.2011 and at 13:50 o’clock and the last one is dated on 11.05.2011 and at 12:20 o’clock. Db3 wavelet gives the best results out of other types of this wavelet applied on the model at various levels. The results of the study are shown in below.

TABLE IV. TIME SERIES TEMPERATURE DATA PREDICTION

PERFORMANCE WITH RESPECT TO TEMPERATURE DATA SETS OF TWO

NEIGHBORING STATIONS (THE SECOND CASE STUDY)

The Linear Regression Model MSE MAE

On the Training Set 0,4918 0,5321

On the test Set 0,7468 0,662

The Linear Regression Model +Wavelet

Model MSE MAE

On the Training Set 0,4572 0,5174

On the Test Set 0,6884 0,6453

In the third study, humidity series is estimated using Poyrali and Akcatat humidity series. For Yakacik station humidity time series recordings were taken every ten minutes and cover 2396 data points in total. This data set was divided into two parts, first 1198 of these data points are defined as the training data set and the last 1198 points are defined as the test data set. The first of the data used is dated on 09.05.2011 and at 21:30 o’clock and the last one is dated on 18.05.2011 and at 05:00 o’clock. Db5 wavelet gives the best results out of other types of this wavelet applied on the model at various levels. The results of the study are shown below.

TABLE V. TIME SERIES HUMIDITY DATA PREDICTION PERFORMANCE

WITH RESPECT TO HUMIDITY DATA SETS OF TWO NEIGHBORING STATIONS

(THE THIRD CASE STUDY)

The Linear Regression Model MSE MAE

On the Training Set 20,5028 3,4997

On the test Set 23,308 3,4757

The Linear Regression Model +Wavelet

Model MSE MAE

On the Training Set 17,9043 3,2996

On the Test Set 21,51 3,3353

The graphical representation of the modeling with the test set

is given bellow.

Figure 7. The graphics about regression+ wavelet transform model, original

signals found on the training and tested on the test sets, Observed and

Calculated Humidity Series in third study.

We also applied explained data reconstruction method to other

types of time series of meteorological data in TARIT project.

Here we present humidity prediction of Yakacik station by

using humidity and temperature measurements of Poyrali and

Akcatat stations. The prediction model equation is shown in

equation 13. Humidity data record was sampled every ten

minutes and contains 1198 measurements at Yakacik station in

the given example application. This data set was divided into

two parts such that first 599 of these data points are defined as

the training data set and the last 599 points are defined as the

test data set. The first part of the data used is dated on

05.05.2011 and at 17:40pm and the last one is dated on

18.05.2011 and at 05:00am. Db3 wavelet yields the best

results amongst the all types of the applied wavelet types at

various levels.

-.= α+K&.+.&+K�.+.� +KL.+.L+KM.+.M (13)

Predicted humidity data reconstruction performance with respect to humidity and temperature time series data of two neighboring stations is shown terms of MAE and MSE in Table VI. It has also seen that the proposed type of wavelet based time series data prediction reduced the error rate more with respect to basic linear regression equivalent of the same data set.

TABLE VI. TIME SERIES HUMIDITY DATA PREDICTION PERFORMANCE

WITH RESPECT TO HUMIDITY AND TEMPERATURE DATA SETS OF TWO

NEIGHBORING STATIONS (THE FOURTH CASE STUDY)

The Linear Regression Model MSE MAE

On the Training Set 16,7777 3,1564

On the test Set 38,6789 4,9781

The Linear Regression Model +Wavelet

Model MSE MAE

On the Training Set 15,6041 3,0174

On the Test Set 37,6228 4,9302

The graphical representation of the modeling with the test set

is given below.

Figure 8. The graphics about regression+ wavelet transform model, original

signals found on the training set and tested on the test set, Observed and

Calculated Humidity Series for the fourth study.

IV. CONCLUSION

We proposed a data reconstruction scheme for fault tolerant

large scale agro-meteorological data acquisition networks. We used regression and wavelet together in our model and the results have shown to be better than the pure regression solution. Reconstruction performance was measured with respect to actual value of the station having missing data that was assumed to have after recovery and the predicted data by using the data of neighboring stations at a chosen distance “r”. Mean square error (MSE) and mean absolute error (MAE) are calculated for the measurement of the reconstruction performance. Prediction performances are given for two different neighborhoods here. In the first case distance r was chosen as 100km. We have shown that MSE was improved 26.98% and MAE was improved 12.79% in the first temperature data set. Improvement rate reduced at closer distance predictions (less than 20km) to 7.82% on MSE and

2.53% on MAE since the even the direct temperature value of neighboring measurement stations were close to actual value of the predicted one. In the third phase of the study, Improvement rate reduced to 7,72% on MSE and 4,04% on MAE In the fourth study, improvement rate reduced to 2,73% on MSE and 0,96% on MAE. We have also seen that the proposed model is applicable to different variants of agrometeorological data set. Humidity data reconstruction is demonstrated for two cases that temperature and humidity data records of the two neighboring stations are used with and without its own temperature data.

The results of different case scenarios have shown that the

linear regression combined with wavelet transform method

yields far better results than the linear regression model alone.

Error boundaries are also acceptable for fault tolerant real time

agricultural information systems depending also upon right

station location assignments. We continue this research by

applying the method at north Anatolia where the topographic

heterogeneity is high hence we will have a manageable error

model for different conditions. Cross correlation of different

data types of the same and neighboring stations are also

considered in the ongoing step of the implementation.

Real data of Turkish Agroinformatic Network Project

(TARIT) chosen from South Eastern Anatolia Stations were

used in this study. Matlab (R2011b) was used for coding,

simulation and analysis besides the in-system application

software compiled in MS C#.

REFERENCES

[1] P. Kumar, E. Fofoula- Georgion, 1193, A Multi Component Decomposition of Spatial Rainfall Fields, 1. Segregration of Large and Small Scale Features Using Wavelet Transform Water Resources Research”, 29,8, 2515- 2532.

[2] M. Kucuk, N. Agiralioglu, 2006, “Modelling of Streamflow Series

Using Wavelet Transform Technique”, itudergisi journal,vol 5, No 2, pp. 69-80, April 2006.

[3] T. Partal, 2007, “Yagis Verilerinin Yapay Sinir Agları ve Dalgacik Donusumu Yontemleri ile Tahmini”, PhD Thesis, Istanbul Technical University,2008, pp.73-74.

[4] W. Briggs, R. Levine,” Wavelets and Field Forecast Verification”,vol. 125,pp.1330-1331,October 1996.

[5] I. Daubechies, “The wavelet transform, time-frequency localization and signal analysis”, IEEE Transactions on Information Theory, 36(5), 961-1005, 1990.

[6] Y. Zhang, Y. Wang, W. Wang, B. Liu, “Doppler ultrasound signal denoising based on wavelet frames”, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 48(3), 709-716, 2001.

[7] S. Mallat, S. Zhong, "Characterization of Signals from Multiscale Edges," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 7, pp.710-732, July 1992.

[8] J. T. Hsu, B. Tian ,C.-Chung Li, Q. Liu, L.-Sen. Pon, M. Sun, R.J. Sclabassi, “Signal Recover from the Approximation Component in the Non-Downsampled Wavelet Transform”, 2003 IEEE International Conference on Neural Networks and Signal Processing, 2003. Vol.1, p.p.704-707, December 2003.

[9] S. Mallat, "A Theory for Multiresolution Signal Decomposition: The Wavelet Representation," IEEETransactions on Pattern Analysis and MachineIntelligence, Vol. 11, No. 7, pp.674-693, July 1989.

[10] D. Walnut, (2002). An Introduction to Wavelet Analysis (2nd ed.). NewYork: NY 10010,USA.

[11] N. Ari, Ş. Ozen, H. Colak, (2008), Dalgacık Teorisi(1st Ed.).Ankara:Palme.

[12] Istanbul Technical University, Ministry of Agriculture and Food; State Planning Office “Crop Yield Monitoring and Prediction (Tarım Rekolte Tahmini ve Kuraklık Izleme) Project”, TARIT , Turkey, 2008-2011

[13] T. Partal, E. Kahya, K. Cigizoglu, “Estimation of Precipitation Data Using Artificial Neural Networks and Wavelet Transform”, itudergisi journal, Vol 7, No 3, pp. 73-85, June 2008.

[14] S. Bilgin, 2008, “Kalp Hızı Degiskenliginin Dalgacik Donusumu ve Yapay Sinir Agları Kullanarak Analizi”, PhD Thesis, Sakarya University,2008.