simple and accurate algorithms - department of electronic
TRANSCRIPT
How to Derive Mean and Mean
Square Error for an Estimator?
Hing Cheung So (蘇慶祥)
http://www.ee.cityu.edu.hk/~hcso
Department of Electronic Engineering City University of Hong Kong
H. C. So Page 1 Nov. 2013
Outline Introduction
Mean and Mean Square Error Formulas for Scalar
Examples for Scalar Estimation
Mean and Mean Square Error Formulas for Vector
Examples for Vector Estimation
List of References H. C. So Page 2 Nov. 2013
Introduction What is Estimation? Estimation refers to accurately finding the values of parameters of interest from the observed data which consist of two components, viz., signal and noise A generic model for estimation of a scalar is:
where is the observation vector, signal is a known function of and noise is an additive random process
The estimation problem is to find given H. C. So Page 3 Nov. 2013
Why Estimation is Needed? Many science and engineering problems can be boiled down to parameter estimation: Radar Radar system transmits an electromagnetic pulse . It is reflected by an aircraft, causing an echo to be received
H. C. So Page 4 Nov. 2013
Time delay is round trip propagation time of radar pulse. If we know , can be obtained as
, is speed of light H. C. So Page 5 Nov. 2013
Mobile Communications
If we know one-way propagation time of the signal traveling between mobile station and base station (BS), then the target position can be obtained using three BSs
H. C. So Page 6 Nov. 2013
Speech Processing
For a voiced speech, it can be modeled as a periodic signal and it is important to estimate its pitch or fundamental frequency for analysis.
0 0.005 0.01 0.015 0.02
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
time (s)
vow
el o
f "a"
H. C. So Page 7 Nov. 2013
Image Processing Estimation of the position and orientation of an object from a camera image is useful when using a robot to pick it up, e.g., bomb-disposal Biomedical Engineering Estimation the heart rate of a fetus and the difficulty is that the measurements are corrupted by the mother’s heart beat as well Seismology Estimation of the underground distance of an oil deposit based on sound reflection due to the different densities of oil and rock layers Astronomy Estimation of the periods of orbits H. C. So Page 8 Nov. 2013
How to Perform Estimation? Least squares (LS) and maximum likelihood (ML) are two standard estimation approaches Consider the model of The LS estimator does not require the probability density function (PDF) of , and its estimate is obtained by minimizing a sum of squared error:
where
H. C. So Page 9 Nov. 2013
To produce the ML estimator, the PDF of is required Assuming that is a zero-mean Gaussian noise, the PDF of the observed vector , which is parameterized by , is
where
The ML estimate is:
When is white with variance , the PDF reduces to
ML estimate is reduced to LS solution H. C. So Page 10 Nov. 2013
How to Assess Estimators? Two standard performance measures for assessing accuracy of an estimator are bias and mean square error (MSE): and It is desired that or , indicating that the estimator is unbiased, and MSE is as small as possible For an unbiased estimator, MSE is equal to variance:
In general:
H. C. So Page 11 Nov. 2013
Consider a simple problem of estimating a DC level from: where has mean 0 and variance We easily suggest three estimators:
H. C. So Page 12 Nov. 2013
It is easy to show:
;
;
;
is the best among the three because it has zero bias and
minimum variance Is optimum? How to compute bias and MSE for more general cases? H. C. So Page 13 Nov. 2013
Cramér-Rao Lower Bound (CRLB) CRLB is performance bound in terms of minimum achievable variance provided by any unbiased estimators Its derivation requires knowledge of the noise PDF and the PDF must have closed-form Although there are other variance bounds, CRLB is simplest Suppose the PDF of where , is The CRLB for can be obtained in two steps:
Compute the Fisher information matrix CRLB for is the entry of , H. C. So Page 14 Nov. 2013
Consider with zero-mean white Gaussian noise:
That is, is the optimum estimator for
H. C. So Page 16 Nov. 2013
Mean and Mean Square Error Formulas for Scalar Recall the signal model:
Suppose the scalar is estimated by minimizing a differentiable cost function constructed from , : This implies
H. C. So Page 17 Nov. 2013
At small estimation error conditions, is close to . Applying Tayler series expansion yields: If is sufficiently smooth around , then Hence
Similarly,
H. C. So Page 18 Nov. 2013
Note: When is a quadratic function:
and
When , is an unbiased estimate of For unbiased estimator:
H. C. So Page 19 Nov. 2013
Examples for Scalar Estimation For simplicity, we assume that the noise is white Gaussian process with variance DC Level Estimation Recall the model:
Using the LS approach, the cost function to be minimized is
H. C. So Page 20 Nov. 2013
To apply the bias and MSE formulas, we compute:
and
Hence:
and
which align with previous analysis H. C. So Page 21 Nov. 2013
Time-Difference-of-Arrival Estimation The simplest model is to estimate the time-difference-of-arrival between two signals: where , and are independent zero-mean white Gaussian variables with , It is clear that is most similar to
. As a result, can be estimated by maximizing the cross-correlation between and :
H. C. So Page 22 Nov. 2013
However, is generally not an integer and thus is a continuous function of Using the convolution theorem, has the form of
Applying the bias and MSE formulas, we obtain:
and
which is also the CRLB H. C. So Page 23 Nov. 2013
Frequency Estimation of a Complex Sinusoid The signal model is:
where , and
A conventional approach for estimating is to search the periodogram peak:
Applying the bias and MSE formulas, we obtain:
and
which is also the CRLB H. C. So Page 24 Nov. 2013
Frequency Estimation of a Real Sinusoid The signal model is:
where , and According to the linear prediction property:
A LS cost function for estimating is then:
H. C. So Page 25 Nov. 2013
The LS estimate of is:
Hence the frequency estimate is
which is known as the modified covariance method H. C. So Page 26 Nov. 2013
Applying the bias and MSE formulas, we obtain: and
if is sufficiently large Hence
where H. C. So Page 27 Nov. 2013
Mean and Mean Square Error Formulas for Vector For estimation of a vector from minimizing , the formulas are generalized as follows:
and
where is the gradient vector and is the Hessian matrix:
H. C. So Page 29 Nov. 2013
As a result, Similarly, the covariance matrix is:
The MSE of is given by entry of H. C. So Page 30 Nov. 2013
Examples for Vector Estimation Estimation of a Linear Model
The linear data model is:
where is known, is unknown vector, and
Employing , the weighted LS cost function is:
Applying the bias and MSE formulas, we obtain:
and
These align with the best linear unbiased estimator (BLUE) H. C. So Page 31 Nov. 2013
Parameter Estimation of a Real Sinusoid
The signal model is:
where , and , while is a white Gaussian process with variance According to ML or LS, we construct:
H. C. So Page 32 Nov. 2013
Applying the bias and MSE formulas, we obtain:
which is the inverse of the Fisher information matrix That is, the estimator is optimum H. C. So Page 33 Nov. 2013
Localization using Range Measurements Consider positioning of a source at by sensors at known coordinates , If we have the one-way propagation time measurements, they can be easily converted to ranges: where and is white The ML or LS cost function is
H. C. So Page 34 Nov. 2013
As a result,
With tedious calculation, we have
which is the inverse of the Fisher information matrix That is, the estimator is optimum H. C. So Page 36 Nov. 2013
Apart from the nonlinear approach, can be linearized:
where
Hence the signal model is now linear:
where
H. C. So Page 37 Nov. 2013
The weighted LS cost function to be minimized is
and the estimate is
Applying the bias and MSE formulas, we obtain:
MSEs of and are given by and entries of
H. C. So Page 39 Nov. 2013
To achieve higher accuracy, the information of should be utilized, which results in a constrained optimization problem:
The solution can be derived using the method of Lagrange multipliers To analyze the performance, the constrained problem can be converted to an unconstrained one by putting the relation of into :
H. C. So Page 40 Nov. 2013
Applying the bias and MSE formulas, we obtain:
which is the inverse of the Fisher information matrix That is, the estimator is also optimum H. C. So Page 41 Nov. 2013
List of References [1] H.C. So, Y.T. Chan, K.C. Ho and Y. Chen, “Simple
formulas for bias and mean square error computation,” IEEE Signal Processing Magazine, vol.30, no.4, pp.162-165, Jul. 2013
[2] S.M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory, Prentice-Hall, NJ: Englewood Cliffs, 1993
[3] V.H. MacDonald and P.M. Schultheiss, “Optimum passive bearing estimation in a spatially incoherent noise environment,'” Journal of the Acoustical Society of America, vol.46, no.1, pt.1, pp.37-43, Jul. 1969
[4] H.C. So, Y.T. Chan and F.K.W. Chan, “Closed-form formulae for optimum time difference of arrival based localization,” IEEE Transactions on Signal Processing,
H. C. So Page 42 Nov. 2013
vol.56, no.6, pp.2614-2620, Jun. 2008 [5] K.W.K. Lui and H.C. So, “Performance of modified
covariance estimator for a single real tone,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol.E90-A, no.9, pp.2021-2023, Sep. 2007
[6] K.W. Cheung, H.C. So, W.-K. Ma and Y.T. Chan, “Least squares algorithms for time-of-arrival based mobile location,” IEEE Transactions on Signal Processing, vol.52, no.4, pp.1121-1128, Apr. 2004
[7] H.C. So, “Source Localization: Algorithms and Analysis,” Handbook of Position Location: Theory, Practice and Advances, Chapter 2, S.A. Zekavat and M. Buehrer, Eds., Wiley-IEEE Press, 2011
H. C. So Page 43 Nov. 2013