empirical frequency-domain optimal parameter estimate for black-box processes

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: REGULAR PAPERS, VOL. 53, NO. 2, FEBRUARY 2006 419

Empirical Frequency-Domain Optimal ParameterEstimate for Black-Box Processes

Kueiming Lo, Member, IEEE, Hidenori Kimura, Fellow, IEEE, Wook-Hyun Kwon, Fellow, IEEE, and Xiaojing Yang

Abstract—Most of the previous signal processing identificationresults have been achieved using either time-domain or fre-quency-domain algorithms. In this study, the two methods werecombined to create a novel identification algorithm called theempirical frequency-domain optimal parameter (EFOP) estimateand the recursive EFOP algorithm for common linear stochasticsystems disturbed with noise. A general prediction error criterionwas introduced in the time-domain estimate. By minimizing thefrequency-domain estimate, some general prediction error criteriawere constructed for Black-box models. Then, the parameterestimation was obtained by minimizing the general predictionerror criterion. This method theoretically provides the globallyoptimum frequency-domain estimate of the model. It has advan-tages in anti-disturbance performance, and can precisely identifya model with fewer sample numbers. Lastly, some simulationswere carried out to demonstrate the validity of the new method.

Index Terms—Black-box models, disturbance noise, fre-quency-domain estimation, general prediction error criterion,time-domain method.

I. INTRODUCTION

AMAJOR TASK of signal processing is modeling andidentification. Basically, a model is constructed from

measured data. Due to signal drift or subject to interference ofnoise and unmodeled dynamics, the observed data are usuallycorrupted. Therefore, a posteriori data information consists ofa finite number of corrupted point frequency response estimatesof the unknown plant. Clearly, large estimation errors wouldresult if the identification is based on corrupted data.

The problem of how to accurately identify a disturbed systemwith noise has been receiving significant attention. Many iden-tification methods have been proposed in the literature. Thesemethods are divided into the time-domain method and fre-quency-domain method. Due to the time-domain method’sdirect and convenient implementation, it is good for designinga controller in the adaptive control. Much research has beenconducted on the time-domain method, of which the prediction

Manuscript received June 23, 2004; revised June 9, 2005. This work was sup-ported by the Funds 2004CB719400, NSFC60474026, JSPS, and ARC at Ts-inghua University, Beijing, China. This paper was recommended by AssociateEditor R. M. Rao.

K. Lo is with the School of Software, Tsinghua University, Beijing 100084,China (e-mail: [email protected]).

H. Kimura is with the Bio-mimetic Control Center, Institute of Phys-ical and Chemical Research (RIKEN), Nagoya 463-0003, Japan (e-mail:[email protected]).

W.-H. Kwon is with the School of Electrical Engineering, Seoul NationalUniversity, Seoul 151-742, Korea (e-mail: [email protected]).

X. Yang is with the Department of Mathematical Sciences, Tsinghua Univer-sity, Beijing 100084, China (e-mail: [email protected]).

Digital Object Identifier 10.1109/TCSI.2005.855737

error (PE) method [1] is popularly used in system identificationand control. Particularly, the celebrated least-squares (LS)method, which is the most common choice among the PE cri-terion, has been the dominant estimation method for parameteridentification. It is also frequently encountered in the modelanalysis area of mechanical engineering and has become themain method of parameter estimation [2].

However, since a linear time-invariant model is often char-acterized by its transfer function, some applications, likemicrowave fields, impedances, etc., are naturally made directlyin the frequency domain, and the properties of a closed-loopsystem can be accurately and intuitively determined bystudying the frequency response function. This view has beenless common in the traditional system identification literature,but has been of great importance in the mechanical engineeringcommunity, vibrational analysis field, and so on. It is veryuseful for describing the signal characteristics in the frequencydomain. Estimating the frequency response of a system fromobserved input–output data is a common requirement. The useof the PE method may also result in large estimation errors. Inparticular, the time-domain method fails to provide a better es-timation of the system transfer function. For this reason, manyrecent studies have investigated system identification usingthe frequency-domain estimate [3]–[6]. With experimentalinformation consisting of a discrete set of corrupted frequencyresponse samples, an alternative approach based on interpola-tion of a given frequency function was formulated in [7]–[10].The subspace-based identification method, which has emergedrecently, identifies state-space models from input–output data.The attractive aspect of this algorithm lies in its usefulness inidentification of multivariable systems. Subspace methods tofit frequency-domain data are treated in [9]–[12]. Nevertheless,these approaches are not easily implemented [10], [11], and donot provide good estimation accuracy [13].

The selection of an identification method is an important de-cision to be made in system identification. However, an identi-fication method, efficient or not, is primarily dependent on thechoice of criteria. This choice must be based on the method’sefficiency in identifying the system, and it should be easy toperform in practical engineering. In the time-domain category,the main identification methods used in engineering are derivedfrom the PE criterion [1], [14]–[16]. Much identification re-search has also been based on the PE criteria (see, for example,[16]–[18], etc.). Several different norms and metrics for the PEcriterion related to the sensibility and robustness of identifica-tion were discussed in [14], [15], and [19]. However, the PEmethod has a potential risk of being stuck in a local minimum,which often results in a poor identification model [20], [21]. The

1057-7122/$20.00 © 2006 IEEE

420 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I: REGULAR PAPERS, VOL. 53, NO. 2, FEBRUARY 2006

design of allpass filters using phase response criteria was con-sidered in [22] and [23]. These approaches are not easily imple-mented. The concept of the general prediction error (GPE) wasproposed in [24] and [25], where FIR models and AR modelswere analyzed.

In this study, the identification of linear discrete systems withnoise disturbance was considered. Based on the linear unbiasedempirical transfer function estimate, the empirical optimal fre-quency-domain estimate was obtained. Since it is difficult to getan analytical solution using frequency-domain methods, bothfrequency-domain and time-domain methods were treated insome studies [7]–[11], [26]. In [11] and [26], the robust identifi-cation from frequency-domain and time-domain measurementswere investigated by using a min-max criterion. In this paper theidentification technique was based on the GPE criterion. Fromthe optimal frequency-domain estimate, some general predic-tion error criteria were constructed for Black-box models. Thetime-domain estimate and the frequency-domain estimate werethen combined to form the empirical frequency-domain optimalparameter (EFOP) estimate and the recursive EFOP estimate fordiscrete systems with noise disturbance. This method provides aglobally optimal frequency-domain estimate and minimizes theGPE criterion. It has obvious advantages in anti-disturbance per-formance and can precisely identify a model from fewer samplenumbers. Lastly, several simulation examples were included toillustrate the method’s reliability.

II. DISCRETE FOURIER TRANSFORM

Most data are collected as samples of the input and outputtime signals. There are occasions when it is natural and benefi-cial to consider the Fourier transforms of the inputs and the out-puts to be the primary data. Let be a finite sequenceand , where is the sample number and

. Then the discrete Fourier transform (DFT) of thesequence is defined as [14]

(1)

Given the frequency values , the inverse DFT(IDFT) yields the time sequence

If and are the finite sequences, then timesignal

is called the convolution (or convolution sum) of the sequencesand . From the above definitions, we obtain

the following properties.

Lemma 1: (Parseval’s Relationship [14]) Let, . Then

(2)

From the convolution definition, it is easy to see that the oper-ation of the DFT convolution has an associative property. How-ever, the commutative property can not be held. Nevertheless,we have the following result.

Lemma 2: Suppose the finite sequence satisfies, . Then

(3)

where and .Proof: By the above definitions

Because , and, from the above relation

III. EMPIRICAL OPTIMAL FREQUENCY-DOMAIN ESTIMATE

Let be an output sequence of a discrete system.is the prediction of output . is the system parameter.

is the prediction error of the system attime . The prediction error vector of the system is defined as

Definition 1: Suppose that is a prediction errorvector of a discrete system. Then a function is a GPE criteriaof the system if the function is a positivedefinite function of the vector with respect to theparameters and .

LO et al.: EFOP ESTIMATE FOR BLACK-BOX PROCESSES 421

There is the basic difference between the GPE criterion andthe PE criterion. In the GPE criterion is a function of ,

, and , while the PE criterion is defined as [14], [15]

where is a norm. It is easy to show that the PE criterion is con-tained in the GPE criterion, but they are not equal. For example,the function

(4)

is a GPE criterion of a linear predictive system, which is called ageneral quadratic criterion, where the matrixis a positive definite symmetric matrix. If is a diagonalmatrix

then the function defined by (4) is the common weightedquadratic criterion. Specifically, the function defined by (4) isalso the standard quadratic criterion if matrix ,where is the identity matrix. Otherwise, if is not adiagonal matrix, it is difficult to choose a norm to satisfy

Thus, the GPE criterion is different from the ordinary PE crite-rion.

A logical idea in identification using the time-domain methodis that a model can be constructed by minimizing its predictionerror. Based on the parameter estimation principle, the PE crite-rion, a well known technique in system identification, was sum-marized in [14]. In this study, our identification method is basedon the GPE criterion.

In the GPE criteria we face the question of how to choose apositive definite function such that the system identificationis more accurate. If relationship (4) is taken as the form of GPEcriterion, the meaning of this question is very clear. In fact, sincethe prediction error vector is determined by the given system, inGPE criterion (4) the matrix is the only object that canbe regulated. The other matrix will result from a dif-ferent estimate. In criterion (4) how is a positive definite sym-metric matrix chosen in order to obtain a more accurateestimate for the given system? This paper presents a method toconstruct such a matrix as well as an efficient GPE cri-teria. This idea is derived from optimizing the transfer functionestimate. A set is defined as

Obviously is a closed subset in . Then we have the fol-lowing results.

Lemma 3: Suppose that is an independent randomsequence with zero mean value and variances . Let

Then, the variance is the minimum if

This is a linear unbiased estimation and a common techniqueused in globally optimal problems. The empirical optimal fre-quency-domain estimation is based on this result. Based on thisLemma some efficient GPE criteria can be constructed for sto-chastic systems.

Suppose that a function is a GPE criterionof a discrete system. The estimation of the system parameteris then taken to be the value that minimizes the GPE criterion

(5)

Estimation (5) is called the EFOP estimate of the parameter ifthe corresponding GPE criterion is constructed with the empir-ical optimal frequency-domain estimation [24].

IV. EFOP ESTIMATE FOR BLACK-BOX MODELS

The linear model structures parameterized in terms of a pa-rameter vector is that

(6)

Here , , and are the output, input, and the distur-bance noise. and are stable filters. is theunit delay operator. If is a white noise, the predictor ofmodel (6) is

(7)

and the prediction error is

(8)

System (6) is a general linear discrete model that can be usedin prediction analysis and model structure [14]. In practice,Black-box models [14], [18] as well as ARMAX models areusually considered. A considerable amount of recent literature(see [16], [27]–[31], etc.) has been devoted to this subject.Many of the research efforts have been focused on how to ac-curately identify this kind of dynamical model. Because outputfeedback signals are involved, the system signals are subject togreat interference. Some previous identification methods, suchas the common LS method or the extended least-squares (ELS)method, could not give the exact estimation if those modelswere disturbed by noise. Hereby, the Black-box models

(9)


are also treated in this study, where , , , andare polynomials of the backward operator

The filters and corresponding to model (9) aregiven as

If the noise is a white noise, it follows from (7) and (8), thepredictor and the prediction error of model (9) that

(10)

(11)

The estimation of the system parameters is based on a se-quence of given input data and the sequence of mea-sured output responses . Since , , , and

are -order, -order, -order, and -order filters, re-spectively, the initial values , ,and are used to calculate model (9). Denote

. For we subse-quently define the initial values as

Remark 1: The above initial values are taken for a guaranteerelationship (A4). This is not an essential requirement, but onlyfor the convenience of theoretical analysis. These values caneasily be satisfied if . They also can be realizedin other cases, for instance the following cases.

1) Given finite input sequence , it is easy to gen-erate a periodic signal with . By this periodic inputsignal the initial output values could be obtained frommodel (9). The initial output values created by thismethod correspond with the above initial requirement ifwe ignore some weaker disturbance. Even if there aresome bounded independent disturbances that inter-fere with the initial system, we only need to add a smallterm with in the covariance of Lemma 4 [14].

2) We can add the postwindowing data [14] after the se-quences and such that

Then the initial values would satisfy the above require-ment when we consider the signal sequencesand .

3) If we use zeros instead of all the initial values we canshow that relationship (A4) has only added a term notmore than [14]. Its influence on the estimationis much less and tends to zero with the sample numberlarge.

Instead of minimizing the prediction error as the PE crite-rion, we establish the GPE criterion via optimizing the empir-ical transfer function estimation. Let be the estimationof the empirical transfer function of model (9). The es-timation errors of the empirical transfer function are

(12)

Lemma 4: Suppose that of model (9) is a whitenoise with variance . Then

(13)

if

if(14)

where .The proof of Lemma 4 is given in Appendix A. Combining

Lemma 3 and Lemma 4 we immediately derive the followingcorollary.

Corollary 1: If the condition of Lemma 4 holds then

(15)

where

Suppose that is a finite input sequence of model (9).are a sequence of the auxiliary signals determined

by the relationship:

(16)

For the initial values of are defined to fitin with .

Let

ifif

(17)

(18)

where ,.


Theorem 1: Suppose that the noise of model (9) iswhite noise with variance . Then

(19)where is the prediction error vector of model (9). Matrix

is determined by relation (18).Proof: Let , , , and

be the discrete Fourier transforms ofthe input sequence , the output sequence ,the auxiliary sequence , and the prediction errorsequence , respectively. Similar to the deduction inAppendix A, from relations (11), (12), and (16) it yields

(20)

(21)

Denote as the convolution betweenthe sequences and . By using Lemma 2, and rela-tions (12), (20), and (21), the discrete Fourier transform of thesequence is that

From Lemma 1, relation (16), and the above relation it followsthat

(22)

(23)

According to the definitions above

Then, by using relations (14), (15), (18), (23), and the aboverelation we have

(24)

Remark 2: Theorem 1 and Corollary 1 show that the perfor-mance function

(25)

is the globally empirical optimum of the estimation errorof the transfer function. This form is the same as

expression (4). Therefore, function is a GPE criterion ofthe globally optimal frequency-domain estimate of model (9).From (5) and (25) the EFOP estimate of the parameters inmodel (9) should be

(26)

Suppose that and are unknownsystem parameters. Denote


Model (9) can be rewritten in a linear regression form

(27)

where is the noise . It is equal to the error, while the signals and matrix are chosen as

(28)

where for the initial values of are definedto fit , and the vector is defined as

withifif

Then we have the next lemma.Lemma 5:

(29)

Proof: By using Lemma 1 and Lemma 2, similar to theproof of (A4) it yields

(30)

and

(31)

From relations (28), (30), and (31), we have

(32)

Combining (24) with (32), result (29) follows.

Lemma 4 shows that performance function (25) can berewritten as

(33)

Comparing (25) with (33), it is easy to see that performancefunction (33) has an advantage in estimating parameters be-cause it contains the token as well as the linear regression

. Therefore, the estimation

(34)

is also the EFOP estimate of model (9).The parameter estimation depends on the choice of matrix

. A different matrix will correspond to a differentestimation. A concrete form of by some Toeplitz matricesis given here.

Remark 3: Although the form of expression (34), a GPE cri-terion for system (9), appears to be similar to the minimizingvariance estimation (MVE), there is also an essential differencebetween them. In the sense of giving the smallest covariancematrix in the MVE, the best choice of the matrix shouldbe the noise covariance matrix. As pointed out in [14], “No-tice that the MVE requires knowledge of the noise covariancematrix, which might not be a realistic assumption.” Therefore,the matrix in the MVE could not be obtained in practice.Even if some knowledge about the system noise is known, theform of the MVE is still different from relationship (34). For ex-ample, if the noise is a white noise with variance , the matrix

in the MVE should be . Then the MVE just becomesthe LS Estimation. However, the matrix in this paper isa Toeplitz matrix and is generated by a system output signal.The simulations will also compare the efficiency of these twomethods.

V. PARAMETER ESTIMATION

According to GPE criterion (33) or (25), the EFOP estimatecan be obtained. The EFOP estimates for some common modelscan be described as follows.

A. Box-Jenkins Models

If the filter , then model (9) becomes theBox–Jenkins (BJ) model [14]

(35)

In this case, and are

The signal is generated by a recursive technique

The initial values of are chosen as before. From (34) it isnot difficult to deduce the following result.


Theorem 2: Suppose that the noise is a sequenceof white noise; then the EFOP estimate of the system parameter

in model (35) is

(36)

B. ARMA Models

Case 1: If and , model (9) be-comes an output error (OE) model [14]

(37)

Then, the signal becomes

In this case and are

If is a white noise, from (37) it is easy to see that

Therefore, it is fair and reasonable to substitute the outputs ofthe system for the weighted signals areif the relative noises are smaller than the output signals. In thiscase we can deduce that the EFOP estimate of model (37) is

(38)where

Case 2: If , then model (9) is

(39)

The parameter and the regressor are still denoted by:

The EFOP estimate of this case can be obtained by using rela-tionship (34) directly. However, this calculation may be compli-cated. Now it can be simplified by the following technique.

The output signals are the measured variables,which are also the random variables that rely on the inputsignals and disturbance noise. If the inputs of the system arethe deterministic signals and the disturbances are the whitenoise, the signals are quasi-stationary signals [14]. The randomvariables could be regarded as approximately theirsingle realization values—the measured values of the system,which are given by a sequence of measured deterministic

signals. Therefore, for , we could obtainapproximately

(40)

where and are the DFT of the sequencesand , respectively. Because matrix

consists of and , and contains unknown filter, the analytical solution of parameter identification can not

be solved from relationship (26). Hence, it is more complicatedwhen the parameter estimation is obtained by function (26)directly. However, if the frequency function is estimatedby the relationship , the estimation error shouldbe . Using relationship (40), similar to theproofs of Theorem 1, the following result can be concluded.Under the above hypothesis the EFOP estimate of model (39) is

(41)where

The validity of this method is also illustrated by the followingsimulation.

Case 3: For the identification of system (9)

we can use a technique similar to the two-stage method [14] tocreate a new method. Let

The parameters , are obtained by rela-tionship (36) and (41), respectively:

(42)


where . The parameter of system (9) can be estimatedby (41). However, it seems to be complicated when a system isidentified by this method. Simple algorithms can be constructedby extending relationship (36) or (41). We have the followingsimpler methods for system (9).

(43)

(44)

where , are defined as previously, and

Remark 4: The identification technique introduced in ourdiscussion seems to require some information about the inter-ference configuration. If the noise filters and are un-known, they can be identified by the method of [14] first. Ingeneral, we can choose filters and as ,

. That also means our algorithms are valid for totalnoise, whether is a white noise or not.

VI. RECURSIVE IDENTIFICATION

From the definition we can see the EFOP estimate is an open-loop algorithm. If the sample number is large, the implemen-tation of the EFOP estimate will take much computational time.The recursive identification has an obvious advantage in car-rying out such an implementation [32]. In this section, we de-scribe a recursive identification for the EFOP estimate. It is alsoapplicable to the adaptive systems when the measurable data areupdated on-line. A black-box model (9) is also considered here.Parameter and regressor are also denoted by

The recursive algorithm for method (42) can be established bythe bootstrap method [14]. For simplicity, we now refer to therecursive algorithm proposed in [32] to create an algorithm formethod (43) or (44). Since the performance functions are similarto the form in [32], we can deduce the recursive algorithms

(45)

(46)

where

where and are defined as in (43) or (44).As pointed out previously, because the output signals or input

signals are stressed in methods (33) as well as (36), (38), and(41)–(46), the EFOP estimate is quite robust against noise in-terference. In particular, these algorithms have an advantage inidentifying the systems affected by color noise interference, al-though these methods are deduced under the hypothesis thatnoise is a white noise. The simulations confirmed that thealgorithm is valid when noise is also a color noise.

VII. SIMULATIONS

To illustrate the behavior of the EFOP estimates, several sim-ulation trials were conducted in the following simulations forcomparison with the previous algorithms. For a real system, theoutput was generated by a system with a given inputsequence . The experimental sample number was2000. Let

be the noise-to-signal ratio, which expresses the extent of dis-turbance of the model signal. Note that is the real model pa-rameter, while , and are the EFOP estimate,the LS estimate, and the extended least-squares (ELS) estimate,respectively.

Example 1: The output error model is given by

(47)

The real system parameters are , and. The input signal is

generated by a square generator and is an approx-imate white noise with variance . The output of thesystem is then generated by (47) with a noise-to-signal ratio


Fig. 1. Simulation of Example 1 with filter D(q).

. The parameter is estimated by the EFOP2 method,the LS method, and the parameter for the recursive pseudolinearregression of system (47), by the ELS estimate. The simulationresults are given in Fig. 1.

The EFOP method can be intuitively compared with the LSmethod and ELS method as shown in Fig. 1. It is easy to see thatthe EFOP method more efficiently identifies a real system (47)than the LS method or the ELS method. Furthermore, with thesample number increasing, the values of the EFOP estimateare closer the real parameters. These facts were also validatedby the following calculated values:

where denotes the average estimate from the 100thEFOP estimate value to the 2000th EFOP2 estimate. and

denote the average estimate from the 100th LS estimatevalue to the 2000th LS estimate and the average estimate fromthe 200th ELS estimate value to the 2000th ELS estimate,respectively. The calculational error is defined by the standarddeviation.

Although in this example, the noise filter was used inthe EFOP estimate, it is not an essential requirement of our newmethod. In fact, if the same input signal and outputsignal in the simulation of Example 1 are taken againand the noise filter in EFOP estimate (38) is taken as 1,the calculated values are the same as above.

These calculationa results demonstrate that the EFOPmethod is more accurate in identifying disturbed systemsthan the LS method and ELS method when the color noise

is ignored. This is also demonstrated by thesimulation shown in Fig. 2.

Fig. 2. Simulation of Example 1 ignored filter D(q).

If the experimental sample number is only 500, the sim-ulation values of the EFOP method, the LS method, and ELSmethod are

respectively. These results as well as the next example also indi-cate that the EFOP method can precisely identify systems usingfewer samples. It has an obvious advantage in system identifi-cation if the experimental data is difficult to obtain.

Example 2: This system was

(48)

The input signal was generated by a pulse generator, too. Thereal system parameters were , , and

. The interferance noise was an approxi-mately white noise. Then the output of system (48) was givenwith a noise-to-signal ratio . The parameter was es-timated by the EFOP method (42), the LS method, and the ELSmethod. The simulation results are shown in Fig. 3.

From Fig. 3, we can see that the EFOP estimation givenby (42) oscillates near the true parameter . Although the es-timation result tends to be steady, with increasing samplenumber , the LS method produces a quite significant error be-tween and . This result is also illustrated by the followingcalculated values


Fig. 3. Simulation of Example 2 with algorithm (42).

Fig. 4. Simulation of Example 2 with algorithm (43).

where denotes the average estimate from the 100th tothe 2000th estimate value of EFOP estimate (42). anddenote the average estimate from the 100th to the 2000th esti-mate value of the LS estimate and ELS estimate, respectively.

Without the two-stage method, extended EFOP methods (43)and (44) can also be used to identify model (48). For instance,the following calculational values and Fig. 4 are yielded bymethod (43)

When recursive algorithms (45) and (46) are also used toidentify model (48), we can see it takes less computational timethan the REFOP algorithm (43). The simulation results of recur-sive algorithm were shown in the following calculational valuesand Fig. 5

From the calculational values the results between algorithm(43) and algorithms (45) and (46) are approximated. Since the

Fig. 5. Simulation of Example 2 with REFOP algorithm.

later is based on recursive algorithm, its computational speedis faster than that of algorithm (43). Furthermore, from Fig. 5we can see the curve of estimate is flater. It tends tothe true parameter with the sample number N increases. Forinstance, at sample 2000, the estimate values are

The above simulation results indicate that the extended EFOPmethod can identify system (48) more efficiently than the LSmethod and ELS method.

VIII. CONCLUSION

In this paper, the identification of common linear stochasticsystems was considered. First, there was a review of some vitalbackground knowledge including the concepts of the GPE cri-terion and the empirical optimal frequency-domain estimate.The identification method was constructed with the GPE cri-terion and the empirical optimal frequency-domain estimate.The discrete Fourier transform is a useful tool in analyzing theproperties of the transfer function and was also used in thisstudy. Then, the paper discussed the identification of Black-boxmodels. From the optimal frequency-domain estimate, severalGPE criteria were deduced from the different ARMA models.The time-domain estimate and the frequency-domain estimatewere then combined to form the EFOP estimate and the recur-sive EFOP algorithm for black-box models. This method pro-vides the globally optimal frequency-domain estimate and min-imizes the GPE criterion. It has obvious advantages in anti-disturbance performance and can precisely identify an ARMAmodel with fewer samples. Lastly, several simulation exampleswere included to illustrate the method’s reliability.

APPENDIX

Let be the discrete Fourier transform of the noise se-quence


Since is white noise with variance , then

(A1)

and for

ifif

(A2)

Because , by the definition of the DFT and theinitial assumption, we have that

(A3)

Take the discrete Fourier transform for both sides of model (9).Similar to the proof of (A3) we derive that

(A4)

where and ., , and are the frequency functions

of the filters , , , and , respectively. Bythese definitions, the relationship

(A5)

follows from (A4). For , relations (13) and(14) are derived from (A1), (A2), and (A5) with no difficulty.

REFERENCES

[1] L. Ljung, “On convergence for prediction error identification methods,”Division of Automatic Control, Lund Institute of Technology, Lund,Sweden, Tech. Rep. 7405, 1974.

[2] W. X. Zheng, “On a least-squares-based algorithm for identification ofstochastic linear systems,” IEEE Trans. Signal Process., vol. 46, no. 6,pp. 1631–1638, Jun. 1998.

[3] T. McKelvey, H. Akcay, and L. Ljung, “Subspace-based identificationof infinite dimensional multivariable systems from frequency-responsedata,” Automatica, vol. 37, pp. 885–902, 1996.

[4] F. Gustafsson, S. Gunnarsson, and L. Ljung, “Shaping frequency-depen-dent time resolution when estimating spectral properties with parametricmethods,” IEEE Trans. Signal Process., vol. 45, no. 4, pp. 1025–1035,Apr. 1997.

[5] B. Ninness, “Integral constraints on the accuracy of least-squares esti-mation,” Automatica, vol. 32, pp. 391–397, 1996.

[6] B. Ninness and J. C. Gomez, “Frequency-domain analysis of trackingand noise performance of adaptive algorithms,” IEEE Trans. SignalProcess., vol. 46, no. 5, pp. 1314–1332, May 1998.

[7] A. J. Helmicki, C. A. Nett, and C. N. Nett, “Control oriented systemidentification: A worst case/deterministic approach in H ,” IEEETrans. Autom. Control, vol. 36, no. 10, pp. 1163–1176, Oct. 1991.

[8] A. J. Helmicki, C. A. Jacobson, and C. N. Nett, “Least squares methodsfor H control-oriented system identification,” IEEE Trans. Autom.Contr., vol. 38, no. 5, pp. 819–826, May 1993.

[9] L. Guvenc and J. Ackermann, “Links between the parameter space andfrequency domain methods of robust control,” Int. J. Robust NonlinearContr., vol. 11, pp. 1435–1453, 2001.

[10] H. Akcay and P. Heuberger, “A frequency-domain iterative identificationalgorithm using general orthonormal basis function,” Automatica, vol.37, pp. 663–674, 2001.

[11] H. Akcay, “A stochastic analysis of robust estimation algorithms inHwith rational basis functions,” Int. J. Robust Nonlinear Contr., vol. 12,pp. 71–86, 2002.

[12] T. Mckelvey, H. Akcay, and L. Ljung, “Subspace-based multivariablesystem identification from frequency response data,” IEEE Trans.Autom. Contr., vol. 41, no. 7, pp. 960–979, Jul. 1996.

[13] W. X. Zheng, “On a least-squares-based algorithm for identification ofstochastic linear systems,” IEEE Trans. Signal Process., vol. 46, no. 6,pp. 1631–1638, Jun. 1998.

[14] L. Ljung, System Identification—Theory for the User. Upper SaddleRiver, NJ: Prentice-Hall, 1999.

[15] H. Akcay, H. Hjalmarsson, and L. Ljung, “On the choice of norms insystem identification,” IEEE Trans. Autom. Contr., vol. 41, no. 9, pp.1367–1372, Sep. 1996.

[16] E. Weyer, “Finite sample properties of system identification of ARXmodels under mixing conditions,” Automatica, vol. 36, pp. 1291–1299,2000.

[17] B. Ninness, “Integral constraints on the accuracy of Least-squares esti-mation,” Automatica, vol. 32, pp. 391–397, 1996.

[18] U. Forssell and L. Ljung, “Identification of unstable systems usingoutput error and Box-Jenkins model structures,” IEEE Trans. Autom.Contr., vol. 45, no. 1, pp. 137–141, Jan. 2000.

[19] R. J. Martin, “A metric for ARMA processes,” IEEE Trans. SignalProcess., vol. 48, no. 4, pp. 1164–1170, Apr. 2000.

[20] T. Söderström and P. Stoica, System Identification. Hemel, U.K.: Pren-tice-Hall International, 1989.

[21] J. Hu, K. Hirasawa, and K. Kumamaru, “A homotopy approach to im-proving PEM identification of ARMAX models,” Automatica, vol. 37,pp. 1323–1334, 2001.

[22] M. Lang and T. I. Laakso, “Simple and robust method for the designof allpass filters using least-squares phase error criterion,” IEEE Trans.Circuits Syst. II, Analog Digit. Signal Process., vol. 41, no. 1, pp. 40–48,Jan. 1994.

[23] R. Kumaresan and A. Rao, “On designing stable allpass filters using ARmodeling,” IEEE Trans. Signal Process., vol. 47, no. 1, pp. 229–231,Jan. 1999.

[24] K. M. Lo and W. H. Kwon, “A new identification approach for FIRmodels,” IEEE Trans. Circuits Syst. II, Analog Digit. Signal Process.,vol. 49, no. 4, pp. 439–446, Apr. 2002.

[25] , “New identification approaches for disturbed models,” Auto-matica, vol. 39, pp. 1627–1634, 2003.

[26] H. Akcay and B. Ninness, “Rational basis functions for robust identi-fication from frequency and time-domain measurements,” Automatica,vol. 34, pp. 1101–1117, 1998.

[27] S. Michalek, M. Wagner, and J. Timmer, “A new approximate likeli-hood estimator for ARMA-filtered hidden Markov models,” IEEE Trans.Signal Process., vol. 48, pp. 1537–1547, 2000.


[28] P. M. T. Broersen, “Autoregressive model orders for Durbin’s MA andARMA estimators,” IEEE Trans. Signal Process., vol. 48, no. 6, pp.2454–2457, Jun. 2000.

[29] , “Finite sample criteria for autoregressive order selection,” IEEETrans. Signal Process., vol. 48, pp. 3550–3558, 2000.

[30] S. Lambert-Lacroix, “On periodic autoregressive processes estimation,”IEEE Trans. Signal Process., vol. 48, no. 12, pp. 1800–1803, Dec. 2000.

[31] P. M. Malila and J. Jarvinen, “On model and filter sensitivity,” Auto-matica, vol. 36, pp. 1905–1908, 2000.

[32] K. M. Lo and H. Kimura, “Recursive estimation methods for discretesystems,” IEEE Trans. Autom. Contr., vol. 48, no. 11, pp. 2019–2024,Nov. 2003.

Kueiming Lo (Guiming Luo) (M’03) receivedthe M.Sc. degree from the University of Scienceand Technology of China, Hefei, China, and thePh.D. degree from the Institute of Systems Science,Chinese Academy of Sciences, Beijing, China, in1988 and 1992, respectively.

He then joined in the Department of MathematicalSciences, Tsinghua University, Beijing, China. FromSeptember 2000 to September 2001, he held a vis-iting Professor position at Seoul National University,Seoul, Korea. From October 2001 to March 2003, he

was with the University of Tokyo, Tokyo, Japan as a PSJS Fellow. Since 2003, hehas been with the School of Software, Tsinghua University. His research inter-ests include system identification, adaptive control, nonlinear control, softwarereliability methods, and model checking.

Hidenori Kimura (F’91) received the graduateand the Doctor of Engineering degrees from theUniversity of Tokyo, Tokyo, Japan, in 1965 and1970, respectively.

He joined the Faculty of Engineering Science,Osaka University, Osaka, Japan, where he wasengaged in research and education of control andsystems theory for 17 years. He was with the War-wick University and Imperial College of Scienceand Technology with the support by British Councilduring 1974–1975. In 1988, he moved to the Faculty

of Engineering, Osaka University, where he was engaged in research on robustcontrol, modeling and signal processing. In 1995, he moved to the University ofTokyo. In 2001, he was also appointed to be a Team Leader of the Bio-mimeticControl Center, The Institute of Physical and Chemical Research (RIKEN),Nagoya, Japan, which is his current position of research. His current researchinterests are biological control systems in various levels of organs includingbrain and cell. He was a Springer Professor of the University of California,Berkeley, in 1996.

Dr. Kimura was the recipient of various awards including SICE paper awardsfrom SICE (five times), George Axelby Award from IEEE Control Systems So-ciety, Paper Prize Award from IFAC (twice), Distinguished Member Award fromIEEE CSS and so on. He is a member of the Science Council of Japan.

Wook-Hyun Kwon (F’99) received B.S. and M.S.degrees in electrical engineering from Seoul NationalUniversity, Seoul, Korea, in 1966 and 1972, respec-tively, and the Ph.D. degree from Brown University,Providence, RI, in 1975.

He was a Research Associate at Brown Universityfrom 1975 to 1976) and an Adjunct Assistant Pro-fessor at the University of Iowa, Ames, from 1976 to1977. He has been with the Seoul National Univer-sity since 1977, where he is currently a Professor inthe School of Electrical Engineering and Computer

Science. He was a Visiting Assistant Professor at Stanford University, Stanford,CA from 1981 to 1982. He has authored about 100 international journal papersand approximately 200 international conference papers, mostly in the areas ofpredictive controls, time-delayed system, finite-impulse response filtering, andreal-time computer applications for automation. He authored a graduate text-book Receding Horizon Control: Model Predictive Control for State Models(Springer, 2005).

Dr. Kwon received the National Academy of Sciences Award in 1997. He be-came a member of the National Academy of Engineering in 1995 and a Fellowof the Korean Academy of Science and Technology (KAST) in 1998. He becamea Fellow of Third World Academy of Science (TWAS) in 2000. He received theBrown University Engineering Alumni Medal (BEAM) award for outstandingachievements in 2003. Since 1991, he has been the Founding Director of theEngineering Research Center for Advanced Control and Instrumentation es-tablished at Seoul National University by the Korean Science and EngineeringFoundation (KOSEF). He was President of the Institute of Control, Automationand Systems Engineers (ICASE) in 1999 and also President of the Korean Insti-tute of Electrical Engineers (KIEE) in 2001. He is now serving the Internationalthe Federation of Automatic Control (IFAC) as President since July 2005, theNational Academy of Engineering of Korea (NAEK) as Vice-President since2002, and the Seoul National University Senate as Chairman since 2005.

Xiaojing Yang received the Ph.D. degree inmathematics from the University of Wuerzburg,Wuerzburg, Germany in 1989.

He was a Lecturer and then an Associate Professorat the Tsinghua University, Beijing, China,wherehe is now a Professor in the Department of Math-ematical Sciences. His research interests includedynamical systems, nonlinear control theory, qual-itative theory of ordinary differential equations,complex analysis, and inequalities. He has authoredmore than 80 papers in the field of mathematics.

empirical frequency-domain optimal parameter estimate for black-box processes

Documents