![Page 1: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/1.jpg)
1
Model Validation
![Page 2: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/2.jpg)
2
Introduction
• Model validation is a crucial step to evaluate the quality of estimated models.
• If validation is unsuccessful, previous steps in the system identification workflow must be redone – e.g., the model structure can be changed.
![Page 3: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/3.jpg)
3
Motivation
• So far, we validated and selected models by examining plots of predicted vs. measured outputs.
• In this section we will learn about additional statistically-based validation tests, namely the correlation tests of the residual error.
![Page 4: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/4.jpg)
4
(1) Whiteness of the residual error
• The residual error is the difference between the measured output and the output estimated by the model:
• The residual error thus represents the information in the measured output which can't be explained by the model.
• For a good model, the residuals must be a set of random
numbers caused by experimental errors. Any trend such as a gradual rise or a slight curve in the plot of the residuals invalidates the results. Therefore, the prediction errors ε(k) should be zero-mean white noise.
)(ˆ)()( kykyk
![Page 5: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/5.jpg)
5
Autocorrelation function of ε(k)
• The autocorrelation function, rε(τ) = E{ ε(k+ τ) ε(k) }, can be estimated from data as:
• If the residual ε(k) is purely random (zero-mean white noise), rε(τ) = 0 for τ≠0, while rε(0) is the variance of the white noise.
• Normalization by the (estimated) variance gives:
N
k
kkN
r1
)()(1
)(ˆ
)0(ˆ)(ˆ
)(
r
rx
![Page 6: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/6.jpg)
6
Testing the Whiteness of ε(k)
• The condition x(τ) = 0 for τ≠0 almost never holds exactly for data taken from an actual experiment. Instead, the auto-correlation function usually exhibits small values for τ≠0.
• However, it can be shown that for large number of data points, N, x(τ) is distributed according to a Gaussian distribution N(0,1/N).
• Therefore, if
then we can assume, with 99% confidence, that x(τ) or rε(τ)=0.
,3
|)(|N
x
![Page 7: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/7.jpg)
7
(2) Independence of the residuals and past inputs
• If the model is accurate, it should entirely explains the influence of inputs u(k) on future outputs y(k+τ).
• Therefore, residual errors ε(k+τ)
are only influenced by the disturbances, and are independent of past inputs u(k).
)(ˆ)()( kykyk
![Page 8: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/8.jpg)
8
Cross-correlation function of ε(k+τ) & u(k)
• If ε(k+τ) are uncorrelated with past input u(k), the cross-correlation function, should be zero for τ > 0.
• Estimation from data and normalization:
)}()({E)( kukr u
)0(ˆ)0(ˆ
)(ˆ)(
0),()(1
)(ˆ1
u
u
N
ku
rr
rx
kukN
r
![Page 9: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/9.jpg)
9
Independence test
Like before, for large N, x(τ) is distributed according to Gaussian N(0,1/N), and therefore if:
then we can assume, with 99% confidence, that x(τ) or rεu(τ) = 0.
,3
|)(|N
x
![Page 10: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/10.jpg)
10
Example
Assume that a real system to be identified is in the output-error (OE) form with order 3:
The identification and validation data are plotted as shown:plot(id); plot(val);
)()()(
)()(
1
1
kekuqF
qBky
![Page 11: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/11.jpg)
11
Example, continued (ARX model)
First, we try an ARX model:mARX = arx(id, [3, 3, 1]); resid(mARX, id);
The model fails the whiteness test. This is because the system is not within the model class, leading to an inaccurate model. Therefore, the model is rejected.
![Page 12: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/12.jpg)
12
Example, continued (ARX model)
Simulating the ARX model on the validation data confirms the fact that the model is poor.
![Page 13: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/13.jpg)
13
Example, continued (OE model)
Now, let us try an OE model:
mOE = oe(id, [3, 3, 1]); resid(mOE, id);
As expected, the OE model passes the two tests because the OE model class contains the real system. The model is therefore validated.
![Page 14: Model Validation 1. Introduction Model validation is a crucial step to evaluate the quality of estimated models. If validation is unsuccessful, previous](https://reader035.vdocuments.site/reader035/viewer/2022062305/5697bf701a28abf838c7da41/html5/thumbnails/14.jpg)
14
Example, continued (OE model)
Simulating the OE model on the validation data confirms the good quality of the model.