1 pattern recognition: statistical and neural lonnie c. ludeman lecture 13 oct 14, 2005 nanjing...
TRANSCRIPT
![Page 1: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/1.jpg)
1
Pattern Recognition:Statistical and Neural
Lonnie C. Ludeman
Lecture 13
Oct 14, 2005
Nanjing University of Science & Technology
![Page 2: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/2.jpg)
2
Lecture 13 Topics
1. Multiple observation Multiple class example: (review) Sufficient statistic space and Likelihood ratio space
2. Calculation of P(error) for 2-class case : several special cases
3. P(error) calculations examples for special cases – 2-class case
![Page 3: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/3.jpg)
3
Example 1: Multiple observation - multiple classes
Given: the pattern vector x is composed of N independent observations of a Gaussian random variable X with the class conditional densities as follows for each component
A zero one cost function is given as
Find:(a) the Bayes decision rule in a sufficient statistic space. (b) the Bayes decision rule in a space of likelihood ratios
![Page 4: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/4.jpg)
4
Solution:
(a) Since the observations are independent the joint conditional density is a product of the marginal densities and given by
for i = 1, 2, 3 and mi = i, i=1, 2, 3
Bayes decision rule is determined form a set of yi(x)
defined for M=3 by
![Page 5: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/5.jpg)
5
Substituting the given properties gives
The region to decide C1 is found by setting the following
inequalities
Therefore the region R1 to decide C
1, reduces to the x that satisfy
![Page 6: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/6.jpg)
6
Similarly the regions R2 and R
3 become
Substituting the conditional densities, taking the ln of both sides and simplifying the decision rule reduces to regions in a sufficient statistic s space as follows
![Page 7: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/7.jpg)
7
Which is shown below in the sufficient statistic s space
An intuitively pleasing result !
s
![Page 8: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/8.jpg)
8
where yi(x) = Cij p(x | Cj) P(Cj)j=1
M
if yi(x) < yj(x) for all j = i
Then decide x is from Ci
(b) Bayes Decision Rule in Likelihood ratio space: M-Class Case derivation
We know that Bayes Decision Rule for the M-Class Case is
![Page 9: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/9.jpg)
9
LM
(x) = p(x | CM
) / p(x | CM
) = 1
Dividing through by p(x | CM) gives sufficient
statistics vi(x) as follows
Therefore the decision rule becomes
![Page 10: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/10.jpg)
10
Bayes Decision Rule in the Likelihood Ratio Space
The dimension of the Likelihood Ratio Space is always one less than the number of classes ( M - 1)
![Page 11: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/11.jpg)
11
Back to Example: Define the likelihood ratios as
Dividing both sides of the inequalities by p(x|C3)
gives the following equations in the Likelihood Ratio space for determining C
1
We have already determined the region to decide C1 as
![Page 12: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/12.jpg)
12
The other regions are determined in the same fashion giving the decision regions in the likelihood ratio space
![Page 13: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/13.jpg)
13
Calculation of Probability of error for the 2-class Gaussian Cases
We know Optimum Bayes Decision Rule is given by
Special Case 1:
![Page 14: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/14.jpg)
14
The sufficient statistic Z conditioned on C1 has the
following mean and variance
![Page 15: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/15.jpg)
15
thus under C1 we have :
a1 =
v1 =
Z ~ N( a1, v
1 )
The conditional variance becomes
![Page 16: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/16.jpg)
16
Similarly the conditional mean and variance under class C2 are
The statistic Z under class C2 is Gaussian and given by
thus under C1 we have :
a2 =
v2 =
Z ~ N( a2, v
2 )
![Page 17: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/17.jpg)
17
Determination of the P(error)
The total Probability Theorem states
where
![Page 18: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/18.jpg)
18
Since the scalar Z is Gaussian the error conditioned on C
1 becomes:
![Page 19: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/19.jpg)
19
Similarly the error conditioned on C2 becomes
Finally the total P(error) becomes for Special Case 1
![Page 20: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/20.jpg)
20
Special case 2: Equal scaled identity Covariance matrices
Using the previous formula the P(error) reduces to
where
(Euclidean distance between the means)
![Page 21: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/21.jpg)
21
Special case 3: Zero- one Bayes Costs and Equal apriori probabilities
Using the previous formula for P(error) gives:
![Page 22: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/22.jpg)
22
Special Case 4:
Then
![Page 23: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/23.jpg)
23
Example: Calculation of probability of Error
Given:
Find: P(error) for the following assumptions
![Page 24: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/24.jpg)
24
(a)
Solution:
![Page 25: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/25.jpg)
25
(b)
Solution:
Substituting the above into the P(error) gives:
![Page 26: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/26.jpg)
26
(c)
Solution:
Substituting the above into the P(error) gives:
![Page 27: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/27.jpg)
27
(d)
Solution:
Substituting the above into the P(error) for the case of equal covariance matrices gives:
![Page 28: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/28.jpg)
28
(d) Solution Continued:
![Page 29: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/29.jpg)
29
Lecture 13 Summary
1. Multiple observation Multiple class example: (review) Sufficient statistic space and Likelihood ratio space
2. Calculation of P(error) for two class case : special cases
3. P(error) calculations examples for special cases - 2 class case
![Page 30: 1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 13 Oct 14, 2005 Nanjing University of Science & Technology](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e525503460f94b481cb/html5/thumbnails/30.jpg)
30
End of Lecture 13