support vector machines (part 1)
DESCRIPTION
Support Vector Machines (part 1). Plan of the lecture. Problem of classification SVM for solving linear problems training classification Application of convolution kernels. Bi bli ography. Corrina Cortez, Vladimir Vapnik Support-Vector Networks. Classification problem. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/1.jpg)
Face Recognition & Biometric Systsems
Support Vector Machines (part 1)
![Page 2: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/2.jpg)
Face Recognition & Biometric Systsems
Plan of the lecture
Problem of classificationSVM for solving linear problems training classification
Application of convolution kernels
![Page 3: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/3.jpg)
Face Recognition & Biometric Systsems
Bibliography
Corrina Cortez, Vladimir VapnikSupport-Vector Networks
![Page 4: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/4.jpg)
Face Recognition & Biometric Systsems
Classification problem
Aim: classification of an element to one of defined classesTwo stages: training classification of samples
Available solutions: Artificial Neural Networks Support Vector Machines other classifiers
![Page 5: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/5.jpg)
Face Recognition & Biometric Systsems
Classification problem
Training set - requirements: classified representative
Training process: aims at finding general rules a risk of overfitting to the training
set (especially when it is not representative)
![Page 6: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/6.jpg)
Face Recognition & Biometric Systsems
Classification problem
Classification of samples: must be preceded by the training stage applies rules derived from the training
Number of classes: SVM solves two-class problems it is possible to solve multi-class
problems basing on two-class problems
![Page 7: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/7.jpg)
Face Recognition & Biometric Systsems
Classification problem
Linearly separable
![Page 8: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/8.jpg)
Face Recognition & Biometric Systsems
Classification problem
Non-linearly separable
![Page 9: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/9.jpg)
Face Recognition & Biometric Systsems
Classification problem
Training with error (soft margin)
![Page 10: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/10.jpg)
Face Recognition & Biometric Systsems
Classification problem
Margin maximisation
![Page 11: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/11.jpg)
Face Recognition & Biometric Systsems
Linear separability
Data set: (y1,x1),...,(yl,xl), yi{-1,1}
Vector w, scalar value b:w • xi + b 1 for yi = 1
w • xi + b -1 for yi = -1
henceyi (w • xi + b) 1
The condition must be fulfilled for the whole data set
![Page 12: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/12.jpg)
Face Recognition & Biometric Systsems
SVM – training
SVM solves linear separable two-class problems other cases transformed to the
basic problem
Optimal hyperplane margin between samples of two
classes margin maximisation
![Page 13: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/13.jpg)
Face Recognition & Biometric Systsems
SVM – training
Optimal hyperplane:w0 • x + b0 = 0
2D example – hyperplane is a line
Margin width (without b):
||
max||
min}1:{}1:{ w
wx
w
wxw
yxyx,bρ
![Page 14: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/14.jpg)
Face Recognition & Biometric Systsems
SVM – training
Optimal width:
Maximisation of , minimisation of w0 • w0
Limitation: yi (w • xi + b) 1
00000
2
||
2
wwww
),bρ(
![Page 15: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/15.jpg)
Face Recognition & Biometric Systsems
SVM – training
Margin:Optimal hyperplane:
yi – class identifier i – Lagrange multipliers
A problem: how to find i?
1)( by ii xw
l
iiiiy
1
00 xw
![Page 16: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/16.jpg)
Face Recognition & Biometric Systsems
SVM – training
Function maximisation:
1 – unitary vector (l – dimensional)D – l x l matrix:
DΛΛ1ΛΛ TTW2
1)(
),...,( 1 lT Λ
jijiij yyD xx
![Page 17: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/17.jpg)
Face Recognition & Biometric Systsems
SVM – training
Optimisation limits:
Optimisation based on the gradient method
0Λ0YΛT
),...,( 1 lT yyY
![Page 18: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/18.jpg)
Face Recognition & Biometric Systsems
SVM – training
Lagrange multipliers : non-zero values for support vectors equal zero for other vectors
(majority)
Training set after the training: support vectors (a small subset of
the training set) coefficients for every vector
![Page 19: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/19.jpg)
Face Recognition & Biometric Systsems
SVM – classification
Calculate y for a vector which is to be classified:
xr, xs – support vectors from both classes
Classification decision
byfl
iiii
1)( xxx
l
isiriii yb
1)(
2
1xxxx
![Page 20: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/20.jpg)
Face Recognition & Biometric Systsems
SVM – limitations
SVM conditions: solves two-class problem linear separability of data
A XOR problem:
![Page 21: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/21.jpg)
Face Recognition & Biometric Systsems
SVM – limitations
Possibilities of enhancement: SVM for non-linear data – too
complicated calculations transformation of the data, so that
they are linearly separable
Mapping into higher dimension example of XOR in 2D mapped into
3D
![Page 22: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/22.jpg)
Face Recognition & Biometric Systsems
Convolution kernelsFunction:Mapping into higher dimension: x (x)Calculations use scalar product of vectors, not the vectors themselvesKernels of convolution may be used instead of scalar products
No need to find function
Nn RR :
)()(),( vuvu K
![Page 23: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/23.jpg)
Face Recognition & Biometric Systsems
Convolution kernels
),( jijiij KyyD xx
Training with convolution kernels
![Page 24: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/24.jpg)
Face Recognition & Biometric Systsems
Convolution kernels
bKyfl
iiii
1),()( xxx
l
isiriii KKyb
1)],(),([
2
1xxxx
xr, xs – support vectors from both classes
Classification with convolution kernels
![Page 25: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/25.jpg)
Face Recognition & Biometric Systsems
Convolution kernels
Linear
Polynomial
RBF (radial basis functions)
2
2||
),( vu
vu
eK
vuvu ),(K
dK )1(),( vuvu
![Page 26: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/26.jpg)
Face Recognition & Biometric Systsems
Summary
ClassifiersBasic problem: two-class linear separable data set solved by the SVM
Enhancement convolution kernels – SVM for non-
linear separable data
![Page 27: Support Vector Machines (part 1)](https://reader035.vdocuments.site/reader035/viewer/2022062800/56814261550346895dae8a98/html5/thumbnails/27.jpg)
Face Recognition & Biometric Systsems
Thank you for your attention!
Next week
Support Vector Machines – continued... multi-class cases soft margin training applications to face recognition