![Page 1: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/1.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Support Vector Machines For
Quaternary Classification
P. Bouboulis, E. Theodoridou, S. Theodoridis
Department of Informatics and TelecommunicationsUniversity of Athens
Athens, Greece
23-09-2013
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 1 / 47
![Page 2: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/2.jpg)
Introduction
Support Vector Machines
The Complex Case
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 2 / 47
![Page 3: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/3.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 3 / 47
![Page 4: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/4.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Reproducing Kernel Hilbert Spaces.
Consider a linear class H of real (complex) valued functions f
defined on a set X (in particular H is a Hilbert space), for which
there exists a function (kernel) κ : X × X → R(C) with the
following two properties:
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 4 / 47
![Page 5: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/5.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Reproducing Kernel Hilbert Spaces.
Consider a linear class H of real (complex) valued functions f
defined on a set X (in particular H is a Hilbert space), for which
there exists a function (kernel) κ : X × X → R(C) with the
following two properties:
1 For every x ∈ X , κ(x , ·) belongs to H.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 4 / 47
![Page 6: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/6.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Reproducing Kernel Hilbert Spaces.
Consider a linear class H of real (complex) valued functions f
defined on a set X (in particular H is a Hilbert space), for which
there exists a function (kernel) κ : X × X → R(C) with the
following two properties:
1 For every x ∈ X , κ(x , ·) belongs to H.
2 κ has the so called reproducing property, i.e.,
f (x) = 〈f , κ(x , ·)〉H, for all f ∈ H, x ∈ X . (1)
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 4 / 47
![Page 7: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/7.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Reproducing Kernel Hilbert Spaces.
Consider a linear class H of real (complex) valued functions f
defined on a set X (in particular H is a Hilbert space), for which
there exists a function (kernel) κ : X × X → R(C) with the
following two properties:
1 For every x ∈ X , κ(x , ·) belongs to H.
2 κ has the so called reproducing property, i.e.,
f (x) = 〈f , κ(x , ·)〉H, for all f ∈ H, x ∈ X . (1)
Then H is called a Reproducing Kernel Hilbert Space (RKHS)
associated to the the kernel κ.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 4 / 47
![Page 8: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/8.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Kernel Trick
The notion of RKHS is a popular tool for treating non-linear
learning tasks.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 5 / 47
![Page 9: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/9.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Kernel Trick
The notion of RKHS is a popular tool for treating non-linear
learning tasks.
Usually this is attained by the so called “kernel trick”.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 5 / 47
![Page 10: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/10.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Kernel Trick
The notion of RKHS is a popular tool for treating non-linear
learning tasks.
Usually this is attained by the so called “kernel trick”.
If
X ∋ x → Φ(x) := κ(x , ·) ∈ HX ∋ y → Φ(y) := κ(y , ·) ∈ H,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 5 / 47
![Page 11: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/11.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Kernel Trick
The notion of RKHS is a popular tool for treating non-linear
learning tasks.
Usually this is attained by the so called “kernel trick”.
If
X ∋ x → Φ(x) := κ(x , ·) ∈ HX ∋ y → Φ(y) := κ(y , ·) ∈ H,
then the inner product in H is given as a function computed on
X :
κ(x , y) = 〈κ(x , ·), κ(y , ·)〉H kernel trick
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 5 / 47
![Page 12: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/12.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Developing Learning Algorithms in RKHS
The black box approach.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 6 / 47
![Page 13: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/13.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Developing Learning Algorithms in RKHS
The black box approach.
Develop the learning Algorithm in X .
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 6 / 47
![Page 14: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/14.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Developing Learning Algorithms in RKHS
The black box approach.
Develop the learning Algorithm in X .
Express it, if possible, in inner products.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 6 / 47
![Page 15: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/15.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Developing Learning Algorithms in RKHS
The black box approach.
Develop the learning Algorithm in X .
Express it, if possible, in inner products.
Choose a kernel function κ.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 6 / 47
![Page 16: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/16.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Developing Learning Algorithms in RKHS
The black box approach.
Develop the learning Algorithm in X .
Express it, if possible, in inner products.
Choose a kernel function κ.Replace inner products with kernel evaluations according to
the kernel trick.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 6 / 47
![Page 17: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/17.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Developing Learning Algorithms in RKHS
The black box approach.
Develop the learning Algorithm in X .
Express it, if possible, in inner products.
Choose a kernel function κ.Replace inner products with kernel evaluations according to
the kernel trick.
Work directly in the RKHS, assuming that the data have
been mapped and live in the RKHS H, i.e.,
X ∋ x → Φ(x) := κ(x , ·) ∈ H.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 6 / 47
![Page 18: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/18.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Advantages
Advantages of kernel-based learning tasks:
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 7 / 47
![Page 19: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/19.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Advantages
Advantages of kernel-based learning tasks:
The original nonlinear task is transformed into a linear one.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 7 / 47
![Page 20: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/20.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Advantages
Advantages of kernel-based learning tasks:
The original nonlinear task is transformed into a linear one.
Different types of nonlinearities can be treated in a unified
way.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 7 / 47
![Page 21: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/21.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 8 / 47
![Page 22: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/22.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Although the theory of RKHS holds for complex spaces
too, most of the kernel-based learning techniques were
designed to process real data only.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 9 / 47
![Page 23: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/23.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Although the theory of RKHS holds for complex spaces
too, most of the kernel-based learning techniques were
designed to process real data only.
Moreover, in the related literature the complex kernel
functions have been ignored.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 9 / 47
![Page 24: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/24.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Although the theory of RKHS holds for complex spaces
too, most of the kernel-based learning techniques were
designed to process real data only.
Moreover, in the related literature the complex kernel
functions have been ignored.
Recently, however, a unified kernel-based framework,
which is able to treat complex data, has been presented.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 9 / 47
![Page 25: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/25.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Although the theory of RKHS holds for complex spaces
too, most of the kernel-based learning techniques were
designed to process real data only.
Moreover, in the related literature the complex kernel
functions have been ignored.
Recently, however, a unified kernel-based framework,
which is able to treat complex data, has been presented.
This machinery transforms the input data into a complex
RKHS, i.e.,
Φ(z) = κC(·, z).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 9 / 47
![Page 26: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/26.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Although the theory of RKHS holds for complex spaces
too, most of the kernel-based learning techniques were
designed to process real data only.
Moreover, in the related literature the complex kernel
functions have been ignored.
Recently, however, a unified kernel-based framework,
which is able to treat complex data, has been presented.
This machinery transforms the input data into a complex
RKHS, i.e.,
Φ(z) = κC(·, z).
and employs the Wirtinger’s Calculus to derive the
respective gradients.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 9 / 47
![Page 27: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/27.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Definitions:
H denotes a complex RKHS.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 10 / 47
![Page 28: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/28.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Definitions:
H denotes a complex RKHS.
H denotes a real RKHS.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 10 / 47
![Page 29: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/29.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Definitions:
H denotes a complex RKHS.
H denotes a real RKHS.
The complex RKHS can be expressed as H = H + iH.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 10 / 47
![Page 30: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/30.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex RKHS
Definitions:
H denotes a complex RKHS.
H denotes a real RKHS.
The complex RKHS can be expressed as H = H + iH.
H is isomorphic to the doubled real space H2.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 10 / 47
![Page 31: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/31.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex Kernels
The complex Gaussian kernel:
κ(z ,w) = exp
(
−∑d
i=1(zi−w∗i)2
σ2
)
,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 11 / 47
![Page 32: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/32.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex Kernels
The complex Gaussian kernel:
κ(z ,w) = exp
(
−∑d
i=1(zi−w∗i)2
σ2
)
,
The Szego kernel: κ(z ,w) = 11−wHz
,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 11 / 47
![Page 33: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/33.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Complex Kernels
The complex Gaussian kernel:
κ(z ,w) = exp
(
−∑d
i=1(zi−w∗i)2
σ2
)
,
The Szego kernel: κ(z ,w) = 11−wHz
,
Bergman kernel: κ(z,w) = 1(1−wHz)2 .
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 11 / 47
![Page 34: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/34.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger Calculus
Complex differentiability is a very strict notion.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 12 / 47
![Page 35: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/35.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger Calculus
Complex differentiability is a very strict notion.
In learning tasks that involve complex data, we often
encounter functions (e.g., the cost functions, which are
defined in R) that ARE NOT complex differentiable.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 12 / 47
![Page 36: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/36.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger Calculus
Complex differentiability is a very strict notion.
In learning tasks that involve complex data, we often
encounter functions (e.g., the cost functions, which are
defined in R) that ARE NOT complex differentiable.
Example: f (z) = |z|2 = zz∗.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 12 / 47
![Page 37: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/37.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger Calculus
Complex differentiability is a very strict notion.
In learning tasks that involve complex data, we often
encounter functions (e.g., the cost functions, which are
defined in R) that ARE NOT complex differentiable.
Example: f (z) = |z|2 = zz∗.
In these cases one has to express the cost function in
terms of its real part fr and its imaginary part fi , and use
real derivation with respect to fr , fi .
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 12 / 47
![Page 38: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/38.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger’s Calculus
This approach leads usually to cumbersome and tedious
calculations.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 13 / 47
![Page 39: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/39.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger’s Calculus
This approach leads usually to cumbersome and tedious
calculations.
Wirtinger’s Calculus provides an alternative equivalent
formulation.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 13 / 47
![Page 40: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/40.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger’s Calculus
This approach leads usually to cumbersome and tedious
calculations.
Wirtinger’s Calculus provides an alternative equivalent
formulation.
It is based on simple rules and principles.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 13 / 47
![Page 41: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/41.jpg)
Introduction
Support Vector Machines
The Complex Case
Reproducing Kernel Hilbert Spaces
Complex RKHS
Wirtinger’s Calculus
This approach leads usually to cumbersome and tedious
calculations.
Wirtinger’s Calculus provides an alternative equivalent
formulation.
It is based on simple rules and principles.
These rules bear a great resemblance to the rules of the
standard complex derivative.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 13 / 47
![Page 42: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/42.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 14 / 47
![Page 43: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/43.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The primal problem
Suppose we are given training data, which belong to two
separate classes C+,C−,i.e.,
{(xn,dn); n = 1, . . . ,N} ⊂ X × {±1}, where if dn = +1, then
the n-th sample belongs to C+, while if dn = −1, then the n-th
sample belongs to C−.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 15 / 47
![Page 44: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/44.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The primal problem
Suppose we are given training data, which belong to two
separate classes C+,C−,i.e.,
{(xn,dn); n = 1, . . . ,N} ⊂ X × {±1}, where if dn = +1, then
the n-th sample belongs to C+, while if dn = −1, then the n-th
sample belongs to C−.
For example:
Figure: Training points belonging to two classes.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 15 / 47
![Page 45: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/45.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The primal problem
The goal of the SVM task is to estimate the maximum margin
hyperplane (wT x + c = 0), that separates the points of the two
classes as best as possible
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 16 / 47
![Page 46: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/46.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The primal problem
The goal of the SVM task is to estimate the maximum margin
hyperplane (wT x + c = 0), that separates the points of the two
classes as best as possible
minimizew∈X ,c∈R
12‖w‖2
H+ C
N
N∑
n=1
ξn
subject to
{
dn
(
wT xn + c)
≥ 1 − ξn
ξn ≥ 0
for n = 1, . . . ,N,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 16 / 47
![Page 47: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/47.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The primal problem
The goal of the SVM task is to estimate the maximum margin
hyperplane (wT x + c = 0), that separates the points of the two
classes as best as possible
minimizew∈X ,c∈R
12‖w‖2
H+ C
N
N∑
n=1
ξn
subject to
{
dn
(
wT xn + c)
≥ 1 − ξn
ξn ≥ 0
for n = 1, . . . ,N,
Note that C is chosen a priori.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 16 / 47
![Page 48: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/48.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Physical justification
minimizew∈X,c∈R
12‖w‖2
H+ C
N
N∑
n=1
ξn
subject to
{
dn
(
wT xn + c)
≥ 1 − ξn
ξn ≥ 0for n = 1, . . . , N,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 17 / 47
![Page 49: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/49.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Physical justification
minimizew∈X,c∈R
12‖w‖2
H+ C
N
N∑
n=1
ξn
subject to
{
dn
(
wT xn + c)
≥ 1 − ξn
ξn ≥ 0for n = 1, . . . , N,
Figure: Linear SVM
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 17 / 47
![Page 50: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/50.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The dual problem
To solve this task, usually we consider the dual problem derived
by the Lagrangian:
maximizea∈RN
N∑
n=1
an −1
2
N∑
n,m=1
anamdndmxTmxn
subject to
N∑
n=1
andn = 0 and an ∈ [0,C/N].
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 18 / 47
![Page 51: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/51.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 19 / 47
![Page 52: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/52.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The kernel trick
Choose a positive definite kernel κR.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 20 / 47
![Page 53: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/53.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The kernel trick
Choose a positive definite kernel κR.
In the dual problem, replace the inner products xTn xm with
the respective kernel evaluations, i.e., κR(xn,xm).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 20 / 47
![Page 54: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/54.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The kernel trick
Choose a positive definite kernel κR.
In the dual problem, replace the inner products xTn xm with
the respective kernel evaluations, i.e., κR(xn,xm).
The application of the kernel trick leads to the nonlinear
SVM:
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 20 / 47
![Page 55: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/55.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
The kernel trick
Choose a positive definite kernel κR.
In the dual problem, replace the inner products xTn xm with
the respective kernel evaluations, i.e., κR(xn,xm).
The application of the kernel trick leads to the nonlinear
SVM:
maximizea∈RN
N∑
n=1
an −1
2
N∑
n,m=1
anamdndmκR(xm,xn)
subject to
N∑
n=1
andn = 0 and an ∈ [0,C/N].
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 20 / 47
![Page 56: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/56.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Mapping to the feature space
The application of the kernel trick to the dual problem is
equivalent to the following procedure:
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 21 / 47
![Page 57: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/57.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Mapping to the feature space
The application of the kernel trick to the dual problem is
equivalent to the following procedure:
Choose a positive definite kernel κR, that is associated to a
specific RKHS H.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 21 / 47
![Page 58: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/58.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Mapping to the feature space
The application of the kernel trick to the dual problem is
equivalent to the following procedure:
Choose a positive definite kernel κR, that is associated to a
specific RKHS H.
Map the points xn to Φ(xn) ∈ H, n = 1, . . . ,N.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 21 / 47
![Page 59: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/59.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
Mapping to the feature space
The application of the kernel trick to the dual problem is
equivalent to the following procedure:
Choose a positive definite kernel κR, that is associated to a
specific RKHS H.
Map the points xn to Φ(xn) ∈ H, n = 1, . . . ,N.
Solve the linear SVM task on the infinite dimensional
RKHS H, for the training data {(Φ(xn),dn); n = 1, . . . ,N}.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 21 / 47
![Page 60: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/60.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
A toy example
−4 −3 −2 −1 0 1 2 3 4 5 6−6
−4
−2
0
2
4
6
−1
−1
−1
−1
0
0
00
0
0
0
1
1
1
Figure: Non linear SVM classification, C = 2, gaussian kernel
(σ = 2).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 22 / 47
![Page 61: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/61.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
A toy example
−4 −3 −2 −1 0 1 2 3 4 5 6−6
−4
−2
0
2
4
6
−1
−1
−1
−1
−1
0
00
0
0 0
0
1
1
1
1
Figure: Non linear SVM classification, C = 5, gaussian kernel
(σ = 2).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 23 / 47
![Page 62: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/62.jpg)
Introduction
Support Vector Machines
The Complex Case
Linear SVMs
Non-linear SVM
A toy example
−4 −3 −2 −1 0 1 2 3 4 5 6−6
−4
−2
0
2
4
6
−1 −1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
00
0
1
1
1
1
1
Figure: Non linear SVM classification, C = 15, gaussian kernel
(σ = 2).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 24 / 47
![Page 63: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/63.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 25 / 47
![Page 64: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/64.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Real Hyperplanes
Recall that in any real Hilbert space H, a hyperplane
consists of all the elements f ∈ H that satisfy
〈f ,w〉H + b = 0, (2)
for some w ∈ H, b ∈ R.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 26 / 47
![Page 65: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/65.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Real Hyperplanes
Recall that in any real Hilbert space H, a hyperplane
consists of all the elements f ∈ H that satisfy
〈f ,w〉H + b = 0, (2)
for some w ∈ H, b ∈ R.
Moreover, any hyperplane of H divides the space into two
parts, H+ = {f ∈ H; 〈f ,w〉H + b > 0} and
H− = {f ∈ H; 〈f ,w〉H + b < 0}.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 26 / 47
![Page 66: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/66.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
Due to this constraint all the efforts so far to generalize realSVMs to more generic Algebras (quaternions, Cliffordalgebras, e.t.c.) has been explored so that:
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 27 / 47
![Page 67: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/67.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
Due to this constraint all the efforts so far to generalize realSVMs to more generic Algebras (quaternions, Cliffordalgebras, e.t.c.) has been explored so that:
The output variable y is retained to be real.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 27 / 47
![Page 68: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/68.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
Due to this constraint all the efforts so far to generalize realSVMs to more generic Algebras (quaternions, Cliffordalgebras, e.t.c.) has been explored so that:
The output variable y is retained to be real.
The set of functions considered is in one way or another of
a special structure, so that the inner product is real.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 27 / 47
![Page 69: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/69.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
Due to this constraint all the efforts so far to generalize realSVMs to more generic Algebras (quaternions, Cliffordalgebras, e.t.c.) has been explored so that:
The output variable y is retained to be real.
The set of functions considered is in one way or another of
a special structure, so that the inner product is real.
The difficulty is that the set of complex numbers is not an
ordered one, and thus one may not assume that a
complex version of the aforementioned relation divides the
space into two parts, as H+ and H− cannot be defined.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 27 / 47
![Page 70: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/70.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
In order to be able to generalize the SVM rationale to
complex spaces, we need first to develop an appropriate
definition for a complex hyperplane.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 28 / 47
![Page 71: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/71.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
In order to be able to generalize the SVM rationale to
complex spaces, we need first to develop an appropriate
definition for a complex hyperplane.
We begin by considering the following two relations,
Re (〈f ,w〉H + c) = 0,
Im (〈f ,w〉H + c) = 0,
for some w ∈ H, c ∈ C, where f ∈ H.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 28 / 47
![Page 72: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/72.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
In order to be able to generalize the SVM rationale to
complex spaces, we need first to develop an appropriate
definition for a complex hyperplane.
We begin by considering the following two relations,
Re (〈f ,w〉H + c) = 0,
Im (〈f ,w〉H + c) = 0,
for some w ∈ H, c ∈ C, where f ∈ H.
It is not difficult to see, that this couple of relations
represent two orthogonal hyperplanes of the doubled real
space, i.e., H2.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 28 / 47
![Page 73: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/73.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
To overcome this constraint and be able to define arbitrarily
placed hyperplanes, we need to employ the so called
widely linear estimation functions, i.e.,
Re (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,
Im (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,
for some w , v ∈ H, c ∈ C, where f ∈ H.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 29 / 47
![Page 74: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/74.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
To overcome this constraint and be able to define arbitrarily
placed hyperplanes, we need to employ the so called
widely linear estimation functions, i.e.,
Re (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,
Im (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,
for some w , v ∈ H, c ∈ C, where f ∈ H.
Depending on the values of w , v , these hyperplanes may
be placed arbitrarily on H2. We define this complex couple
of hyperplanes as the set of all f ∈ H that satisfy either one
of the two relations, for some w , v ∈ H, c ∈ C.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 29 / 47
![Page 75: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/75.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
The aforementioned arguments demonstrate the significant
difference between complex linear estimation and widely
linear estimation functions, which has been pointed out by
many other authors, in the context of regression tasks.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 30 / 47
![Page 76: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/76.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
The aforementioned arguments demonstrate the significant
difference between complex linear estimation and widely
linear estimation functions, which has been pointed out by
many other authors, in the context of regression tasks.
In the current context of classification, we have just seen
that confining to complex linear modeling is quite
restrictive, as the corresponding couple of complex
hyperplanes are always orthogonal.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 30 / 47
![Page 77: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/77.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
The aforementioned arguments demonstrate the significant
difference between complex linear estimation and widely
linear estimation functions, which has been pointed out by
many other authors, in the context of regression tasks.
In the current context of classification, we have just seen
that confining to complex linear modeling is quite
restrictive, as the corresponding couple of complex
hyperplanes are always orthogonal.
On the other hand, the widely linear case is more general
and covers all cases.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 30 / 47
![Page 78: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/78.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
The complex couple of hyperplanes divides the space H (or
H2) into four parts, i.e.,
H++ =
{
f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) > 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) > 0
}
,
H+− =
{
f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) > 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) < 0
}
,
H−+ =
{
f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) < 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) > 0
}
,
H−− =
{
f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) < 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) < 0
}
.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 31 / 47
![Page 79: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/79.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Generalization
Figure: A complex couple of hyperplanes separates the space of
complex numbers (i.e., H = C) into four parts.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 32 / 47
![Page 80: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/80.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 33 / 47
![Page 81: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/81.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
The problem
Suppose we are given training data, which belong to four
separate classes C++,C+−,C−+,C−−, i.e.,
{(zn,dn); n = 1, . . . ,N} ⊂ X × {±1 ± i)}. If dn = +1 + i , then
the n-th sample belongs to C++, i.e., zn ∈ C++, if dn = 1 − i ,
then zn ∈ C+−, e.t.c.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 34 / 47
![Page 82: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/82.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
The problem
Suppose we are given training data, which belong to four
separate classes C++,C+−,C−+,C−−, i.e.,
{(zn,dn); n = 1, . . . ,N} ⊂ X × {±1 ± i)}. If dn = +1 + i , then
the n-th sample belongs to C++, i.e., zn ∈ C++, if dn = 1 − i ,
then zn ∈ C+−, e.t.c.
As zn is complex, we denote by xn its real part and by yn its
imaginary part respectively, i.e.,
zn = xn + iyn, n = 1, . . . ,N.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 34 / 47
![Page 83: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/83.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
The problem
Suppose we are given training data, which belong to four
separate classes C++,C+−,C−+,C−−, i.e.,
{(zn,dn); n = 1, . . . ,N} ⊂ X × {±1 ± i)}. If dn = +1 + i , then
the n-th sample belongs to C++, i.e., zn ∈ C++, if dn = 1 − i ,
then zn ∈ C+−, e.t.c.
As zn is complex, we denote by xn its real part and by yn its
imaginary part respectively, i.e.,
zn = xn + iyn, n = 1, . . . ,N.
Our objective is to develop an SVM rationale for the complex
training data.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 34 / 47
![Page 84: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/84.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
Consider the complex RKHS, H, with respective kernel κC.
Following a similar rationale to the real case, we transform
the input data from X to H, via the feature map ΦC.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 35 / 47
![Page 85: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/85.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
Consider the complex RKHS, H, with respective kernel κC.
Following a similar rationale to the real case, we transform
the input data from X to H, via the feature map ΦC.
The goal of the SVM task is to estimate a complex couple
of maximum margin hyperplanes, that separates the points
of the four classes as best as possible. To this end, we
formulate the primal complex SVM as
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 35 / 47
![Page 86: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/86.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
Consider the complex RKHS, H, with respective kernel κC.
Following a similar rationale to the real case, we transform
the input data from X to H, via the feature map ΦC.
The goal of the SVM task is to estimate a complex couple
of maximum margin hyperplanes, that separates the points
of the four classes as best as possible. To this end, we
formulate the primal complex SVM as
minw ,v ,c
1
2‖w‖2
H+ 1
2‖v‖2
H+ C
N
N∑
n=1
(ξrn + ξi
n)
s. to
d rn Re (〈ΦC(zn),w〉H + 〈Φ∗
C(zn), v〉H + c) ≥ 1 − ξr
n
d in Im (〈ΦC(zn),w〉H + 〈Φ∗
C(zn),w〉H + c) ≥ 1 − ξi
n
ξrn, ξ
in ≥ 0
for n = 1, . . . ,N.
(3)
for some C > 0.P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 35 / 47
![Page 87: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/87.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
Consequently, the Lagrangian function becomes
L(w , v , a, a,b, b) =1
2‖w‖2
H +1
2‖v‖2
H +C
N
N∑
n=1
(ξrn + ξi
n)
−
N∑
n=1
an (drn Re (〈ΦC(zn),w〉H + 〈Φ∗
C(zn), v〉H + c)− 1 + ξrn)
−
N∑
n=1
bn
(
d in Im (〈ΦC(zn),w〉H + 〈Φ∗
C(zn),w〉H + c)− 1 + ξin
)
−
N∑
n=1
ηnξrn −
N∑
n=1
θnξin,
where an,bn, ηn, θn are the positive Lagrange multipliers of the
respective inequalities, for n = 1, . . . ,N.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 36 / 47
![Page 88: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/88.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
Employing the notion of Wirtinger’s calculus to derive therespective gradients and exploiting the saddle point conditionsof the Lagrangian function, it turns out that the dual problemcan be split into two separate maximization tasks:
maximizea
N∑
n=1
an −1
2
N∑
n,m=1
anamdrnd
rmκ
rC(zm, zn)
subject to
N∑
n=1
andrn = 0
0 ≤ an ≤ CN
for n = 1, . . . , N
(4a)
and
maximizea
N∑
n=1
bn −1
2
N∑
n,m=1
bnbmdind
imκ
rC(zm, zn)
subject to
N∑
n=1
bndin = 0
0 ≤ bn ≤ CN
for n = 1, . . . , N,
(4b)
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 37 / 47
![Page 89: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/89.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
We observe that these problems are equivalent with two distinct
real SVM (dual) tasks employing the induced real kernel κrC
:
κrC(z, z
′) = 2 Re(κC(z , z′)), (5)
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 38 / 47
![Page 90: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/90.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
We observe that these problems are equivalent with two distinct
real SVM (dual) tasks employing the induced real kernel κrC
:
κrC(z, z
′) = 2 Re(κC(z , z′)), (5)
One may
split the (output) data to their real and imaginary parts,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 38 / 47
![Page 91: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/91.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
We observe that these problems are equivalent with two distinct
real SVM (dual) tasks employing the induced real kernel κrC
:
κrC(z, z
′) = 2 Re(κC(z , z′)), (5)
One may
split the (output) data to their real and imaginary parts,
solve two real SVM tasks employing any one of the
standard algorithms and, finally,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 38 / 47
![Page 92: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/92.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complex formulation
We observe that these problems are equivalent with two distinct
real SVM (dual) tasks employing the induced real kernel κrC
:
κrC(z, z
′) = 2 Re(κC(z , z′)), (5)
One may
split the (output) data to their real and imaginary parts,
solve two real SVM tasks employing any one of the
standard algorithms and, finally,
combine the solutions to take the complex labeling
function:
g(z) = signi
(
N∑
n=1
(and rn + ibnd i
n)κrC(zn, z) + cr + ic i
)
,
where signi
(z) = sign(Re(z)) + i sign(Im(z)).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 38 / 47
![Page 93: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/93.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Pure Complex SVM
Figure: Pure Complex Support Vector Machines.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 39 / 47
![Page 94: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/94.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complexification
An alternative path is the so called complexification
procedure.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 40 / 47
![Page 95: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/95.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complexification
An alternative path is the so called complexification
procedure.
We employ a real kernel κR and transform the input data
from X to the complexified space H, i.e.,
x → ΦR(x) + iΦR(x).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 40 / 47
![Page 96: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/96.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complexification
An alternative path is the so called complexification
procedure.
We employ a real kernel κR and transform the input data
from X to the complexified space H, i.e.,
x → ΦR(x) + iΦR(x).
We can similarly deduce that the dual of the complexified
SVM task is equivalent to two real SVM tasks employing
the kernel 2κR.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 40 / 47
![Page 97: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/97.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Complexification
An alternative path is the so called complexification
procedure.
We employ a real kernel κR and transform the input data
from X to the complexified space H, i.e.,
x → ΦR(x) + iΦR(x).
We can similarly deduce that the dual of the complexified
SVM task is equivalent to two real SVM tasks employing
the kernel 2κR.
We conclude that, in both cases, we end up with two real
SVM tasks (although employing different types of kernels).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 40 / 47
![Page 98: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/98.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Binary Classification
Although both scenarios are developed naturally for
quaternary classification, they can be easily adapted to the
binary case also.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 41 / 47
![Page 99: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/99.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Binary Classification
Although both scenarios are developed naturally for
quaternary classification, they can be easily adapted to the
binary case also.
This can be done by considering that the labels of the data
are real numbers (i.e., dn ∈ R) taking the values ±1. In this
case we solve one problem instead of two.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 41 / 47
![Page 100: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/100.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Outline
1 Introduction
Reproducing Kernel Hilbert Spaces
Complex RKHS
2 Support Vector Machines
Linear SVMs
Non-linear SVM
3 The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 42 / 47
![Page 101: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/101.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
MNIST
We use the popular MNIST database of handwritten digits.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 43 / 47
![Page 102: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/102.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
MNIST
We use the popular MNIST database of handwritten digits.
Each digit is encoded as an image file with 28 × 28 pixels.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 43 / 47
![Page 103: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/103.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
MNIST
We use the popular MNIST database of handwritten digits.
Each digit is encoded as an image file with 28 × 28 pixels.
MNIST contains 60000 handwritten digits (from 0 to 9) for
training and 10000 handwritten digits for testing.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 43 / 47
![Page 104: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/104.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
MNIST
We use the popular MNIST database of handwritten digits.
Each digit is encoded as an image file with 28 × 28 pixels.
MNIST contains 60000 handwritten digits (from 0 to 9) for
training and 10000 handwritten digits for testing.
To exploit the structure of complex numbers, we perform a
Fourier transform to each training image and keep only the
100 most significant coefficients.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 43 / 47
![Page 105: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/105.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
We compare a standard one-versus-all SVM scenario that
exploits the original (real) data (images of 28 × 28 = 784
pixels) with
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 44 / 47
![Page 106: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/106.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
We compare a standard one-versus-all SVM scenario that
exploits the original (real) data (images of 28 × 28 = 784
pixels) with
a complex one versus all variant exploiting the
complexified binary SVM, where we use only the 100 most
significant (complex) Fourier coefficients of each picture.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 44 / 47
![Page 107: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/107.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
We compare a standard one-versus-all SVM scenario that
exploits the original (real) data (images of 28 × 28 = 784
pixels) with
a complex one versus all variant exploiting the
complexified binary SVM, where we use only the 100 most
significant (complex) Fourier coefficients of each picture.
In both scenarios we use the first 6000 digits of the MNIST
training set to train the learning machines and test their
performances using the 10000 digits of the testing set.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 44 / 47
![Page 108: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/108.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
We compare a standard one-versus-all SVM scenario that
exploits the original (real) data (images of 28 × 28 = 784
pixels) with
a complex one versus all variant exploiting the
complexified binary SVM, where we use only the 100 most
significant (complex) Fourier coefficients of each picture.
In both scenarios we use the first 6000 digits of the MNIST
training set to train the learning machines and test their
performances using the 10000 digits of the testing set.
We used the gaussian kernel with t = 1/64 and
t = 1/1402 respectively.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 44 / 47
![Page 109: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/109.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
We compare a standard one-versus-all SVM scenario that
exploits the original (real) data (images of 28 × 28 = 784
pixels) with
a complex one versus all variant exploiting the
complexified binary SVM, where we use only the 100 most
significant (complex) Fourier coefficients of each picture.
In both scenarios we use the first 6000 digits of the MNIST
training set to train the learning machines and test their
performances using the 10000 digits of the testing set.
We used the gaussian kernel with t = 1/64 and
t = 1/1402 respectively.
The SVM parameter C has been set equal to 100.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 44 / 47
![Page 110: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/110.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
The error rate of the standard real-valued scenario is
3.79%, while the error rate of the complexified
(one-versus-all) SVM is 3.46%.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 45 / 47
![Page 111: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/111.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
The error rate of the standard real-valued scenario is
3.79%, while the error rate of the complexified
(one-versus-all) SVM is 3.46%.
In both learning tasks we used the SMO algorithm to train
the SVM.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 45 / 47
![Page 112: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/112.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
First Experiment
The error rate of the standard real-valued scenario is
3.79%, while the error rate of the complexified
(one-versus-all) SVM is 3.46%.
In both learning tasks we used the SMO algorithm to train
the SVM.
The total amount of time needed to perform the training of
each learning machine is almost the same for both cases
(the complexified task is slightly faster).
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 45 / 47
![Page 113: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/113.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
This is a quaternary classification problem.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 46 / 47
![Page 114: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/114.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
This is a quaternary classification problem.
Using the complex approach, such a problem can be
solved using only 2 distinct SVM tasks, instead of the 4
SVM tasks needed by the standard 1-versus-all or the
1-versus-1 strategies.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 46 / 47
![Page 115: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/115.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
This is a quaternary classification problem.
Using the complex approach, such a problem can be
solved using only 2 distinct SVM tasks, instead of the 4
SVM tasks needed by the standard 1-versus-all or the
1-versus-1 strategies.
We compare a complex quaternary SVM task with the
1-versus-all scenario.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 46 / 47
![Page 116: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/116.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
This is a quaternary classification problem.
Using the complex approach, such a problem can be
solved using only 2 distinct SVM tasks, instead of the 4
SVM tasks needed by the standard 1-versus-all or the
1-versus-1 strategies.
We compare a complex quaternary SVM task with the
1-versus-all scenario.
To this end we use the first 6000, 0, 1, 2 and 3 digits of the
MNIST training set and compare the performances of the
two algorithms using the respective digits of the MNIST
training set.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 46 / 47
![Page 117: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/117.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
The error rate of the 1-versus-all SVM was 0.721%,
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 47 / 47
![Page 118: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/118.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
The error rate of the 1-versus-all SVM was 0.721%,
while the error rate of the complex SVM was 0.866%.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 47 / 47
![Page 119: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/119.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
The error rate of the 1-versus-all SVM was 0.721%,
while the error rate of the complex SVM was 0.866%.
In terms of speed the 1-versus-all SVM task required about
double the time for training, compared to the complex SVM.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 47 / 47
![Page 120: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/120.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
The error rate of the 1-versus-all SVM was 0.721%,
while the error rate of the complex SVM was 0.866%.
In terms of speed the 1-versus-all SVM task required about
double the time for training, compared to the complex SVM.
This is expected, as the latter solves half as many distinct
SVM tasks as the first one.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 47 / 47
![Page 121: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/121.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
The error rate of the 1-versus-all SVM was 0.721%,
while the error rate of the complex SVM was 0.866%.
In terms of speed the 1-versus-all SVM task required about
double the time for training, compared to the complex SVM.
This is expected, as the latter solves half as many distinct
SVM tasks as the first one.
In both experiments we used the gaussian kernel with
t = 1/49 and t = 1/1602 respectively.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 47 / 47
![Page 122: Complex Support Vector Machines For Quaternary Classification](https://reader034.vdocuments.site/reader034/viewer/2022042714/557cbd49d8b42a1b0c8b4fd9/html5/thumbnails/122.jpg)
Introduction
Support Vector Machines
The Complex Case
Complex Hyperplanes
Problem formulation
Experiments
Second Experiment - Quaternary Classification
The error rate of the 1-versus-all SVM was 0.721%,
while the error rate of the complex SVM was 0.866%.
In terms of speed the 1-versus-all SVM task required about
double the time for training, compared to the complex SVM.
This is expected, as the latter solves half as many distinct
SVM tasks as the first one.
In both experiments we used the gaussian kernel with
t = 1/49 and t = 1/1602 respectively.
The SVM parameter C has been set equal to 100 in this
case also.
P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 47 / 47