complex support vector machines for quaternary classification

Introduction

Support Vector Machines

The Complex Case

Complex Support Vector Machines For

Quaternary Classification

P. Bouboulis, E. Theodoridou, S. Theodoridis

Department of Informatics and TelecommunicationsUniversity of Athens

Athens, Greece

23-09-2013

P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 1 / 47

Introduction

The Complex Case

Outline

1 Introduction

Reproducing Kernel Hilbert Spaces

Complex RKHS

2 Support Vector Machines

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Complex RKHS

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Complex RKHS

Reproducing Kernel Hilbert Spaces.

Consider a linear class H of real (complex) valued functions f

defined on a set X (in particular H is a Hilbert space), for which

there exists a function (kernel) κ : X × X → R(C) with the

following two properties:

Introduction

The Complex Case

Complex RKHS

1 For every x ∈ X , κ(x , ·) belongs to H.

Introduction

The Complex Case

Complex RKHS

2 κ has the so called reproducing property, i.e.,

f (x) = 〈f , κ(x , ·)〉H, for all f ∈ H, x ∈ X . (1)

Introduction

The Complex Case

Complex RKHS

2 κ has the so called reproducing property, i.e.,

f (x) = 〈f , κ(x , ·)〉H, for all f ∈ H, x ∈ X . (1)

Then H is called a Reproducing Kernel Hilbert Space (RKHS)

associated to the the kernel κ.

Introduction

The Complex Case

Complex RKHS

Kernel Trick

The notion of RKHS is a popular tool for treating non-linear

learning tasks.

Introduction

The Complex Case

Complex RKHS

Kernel Trick

learning tasks.

Usually this is attained by the so called “kernel trick”.

Introduction

The Complex Case

Complex RKHS

Kernel Trick

learning tasks.

X ∋ x → Φ(x) := κ(x , ·) ∈ HX ∋ y → Φ(y) := κ(y , ·) ∈ H,

Introduction

The Complex Case

Complex RKHS

Kernel Trick

learning tasks.

X ∋ x → Φ(x) := κ(x , ·) ∈ HX ∋ y → Φ(y) := κ(y , ·) ∈ H,

then the inner product in H is given as a function computed on

κ(x , y) = 〈κ(x , ·), κ(y , ·)〉H kernel trick

Introduction

The Complex Case

Complex RKHS

Developing Learning Algorithms in RKHS

The black box approach.

Introduction

The Complex Case

Complex RKHS

Develop the learning Algorithm in X .

Introduction

The Complex Case

Complex RKHS

Express it, if possible, in inner products.

Introduction

The Complex Case

Complex RKHS

Choose a kernel function κ.

Introduction

The Complex Case

Complex RKHS

Choose a kernel function κ.Replace inner products with kernel evaluations according to

the kernel trick.

Introduction

The Complex Case

Complex RKHS

Choose a kernel function κ.Replace inner products with kernel evaluations according to

the kernel trick.

Work directly in the RKHS, assuming that the data have

been mapped and live in the RKHS H, i.e.,

X ∋ x → Φ(x) := κ(x , ·) ∈ H.

Introduction

The Complex Case

Complex RKHS

Advantages

Advantages of kernel-based learning tasks:

Introduction

The Complex Case

Complex RKHS

Advantages

The original nonlinear task is transformed into a linear one.

Introduction

The Complex Case

Complex RKHS

Advantages

The original nonlinear task is transformed into a linear one.

Different types of nonlinearities can be treated in a unified

Introduction

The Complex Case

Complex RKHS

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Complex RKHS

Although the theory of RKHS holds for complex spaces

too, most of the kernel-based learning techniques were

designed to process real data only.

Introduction

The Complex Case

Complex RKHS

Moreover, in the related literature the complex kernel

functions have been ignored.

Introduction

The Complex Case

Complex RKHS

Recently, however, a unified kernel-based framework,

which is able to treat complex data, has been presented.

Introduction

The Complex Case

Complex RKHS

This machinery transforms the input data into a complex

RKHS, i.e.,

Φ(z) = κC(·, z).

Introduction

The Complex Case

Complex RKHS

This machinery transforms the input data into a complex

RKHS, i.e.,

Φ(z) = κC(·, z).

and employs the Wirtinger’s Calculus to derive the

respective gradients.

Introduction

The Complex Case

Complex RKHS

Definitions:

H denotes a complex RKHS.

Introduction

The Complex Case

Complex RKHS

Definitions:

H denotes a real RKHS.

Introduction

The Complex Case

Complex RKHS

Definitions:

The complex RKHS can be expressed as H = H + iH.

Introduction

The Complex Case

Complex RKHS

Definitions:

The complex RKHS can be expressed as H = H + iH.

H is isomorphic to the doubled real space H2.

Introduction

The Complex Case

Complex RKHS

Complex Kernels

The complex Gaussian kernel:

κ(z ,w) = exp

−∑d

i=1(zi−w∗i)2

Introduction

The Complex Case

Complex RKHS

Complex Kernels

κ(z ,w) = exp

−∑d

i=1(zi−w∗i)2

The Szego kernel: κ(z ,w) = 11−wHz

Introduction

The Complex Case

Complex RKHS

Complex Kernels

κ(z ,w) = exp

−∑d

i=1(zi−w∗i)2

The Szego kernel: κ(z ,w) = 11−wHz

Bergman kernel: κ(z,w) = 1(1−wHz)2 .

Introduction

The Complex Case

Complex RKHS

Wirtinger Calculus

Complex differentiability is a very strict notion.

Introduction

The Complex Case

Complex RKHS

Wirtinger Calculus

In learning tasks that involve complex data, we often

encounter functions (e.g., the cost functions, which are

defined in R) that ARE NOT complex differentiable.

Introduction

The Complex Case

Complex RKHS

Wirtinger Calculus

Example: f (z) = |z|2 = zz∗.

Introduction

The Complex Case

Complex RKHS

Wirtinger Calculus

Example: f (z) = |z|2 = zz∗.

In these cases one has to express the cost function in

terms of its real part fr and its imaginary part fi , and use

real derivation with respect to fr , fi .

Introduction

The Complex Case

Complex RKHS

Wirtinger’s Calculus

This approach leads usually to cumbersome and tedious

calculations.

Introduction

The Complex Case

Complex RKHS

calculations.

Wirtinger’s Calculus provides an alternative equivalent

formulation.

Introduction

The Complex Case

Complex RKHS

calculations.

formulation.

It is based on simple rules and principles.

Introduction

The Complex Case

Complex RKHS

calculations.

formulation.

It is based on simple rules and principles.

These rules bear a great resemblance to the rules of the

standard complex derivative.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

Suppose we are given training data, which belong to two

separate classes C+,C−,i.e.,

{(xn,dn); n = 1, . . . ,N} ⊂ X × {±1}, where if dn = +1, then

the n-th sample belongs to C+, while if dn = −1, then the n-th

sample belongs to C−.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

Suppose we are given training data, which belong to two

separate classes C+,C−,i.e.,

{(xn,dn); n = 1, . . . ,N} ⊂ X × {±1}, where if dn = +1, then

the n-th sample belongs to C+, while if dn = −1, then the n-th

sample belongs to C−.

For example:

Figure: Training points belonging to two classes.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

The goal of the SVM task is to estimate the maximum margin

hyperplane (wT x + c = 0), that separates the points of the two

classes as best as possible

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

minimizew∈X ,c∈R

12‖w‖2

subject to

wT xn + c)

≥ 1 − ξn

ξn ≥ 0

for n = 1, . . . ,N,

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The primal problem

minimizew∈X ,c∈R

12‖w‖2

subject to

wT xn + c)

≥ 1 − ξn

ξn ≥ 0

for n = 1, . . . ,N,

Note that C is chosen a priori.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

Physical justification

minimizew∈X,c∈R

12‖w‖2

subject to

wT xn + c)

≥ 1 − ξn

ξn ≥ 0for n = 1, . . . , N,

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

Physical justification

minimizew∈X,c∈R

12‖w‖2

subject to

wT xn + c)

≥ 1 − ξn

ξn ≥ 0for n = 1, . . . , N,

Figure: Linear SVM

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The dual problem

To solve this task, usually we consider the dual problem derived

by the Lagrangian:

maximizea∈RN

an −1

anamdndmxTmxn

subject to

andn = 0 and an ∈ [0,C/N].

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick

Choose a positive definite kernel κR.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick

In the dual problem, replace the inner products xTn xm with

the respective kernel evaluations, i.e., κR(xn,xm).

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick

The application of the kernel trick leads to the nonlinear

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

The kernel trick

The application of the kernel trick leads to the nonlinear

maximizea∈RN

an −1

anamdndmκR(xm,xn)

subject to

andn = 0 and an ∈ [0,C/N].

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

Mapping to the feature space

The application of the kernel trick to the dual problem is

equivalent to the following procedure:

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

Choose a positive definite kernel κR, that is associated to a

specific RKHS H.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

specific RKHS H.

Map the points xn to Φ(xn) ∈ H, n = 1, . . . ,N.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

specific RKHS H.

Map the points xn to Φ(xn) ∈ H, n = 1, . . . ,N.

Solve the linear SVM task on the infinite dimensional

RKHS H, for the training data {(Φ(xn),dn); n = 1, . . . ,N}.

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

A toy example

−4 −3 −2 −1 0 1 2 3 4 5 6−6

Figure: Non linear SVM classification, C = 2, gaussian kernel

(σ = 2).

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

A toy example

−4 −3 −2 −1 0 1 2 3 4 5 6−6

(σ = 2).

Introduction

The Complex Case

Linear SVMs

Non-linear SVM

A toy example

−4 −3 −2 −1 0 1 2 3 4 5 6−6

−1 −1

(σ = 2).

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Real Hyperplanes

Recall that in any real Hilbert space H, a hyperplane

consists of all the elements f ∈ H that satisfy

〈f ,w〉H + b = 0, (2)

for some w ∈ H, b ∈ R.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Real Hyperplanes

Recall that in any real Hilbert space H, a hyperplane

consists of all the elements f ∈ H that satisfy

〈f ,w〉H + b = 0, (2)

for some w ∈ H, b ∈ R.

Moreover, any hyperplane of H divides the space into two

parts, H+ = {f ∈ H; 〈f ,w〉H + b > 0} and

H− = {f ∈ H; 〈f ,w〉H + b < 0}.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

Due to this constraint all the efforts so far to generalize realSVMs to more generic Algebras (quaternions, Cliffordalgebras, e.t.c.) has been explored so that:

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The output variable y is retained to be real.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The set of functions considered is in one way or another of

a special structure, so that the inner product is real.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The set of functions considered is in one way or another of

a special structure, so that the inner product is real.

The difficulty is that the set of complex numbers is not an

ordered one, and thus one may not assume that a

complex version of the aforementioned relation divides the

space into two parts, as H+ and H− cannot be defined.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

In order to be able to generalize the SVM rationale to

complex spaces, we need first to develop an appropriate

definition for a complex hyperplane.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

We begin by considering the following two relations,

Re (〈f ,w〉H + c) = 0,

Im (〈f ,w〉H + c) = 0,

for some w ∈ H, c ∈ C, where f ∈ H.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

We begin by considering the following two relations,

Re (〈f ,w〉H + c) = 0,

Im (〈f ,w〉H + c) = 0,

for some w ∈ H, c ∈ C, where f ∈ H.

It is not difficult to see, that this couple of relations

represent two orthogonal hyperplanes of the doubled real

space, i.e., H2.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

To overcome this constraint and be able to define arbitrarily

placed hyperplanes, we need to employ the so called

widely linear estimation functions, i.e.,

Re (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

Im (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

for some w , v ∈ H, c ∈ C, where f ∈ H.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

To overcome this constraint and be able to define arbitrarily

placed hyperplanes, we need to employ the so called

widely linear estimation functions, i.e.,

Re (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

Im (〈f ,w〉H + 〈f ∗, v〉H + c) = 0,

for some w , v ∈ H, c ∈ C, where f ∈ H.

Depending on the values of w , v , these hyperplanes may

be placed arbitrarily on H2. We define this complex couple

of hyperplanes as the set of all f ∈ H that satisfy either one

of the two relations, for some w , v ∈ H, c ∈ C.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The aforementioned arguments demonstrate the significant

difference between complex linear estimation and widely

linear estimation functions, which has been pointed out by

many other authors, in the context of regression tasks.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

In the current context of classification, we have just seen

that confining to complex linear modeling is quite

restrictive, as the corresponding couple of complex

hyperplanes are always orthogonal.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

In the current context of classification, we have just seen

that confining to complex linear modeling is quite

restrictive, as the corresponding couple of complex

hyperplanes are always orthogonal.

On the other hand, the widely linear case is more general

and covers all cases.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

The complex couple of hyperplanes divides the space H (or

H2) into four parts, i.e.,

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) > 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) > 0

H+− =

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) > 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) < 0

H−+ =

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) < 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) > 0

H−− =

f ∈ H;Re (〈f ,w〉H + 〈f ∗, v〉H + c) < 0,Im (〈f ,w〉H + 〈f ∗, v〉H + c) < 0

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Generalization

Figure: A complex couple of hyperplanes separates the space of

complex numbers (i.e., H = C) into four parts.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The problem

Suppose we are given training data, which belong to four

separate classes C++,C+−,C−+,C−−, i.e.,

{(zn,dn); n = 1, . . . ,N} ⊂ X × {±1 ± i)}. If dn = +1 + i , then

the n-th sample belongs to C++, i.e., zn ∈ C++, if dn = 1 − i ,

then zn ∈ C+−, e.t.c.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The problem

As zn is complex, we denote by xn its real part and by yn its

imaginary part respectively, i.e.,

zn = xn + iyn, n = 1, . . . ,N.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The problem

As zn is complex, we denote by xn its real part and by yn its

imaginary part respectively, i.e.,

zn = xn + iyn, n = 1, . . . ,N.

Our objective is to develop an SVM rationale for the complex

training data.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

Consider the complex RKHS, H, with respective kernel κC.

Following a similar rationale to the real case, we transform

the input data from X to H, via the feature map ΦC.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

The goal of the SVM task is to estimate a complex couple

of maximum margin hyperplanes, that separates the points

of the four classes as best as possible. To this end, we

formulate the primal complex SVM as

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

The goal of the SVM task is to estimate a complex couple

of maximum margin hyperplanes, that separates the points

of the four classes as best as possible. To this end, we

formulate the primal complex SVM as

minw ,v ,c

2‖w‖2

2‖v‖2

(ξrn + ξi

d rn Re (〈ΦC(zn),w〉H + 〈Φ∗

C(zn), v〉H + c) ≥ 1 − ξr

d in Im (〈ΦC(zn),w〉H + 〈Φ∗

C(zn),w〉H + c) ≥ 1 − ξi

ξrn, ξ

in ≥ 0

for n = 1, . . . ,N.

for some C > 0.P. Bouboulis, E. Theodoridou, S. Theodoridis CSVR 35 / 47

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

Consequently, the Lagrangian function becomes

L(w , v , a, a,b, b) =1

2‖w‖2

2‖v‖2

(ξrn + ξi

an (drn Re (〈ΦC(zn),w〉H + 〈Φ∗

C(zn), v〉H + c)− 1 + ξrn)

d in Im (〈ΦC(zn),w〉H + 〈Φ∗

C(zn),w〉H + c)− 1 + ξin

ηnξrn −

θnξin,

where an,bn, ηn, θn are the positive Lagrange multipliers of the

respective inequalities, for n = 1, . . . ,N.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

Employing the notion of Wirtinger’s calculus to derive therespective gradients and exploiting the saddle point conditionsof the Lagrangian function, it turns out that the dual problemcan be split into two separate maximization tasks:

maximizea

an −1

anamdrnd

rC(zm, zn)

subject to

andrn = 0

0 ≤ an ≤ CN

for n = 1, . . . , N

maximizea

bn −1

bnbmdind

rC(zm, zn)

subject to

bndin = 0

0 ≤ bn ≤ CN

for n = 1, . . . , N,

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

We observe that these problems are equivalent with two distinct

real SVM (dual) tasks employing the induced real kernel κrC

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

One may

split the (output) data to their real and imaginary parts,

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

One may

solve two real SVM tasks employing any one of the

standard algorithms and, finally,

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complex formulation

κrC(z, z

′) = 2 Re(κC(z , z′)), (5)

One may

solve two real SVM tasks employing any one of the

standard algorithms and, finally,

combine the solutions to take the complex labeling

function:

g(z) = signi

(and rn + ibnd i

n)κrC(zn, z) + cr + ic i

where signi

(z) = sign(Re(z)) + i sign(Im(z)).

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Pure Complex SVM

Figure: Pure Complex Support Vector Machines.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification

An alternative path is the so called complexification

procedure.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification

procedure.

We employ a real kernel κR and transform the input data

from X to the complexified space H, i.e.,

x → ΦR(x) + iΦR(x).

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification

procedure.

We can similarly deduce that the dual of the complexified

SVM task is equivalent to two real SVM tasks employing

the kernel 2κR.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Complexification

procedure.

We can similarly deduce that the dual of the complexified

SVM task is equivalent to two real SVM tasks employing

the kernel 2κR.

We conclude that, in both cases, we end up with two real

SVM tasks (although employing different types of kernels).

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Binary Classification

Although both scenarios are developed naturally for

quaternary classification, they can be easily adapted to the

binary case also.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Binary Classification

Although both scenarios are developed naturally for

quaternary classification, they can be easily adapted to the

binary case also.

This can be done by considering that the labels of the data

are real numbers (i.e., dn ∈ R) taking the values ±1. In this

case we solve one problem instead of two.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Outline

1 Introduction

Complex RKHS

Linear SVMs

Non-linear SVM

3 The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

We use the popular MNIST database of handwritten digits.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Each digit is encoded as an image file with 28 × 28 pixels.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

MNIST contains 60000 handwritten digits (from 0 to 9) for

training and 10000 handwritten digits for testing.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

MNIST contains 60000 handwritten digits (from 0 to 9) for

training and 10000 handwritten digits for testing.

To exploit the structure of complex numbers, we perform a

Fourier transform to each training image and keep only the

100 most significant coefficients.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

We compare a standard one-versus-all SVM scenario that

exploits the original (real) data (images of 28 × 28 = 784

pixels) with

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

pixels) with

a complex one versus all variant exploiting the

complexified binary SVM, where we use only the 100 most

significant (complex) Fourier coefficients of each picture.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

pixels) with

In both scenarios we use the first 6000 digits of the MNIST

training set to train the learning machines and test their

performances using the 10000 digits of the testing set.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

pixels) with

We used the gaussian kernel with t = 1/64 and

t = 1/1402 respectively.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

pixels) with

We used the gaussian kernel with t = 1/64 and

t = 1/1402 respectively.

The SVM parameter C has been set equal to 100.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

The error rate of the standard real-valued scenario is

3.79%, while the error rate of the complexified

(one-versus-all) SVM is 3.46%.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

In both learning tasks we used the SMO algorithm to train

the SVM.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

First Experiment

In both learning tasks we used the SMO algorithm to train

the SVM.

The total amount of time needed to perform the training of

each learning machine is almost the same for both cases

(the complexified task is slightly faster).

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Second Experiment - Quaternary Classification

This is a quaternary classification problem.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

Using the complex approach, such a problem can be

solved using only 2 distinct SVM tasks, instead of the 4

SVM tasks needed by the standard 1-versus-all or the

1-versus-1 strategies.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

We compare a complex quaternary SVM task with the

1-versus-all scenario.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

We compare a complex quaternary SVM task with the

1-versus-all scenario.

To this end we use the first 6000, 0, 1, 2 and 3 digits of the

MNIST training set and compare the performances of the

two algorithms using the respective digits of the MNIST

training set.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

The error rate of the 1-versus-all SVM was 0.721%,

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

while the error rate of the complex SVM was 0.866%.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

In terms of speed the 1-versus-all SVM task required about

double the time for training, compared to the complex SVM.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

This is expected, as the latter solves half as many distinct

SVM tasks as the first one.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

In both experiments we used the gaussian kernel with

t = 1/49 and t = 1/1602 respectively.

Introduction

The Complex Case

Complex Hyperplanes

Problem formulation

Experiments

In both experiments we used the gaussian kernel with

t = 1/49 and t = 1/1602 respectively.

The SVM parameter C has been set equal to 100 in this

case also.

complex support vector machines for quaternary classification

x x rc

complex case outline

h kernel trick

function kernel

h x y y

theodoridis csvr

notion of rkhs

reproducing property

Education

support vector machines -...

support vector machines hyperplane...

kernel methods: support vector machines maximum margin...

support vector machines kernel machines

using support vector machines, convolutional … support...

support vector machines & kernel machines

using support vector machines, convolutional neural ... ·...

support vector machines