nips 2000 workshop on kernel methods frame, reproducing kernel and learning alain rakotomamonjy...

NIPS 2000 Workshop on Kernel methods

Frame, Reproducing Kernel and Learning

Alain RakotomamonjyStéphane Canu

http://asi.insa-rouen.fr/~arakotomAlain.Rakoto,[email protected]

Perception, Systèmes et InformationInsa de Rouen,76801 St Etienne du RouvrayFrance


Motivations

Wavelet-based approximation (wavelet or ridgelet networks) are regularization networks?

Construction of multiresolution scheme of approximation

kernel adapted to the structures of function to be learned


Motivations Ctd.

Frame based framework for learning

Approximating highly oscillating structure Without losing

regularity in smooth region


Road Map

Introduction on FrameFrom Frame to KernelsFrom Frame kernels to learningConclusions and perspectives


Frame : A definition

H : Hilbert Space dot product nn A sequence of elements of H

nn is a frame of H if there exists A,B > O s.t

n

2

H

2

Hn2

HfB,ffAHf

A,B are the frame bounds

H,


Frame : definition Ctd.

Frame intepretation

Frame allows stable representationas for all f in H

n

n*n,ff

Frame = "Basis" + linear dependency + redundancy

n*n being a dual frame of n in H


Particular cases of Frame

Tight FrameFrame with bounds s.t A=B

Orthonormal BasisA=B=1

Riesz BasisFrame elements are linearly independent

n

2

H

2

Hn f,f,Hf

]np[, n*p

n*n A

1


Examples of Frame

Tight Frame of IR2

Frame of L2(IR)

11 e

21

2 e23

2e

21

3 e23

2e

3

1n

22

n2 f

23,f,IRf

2Zn,jk,j )t(

j

jo

jk,j aanut

a

1)t( is an admissible wavelet

12

3


Road Map



Frameable RKHS

Condition for having a RKHSSuppose H is a Hilbert space of function IRIR n

nnand a frame of H

Hn

n*n )t()(,t

The Reproducing Kernel is

n

n*n )t()s()t,s(K

H is a RKHS if Htt fM)t(ft.s0M,t,Hf

On a frameable Hilbert Space, this is equivalent to


Construction of Frameable RKHS

A Practical way to build a RKHSF is a Hilbert Space of function IRIR:f n

N..1nn A finite set of F elements such that

Fn,N...1n

M)t(t,N...1n,IRM n

Fn ,,span is a RKHS with {n} as frame elements


Example of Frameable RKHS

frameable RKHS included in L2(IR)i : L2 function (e.g i is a wavelet) span {i}i=1…N is a RKHS

span a RKHS with kernel

3

1ii

*i )t()s()t,s(K

3 wavelets at same scale jExample


Road Map



Semiparametric EstimationContext Learning from training set (xi,yi)i=1..N

One looks for the minimizer of the risk functional 2

H

N

1iii f)x(f,yC

in a space H + span{i}i=1…m H being a RKHS

m

1j

N

1iiHijj

* )x,x(Kb)x(a)x(f

Under general conditions,

Semiparametric framework

span{i}i=1…m : parametric hypothesis space


Semiparametric EstimationParametric hyp. space is a frameable

RKHSP is a frameable RKHS spanned by {n}, with P H, H RHKS

PPH PPKKK

2

P

N

1iii f)x(f,yC

One looks for the minimizer in H of

m

1j

N

1iiPijj

* )x,x(Kb)x(a)x(f

As spaces are orthogonal, backfitting is sufficient for estimating f*

Semiparametric estimation on H with P as a parametric hyp. space


Semiparametric Estimation

H= P + N P : Frameable RKHS, N : Frameable RKHS

N: "unknown component" to be regularized

P : "known component" not to regularized

Frame view point H frameable

H defined by kernel K

H

NPH KN=KH-KP

P N : due to linear dependency of frame

P : Frameable RKHS


Multiscale approximation

H a frameable RKHS

1m...1iFHH 1i1ii

m1m10 HHHH And any space Hi or Fi is a RKHS

Hi : Trend Spaces Fi : Details Spaces

H is splitted in different spaces {Fi}i=1…m-1 and H0


Multiscale Approximation Ctd.

At each step j, trend obtained at step j-1 is decomposed in trend and details

H

H2 F2

H1 F1

H0 F0

ii y,x

k

k,2k,2d k

ik,2k,2 )x,x(Kc Resid.

k

k,1k,1d k


k

k,0k,0d k


f*


Multiscale Approximation Ctd.

ValidityAt each step, representer Theorem

Hypothesis must be verified

Solution

Trend

kkk

Details

1m

0j

N

1iijj,i

*

0

)x(d)x,x(Kc)x(f


Illustration on toy problem

)2t(5))2t(5sin(

)5t())5t(sin()xsin()x(s

Function to be learnedData xi : N points from the random sampling of [0, 10]

),0(N)x(sy ii

Algorithm - SVM Regression- Multiscale Regularisation networks on Frameable RKHS

Sin/Sinc based kernelWavelet based kernel


Results

SVM Wavelet Kernel Sinc Kernel

L2 error

1 ± 0.096

1 ± 0.028

0.9297 ± 0.312

0.8280± 0.025

0.5115 ± 0.098

0.7252± 8.022

N=902 Results are averagerad over 300 experiments and

normalized with regards to SVM performance


Plots of typical results


Road Map



Summary new design of kernel based on frame

elements algorithm for multiscale learning

But no explicit definition of kernel

Time-consuming


Future work

Multidimensional extensionTight Frame of multidimensional wavelet

Using a priori knowledge on the learning problem

How to choose the frame elements?Theoretical justification and analysis

of multiscale approximation

nips 2000 workshop on kernel methods frame, reproducing kernel and learning alain rakotomamonjy...

Documents