Supervisor: Dr. András Lőrincz

Eötvös Loránd UniversityNeural Information Processing GroupBudapest, Hungary

Independent Subspace Analysis

Barnabás Póczos

MPI, Tübingen, 24 July 2007.

2

Independent Component Analysis

Estimation

y(t)=Wx(t)

Sources Observation

x(t) = As(t)s(t)A

Mixing

3

Some ICA Applications

Blind source separation Image denoising Medical signal processing – fMRI, ECG, EEG Modeling of the hippocampus Modeling of the visual cortex Feature extraction Face recognition Time series analysis Financial applications

4

Independent Subspace Analysis

A 2 Rmd£md W 2Rmd£md

Sources Observation Estimation

s1 2Rd

s2 2Rd

sm 2 Rd

x1 2 Rd

x2 2 Rd

xm 2 Rd

y1 2 Rd

y2 2 Rd

ym 2 Rd

s sT ; : : : ; smT T 2 Rdm x xT ; : : : ; xmT T 2Rdm x As

y yT ; : : : ; ymT T 2Rdm yWx

5

Independent Subspace Analysis The Efforts Cardoso, 1998

Conjecture, ICA preprocessing followed by permutation Akaho et al, 1999

Processing of EEG-fMRI data 2-dimensional Edgeworth-expansion

Bach & Jordan, 2003 kernel methods for mutual information estimation

Theis, 2003 Ambiguity issues, uniqueness of ISA

Hyvärinen & Köster, 2006 FastISA: A fast fixed-point algorithm for independent

subspace analysis

6

Independent Subspace AnalysisThe Ambiguity

Ambiguity of ICA: Sources can be recovered only up to:

arbitrary permutation arbitrary scaling factors sign

Ambiguity of ISA: Sources can be recovered only up to:

arbitrary permutation arbitrary invertible transformation

7

Independent Subspace Analysispairwise independence ≠ joint independence

In ICA case pairwise independence of the sources = joint independence of the sources (Comon, 1994)

In ISA case pairwise independence of the subspaces ≠ joint independence of the subspaces

Proof: Let 3 of 3-dimensional independent sources, where the elements of each subspace are pairwise independent.Than is a wrong ISA solution.

fs; s; sg; fs; s; sg; fs; s; sg

fs; s; sg; fs; s; sg; fs; s; sg

8

The ISA Cost Functions

Mutual Information:

py pydyHy ¡R

Shannon-entropy:

RIy; : : : ;ym py

py¢¢¢pymdy

9

The ISA Cost Functions

Multidimensional Entropy Estimation

11

Multi-dimensional Entropy Estimations, Method of Kozahenko and Leonenko

Then the nearest neighbour entropy estimation:

This estimation is means-square consistent, but not robust.Let us try to use more neighbours!

Hz n

nPj

nkN;j ¡ zjk CE

CE ¡1Re¡t tdt

fz; : : : ; zng n z 2 Rd N;j zj

12

Multi-dimensional Entropy Estimations

Let us apply Rényi’s-entropy for estimating the Shannon-entropy:

H®¡®

R®!H® ¡

R

Let us use - K-nearest neighbors - geodesic spanning trees for estimating the multi-dimensional Rényi’s entropy.

f®zdz

fz fzdz

13

Beadword - Halton - Hammersley Theorem

fz; : : : ; zng n z 2 Rd Nk;j k zj

° d¡ d®

¡®

kn®

nPj

Pv2Nk;j

kv¡zjk°! H®z c

n!1

14

Multi-dimensional Entropy Estimations Using Geodesic Spanning Forests

Build first an Euclidean neighbourhood graph use the edges of the k nearest nodes to each node

Find geodesic spanning forests on this graph (minimal spanning forests of the Euclidean neighbourhood

graph)

zp

15

Euclidean Graphs

Euclidean neighbourhood graph

Weight of minimal (γ-weighted) Euclidean spanning forest:

Where is the set of all γ-weighted Euclidean spanning forests

L°z T2T

Pe2T

kek°

T

fz; : : : ;zng n z 2 Rd

E fe ep; q zp¡ zq 2 Rd;zq 2 Nk;pg

° d¡ d® d°

L°zn® ! H®z c n!1

16

Estimation of the Shannon-entropy

Mutual Information Estimation

18

Kernel covariance (KC) A. Gretton, R. Herbrich, A. Smola, F. Bach, M. Jordan

The calculation of the supremum over function sets is extremely difficult. We can ease it using Reproducing Kernel Hilbert Spaces.

19

RKHS construction for x, y stochastic variables.

20

Kernel covariance (KC)

And what is more, after some calculation we get, that

The ISA Separation Theorem

22

ISA Separation Theorem

23

The ISA Separation Theorem

Numerical Simulations

25

Numerical Simulations 2D Letters (i.i.d.)

Sources ObservationEstimated sources

Performance matrix

26

Numerical Simulations3D Curves (i.i.d.)

Sources ObservationEstimated sources

Performance matrix

27

Numerical SimulationsFacial images (i.i.d.)

Sources ObservationEstimated sources

Performance matrix

28

Numerical Simulations

29

These methods for entropy estimation need i.i.d. processes.

What can we do with τ-order AR sources?

Then the innovations are i.i.d. processes:

and the mixing matrix is the same for the innovations, so we can use ISA on innovations.

sit Fsit ¡ : : : F¿ sit¡ ¿ ¹t

sit sit¡Esitjsit¡ ; sit¡ : : : ¹t

Ast xt ¡Extjxt¡ ; xt ¡ : : : xt

Working on Innovations

30

Results Using Innovations

Performance using innovations

Estimated sourcesby plain ISA

Original ‘AR’ sources

Mixed sources

Estimated sourcesusing ISA on innovations

Performance of plain ISA

31

Undercomplete Blind Subspace Deconvolution

Multi-dimensional generalization of the undercomplete Blind Source Deconvolution (BSD)

32

BSSD reduction to ISA

33

BSSD reduction to ISA

ISA task!

34

BSSD Results

Database:

Convolved mixture Performance Estimation

35

Post nonlinear ISA

Has it any sense?

36

Post nonlinear ISA

Separability theorem:

37

Post nonlinear ISA, results

Original

Observed

Nonlinear functions Hinton diagram Estimated functions

38

ISA for facial components

In our database we had 800 different front view faces with the 6 basic facial expressions. We had thus 4,800 images in total. All images were sized to 40 £ 40 pixel.

A large 4800 £ 1600 data matrix was compiled; rows of this matrix were 1600 dimensional vectors formed by the pixel values of the individual images. The columns of this matrix were considered as mixed signals.

The observed 4800 dimensional signals were compressed by PCA to 60 dimensions and we searched for 4 pieces of ISA subspaces.

39

ISA for facial components

Estimated subspaces

mouth

eye brushes

eyes

facial profiles

40

Ongoing Projects & Future Plans

Multilinear (tenzorial) ISA BSSD in the Fourier domain

EEG, fMRI data processing

Low-dimensional embedding with ICA / ISA constraints Low-dimensional embedding of time series

Variational Bayesian Hidden Markov Factor Analyzer

41

Thanks for your attention!

42

References

Independent Process Analysis without Combinatorial Efforts. Z. Szabó, B. Póczos and A. Lőrincz. (ICA2007, accepted)

Post Nonlinear Independent Subspace Analysis.Z. Szabó, B. Póczos, G. Szirtes and A. Lőrincz. 2007. (ICANN 2007, accepted)

Undercomplete BSSD via Linear Prediction. Z. Szabó, B. Póczos and A. Lőrincz. 2007. (ECML 2007, accepted)

Undercomplete Blind Subspace Deconvolution Z. Szabó, B. Póczos and A. Lőrincz. 2007.Journal of Machine Learning Research. vol. 8, pp. 1063-1095.

Independent subspace analysis using geodesic spanning trees B. Póczos and A. LőrinczProc. of ICML 2005, Bonn, ICML: 673-680

Cross-Entropy Optimization for Independent Process AnalysisZ. Szabó, B. Póczos and A. LőrinczProc. of ICA 2006, Charleston, SC: LNCS 3889, 909-916, Springer Verlag

43

References

Noncombinatorial estimation of independent auto-regressive sourcesB. Póczos and A. Lőrincz 2005.Neurocomputing vol. 69, pp. 2416-2419

Independent Subspace Analysis on Innovations B. Póczos, B. Takács and A. LőrinczProc. of. ECML 2005, Porto, LNAI 3720: 698-706, Springer-Verlag

Independent subspace analysis using k-nearest neighborhood distances B. Póczos and A. LőrinczProc. of ICANN 2005, Warsaw, LNCS 3697: 163-168, Springer-Verlag

Separation theorem for independent subspace analysis with sufficient conditions.Z. Szabó, B. Póczos and A. Lőrincz.Technical report, Eötvös Loránd University, Budapest.

Cost component analysisAndrás Lőrincz and Barnabás Póczos 2003.International Journal of Neural Systems, Vol. 13, pp. 183-192.

44