linear algebra · 2020. 11. 30. · linear algebra iwe begin with recollections from the theory of...

Linear Algebra

I We begin with recollections from the theory of linear algebra ⇒ Fields. Vector Spaces. Algebras

I Recast linear information processing as operators in the algebra of endomorphisms of a vector space

1

Fields

Field (“Definition”)

A field F is a set where a sum and a multiplication are defined

I Define numbers and the operations we perform on them ⇒ Reals F = R. Complexes F = C

I There are two operations defined ⇒ The sum and the product

2

Fields

Field (Definition)

A field F is a set with two binary operations: Addition (+) and multiplication (·). Such that:

⇒ Both are associative ⇒ α + (β + γ) = (α + β) + γ, and α · (β · γ) = (α · β) · γ

⇒ Both are commutative ⇒ α + β = β + α, and α · β = β · α

⇒ Both have identity elements ⇒ α + 0 = α, and α · 1 = α

⇒ Additive inverse ⇒ For all α ∈ F , there exists −α such that α + (−α) = 0

⇒ Multiplicative inverse ⇒ For all α ∈ F , α 6= 0, there exists α−1 such that α · (α−1) = 1

⇒ Distributive property ⇒ α · (β + γ) = (α · β) + (α · γ)

2

Vector Spaces

Vector Space (“Definition”)

A vector space M over the field F is a set whose elements can be added together and can be also

multiplied by elements of the field F . A set of arrows.

I Define the signals we want to process ⇒ Vectors in Rn. Functions in L2([0, 1]). Sequences

I In addition to the field operations we add two more operations

⇒ The addition of signals and the multiplication of signals by scalars

3

Vector Spaces

Vector Space (Definition)

A vector space M over the field F is a set with two operations: Vector addition (+) and scalar

multiplication (×). Such that

⇒ Vector addition is associative ⇒ x + (y + z) = (x + y) + z

⇒ Vector addition is commutative ⇒ x + y = y + x

⇒ Vector addition has an identity element ⇒ x + 0 = x

⇒ Vector addition has an inverse ⇒ For all x ∈ M, there exists −x such that x + (−x) = 0

3

Vector Spaces

Vector Space (Definition)

A vector space M over the field F is a set with two operations: Vector addition (+) and scalar

multiplication (×). Such that

⇒ Scalar and field multiplication are compatible ⇒ α× (β × x) = (α · β)× x

⇒ Scalar multiplication has an identity element ⇒ 1× x = x

⇒ Distributive property w.r.t vector addition ⇒ α× (x + y) = (α× x) + (α× y)

⇒ Distributive property w.r.t field addition ⇒ (α + β)× x = (α× x) + (β × x)

3

Associative Algebras

Associative Algebra (Definition)

An associative algebra A is a vector space with a bilinear map A×A→ A mapping (a, b)→ a ∗ b

and such that (a ∗ b) ∗ c = a ∗ (b ∗ c).

I An algebra with unity is one with an identity element 1 such that 1 ∗ a = a ∗ 1 = a

I The algebra is commutative if for all a, b ∈ A we have a ∗ b = b ∗ a

I Add a fifth operation to define the linear transformation of a signal ⇒ Endomorphisms

4

Signals and Vector Spaces

I Signals are the entities we want to process ⇒ They are elements x of a vector space M

⇒ We can add two signals ⇒ y = x1 + x2

⇒ We can scale signals with elements of a field ⇒ y = α× x

I Set of vectors with n components ⇒ M = Rn ⇒ Graph signals. Or discrete signals

I Set of finite energy functions in [0, 1] ⇒ M = L2([0, 1]) ⇒ Graphon signals

I Sequences (discrete time). Functions in R (continuous time). Sequences with two indexes (images)

5

Endomorphisms

I An endomorphism e is a linear map from the vector space M into itself

e(α1 × x1 + α2 × x2) = (α1 × ex1) + (α2 × ex2)

I If M = Rn ⇒ Square matrix multiplications ⇒ y = Ex

I If M = L2([0, 1]) ⇒ Linear functionals ⇒ y(u) =

∫ 1

0

E(u, v)x(v) dvM

x ex

End(M)

e

I The space of all the endomorphisms in M is End(M) ⇒ All Matrices. Or all linear functionals

6

The Set of Endomorphisms is a Vector Space

I The set End(M) of endomorphisms of a vector space M is (another) vector space

I The sum operation yields the endomorphism e = e1 + e2 defined as

ex = e1x + e2x

I Scalar multiplication yields the endomorphism e′ = αe defined as

e′x = α× (ex)M

x ex

End(M)

e

I The set of square matrices of given dimension or the set of linear functionals is a vector space

7

The Set of Endomorphisms is an Algebra

I It has more structure, though ⇒ End(M) is also an associative algebra with unity

I Product e = e1 ∗ e2 ⇒ Composition of endomorphisms ⇒ ex = e1(e2x)

I The product of two matrices ⇒ E = E1E2

I Composed functional ⇒ y(w) =

∫ 1

0

E1(w , v)

[∫ 1

0

E2(u, v)x(v)dv

]du

M

x ex

End(M)

e

I Linear Algebra ⇒ Process signals (vectors) by composing linear maps (endomorphisms)

8

Signal Processing in the Algebra of Endomorphisms

I An endomorphism is the set of all linear transformations that can be applied to a signal

I There is no structure in the space of endomorphisms ⇒ Learning in End(M) does not scale

I Introducing structure ≡ Restricting the set of allowable endomorphisms

⇒ We accomplish this with an Algebra and a Representation to define sets of convolutional filters

9

Algebraic Signal Processing

I The algebra of endomorphisms of a vector space does not leverage signal structure

I Convolutional filters use algebras and homomorphisms to restrict allowable linear transformations

10

From Linear Algebra to Signal Processing

I The signals we want to process live in a vector space M

I Linear processing with elements e of the algebra End(M)

I This is too general ⇒ Restrict allowable operations

⇒ To those that represent another (more restrictive) algebra

I Have to define homomorphisms and representationsM

x ex

End(M)

e = ρ(a)

A

a

ρ

11

Homomorphisms Between Algebras

Algebra Homomorphism

Let A and A′ be two algebras. A homomorphism is a map ρ : A→ A′ that preserves the operations

of A. I.e., for all a, b ∈ A we have that

⇒ The homomorphism preserves the sum ⇒ ρ(a+ b) = ρ(a) +′ ρ(b)

⇒ The homomorphism preserves the product ⇒ ρ(a ∗ b) = ρ(a) ∗′ ρ(b)

⇒ The homomorphism preserves the scalar product ⇒ ρ(α× a) = α×′ ρ(a)

I Doing operations on the algebra A is the same as carrying operations on the algebra A′

⇒ The converse need not be true. Algebra A′ may be “richer.” May have more elements

12

Representations and Filters

Representation

Consider an algebra A, a vector space M, and a homomorphism ρ from A to End(M),

ρ : A→ End(M).

The pair (M, ρ) is said to be a representation of the associative algebra A

I Ties the abstract algebra A to concrete operations on signals that live in the vector space M

I We say that a ∈ A is a filter ⇒ The action of a on signal x produces the filtered signal y = ρ(a)x

13

Algebra of Polynomials of a Single Variable

I A polynomial over the field F is an expression of the form ⇒ a =K∑

k=0

aktk

I The coefficients ak are elements of the field F . The sumK∑

k=0

and the powers tk are just symbols.

I The Algebra of polynomials over F is the set of all polynomials along with the operations

Scalar multiplication Vector sum Algebra product

(α× a) =K∑

k=0

(α · ak )tk (a + b) =K∑

k=0

(ak + bk )tk (a ∗ b) =K∑

k=0

[k∑

j=0

aj · bk−j

]tk

14

Graph Signal Processing (GSP)

I Signals x in Vector space M = Rn ⇒ Endomorphisms End(M) = Rn×n (square matrices E)

I Processing with E is too general ⇒ Suppose that x is supported on a graph with shift operator S

I Define the homomorphism ρ from the algebra of polynomials to End(M) = Rn×n in which we map

a =K∑

k=0

aktk → ρ(a) =

K∑k=0

akSk

I Algebra of polynomials + Homomorphism ρ ≡ Graph signal processing on shift operator S

15

Algebraic Signal Processing (ASP)

I An Algebraic SP model is a triplet (A,M, ρ)

I A is an Algebra with unity where filters h ∈ A live

⇒ It defines the rules of convolutional signal processing

I M is a vector space

⇒ The space containing the signals x we want to process

I ρ is a homomorphism from A to the endomorphisms of M

⇒ Instantiates the abstract filter h in the space End(M)

M

x ρ(h)x

End(M)

ρ(h)

A

h

ρ

I Any h ∈ A is a filter which operates on signals according to the homomorphism ⇒ y = ρ(h)x

16

Polynomials in an Algebra and Polynomial Functions

17

Polynomials in an Algebra

I Given an element of an algebra a ∈ A and a set of coefficients hk ∈ F , a polynomial is

pA(a) = h0 × 1 + h1 × a + h2 × (a ∗ a) + h3 × (a ∗ a ∗ a) + . . . =∑k

hk ak

I We know that pA(a) ∈ A because we start from a ∈ A and use algebra operations throughout

I The element pA(a) ∈ A is generated from a ∈ A using the operations of the algebra (×,+, ∗)

18

Polynomials in a Different Algebra

I We use the algebra’s operations ⇒ If we change the algebra, the “same” polynomial is different

I For a′ ∈ A′ the “same” polynomial performs different operations to generate pA′(a′) ∈ A′

pA′ (a′) = h0 ×′ 1 +′ h1 ×′ a +′ h2 ×′ (a ∗′ a) +′ h3 ×′ (a ∗′ a ∗′ a) +′ . . . =

∑k

hk (a′)k

I We use the operations (×′,+′, ∗′) of the algebra A′.

19

Polynomial Functions over a Field

I A related object is the polynomial function over the field F

I Consider given coefficients hk ∈ F and a variable λ ∈ F . The polynomial function pF takes values

pF (λ) = h0 · 1 + h1 · λ + h2 · (λ · λ) + h3 · (λ · λ · λ) + . . . =∑k

hk λk

I It is function of λ ⇒ pF : F → F . Defined in terms of the operations (·,+) of the field F .

I Resemblance to frequency responses is not coincidental

20

Polynomials with Multiple Elements in a Commutative Algebra

I Generalize to multiple elements ⇒ For set of elements A = a1 . . . ar ⊆ A define the polynomial

pA(A) =∑

k1,...kr

hk1,...,kr ak11 . . . akrr

I Associated with the set of coefficients hk1,...,kr ∈ F . Algebra Operations. An element of the algebra

I For the same set of coefficients we define the polynomial function pF (λ1, . . . , λr ) = pF (L)

pF (L) =∑

k1,...kr

hk1,...,krλk11 . . . λkr

r

I A different (simpler) object.. A function with variables λi ∈ F . Operations performed on the field F

21

Generators, Shift Operators, and Frequency Representations

I Algebraic Signal Processing is an abstraction of Convolutional Information Processing

I Three central components ⇒ generators, shift operators, and frequency representations

22

Generators of an Algebra

Definition (Generators)

The set G ⊆ A generates A if all h ∈ A can be represented as polynomials of the elements of G,

h =∑

k1,...kr

hk1,...,kr g1k1 . . . gr

kr = pA(G)

I The elements g ∈ G are the generators of A. And h = pA (G) is the polynomial that generates h

⇒ Filters can be built from the generating set using the operations of the algebra

I Given the algebra, the generators are given ⇒ Filter h is completely specified by its coefficients

23

Generators of the Algebras of Polynomials

I The algebra of polynomials of a single variable t is generated by the polynomial g = t

⇒ Algebra elements are expressions h =∑k

hktk ⇒ They can be generated as h =

∑k

hktk

I Algebra of polynomials of two variables x and y is generated by the polynomials g1 = x and g2 = y

⇒ Algebra elements are expressions h =∑k

hklxky l ⇒ Can be generated as h =

∑k

hklxky l

24

Shift Operators

Definition (Shift Operators)

Let (M, ρ) be a representation of A and G⊆ A a generator set of A. We say S is a shift operator if

S = ρ(g), for some g ∈ G

The set S = {ρ(g), g ∈ G} is the shift operator set of the representation (M, ρ) of algebra A.

I Generators g of Algebra A mapped to shift operators S in the space End(M) of endomorphisms of M

25

Polynomials on Shift Operators

Theorem (Filters as Polynomials on Shift Operators)

Let (M, ρ) be a representation of A with generators gi ∈ G and shift operators Si = ρ(gi ) ∈ S.

The representation ρ(h) of filter h is a polynomial on the shift operator set,

h = pA(G) =∑

k1,...kr

hk1,...,kr g1k1 . . . gr

kr ⇒ ρ(h) = pM(S) =∑

k1,...kr

hk1,...,kr S1k1 . . .Sr

kr

Proof: The theorem is true because the homomorphism ρ preserves the operations of the algebra

ρ(h) = ρ

( ∑k1,...kr

hk1,...,kr g1k1 . . . gr

kr

)=

∑k1,...kr

hk1,...,kr ρ(g1)k1 . . . ρ(gr )kr �

26


I ASP ⇒ Vector space ≡ Signals + Algebra ≡ Filters + Homomorphism ≡ Filter Instantiation.

I In principle, the homomorphism ρ : A→ End(M) has to be

specified for all possible filters h ∈ A.

I In reality, it suffices to specify ρ for the generator set

gi ⇒ Si = ρ(gi )

I Other filters are polynomials on gi ⇒ h = pA(G)

I Which instantiate to polynomials on Si ⇒ ρ(h) = pM(S) M

x

End(M)A

ρ(h)x

ρ(h)h

ρ

27






gi ⇒ Si = ρ(gi )



x

End(M)A

ρ(h1)x

ρ(h1)h1

ρ

ρ(h2)x

ρ(h2)h2

ρ

ρ(h3)x

ρ(h3)h3

ρ

27






gi ⇒ Si = ρ(gi )



x

End(M)A

ρ(g1)x

ρ(g1)g1

ρ

ρ(g2)x

ρ(g2)g2

ρ

27

Graph Signal Processing

I GSP ⇒ Signals in Rn + Algebra of Polynomials + Homomorphism ρ defined by the map

h =K∑

k=0

hktk → ρ(h) =

K∑k=0

hkSk

I Equivalent to the (much) simpler specification of the homomorphism ⇒ ρ(t) = S

⇒ This is possible because the algebra of polynomials is generated by g = t

28

Frequency Representation of an Algebraic Filter

Definition (Frequency Representation)

In an algebra A with generators gi ∈ G we are given the filter h expressed as the polynomial

h =∑

k1,...kr

hk1,...,kr g1k1 . . . gr

kr = pA(G)

The frequency representation of h over the field F 1 is the polynomial function with variables λi ∈ L

pF (L) =∑

k1,...kr

hk1,...,kr λ1k1 . . . λr

kr

I The two polynomials are different creatures ⇒ The frequency representation is a simpler object

1 The field is unspecified in the definition. But unless otherwise noted F refers to the field over which the vector space M is defined

29

Three Polynomials (or More)

I The central components of an ASP model are three different polynomials

⇒ The filter. The filter’s instantiation on the space of endomorphisms The frequency response

I These three polynomials have the same coefficients. They are related. But similar though they look

⇒ They are different objects. They utilize different operations. They have different meanings.

30

Polynomial 1: The Filter

P1: The Filter

I A polynomial on the elements gi of the generator set G of the algebra A

pA(G) =∑

k1,...kr

hk1,...,kr g1k1 . . . gr

kr

I Sum, product, and scalar product are the operations of the algebra A

I The abstract definition of a filter. Untethered to a specific signal model

31

Polynomial 2 (or More): The Instantiation of the Filter

P2: The Instantiation of the Filter in the space of Endomorphisms End(M)

I A polynomial on the elements Si = ρ(gi ) of the shift operator set S

pM(S) =∑

k1,...kr

hk1,...,kr S1k1 . . .Sr

kr

I Sum, product, and scalar product are the operations of the algebra of Endomorphisms of M

I The concrete effect that a filter has on a signal x. Tethered to a specific signal model

I “Or more” ⇒ The same abstract filter can be instantiated in multiple signal models

32

Polynomial 3: The Frequency Response

P3: The Frequency Response

I A polynomial function where the variables λi ∈ L take values on the field F

pF (L) =∑

k1,...kr

hk1,...,kr λ1k1 . . . λr

kr

I Sum and product are the operations of field F . ⇒ E.g, a polynomial with real variables

I Simpler representation of a filter. Untethered to a specific signal model (except for the field)

I The tool we use for analysis. ⇒ To explain discriminability, stability and transferability

33

The Three (or More) Polynomials in GSP

(P1) Abstract filter ⇒ pA(t) =K∑

k=0

hktk . Abstract definition. Untethered to any specific graph

(P2) Filter instantiated on a graph ⇒ pM(S) =K∑

k=0

hkSk . Concrete instantiation. Tethered to S

⇒ On another graph ⇒ pM(S) =K∑

k=0

hk Sk . Concrete instantiation. Tethered to S

⇒ On a graphon ⇒ pM(TW ) =K∑

k=0

hkT(k)W . Concrete instantiation. Tethered to graphon W

(P3) Frequency response ⇒ pF (λ) =K∑

k=0

hkλk . Simple function of one variable. Same for all instances

34

Convolutional Information Processing

I Algebraic filters are a generic abstraction of the common features of convolutional signal processing

I Graph, time, and image convolutions can be expressed as particular cases of algebraic filters

35

Specification of ASP Models

I To specify an ASP model we need to specify

⇒ A vector space M where signals x live

⇒ An algebra A where convolutional filters h live

⇒ A homomorphism ρ mapping filters to End(M)

I The signal x is processed to the output ⇒ y = ρ(h)xM

x

End(M)A

hρ(h)

ρ

ρ(h)x

36

Specification of ASP Models

I To specify an ASP model we need to specify

⇒ A vector space M where signals x live

⇒ An algebra A where convolutional filters h live

⇒ The images S = ρ(g) of the algebra generators g

I The signal x is processed to the output ⇒ y = ρ(h)xM

x

End(M)A

gρ(g)

ρ

36


Task

Process signals x that are supported on a graph with n nodes. A matrix representation of the

graph is given in the matrix S.

1

2

3

4

5

6

7

8

9

10

11x1

x2

x3

x4

x5

x6

x7

x8

x9

x10

x11

37


I GSP in the graph S is a particular case of ASP in which

⇒ M = Rn ⇒ Vectors with n components

⇒ A = P(t) ⇒ The algebra of polynomials h =∑k

hktk

⇒ Shift operator ρ(t) = S ⇒ Resulting in filters

ρ(h) = ρ

(∑k

hktk

)=∑k

hkSkRn

x

Rn×nP(t)

∑hk t

k

∑hkSk

ρ

∑hkSkx

tρ(t) = S

ρ

I Processing x with filter ρ(h) yields output ⇒ y = ρ(h)x = ρ

(∑k

hktk

)x =

∑k

hkSkx

38

Graphon Signal Processing (WSP)

Task

Process signals x supported on a graphon W . The signals have finite energy. I.e, x ∈ L2([0, 1])

0 1 u

x(u)

39


I WSP in the graphon W is a particular case of ASP

⇒ M = L2([0, 1]) ⇒ Finite-energy functions in [0, 1]


hktk

L2([0, 1])

x

End(L2([0, 1]))P(t)

∑hk t

k

∑hkT

kW

ρ

∑hkT

kW x

tρ(t) = TW

ρ

40


I WSP in the graphon W is a particular case of ASP

⇒ Shift operator is ρ(t) = TW defined as

(Twx) (u) =

∫ 1

0

W (u, v)x(v)dv

⇒ This mapping of the generator t yields filters

ρ(h) = ρ

(∑k

hktk

)=∑k

hkTkw

where T kw represents k applications of Tw

L2([0, 1])

x

End(L2([0, 1]))P(t)

∑hk t

k

∑hkT

kW

ρ

∑hkT

kW x

tρ(t) = TW

ρ

I Processing x with filter ρ(h) yields output ⇒ y = ρ(h)x = ρ

(∑k

hktk

)x =

∑k

hkTkW x

40

Discrete Time Signal Processing (DTSP)

Task

Process sequences X with values (X )n = xn for integer indexes n ∈ Z. The sequences have finite

energy. We say that X ∈ L2(Z)

41


I DTSP is a particular case of ASP in which

⇒ M = L2(Z) ⇒ Finite-energy sequences in Z


hktk

L2(Z)

X

End(L2(Z))P(t)

∑hk t

k

∑hkS

k

ρ

∑hkS

kX

tρ(t) = S

ρ

42


I DTSP is a particular case of ASP in which

⇒ Shift operator is a time shift ρ(t) = S such that

(SX )n = (X )n−1

⇒ This mapping of the generator t yields filters

ρ(h) = ρ

(∑k

hktk

)=∑k

hkSk

where Sk represents k applications of SL2(Z)

X

End(L2(Z))P(t)

∑hk t

k

∑hkS

k

ρ

∑hkS

kX

tρ(t) = S

ρ

I Processing X with ρ(h) yields ⇒(Y)n

=(ρ(h)X

)n

=

(∑k

hkSkX

)n

=∑k

hk(X)n−k

42

Image Processing (IP)

Task

Process images, defined as sequences X with values (X )mn = xmn that depend on two integer

indexes m, n ∈ Z. The sequences have finite energy. We say that X ∈ L2(Z2)

43


I IP is a particular case of ASP in which

⇒ M = L2(Z2) ⇒ Finite-energy sequences in Z2

⇒ A=P(x , y) ⇒ Two-letter polynomials h=∑k

hklxky l

L2(Z2)

X

End(L2(Z2))P(x, y)

∑hkl x

ky l∑

hklSkx S

ly

ρ

∑hklS

kx S

lyX

x ρ(x) = Sx

ρ

y

ρ(y) = Sy

ρ

44


I IP is a particular case of ASP in which

⇒ Two shift operators ρ(x) = Sx and ρ(y) = Sy

(SxX )mn = (X )(m−1)n (SyX )mn = (X )m(n−1)

⇒ This mapping of the generators x and y yields filters

ρ(h) = ρ

(∑k

hktk

)=∑k

hklSkx S

ly

Skx or Sk

x represent k or l applications of Sx orSlL2(Z2)

X

End(L2(Z2))P(x, y)

∑hkl x

ky l∑

hklSkx S

ly

ρ

∑hklS

kx S

lyX

x ρ(x) = Sx

ρ

y

ρ(y) = Sy

ρ

I Processing X yields ⇒(Y)mn

=(ρ(h)X

)mn

=

(∑kl

hklSkx S

lyX

)mn

=∑k

hk(X)

(m−k)(n−l)

44

ASP is a Generic Analysis Tool

I Algebraic SP encompasses Graph SP, graphon SP, Time SP, and Image SP as particular cases

⇒ Other particular cases exist. Notably, Group SP

I ASP provides a framework for fundamental analyses that hold for all forms of convolutional filters

45

Algebraic Neural Networks

I We introduce Algebraic Neural Networks (AlgNNs) to generalize neural convolutional networks

46


I Algebraic Neural Networks (AlgNNs) are stacked layered structures

⇒ Each layer consists of the algebraic signal model (A`,M`, ρ`) and σ` ⇒ nonlinearity and pooling

I The output at layer ` in the AlgNN is given by

x` = σ` (ρ`(a`)x`−1) with a` ∈ A`

I The operation is equivalent to

x` = Φ(x`−1,P`,S`)

⇒ P` ⊂ A` represents the properties of the filters and S` is the shifts associated to (M`, ρ`)

47


I Algebraic Neural Network {(A`,M`, ρ`)}3`=1 architecture with three layers.

(A1,M1, ρ1)

(A2,M2, ρ2)

(A3,M3, ρ3)

x

y1 = ρ1(a1) x z1 = η1

[y1

]x1 = P1

[z1

]y1 z1

y2 = ρ2(a2) x1 z2 = η2

[y2

]x2 = P2

[z2

]y2 z2

y3 = ρ3(a3) x2 z3 = η3

[y3

]x3 = P3

[z3

]y3 z3

x1

x1

x2

x2

x3

48

Convolutional Features

I The processing at each layer can be performed by a family of filters

xf` = σ`

(F∑g=1

ρ`(agf` )xg`−1

)

⇒ where agf` is the filter processing gth feature xg`−1 and F` is the number of features

I Layers may use (different) specific algebraic signal models (A`,M`, ρ`)

I Trainable parameters are the filters {afg` }`fg . Numerically, we train directly on ρ`(afg` ).

49

Pooling

I The pooling increases the computational efficiency and improve the performance

I The operation is attributed to the composition operator σ` = P` ◦ η`

⇒ η` is a pointwise nonlinearity and P` is a pooling operator

I The operator σ` is only assumed to be Lipschitz and to have zero as a fixed point σ`(0) = 0

I The pooling operator P` projects elements from a given vector space M` to another M`+1

50

Example 1: CNN

I Traditional CNNs particularize the algebraic signal model in AlgNNs as the typical signal model.

I Make M = CN and A the polynomial algebra in the variable t and module tN − 1

⇒ S = C is the cyclic shift operator satisfying C k = C k mod N .

I The f th feature at layer ` is given by

xf` = σ`

(F∑g=1

K−1∑i=0

hgfi C i

`xg`−1

)

I In this case P` is a sampling operator while η`(u) = max{0, u} is the ReLU.

51

Example 2: GNN

I The GNNs particularize the algebraic signal model in AlgNNs as the graph signal model.

I Let M = CN with components xn of x ∈ M associated with graph nodes

I Let A be the polynomial algebra with elements a =∑K−1

k=0 hktk

⇒ The homomorphism filter is given by ρ(a) =∑K−1

k=0 hkSk

⇒ S ∈ CN×N is the graph matrix representation, e.g., adjacency matrix, Laplacian matrix, etc.

I The f th feature at layer ` is given by

xf` = σ`

(F∑g=1

K−1∑k=0

hgfk Skxg`−1

)

52

Perturbation Models

I We define perturbations in the context of algebraic signal processing

53

Representations and Shift Operators

I To define perturbations (deformations) in ASP we recall the notion of generator of an algebra

I Generators: The set G ⊆ A generates A if all a ∈ A are polynomial functions of elements of G

I Shift Operators: The set S of homomorphism images S = ρ(g) is the set of shift operators

I Definitions of generators and shift operators allows writing filters as polynomials on shift operators

ρ(a) = pM(ρ(G)

)= pM

(S)

= p(S)

54

Perturbations in Algebraic Signal Models

I We define perturbations of Algebraic models as perturbations of shift operators ⇒ S = S + T(S)

I The ASP model (A,M, ρ) is consequently perturbed to the triplet (A,M, ρ) such that

ρ(a) = pM(ρ(g)

)= pM

(S)

That is, the polynomials that define filters are the same. But they use the perturbed shift operator

I Graphs ⇒ Shift operator S represents a graph ⇒ Perturbed operator S represents different graph

I Time ⇒ S represents translation equivariance ⇒ S represents quasi-translation equivariance

55

Perturbation Definition: Remarks

I Our definition limits the perturbation of the homomorphism to the perturbation of the shift operator

⇒ Motivated by practice: seen in graph, discrete time, group and graphon signals

I The model perturbs the homomorphism ρ but not the algebra A ⇒ A define the type of operations

I Notice that a perturbation x = Tx acting on the signal x can be interpreted as a transformation of S

Sx = S(Tx) = (ST )x = Sx

56

Perturbation Model: Deformations

I We will consider a first order generic model of small perturbation for the shift operator(s) S

I It includes an absolute or additive perturbation T0 and a relative or multiplicative perturbation T1S

T(S) = T0 + T1S

I The operators T0 and T1 are compact normal with norms satisfying that ‖T0‖ ≤ 1 and ‖T1‖ ≤ 1

I ‖T0‖ ≤ 1, ‖T1‖ ≤ 1 not strong requirements as our interest is on perturbations with ‖T(S)‖ � 1

57

Example: perturbations of graph filters

I Apply the same filter h to the same signal x on different graphs shift operators S and S

H(S) x =K−1∑k=0

hkSkx H(S) x =K−1∑k=0

hk Skx

I Filter H(S) x ⇒ Coefficients hk . Input signal x. Instantiated on shift S

I Filter H(S) x ⇒ Same Coefficients hk . Same Input signal x. Instantiated on perturbed shift S

I Perturbed model S is the matrix S = T0 + T1S ⇒ T0 additive and T1 relative perturbations

58

Stability Theorems

I We define the notion of stability in the context of algebraic signal processing

I We discuss the stability properties of algebraic filters and algebraic neural networks

59

Stability: recalling basic notions and notation

I In the algebraic signal model (A,M, ρ) filters are defined by the operators ρ(a) ∈ End(M), a ∈ A

I If A is generated by the set G ⊂ A ⇒ a ∈ A can be written as a = p(g) with g ∈ G, p: polynomial

I Any filter H ∈ End(M) is defined by operators p(S) where S = ρ(g) ⇒ Filters are functions of S

I Filters are polynomial functions of the shift operators S ∈ S ⇒ We use denote filters as p(S)

60

Stability

Stable Operators:

We say operator p(S) is stable if there exist constants C0, C1 > 0 such that

∥∥∥p(S)x− p(S)x∥∥∥ ≤ [C0 sup

S∈S‖T(S)‖+ C1 sup

S∈S

∥∥DT(S)∥∥+O

(‖T(S)‖2

)]∥∥x∥∥

for all x ∈M and DT(S) denoting the Frechet derivative of T.

I∥∥∥p(S)x− p(S)x

∥∥∥ is bounded by the size of the deformation. Measured by value and rate of change

I Stability is not a given ⇒ Counter examples in GNN and processing of time signals.

61

Lipschitz and Integral Lipschitz Filters

I Filters are polynomials on shift operators ⇒ Isomorphic to polynomials with complex variables

I Lipschitz Filter: Polynomial p : C→ C is Lipschitz if ‖p(λ)− p(µ)‖ ≤ L0‖λ− µ‖ for some L0

I Integral Lipschitz: Polynomial p : C→ C is Integral Lipschitz if

∥∥∥∥λdp(λ)

dλ

∥∥∥∥ ≤ L1 for some L1

I Restricted attention to algebras with a single generator. Generalizations are cumbersome but ready

62

Stability of Algebraic Filters

Theorem (Stability of Algebraic Filters)

A filter that is Lipschitz and Integral Lipschitz is stable, i.e.

∥∥∥p(S)x− p(S)x∥∥∥ ≤ [(1 + δ)

(L0 sup

S‖T(S)‖+ L1 sup

S‖DT(S)‖

)+O(‖T(S)‖2)

]‖x‖

I Good news ⇒ Algebraic filters can be made stable to perturbations

I Alas, we either have stability or discriminability. Integral Lipschitz Filter ⇒∥∥∥∥λdp(λ)

dλ

∥∥∥∥ ≤ L1

I Commutativity factor affects stability constant but does not generate instability

63

Stability of Algebraic Neural Networks, Single Layer

Theorem (Stability of Algebraic Neural Networks, Single Layer)

Let Φ`(S, x) and Φ`(S, x) be the operators associated with layer ` of an Algebraic NN. If the layer

filters are Lipschitz and Integral Lipschitz,

∥∥∥Φ`(S, x)− Φ`(S, x)∥∥∥ ≤ [(1 + δ)

(L0 sup

S‖T(S)‖+ L1 sup

S‖DT(S)‖

)+O(‖T(S)‖2)

]‖x‖

I Good news ⇒ Algebraic NNs can be made stable to perturbations. It’s the same bound

I Individual layers lose discriminability. Integral Lipschitz Filter ⇒∥∥∥∥λdp(λ)

dλ

∥∥∥∥ ≤ L1

I Nonlinearity mixes frequency components ⇒ Recover discriminability in subsequent layers

64

Stability of Algebraic Neural Networks, Multiple Layers

Theorem (Stability of Algebraic Neural Networks, Multi-Layer)

Let Φ(S, x) and Φ(S, x) be the operators associated with an Algebraic NN on L layers. If the layer

filters are Lipschitz and Integral Lipschitz,

∥∥∥Φ(S, x)− Φ(S, x)∥∥∥ ≤ L

[(1 + δ)

(L0 sup

S‖T(S)‖+ L1 sup

S‖DT(S)‖

)+O(‖T(S)‖2)

]‖x‖

I It is still the same bound ⇒ simply scaled by the number of layers L

65

Stability of Algebraic Neural Networks, Multiple Generators

Theorem (Stability of Algebraic Neural Networks, Multiple Generators)

Let Φ(S, x) and Φ(S, x) be the operators associated with an Algebraic NN with M generators on

L layers. If the layer filters are Lipschitz and Integral Lipschitz,

∥∥∥Φ(S, x)− Φ(S, x)∥∥∥ ≤ ML

[(1 + δ)

(L0 sup

S‖T(S)‖+ L1 sup

S‖DT(S)‖

)+O(‖T(S)‖2)

]‖x‖

I It is still the same bound ⇒ simply scaled by the number of generators M and layers L

66

Spectral Representations

I In this lecture we discuss of notion of spectral decompositions in algebraic signal models

67

Recalling basic concepts

I Recalling algebraic signal model (A,M, ρ) ⇒ A: algebra, M: vector space and ρ: homomorphism

I Where (M, ρ) is a representation of A ⇒ Each representation of A defines an algebraic signal model

I Given (M1, ρ1) and (M2, ρ2) representations of A ⇒ we can define a direct sum (M1 ⊕M2, ρ)

I Where the action of ρ(a) on M1 ⊕M2 is given according to ρ(a)(x1 ⊕ x2) = (ρ1(a)x1 ⊕ ρ2(a)x2)

68

Subrepresentations and Irreducible Representations

Definition (Subrepresentation)

Let (M, ρ) be a representation of A. Then, a representation (U, ρ) of A is a subrepresentation of

(M, ρ) if U ⊆ M and U is invariant under all operators ρ(a) for all a ∈ A, i.e. ρ(a)u ∈ U for all

u ∈ U and a ∈ A.

Definition (Irreducible Representations)

A representation (M, ρ) with M 6= 0 is irreducible or simple if the only subrepresentations of

(M, ρ) are (0, ρ) and (M, ρ).

69

Intertwining Operators

Definition (Intertwining operators)

Let (M1, ρ1) and (M2, ρ2) be two representations of an algebra A. A homomorphism or interwining

operator φ : M1 → M2 is a linear operator which commutes with the action of A, i.e.

φ(ρ1(a)v) = ρ2(a)φ(v),

And, φ is said to be an isomorphism of representations if it is an isomorphism of vectors spaces.

I If (M1, ρ1) and (M2, ρ2) are isomorphic representations we use the notation (M1, ρ1) ∼= (M2, ρ2)

70

Fourier Decompositions

Definition (Fourier decomposition)

For (A,M, ρ) we say that there is a spectral or Fourier decomposition of (M, ρ) if

(M, ρ) ∼=⊕

(Ui ,φi )∈Irr{A}

(Ui , φi )⊕m(Ui ,M)

where m(Ui ,M) indicates the number of irreducible subrepresentations of (M, ρ) that are iso-

morphic to (Ui , φi ). Any signal x ∈ M can be therefore represented by the map ∆ given by

∆ : M →⊕

i U⊕m(Ui ,M)i

x 7→ x

The projection of x in each Ui are the Fourier components represented by x(i).

71


I It is worth pointing out that the Fourier decomposition of any representation is defined by two sums

(M, ρ) ∼=⊕

(Ui ,φi )∈Irr{A}(Ui , φi )⊕m(Ui ,M)

I Non isomorphic subrepresentations (external) and one (internal) for Isomorphic subrepresentations

I⊕

on the frequencies of the representation while⊕

on components associated to a given frequency

I ∆ is an intertwining operator ⇒ ∆(ρ(a)x) = ρ(a)∆(x) ⇒ convolution ρ(a)x = ∆−1(ρ(a)∆(x))

72


I The projection of ρ(a)x on Ui is given by φi (a)x(i) where φi : A→ End(Ui ) is a homomorphism

I The collection of the projections of ρ(a)x on Ui for all i is the spectral representation of ρ(a)x

I The product in φi (a)x(i) translates into different operations depending on the dimension of Ui

I If dim(Ui ) = 1, the operator φi (a) is a scalar while if dim(Ui ) > 1 and finite φi (a) is a matrix

73

Remarks

I The link between A and Fourier decomposition is exclusively given by φi (a) which is acting on x(i)

I Not possible by selecting filters in A to modify the spaces Ui in the spectral decomposition of (M, ρ)

I Two sources of differences between two operators ρ(a) and ρ(a) associated to (M, ρ) and (M, ρ)

I One source of difference can be modified by selecting subsets of A and this is embedded in φi and φi

I Second source of difference ⇒ the difference between the spaces Ui and Ui and cannot be modified

74

Examples

I In CNNs the filtering is defined by the polynomial algebra A = C[t]/(tN − 1), therefore, we have

ρ(a)x =N∑i=1

φi

(K−1∑k=0

hktk

)x(i)ui =

N∑i=1

K−1∑k=0

hk(e−

2πijN

)kx(i)ui ,

I ui (v) = 1√Ne

2πjviN is the ith column vector of the DFT matrix and φi (t) = e−

2πjiN its eigenvalue

I In GNNs the filtering is defined by the polynomial algebra A = C[t], therefore we have

ρ(a)x =N∑i=1

φi

(K−1∑k=0

hktk

)x(i)ui =

N∑i=1

K−1∑k=0

hkλik x(i)ui

I ui is the ith eigenvector of ρ(t) = S, which is the matrix graph, and φi (t) = λi its ith eigenvalue

75

linear algebra · 2020. 11. 30. · linear algebra iwe begin with recollections from the theory of...

Documents