Slide 1

Multi-DimensionalNeural Networks:

Unified Theory

Garimella RamamurthyAssociate Professor

IIIT-HyderebadIndia

Slide 2

Important Publication

• Book based on • my MASTERPIECE

• Title: “Multi-Dimensional Neural • Networks: Unified Theory”• Publisher: New Age International • Publishers

Slide 3

1. One-dimensional neural networks• McCulloch – Pitts• Perceptron (single layer)• XOR problem• Multi-layer Perceptron• Hopfield neural network

2. One-dimensional neural networks• Boolean logic theory

3. One-dimensional neural networks• Coding theory

4. One-dimensional neural networks• Control theory

Outline of the Talk

Slide 4

1. Multi-dimensional logic theory

• Neural networks

2. Multi-dimensional coding theory

• Neural networks

3. Tensor state space representation

4. Multi-dimensional control theory

• Neural networks

5. Unified theory in multi-dimensions

6. Implications

Slide 5

1. One-Dimensional Neural NetworksGoal: To synthesize units which mimic the functions of biological neurons• McCulloch – Pitts Neuron:

Limitation:- Doesn’t mimic all the functions of a biological neuron.- No training.

1

N

j jj

x w=

∑y = Sign

y

Artificialneuron

x1x2

xN-1

xN

wN-1

wN

...

w1

w2

Slide 6

• Perceptron (Rosenblatt)– Incorporation of training by variation of weights, i.e.,

wj(n+1) = wj(n) + η xj ( t – o ); t…target output– o….actual output– Convergence of weights using the learning law.

neuron1

neuron2

...

x1

x2

xN

w11

w21

wN2

wN1

w22

w12

i.e., the input pattern space is divided Into several regions using hyperplanes.

Slide 7

• Minsky:

i.e., it is concluded that XOR gate cannot be

synthesized using a single layer Perceptron.

XOR problem

(1,1)(0,1)XOR gate

x2

x1(1,0)(0,0)

Slide 8

• Multi-Layer Perceptron:

• Hornik – Stinchcombe – White Theorem. i.e., even non-linearly separable patterns can be classified using a multi-layer Perceptron, i.e., XOR problem disappears.

• Functional Link Networks: patterns not linearly separable in a lower dimensional space can be linearly separable in a higher dimensional space.

.

.

.

.

.

.

.

.

.

Input layer Hidden layer Output layer

Slide 9

• Hopfield/Amari Neural Network:

G(V ,E ) : Fully connected undirected graph

- Let there be N nodes/neurons ( )V N=

11

N

i ij j ij

V ( n ) Sign W V ( n ) T=

+ = −

∑

- Modes of operation:- serial mode- parallel mode

- Convergence theorem:- In the serial mode network always converges starting in an initial state.- In parallel mode convergence or cycle of length 2 appears.

- Energy function is non-decreasing.

Slide 10

Control, Communication andComputation Theories

• Control Theory : Move a system from • one point in state space to • another point such that certain• objective function is optimized • Communication Theory: Convey a

message from one point in space to another point reliably.

Slide 11

Control,…Computation Theories: Neural Nets

• Computation Theory: Process a • set of input symbols and produce• a set of output symbols based • on some information processing • operation• Question: Are the above • theories related to the THEORY• of NEURAL NETWORKS

Slide 12

2. One-Dimensional Neural Networks: Boolean logic theory

• Logic Gates: AND, OR, NOT, NAND, NOR, XOR (2 inputs or multiple inputs).

• Relationship between neural networks and logic gates: (Chakradhar, et al.).– Given a logic gate, there exists a neural network such

that the inputs are mapped to stable states, (which constitute the logic gate outputs).

i.e., A Hopfield network can be utilized to synthesize a logic gate.

Slide 13

1. One-Dimensional Neural Networks: Coding theory

• Given a Hopfield Neural Network (optimizing quadratic energy function), there exists a graph-theoretic code (encoder) such that the input information vector is mapped to a codeword (and vice versa).

• Give a generalized neural network (optimizing an energy function higher than a quadratic form), there exists a linear code (encoder) such that every stable state (mapping an input information vector) corresponds to a codeword (and vice versa).

• Also maximum likelihood decoding corresponds to determining the global optimum of energy function.

Slide 14

4. One-Dimensional Neural Networks: Control theory

• Problem Statement: Given a linear time varying system, determine a finite sequence of input values (bounded in magnitude by unity). Such that the total output energy over a finite horizon is as maximum as possible.

• Solution: (Using Pontriyagin’s maximum principle). The optimal input vector constitutes the stable state of a Hopfield neural network, i.e.,

where R … Energy Density Matrix of Linear System.

u Sign( Ru )=

Slide 15

5. Multi-Dimensional Logic Theory: Neural Networks

• Q: How can logic gate definition be extended to a two-dimensional array of 0’s and 1’s?

Some possibilities in the case of a matrix of 0’s and 1’s.

• Such an approach is heuristic and the logic gate definition is non-unique.

• Difficulties are compounded for a multi-dimensional logic gate definition

Generalization of one-dimensional definition.

Slide 16

• Energy Function: Quadratic or higher degree form.

Multi-dimensionalLogic gate

Multi-dimensionalNeural network

m-d logicgate

outputs

m-darray

ofinputs

m-d array

ofinputs

m-d neural network

stable state

Slide 17

6. Multi-Dimensional Coding Theory: Neural Networks

• Multi-dimensional Hopfield neural network. Maximization of energy function (quadratic form) defined on the m-d hypercube. Encoder of a multi-dimensional graph theoretic code.(Application: Multi-dimensional associative memory.)

• Multi-dimensional generalized neural network. Maximization of higher degree energy function on the multi-dimensional bounded lattice. Encoder of a multi-dimensional linear/non-linear code

• Stable states m-d

codewords.

Slide 18

7. Tensor State Space Representation: Multi-dimensional system theory

• One-dimensional linear systems.– State space representation (Kalman)

• Earlier efforts in 2-dimensional system theory: (applications in image processing, etc.).– No notion of causality ⇒ quarter plane and half plane

causality.

Slide 19

Main Contribution:

Tensor state space representation

1 2 1 2 1 2

1 1 1 1

1 1 1 1

1 1

i ,i , ,in; j , j , , jn j , j , , jn

i , ,in; j , jn j , , jn i , in

j , jn i , ,in; j , , jn j , , jn

j , , jn j , , jn

A X ( t )

B U ( t ) X ( t )

Y ( t ) C X ( t )

D U ( t )

⊗

+ ⊗ =

= ⊗

+ ⊗

˙

Slide 20

8. Multi-Dimensional Control Theory: Neural networks• Problem formulation: Given a multi-dimensional linear

system with TSSR, with input tensors subjected to a bounded amplitude constraint, maximize the TOTAL output energy over a FINITE horizon.

Solution: The optimal control tensors constitute the stable states of a multi-dimensional Hopfield neural network.

⇒ concepts such as controllability, observability, stability are generalized from one-dimension to multi-dimensions with TSSR.

Slide 21

9. Unified Theory of Control: Communication and computation• Mathematical cybernetics:

Optimal control tensors, optimal codeword tensors, optimal switching function tensorsconstitute the stable states of generalizedmulti-dimensional neural networks.

optimalcontroltensors

optimalm-d logicgate outputtensors

optimal codeword tensors

stablestates of

m-dneural networks

Slide 22

10. Implications:

• Applications:– 3-d/m-d associative memory.

– Powerful robots.

• Theoretical implications:– Relating work on topological groups and maximum

principle.

– Functional analysis.

– Infinite dimensional generalization.