to be is the only way to know vishwanathan mohan buddhaa ... · e w ij v i v j g v dv iv i 11 0 1...

Vshuu/Work/Genova/PPTW_gen 14/05/051

To be is the only way to knowBuddhaa

To be is the only way to knowBuddhaa

Vishwanathan Mohan

PhD Student, IIT


CHNNElectronic Model

DynamicsEnergetic CostsIEEE MWSCAS

AHNNOnline Adaptation

Dynamics CHAR Recog

JACII

Link Between Energy

And Computation

NIPS 2001

HOPFIELDNEURAL

NETWORK(HNN)

Spike Based

Neuromorphic Designs

SOMSOM

Multilevel Activation based HNN

6 Level AVLSI CKT

Pattern Recognition

Phantom Limbs

Pattern Recognition

Phantom Limbs


Low Power Chip DesignLow Power Chip Design

Device levelDevice level System levelSystem level

Obvious culpritsObvious culprits

Computer as a Computer as a

Thermodynamic EngineThermodynamic Engine

A B Out0 0 00 1 01 0 01 1 1

A B Out0 0 00 1 01 0 01 1 1

AND GateAND Gate Conventional Digital Designs

are Logically Irreversible

Physical Irreversibility

dU = dQ +dW

Conservative Logic

Dynamic Charge Recovery Logic

Electronic Letters

Jour. Ckts and Sys, 2004


A model of associative memory.Fully connected net of MP neuronsDissipative dynamics: has Lyapunov fn.Dynamics can be modeled into an electronic equivalent circuit.

Hopfield Neural NetworkHopfield Neural Network


Hpofeild Neuarl Netwrok

(W (i ,j)= W (j, i))(W (i ,i)= 0)

)( ii ugV λ=Vi = state of i’th neuron;

Wij = strength of connection between

Output of Jth neuron to inputOf Ith neuron;

g(i) = sigmoid function;

∑=

+−=N

jjiji

i Vwudtdu

1τ

Ui


a1 = [1 -1 1 -1]

∑=

=P

p

pj

piij NSSw

1/

0 -1 1 -1-1 0 -1 11 -1 0 -1

-1 1 -1 0

Equivalent ResistancesStorage Algorithm

One Shot

Retrieval

Network’s initial state, V(0), is a noisy version of a stored pattern, Sp. Noise is added by flipping a fixed fraction (15%) of bits in Sp. Ideally, for V(0) close to Sp, the final state, Vfinal, of the network must equal Sp.

Sequence of states in a recurrent network is Non deterministic .In addition there can be many equilibrium states that can be potentially reached after many such nondeterministic amount of transitions.

HNN as Associative Memory


Does a solution exist ? If it does ,will the system converge in finite time ? What is the solution.

“ If a bounded function of state variables of a dynamical system can be found, such that all state changes result in a decrease in the value of the function , then the system has a stable solution”.

∑ ∑∫∑∑= =

−

= =

−+−=N

i

N

iii

VN

i

N

jjiij VIdVVgVVwE i

1 10

1

1 1)(1

21

λ

0))(('

)(

2

11

1 1

≤−=−=

+−−=

∑∑

∑ ∑

==

= =

dtdu

ugdt

dVdtdu

dtdV

IuVwdtdE

iN

ii

iN

i

i

iN

i

N

jiijij


∑=

+−=N

jjiji

i Vwudtdu

1τ

)( ii ugV λ=

0)(

1=

−++ ∑

=

N

j ij

jiji

t

ii

RVu

Ru

dtdu

Cγ

⎟⎟⎠

⎞⎜⎜⎝

⎛+

=

∑=

N

j ijt RR

C

1

11τ

⎟⎟⎠

⎞⎜⎜⎝

⎛+

=

∑=

N

j ijt

ijij

ij

RR

Rw

1

11

)1(γ

⎟⎟⎠

⎞⎜⎜⎝

⎛−

=

∑=

N

jijt

ij

ij wR

w

R

11

1

Taking modulus on both sides,

⎟⎟⎠

⎞⎜⎜⎝

⎛+

=

∑=

N

j ijt

ijij

RR

Rw

1

11

1

Taking summation over j,

⎟⎟⎠

⎞⎜⎜⎝

⎛+

=

∑

∑∑

=

=

= N

j ijt

N

j ijN

jij

RR

Rw

1

1

1 11

1

Rij, expressing it in termsof network weights.

⎟⎟⎠

⎞⎜⎜⎝

⎛−

=

∑=

N

jijt

ij

ij wR

w

R

11

1

∑∑==

+⎟⎟⎠

⎞⎜⎜⎝

⎛+−=

N

j ij

jijN

j ijti

i

Rv

RRu

dtdu

C11

11 γ


∑∑==

+⎟⎟⎠

⎞⎜⎜⎝

⎛+−=

N

j ij

jijN

j ijti

i

Rv

RRu

dtdu

C11

11 γ

∑=

+−=N

jjiji

i Vwudtdu

1τ

Visual C++Visual C++

Foreign NetlistForeign Netlist

SCAD (Linear Technology)SCAD (Linear Technology)


Is there a relationship between: 1) HNN performance as an associative memory and 2) the concomitant energy dissipation in the electronic model?

‘Link between energy and computation in a physical model of Hopfield Network’ VS Chakravarthy, Vishwanathan Mohan, Uday Shankar NIPS 2001, Singapore.

1) L .Sokoloff, Metabolic Probes of Central Nervous SystemActivity in Experimental Animals and Man, Sunderland, MA:Sinauer Associates, 1984.2)J.G.McElligott and R Melzack, “Localized Thermal Changesevoked in the brain by Visual and Auditory Stimulation,”Experimental Neurology, vol. 17, pg 293-312, 1967


In the brain there is a tight coupling between neural activity, local

cerebral blood flow, and local glucose metabolism .

Our experiments on possibility of such a link in neural network

models show a consistent correlation between Energy dissipated and

performance of Hopfield Neural Network.

IS the brain exploiting some relationship between computation and

energy dissipation?

If there is a minimal cost of computation ,has the brain evolved so

as to optimize its energy expenditure in order to perform its

functions ?

Irreversible computational process have inevitable energetic costs

Rolf Launder (1961), IBM Joun. Of Research


COMPLEX HOPFIELD NEURAL NETWORKCOMPLEX HOPFIELD NEURAL NETWORK : : OverviewOverview

Extension of Hopfield model from real number to Complex numberExtension of Hopfield model from real number to Complex numberdomain .domain .

Interesting as real Neural systems also exhibit a range of Interesting as real Neural systems also exhibit a range of phenomenon like oscillations ,Phase lock and Chaos seldom touchephenomenon like oscillations ,Phase lock and Chaos seldom touched d by dissipative models with fixed point dynamics.by dissipative models with fixed point dynamics.

Exhibits both Fixed point and Oscillatory behavior controlled bExhibits both Fixed point and Oscillatory behavior controlled by a y a continuous mode parameter continuous mode parameter -- ‘‘٧٧’’

Can be used as associative memoryCan be used as associative memory

At one extreme network has conservative dynamics and at other At one extreme network has conservative dynamics and at other extreme , dynamics are dissipative and governed by Lyapunov extreme , dynamics are dissipative and governed by Lyapunov function.function.

Work with mammalian olfactory cortex revealed that odors are stored as oscillatory states.

Freeman (1987)


COMPLEX HOPFIELD NEURAL NETWORKCOMPLEX HOPFIELD NEURAL NETWORK : : DynamicsDynamics

ZJ : Complex variable representing state of Jth

Neuron.

Vj : Output of Jth Neuron

TJK :Strength of connection between Jth and Kth neurons

α and β are complex coefficients such that , β = γ + i (1 – γ ) and α = λ βWhere λ is a large positive number, g(.) is Tanh activation function

jjkjktZj IZVT

k+−∑=∂

∂ βτa

)( *jj ZgV α=


Vishwanathan Mohan and VS Chakravarthy .,”Inevitable Energy Costs of Storage Capacity Enhancement in an Oscillatory Neural Network,” IEEE Circuits and Syst, 2003

When the oscillations in CHNN When the oscillations in CHNN stabilize outputs of individual neurons stabilize outputs of individual neurons attain fixed phase relationships. attain fixed phase relationships.

Situation changes dramatically Situation changes dramatically when 2 or more patterns are stored. when 2 or more patterns are stored. Instead of oscillating between a Instead of oscillating between a specific pattern and its negative, the specific pattern and its negative, the network wanders chaotically from one network wanders chaotically from one stored pattern to another.stored pattern to another.

When the same circuit is When the same circuit is operated in static mode however, operated in static mode however, We are able to successfully store We are able to successfully store and recall much more patterns in and recall much more patterns in the presence of noise (0.14, the presence of noise (0.14, Hopfield)Hopfield)

Interesting thing is that the Interesting thing is that the same electronic circuit also same electronic circuit also dissipates much higher energy to dissipates much higher energy to do so!do so!


Adaptive HNN (Online Hebbian adaptation during retrieval phase at a slower timescale)

•Two phases: Storage phase and retrieval phase•In connectionist networks weights are frozen once the training is completed and network is used for retrieval •In AHNN weights also change during retrieval, Hence losing information during retrieval

( , , )ijij i j

dw F w V Vdt

=


w

ccw R

V VC−

= jiVVdt

d

jiVVdtd

+−= cc

ww VV

CR

cij vW ∝

( )TGSiDSij VVKW

LRR−

≅=

ijij R

W 1∝

ijij i j

dw w V Vdt

ρ α= − +


Comparative retrieval performance of the HNN and AHNN

Vishwanathan Mohan, , Joshi, Y.V, Anand, I., “Studies on an Electronic analog of a Recurrent Neural Network with retrieval Phase weight adaptations,” Journal of Advanced Computational Intelligence and Intelligent Informatics ,Editors L A Zadeh, T.Fukuda


SCIS 2004, Yokohama


Multi Level AMSpike Based Electronic Models


Fixed PointHNN

AHNNFixed Point

HNN + Learning

GHNNMulti Level Associative

Memory

CHNNOscillatoryAssociative

Memory

Spike Based

Neural Computation


Self Organizing MapsSelf Organizing Maps


900 Neurons Competing, Cooperating, and Adapting

Inspired by ‘PHANTOMS IN THE BRAIN’


A Glance at NatureA Glance at NatureWhen When the only the only tool you tool you have is a have is a Hammer Hammer ,Every ,Every problem problem begins begins and ends and ends with a with a Nail.Nail.

Pentium IV: 108 Transistors100W /CM2

Pentium 4: 10Pentium 4: 108 8 TranTran100W/cm100W/cm22


Vishwanathan Mohan, Joge.A. ‘Efficient Datapath Design using 6*6 Conservative reversible gates’, Journal of Circuits and Systems (World Scientific)


Dynamic Charge Recovery Logic

Vishwanathan Mohan, ‘Dynamic Charge Recovery Logic’ Submitted to the Journal: Electronic Letters, Dec 2005

CL=0.1pf

0.1 Micron NWELL Process

Dev density :

CMOS : 2N SCRL:4N+4 DCRL: 2N+8

Power : 90% CMOS and 33% SCRL


Vish to acknowledge the help, suggestions and encouragement provided by :

Govt of India (MHRD, AICTE)

Dr. M.Frank (FSU), Dr. Hiroshi Tsujimo (HRI),

Dr. V.S.Chakravarthy (IITM), M.J.Patil (Texas Inst, USA)

Dr.Y.V.Joshi (SGGSIE&T), GowriShankar (SHARP SD Labs )

Family, Friends……………………………………………………………………………

………………………………. Billions of Stars and the Moon……………..

www.geocities.com/vish_m2000/VISHUU


Mathematical AppendixMathematical Appendix

Lemma 1. If Lemma 1. If λλııRe [Z]Re [Z]ıı >> 1 and g(>> 1 and g(λλ z)=tanh(z)=tanh(λλz) ,then z) ,then g(g(λλ z)z)≈≈ tanh(tanh(λλ(Re [Z]))(Re [Z]))


Single Neuron oscillatorSingle Neuron oscillator

Dynamics of single neuron is described byDynamics of single neuron is described by ::

Separating real and imaginary parts and invoking Lemma1 we have,Separating real and imaginary parts and invoking Lemma1 we have,

(a) (a) which is a conservative system of form which is a conservative system of form

and describes motion of a particle in a potential Vand describes motion of a particle in a potential Vpot pot given bygiven by

))log(cosh(21)( 2 ywyyfVpot λ

λ+=−= ∫

ForFor γγ=0 the resistors disappear and =0 the resistors disappear and circuit conserves energy. Applying circuit conserves energy. Applying KCL at node y gives,KCL at node y gives,

Same as (a)Same as (a)

Figure 1 : Electrical Analog of Neuron near Figure 1 : Electrical Analog of Neuron near γγ > 0> 0


viHow do living beings extend their genetically predetermined set of actions and percepts with new ones, thus responding effectively to anever-changing, potentially hostile environment? What is really going on in their nervous systems while they sense, act and adapt –in other words, while their intelligence builds up?Medium is the message

Marshal Mc Luhan (1948)

to be is the only way to know vishwanathan mohan buddhaa ... · e w ij v i v j g v dv iv i 11 0 1...

Documents