1 human brain and models of ann

8/3/2019 1 Human Brain and Models of ANN

1/25

Definitions of Certain Key Terms

Neuron: The basic nerve cell or computing unit for

biologic information processing.Action potential: The pulse of electric potential generatedacross the membrane of a neuron following the applicationof a stimulus greater than the threshold value.Axon: The output node of a neuron that carries the actionpotential to other neurons in the network.Axon hill cock: The starting point of the axon.Dendrite: The input part of the neuron that carries a

temporal summation of action potential to soma.Soma: The cell body of the neuron (that processes theinputs from dendrites).Somatic gain: The parameter that changes the slope of thenon-linear activation function, used in the architecture ofneuron.Synapse: The junction point between the axon of a pre-

synaptic neuron and the dendrite of a post-synaptic neuron.It is the axon-dendrite contact organ.Synaptic and somatic learning: Synaptic learning is thecomponent of the learning that determines the optimumsynaptic weights based on the minimization of certain performance index of error. Somatic learning consists ofthe adaptation of the optimum value of the slope of thenon-linear activation function.


2/25

2

1. Neuro Computing

A human brain consists of approximately11

10 computingelements called neurons. They communicate through aconnection network of axons and synapses, having adensity of approximately 410 synapses per neuron. Thehuman brain is thus a densely connected electricalswitching network, conditioned largely by the biochemicalprocess. The neuron is thus the fundamental building blockof a biological neural network and operates in a chemical

environment. A typical neuron cell has three major regions:the soma (cell body), the axon and the dendrites. Thedendrites form a dendrite tree, which is a very fine bush ofthin fibers around the neuron body. Dentrites receiveinformation from the cell body through axons (long fibersthat serve as transmission lines). An axon is a longcylindrical connection that carries impulses from the

neuron. The end part of the axon splits in to a fineelements, each branch of which terminates in a small end bulb almost touching the dendrites of the neighboringneurons. This axon- dendrite contact is termed as asynapse. The synapse is where the neuron introduces itssignal (in terms of electrical impulses) to the neighboringneuron. Further more the neuron is covered by a thinmembrane.

A neuron will respond to the total of its inputs aggregatedover a short time interval (period of latent summation). Theneuron will respond if the total potential of its membranereaches a certain level. The neurons generate a pulse


3/25

3

response and send it to its axon only under the satisfactionof certain conditions. The incoming impulse may beexcitatory if they cause firing, or inhibitory if they hinder

the firing. The precise condition for firing is that theexcitation should exceed the inhibition by the amountcalled the threshold of a neuron (a typical value for thethreshold is 40 mV.).

The incoming impulses to neuron can only be generated bythe neighboring neurons or by the neuron itself (byfeedback). Usually a certain number of impulses are

required for a neuron to fire. Impulses that are closelyspaced in time and arrive synchronously are more likely tocause a neuron to fire. Observations showed that biologicalneural networks perform temporal integration andsummation of electrical signals. The resulting spatio-temporal processing performed by the biological neuralnetworks is a complex process and is less structured than in

digital computations. Furthermore the electrical impulsesare not synchronized in time as opposed to the synchronousdiscipline of digital computation. One importantcharacteristic feature 0f the biological neuron is that themagnitude of the signal generated does not differsignificantly. The signal in the nerve fiber is either absentor has a maximum value. This means that the informationis transmitted between the nerve cells in the form of binarysignals.

After carrying a pulse, an axon fiber undergoes a state ofcomplete inactivity for a certain time called the refractoryperiod. For this time interval the nerve does not conduct


4/25

4

any signals, regardless of the intensity of excitations. Therefractory period is not uniform over the cells. The timeunits for modeling biological neurons may be of the order

of milliseconds. Also there are different types of neuronsand different ways in which they are connected.

Now understand that we are dealing with a dense networkof interconnected neurons that release asynchronoussignals, which are not only fed forward to the neighboringneurons but also fed back to the generating neuron itself.Thus the picture of the real phenomena in the biological

neural network becomes involved.

The brain is a highly nonlinear, complex, and parallel,information processing system. Human brain has the abilityto arrange its structural constituents (neurons) to performcertain operations like pattern recognition, perception andmotor control, many times faster than the fastest computer

available today. In what follows an example of suchoperation by human brain is explained.

Consider the human vision which is an information processing task. The visual system continuously gives therepresentation of the environment around us and supply theinformation needed to react to it. The human brainroutinely accomplishes these perceptual recognition tasksin approximately 100-200 msec. A digital computer willtake days to perform a much less complex task. Considerfor example, the sonar of a bat, which is an active echorecognition system. The bat sonar gives information likehow far away the target are, the relative velocity of the


5/25

5

target, the size of the target, the size of the various featuresof the target, and the azimuth & elevation of the target.

The vestibule-ocular reflex (VOR) system is a part of thevision operations performed by the human eye and thebrain. The function of the VOR is to maintain the stabilityof the retinal image by making the eye rotations opposite tothe head rotations. There are pre-motor neurons and motorneurons which carry out any muscle movement. The pre-motor neurons in the vestibular nuclei receives and processhead rotation (inputs) signals and sends the results to the

eye muscle motor neurons responsible for eye rotations.Since the above input and output signals are well defined itis possible to modal such a vestibule-ocular reflex (VOR)system.

In what follows two questions are asked.

1.1 Why Neurons are very slow?

1.The axon is a long insulated conductor. It is a fewmicrons in diameter filled with a much poorerconductor than copper, even a few millimeters willhave high resistance.

2. No insulation is perfect. Some current will leakthrough the membrane

3.A cell membrane is an insulating sheet tens of anngstroms thick with conductors on both sides. Themembrane material has a high dielectric constant. Sowe should expect a large membrane capacity (a typicalvalue would be 1 Q F per 2cm ).


6/25

6

Now, the time constant which is proportional to the productof the resistance and capacitance is also high.

1.2 Why the action potential is all-or-none?

A neuron will respond to the total of its inputs aggregatedover a short time interval (period of latent summation). Theneuron will respond if the total potential of its membranereaches a certain level. The neurons generate a pulseresponse and send it to its axon only under the satisfaction

of certain conditions. The incoming impulse may beexcitatory if they cause firing, or inhibitory if they hinderthe firing. The precise condition for firing is that theexcitation should exceed the inhibition by the amountcalled the threshold of a neuron (a typical value for thethreshold is 40 mV.).

1.3 Computation by human brain

We may have the complete knowledge of the neuralarchitecture and arrangements, yet the characterisation ofthe high-level computation of the human mind remains amystery. This is because the electro chemical transmissionof signals and the adjustments of the synaptic (connection

weights) are involved and it is complex. This paradoxicalsituation of human mind can be roughly explained asfollows:


7/25

7

Imagine connecting a logic analiser to a working CPU witha completely known and well documented architecture. Letall the signal flow from the logic analyzer to the CPU and

from CPU to the logic analyzer is known and isdocumented and analyzed. The knowledge of this activityin the micro level is insufficient to explain the computationthat is taking place in the macro level.

Note, however, that the primary purpose, application, andobjective of the human brain is survival. The time-evolved performance of human intelligence reflects an attempt to

optimize this objective. The distinguishing characteristicsdoes not, however, reduce our interest in biologicalcomputation since,

1.The brain integrates and stores experiences, whichcould be previous classification or associations ofinput data. In this sense it self organizes experience.

2.The brain considers new experiences in the context ofstored experiences.

3.The brain is able to make accurate predictions aboutnew situations on the basis of previously self

organized experiences.4.The brain does not require perfect information. It is

tolerant of deformations of input patterns orperturbations in input data.

5.The brain seems to have available, perhaps unused,neurons ready for use.

6.The brain does not provide, through microscopic ormacroscopic examination of its activity, much usefulinformation about its operation at high level.


8/25

8

7.The brain tends to cause behavior that homeostatic,meaning in a state of equilibrium (stable) or tendingtowards such a state. This is an interesting feature

found in some recurrent neural networks such as inHopfield and grossberg networks.

1.4 The Artificial Neural Network

The idea of artificial neural network has been motivatedfrom the recognition that the human brain computes inentirely different way from the conventional digitalcomputer. Such a neural network is defined as follows:A neural network is a massively parallel distributed

processor made up of simple processing units, which has a

natural propensity for storing experimental knowledge and

making it available for use. It resembles the brain in two

aspects:

(1) Knowledge is acquired by the network from its

environment through a

Process of learning(2 ) Interneuron connection strengths, called synaptic

weights, are used to store the acquired knowledge

1.5 Representation of knowledge

K

nowledge refers to stored information or modals used bya person or machine to interpret, predict, and appropriatelyrespond to the outside world. The neural network will thuslearn the environment in which it is embedded. Such aknowledge learned is of two kinds:


9/25

9

1.The known world state, or the facts about what is andwhat has been, Known. This kind of knowledge isreferred to as prior information.

2.M

easurements (observations) obtained by usingsensors designed to probe the environment. Thisinformation provides examples to train the neuralnetwork.

The examples may be labeled or unlabelled. In labeledexamples, each example representing an input signal is paired with a target or desired response. Unlabelled

examples consists of different realisations of the inputsignal by itself. The neural network will then acquireknowledge by training using these examples that arelabeled or unlabelled.

The knowledge representation inside the neural network israther complicated. In what follows four rules are explained

which are of common sense in nature.

Rule 1. It is obvious that similar inputs from similarclass usually produce similar representations inside thenetwork and therefore they should be classified asbelonging to the same category.

One usually used measure of similarity is the Euclideandistance. The Euclidean distance between a pair ofvectors ix and jx in the Euclidean space

mR is given by


10/25

10

2/1

1

2

)(

)(

-

!

!

!

m

k jkik

jiji

xx

d xxxx

The similarity between the two inputs is defined as thereciprocal of the Euclidean distance between the twovectors. Lesser the distance more similar the inputs are.

Rule 2. Second rule is just opposite of the first rule.Items to be separated as separate classes should be given

widely different representations in the network.Consequently, the more is the Euclidean distance theinputs are more separate.Rule 3. If a particular feature is important, thenmore number of neurons should be used for therepresentation of that event in the network.Rule 4. Prior informations and invariance should be

built in to the network, so that they need not be learnedand these results in the reduction of the networkarchitecture. The free parameters to be adjusted arereduced and this results in less number of building blocks and less cost. Here we are talking aboutspecialized networks. Biological neural networks arespecialized indeed.

There are no general rules for incorporating prioriformatios and invariance. It is possible to incorporate prior information in to the network architecture byweight sharing and localized connections. Invariance


11/25

11

actually means invariance to transformations. Invarianceto transformations can be achieved(i) by structure

(ii) by training

1.6 Characteristics of Neural network1. Generalization

A neural network derives its computing power due to (i) Itsmassive parallel-distributed structure (ii) Its ability to learn.Thus we train the network using some training examples.The network will give an appropriate response if we give

an example that is not included in the training examplesused for training.2. Nonlinearity

The basic model of a neural network is nonlinear if theactivation function is nonlinear (that is usually the case). Nonlinearity is an important feature, since the underlying physical mechanism is nonlinear. Furthermore the

nonlinearity is distributed trough out the network.3. Adaptation

y A neural network is inherently adaptivey When a neural network is doing a task two features are

involved; space and the timey The training of a neural network is usually done in a

stationary environment

y But the environment will change continuouslyy So a spatiotemporal training is required. The synapticweights of the network (weight space) will changecontinuously


12/25

12

y As a result when the environment changes the trainingexamples as well as the weight space changes.

y This is a continuous process in all animals

y Such a continuous change is also possible in anartificial neural network.y In other words the training process in an artificial

neural network is continuous and the free parametersof the system should continuously adapt to theenvironment

y The question that arises is how often this adaptation

should take place? That depends on the applicationy An unsupervised training will be better thansupervised training, as is the case in human brain?

1.7 Models of a neuron

A neuron is an information-processing unit. A neuralnetwork consists of a number of such units. The figure

shows the modelof a neuron. One can identify three basicingredients of such a neuron model.(i) A set of connecting links called synapses between

the input signals mjxj ,...,2,1; ! and neuron k. Such

synapses are characterised by their synaptic weightsmjwkj ,...,2,1; ! . Note that the subscripts ofw are

kj and not jk, the meaning of which will be clear

when we deal with the back propagation algorithmfor training the neuron.

(ii) An adder which sums up the input signals weighed(multiplied) by their respective synapses.


13/25

13

kmw

1kw

2kw

(iii) It is required to limit the amplitude of the output ofthe neuron to some finite value. The amplitude ofthe output of a neuron may be limited to the range

[0,1] or [-1,1]. This operation is carried out by asquashing function called the nonlinear activationfunction.

1x bias kb

2x kv ky

/ /

mx

The above neuron model also includes an externally

applied bias term kb . The effect of the bias term is toincrease or lower the net input of the activation function asshown in figure.

Induced local field, kv 0"kb

0!kb 0kb

0

Linear combiners output, ku

(.)N


14/25

14

kmw

0kw

1kw

We will describe the neuron k by using the following setof equations:

)v(

)bu(y

xwu

k

kkk

m

1jjkjk

N!

N!

!!

To incorporate the bias term as an input term, the neuron

model may be modified. Accordingly the equations aremodified as

)(

0

kk

m

jjkjk

vy

xwu

N!

! !

0

x kk

bw !0

1x kv ky

/ /

(.)N

mx


15/25

15

1.8 Signal Flow Graph of a Neuron

The signal flow graph of a single neuron is shown in the

figure below. One can identify the source nodes, thecomputation node and the communication links from thefigure.

10 !x

1x

2x kv (.)N ky /

mx

Signal flow graph of a neuron

1.9 Types of Activation function

Three types of activation functions are explained below.

1.Threshold Function:As shown in the figure, we have

u

! 01

00)( k

k

vifvif

vN )(vN 1

kv


16/25

16

u

!

0101

)( kk

vifvif

vN 1

)(vN This type of neuron model is kv known asMcCulloch pits model

-1

2.Piecewise-Linear Function

e

""

u

!

2121

21 2

1

0

1

)(

k

k

k

vif

vifv

vif

vN )(vN

21 2

1 kv

3. Sigmoid Function ( Logistic Function )

The S-shaped sigmoid function is the most commonly usedactivation function.

)v( kN

)kva(k

e1

1)v(

!N

kv


17/25

17

Note:-

1. The sigmoid function is differentiable, where as the

threshold function is not. Differentiability is animportant feature of the neural network theory.2. As 1to0)v(,toa k pNggp , and it reduces to

threshold function.3. The logistic function coined its name from the the

transcendental law of logistic growth. Measured inappropriate units, all growth process are supposed tobe represented by the logistic distribution function

FE!

te1

1)t(F

Where t represents time and FE , are constans.

Another example of the odd sigmoid function which rangesfrom -1 to +1 is the hyperbolic tangent function (thesigmum function) given by the expression

11

2

1

1

2tan)(

!

!

!

aveave

ave

avhvN

This is bipolar continuous activation function between 1

and 1. With sgpa , we have a bipolar hard limitingactivation function with output as either 1 or 1.


18/25

18

1.10 Exercises:

1. Show that the derivative of the logistic function w.r.t v is)]v(1[)v(a)v( NN!Nd

What is the value of this derivative at the origin?

)v(1)v(ae1

ea

ea

1

e1

ea

vd

)v(d)v(

e1

1)v(

va

va

va

2va

va

)va(

NN!

!

!

N!Nd

!N

At 21)v(,0v !N!

Therefore,

`4

a)()1(a)0( 2

1

2

1

!

!Nd

2. Show that the derivative of the tansigmoid function w.r.tv is

)]v(1[)v( 22a N!Nd

What is the value of this derivative at the origin?


19/25

19

2

a)0(

)v(12

a2

avhtan1

2

a

2

avhsec

2

a)v(

2

avhtan)v(

2

22

!Nd

N!

!

!Nd

!N

3. In logistic activation function, the presence of theconstant a has the same effect of multiplying all the inputswith a.

!

!

!N

ii1

ii1

)va(

)xawexp(1

1

)xwaexp(1

1e1

1)v(

4. Show that

(i) A linear neuron may be approximated as a neuronwith sigmoidal activation function with small synapticweights.

( Hint: For small values of x, x1e x } )

(ii) a McCulloch-Pits modal of a neuron may beapproximated as a neuron with sigmoidal activationfunction with large synaptic weights.


20/25

20

(a) What a single neuron can do?

We will Pause an Identification Problem

Consider a dynamic system with m inputsx(i)such that

T

m nxnxnxix )](),(),([)( 21 ..!

Let we do not know anything about the systemother than that it produces a single output d(n),when simulated by the input vector. Thus theexternal behavior of the system is representedby:

_ a..,,,2,1);(),(: pindnx !

Now we will pose the problem:How to design a multiple input-single outputmodal of the dynamic system using a singleneuron (perceptron)

If we assume the neuron is linear (with linearactivation function), the output y(n)is the sameas the induced local field v(n); ie


21/25

21

!

!!m

k

kk nxnwnvny

0

)()()()(

where (n)wk is the m synaptic weightsmeasured at the time n. Now we have the errore(n)=d(n)-y(n). Now the adaptation of thesynaptic weights is straight forward using the

unconstrained optimization techniques likesteepest descent, Newtons method, Gauss-Newtons method etc.

)(0 nx )(1 nw

)(1 nx )(2 nw / 1 )(nd -1/ )(nv )(ny

)(nxj )(nwj )(ne

)(nwm

)(nxm


22/25

22

1.11 Logic operations performed by ANN

Logical AND :

Consider the truth table illustrating an AND gate

1x

1w v y

2x 2w hard limiter

b

5.1,1

1

2

1 !

-

!

-

b

w

w

Logical OR :

Consider the truth table illustrating the OR gate

2x 1x y

0 0 00 1 0

1 0 0

1 1 1


23/25

23

5.0b,1

1

w

w

2

1 !

-

!

-

Note: The implementations of AND and OR logicfunctions differ only by the value of the bias

Complement

1-w ! v y

x hard limiter

5.0!b

Exercises

1. Show the implementation of NAND ad NOR gates.

2. Try the implementation of an XOR

1.12 Memory Cell

2x 1x y

0 0 0

0 1 11 0 1

1 1 1

x y

1 0

0 1


24/25

24

kx kk x!10

kb

A single neuron with single input with both weight and bias

values of unity, computekk

x!1

0 . Such a simple networkthus behaves as a single register cell able retain the inputfor one time period. As a consequence, once a feedbackloop is closed around the neuron as shown in the figure, weobtain a memory cell. An excitatory input of 1 initializesthe firing in the memory cell, and an inhibitory input ofone initializes a non-firing state. The output value, at the

absdence of inputs, is then sustained indefinitely. This is because the output of zero fed back to the input does notcause firing at the next instant while the output of 1 does.

1.13 Network Architectures

1. Single layer feed forward network


25/25

25

10

20

x /

m0

Input layer of Output layer

source nodes of neurons

1. Multi layer feed forward network

10

x 20 /

Input layer of later of layer of output

source nodes hidden neurons neurons

1 human brain and models of ann

Documents