neural network theory neuronal dynamics Ⅰ : activations and signals...
Post on 18-Jan-2018
319 Views
Preview:
DESCRIPTION
TRANSCRIPT
NEURAL NETWORK THEORY
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
欢迎大家提出意见建议! 2003.10.15
2
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
NEURONS AS FUNCTIONS
Neurons behave as functions.Neurons transduce an unbounded input activation x(t) at time t into a bounded output signal S(x(t)).
3
NEURONS AS FUNCTIONSThe transduction description: a sigmoidal or S-shaped curve the logistic signal function:
cxexS
1
1)(
)()(' 001 cScSdxdSS
The logistic signal function is sigmoidal and strictly increases for positive scaling constant c >0.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
4
NEURONS AS FUNCTIONSS(x)
x
0-∞ - + +∞ Fig.1 s(x) is a bounded monotone-nondecreasing function of x
If c→+∞ , we get threshold signal function (dash line),Which is piecewise differentiable
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
5
SIGNAL MONOTONICITY
In general, signal functions are monotone nondecreasing S’>=0. This means signal functions have an upper bound or saturation value.The staircase signal function is a piecewise-differentiableMonotone-nondecreasing signal function.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
6
SIGNAL MONOTONICITYAn important exception: bell-shaped signal function or Gaussian signal functions
02
cexS cx)(
xScxeS cx ',2'2
The sign of the signal-activation derivation s’ is opposite the sign of the activation x. We shall assume signal functions are monotone nondecreasing unless stated otherwise.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
7
SIGNAL MONOTONICITYGeneralized Gaussian signal function define potential or radial basis function :
])(2
1exp[)( 22
n
j
ijj
ii xxS
n
n Rxxx ),,( 1
),,( in
ii 1
input activation vector:
variance:
mean vector:
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
2i
)(xSi
we shall consider only scalar-input signal functions:
)( ii xS
8
SIGNAL MONOTONICITYneurons are nonlinear but not too much so ---- a property as semilinearityLinear signal functions - make computation and analysis comparatively easy - do not suppress noise - linear network are not robustNonlinear signal functions - increases a network’s computational richness - increases a network’s facilitates noise suppression - risks computational and analytical intractability - favors dynamical instability
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
9
SIGNAL MONOTONICITYSignal and activation velocities the signal velocity: =dS/dt
Signal velocities depend explicitly on action velocities
S
xSdtdx
dxdSS '
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
10
BIOLOGICAL ACTIVATIONS AND SIGNALS
Fig.2 Neuron anatomy神经元 (Neuron) 是由细胞核 (cell nucleus) ,细胞体 (soma) ,轴突 (axon) ,树突 (dendrites) 和突触 (synapse) 所构成的
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
11
X= ( x1 , x2 ,…, xn )W= ( w1 , w2 ,…, wn )net=∑xiwinet=XW
x2 w2 ∑ f o=f ( net)
xn wn
…
net=XW
x1 w1
BIOLOGICAL ACTIVATIONS AND SIGNALS
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
12
BIOLOGICAL ACTIVATIONS AND SIGNALSCompetitive Neuronal Signal
11
21
1
cxcx exS
exS )()(
logical signal function ( Binary Bipolar )
The neuron “wins” at time t if , “loses” ifand otherwise possesses a fuzzy win-loss status between 0 an 1.
a. Binary signal functions : [0,1]b. Bipolar signal functions : [-1,1] McCulloch—Pitts (M—P) neurons
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
1))(( txS 1))(( txS
13
NEURON FIELDSNeurons within a field are topologically ordered, often by proximity. zeroth-order topology : lack of topological structure Denotation: , , neural system samples the function m times to generate the associated pairs , ... ,The overall neural network behaves as an adaptive filter and sample data changed network parameters.
XF YF ZF },{ YX FF },,{ ZYX FFF
),( mm yx),( 11 yx
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
pn RRf :YX FF
f
14
NEURONAL DYNAMICAL SYSTEMSDescription:a system of first-order differential or difference equations that govern the time evolution of the neuronal activations or membrane potentialsActivation differential equations: niFFgx YXii ,,),,( 21
piFFhy YXii ,,),,( 21
in vector notation:niFF YXi ,,),,( 21gx
piFF YXi ,,),,( 21hy
(1)(2)
(3)(4)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
15
NEURONAL DYNAMICAL SYSTEMSNeuronal State spaces
nn Rtxtxt ))(,),(()( 1X
pp Rtytyt ))(,),(()( 1Y
So the state space of the entire neuronal dynamical system is: pn RR
Augmentation:
ZYX FFF ]|[pn
z RF
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
},{ YX FF
16
NEURONAL DYNAMICAL SYSTEMSSignal state spaces as hyper-cubesThe signal state of field at time t:
XF
)))((,)),((())(( txStxStXS nXn
X 11
The signal state space: an n-dimensional hypercubeThe unit hypercube : or ,The relationship between hyper-cubes and the fuzzy set : , subsets of correspond to the vertices of
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
nI n]1,0[
pnyx IIFF , pn
yx IFF | py
nx IFIF ,
nxxxX ,,, 21 n2 X n2nI
17
NEURONAL DYNAMICAL SYSTEMSNeuronal activations as short-term memory
Short-term memory(STM) : activationLong-term memory(LTM) : synapse
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
18
1 、 Liner Function S(x) = cx + k , c>0
SIGNAL FUNCTION (ACTIVATION FUNCTION)
x
S
o
k
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
19
2. Ramp Function r if x≥θS(x)= cx if |x|<θ
-r if x≤-θr>0, r is a constant.
SIGNAL FUNCTION (ACTIVATION FUNCTION)
r
-r
θ -θ x
S
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
20
SIGNAL FUNCTION (ACTIVATION FUNCTION) 3 、 threshold linear signal function: a special Ramp Function
Another form:elsecxifcxif
cxxS 0
101
)(
)),max(,min()( cxxS 01
0cS '
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
21
SIGNAL FUNCTION (ACTIVATION FUNCTION) 4 、 logistic signal function:
xcxc
xc
cx
ee
ee
xS22
2
11)(
Where c>0.01 )(' ScSS
So the logistic signal function is monotone increasing.
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
22
SIGNAL FUNCTION (ACTIVATION FUNCTION) 5 、 threshold signal function:
Where T is an arbitrary real-valued threshold,and k indicates the discrete time step.
TxifTxifTxif
xSxSk
k
k
kk
1
1
1
1
0)(
1)(
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
23
SIGNAL FUNCTION (ACTIVATION FUNCTION) 6 、 hyperbolic-tangent signal function:
Another form:
)tanh()( cxxS
01 2 )(' ScS
cxcx
cxcx
eeeecx
)tanh(
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
24
SIGNAL FUNCTION (ACTIVATION FUNCTION) 7 、 threshold exponential signal function:
When ,
),min()( cxexS 1
1cxe
0 cxceS '
02 cxecS ''
0 cxnn ecS )(
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
25
SIGNAL FUNCTION (ACTIVATION FUNCTION) 8 、 exponential-distribution signal function:
When ,
),max()( cxexS 10
0x
0 cxceS '
0'' 2 cxecS
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
26
SIGNAL FUNCTION (ACTIVATION FUNCTION) 9 、 the family of ratio-polynomial signal function: An example
For ,
),max()( n
n
xcxxS
0
1n
02
1
)(' n
n
xccnxS
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
27
SIGNAL FUNCTION (ACTIVATION FUNCTION)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
28
SIGNAL FUNCTION (ACTIVATION FUNCTION)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
29
PULSE-CODED SIGNAL FUNCTIONDefinition:
(5)
(6)
t ts
ii dsesxtS )()(
t ts
jj dsesytS )()(
tatpulsenoiftatoccurspulseaif
txi
01
)(
where
(7)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
30
PULSE-CODED SIGNAL FUNCTIONPulse-coded signals take values in the unit interval [0,1].Proof: when
0)(txi0
t ts
ii dsesxtS )()(
1)( txi
1
ts
s
tst tsi eedsetS lim)(
when
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
31
PULSE-CODED SIGNAL FUNCTION Velocity-difference property of pulse-coded signals
The first-order linear inhomogenous differential equation: (8)
(9)
)()( tqxtpx The solution to this differential equation:
t tst dsesqextx0
0 )()()(
A simple form for the signal velocity:)()()( tStxtS iii
)()()( tStytS jjj
(10)(11)
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
t ts
ii dsesxtS )()( (5)
32
PULSE-CODED SIGNAL FUNCTION
The central result of pulse-coded signal functions:The instantaneous signal-velocity equals the current pulse minus the current expected pulse frequency. ------------- the velocity-difference property of pulse-coded signal functions
NEURONAL DYNAMICS Ⅰ: ACTIVATIONS AND SIGNALS
)()()( tStxtS iii (10)
top related