![Page 1: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/1.jpg)
Classical neurodynamics
Sebastian Seung
![Page 2: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/2.jpg)
€
τdxidt
+ xi = f bi + Wijx jj∑
$
% &
'
( )
activity
synaptic strength
activation function
elementary time constant
external input
A model of neural networks
• xi are rate-like variables
![Page 3: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/3.jpg)
Spikes vs. rates
• “Quantum” neurodynamics – neural activity is quantized – comes in packets called “spikes”
• “Classical” neurodynamics – neural activity is continuous ly graded – described by rates
![Page 4: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/4.jpg)
A model neuron
• Each synapse is a current source controlled by presynaptic spiking.
• The dendrite adds the currents of multiple synapses.
• The total current drives spiking in the axon.
![Page 5: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/5.jpg)
Voltage clamp measurement of synaptic transmission
![Page 6: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/6.jpg)
A synapse as a current source
• Each action potential causes a synapse to inject a stereotyped current into the postsynaptic neuron.
• Let the “synaptic strength” W denote the total charge injected by the synapse due to a single presynaptic spike.
![Page 7: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/7.jpg)
Decaying exponential model
€
g t( ) =τ−1e− t /τ , t > 0,0, t ≤ 0.
% & ' €
I t( ) =Wg t( )
![Page 8: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/8.jpg)
€
I t( ) =W g t − t a( )a∑
Temporal summation
• Approximation: currents from successive spikes add linearly.
![Page 9: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/9.jpg)
Currents of divergent synapses share the same time
course.
![Page 10: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/10.jpg)
Normalized current
• Assumption: all synapses have the same time constant τ
€
Iij t( ) =Wij g t − t ja( )
a∑ =Wij x j t( )
€
x j t( ) = g t − t ja( )
a∑
![Page 11: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/11.jpg)
“Quantum” vs. “classical”
![Page 12: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/12.jpg)
Leaky integrator model
• If there is a spike, • Otherwise, exponential decay with time
constant τ
• Normalized current is a low-pass filtered version of the presynaptic spike train.
€
τdx j
dt+ x j = δ t − t j
a( )a∑€
x j := x j +1/τ
![Page 13: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/13.jpg)
Firing rate approximation
• Normalized current is a low-pass filtered version of the firing rate.
• Accurate if rate is large compared to 1/τ
€
τdx j
dt+ x j = δ t − t j
a( )a∑
≈ν j
![Page 14: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/14.jpg)
€
Ii = Iijj∑
= Wij x jj∑
Summation of currents from convergent synapses
• Approximation: linear superposition of currents from different synapses onto the same postsynaptic neuron.
![Page 15: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/15.jpg)
Spike generation
• Regard the soma as a device that transduces current into rate of action potential firing.
• The detailed dynamics of action potential generation (voltage, etc.) is implicit.
![Page 16: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/16.jpg)
Somatic current injection
• Simulation of synaptic current
![Page 17: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/17.jpg)
Repetitive spiking
• Layer V pyramidal neurons in rat sensorimotor cortex
Schwindt, O’Brien, and Crill. J. Neurophysiol. 77:2484 (1997)
![Page 18: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/18.jpg)
f-I curve
Schwindt, O’Brien, and Crill. J. Neurophysiol. 77:2484 (1997)
![Page 19: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/19.jpg)
Putting it all together
€
τdxidt
+ xi = f Ii( )
Ii = bi + Wij x jj∑
normalized current sourced by synapses from neuron i
input current to neuron i
![Page 20: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/20.jpg)
Biophysical interpretation
• f is firing rate as a function of current • x is normalized synaptic current
(measured in Hz) • τ is synaptic time constant • W is the charge/presynaptic spike
€
τdxidt
+ xi = f bi + Wijx jj∑
$
% &
'
( )
![Page 21: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/21.jpg)
Activation functions
![Page 22: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/22.jpg)
(Half-wave) rectification
• Threshold for firing. • Rises smoothly from zero. • No saturation. • Popular for modeling the visual system. • “Semilinear”: linear concepts
applicable.
€
u[ ]+ =max u,0{ } =u, u ≥ 0,0, u ≤ 0.$ % &
![Page 23: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/23.jpg)
Step function
• Jumps to nonzero value at threshold. • No increase in rate thereafter. • Computation with binary variables. • Popular with computer scientists.
€
H u( ) =1, u > 0,0, u ≤ 0.# $ %
![Page 24: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/24.jpg)
Sigmoidal function
• Compromise – Binary for large inputs. – Linear for small inputs.
• “S-shaped” • Logistic function
– 0 to 1 • Hyperbolic tangent
– -1 to 1
€
σ u( ) =1
1+ e−u
€
tanhu =eu − e−u
eu + e−u
![Page 25: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/25.jpg)
Rates vs. spikes
• When can the quantization of light be neglected? – “lots of photons” and little synchrony
• When can the quantization of neural activity be neglected? – “lots of spikes” and little synchrony
![Page 26: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/26.jpg)
Signal flow in neurons (classical view)
• dendrite: passive spread of current from synapses to soma
• axon: active propagation of spikes to synapses
dendrite soma or cell body axon
current spikes
![Page 27: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/27.jpg)
Deficiencies of the rate model
• Soma – Spike frequency adaptation
• Dendrite – Passive nonlinearities – Active nonlinearities
• Short-term synaptic plasticity
![Page 28: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/28.jpg)
Spike frequency adaptation
Schwindt, O’Brien, and Crill. J. Neurophysiol. 77:2484 (1997)
![Page 29: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/29.jpg)
Short-term depression
Tsodyks & Markram, PNAS 94:719 (1997)
![Page 30: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/30.jpg)
Further reading
• C. Koch, Biophysics of Computation, Oxford Univ. Press (1999).
• B. Ermentrout, Reduction of Conductance-Based Models with Slow Synapses to Neural Nets, Neural Comput. 6, 679-695 (1995).
• O. Shriki, D. Hansel, H. Sompolinsky, Rate Models for Conductance-Based Cortical Neuronal Networks, Neural Comput. 15, 1809-1841 (2003).
![Page 31: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/31.jpg)
Is the solar system stable?
Joseph-Louis Lagrange 1736-1813
Pierre-Simon Laplace 1749-1827
![Page 32: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/32.jpg)
Laplace-Lagrange secular theory
• Fast variable: orbit of planet around sun • Slow variable: precession of orbit
![Page 33: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/33.jpg)
Slow and rapid variables
• “separation of time scales”
ds
dt=
1
⌧S(r, s)
dr
dt= R(r, s)
![Page 34: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/34.jpg)
Time average for constant s
• For any constant s, define rs(t) as the solution with initial value rs(0)
• Assume that this time average does not depend on the initial value rs(0)
S̄(s) = limT!1
1
T
Z T
0S(rs(t), s)dt
dr
dt= R(r, s)
![Page 35: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/35.jpg)
Averaging theorem
• The solutions of these equations
differ by O(1/τ) over a time interval of O(τ)
ds
dt=
1
⌧S(r, s)
dr
dt= R(r, s)
ds
dt=
1
⌧S̄(s)
![Page 36: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/36.jpg)
Further reading
• J. Laskar. Is the solar system stable? Prog. Math. Phys. 66, 239-70 (2013).
• V. M. Volosov. Averaging in systems of ordinary differential equations. Russ. Math. Surv. 17 (1962).
![Page 37: Classical neurodynamics - Seung Labseunglab.org/wp-content/uploads/2015/02/rateeqs.pdf · Spikes vs. rates • “Quantum” neurodynamics – neural activity is quantized – comes](https://reader034.vdocuments.site/reader034/viewer/2022051509/5ad9c3357f8b9a6d318bc32b/html5/thumbnails/37.jpg)
Napoleon: You have written this huge book on the system of the world without once mentioning the author of the universe. Laplace: Sire, I had no need of that hypothesis. (When told by Napoleon about the incident) Lagrange: Ah, but that is a fine hypothesis. It explains so many things.