image stabilization by bayesian dynamics yoram burak sloan-swartz annual meeting, july 2009
TRANSCRIPT
![Page 1: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/1.jpg)
Image Stabilization by Bayesian Dynamics
Yoram BurakSloan-Swartz annual meeting, July 2009
![Page 2: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/2.jpg)
What does neural activity represent?
In Bayesian models: probabilities
Direction of motion: single, static variable
Accumulated evidence in area LIPShadlen and Newsome (2001)
![Page 3: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/3.jpg)
What does neural activity represent?
In Bayesian models: probabilities
Direction of motion: single, static variable
What about multi-dimensional, dynamic quantities?
Accumulated evidence in area LIPShadlen and Newsome (2001)
![Page 4: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/4.jpg)
Foveal vision and fixational drift
![Page 5: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/5.jpg)
Foveal vision and fixational drift
By XaqPitkow
- between micro-saccades -~20 receptive fields
Image from: X. Pitkow
- between spikes (100 Hz) -~2-4 receptive fields !
Fixational drift is large in the fovea:
cone separation: 0.5 arcmin
![Page 6: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/6.jpg)
Foveal vision and fixational drift
By XaqPitkow
- between micro-saccades -~20 receptive fields
Image from: X. Pitkow
- between spikes (100 Hz) -~2-4 receptive fields !
Downstream areas require knowledge
of trajectory to interpret spikes
Fixational drift is large in the fovea:
cone separation: 0.5 arcmin
![Page 7: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/7.jpg)
Joint decoding of image and position
Bayesian:
Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)
N x 2 probabilities
# positions
![Page 8: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/8.jpg)
Bayesian:
Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)
N x 2 probabilities
Unconstrained image 30 x 30 binarypixels
# positions
N x 2900 probabilities
Joint decoding of image and position
![Page 9: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/9.jpg)
Bayesian:
Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)
N x 2 probabilities
Unconstrained image 30 x 30 binarypixels
# positions
N x 2900 probabilities
Can the brain apply a Bayesian approach to this problem?
Joint decoding of image and position
![Page 10: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/10.jpg)
Can the brain apply a Bayesian approach to this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
![Page 11: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/11.jpg)
Can the brain apply a Bayesian approach to this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
![Page 12: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/12.jpg)
Decoding strategy
Discards information about correlations
Factorized representation:
![Page 13: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/13.jpg)
Decoding strategy
Discards information about correlations
minimizeDKL
Factorized representation:
Exact if trajectory is known.
evidence, diffusion
Update dynamics:
![Page 14: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/14.jpg)
Decoding strategy
Discards information about correlations
minimizeDKL
Factorized representation:
Exact if trajectory is known.
evidence, diffusion
evidence - Poisson spiking (rate λ1 for on pixels, λ0 for off)diffusion - Random walk (diffusion coefficient D)
Retinal encoding model:
Update dynamics:
![Page 15: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/15.jpg)
Decoding strategy
Discards information about correlations
Neural Implementation - Two populations: where , what
For 30 x 30 pixels: N × 2900 → N + 900
quantities.
Factorized representation:
![Page 16: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/16.jpg)
Update rulesUpdate of what neurons:
multiplicative gating
Ganglion cells
What
Where
nonlinearity
![Page 17: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/17.jpg)
Update rulesUpdate of what neurons:
Update of where neurons:
multiplicative gating
Ganglion cells
What
Where
Where
What
multiplicative gating
Ganglion cells
+ diffusion
nonlinearity
![Page 18: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/18.jpg)
Demo
image
retina
m x m binary pixels
2d diffusion (D)
Poisson spikes:100 Hz (on), 10 Hz (off)
Decoder
![Page 19: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/19.jpg)
Demo
![Page 20: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/20.jpg)
Decoding strategy
Performance in parameter space
What are the biological implications?
Can the brain apply a Bayesian approach to this problem?
![Page 21: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/21.jpg)
Performance
D D
Con
verg
en
ce t
ime [
s]
acc
ura
cy
Performance degrades with larger D (and smaller λ)
![Page 22: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/22.jpg)
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
m = 5, 10, 30, 50, 100
acc
ura
cy
![Page 23: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/23.jpg)
Demo
![Page 24: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/24.jpg)
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
acc
ura
cy
m = 5, 10, 30, 50, 100
![Page 25: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/25.jpg)
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
acc
ura
cy
m = 5, 10, 30, 50, 100
![Page 26: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/26.jpg)
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
acc
ura
cy
m = 5, 10, 30, 50, 100
![Page 27: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/27.jpg)
Performance
D/m D/m
Con
verg
en
ce t
ime [
s]
acc
ura
cy
scales with linear image size m
m x m pixels
![Page 28: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/28.jpg)
Performance
D/m D/m
Con
verg
en
ce t
ime [
s]
acc
ura
cy
scales with linear image size m
Analytical scaling:
D*
m x m pixels
![Page 29: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/29.jpg)
Performance
Performance improves with image size.
Success for images 10 x 10 or larger
Prediction for psychophysics:
Degradation in high acuity tasks when visual scene
contains little background detail.
![Page 30: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/30.jpg)
Temporal response of Ganglion cells
Common view: fixational motion important to activate cells, due to biphasic response
f(t)
t
Temporal response makes decoding much more difficult.
50 ms
Need history
Non-Markovian:
![Page 31: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/31.jpg)
Temporal response of Ganglion cells
Approach: Choose decoder that is Bayes optimal if the trajectory is known.
What
Ganglion
“filteredtrajectory”
Where
history dependent decoder / naive decoder
Converg
ence
tim
e [
s]
acc
ura
cy
D D
![Page 32: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/32.jpg)
Temporal response of Ganglion cells
Is fixational motion beneficial?
Known trajectory , perfect inhibitory balanceC
onverg
ence
tim
e [
s]
D
Optimal D - order of magnitude smaller than biological value
![Page 33: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/33.jpg)
Can the brain apply a Bayesian approach to this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
![Page 34: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/34.jpg)
Network architecture
Each ganglion cell innervates multiple what & where cells(spread: ~10 arcmin)
WhereWhat
Ganglion
Reciprocal, multiplicative gating
![Page 35: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/35.jpg)
Activity:
What neuronsSlow dynamics, evidence accumulation
Where neuronsFewer. Highly dynamic activityTonic, sparse in retinal stabilization conditions.
![Page 36: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/36.jpg)
Activity:
What neuronsSlow dynamics, evidence accumulation
Where neuronsFewer. Highly dynamic activityTonic, sparse in retinal stabilization conditions.
Where in the brain?
Monocular
LGN?
V1?
If so, suggests LGN or V1
Modulatory inputs to relay cells (gating?)
Lateral connectivity in where network, Increase in number of neurons.
![Page 37: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/37.jpg)
SummaryStrategy for stabilization of foveal visionFactorized Bayesian approach to multi-dimensional inference
![Page 38: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/38.jpg)
SummaryStrategy for stabilization of foveal vision
Explicit representation of stabilized image“What” and “where” populations
Factorized Bayesian approach to multi-dimensional inference
![Page 39: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/39.jpg)
SummaryStrategy for stabilization of foveal vision
Explicit representation of stabilized image“What” and “where” populations
Good performance at 1 arcmin resolutionProblem is easier for large images, for coarser reconstruction
Factorized Bayesian approach to multi-dimensional inference
![Page 40: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/40.jpg)
SummaryStrategy for stabilization of foveal vision
Explicit representation of stabilized image“What” and “where” populations
Good performance at 1 arcmin resolutionProblem is easier for large images, for coarser reconstruction
Factorized Bayesian approach to multi-dimensional inference
Network architecture:Many-to-one inputs from retina, multiplicative gating (what/where)
![Page 41: Image Stabilization by Bayesian Dynamics Yoram Burak Sloan-Swartz annual meeting, July 2009](https://reader036.vdocuments.site/reader036/viewer/2022062301/5697bf821a28abf838c85cdf/html5/thumbnails/41.jpg)
Uri Rokni
Haim Sompolinsky Markus Meister
Special thanks - the Swartz foundation
Acknowledgments