image stabilization by bayesian dynamics yoram burak sloan-swartz annual meeting, july 2009

Image Stabilization by Bayesian Dynamics

Yoram BurakSloan-Swartz annual meeting, July 2009

What does neural activity represent?

In Bayesian models: probabilities

Direction of motion: single, static variable

Accumulated evidence in area LIPShadlen and Newsome (2001)

What does neural activity represent?

In Bayesian models: probabilities

Direction of motion: single, static variable

What about multi-dimensional, dynamic quantities?

Accumulated evidence in area LIPShadlen and Newsome (2001)

Foveal vision and fixational drift

Foveal vision and fixational drift

By XaqPitkow

- between micro-saccades -~20 receptive fields

Image from: X. Pitkow

- between spikes (100 Hz) -~2-4 receptive fields !

Fixational drift is large in the fovea:

cone separation: 0.5 arcmin

Foveal vision and fixational drift

By XaqPitkow

- between micro-saccades -~20 receptive fields

Image from: X. Pitkow

- between spikes (100 Hz) -~2-4 receptive fields !

Downstream areas require knowledge

of trajectory to interpret spikes

Fixational drift is large in the fovea:

cone separation: 0.5 arcmin

Joint decoding of image and position

Bayesian:

Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)

N x 2 probabilities

# positions

Bayesian:

Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)

N x 2 probabilities

Unconstrained image 30 x 30 binarypixels

# positions

N x 2900 probabilities

Joint decoding of image and position

Bayesian:

Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)

N x 2 probabilities

Unconstrained image 30 x 30 binarypixels

# positions

N x 2900 probabilities

Can the brain apply a Bayesian approach to this problem?

Joint decoding of image and position

Can the brain apply a Bayesian approach to this problem?

Decoding strategy

Performance in parameter space

What are the biological implications?

Can the brain apply a Bayesian approach to this problem?

Decoding strategy

Performance in parameter space

What are the biological implications?

Decoding strategy

Discards information about correlations

Factorized representation:

Decoding strategy

Discards information about correlations

minimizeDKL

Factorized representation:

Exact if trajectory is known.

evidence, diffusion

Update dynamics:

Decoding strategy

Discards information about correlations

minimizeDKL

Factorized representation:

Exact if trajectory is known.

evidence, diffusion

evidence - Poisson spiking (rate λ1 for on pixels, λ0 for off)diffusion - Random walk (diffusion coefficient D)

Retinal encoding model:

Update dynamics:

Decoding strategy

Discards information about correlations

Neural Implementation - Two populations: where , what

For 30 x 30 pixels: N × 2900 → N + 900

quantities.

Factorized representation:

Update rulesUpdate of what neurons:

multiplicative gating

Ganglion cells

What

Where

nonlinearity

Update rulesUpdate of what neurons:

Update of where neurons:

multiplicative gating

Ganglion cells

What

Where

Where

What

multiplicative gating

Ganglion cells

+ diffusion

nonlinearity

Demo

image

retina

m x m binary pixels

2d diffusion (D)

Poisson spikes:100 Hz (on), 10 Hz (off)

Decoder

Demo

Decoding strategy

Performance in parameter space

What are the biological implications?

Can the brain apply a Bayesian approach to this problem?

Performance

D D

Con

verg

en

ce t

ime [

s]

acc

ura

cy

Performance degrades with larger D (and smaller λ)

Performance

D D

Con

verg

en

ce t

ime [

s]

Faster and more accurate for larger images

m = 5, 10, 30, 50, 100

acc

ura

cy

Demo

Performance

D D

Con

verg

en

ce t

ime [

s]

Faster and more accurate for larger images

acc

ura

cy

m = 5, 10, 30, 50, 100

Performance

D D

Con

verg

en

ce t

ime [

s]

Faster and more accurate for larger images

acc

ura

cy

m = 5, 10, 30, 50, 100

Performance

D D

Con

verg

en

ce t

ime [

s]

Faster and more accurate for larger images

acc

ura

cy

m = 5, 10, 30, 50, 100

Performance

D/m D/m

Con

verg

en

ce t

ime [

s]

acc

ura

cy

scales with linear image size m

m x m pixels

Performance

D/m D/m

Con

verg

en

ce t

ime [

s]

acc

ura

cy

scales with linear image size m

Analytical scaling:

D*

m x m pixels

Performance

Performance improves with image size.

Success for images 10 x 10 or larger

Prediction for psychophysics:

Degradation in high acuity tasks when visual scene

contains little background detail.

Temporal response of Ganglion cells

Common view: fixational motion important to activate cells, due to biphasic response

f(t)

t

Temporal response makes decoding much more difficult.

50 ms

Need history

Non-Markovian:

Temporal response of Ganglion cells

Approach: Choose decoder that is Bayes optimal if the trajectory is known.

What

Ganglion

“filteredtrajectory”

Where

history dependent decoder / naive decoder

Converg

ence

tim

e [

s]

acc

ura

cy

D D

Temporal response of Ganglion cells

Is fixational motion beneficial?

Known trajectory , perfect inhibitory balanceC

onverg

ence

tim

e [

s]

D

Optimal D - order of magnitude smaller than biological value

Can the brain apply a Bayesian approach to this problem?

Decoding strategy

Performance in parameter space

What are the biological implications?

Network architecture

Each ganglion cell innervates multiple what & where cells(spread: ~10 arcmin)

WhereWhat

Ganglion

Reciprocal, multiplicative gating

Activity:

What neuronsSlow dynamics, evidence accumulation

Where neuronsFewer. Highly dynamic activityTonic, sparse in retinal stabilization conditions.

Activity:

What neuronsSlow dynamics, evidence accumulation

Where neuronsFewer. Highly dynamic activityTonic, sparse in retinal stabilization conditions.

Where in the brain?

Monocular

LGN?

V1?

If so, suggests LGN or V1

Modulatory inputs to relay cells (gating?)

Lateral connectivity in where network, Increase in number of neurons.

SummaryStrategy for stabilization of foveal visionFactorized Bayesian approach to multi-dimensional inference

SummaryStrategy for stabilization of foveal vision

Explicit representation of stabilized image“What” and “where” populations

Factorized Bayesian approach to multi-dimensional inference

SummaryStrategy for stabilization of foveal vision

Explicit representation of stabilized image“What” and “where” populations

Good performance at 1 arcmin resolutionProblem is easier for large images, for coarser reconstruction

Factorized Bayesian approach to multi-dimensional inference

SummaryStrategy for stabilization of foveal vision

Explicit representation of stabilized image“What” and “where” populations

Good performance at 1 arcmin resolutionProblem is easier for large images, for coarser reconstruction

Factorized Bayesian approach to multi-dimensional inference

Network architecture:Many-to-one inputs from retina, multiplicative gating (what/where)

Uri Rokni

Haim Sompolinsky Markus Meister

Special thanks - the Swartz foundation

Acknowledgments