bayesian perception

Bayesian Perception

General Idea

Ernst and Banks, Nature, 2002

General Idea

• Bayesian formulation:

, || ,

,

| |

,

| |

P t v w P wP w t v

P t v

P t w P v w P w

P t v

P t w P v w P w

ConditionalIndependence assumption

ˆ arg max | ,w

w P w t v

General Idea

Ernst and Banks, Nature, 2002

w

v=w+n

t=w+n

+

+

noise

noise

w?

Generative model: , |P t v w

General Idea

2

2

2

2

2 2

2 2

2 22 2

2 2

2 2 2 2 2

2 2

2 22

2 2

2 2 2 2

2 2

2 2

| exp2

log |2

log | |2 2

2

2

2

2

2 /

v

v

v t

t v

v t

t v t v

v t

t v

t v

v t t v

t v

t v

v wP v w

v wP v w

v w t wP v w P t w

v w t w

w v t w C

v tw w C

v tw

2

2 2 2 22 /v t t v

C

General Idea

Mean and variance

2 2

2 2 2 2t v

t v t v

w v t

22 2

2 2

2 2 2 2log | ,

2 /

t v

t v

v t t v

v tw

P w v t

General Idea

Width

Prob

abil

ity

VisualP(v|w)

TouchP(t|w)

tv

2 2

2 2 2 2t v

t v t v

w v t

General Idea

Mean and variance

2 2

2 2 2 2t v

t v t v

w v t

22 2

2 2

2 2 2 2log | ,

2 /

t v

t v

v t t v

v tw

P w v t

2 2

2

2 2v t

w

t v

Optimal Variance

Variance

2 2 2

1 1 1

w v t

w v tI I I

22 2

2 2

2 2 2 2log | ,

2 /

t v

t v

v t t v

v tw

P w v t

Fisher information sums for independent signals

General Idea

0 67 133 2000

0.05

0.1

0.15

0.2

Th

resh

old

(S

TD

)

Visual noise level (%)

Measured bimodal STD

Predicted by the Bayesian model

Unimodal visual STD

Unimodal Tactile STD

Ernst and Banks, Nature, 2002 Note: unimodal estimates may not be optimal but the multimodal estimate is optimal

Adaptive Cue Integration

• Note: the reliability of the cue change on every trial

• This implies that the weights of the linear combination have to be changed on every trial!

• Or do they?

2 2

2 2 2 2t v

t v t v

w v t

General Idea

• Perception is a statistical inference

• The brain stores knowledge about P(I,V) where I is the set of natural images, and V are the perceptual variables (color, motion, object identity)

• Given an image, the brain computes P(V|I)

, ||

P P PP

P P

I V I V VV I

I I

General Idea

• Decisions are made by collapsing the distribution onto a single value:

• or

ˆ |P dV V I V V

ˆ arg max |PV

V V I

Key Ideas

• The nervous systems represents probability distributions. i.e., it represents the uncertainty inherent to all stimuli.

• The nervous system stores generative models, or forward models, of the world (P(I|V)), and prior knowlege about the state of the world (P(V))

• Biological neural networks can perform complex statistical inferences.

Motion Perception

The Aperture Problem


Horizontal velocity (deg/s)V

erti

cal v

eloc

ity

(deg

/s)



erti

cal v

eloc

ity

(deg

/s)

Standard Models of Motion Perception

• IOC: interception of constraints

• VA: Vector average

• Feature tracking



erti

cal v

eloc

ity

(deg

/s)

IOCVA



erti

cal v

eloc

ity

(deg

/s)

VA

IOC



erti

cal v

eloc

ity

(deg

/s)

IOCVA


• Problem: perceived motion is close to either IOC or VA depending on stimulus duration, eccentricity, contrast and other factors.


• Example: Rhombus

Horizontal velocity (deg/s)

Ver

tica

l vel

ocit

y (d

eg/s

) IOCVA


Ver

tica

l vel

ocit

y (d

eg/s

) IOCVA

Percept: VAPercept: IOC

Moving Rhombus

Bayesian Model of Motion Perception

• Perceived motion correspond to the MAP estimate

* arg max |

|| |

, |i ii

P I

P I PP I P I P

P I

P P I x y

vv v

v vv v v

v v

Prior

• Human observers favor slow motions

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

2 2exp / 2 pP v v

Likelihood

• Weiss and Adelson

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

, , |i i iP I x y t v

Likelihood

, , , ,

, , , ,

x y

x y

I x y t I x t y t t t

I x y t I x t y t t t

v v

v v

, , , ,

, , , ,

i x i y i i x x y y t

i i i x i y x x y y t

x x y y t

I x t y t t t I x y t I t I t I t

I x y t I x t y t t t I t I t I t

I I I t

v v v v

v v v v

v v

2

2

, , , ,, , | exp

2

i i x y

i i

I x y t I x t y t t tP I x y t

v vv

2

2, , | exp

2x x y y t

i i

I I IP I x y t

v vv

Likelihood

2

2

2

2,

, , | , , |

1exp

2

1exp ,

2

i ii

x x y y ti

x x y y t

x y

P I x y t P I x y t

I I I

w x y I I I dxdy

v v

v v

v v

Binary maskPresumably, this is set by segmentation cues

Posterior

2 2 22 2

,

log | , , log , , | log

1 1

2 2x x y y t x yx y p

P I x y t P I x y t P

I I I

v v v

v v v v

Bayesian Model of Motion Perception

• Perceived motion corresponds to the MAP estimate

*

22

2

*

22

2

arg max | , | , |ii

i

x x yp x t

y tx y y

p

P I P I P P I x y

I I II I

I II I I

vv v v v v

v

Only one free parameter

Likelihood

2 2 22 2

, ,

, ,

2 2

,,

22

2, ,

2 22

2, ,

1 1( )

2 2

2 2( ) 1 1

2 2 22

1

x x y y t x yx y x yp

x x y y t x xx y x y

p yx x y y t yx yx y

x y x tx y x yp

y x yx y x y p

L I I I

I I I IL

I I I I

I I I I I

I I I

v v v v v

v v vv

v vv v

v,

,

0

xx y

t yx y

I I

Motion through an Aperture

• Humans perceive the slowest motion.

• More generally: we tend to perceive the most likely interpretation of an image

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

Motion through an Aperture

ML

MAP

Prior Posterior

Likelihood

Motion and Constrast

• Humans tend to underestimate velocity in low contrast situations

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

Motion and Contrast

ML

MAP

Prior Posterior

HighContrast

Likelihood

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

Motion and Contrast

ML

MAP

Prior Posterior

LowContrast

Likelihood

Motion and Contrast

• Driving in the fog: in low contrast situations, the prior dominates

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

Moving Rhombus

IOC

MAP

Prior Posterior

HighContrast

Likelihood

Moving Rhombus

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

-50 0 50

-50

0

50

Horizontal Velocity

Ver

tica

l Ve

loci

ty

IOC

MAP

Prior Posterior

LowContrast

Likelihood

Moving Rhombus

Moving Rhombus

• Example: Rhombus


Ver

tica

l vel

ocit

y (d

eg/s

) IOCVA


Ver

tica

l vel

ocit

y (d

eg/s

) IOCVA

Percept: VAPercept: IOC

Barberpole Illusion

Plaid Motion: Type I and II

Plaids and Contrast

Lower contrast

Plaids and Time

• Viewing time reduces uncertainty

Ellipses

• Fat vs narrow ellipses

Ellipses

• Fat vs narrow ellipses

• All motions agree

Ellipses

Ellipses

• Adding unambiguous motion

Other Prior

• Prior on direction of lightning

Generalization

• All computation are subject to uncertainty (ill-posed)

• This includes syntax processing, language acquisition… etc.

• Solution: compute with probability distributions

Binary Decision Making

Shadlen et al.

Race Model

• Standard theory: some signal is accumulated (or integrated) to a bound. Also known as race models.

• The signal to be integrated could be the response of sensory neurons.

Bayesian Strategy

• The ‘diffusion to bound’ model of Shadlen et al.

High motion strength

High m

otion stre

ngth

Low motion strength

Time

~1 secStimulus

onStimulus

off

Spikes/s

Time

~1 secStimulus

onStimulus

off

Spikes/s

Low motion strength

A Neural Integrator for Decisions?

MT: Sensory Evidence

Motion energy

“step”

LIP: Decision Formation

Accumulation of evidence

“ramp”

Threshold

Diffusion to bound model

Positive bound

Negative bound

Proposed by Wald, 1947 and Turing (WW II, classified); Stone, 1960; then Laming, Link, Ratcliff, Smith, . . .


Positive bound or Criterion to answer “1”

Negative bound or Criterion to answer “2”

Momentary evidencee.g.,

∆Spike rate:MTRight– MTLeft

Accumulated evidencefor Rightward

andagainst Leftward

Criterion to answer “Right”

Criterion to answer “Left”


Shadlen & Gold (2004)Palmer et al (2005)

kC

C is motion strength (coherence)Seems arbitrary but why not?

MT responses

60

40

20

0

Firi

ng r

ate

Direction (deg)

Height scales with coherence

MT MTRight Leftr r

Right Left


• Performance reaction time trade-off

Best fitting chronometric function“Diffusion to bound”

t(C) B

kCtanh(BkC) tnd

Predicted psychometric function “Diffusion to bound”

P 1

1 e 2k C B

Average LIP activity in RT motion task

Roitman & Shadlen, 2002 J. Neurosci.

choose Tin

choose Tout

Note the clear asymmetry

Bayesian Strategy

• The Bayesian strategy in this case consists in computing the posterior distribution given all activity patterns from MT up to the current time,

MT:1MT

:1 MT:1

MT

MT MT MT1:1 1:1

MT MT1:1

MT MT1:1

||

|

| , |

| |

| |

t

t

t

t

t t t

t t

t t

p s p sp s

p

p s p s

p s p s p s

p s p s p s

p s p s

rr

r

r

r r r

r r

r r

Bayesian Strategy

• Race models and Bayesian approach

MT MT MT:1 1:1

MT MT MT:1 1:1

MT

1

| | |

log | log | log |

log |

t t t

t t t

t

p s p s p s

p s p s p s

p s

r r r

r r r

r

Temporal sum

Unless is related to …

But not over rMT, or MT MTRight Leftr r

MTlog |p srMT MTRight Leftr r

Bayesian Strategy

• Are neurons computing log likelihood?

• The difference of activity between two neurons with preferred directions 180 deg away is proportional to a log likelihood ratio.

Bayesian Strategy

• Is the log likelihood ration proportional to ?

2MT MTR L R LMT MT

R L 2

2MT MTR L R LMT MT

R L 2

R R| exp

2

L L| exp

2

p R

p L

r rr r

r rr r

MT MTR, L, r r

Coherence level

MT MT2 2R L MT MT MT MT

R L R L R L R LMT MTR L

|log R R L L

|

p R

p L

r rr r r r

r r

MT MTR L R LR R r r

MT MTR LC r r

Bayesian Strategy

• Note that if you know , you still don’t know the log likelihood ration unless you’re given the coherence level.

• Therefore, the animal can’t know its confidence level (the log likelihood ratio) unless it estimates C…

• Another important point: if we stop the race at a fixed level of we stop at different levels of log likelihood ratio depending on the coherence. This is why performance gets better when coherence increases, even though we always stop at the same activity threshold.

MT MTR, L, r r

MT MTR, L, r r

Decision Making

• Does that mean the animal does not know how much to trust its own decision?

• Does that mean the brain does not encode uncertainty or probability distribution?

• Seems unlikely…

• To be continued…

bayesian perception

Documents

slowest motion

generative models

forward models

general ideaperception

bayesian modelernst

map estimateonly

world piv

horizontal velocityve