Learning and Tuning of Neurons in Inferior Temporal Cortex

Bharathi JagadeeshDepartment of Physiology & BiophysicsUniversity of WashingtonSeattle, Washington

Learning and Neural Plasticity in the Adult Visual SystemSociety for Neuroscience

San Diego, California

V1

PGPF

V3

V1 V2 V4 TEO TE

TE

Ventral or “What” processing stream

Macaque temporal lobe

Pictures of people, places, and things

100

s/s

300 ms

best next best

worst

What does the selectivity in IT mean?

V1 V2 V4 TEO ITre

spo

nse

Perceptual similarity

IT

Perceptual similarity

• Image characteristics

• Experience

Perceptual similarity

Neural responses in IT

Proposed relationship

Similarity of stimuli should explain selectivity in IT cortex

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Proposed relationship

Measuring perceptual similarity

How do we use perceptual similarity algorithms?

Image similarity algorithms

Wang et al (2001)

SIMPlicity algorithm

1. Image divided into 4x4 pixel blocks, feature vector is calculated for each block.

2. Feature vector 6-dimensional: Color dimensions, (LUV space, 3 dimensions) , Spatial frequency, wavelet analysis on the L component of the image (3 dimensions)

3. Number of regions using a k-means algorithm.

4. The similarity between two images computed by comparing regions using Integrated Region Matching (IRM).

5. The SIMPLIcity (similarity) distance is weighted sum of similarity between regions.

25.8 32.2

38.4

33.559.9

47.5

31.2

57.9

30.0

40.954.9

32.1

Calculate image distances between images

Prediction

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Proposed relationship

?

3.54 4.07 4.07 4.34 4.92

5.13 5.24 5.56 5.72 5.90 5.93

6.05 6.39 6.42 6.49 7.00 7.12

7.33 7.34 7.49 7.50 7.50 7.51

SIMPlicity retrieves targets

50 100 150 200

0.25

0.5

0.75

1.00SIMPLIcity

Number of Relevant Images Retrieved

Pre

cisi

on

Algorithms predict perceptual similarity

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Proposed relationship

?

Do perceptual similarity algorithms explain neural responses in IT cortex?

100

s/s

300 ms

best next best

worst

Example cell: image distance between best/next and best/worst

best-next best

best-worst

0

15

30

45

SIM

PLI

city

best

next best

best

worst

SIMPLIcity

(best and next best stimuli)

best

worst

next best

best

Population: Distance between Best-worst v. Best-Next Best

0 50 1000

50

100S

IMP

LIci

ty

( be

st a

nd w

orst

stim

uli)

30

40

50

best-next best-worst

Do other similarity algorithms explain neural responses in IT cortex?

50 100 150 200

0.25

0.50

0.75

1.00

Number of Relevant Images Retrieved

Pre

cisi

on

Contrast: Doesn’t retrieve targets

0.0 0.2 0.4 0.6

0.0

0.2

0.4

0.6

RMS contrast difference (best and next best stimuli)

RM

S c

ontr

ast

diff

eren

ce

(be

st a

nd w

orst

stim

uli)

And, doesn’t explain IT responses

best

worst

next best

best

0

0.1

0.2

best-next best-worst

EMD, another similarity metric: Retrieves targets

50 100 150 200

0.25

0.50

0.75

1.00

Number of Relevant Images Retrieved

Pre

cisi

on

And, also explains IT responses

0 50 1000

20

40

60

80

100

EMD (best and next best stimuli)

EM

D

(be

st a

nd w

orst

stim

uli)

best next best

20

30

40

best-next best-worst

Prediction

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Proposed relationship

Behavior

?

Fixation250-500ms

Response

Stimulus16-512ms

Mask256 ms Delay

100-500msDelay500-1200ms

Delayed match to sample (DMS) (easy pair)

Fixation250-500ms

Response

Stimulus16-512ms

Mask256 ms Delay

100-500msDelay500-1200ms

DMS (difficult pair)

Measure “perceptual similarity”

Performance @ 50 ms stimulus presentation

Low performance High performance

71% 96%

Measure neural selectivity

Average neural response difference in passive fixation task

Performance @ 50 ms stimulus presentation

Low performance High performance

71% 96%

62% 86%

0.6 0.7 0.8 0.9 10.6

0.7

0.8

0.9

1

Behavioral performance

Neu

ral R

OC

r = 0.57

Neural performance v Behavior

Prediction

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Individual correlations

0

0.2

0.4

Behavior v

neuron

Algorithmv

neuron

0.6C

orr

ela

tion

r

Prediction

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Partial correlations

0

0.2

0.4

0.6

Behavior v

neuron

Algorithmv

neuron

Pa

rtia

l cor

rela

tion

r

Perceptual similarity correlated with IT neuron response similarity

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

How does training change the relationship to the algorithm?

No Go

Go

FixationPredictor

DelayChoice

Barrelease

Erickson CA, Desimone R (1999)

Passive Association Task

Go Trials No Go TrialsPredictor Choice Predictor Choice

Go 359 ms

Go377 ms

Go376 ms

Go369 ms

No Go

No Go

No Go

No Go

Valid Association Trials

Invalid Association Trials

Go 359/473 ms

Go377/465 ms

Go376/436 ms

Go369/363 ms

No Go

No Go

No Go

No Go

Go Trials No Go TrialsPredictor Target Predictor Target

Response to choice stimulus is correlated with response to predictor

Neural response to predictor

Ne

ura

l re

spo

nse

to c

hoi

ce

Erickson & Desimone (1999)

Dissimilar stimuli produce similar responses

Erickson & Desimone (1999)

Training breaks relationship to algorithm

35

37

39

41

43

Similar Responses

Differentresponses

Imag

e di

stan

ce35

37

39

41

43

Similar responses

Different responses

Imag

e di

stan

ce

Association task: TrainingPassive fixation: No training

Data from Erickson & Desimone (1999)

Specific conclusions

•Perceptual similarity algorithms measure (at least partially) the perceptual similarity of stimuli.

•The same algorithms explain (at least partially) the neural response similarity in IT.

•But, neural response similarity is better correlated with discrimination performance (measured perceptual similarity) than it is with the image similarity algorithms.

•And, training that modifies the processing of stimuli breaks the relationship between image similarity and neural response similarity.

Katie AhlSarah AllredYan Liu, M.D., Ph.D.Jen Skiver Thompson

Andrew Derrington, Ph.D.Cynthia Erickson, Ph.D.J.M.Jagadeesh, Ph.D.Amanda Parker, Ph.D.

Jamie BullisRebecca MeaseAmber McAlister

Current members

Rotation students & former members

Visiting scholars and collaborators

Jagadeesh labUniversity of Washington

Divider

Methods

Record from single neurons in the non-human primate brain, while the primate performs visual tasks.

Monitor eye movements so that visual stimulus at the retina is known.

In my lab, we record from inferotemporal cortex in the macaque, and are of cortex thought to be important for perception of people places and things.

100

s/s

300 ms

best next best

worst

B

0 50 1000

50

100

SIMPLIcity

(within effective group)

SIM

PLI

city

(eff

ectiv

e v

inef

fect

ive

grou

ps)

35

37

39

41

43

Eff Eff-Ineff

Ima

ge

dis

tan

ce

Population within v across groups

0 100 200 300 400 500 6000

10

20

30

40

time in ms after stimulus onset

spik

es

pe

r se

con

d

rank 1-8 rank 10-17rank 18-24

Histograms, population, sorted by emd rank to best

-1 -0.5 0 0.5 1

0

20

40

60

80

r-value

freq

uenc

y

Population: all comparisons

20 40 60 800

20

40

60

80

SIMPLIcity to best

ne

ura

l re

spo

nse

(sp

ike

s/se

con

d)

20 40 60 800

20

40

60

80

SIMPLIcity to worst

ne

ura

l re

spo

nse

(sp

ike

s/se

con

d)

Example : SIMPlicity correlated with neural response

best worst

0

20

40

60

80

20

40

60

80

-1 -0.5 0 0.5 10

20

40

60

80

-100 -50 0 50 1000

20

40

60

80

R-value

fre

qu

en

cyfr

eq

ue

ncy

Slope

best

worst

Population: SIMPlicity is correlated with neural response

0 5 10 15 20 25

0

0.03

0.06

0.09

0.12

0.15

Stimulus Rank (best to worst)

Ave

rag

e r

-va

lue

(a

bso

lute

va

lue

)

SIMPLIcity

shuffle

Population: correlation to each

EMD: Does retrieve images, also explains neural response

50 100 150 200

0.25

0.5

0.75

1

Number of Relevant Images Retrieved

Pre

cisi

on

avg EMD example EMDchance

0 50 1000

20

40

60

80

100

EMD (best and next best stimuli)

EM

D

(b

est

an

d w

ors

t st

imu

li)

A B

25.8 17.7

32.2 19.3

38.4 29.2

33.5 24.9

59.9 37.8

47.5 30.4

31.225.1

57.9 35.7

30.0 22.3

40.9 31.5

54.9 39.0

32.1 25.6

Individual correlations

0.0

0.2

0.4

0.6

neuron/behavior

Image sim

/neuron

Image sim

/behavior

core

lati

on

, r

Individual correlations

0

0.2

0.4

beh/neu emd/beh emd/neu

0.6

Partial correlations

0

0.2

0.4

0.6

beh/neu img/beh img/neu

Invalid v valid latencies

300

400

500

600

700

300 400 500 600 700

valid latencies (ms)

inva

lid la

tenc

ies

(ms)

Valid v Invalid trial latencies

300

400

500

600

700

300 400 500 600 700

valid latencies (ms)

invalid

late

ncie

s (

ms)

90% of trials

10% of trials

With training, IT relationship to algorithmic image similarity breaks

10 20 30 40 50 6010

20

30

40

50

60

within effective

across

0 50 1000

50

100

within effective

across

Data from Erickson and Desimone (1999)

V1 V2 V4 TEO IT

orientation

resp

on

se

V1

Proposal: IT neurons represent the perceptual similarity of stimuli.

Prediction

Perceptual similarity algorithms

Perceptual similarity

Neural responses in IT

Proposed relationship

Behavior

?

Retrieval

0

0.1

0.2

0 0.5 1

Imag

e di

ff

Characteristics of algorithms

1) Rely heavily on color content of images

2) Pairwise comparison, not a parametric space

3) Ignores “cognitive” information

But, works for realistic images, because images that are similar to one another in the real world tend to share patterns of colors.

Algorithms predict perceptual similarity

Test of algorithm in “spiked” databases.

Correlation between algorithm and human sorting of images.

Correlation between algorithm and monkey performance in discrimination task.