Cortical circuits for vision

Jamie Mazer

Neurobiology of Cortical Systems

Lecture 7

March 12, 2012

Readings for Thursday

How much of cortex is visual? (in primates)

Van Essen flat mapof macaque cortex

Primates are likely an extremeexample or an upper bound..

How much of cortex is visual?

Van Essen flat mapof macaque cortex

“simplified” Felleman & Van Essen hierarchy

Key concepts

• phenomenon vs. implementation vs. function

• “centrally synthesized maps”– everything we perceive must be encoded by the retina

– if so, what’s all that visual cortex doing?

– generating explicit sensory representations

– “emergent” properties seem to be a key feature of high-level sensory cortical function

– Question: Is cortex required to generate explicit or abstract properties?

– Answer: What’s emergent in the retina? What about animals with not cortex, like birds and fish?

• are there common “motifs” across sensory modalities?– computational maps in other modalites?

– what about other species? are they unique to cortex?

Retinal bipolar cells receptive fields

Retinal ganglion cell RFs (only retinal output)

Receptive fields and center-surround opponency

Receptive fields and center-surround opponency

Center-surround organization Observed phenomenon? Implementation? Function?

Receptive fields and center-surround opponency

Center-surround organization Observed phenomenon? Characteristic RF structure Implementation? Lateral inhibition Function? Spatial derivative; contrast enhancement

Behavioral consequences of center surround organization

herring gridmach bands

Behavioral consequences of center surround organization

herring gridmach bands

Thalamus: dLGN

What changes between the photoreceptors and LGN?

• transition from receptor potentials to spiking• center-surround spatial receptive fields• “color opponency” (B-Y/R-G) instead of simple cone-

based wavelength tuning• segregation into parallel processing streams

– sustained and transient– fast and slow– on and off channels– color and luminance

Which brings us to primary visual cortex (BA 17; V1)

m

visualassociation

primaryvisual

Topographic organization of V1

- retinotopy- orientation columns- occular dominance columns- non-oriented blobs (L2)- orientation topography

Thalamocortical projections and the canonical microcircuit

Primary visual cortex: simple cell orientation tuning

hubel & wiesel 1968

orientation tuned V1 neuron

MOVIE

Primary visual cortex: simple cell orientation tuning

hubel & wiesel 1968

orientation tuned V1 neuronhubel & wiesel model

Primary visual cortex: simple cell orientation tuning

hubel & wiesel 1968

orientation tuned V1 neuronhubel & wiesel model

Key failures for the feedforward model?

- contrast invariant orientation tuning

Primary visual cortex: simple cell orientation tuning

hubel & wiesel 1968

orientation tuned V1 neuronhubel & wiesel model

Hubel & Wiesel Interpretation Observed phenomenon?

preferred orientation Implementation?

linear summation of LGN cells Function?

feature detectors for edges

Primary visual cortex: simple cell orientation tuning

hubel & wiesel 1968

orientation tuned V1 neuron hubel & wiesel model

Spatial Vision Interpretation Observed phenomenon?

preferred orientation Implementation?

quasi-linear combination of LGN cells Function?

spatiotemporal filtering

• cells prefer light increments or decrements

• cells have orientation tuning

• cells have a width tuning

• cells have length tuning

• cells have speed tuning

• cells are feature detectors where the feature is a bar of a particular orientation, size and speed

• intuitively obvious, simple to understand, seems to imply obvious behavioral function

• cells prefer light increments or decrements

• cells have orientation tuning

• cells have spatial frequency tuning

• cells have temporal frequency tuning

• cells are half-wave rectified spatiotemporal filters (Gabors)

• requires some math chops to understand, but has predictive power

Feature detector model “spatial vision” model

Primary visual cortex: spatial frequency tuning

Robson, DeValois, Maffei etc..

Feature detector model “spatial vision” model

• cells prefer light increments or decrements

• cells have orientation tuning

• cells have a width tuning

• cells have length tuning

• cells have speed tuning

• cells are feature detectors where the feature is a bar of a particular orientation, size and speed

• intuitively obvious, simple to understand, seems to imply obvious behavioral function

• cells prefer light increments or decrements

• cells have orientation tuning

• cells have spatial frequency tuning

• cells have temporal frequency tuning

• cells are half-wave rectified spatiotemporal filters

• requires some math chops to understand, but has predictive power

Primary visual cortex: simple complex

hubel & wiesel 1968

simple

complex

Primary visual cortex: simple complex

hubel & wiesel 1968

simple

complex

MOVIE

Primary visual cortex: simple complex

hubel & wiesel 1968

simple

complex

hypercomplex+length tuning+length tuning+length tuning

Primary visual cortex: simple complex

hubel & wiesel 1968

“simple cells”pool center-surround neuronsto form orientation selectivity

“complex cells”pool simple cells to becomeposition or phase invariant.

and turtles all the way down…

Complex cells and the F1/F0 ratio

cats

monkeys

Skottun et al, 1991

What’s the spatial vision model got to say?

Complex cells and the F1/F0 ratio

Skottun et al, 1991

cats

monkeys

Mechler & Ringach, 2002

is this all an artifact?

Reverse correlation and the spike triggered average

Jones & Palmer, 1987

Reverse correlation and the spike triggered average

Jones & Palmer, 1987

Reverse correlation and the spike triggered average

Jones & Palmer, 1987

V1 neurons are Gabor’s and Gabor’s are optimal…

Daugman, 1985

V1 neurons are Gabor’s and Gabor’s are optimal…

Daugman, 1985

Where do Gabor’s come from and the efficient coding hypothesis

Barlow, 1972

Where do Gabor’s come from and the efficient coding hypothesis

Where do Gabor’s come from and the efficient coding hypothesis

Vinje & Gallant, 2000

Where do Gabor’s come from and the efficient coding hypothesis

Haider et al, 2010

What have we established?• simple cells

– simple cells are partially assembled from LGN afferents

– one basic flavor: Gabor

• they are bar-detectors as well (glass half empty), but

• the Gabor-model seems like a more compact framework

• complex cells– complex cells are assembled from simple cells

– strict dichotomy not likely, more likely is,

• thalamocortical direct recipient simple cells, and,

• cells that are a combination of simple and non-simple innputs

• coding in V1– sparseness is a hallmark of an efficient code

– simple cells can be learned by maximizing sparseness

– sparseness in V1 is based on center-surround (intracortical) inhibitory interations

– the neural representation is awful close to what the computer vision people call a wavelet or multiscale pyramid and is the basis for things like MPG and JPG compression…

• perhaps we need more data from more complex stimuli?

Reverse correlation, complex cells and natural scenes

Reverse correlation, complex cells and natural scenes

Problems:1. STA doesn’t really work for

natural (non-white) stimuli2. the STA is just plain “wrong”

for complex cells

Linear receptive field maps in early vision

DeAngelis et al, 1995

still orientation tuned!where’s it coming from?

Reverse correlation, complex cells and natural scenes

Problems:1. STA doesn’t really work for

natural (non-white) stimuli2. the STA is just plain “wrong”

for complex cells

Reverse correlation, complex cells and natural scenes

Problems:1. STA doesn’t really work for

natural (non-white) stimuli2. the STA is just plain “wrong”

for complex cells

Spike Triggered Covariance (STC)

What have we established?• simple cells

– simple cells are partially assembled from LGN afferents

– one basic flavor: Gabor

• they are bar-detectors as well (glass half empty), but

• the Gabor-model seems like a more compact framework

• complex cells– complex cells are assembled from simple cells

– strict dichotomy not likely, more likely is,

• thalamocortical direct recipient simple cells, and,

• cells that are a combination of simple and non-simple innputs

• coding in V1– sparseness is a hallmark of an efficient code

– simple cells can be learned by maximizing sparseness

– sparseness in V1 is based on center-surround (intracortical) inhibitory interations

– the neural representation is awful close to what the computer vision people call a wavelet or multiscale pyramid and is the basis for things like MPG and JPG compression…

• perhaps we need more data from more complex stimuli?– STRFs, STC, regression analysis, MII etc all provide new tools could complex cells and complex stimuli…

• what did I not talk about??

Direction selectivity

hubel&wiesel 1968

MOVIE

Direction selectivity is Gabor-ish too (vs. Reichardt Detector)

DeAngelis et al, 1993. 1995

Disparity/depth tuning

focal plane

near

far

foveae G. Poggio et al

MOVIE

Disparity too fits in the spatial vision view…

Note: Complex cells see anti-correlated bars differently than correlated, not true for perception…

Ohzawa et al, 1997

Is spatial vision everything?

• the high-dimensional Gabor filter model explains a lot of the neurophysiological and psychophysical data, but..– finding the right dimensions is non-trivial as we’ll see next week.

– even when the dimensions are likely identified, it’s essentially a linear or quasi-linear model and doesn’t explain a range of observed phenomena, even in V1…

Center-surround interactions in V1 – generally NOT accounted for by the standard spatial vision model.

• end-stopping, length-tuning, “hypercomplexity” (H&W)• cross-orientation inhibition (Silito et al)• divisive gain control (Carandini paper!)• curvature processing (Dobbins & Zucker)• target pop-out (Knierim & Van Essen)• attention and/or figure segmentation (Lamme et al)

What are the emergent properties of V1?

• new features extracted– orientation– binocular disparity (depth)– direction selectivity– spatial frequency– color (really “transformed”)

• new maps– orientation (columns)– ocular dominance– segregation of color info in blobs

What have we established?• simple cells

– simple cells are partially assembled from LGN afferents

– one basic flavor: Gabor

• they are bar-detectors as well (glass half empty), but

• the Gabor-model seems like a more compact framework

• complex cells– complex cells are assembled from simple cells

– strict dichotomy not likely, more likely is,

• thalamocortical direct recipient simple cells

• cells that are a combination of simple and non-simple innputs

• coding in V1– sparseness is a hallmark of an efficient code

– simple cells can be learned by maximizing sparseness

– sparseness in V1 is based on center-surround (intracortical) inhibitory interations

– the neural representation is awful close to what the computer vision people call a wavelet or multiscale pyramid and is the basis for things like MPG and JPG compression…

• perhaps we need more data from more complex stimuli?– STRFs, STC, regression analysis, MII etc all provide new tools could complex cells and complex stimuli…

• V1 exhibits multiple emergent properties

• What happens when you lose V1??

• How much of this interpretation is primate-centric?

Primate-centric view?

Andermann et al., 2011

Neill & Stryker, 2010

Shuler & Bear, 2006

- Rodents have striate and extrastriate analogues or homologues- Tuning is similar, but not identical- “Extra-retinal” effects seem more pronounced

Readings for Thursday