THE ENCODING OF PARTS AND WHOLES

IN THE VISUAL CORTICAL HIERARCHY

JOHAN WAGEMANS

LABORATORY OF EXPERIMENTAL PSYCHOLOGY UNIVERSITY OF LEUVEN, BELGIUM

IEA-PARIS, FRANCE

“ADAPTIVE COMPUTATIONS” SANTORINI, MAY 31, 2012

Some examples

Visual hierarchy

• features

• parts

• wholes, e.g. – objects – faces – scenes

Cortical hierarchy: Mainstream view

based on single-unit recordings (Hubel & Wiesel) tuning properties of different types of cells in different

areas of the brain functional specialization hierarchical organization

confirmed in human fMRI (modules, maps) standard view in several approaches

Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.

Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.

Serre, T., Oliva, A., & Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. Proceedings of the National Academy of Science of the USA, 104, 6424-6429.

Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.

Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.

Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.

Alternative views possible

Gestalt theory (Berlin school: Wertheimer, Köhler, Koffka) wholes are not only more than the sum of the parts wholes are also different from the sum of the parts 2-sided dependency wholes come first (e.g., global precedence effect)

more recent views Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse

hierarchies in the visual system. Neuron, 36, 791-804. Bar, M. et al. (2006). Top-down facilitation of visual recognition. Proceedings of

the National Academy of Science of the USA, 103, 449-454.

interesting characteristics from viewpoint of Gestalt theory wholes come first highly interactive highly dynamic

Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.

Bar, M. et al. (2006). Top-down facilitation of visual recognition. PNAS, 103, 449-454.

The problem

• How to understand the relationships between

parts and wholes in visual experience

• How to understand the encoding of parts and wholes in the hierarchy of visual cortex

• How to understand the relationships between cortical encoding and visual experience

My proposal

• There are different types of “Gestalts” with their own

relationships between parts and wholes, both in visual experience and in their neural encoding

• Some Gestalts seem to be encoded in low-level areas based on feedback from higher-order regions – Kourtzi, Z., Tolias, A. S., Altmann, C. F., Augath, M., &

Logothetis, N. K. (2003). Integration of local features into global shapes: Monkey and human fMRI studies. Neuron, 37, 333-346.

– Murray, S. O., Boyaci, H., & Kersten, D. (2006). The representation of perceived angular size in human primary visual cortex. Nature Neuroscience, 9, 429-434.

Eliminative versus preservative Gestalts

1. “eliminative Gestalts” – “wholes” dominate and “parts” disappear from

experience – “wholes” emerge in higher areas of the brain and

encoding of “parts” is then suppressed

2. “preservative Gestalts” – functional “wholes” also arise spontaneously and

“parts” become less functional – but still, the encoding of these “wholes” at higher levels

of the cortical hierarchy does not suppress the encoding of the “parts”

Two examples of eliminative Gestalts

• Motion silencing – Suchow, J. W., & Alvarez, G. A. (2011). Motion silences

awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019

– Poljac*, E., de-Wit*, L., & Wagemans, J. (2012). Perceptual wholes can reduce the conscious accessibility of their parts. Cognition, 123, 308-312. (*joint first authors) doi:10.1016/j.cognition.2012.01.001

• Bistable diamond – Fang, F., Kersten, D., & Murray, S. O. (2008). Perceptual

grouping and inverse fMRI activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2

– de-Wit, L. H., Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2012). Bi-stable Gestalts reduce activity in the whole of V1 not just the retinotopically predicted parts. Journal of Vision. Under revision.

Suchow & Alvarez (2011)

• Suchow, J. W., & Alvarez, G. A. (2011). Motion silences awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019

• “Best Illusion of the Year 2011”

Demonstration

Demonstration

More demonstrations

Methods

• Stimuli: – 100 dots – first stationary, then rotating back and forth for 30° – 2 phases alternating every 3 s

• Task: – observers had to adjust the rate of change during the

stationary phase to match the apparent rate of change in the moving phase

– rate of change (“silencing factor”) between 0.1 (static perceived as changing slower) and 10 (static perceived as changing faster)

Results

Interpretation

• Suchow & Alvarez:

– local mechanisms with small receptive fields – because a fast-moving object spends little time at any

one location, a local detector is afforded only a brief window in which to assess the changing object

• alternative interpretation: – objecthood – when a good “whole” is formed, the details of the

“parts” are fundamentally less accessible to conscious perception

Our study

• Poljac*, E., de-Wit*, L., & Wagemans, J. (2012). Perceptual wholes can reduce the conscious accessibility of their parts. Cognition, 123, 308-312.

(*joint first authors) doi:10.1016/j.cognition.2012.01.001

• motivation: to test this alternative interpretation with biological motion

Why biological motion?

• a prototypical case of a complex hierarchical stimulus (Johansson, 1973; Cutting & Proffitt, 1982) – multiple elements, each with their own spatio-temporal

trajectories – organized quickly and efficiently in a hierarchical configuration,

in which the motion of the local elements are coded relative to a more global structural description

• the perceptual Gestalt is constructed automatically by the

visual system (Thornton & Vuong, 2004)

• the construction of the perceptual whole implies a more efficient representation of the relationships between the parts (Tadin et al., 2002)

• inversion allows control over low-level motion trajectories (Sumi, 1984; Pavlova & Sokolov, 2000)

Demonstrations

Demonstrations

Demonstrations

Methods

• Stimuli:

– motion captured point-light treadmill walkers (Vanrie & Verfaillie, 2004)

– 70 colored dots (“confetti walker”) – upright, inverted, phase-scrambled

• Task: adjust rate of change in the test figure until it matches the range of change in the comparison figure

• Direct comparison of dynamic and static – comparison figure: scrambled – test figures: upright or inverted

Results

Discussion

• on top of the effect of static vs moving, there is a clear effect of configurality (“goodness” of the whole percept)

• cost of objecthood: the more strongly the parts are integrated into the perception of a whole object, the less accessible the changing features of the parts are (e.g., also embedded figures)

Eliminative Gestalts

• excellent example: bistable diamond

• Murray et al. key papers: – Murray, S. O., Kersten, D., Olshausen, B. A., Schrater,

P., & Woods, D. L. (2002). Shape perception reduces activity in human primary visual cortex. Proceedings of the National Academy of Sciences, 99, 15164-15169.

– Fang, F., Kersten, D., & Murray, S. O. (2008).

Perceptual grouping and inverse fMRI activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2

Demonstration

Nice features

• perceptual bi-stability: – “parts” seen to move vertically – “whole” seen to move horizontally

• switching relatively slow, perceptual states rather clear

• stable individual differences

• studied rather extensively at psychophysical level, e.g. – Lorenceau, J. & Shiffrar, M. (1992). The influence of

terminators on motion integration across space. Vision Research, 32, 263-273.

Murray et al.: Design

• present bistable diamonds

• ask observers to indicate perception of “parts”

(line segments) or “whole” (diamond)

• record BOLD responses (fMRI) in different areas and relate these to the reported percepts

Murray et al.: Results

Murray et al.: Results

Discussion

• convincing demonstration of inverse activity patterns in V1 and LOC

• interpretation? – perception of “parts” suppressed by perception of

“whole” – predictive coding framework: “explaining away”

• however – using MVPA decoding, we have shown that the

reduction of activity in V1 is global, not retinotopically specific

Preservative Gestalts

• excellent example: configural-superiority effect

• Pomerantz et al. key papers:

– Pomerantz, J. R., Sager, L. C., & Stoever, R. J. (1977). Perception of wholes and their component parts: Some configural superiority effects. Journal of Experimental Psychology: Human Perception and Performance, 3, 422-435.

– Pomerantz, J. R., & Portillo, M. C. (2011). Grouping and emergent features in vision: Toward a theory of basic Gestalts. Journal of Experimental Psychology: Human Perception and Performance, 37, 1331-1349.

• neural basis?

Kubilius et al. (2011)

• Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2011).

Emergence of perceptual Gestalts in the human visual cortex: The case of the configural superiority effect. Psychological Science, 22(10), 1296-1303.

• behavioral results

• fMRI decoding results

Behavioral results

parts corner whole

Scanning protocol

fMRI results: Retinotopic mapping

MVPA: decoding

fMRI results: decoding

fMRI results: decoding

Discussion

• behavioral configural-superiority effect

• neural configural-superiority effect: – better coding of “wholes” than “parts” in higher shape-

selective regions – better coding of “parts” than “wholes” in lower-level

retinotopic regions

• general conclusions: – at least some Gestalts emerge only at higher stages of

visual information processing – feedforward processing may be sufficient to produce

some Gestalts

Conclusion

• the encoding of parts, wholes and their relationships constitutes a serious challenge to the visual system

• the visual system appears to have developed flexible mechanisms (“adaptive computations”) with different characteristics – sometimes wholes are encoded in low-level areas

(feedback?) – sometimes wholes are encoded in high-level areas, while

parts are suppressed in low-level areas – sometimes wholes are encoded in high-level areas, while

parts are preserved in low-level areas

• further research is needed to establish the specific properties of these cases (computational reasons, boundary conditions, etc.)

THANK YOU

JOHAN.WAGEMANS@PSY.KULEUVEN.BE

WWW.GESTALTREVISION.BE

The encoding of parts and wholes �in the visual cortical hierarchy�Some examplesSlide Number 3Slide Number 4Slide Number 5Slide Number 6Visual hierarchyCortical hierarchy: Mainstream viewFelleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.Grill-Spector, K., & Malach, R. (2004). The human visual cortex. �Annual Review of Neuroscience, 27, 649-677.Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.Alternative views possibleHochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.Bar, M. et al. (2006). Top-down facilitation of visual recognition. �PNAS, 103, 449-454.The problemMy proposalEliminative versus preservative GestaltsTwo examples of eliminative GestaltsSuchow & Alvarez (2011)DemonstrationDemonstrationMore demonstrationsMethodsResultsInterpretationOur studyWhy biological motion?DemonstrationsDemonstrationsDemonstrationsMethodsResultsDiscussionEliminative GestaltsDemonstrationNice featuresMurray et al.: DesignMurray et al.: ResultsMurray et al.: ResultsDiscussionPreservative GestaltsKubilius et al. (2011)Behavioral resultsScanning protocolfMRI results: Retinotopic mappingMVPA: decodingfMRI results: decodingfMRI results: decodingDiscussionConclusionThank You