PART-WHOLE RELATIONSHIPS

IN VISUAL PERCEPTION

JOHAN WAGEMANS

LABORATORY OF EXPERIMENTAL PSYCHOLOGY UNIVERSITY OF LEUVEN

PRINCETON, NOVEMBER 22, 2011

Some examples

Some examples

Some examples

Some examples

Overview

• Part 1: historical and conceptual background

• Part 2: some recent empirical studies – bistable diamonds

• Murray et al. (2002) • Fang et al. (2008)

– motion silencing • Suchow & Alvarez (2011) • Poljac et al. (subm.)

– configural-superiority effect • Kubilius et al. (2011)

• Part 3: general discussion

HISTORICAL AND CONCEPTUAL BACKGROUND

PART 1

How Gestalt psychology started

Wertheimer, M. (1912). Experimentelle Studien über das Sehen von Bewegung. Zeitschrift für Psychologie, 61, 161-265. phi motion

Steinman, R. M., Pizlo, Z., & Pizlo, F. J. (2000). Phi is not beta, and why Wertheimer’s discovery launched the Gestalt revolution. Vision Research, 40, 2257-2264.

http//psych.purdue.edu/magniphi/ key role

phi as pure motion, not a displacement between two objects

phi as a process, “an across in itself”, that cannot be composed from the usual optical contents

Max Wertheimer (1880-1943)

A radical vision

emerging Gestalt theory not Gestalt qualities added to the primary sensations not Gestalts as more than the sum of the parts but Gestalts as different from the sum of the parts often the whole is grasped even before the individual parts

enter consciousness a structured unit emerges as a whole

psychological facts and physiological hypotheses went hand-in-hand continuous whole-processes rather than associated

combinations of elementary excitations specifically: some kind of physiological short circuit, and a

flooding back of the current flow, creating a unitary continuous whole-process

Some early Gestalt history

Koffka, K. (1915). Beitrage zur Psychologie der Gestalt. III. Zur Grundlegung der Wahrnehmungspsychologie. Eine Auseinandersetzung mit V. Benussi. Zeitschrift für Psychologie, 73, 11-90. implications of this view primary relations

no longer stimulus ~ sensation but stimulus pattern ~ perceived whole

perceived wholes not constructed in the mind from elementary sensations but direct experience-correlates emerging in the brain

Kurt Koffka (1886-1941)

Some early Gestalt history

Köhler, W. (1920). Die physischen Gestalten in Ruhe und im stationären Zustand. Eine natur-philosophische Untersuchung. Braunschweig, Germany: Vieweg.

decisive step: real physical Gestalts in the brain

strong Gestalts the mutual dependence among the parts is so great that

no displacement or change of state is possible without influencing all the other parts of the system

in fact: there are no parts at all, only interacting moments of structure that carry one another

psychophysical isomorphism psychological facts and the brain events that underlie

them are similar in all of their structural characteristics specifically: visual Gestalts result from a single Gestalt

process in which the whole optic sector from the retina onwards is involved, including transverse functional connections

in fact: the brain described as a self-organizing physical system

Wolfgang Köhler

(1887-1967)

From speculation to facts

Köhler, W., & Held, R. (1949). The cortical correlate of pattern vision. Science, 110, 414-419.

first recordings of visual currents, picked up by an electrode at the scalp of human observers

“electrical field theory”

Lashley, K. S., Chow, K. L., & Semmes, J. (1951). An examination of the electrical field theory of cerebral integration. Psychological Review, 58, 123-136.

more direct test of electrical field theory rationale: insulate part of a cortical field and test for consequent

disturbances of function metallic strips and pins inserted in macaque cortex almost no effect on post-operative retention of object discrimination devastating blow to

electrical field theory basic isomorphism postulate of Gestalt theory

Single-neuron doctrine

Hubel & Wiesel: big success

Barlow, H. (1972): Single units and sensation: A neuron doctrine for perceptual psychology. Perception, 1, 371-394.

our perceptions are caused by the activity of a small number of neurons the activity of a single neuron is related quite simply to our subjective

experience

reductionist, elementalist approach which Gestalt theorists had banished

mapping of responses of single neurons in LGN, striate and extrastriate cortex in cat and monkey

tuning to simple stimulus attributes (e.g., orientation)

single neurons interpreted as “feature detectors” (e.g., line detectors, edge detectors)

Further developments

single-unit recording flourished tuning properties of different types of cells in different areas of

the brain functional specialization hierarchical organization

retinotopy decreases invariance increases complexity of the “features” increases

confirmed in human fMRI (modules, maps) standard view

Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.

Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.

Serre, T., Oliva, A., & Poggio, T. (2007). A feedforward architecture accounts for rapid categorization. Proceedings of the National Academy of Science of the USA, 104, 6424-6429.

Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.

Grill-Spector, K., & Malach, R. (2004). The human visual cortex. Annual Review of Neuroscience, 27, 649-677.

Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.

Re-emergence of Gestalt issues

surround influences from outside classic receptive field (cRF) Allman, J., Miezin, F., & McGuinness, E. (1985). Direction- and

velocity-specific responses from beyond the classical receptive field in the middle temporal visual area (MT). Perception, 14, 105-126.

neural responses to illusory contours

von der Heydt, R., Peterhans, E., & Baumgartner, G. (1984). Illusory contours and cortical neuron responses. Science, 224, 1260-1262.

neural responses to figure-ground organization

Zhou, H., Friedman, H. S., & von der Heydt, R. (2000). Coding of border-ownership in monkey visual cortex. Journal of Neuroscience, 20, 6594-6611.

Allman, J. et al. (1985). Direction- and velocity-specific responses from beyond the classical receptive field in the middle temporal visual area (MT).

Perception, 14, 105-126.

Allman, J. et al. (1985). Direction- and velocity-specific responses from beyond the classical receptive field in the middle temporal visual area (MT).

Perception, 14, 105-126.

von der Heydt, R. et al. (1984). Illusory contours and cortical neuron responses. Science, 224, 1260-1262.

Zhou, H. et al. (2000). Coding of border-ownership in monkey visual cortex. Journal of Neuroscience, 20, 6594-6611.

A modern view on a radical vision

more general models Hochstein, S., & Ahissar, M. (2002). View from the top:

Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.

Bar, M. et al. (2006). Top-down facilitation of visual recognition. Proceedings of the National Academy of Science of the USA, 103, 449-454.

interesting characteristics from viewpoint of Gestalt theory “wholes” come first highly interactive highly dynamic not limited to visual areas

Hochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.

Bar, M. et al. (2006). Top-down facilitation of visual recognition. PNAS, 103, 449-454.

Interim conclusions

• “parts” versus “wholes” constitutes a prominent theme in vision science, from Gestalt psychology up until today

• the pendulum seems to have swung back-and-forth

• new attempts towards a synthesis are emerging

• new research inspired by these views

Some conceptual points

• “parts” and “wholes” are used generically (not

just parts of objects)

• parts: features, components, constituents • wholes: objects, compositions, configurations,

Gestalts

• parts are generally smaller than wholes

• hierarchical relationship: – parts belong to wholes – wholes consist of parts

BISTABLE DIAMONDS: MURRAY ET AL. (2002) FANG ET AL. (2008)

PART 2A

Murray et al.: Key papers

• Murray, S. O., Kersten, D., Olshausen, B. A.,

Schrater, P., & Woods, D. L. (2002). Shape perception reduces activity in human primary visual cortex. Proceedings of the National Academy of Sciences, 99, 15164-15169.

• Fang, F., Kersten, D., & Murray, S. O. (2008).

Perceptual grouping and inverse fMRI activity patterns in human visual cortex. Journal of Vision, 8(7):2, 2-9. doi: 10.1167/8.7.2

Demonstration

Demonstration

Demonstration

Nice features

• perceptual bi-stability: – “parts” seen to move vertically – “whole” seen to move horizontally

• switching relatively slow, perceptual states rather clear

• stable individual differences

• studied rather extensively at psychophysical level, e.g. – Lorenceau, J. & Shiffrar, M. (1992). The influence of

terminators on motion integration across space. Vision Research, 32, 263-273.

Murray et al.: Design

• present bistable diamonds

• ask observers to indicate perception of “parts”

(line segments) or “whole” (diamond)

• record BOLD responses (fMRI) in different areas and relate these to the reported percepts

Murray et al.: Results

Murray et al.: Results

Murray et al.: Results

Murray et al.: Results

Interim conclusions

• convincing demonstration of inverse activity patterns in V1 and LOC

• interpretation? – perception of “parts” suppressed by perception of

“whole” – direct interaction between V1 and LOC? – are reductions of V1 activity caused by feedback from

LOC? – are reductions in V1 necessary for the perception of the

whole?

MOTION SILENCING: SUCHOW & ALVAREZ (2011) POLJAC, DE-WIT, & WAGEMANS (SUBM.)

PART 2B

Suchow & Alvarez (2011)

• Suchow, J. W., & Alvarez, G. A. (2011). Motion silences awareness of visual change. Current Biology, 21(2), 140-143. doi:10.1016/j.cub.2010.12.019

• “Best Illusion of the Year 2011”

Demonstration

Demonstration

More demonstrations

Methods

• Stimuli: – 100 dots – first stationary, then rotating back and forth for 30° – 2 phases alternating every 3 s

• Task: – observers had to adjust the rate of change during the

stationary phase to match the apparent rate of change in the moving phase

– rate of change (“silencing factor”) between 0.1 (static perceived as changing slower) and 10 (static perceived as changing faster)

Results

Interpretation

• Suchow & Alvarez:

– local mechanisms with small receptive fields – because a fast-moving object spends little time at any

one location, a local detector is afforded only a brief window in which to assess the changing object

• alternative interpretation: – objecthood – when a good “whole” is formed, the details of the

“parts” are fundamentally less accessible to conscious perception

Larger framework

• this idea is consistent with

– rapid access to the gist of a given image (e.g., Thorpe et al., 1996)

– sometimes surprisingly limited access to parts that make up the interpretation (e.g., Rensink et al., 1997)

• perhaps these reflect 2 sides of the same coin: providing only the most critical information (Hochstein & Ahissar, 2002)

Our study

• motivation: to test this alternative interpretation and more general idea with biological motion

• Poljac, E., de-Wit, L., & Wagemans, J. (submitted). Perceptual wholes can reduce the awareness of their changing parts.

Why biological motion?

• a prototypical case of a complex hierarchical stimulus (Johansson, 1973; Cutting & Proffitt, 1982) – multiple elements, each with their own spatio-temporal

trajectories – organized quickly and efficiently in a hierarchical configuration,

in which the motion of the local elements are coded relative to a more global structural description

• the perceptual Gestalt is constructed automatically by the

visual system (Thornton & Vuong, 2004)

• the construction of the perceptual whole implies a more efficient representation of the relationships between the parts (Tadin et al., 2002)

• inversion allows control over low-level motion trajectories (Sumi, 1984; Pavlova & Sokolov, 2000)

Demonstrations

Demonstrations

Demonstrations

Methods

• Stimuli:

– motion captured point-light treadmill walkers (Vanrie & Verfaillie, 2004)

– 70 colored dots (“confetti walker”) – upright, inverted, phase-scrambled

• Task: adjust rate of change in the test figure until it matches the range of change in the comparison figure

• Experiment 1: all static – comparison figure: scrambled – test figures: upright, inverted or scrambled

• Experiment 2: dynamic and static – comparison figure: scrambled – test figures: upright or inverted

Results: Experiment 1

Results: Experiment 2

Discussion

• on top of the effect of static vs moving, there is

a clear effect of configurality (“goodness” of the whole percept)

• cost of objecthood: the more strongly the parts are integrated into the perception of a whole object, the less accessible the changing features of the parts are (e.g., also embedded figures)

Interim conclusions

• two sides of the same coin:

– the rapid extraction of a perceptual Gestalt – the inaccessibility of the parts that make up

that Gestalt

• general principle: – human vision provides only the most useful

level of abstraction to conscious awareness

CONFIGURAL-SUPERIORITY EFFECT KUBILIUS ET AL. (2011)

PART 2C

Point of departure

• configural-superiority effect

– Pomerantz, J. R., Sager, L. C., & Stoever, R. J. (1977). Perception of wholes and their component parts: Some configural superiority effects. Journal of Experimental Psychology: Human Perception and Performance, 3, 422-435.

– Pomerantz, J. R., & Portillo, M. C. (2011). Grouping and emergent features in vision: Toward a theory of basic Gestalts. Journal of Experimental Psychology: Human Perception and Performance, 37, 1331-1349.

Kubilius et al. (2011)

• Kubilius, J., Wagemans, J., & Op de Beeck, H. P. (2011). Emergence of perceptual Gestalts in the human visual cortex: The case of the configural superiority effect. Psychological Science, 22(10), 1296-1303.

• behavioral results

• fMRI decoding results

Behavioral results

parts corner whole

Scanning protocol

fMRI results: Retinotopic mapping

MVPA: decoding

fMRI results: decoding

fMRI results: decoding

Interim conclusions

• behavioral configural-superiority effect

• neural configural-superiority effect: – better coding of “wholes” than “parts” in higher shape-

selective regions – better coding of “parts” than “wholes” in lower-level

retinotopic regions

• general conclusions: – at least some Gestalts emerge only at higher stages of

visual information processing – feedforward processing may be sufficient to produce

some Gestalts

GENERAL DISCUSSION

PART 3

Some tentative conclusions

• first set of studies: – “wholes” dominate and “parts” disappear from experience – “wholes” emerge in higher areas of the brain and encoding of “parts” is

then suppressed

• second set of studies: – functional “wholes” also arise spontaneously and “parts” become less

functional – but still, the encoding of these “wholes” at higher levels of the cortical

hierarchy does not suppress the encoding of the “parts”

• more generally: – not all Gestalts are created equal – probably not all Gestalts are created equally – we seem to get better handles to understand the relations between

“parts” and “wholes” in visual perception – a lot of work remains to be done

Announcement: VSS symposium

Friday, May 11, 2012 “Part-whole relationships

in visual cortex”

• Johan Wagemans: “Part-whole relationships in vision science: A brief historical review and conceptual analysis”

• Charles E. Connor: “Ventral pathway visual cortex: Representation by parts in a whole object reference frame”

• Scott O. Murray: “Long-range, pattern-dependent contextual effects in early human visual cortex”

• James R. Pomerantz: “The computational and cortical bases for configural superiority”

• Jacob Feldman: “Computational integration of local and global form” • Shaul Hochstein: “The rise and fall of the Gestalt gist”

THANK YOU

JOHAN.WAGEMANS@PSY.KULEUVEN.BE

WWW.GESTALTREVISION.BE

Part-Whole Relationships �in Visual Perception�Some examplesSome examplesSome examplesSome examplesOverviewhistorical and conceptual backgroundHow Gestalt psychology startedA radical visionSome early Gestalt historySome early Gestalt historyFrom speculation to factsSingle-neuron doctrineFurther developmentsFelleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1, 1-47.Grill-Spector, K., & Malach, R. (2004). The human visual cortex. �Annual Review of Neuroscience, 27, 649-677.Serre, T. et al. (2007). A feedforward architecture accounts for rapid categorization. PNAS, 104, 6424-6429.Re-emergence of Gestalt issuesAllman, J. et al. (1985). Direction- and velocity-specific responses from beyond the classical receptive field in the middle temporal visual area (MT). Perception, 14, 105-126.Allman, J. et al. (1985). Direction- and velocity-specific responses from beyond the classical receptive field in the middle temporal visual area (MT). Perception, 14, 105-126.von der Heydt, R. et al. (1984). Illusory contours and cortical neuron responses. Science, 224, 1260-1262.Zhou, H. et al. (2000). Coding of border-ownership in monkey visual cortex. Journal of Neuroscience, 20, 6594-6611.A modern view on a radical visionHochstein, S., & Ahissar, M. (2002). View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron, 36, 791-804.Bar, M. et al. (2006). Top-down facilitation of visual recognition. �PNAS, 103, 449-454.Interim conclusionsSome conceptual pointsBistable diamonds:�Murray et al. (2002)�Fang et al. (2008)Murray et al.: Key papersDemonstrationDemonstrationDemonstrationNice featuresMurray et al.: DesignMurray et al.: ResultsMurray et al.: ResultsMurray et al.: ResultsMurray et al.: ResultsInterim conclusionsMotion silencing:�Suchow & alvarez (2011)�Poljac, De-wit, & wagemans (subm.)Suchow & Alvarez (2011)DemonstrationDemonstrationMore demonstrationsMethodsResultsInterpretationLarger frameworkOur studyWhy biological motion?DemonstrationsDemonstrationsDemonstrationsMethodsResults: Experiment 1Results: Experiment 2DiscussionInterim conclusionsConfigural-superiority effect�Kubilius et al. (2011)Point of departureKubilius et al. (2011)Behavioral resultsScanning protocolfMRI results: Retinotopic mappingMVPA: decodingfMRI results: decodingfMRI results: decodingInterim conclusionsGeneral discussionSome tentative conclusionsAnnouncement: VSS symposiumThank You