mental imagery and human memory - universität...

Mental Imagery andHuman Memory

Seminar Spatial CognitionSummer Term 2003

Thomas Barkowsky

What is mental imagery?

The trunk exampleYou've got a car, numerous pieces of baggage, sports equipment, …

A geographic exampleIn which direction with respect to Bremen are located:

Berlin?London?Vienna? …

Outlook

Mental imagerydefinitionproperties

Human memorylong- term memoryworking memoryshort- term memory

Imagery Modelspsychologyartificial intelligence

Mental imagery – A definition

Mental imagery is "the mental invention or recreation of an experience that in at least some respects resembles the experience of actually perceiving an object or an event, either in conjunction with, or in absence of, direct sensory stimulation" (Finke, 1989)

… invention or recreation …

mental image constructionbased on memorycombined from elements

image inspectionutilization of image constructed

image modificationalternatives in constructed and inspected images

… of an experience …

issue of interest: cognitive principles (not experience!)indirect measurement!are mental images 'real'?cf. imagery debate

… visual perception …

imagery tightly coupled with visionimagery with perception

e.g. combination of seen and mentally constructed elements

imagery without perceptionpure construction from memory

no perception without imageryimagery precondition for vision processes

Mental rotation

[Shepard & Metzler, 1971]

Mental scanning

[Kosslyn et al., 1978]Learn artificial mapScanning the imagined mapReaction time proportional to distance

Conclusion

Spatio-analogical representation in the mindCritics: difference between

experienceexperimental behaviorphysical realization in the mind / brain

The imagery debate

Do 'mental images' have a specific representation format?

dual codingjust propositional coding

e.g. "who 'looks' at the internal picture?"evidence from patients with brain damagesmental imagery quite accepted

Mental images: Characteristics

not epiphenomenalquasi-pictorial mental representation structures

not retrieved in a ready-made formmental constructionsmental storage differs from form in usageuse of pieces of knowledge

Mental images: Characteristics

composition from well-organized pieces of knowledge

hierarchical structure in long-term memoryorganization in mental storage influences image construction time

image construction from both pictorial and propositional information

e.g. image construction from verbal descriptions

Unifying principles

implicit encodingperceptual equivalencespatial equivalencetransformational equivalencestructural equivalence

[Finke, 1989]

Implicit encoding

imagery allows to exhibits information not explicitly stored in memoryretrieve implicit information from memoryadvantageous w.r.t. space requirementcf. diagrammatic reasoning in AI

Perceptual equivalence

"Imagery is functionally equivalent to perception to the extent that similar mechanisms in the visual system are activated when objects or events are imagined as when the same objects or events are actually perceived"Imagery is essential for vision, not the other way around!

Spatial equivalence

"The spatial arrangement of the elements of a mental image correspond to the way objects and their parts are arranged on actual physical surfaces or in an actual physical space"Visual buffer structure both in vision and in imagery

Transformational equivalence

Cf. mental rotation"Imagined transformations and physical transformations exhibit corresponding dynamic characteristics and are governed by the same laws of motion"Cf. mental animationAlso used in diagrammatic reasoning systems

Structural equivalence

Structure of images corresponds to that of real objects

organized, coherent structurereorganizationreinterpretation

e.g. in configuration tasks, anticipation and planning of motion and change

Human memory

long-term memoryworking memoryshort-term memory

Working memory [Baddeley 1986]

information from the sensesinformation from long-term memory(at least) 2 sub-systems

visuo-spatial sketchpadarticulatory loop

Centralexecutive

Articulatoryloop

Visuo- spatialscratchpad

Central executive

mediates betweenworking memory subsystems long-term memory

controls rehearsal processestranslates between modalities

Visuo-spatial sketchpad

short-term memorytransient structure

fading (approx 1.5 sec)rehearsal

subdivision by Logie, 1995visual informationspatial information

Capacity restriction in working memory

7+/-2 items (Miller, 1956)4 items (Cowan, 2001)swapping with long-term memorychunking in working memory

Long-term memory

network structurespreading activationhierarchical organization

conceptual hierarchiesindividual hierarchies

Memory for mental imagery

imagery takes place in working memoryimage construction and modification

information from long-term memoryactivation of information

Memory for mental imagery (contd.)

working memory consists of long-term memory and short-term memoryimage construction in visual buffer = visuo-spatial sketchpadimage inspection in visual bufferimagery processes in central executive

Integration of memory systems

short-termmemory

(not activated)

long-termmemory

(activated)workingmemory

activation

representation/

maintenance

Imagery models

Psychological modelsimplemented (Kosslyn, 1980)conceptual (Kosslyn, 1994)

Artificial intelligence modelssimulation (Funt, 1980)spatial reasoning (Khenkhar, 1991)computational imagery (Glasgow & Papadias, 1992)

Kosslyn, 1980

cathode-ray tube metaphorsurface vs. deep representation

surface representationshort- term / working memoryquasi- pictorial positional structure (raster matrix)limited spatial extendapprox. circular shapedefinite grain and limited resolutionresolution decreases towards peripheryfading of image parts

Kosslyn, 1980 (contd.)

deep representationlong-term memory2 kinds of deep representations

perceptual / literal image representationskeletal encodingsindividual encodingshierarchical organization

discursive description (propositions)qualitative spatial descriptionsinformation about parts, location, and category

Kosslyn, 1980 (contd.)

image constructionPICTURE, FIND, PUT, IMAGE (control)

image inspectionLOOKFOR (control), RESOLUTION, REGENERATE

image modificationZOOM, PAN, ROTATE, SCAN

Kosslyn, 1994

conceptual modelbased on high-level vision processes (neurospsychology)interaction between subsystemsimplementation only of partial aspects

Kosslyn, 1994 (contd.)

Kosslyn, 1994 (contd.)

1. pattern code

2.

2.

3.

3. 4.

image construction

Kosslyn, 1994 (contd.)

1. pattern code

2.

2. 3. 4.

adding additional parts

5.

Kosslyn, 1994 (contd.)

2.

image inspection

1.

1.2.

spatial properties

object properties3.

Funt, 1980 (WHISPER)Reasoning about mechanicsPrediction of motionPositional representation of initial stateRepresentation of intermediate statesComponents

diagramhigh-level reasoner'retina'

Funt, 1980 (contd.)high-level reasoner

procedural propositional problem solverqualitative physical knowledge

Funt, 1980 (contd.)

'retina'positional diagrammatic structurecircular structure of concentric ringsparallel processorsneighborhood structuresupervising processorvariable position

Funt, 1980 (contd.)

retina: percptual primitivesfocus center of entitycheck for symmetrycheck for congruencescaling, rotationconnections, collisionsboundary features (convexity, slope)identify neighboring entities

simulation of tentative movement of objects

Depic-2D (Khenkhar, 1991)

hybrid system: propositional and pictorial reasoningregular rectangular cell structuredepiction of objects in cells

Depic-2D (Khenkhar, 1991)

depiction of objects in cellsuse of neighborhood structure

spreading activatione.g. buffering

Depic-2D (Khenkhar, 1991)

Computational imagery (Glasgow & Papadias, 1992)

diagrammatic reasoning architecture for technical applications

chemistrygeography

3 subsystems

visualrepresentation

interpret

store

spatialrepresentation

deeprepresentation

perceive reconstruct

Computational imagery (contd.)

surface representation subdividedspatial representation

'where' aspects: location w.r.t. other objectsvisual representation

'what' aspects: shape, color, texture

visualrepresentation

interpret

store

spatialrepresentation

deeprepresentation

perceive reconstruct

Computational imagery (contd.)

deep representationlong-term storage (before and after processing)hierarchical organization

Computational imagery (contd.)visual representation

positional structure: occupancy arraysshape, distance, (relative) distancestexture, color, surface orientation3D, viewer independentstandard computer graphics operations

Computational imagery (contd.)

spatial representationqualitative relationships between objectsrelational structure: symbolic arrays

Computational imagery (contd.)

topology and (coarse) orientationno exact shapes and sizeshierarchiesprocesses for

information transferplacing, detecting, moving, deletingchecking for adjacencypattern recognition by attention control

Thanks!

questions?comments?