Extending Cognitive Architectures with Mental Imagery

AGI-09Scott LathropJohn Laird 1

2

WHY MENTAL IMAGERY?

Cognitive Architectures • Amodal, symbolic representations & computations• No general reasoning with perceptual-based

representations (Barsalou 1999, 2008)

Resulting in… • Ad-hoc reasoning in tasks rich with spatial and visual

properties• “Bolted-on” task-dependent components

What we desire…• “Best of both worlds” multi-representational, task-

independent approach• Functionality• Computational advantage

• Link perceptual-based thought and cognition

3

MENTAL IMAGERY REPRESENTATIONS

Representation Processing Uses ExampleSymbolic

Amodal

Symbolic manipulation

General Reasoning/Control

Qualitative Spatial & Visual Reasoning

feature (letter-B, line)color(letter-B, blue)

feature(box, corner)color(box, yellow)

on(letter-B, box)Quantitative spatial

Amodal/Perceptual-based

(Spatial Imagery)

Mathematical manipulation

Dynamical Systems (motion models)

Spatial Reasoning(Generic Shapes)

Facilitate building visual depictive representation

letter-B location <2,3,-2> orient 0 length 2 height 4 box location <0,0,0> orient 20 length 6 height 4 width 2

Visual depictive

Modality-specific/Perceptual-based

(Visual Imagery)

Mathematical manipulation

Depictive manipulation

Spatial Reasoning(Specific Shapes)

Visual Feature Recognition

4

SCOUT DOMAINEnemy Scout-

2

Enemy Scout-

1

Scout-1

(Agent)

Scout-2(Teammat

e)

Enemy Scout-

3

Actual Situation

Scout-2(Teammate)

Enemy Scout-1

Teammate’s ViewObserved

EnemyScout-2

Scout-1(Agent)

Agent’s View

ObservedEnemy Scout-2

Hypothesized

Enemy Scout

Hypothesized

Enemy Scout

“Key”Terrain

Imagined Situation

SCOUT DOMAIN

5

6

ARCHITECTURE

SVI

• Quantitative spatial • Scene-graph•Viewpoint

Object Map STM

Refresher

Stimulus-Based

Refresh

“Rendered”

Image

Visual-Spatial STMObjec

t SetObjA,ObjB,…

Feature

Set

Spatial

Set

Feature

Set

Object

SetObjA1,ObjA2,…

Spatial

Set

• Visual depictive• Set of 2D bitmaps

Visual Buffer STM

• Shape•Texture• Spatial configuration

Perceptual LTM

“What” Inspectors

“Where” Inspectors

Soar

Symbolic STM•Goals

•Current State

Process Productions

MemorySymbolsVisual Symbols

DataPath

Control Path

Symbolic (Procedural) LTM IMAGERY OPERATORSConstruct

GenerateTransformInspect

TASK OPERATORS

Constructor

Manipulator

8

Pixel-level rewrites (Furnas, 2000)• Production-like rules (LHS/RHS) • Shared image (i.e. working memory)• LHS and RHS are pixel patterns• Conflict resolution scheme (sequencing)• Processing may match other

orientations• Active manipulation of shapes

Contrast with other image processing• Computer vision

o Sentential manipulations o Filters (e.g. Gaussian) where each pixel

is rewritten as a specific function of its neighbors’ values

• Cellular Automatao Finite State Machineso Next state of a cell based on current

state and state of its neighborso Rather than rewriting many local

configurations at once

= “don’t care”

“2x2 reduce

”

“nibble up”

“nibble left”

DEPICTIVE MANIPULATIONS(1 of 2)

9

Takes specific shape of no-go terrain and obstacles (yellow) into account Attention window shift controlled by Soar Pixel rewrite rules sent from Soar to SVI Determine approximate path coverage by overlaying each scout’s view frustum Simulate alternate view orientations to determine “best” path coverage

*

Steep Terrain

All Other

Obstacles

Mark Obstacles

**

**

Layer 0-1

*

Distance Field Flood

Layer 0-1

Layer 1-2

Mark Path Start-

Layer 2

Layer 2-1

Layer 1-0

Layer 1-0

DEPICTIVE MANIPULATIONS(2 of 2)

10

0 50 100

150

200

250

300

350

-1-0.8-0.6-0.4-0.2

00.20.40.6

Soar+SVISoar-SVIObserverTracker

Simulation Time

Info

rmat

ion

RESULTSAMOUNT OF INFORMATION

SUMMARY General reasoning with symbolic and perceptual-

based representations where the reasoning emerges from the combination of the representations

Soar• Symbolic representations and computations• Goals and high-level knowledge• Controls imagery processing

Spatial and Visual Imagery (SVI)• Quantitative spatial and visual depictive representations and

computations• Leverages perceptual mechanisms

Future Work - Push architecture closer to sensory input (Robotics)

11