The Appeal of Parallel The Appeal of Parallel Distributed ProcessingDistributed Processing

J.L. McClelland, D.E. Rumelhart, and G.E. Hinton

인지과학 협동과정 강소영

ContentsContents

1. Introduction 2. Parallel Distributed Processing 3. Examples of PDP Models 4. Representation and Learning In PDP Models 5. Origins of Parallel Distributed Processing

1. Introduction1. Introduction

Multiple Simultaneous Constraints Reaching and Grasping The Mutual Influence of Syntax and Semantics Simultaneous Mutual Constraints in Word Recognition Understanding the Interplay of Multiple Sources of

Knowledge

The Mutual Influence of The Mutual Influence of Syntax and SemanticsSyntax and Semantics Syntactic constraint

The boy the man chased kissed the girl

Semantic constraint I saw the grand canyon flying to New York I saw the sheep grazing in the field

Mutual constraint within each of these domains I like the joke I like the drive I like to joke I like to drive

Simultaneous Mutual Constraints Simultaneous Mutual Constraints in Word Recognitionin Word Recognition

Selfridge’s example Paradox:

How can we get the process started?

Solution: our perceptual system is capable of exploring all these

possibilities without committing itself to one until all of the constraints are taken into account

Understanding the Interplay of Understanding the Interplay of Multiple Sources of KnowledgeMultiple Sources of Knowledge

Knowledge Structure scripts (Schank 1976) frames (Minsky 1975) schemata (Norman and Bobrow 1976; Rumelhart 1975)

Most everyday situations cannot be rigidly assigned to just a single script Interplay between a number of different sources of in

formation ex) birthday party at a restaurant

The generative capacity of human understanding in novel situations --> interact with each other

2. 2. Parallel Distributed Parallel Distributed ProcessingProcessing Properties of the tasks that people are good at.

A number of different pieces of information must be kept in mind at once.

Each plays a part, constraining others and being constrained by them

Assumption of PDP model: interactions of a large number of simple processing

elements each sending excitatory and inhibitory signals to other

units.

Elements of model unit, activation, interaction among units

PDP Models: Cognitive Science or NeuroscPDP Models: Cognitive Science or Neuroscience ience

The appeal of PDP : Computationally sufficient and psychologically

accurate mechanistic accounts of the phenomena of human cognition

PDP models have radically altered the way we think about the time course of processing the nature of representation the mechanisms of learning

Microstructure of CognitionMicrostructure of Cognition

Parallel Distributed Model offer alternatives to serial models of the microstructure of cognition. They do not deny that there is a macrostructure

Objects referred to in macrostructural models of cognitive processing are seen as approximate descriptions of emergent properties of the microstructure

3. Examples of PDP Models3. Examples of PDP Models

Recent application of PDP Motor control, perception, memory, language

PDP mechanisms are used to provide natural accounts of the exploitation of multiple, simultaneous, mutual constraint

3.1 Motor Control 3.1 Motor Control

Hinton’s stick person Two constraints on the task

the tip of the forearm must touch the object center of gravity over the foot

Each processor receives two information how far the tip of the hand was from the target where the center of gravity was with respect to the foot

Combination of joint angles

3.2 Perception3.2 Perception

Stereoscopic Vision Random Dot Stereogram --> Depth Perception

Perceptual Completion of Familiar Patterns Completion of Novel Patterns

Marr and Poggio (1976) explain the perception of depth in random-dot stereo

grams Two general principles about the visual world

Stereoscopic VisionStereoscopic Vision

Perceptual Completion of Perceptual Completion of Familiar PatternsFamiliar Patterns Perception is influenced by familiarity

Less time ambiguous lower level information to fill in missing lower-level information phonemic restoration effect

visual perception of words (McClelland and Rumelhart 1981)

Assumption of model detectors for the visual features

Two hypotheses or activation mutually consistant support each other mutually inconsistant weaken each other

two kinds of inconsistency between-level inconsistency

between-level inhibition

mutual exclusion competitive inhibition

Completion of Novel PatternsCompletion of Novel Patterns

Result of word perception model exhibits perceptual facilitation to pronounceable nonwo

rds as well as words general principles or rules can emerge from the interact

ions of simple processing elements. does not implement exactly any of the systems of ortho

graphic rules that have been proposed by linguists or psychologists

PDP models may provide more accurate accounts of the details of human performance than models based on a set of rules representing human competence

3.3 Retrieving Information 3.3 Retrieving Information From MemoryFrom Memory

Content Addressability Graceful Degradation Default Assignment Spontaneous Generalization

Jets and Sharks ModelJets and Sharks Model

4. Representation and 4. Representation and Learning In PDP ModelsLearning In PDP Models What is the stored knowledge that gives rise to

that pattern of activation? The difference between PDP models and other

models of cognitive processes others: knowledge is stored as a static copy of a pattern PDP:

the patterns themselves are not stored what is stored is the connection strengths between

units that allow these patterns to be re-created

Local Versus Distributed Local Versus Distributed RepresentationRepresentation Distributed Representation

The knowledge about any individual pattern is not stored in the connections of a special unit reserved for that pattern, but is distributed over the connections among a large number of processing units.

Units are conceptual primitives Units have no particular meaning as individuals

Pattern Associator --> Hebbian Rule

Attractive Properties of Pattern AsAttractive Properties of Pattern Associator Modelssociator Models Uncorrelated patterns do not interact with each oth

er, but more similar ones do if we present the same pair of patterns over and ov

er, but each time we add a little random noise to each element of each member of the pair, the system will automatically learn to associate the central tendency of the two patterns and will learn to ignore the noise

What will be stored will be an average of the similar patterns with the slight variations removed.

Extracting the Structure of an EnsExtracting the Structure of an Ensemble of Patternsemble of Patterns Distributed Model

if there are regularities in the correspondences between pairs of patterns, the model will naturally extract these regularities.

Language Learning Model - learning past tense creation of regular past tenses of new verbs overregularization of the irregular verbs same phenomena as what is shown in children’s past te

nse acquisition we can see how the acquisition of performance that con

forms to linguistic rules can emerge from a simple, local, connection strength modulation process

5. Origins of Parallel 5. Origins of Parallel Distributed ProcessingDistributed Processing

Jackson(1869/1958) and Luria(1966) distributed, multilevel conceptions of processing systems dynamic functional system

Hebb(1949) and Lashley(1950) “there are no special cells reserved for special memories”

Rosenblatt(1959, 1962) and Selfridge(1955) perceptron Pandemonium : importance of interactive processing

Anderson, Grossberg, Longuet-Higgins (60’s, 70’s) concept learning competitive learning mechanism distributed memory models

Marr and Poggio(1976) Morton’s logogen model(1969)

one of the first models to capture concretely the principle of interaction of different sources of information

Marseln-Wilson(1978) empirical demonstrations of interaction between different levels

of language processing

Levin’s Proteus model(1976) virtues of activetion-competition model

Feldman and Ballard(1982) Hofstadter(1979, 1985) Sutton and Barto(1981) --> delta rule Hopfield(1982)