episodic memory: why and how?

MÁTÉ LENGYEL

Computational and Biological Learning LabDepartment of Engineering

University of Cambridge

EPISODIC MEMORY: WHY AND HOW?(THE POWERS AND PERILS OF BAYESIAN INFERENCE IN THE BRAIN)

Máté Lengyel: Episodic memory: why and how? BCCN 2009, 3 October 2009 http://www.eng.cam.ac.uk/~m.lengyel

EPISODIC MEMORY: AN EXAMPLE

2

I raised to my lips a spoonful of the tea in which I had soaked a morsel of the cake. ... And suddenly the memory returns. The taste was that of the little crumb of madeleine which on Sunday mornings at Combray, when I went to say good day to her in her bedroom, my aunt Léonie used to give me, dipping it first in her own cup of real or of lime-flower tea.

Marcel Proust: À la recherche du temps perdu

http://www.eng.cam.ac.uk/~m.lengyel




2



PART I: WHY DO WE HAVE SUCH MEMORIES?! specific personal experiences! organised into sequences of events





2



PART I: WHY DO WE HAVE SUCH MEMORIES?! specific personal experiences! organised into sequences of events

PART II: HOW DOES THIS HAPPEN?! memories laid down in the past ! recalled in response to a cue




ON THE USE OF MEMORIES

3





3

ah, those nice days back in Combray





3

experiencedata

predictions

ah, those nice days back in Combray

representation in memorysufficient statistics





3

experiencedata

predictions

if I soak my cake in my tea ! it will taste good

representation in memorysufficient statistics





3

experiencedata

predictions


model of the environmentsufficient statistics

‘semantic’ memory:





3

experiencedata

predictions

‘episodic’ memory:select episodes

data points








3

experiencedata

predictions


data points?if I soak my cake in my tea !

it will taste good







3

experiencedata


data points

planning for the futuresequential decision making

?if I soak my cake in my tea !

it will taste good







3

experiencedata


data points


?

what shall I do next to taste something nice in the end?







3

experiencedata


data points


?




! delayed rewards! temporal credit

assignment





3

experiencedata

learning control


data points


?





assignment





3

experiencedata

learning control


data points


?




clever hard


assignment





3

experiencedata

learning control


data points


?




clever

dull

hard

easy! delayed rewards! temporal credit

assignment



Máté Lengyel: Episodic memory: why and how? BCCN 2009, 3 October 2009 http://www.eng.cam.ac.uk/~m.lengyel 4

SEQUENTIAL DECISION-MAKING UNDER UNCERTAINTYLengyel & Dayan, NIPS 2007




SEMANTIC MEMORY: MODEL-BASED CONTROL

5

Lengyel & Dayan, NIPS 2007





5

! learns a model of the environment(posterior distribution over parameters, etc)






5

T=0







5

T=0T=1







5

T=0T=1T=100







5

T=0T=1T=100


! selects actions by recursive ‘mental simulation’






5

T=0T=1T=100



! tree search implies combinatorial explosion






5

T=0T=1T=100




! approximations are necessary






5

T=0T=1T=100




! approximations are necessary

! effective computational noise





EPISODIC MEMORY: MODEL-FREE CONTROL

6






6

! stores specific episodes retrospectively(state—action—…—reward sequences)






6

T=0! stores specific episodes retrospectively

(state—action—…—reward sequences)






6

T=0T=1! stores specific episodes retrospectively







6

T=0T=1T=100! stores specific episodes retrospectively







6



! selects action that yielded maximalultimate reward in past episodes startingfrom current state






6



! selects action that yielded maximalultimate reward in past episodes startingfrom current state

compatible with:

! hippocampal involvement in" processing sequential memories

(Fortin et al, 2002; Ergorul & Eichenbaum, 2006; Manns et al, 2007, Lehn et al, 2009)

" imagining new experiences(Hassabis et al, 2007)

! awake forward replay at decision points (Johnson & Redish, 2007)

! reward and episodic information integrated (Lansink et al, 2009; Rossato et al, 2009)





COMPARING THE TWO SYSTEMS

7






7

• vary complexity of the environment (A, B, and D)






7


number of actionsA






7


number of actionsA branching factor

B






7


depthD


B






7


• vary amount of experience available to semantic and episodic memory systems

depthD


B






7



• compute average performance of three systems

depthD


B






7




1. perfect semantic memory-based control ! theoretical upper bound

depthD


B






7




1. perfect semantic memory-based control ! theoretical upper bound2. approximate semantic memory-based controller

depthD


B






7




1. perfect semantic memory-based control ! theoretical upper bound2. approximate semantic memory-based controller 3. episodic memory-based controller

depthD


B





THE PERFECT MODEL-BASED SYSTEM

8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB

core idea: analysing a sub-treelet






8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



value of state





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



values of available action





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



values of successor states





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



µV ′

σV ′





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



µV ′

σV ′

averaging





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



µV ′

σV ′

µQ

σQ

averaging

CQ





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



µV ′

σV ′

µQ

σQ

averagingmax

CQ





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB



µV ′

σV ′

µQ

σQ

µV

σV

averagingmax

CQ





8

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB

core idea: analysing a sub-treelet whole environment


µV ′

σV ′

µQ

σQ

µV

σV

averagingmax

CQ





8

more complex environment ! more potential for high rewards

V ′1

V ′2

V ′B

!

V

!

Q1

QA

p11

pAB

core idea: analysing a sub-treelet whole environment


µV ′

σV ′

µQ

σQ

µV

σV

averagingmax

CQ




THE EFFECTS OF APPROXIMATIONS

9

true action values ! noisy versions

Q̃ = η1Q + η2z z ∼ N (0, 1)






9


Q̃ = η1Q + η2z z ∼ N (0, 1)


Q1 Q2





9


Q̃ = η1Q + η2z z ∼ N (0, 1)


P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2





9


Q̃ = η1Q + η2z z ∼ N (0, 1)actual reward dependson Q of the actionwith highest Q̃


P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2





9


Q̃ = η1Q + η2z z ∼ N (0, 1)

noise-to-signal ratio: ω2 =η22

η21 σ2

Q

actual reward dependson Q of the actionwith highest Q̃


P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2





9


Q̃ = η1Q + η2z z ∼ N (0, 1)


η21 σ2

Q



σQ

P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2

ω ω





9

single state


Q̃ = η1Q + η2z z ∼ N (0, 1)


η21 σ2

Q



σQ

P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2

ω ω





9

incre

asin

g le

vels o

fco

mputa

tional n

oise

single state whole environment


Q̃ = η1Q + η2z z ∼ N (0, 1)


η21 σ2

Q



σQ

P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2

ω ω





9

incre

asin

g le

vels o

fco

mputa

tional n

oise

deeper environment ! noise is more deleterious

single state whole environment


Q̃ = η1Q + η2z z ∼ N (0, 1)


η21 σ2

Q



σQ

P(Q̃2 | Q2

)P

(Q̃1 | Q1

)

Q1 Q2

ω ω




! "!! #!! $!! %!! &!!!

!'&

"

!"

! "!! #!! $!! %!! &!!!

!'"

!'#

!#

! "!! #!! $!! %!! &!!!

"!

#!

$!

()*+,-!./!+/)0123,.)/

4!

THE EFFECT OF LEARNING

10

key idea: ignorance about the environment ! additional noise





! "!! #!! $!! %!! &!!!

!'&

"

!"

! "!! #!! $!! %!! &!!!

!'"

!'#

!#

! "!! #!! $!! %!! &!!!

"!

#!

$!

()*+,-!./!+/)0123,.)/

4!


10


number of times each state-action pair

is visited





! "!! #!! $!! %!! &!!!

!'&

"

!"

! "!! #!! $!! %!! &!!!

!'"

!'#

!#

! "!! #!! $!! %!! &!!!

"!

#!

$!

()*+,-!./!+/)0123,.)/

4!


10


number of times each state-action pair

is visited

time requiredto collect it

∝ A BD





THE EFFECTS OF COMPUTATIONAL + IGNORANCE NOISE

11






11

incre

asin

g le

vels o

fco

mputa

tional n

oise






11

approximations: more adverse effects early in learning!

incre

asin

g le

vels o

fco

mputa

tional n

oise






11

approximations: more adverse effects early in learning!

incre

asin

g le

vels o

fco

mputa

tional n

oise

room for alternative decision making systems in low-data limit





EPISODIC VS. SEMANTIC MEMORY-BASED DECISION MAKING

12






12

episodic advantage early in learning






12

incre

asin

g e

nviro

nm

enta

l com

ple

xity


Amount of experience






12

incre

asin

g e

nviro

nm

enta

l com

ple

xity


lasts longer for more complex environments

Amount of experience





SUMMARY

13




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memoryneocortex




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory

neocortex

hippocampus




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory

‘value’ or procedural memory

neocortex

hippocampus

striatum




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory


‘consolidation’

neocortex

hippocampus

striatum




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory


‘consolidation’

consolidation: transfer of control rather than transfer of memories?

neocortex

hippocampus

striatum




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory


‘consolidation’

‘competing memory systems’


neocortex

hippocampus

striatum




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory


‘consolidation’

‘habitization’


Daw et al, 2005


neocortex

hippocampus

striatum




envi

ronm

enta

l co

mple

xit

y

learning

SUMMARY

13

semantic memory

episodic memory


‘consolidation’

‘habitization’


Daw et al, 2005


neocortex

hippocampus

striatum

environmental non-stationarity




OPEN QUESTIONS

14




OPEN QUESTIONS

14

! granularity of integration of episodic with model-based system




OPEN QUESTIONS

14


! episodes store outcomes (goal-directed) or rewards (habitual)




OPEN QUESTIONS

14



! arbitration between parallel systems




OPEN QUESTIONS

14



! arbitration between parallel systems! need to represent uncertainty (Daw et al, 2005)




OPEN QUESTIONS

14




! replay during sleep not for consolidation




OPEN QUESTIONS

14




! replay during sleep not for consolidation ! keeping memory representations in register (Káli & Dayan, 2005)



episodic memory: why and how?

Documents