summary of analogical reinforcement learning
TRANSCRIPT
Summary of Analogical Reinforcement Learning
James M. Foster & Matt Jones
Methodology of Cognitive Science
Jin Hwa Kim Program in Cognitive Science
Seoul National University
Introduction
1. Background
- To develop a computational understanding of how people learn abstract concepts (analogical reasoning),
- Recognizing structural similarities among relational system [Doumas et al., 2008; Gentner, 1983; Hummel & Holyoak, 2003]
2. Goal
- Integrate analogical learning with reinforcement learning (RL), which has neurophysiological support [e.g. Schultz et al., 1997]
- Representation through similarity-based generalization
Fork Analogy
[Foster & Jones, 2013]
Figure 1: An example of structural alignment between two chess positions. Both positions contain instances of the abstract concept of a FORK: black’s piece is simultaneously attacking both of white’s pieces. These attacking relations are represented by the red arrows. Cyan lines indicate the mapping between the two episodes. The mapping satisfies parallel connectivity because it respects the bindings between relations and their role-fillers.
Structural Alignment
1. It is critical to learn abstract concepts
- Perceived similarity depends on it. [Markman & Gentner, 1993]
- For analogical transfer [Gick & Holyoak, 1980](e.g. in a different, but aligned position)
- Lead to schema induction
2. Need for selection
- Fork is an important not (only) because its frequency, but because it has significant wining chance.
- Let a reward of RL judges analogical schema.
Reinforcement Learning
1. Method
- Learning values for different states or actions, which represent the totally expected future reward
- Incrementally from temporal-difference (TD) error
2. Limitation
- Too many states for learning values Using a subset of the values for each exemplar states
Analogical RL
1. Synergy of Analogy and RL
- Structural alignment yields an abstract form of psychological similarity that can support sophisticated generalization [Gick & Holyoak, 1980; Markman & Gentner, 1993](e.g. quickly learn to create forks and to avoid forks by the opponent)
- Drive analogical learning by guiding schema induction (e.g. sequential attacking strategy)
2. Generating schemas
- Initially exemplars are episodic states.
- Whenever the structural alignment between a state and an exemplar produce a sufficient reduction in prediction error, the induced schema is added to the collection of exemplars.
- Thus the model’s representation transits from initially episodic to more abstract and conceptual.
Model
Figure 2: Model operation. Each candidate after-state is evaluated by analogical comparison to stored exemplars, followed by similarity-weighted averaging among the learned exemplar values. Learning is by TD error applied to the exemplar values. On some trials, especially useful analogies produce new schemas that are added to the exemplar pool. In the example here, s and E both have guaranteed wins for X by threatening a win in two ways. The induced schema embodies this abstract structure. Dots with red arrows indicate ternary “same-rank” relations. r = reward; γ = temporal discount parameter; α = learning rate; other variables are defined in the text.
Simulation
1. Three models
1) Featural model is the simplest one using only literal mappings.
2) Relational model includes rigid rotation and reflection of the board.
3) Schema model includes two hand-coded schemas in addition to the above relational model. (111, immediate win; and 022, risks immediate loss)
2. Limitation
- The purpose of the schema model was to test the utility of having schemas that capture task-relevant structures.
- To integrate a solution into the present model soon.
Results
Figure 3: Learning curves. A: 50 copies of each model. B: Single copies of the two slower models over extended training.
Discussion
1. Limitations
- Not included a mechanism for inducing new schemas.
- Learning is slow due to the nature of gradient descent operating.(ref. One-shot learning with bayesian networks [Maas & Kemp, 2009])
2. Bottom-up and Top-down
- Whereas the analogical RL builds up schemas from simpler representations, the Bayesian approach takes a top-down approach.
- The question is how the human mind discovers new, abstract concepts.
!!
†
Q/A
1. Analogical RL의 실례로 체스의 경우를 들고 있습니다. fork의 경우들을 학습하는 대신에 fork 개념 자체를 학습하는 것으로 설명하고 있는데, 다소 이해가 어렵습니다. 개념을 직접 학습하는데에 fork의 경우 등의 사전지식이 불필요함을 이야기하는 것인지, 단지 "모든 경우의 수"를 다 알 필요는 없다는 뜻인지 궁금합니다.
Q/A
2. RL이 temporal difference error를 사용하는 것에 대해 인간의 뇌가 이 방식을 사용한다는 증거가 있기 때문이라 말합니다. 인간의 뇌가 이 방식을 사용한다는 점에 대한 보다 자세한 설명 부탁드리고요. [Schultz et al., 1997]