Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Monte Carlo Tree Search and Its Applications

Max Magnuson

Computer Science Senior SeminarDivision of Science and Mathematics

University of Minnesota, MorrisMorris, Minnesota, USA

April 25, 2015

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Kasparov vs Deep Blue

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Kasparov vs Deep Blue

Great display of artifical intelligence (AI)Techniques employed by IBM

I Brute force deterministic approachI Human knowledge

LimitationI Scalability into larger search spaces

Monte Carlo tree search (MCTS) is an alternative method

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Outline

Introduction

Naive MCTS Implementation

Applying MCTS to Go

Applying MCTS to Narrative Generation

Conclusion

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Monte Carlo Tree Search (MCTS)

I Combines random sampling and game treesI Lightweight random simulationsI Probabilistic not deterministicI Useful for problems with larger search spaces

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Applying MCTS to Go

GoI Board game about positional advantageI Game board for Chess:

I 8x8I Average possible configurations for a game of Chess:

I 10120

I Game board for Go:I 19x19

I Average possible configurations for a game of Go:I 10761

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Applying MCTS to Narrative Generation

I Useful ApplicationsI Video game replay valueI Educational applications

I The search space scales with the number of characters,items, locations, and actions

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Outline

Introduction

Naive MCTS Implementation

Applying MCTS to Go

Applying MCTS to Narrative Generation

Conclusion

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

TicTacToe Diagram

X

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

TicTacToe Diagram

XX

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

TicTacToe Diagram

XX X

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

TicTacToe Diagram More Levels

X

X X

X

O

X

O

O

X X

O

O X

O

X

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

TicTacToe Diagram

XX X

XO

X

O

OX X

OO X

OX

XO OXO XX

O X OX OXO XX

O XO XX

OWin Draw Loss

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

TicTacToe Diagram

XX X

XO

X

O

OX X

OO X

OX

XO OXO XX

O X OX OXO XX

O XO XX

OWin Draw Loss

Current Game State

Potential Game States

Potential End States

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Tree Structure

XX X

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Sampling

XX X

O XO X

OX

O X OX XOO XX

O XO XOX OX

O XX XXOO

X O XX O

XO

X OX O

OX

X O OOX XX

X XO OO

X

X O XO XOX OX

1/3 2/3 0/3

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

2/5

0/1 2/3

1/1 1/1

1

Selection Expansion Simulation Backpropagation

2/5

0/1 2/3

1/1 1/1

?

2/5

0/1 2/3

1/1 1/1

?

3/6

0/1 3/4

2/2 1/1

1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/2 1/1

1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/2 1/1

1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/2 1/1

1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/2 1/1

1/1 ?

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/2 1/1

1/1 ?

0

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/2 1/1

1/1 0/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/4

2/3 1/1

1/1 0/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/6

0/1 3/5

2/3 1/1

1/1 0/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1

Selection

Expansion

Simulation

BackpropagationBackpropagationBackpropagationBackpropagationBackpropagation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1 ?

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1 ?

1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 1/1

1/1 0/1 1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 3/5

2/3 2/2

1/1 0/1 1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

3/7

0/1 4/6

2/3 2/2

1/1 0/1 1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Four Steps Diagram

4/8

0/1 4/6

2/3 2/2

1/1 0/1 1/1

Selection

Expansion

Simulation

Backpropagation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

What Happens When We Choose a Move?

Now we have:I A tree structureI A method of generating the tree

What happens when we need to choose a move?

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Choosing a Move

4/8

0/1 4/6

2/3 2/2

1/1 0/1 1/1

A B

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Choosing a Move

4/8

0/1 4/6

2/3 2/2

1/1 0/1 1/1

A B

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Choosing a Move

4/8

0/1 4/6

2/3 2/2

1/1 0/1 1/1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Choosing a Move

4/6

2/3 2/2

1/1 0/1 1/1

New State of the Game

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Exploration vs Exploitation

I We might overlook better pathsI Exploration vs Exploitation

I Exploration looks at more optionsI Exploitation focuses on the most promising path

I Must find a balance between the two

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Upper Confidence Bound Applied to Trees (UCT)

UCT (node)=W (node)N(node)︸ ︷︷ ︸

Value of the Node

+ C

√ln(N(parentNode))

N(node)︸ ︷︷ ︸Exploration Bonus

I W represents the number of simulated winsI N represents the total number of simulationsI C is an experimental constantI Used during tree traversalI Balances exploration vs exploitation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Outline

Introduction

Naive MCTS Implementation

Applying MCTS to Go

Applying MCTS to Narrative Generation

Conclusion

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

MCTS applied to Go

What variations can we make specific to Go?In Go each player takes turn placing pieces on a game board

I How much does the order of these moves matter?I Can we use this to improve MCTS in the context of Go?

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Tree Redundancy

X XO

XO

X

XO

X

XO

X

A B C

C B A

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Rapid Action Value Estimate (RAVE)

I Takes advantage of tree redundancyI Moves have no contextual dependenciesI Stores the value of a move within a subtree at each node

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

RAVE Diagram

Z

1 1 100

A B

A

B

B

MCTS(A) = 1/3 = 0.33

MCTS(B) = 2/3 = 0.67

RAVE(A) = 2/4 = 0.5

RAVE(B) = 2/5 = 0.4

1 0

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

MCTS Values

Z

1 1 100

A B

A

B

B

MCTS(A) = 1/3 = 0.33

MCTS(B) = 2/3 = 0.67

RAVE(A) = 2/4 = 0.5

RAVE(B) = 2/5 = 0.4

1 0

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

MCTS Values

Z

1 1 100

A B

A

B

B

MCTS(A) = 1/3 = 0.33

MCTS(B) = 2/3 = 0.67

RAVE(A) = 2/4 = 0.5

RAVE(B) = 2/5 = 0.4

1 0

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

RAVE Values

Z

1 1 100

A B

A

B

B

MCTS(A) = 1/3 = 0.33

MCTS(B) = 2/3 = 0.67

RAVE(A) = 2/4 = 0.5

RAVE(B) = 2/5 = 0.4

1 0

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

RAVE Values

Z

1 1 100

A B

A

B

B

MCTS(A) = 1/3 = 0.33

MCTS(B) = 2/3 = 0.67

RAVE(A) = 2/4 = 0.5

RAVE(B) = 2/5 = 0.4

1 0

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

MCTS RAVE Comparison

Z

1 1 100

A B

A

B

B

MCTS(A) = 1/3 = 0.33

MCTS(B) = 2/3 = 0.67

RAVE(A) = 2/4 = 0.5

RAVE(B) = 2/5 = 0.4

1 0

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

RAVE

I Very powerful approachI Each simulation provides us with more informationI Sometimes we do need contextual dependencies

I Example: Close tactical battles

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

MC RAVE

I Combines MCTS values with RAVE valuesI Uses a weighted averageI Favors RAVE values when fewer simulations have been

performedI Contextual dependencies are unknown

I Favors MCTS values when more simulations have beenperformed

I Contextual dependencies are more developed

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Go Results

I Deterministic approaches could hardly defeat low levelamateurs

I Computer Go programs use MC RAVEI MoGoI Crazy Stone

I Can compete against top pros in 9x9 GoI Can compete against top pros in handicapped 19x19 Go

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Outline

Introduction

Naive MCTS Implementation

Applying MCTS to Go

Applying MCTS to Narrative Generation

Conclusion

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Narrative Generation

Kartal et al. applied MCTS to Narrative GenerationI Crime storyI User defines the set up and goals for the story

I Example Setup: The detective starts in his officeI Example Goal: The killer must be arrested

Unlike Go and other gamesI Slightly different tree structureI Evaluation function needed

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Actions

I Actions drive the storyI Actions are believable based on context

I Example: Inspector searches for cluesI Example: Character A kills Character B

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Example Actions

Move(A, P): A moves to place P.Kill(A, B): B’s health to zero(dead).Earthquake(P): An earthquake strikes at place P.

I Actions take the place of movesI No clear end stateI The researchers used a set threshold during simulation

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Evaluation function

I Method of giving nodes valueI Incorporates believability and goal completionI Ensures stories are interesting

Value(story) = Believability(story) * GoalCompletion(story)I Believability is the mathematical product of every action in

a storyI The value is between 0 and 1

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Narrative Generation Test

MCTS compared against three deterministic algorithmsI Breadth-first searchI Depth-first searchI Best-first search

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Breadth-First Search

Move(

A, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Breadth-First Search

Move(

A, P)

Kill

(A, B

)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Breadth-First Search

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Breadth-First Search

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

Move(

A, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Breadth-First Search

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

Move(

A, P)

Kill

(A, B

)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Breadth-First Search

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Depth-First Search

Mov

e(A

, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Depth-First Search

Mov

e(A

, P)

Mov

e(A

, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Depth-First Search

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Depth-First Search

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Depth-First Search

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Depth-First Search

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Mov

e(A

, P)

Kill(A, B)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Best-First Search

0.40.8 0.2

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Best-First Search

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

0.8 0.20.4

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Best-First Search

0.8

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

0.4 0.8 0.2

Move(

A, P)

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Best-First Search

0.8

Move(

A, P)

Kill

(A, B

)

EarthQuake(P)

0.4 0.2

Move(

A, P)

0.8

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Best-First Search

0.8K

ill(A

, B)

Move(

A, P)

0.8

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Test Conditions

Goals for the narrative:I At least two people are killedI The killer is arrested

Each algorithm was given two budgetsI 100,000 nodesI 3 million nodes

Each algorithm ran three timesThe score of the narratives were averaged

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Results

MCTSBreadth-first

Depth-first

Best-first

LowBudget

0.07 0.05

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Stories Produced by MCTS

I Stories from MCTS tended to be believableI Completed both user defined goalsI Some problemsI Overall reasonable narratives

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Low Scoring Example from Breadth-First

Sherlock moved to Alice’s House. An Earthquake occurredat Alice’s House! Sherlock and Alice both died due to theearthquake.

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Outline

Introduction

Naive MCTS Implementation

Applying MCTS to Go

Applying MCTS to Narrative Generation

Conclusion

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Conclusion

I MCTS successful in extending AI capabilitiesI Tackles problems with larger search spacesI Effective in Go and narrative generationI Applicable to other problems

I Can outperform humans in many puzzlesI Real time gamesI Super Mario Brothers

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

Introduction Naive MCTS Implementation Applying MCTS to Go Applying MCTS to Narrative Generation Conclusion

Any Questions?

Magnuson U of Minn, Morris

Monte Carlo Tree Search and Its Applications

IntroductionNaive MCTS ImplementationApplying MCTS to GoApplying MCTS to Narrative GenerationConclusion