Pondering Probabilistic Pondering Probabilistic Play Policies for PigPlay Policies for Pig

Todd W. NellerGettysburg College

Sow What’s This All Sow What’s This All About?About?

•The Dice Game “Pig”•Odds and Ends•Playing to Win–“Piglet”–Value Iteration

•Machine Learning

Pig: The GamePig: The Game

•Object: First to score 100 points

•On your turn, roll until:–You roll 1, and score NOTHING.–You hold, and KEEP the sum.

•Simple game simple strategy?

•Let’s play…

Playing to ScorePlaying to Score

•Simple odds argument–Roll until you risk more than you stand to gain.

–“Hold at 20”•1/6 of time: -20 -20/6•5/6 of time: +4 (avg. of 2,3,4,5,6) +20/6

Hold at 20?Hold at 20?

•Is there a situation in which you wouldn’t want to hold at 20?–Your score: 99; you roll 2–Case scenario •you: 79 opponent:99•Your turn total stands at 20

What’s Wrong With What’s Wrong With Playing to Score?Playing to Score?

•It’s mathematically optimal!•But what are we optimizing?•Playing to score Playing to win

•Optimizing score per turn Optimizing probability of a win

PigletPiglet

•Simpler version of Pig with a coin

•Object: First to score 10 points•On your turn, flip until:–You flip tails, and score NOTHING.–You hold, and KEEP the # of heads.

•Even simpler: play to 2 points

Essential InformationEssential Information

•What is the information I need to make a fully informed decision?–My score–The opponent’s score–My “turn score”

A Little NotationA Little Notation

•Pi,j,k – probability of a win ifi = my scorej = the opponent’s scorek = my turn score

•Hold: Pi,j,k = 1 - Pj,i+k,0

•Flip: Pi,j,k = ½(1 - Pj,i,0) + ½ Pi,j,k+1

Assume RationalityAssume Rationality

•To make a smart player, assume a smart opponent.

• (To make a smarter player, know your opponent.)

• Pi,j,k = max(1 - Pj,i+k,0, ½(1 - Pj,i,0 + Pi,j,k+1))

•Probability of win based on best decisions in any state

The Whole StoryThe Whole Story

P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1))P0,0,1 = max(1 – P0,1,0, ½(1 – P0,0,0 + P0,0,2))P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1))P0,1,1 = max(1 – P1,1,0, ½(1 – P1,0,0 + P0,1,2))P1,0,0 = max(1 – P0,1,0, ½(1 – P0,1,0 + P1,0,1))P1,1,0 = max(1 – P1,1,0, ½(1 – P1,1,0 + P1,1,1))

The Whole StoryThe Whole Story

P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1))P0,0,1 = max(1 – P0,1,0, ½(1 – P0,0,0 + P0,0,2))P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1))P0,1,1 = max(1 – P1,1,0, ½(1 – P1,0,0 + P0,1,2))P1,0,0 = max(1 – P0,1,0, ½(1 – P0,1,0 + P1,0,1))P1,1,0 = max(1 – P1,1,0, ½(1 – P1,1,0 + P1,1,1))

These are winning states!

The Whole StoryThe Whole Story

P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1))

P0,0,1 = max(1 – P0,1,0, ½(1 – P0,0,0 + 1))P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 +

P0,1,1))P0,1,1 = max(1 – P1,1,0, ½(1 – P1,0,0 + 1))P1,0,0 = max(1 – P0,1,0, ½(1 – P0,1,0 + 1))P1,1,0 = max(1 – P1,1,0, ½(1 – P1,1,0 + 1))Simplified…

The Whole StoryThe Whole Story

P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1))P0,0,1 = max(1 – P0,1,0, ½(2 – P0,0,0))P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1))P0,1,1 = max(1 – P1,1,0, ½(2 – P1,0,0))P1,0,0 = max(1 – P0,1,0, ½(2 – P0,1,0))P1,1,0 = max(1 – P1,1,0, ½(2 – P1,1,0))And simplified more into a hamsome set

of equations…

How to Solve It?How to Solve It?

P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1))P0,0,1 = max(1 – P0,1,0, ½(2 – P0,0,0))P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1))P0,1,1 = max(1 – P1,1,0, ½(2 – P1,0,0))P1,0,0 = max(1 – P0,1,0, ½(2 – P0,1,0))P1,1,0 = max(1 – P1,1,0, ½(2 – P1,1,0))P0,1,0 depends on P0,1,1 depends on P1,0,0 depends on P0,1,0

depends on …

A System of A System of PigquationsPigquations

P0,1,1

P0,0,1P1,1,0P0,1,0

P1,0,0P0,0,0

Dependencies betweennon-winning states

How Bad Is It?How Bad Is It?

•The intersection of a set of bent hyperplanes in a hypercube

• In the general case, no known method (read: PhD research)

• Is there a method that works (without being guaranteed to work in general)? –Yes! Value Iteration!

Value IterationValue Iteration

•Start out with some values (0’s, 1’s, random #’s)

•Do the following until the values converge (stop changing):–Plug the values into the RHS’s–Recompute the LHS values

•That’s easy. Let’s do it!

Value IterationValue Iteration

P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1))P0,0,1 = max(1 – P0,1,0, ½(2 – P0,0,0))P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1))P0,1,1 = max(1 – P1,1,0, ½(2 – P1,0,0))P1,0,0 = max(1 – P0,1,0, ½(2 – P0,1,0))P1,1,0 = max(1 – P1,1,0, ½(2 – P1,1,0))• Assume Pi,j,k is 0 unless it’s a win• Repeat: Compute RHS’s, assign to

LHS’s

But That’s GRUNT But That’s GRUNT Work!Work!

•So have a computer do it, slacker!

•Not difficult – end of CS1 level•Fast! Don’t blink – you’ll miss it•Optimal play:–Compute the probabilities–Determine flip/hold from RHS

max’s– (For our equations, always FLIP)

Piglet SolvedPiglet Solved

•Game to 10•Play to Score: “Hold at 1”

•Play to Win:

0

2

4

6

8

10

Hold Value

Little Pig Hold Values

You

Opponent0 1 2 3 4 5 6 7 8 9

0 2 2 2 2 2 2 2 2 3 101 1 2 2 2 2 2 2 3 3 92 1 1 2 2 2 2 2 2 3 83 1 1 1 2 2 2 2 2 3 74 1 1 1 1 2 2 2 2 2 65 1 1 1 1 1 2 2 2 2 56 1 1 1 1 1 1 2 2 2 47 1 1 1 1 1 1 1 1 3 38 1 1 1 1 1 1 1 2 2 29 1 1 1 1 1 1 1 1 1 1

Pig ProbabilitiesPig Probabilities

•Just like Piglet, but more possible outcomes

•Pi,j,k = max(1 - Pj,i+k,0, 1/6(1 - Pj,i,0 + Pi,j,k+2 + Pi,j,k+3

+ Pi,j,k+4 + Pi,j,k+5 + Pi,j,k+6))

Solving PigSolving Pig

•505,000 such equations•Same simple solution method (value iteration)

•Speedup: Solve groups of interdependent probabilities

•Watch and see!

Pig Sow-lutionPig Sow-lution

Pig Sow-lutionPig Sow-lution

Reachable StatesReachable States

Player 2 Score (j) = 30

Reachable StatesReachable States

Sow-lution forSow-lution forReachable StatesReachable States

Probability ContoursProbability Contours

SummarySummary

•Playing to score is not playing to win.

•A simple game is not always simple to play.

•The computer is an exciting power tool for the mind!

When Value IterationWhen Value IterationIsn’t EnoughIsn’t Enough

•Value Iteration assumes a model of the problem–Probabilities of state transitions– Expected rewards for transitions

•Loaded die?•Optimal play vs. suboptimal

player? •Game rules unknown?

No Model? Then No Model? Then LearnLearn!!

•Can’t write equations can’t solve

•Must learn from experience!•Reinforcement Learning– Learn optimal sequences of

actions– From experience–Given positive/negative feedback

Clever Mabel the CatClever Mabel the Cat

Clever Mabel the CatClever Mabel the Cat

•Mabel claws new La-Z-Boy BAD!

•Cats hate water spray bottle = negative reinforcement

•Mabel claws La-Z-Boy Todd gets up Todd sprays Mabel Mabel gets negative feedback

•Mabel learns…

Clever Mabel the CatClever Mabel the Cat

•Mabel learns to run when Todd gets up.

•Mabel first learns local causality:–Todd gets up Todd sprays Mabel

•Mabel eventually sees no correlation, learns indirect cause

•Mabel happily claws carpet. The End.

BackgammonBackgammon

•Tesauro’s Neurogammon–Reinforcement Learning + Neural

Network (memory for learning)– Learned backgammon through

self-play–Got better than all but a handful

of people in the world!–Downside: Took 1.5 million

games to learn

Greased PigGreased Pig

•My continuous variant of Pig•Object: First to score 100 points•On your turn, generate a

random number from 0.5 to 6.5 until:–Your rounded number is 1, and

you score NOTHING.–You hold, and KEEP the sum.

•How does this change things?

Greased Pig ChallengesGreased Pig Challenges

•Infinite possible game states

•Infinite possible games•Limited experience•Limited memory•Learning and approximation challenge

SummarySummary

•Solving equations can only take you so far (but much farther than we can fathom).

•Machine learning is an exciting area of research that can take us farther.

•The power of computing will increasingly aid in our learning in the future.