cmsc 100 heuristic search and game playing professor marie desjardins tuesday, november 13, 2012 thu...

CMSC 100CMSC 100

Heuristic Search and Game PlayingHeuristic Search and Game Playing

Professor Marie desJardins

Tuesday, November 13, 2012Thu 10/25/121E-Commerce and Databases

2

Summary of TopicsSummary of Topics What is heuristic search?

Examples of search problems

Search methods Uninformed search Informed search Local search Game trees

3

Building Goal-Based Building Goal-Based Intelligent AgentsIntelligent Agents

To build a goal-based agent we need to answer the following questions: What is the goal to be achieved? What are the actions? What relevant information is needed to describe the state

of the world, describe the available transitions, and solve the problem?

Initialstate

Goalstate

Actions

4

Representing StatesRepresenting States What information is needed to sufficiently describe all

relevant aspects to solving the goal? That is, what knowledge needs to be represented in a state

description to adequately describe the current state or situation of the world?

The size of a problem is usually described in terms of the number of states that are possible. Tic-Tac-Toe has about 39 states. Checkers has about 1040 states. Rubik’s Cube has about 1019 states. Chess has about 10120 states in a typical game.

5

Real-world Search Real-world Search ProblemsProblems

Route finding (GPS algorithms, Google Maps)

Touring (traveling salesman)

Logistics

VLSI layout

Robot navigation

Learning

6

8-Puzzle8-PuzzleGiven an initial configuration of 8 numbered tiles on a

3 x 3 board, move the tiles into a desired goal configuration of the tiles.

7

8-Puzzle Encoding8-Puzzle Encoding State: 3 x 3 array configuration of the tiles on the board.

4 Operators: Move Blank Square Left, Right, Up or Down. This is a more efficient encoding of the operators than one in

which each of four possible moves for each of the 8 distinct tiles is used.

Initial State: A particular configuration of the board.

Goal: A particular configuration of the board.

What does the state space look like?

8

Solution CostSolution Cost A solution is a sequence of operators that is associated

with a path in a state space from a start node to a goal node.

The cost of a solution is the sum of the arc costs on the solution path. If all arcs have the same (unit) cost, then the solution cost is

just the length of the solution (number of steps / state transitions)

9

Evaluating Search Evaluating Search StrategiesStrategies

Completeness Guarantees finding a solution whenever one exists

Time complexity How long (worst or average case) does it take to find a solution?

Usually measured in terms of the number of nodes expanded

Space complexity How much space is used by the algorithm? Usually measured in

terms of the maximum size of the “nodes” list during the search

Optimality/Admissibility If a solution is found, is it guaranteed to be an optimal one? That is,

is it the one with minimum cost?

10

Search MethodsSearch Methods Uninformed search strategies

Also known as “blind search,” uninformed search strategies use no information about the likely “direction” of the goal node(s)

Variations on “generate and test” or “trial and error” approach Uninformed search methods: breadth-first, depth-first, uniform-cost

Informed search strategies Also known as “heuristic search,” informed search strategies use

information about the domain to (try to) (usually) head in the general direction of the goal node(s)

Informed search methods: greedy search, (A*) (Informed search is what your GPS does!!)

11

Search Methods cont.Search Methods cont.

Local search strategies Pick a starting solution (that might not be very good) and

incrementally try to improve it Local search methods: hill-climbing, genetic algorithms

Game trees Search strategies for situations where you have an opponent who

gets to make some of the moves Try to pick moves that will let you win most of the time by “looking

ahead” to see what your opponent might do

Uninformed Search

13

Maze ExampleMaze ExampleS

G

Reach the goal as cheaply as possibleStart at “S”Try to get to “G”Blue area costs ten dollars to move intoEvery other area costs one dollarNo point in retracing steps... (treat dead ends as having cost ∞)

14

Maze Search TreeMaze Search Tree

S

G

Search tree == all possible pathsTry left, then straight, then backNever reverse direction

S

GS

G

S

G

G

S

G

S

G

` S

G

S G

CHALLENGE: How to explore (generate) only the part of this search tree that actually leads to a solution?That’s what SEARCH does!

Cost: 0 Cost: 1

Cost: 2

Cost: 3 Cost: 4 Cost: ∞

Cost: 23

Cost: ∞Cost: 2

S

G

Cost: 22

15

Depth-First Search Depth-First Search (DFS)(DFS)

Enqueue nodes on nodes in LIFO (last-in, first-out) order. That is, nodes used as a stack data structure to order nodes. Like walking through a maze, always following the rightmost

branch When search hits a dead end, back up one step at a time

Can find long solutions quickly if lucky (and short solutions slowly if unlucky!)

16

DFS Maze Solution (Step DFS Maze Solution (Step 0)0)

S

G

Cost: 0

CurrentFrontier(revisitlater)

17


S

G

S

G

Cost: 0 Cost: 1

18


S

G

S

GS

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 2

19


S

G

S

GS

G

S

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 3

Cost: 2

S

G

Cost: 22

20


S

G

S

GS

G

S

G

S

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 3 Cost: 4

Cost: 2

S

G

Cost: 22

21


S

G

S

GS

G

S

G

S

G

S

G

` S

G

Cost: 0 Cost: 1

Cost: 2


Cost: 2

S

G

Cost: 22

22


S

G

S

GS

G

S

G

S

G

S

G

` S

G

Cost: 0 Cost: 1

Cost: 2


Cost: 2

S

G

Cost: 22

23


S

G

S

GS

G

S

G

G

S

G

S

G

` S

G

Cost: 0 Cost: 1

Cost: 2


Cost: 23

Cost: 2

S

G

Cost: 22

24

Breadth-First SearchBreadth-First Search

Enqueue nodes on nodes in FIFO (first-in, first-out) order. Like having a team of “gremlins” who can keep track of

every branching point in a maze Only one gremlin can move at a time, and breadth-first

search moves them each in turn, duplicating the gremlins if they reach a “fork in the road” – one for each fork!

Finds the shortest path, but sometimes very slowly (and at the cost of generating lots of gremlins!)

25

BFS Maze Solution (Step BFS Maze Solution (Step 0)0)

S

G

Cost: 0

26


S

G

S

G

Cost: 0 Cost: 1

27


S

G

S

GS

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 2

28


S

G

S

GS

G

S

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 3

Cost: 2

S

G

Cost: 22

29


S

G

S

GS

G

S

G

S

G

S G

Cost: 0 Cost: 1

Cost: 2

Cost: 3

Cost: ∞Cost: 2

S

G

Cost: 22

30


S

G

S

GS

G

S

G

S

G

S

G

S G

Cost: 0 Cost: 1

Cost: 2

Cost: 3 Cost: 4

Cost: ∞Cost: 2

S

G

Cost: 22

31


S

G

S

GS

G

S

G

G

S

G

S

G

S G

Cost: 0 Cost: 1

Cost: 2

Cost: 3 Cost: 4

Cost: 23

Cost: ∞Cost: 2

S

G

Cost: 22

32

Uniform Cost SearchUniform Cost Search Enqueue nodes by path cost. That is, let priority = cost

of the path from the start node to the current node. Sort nodes by increasing value of cost (try low-cost nodes first)

Called “Dijkstra’s Algorithm” in the algorithms literature; similar to “Branch and Bound Algorithm” from operations research

Finds the very best path, but can take a very long time and use a very large amount of memory!

33

UCS Maze Solution (Step UCS Maze Solution (Step 0)0)

S

G

Cost: 0

34


S

G

S

G

Cost: 0 Cost: 1

35


S

G

S

GS

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 2

36


S

G

S

GS

G

S

G

S

G

Cost: 0 Cost: 1

Cost: 2

Cost: 3

Cost: 2

S

G

Cost: 22

37


S

G

S

GS

G

S

G

S

G

S G

Cost: 0 Cost: 1

Cost: 2

Cost: 3

Cost: ∞Cost: 2

S

G

Cost: 22

38


S

G

S

GS

G

S

G

S

G

S

G

` S

G

S G

Cost: 0 Cost: 1

Cost: 2


Cost: ∞Cost: 2

S

G

Cost: 22

39


S

G

S

GS

G

S

G

S

G

S

G

` S

G

S G

Cost: 0 Cost: 1

Cost: 2


Cost: ∞Cost: 2

S

G

Cost: 22

40


S

G

S

GS

G

S

G

G

S

G

S

G

` S

G

S G

Cost: 0 Cost: 1

Cost: 2


Cost: 23

Cost: ∞Cost: 2

S

G

Cost: 22

41

Holy Grail SearchHoly Grail Search

If only we knew where we were headed…

Informed SearchInformed Search

43

What’s a Heuristic?What’s a Heuristic?

From WordNet (r) 1.6 heuristic adj 1: (computer science) relating to or

using a heuristic rule 2: of or relating to a general formulation that serves to guide investigation [ant: algorithmic] n : a commonsense rule (or set of rules) intended to increase the probability of solving some problem [syn: heuristic rule, heuristic program]

44

Informed Search: Use Informed Search: Use What You Know!What You Know!

Add domain-specific information to select the best path along which to continue searching

Define a heuristic function that estimates how close a node n is to the goal node

Incorporate the heuristic function into search to choose the most promising branch along which to search: Greedy search: Go in the most promising direction of any frontier

node (ignoring how much it cost you to get to that frontier node) A* search: Combine the work you’ve done so far to get to a

frontier node, plus the heuristic estimate to the goal node, and pick the most promising in terms of total cost

Local SearchLocal Search

46

Local SearchLocal Search Another approach to search involves starting with

an initial guess at a solution and gradually improving it until it is a legal solution or the best that can be found.

Also known as “incremental improvement” search

Some examples: Hill climbing Genetic algorithms

47

Hill Climbing on a Surface of Hill Climbing on a Surface of StatesStates

Height Defined by Evaluation Function

48

Drawbacks of Hill Drawbacks of Hill ClimbingClimbing

Problems: Local Maxima: peaks that aren’t the highest point in the

space Plateaus: the space has a broad flat region that gives the

search algorithm no direction (random walk) Ridges: flat like a plateau, but with dropoffs to the sides;

steps to the North, East, South and West may go down, but a step to the NW may go up.

Remedies: Random restart Problem reformulation

Some problem spaces are great for hill climbing and others are terrible.

49

Genetic AlgorithmsGenetic Algorithms Start with k random states (the initial population)

New states are generated by “mutating” a single state or “reproducing” (combining) two parent states (selected according to their fitness)

Encoding used for the “genome” of an individual strongly affects the behavior of the search

Genetic algorithms / genetic programming are a large and active area of research

Game Game PlayingPlaying

51

Why Study Games?Why Study Games? Clear criteria for success

Offer an opportunity to study problems involving {hostile, adversarial, competing} agents.

Historical reasons

Fun

Interesting, hard problems thatrequire minimal “initial structure”

Games often define very large search spaces chess 35100 nodes in search tree, 1040 legal states

52

State of the ArtState of the Art How good are computer game players?

Chess: Deep Blue beat Gary Kasparov in 1997 Garry Kasparav vs. Deep Junior (Feb 2003): tie! Kasparov vs. X3D Fritz (November 2003): tie!

http://www.cnn.com/2003/TECH/fun.games/11/19/kasparov.chess.ap/ Checkers: Chinook (an AI program with a very large endgame database)

is the world champion (checkers is “solved”!) Go: Computer players are competitive at a professional level Bridge: “Expert-level” computer players exist (but no world champions

yet!) Poker: Computer team beat a human team, using statistical modeling and

adaptation:http://www.cs.ualberta.ca/~games/poker/man-machine/

Good places to learn more: http://www.cs.ualberta.ca/~games/ http://www.cs.unimass.nl/icga

53

ChinookChinook Chinook is the World Man-Machine Checkers

Champion, developed by researchers at the University of Alberta.

It earned this title by competing in human tournaments, winning the right to play for the (human) world championship, and eventually defeating the best players in the world.

Visit http://www.cs.ualberta.ca/~chinook/ to play a version of Chinook over the Internet.

The researchers have fully analyzed the game of checkers, and can provably always win if they play black

“One Jump Ahead: Challenging Human Supremacy in Checker,s” Jonathan Schaeffer, University of Alberta

54

Typical Game SettingTypical Game Setting

2-person game

Players alternate moves

Zero-sum: one player’s loss is the other’s gain

Perfect information: both players have access to complete information about the state of the game. No information is hidden from either player.

No chance (e.g., using dice) involved

Examples: Tic-Tac-Toe, Checkers, Chess, Go, Nim, Othello

Not: Bridge, Solitaire, Backgammon, ...

55

Let’s Play Nim!Let’s Play Nim! Seven tokens (coins, sticks, whatever)

Each player must take either 1 or 2 tokens

Whoever takes the last token loses

You can go first…

56

How to Play a GameHow to Play a Game

A way to play such a game is to: Consider all the legal moves you can make Compute the new position resulting from each move Evaluate each resulting position and determine which is best Make that move Wait for your opponent to move and repeat

Key problems are: Representing the “board” Generating all legal next boards Evaluating a position

57

Evaluation FunctionEvaluation Function

Evaluation function or static evaluator is used to evaluate the “goodness” of a game position. Contrast with heuristic search where the evaluation function

was a non-negative estimate of the cost from the start node to a goal and passing through the given node

The zero-sum assumption allows us to use a single evaluation function to describe the goodness of a board with respect to both players. f(n) >> 0: position n good for me and bad for you f(n) << 0: position n bad for me and good for you f(n) near 0: position n is a neutral position f(n) = +infinity: win for me f(n) = -infinity: win for you

58

Evaluation Function Evaluation Function ExamplesExamples

Example of an evaluation function for Tic-Tac-Toe: f(n) = [# of 3-lengths open for me] - [# of 3-lengths open for you] where a 3-length is a complete row, column, or diagonal

Alan Turing’s function for chess f(n) = w(n)/b(n) where w(n) = sum of the point value of white’s

pieces and b(n) = sum of black’s Most evaluation functions are specified as a weighted sum of

position features:f(n) = w1*feat1(n) + w2*feat2(n) + ... + wn*featk(n)

Example features for chess are piece count, piece placement, squares controlled, etc.

Deep Blue had over 8000 features in its evaluation function

59

Game TreesGame Trees

Problem spaces for typical games are represented as trees

Root node represents the current board configuration; player must decide the best single move to make next

Static evaluator function rates a board position. f(board) = real number with f>0 “white” (me), f<0 for black (you)

Arcs represent the possible legal moves for a player

If it is my turn to move, then the root is labeled a "MAX" node; otherwise it is labeled a "MIN" node, indicating my opponent's turn.

Each level of the tree has nodes that are all MAX or all MIN; nodes at level i are of the opposite kind from those at level i+1

60

Minimax ProcedureMinimax Procedure

Create start node as a MAX node with current board configuration

Expand nodes down to some depth (a.k.a. ply) of lookahead in the game

Apply the evaluation function at each of the leaf nodes

“Back up” values for each of the non-leaf nodes until a value is computed for the root node At MIN nodes, the backed-up value is the minimum of the values associated

with its children. (Best move for the MIN player) At MAX nodes, the backed-up value is the maximum of the values associated

with its children. (Best move for the MAX player)

Pick the operator associated with the child node whose backed-up value determined the value at the root

61

Nim-4: First PlyNim-4: First Ply4 MAX

2 MIN3 MIN

State: # coins left, whose turn it isWin for MAX: +1Win for MIN: -1Left branch: take 1 coinRight branch: take 2 coins

62

Nim-4: Second PlyNim-4: Second Ply4 MAX

2 MIN3 MIN

1 MAX2 MAX 0 MAX1 MAX

+1

63

Nim-4: Third PlyNim-4: Third Ply4 MAX

2 MIN3 MIN


0 MIN1 MIN 0 MIN0 MIN

-1 -1 -1

+1

64

Nim-4: Fourth PlyNim-4: Fourth Ply4 MAX

2 MIN3 MIN



0 MAX

+1

-1 -1 -1

+1

Complete game tree!All “leaf nodes” are terminal states (end of the game), so no need to evaluate intermediate (non-end) states

65

Backup to Level 3Backup to Level 34 MAX

2 MIN3 MIN



0 MAX

+1

+1 -1 -1 -1

66


2 MIN3 MIN



0 MAX

+1

+1 -1 -1 -1

+1-1-1+1

67


2 MIN3 MIN



0 MAX

+1

+1 -1 -1 -1

+1-1-1+1

-1 -1

68

Backup to Level 0 Backup to Level 0 (Root)(Root)

4 MAX

2 MIN3 MIN



0 MAX

+1

+1 -1 -1 -1

+1-1-1+1

-1

-1 -1

MAX always loses!(unless MIN does something stupid...)

cmsc 100 heuristic search and game playing professor marie desjardins tuesday, november 13, 2012 thu...

Documents

heuristic search

blind search

greedy search

search methods cont

local search strategiespick

solution costa solution

solution path

building goal