ontologies reasoning components agents simulations agents solving problems by environment state...

OntologiesReasoningComponentsAgentsSimulations

Agents Solving Problems byAgents Solving Problems byEnvironment State Space SearchEnvironment State Space Search

Jacques Robin

OutlineOutline

Search Agent Formulating Decision Problems as Navigating State Space to Find Goal

State Generic Search Algorithm Specific Algorithms

Breadth-First Search Uniform Cost Search Depth-First Search Backtracking Search Iterative Deepening Search Bi-Directional Search Comparative Table

Limitations and Difficulties Repeated States Partial Information

Search AgentsSearch Agents

Generic decision problem to be solved by an agent: Among all possible action sequences that I can execute, which ones will result in changing the environment from its current state, to another state that matches my goal?

Additional optimization problem: Among those action sequences that will change the environment from its current state

to a goal state which one can be executed at minimum cost? or which one leads to a state with maximum utility?

Search agent solves this decision problem by a generate and test approach: Given the environment model, generate one by one (all) the possible states (the state space) of the environment

reachable through all possible action sequences from the current state, test each generated state to determine whether it satisfies the goal or maximize utility

Navigation metaphor: The order of state generation is viewed as navigating the entire environment state

space

Off-Line Goal-Based Search AgentOff-Line Goal-Based Search Agent

AgentAgent

En

viro

nm

en

t

Sensors

Effectors

Action Sequence EffectPrediction: Search Algorithm

S

Percept Interpretation:Environment Initialization

P

goalTest():Boolean

E: EnvironmentModel

S: ActionSequence

Off-Line Utility-Based Search AgentOff-Line Utility-Based Search Agent

AgentAgent

En

viro

nm

en

t

Sensors

Effectors

Action Sequence EffectPrediction: Search Algorithm

S

Percept Interpretation:Environment Initialization

P

utility():Number

E: EnvironmentModel

S: ActionSequence

On-Line Goal-Based Search AgentOn-Line Goal-Based Search Agent

AgentAgent

En

viro

nm

en

t

Sensors

Effectors

Single Action EffectPrediction: Search Algorithm

A

Percept Interpretation:Environment Update

P

goalTest():Boolean

E: EnvironmentModel

On-Line Utility-Based Search AgentOn-Line Utility-Based Search Agent

AgentAgent

En

viro

nm

en

t

Sensors

Effectors

Single Action EffectPrediction: Search Algorithm

Percept Interpretation:Environment Update

P

utility():Number

E: EnvironmentModel

A

Example of Natural Born Search ProblemExample of Natural Born Search Problem

Example of Applying Navigation MetaphorExample of Applying Navigation Metaphorto Arbitrary Decision Problemto Arbitrary Decision Problem

N-Queens problem: how to arrange n queens on a NxN chess board in a pacific configuration where no queen can attack another?

Navigation Metaphor

Problem Algorithm

Q Q

Q

Q

Q

Q

Q

Q

Q

Q

InsertQueen

InsertQueen

InsertQueen

InsertQueen

Q

Q

Q Q

Q

Q

Q

Q

Q Q

Q Q MoveQueen

MoveQueen

Full-State Formulation: local navigation in thefull-State Space

Partial-State Formulation: global navigation in partial-State Space

Search Problem TaxonomySearch Problem Taxonomy

Search Problem

overlapping, complete

Goal-SatisfactionSearch Problem

Test: GoalSatisfactionTest

OptimizationSearch Problem

Test: utilityMaximizationTest

Full-State SpaceSearch Problem

initState: FullStatetypicalState: FullState

action: ModifyStateAction

Partial-State SpaceSearch Problem

initialState: EmptyStatetypicalState: PartialState

goalState: FullStateaction: RefineStateAction

disjoint, complete

disjoint, complete

State FindingSearch Problem

solution: State

Path FindingSearch Problem

solution: Path

disjoint, complete

Off-LineSearch Problem

solution: Sequence(Action)

On-LineSearch Problem

solution: Action

Fully InformedSearch Problem

sensor: CompletePerfectSensoraction: DeterministicAction

envtModel: CompleteEnvModel

SensorlessSearch Problem

sensor: MissingSensoraction: DeterministicAction

envModel: NonMissing

ContingencySearch Problem

sensor: PartialSensoraction: DeterministicAction

envModel: CompleteEnvModel

ExplorationSearch Problem

sensor: PartialSensoraction: DeterministicAction

envModel: Missing

disjoint, incomplete

Search Graphs and SearchTreesSearch Graphs and SearchTrees

The state space can be represented as a search graph or a search tree Each search graph or tree node represents an environment state Each search graph or tree arc represents an action changing the environment from its

source node to its target node Each node or arc can be associated to a utility or cost Each path from one node to another represents an action sequence In search tree, the root node represents the initial state and some leaves represent goal

states The problem of navigating the state space then becomes one of generating and maintaining

a search graph or tree until a solution node or path is generated The problem search graph or tree:

An abstract concept that contains one node for each possible environment state Can be infinite

The algorithm search graph or tree: A concrete data structure A subset of the problem search graph or tree that contains only the nodes generated and

maintained (i.e., explored) at the current point during the algorithm execution (always finite) Problem search graph or tree branching factor: average number of actions available to the

agent in each state Algorithm effective branching factor: average number of nodes effectively generated as

successors of each node during search

Search Graph and Tree Examples:Search Graph and Tree Examples:Vacuum Cleaner WorldVacuum Cleaner World

c,d

d

c

d

c

d

c,d

d

d

c,d

c,d

dc

d

d

c,d

c

d c,d

d

c

c,d

d

d

c,d

c

dc

d

d

c,d

c,d

dc

d

d

c,d

c,d

dd

c

d

c,dc

dc

d

d

c,d

c

dc

d

d

c,d

c

d c c,d

c,d

dc

d

d

c,d

c,d d

c

c,d

d

d

c

d

c

d

c,d

u s d

u ds u ds u ds

c,d

d

c

d

d

c,d c,d

c

d

c

c,d

c

u

uu u

d

ddd

s

s

d

s

u

s

d

u

dudu

s

s

ss

Search MethodsSearch Methods

Searching the entire space of the action sequence reachable environment states is called exhaustive search, systematic search, blind search or uninformed search

Searching a restricted subset of that state space based on knowledge about the specific characteristics of the problem or problem class is called partial search

A heuristic is an insight or approximate knowledge about a problem class or problem class family on which a search algorithm can rely to improve its run time and/or space requirement

An ordering heuristic defines: In which order to generate the problem search tree nodes, Where to go next while navigating the state space (to get closer faster to a solution

point) A pruning heuristic defines:

Which branches of the problem search tree to avoid generating alltogether, What subspaces of the state space not to explore (for they cannot or are very unlikely to

contain a solution point) Non-heuristic, exhaustive search is not scalable to large problem instances

(worst-case exponential in time and/or space) Heuristic, partial search offers no warrantee to find a solution if one exist, or to

find the best solution if several exists.

Formulating a Agent Decision ProblemFormulating a Agent Decision Problemas a Search Problemas a Search Problem

Define abstract format of a generic environment state, ex., a class C Define the initial state, ex., a specific object of class C Define the successor operation:

Takes as input a state or state set S and an action A Returns the state or state set R resulting from the agent executing A in S

Together, these 3 elements constitute an intentional representation of the state space The search algorithm transforms this intentional representation into an

extensional one, by repeatedly applying the successor operation starting from the initial state

Define a boolean operation testing whether a state is a goal, ex., a method of C

For optimization problems: additionally define a operation that returns the cost or utility of an action or state

Problem Formulation Most Crucial FactorProblem Formulation Most Crucial Factorof Search Efficiencyof Search Efficiency

Problem formulation is more crucial than choice of search algorithm or choice of heuristics to make an agent decision problem effectively solvable by state space search

8-queens problem formulation 1: Initial state: empty board Action: pick column and line of one queen Branching factor: 64 State-space: ~648

8-queens problem formulation 2: Initial state: empty board Action: pre-assign one column per queen,

pick only line in pre-assigned column Branching factor: 8 State-space: ~88

Generic Exhaustive Search AlgorithmGeneric Exhaustive Search Algorithm

1.Initialize the fringe to the root node representing the initial state2.Until goal node found in fringe, repeat:

a.Choose one node from fringe to expand by calling its successor operationb.Extend the current fringe with the nodes generated by this successor operationc. If optimization problem, update path cost or utility value

3.Return goal node or path from root node to goal node

Specific algorithms differ in terms of the order in which they respectively expand the fringe nodes

Arad Fagaras Oradea R.Vilcea Arad Lugoj Arad Oradea

Sibiu Timisoara Zenrid

Arad fringe

Generic Exhaustive Search AlgorithmGeneric Exhaustive Search Algorithm

1.Initialize the fringe to the root node representing the initial state2.Until goal node found in fringe, repeat:

a.Choose one node from fringe to expand by calling its successor operationb.Extend the current fringe with the nodes generated by this successor operationc. If optimization problem, update path cost or utility value

3.Return goal node or path from root node to goal node

Specific algorithms differ in terms of the order in which they respectively expand the fringe nodes

Arad Fagaras Oradea R.Vilcea Arad Lugoj Arad Oradea

Sibiu Timisoara Zenrid

Arad open-list

fringe

Search Algorithms CharacteristicsSearch Algorithms Characteristics and Performance and Performance

Complete: guaranteed to find a solution if one exists Optimal (for optimization problem): guaranteed to find the best (highest

utility or lowest cost) solution if one exists Input parameters to complexity metrics:

b = problem search tree branching factor d = depth of highest solution (or best solution for optimization problems) in

problem search tree m = problem search tree depth (can be infinite)

Complexity metrics of algorithms: TimeComplexity(b,d,m) = number of expanded nodes SpaceComplexity(b,d,m) = maximum number of nodes needed in memory at

one point during the execution of the algorithm

Exhaustive Search AlgorithmsExhaustive Search Algorithms

Breadth-First: Expand first most shallow node from fringe Uniform Cost: Expand first node from fringe of lowest cost (or highest

utility) path from the root node Depth-First: Expand first deepest node from fringe Backtracking: Depth first variant with fringe limited to a single node Depth-Limited: Depth-first stopping at depth limit N. Iterative Deepening: Sequence of depth limited search at increasing

depth Bi-directional:

Parallel search from initial state and from goal state Solution found when the two paths under construction intersect

Breadth-First SearchBreadth-First Search

A

B C

E FD G

K MI OJ LH N

Fringe

Breadth-First SearchBreadth-First Search

A

B C

E FD G

K MI OJ LH N

Fringe

Expanded

Uniform Cost SearchUniform Cost Search

A C E

B

D

Problem Graph:

1 10

515

5 5

A

B C D

1 5 15

11

A

B C D

5 15

EB

A

B C D

11 10

15

EB Ec

A

Depth-First SearchDepth-First Search

A

B C

E FD G

J LH NK MI O

Backtracking SearchBacktracking Search

A

B C

E FD G

J LH NK MI O

Depth-First SearchDepth-First Search

A

B C

E FD G

J LH NK MI O


A

B C

E FD G

J LH NK MI O


A

B C

E FD G

J L NK MI O


A

B C

E F G

J L NK M O


A

B C

E F G

L NK M O


A

C

F G

L NM O

Iterative DeepeningIterative Deepening

A

A

B C

A

B C

D E D E

A

B C

D E

A

B C

D E D E

AL = 0

AL = 1 A

B C

A

B C

AL = 2 A

B C

A

B C

D E

A

B C

D E

A

B C

D E D E

Bi-Directional SearchBi-Directional Search

Two parallel searches: one from the current state and one from the goal state

When they reach a common node: a path from current to goal has been found

Time complexity halved: O(bd/2) + O(bd/2) = O(bd/2) << O(bd) But not always

possible: Irreversible

actions Large number

of intentionallyspecifiedgoal states

Comparing Search AlgorithmsComparing Search Algorithms

C* = cost of optimal solution a actions(agent), cost(a) e

Breadth-First

Uniform-Cost

Depth-First

Back-tracking

Iterative Deepening

Bi-Directional

Complete if b finite if all step costs

positives

no no if b finite depends on search used

on each directionOptimal if all steps

share same cost

yes no no if all steps share

same cost

Time Complexit

y

O(bd+1) O(bC*/e) O(bm) O(bm) O(bd) O(bd/2)

Space Complexit

y

O(bd+1) O(bC*/e) O(b.m)

O(m) O(b.d) O(bd/2)

Searching EnvironmentsSearching Environmentswith Repeated Stateswith Repeated States

In some search problems, each environment state is only reachable through a single action sequence ex, such as the N-Queens problem in the pre-assigned column formulation, However, for most problems, the same environment state can be reached through several, many or even an

infinity of distinct action sequences, This leads to the repeated generation of the same nodes in different branches

of the search tree

saa

sab

saz

sba

sbb

sbz

sza

szb

szz

saa

sab sba szasaz

saa sbb saa sbb sbz saa sbz szz saaszb szb szz

Effects of Repeated States onEffects of Repeated States onExhaustive Search AlgorithmsExhaustive Search Algorithms

They must be modified to include test comparing newly generated nodes with already generated ones and prune the repeated ones from the frontier

Without such test: Breadth-first and uniform-cost search fill memory far sooner with repeated

nodes and likely before reaching the depth of the first goal node Depth-first and backtracking search can enter in deepening loop, never

returning even in the presence of shallow goal nodes With such test

Depth-first, backtracking and iterative deepening search loose their linear worst-case space complexity,

for any guarantee to avoid all loops may require keeping an exponential number of expanded nodes in memory,

in practice, the probability of loop occurrence is traded-off for size of the expanded node history

Sensorless and Contigency Search Sensorless and Contigency Search ProblemsProblems

All algorithms presented until now are adequate only for off-line, fully informed search problems

How to deal with partial knowledge of: The current environment state? The effects of the available actions?

Simple yet not scalable solutions: Search in the possible state set space Each node in the search tree or graph corresponds to a set of possible states Each action labeled arc in the search tree or graph leads from one set of

possible states to another set of possible states Scalable solution:

Knowledge-based agents, probabilistic planning How to deal with lack of environment model?

Adaptive agents, machine (reinforcement) learning

Searching in the Possible States Set Space Searching in the Possible States Set Space

Heuristic Search:Heuristic Search:Definition and MotivationDefinition and Motivation

Definition: Non-exhaustive, partial state space search strategy, based on approximate heuristic knowledge of the search problem class (ex, n-queens,

Romania touring) or family (ex, unordered finite domain constraint solving) allowing to leave unexplored (prune) state space zones that are either guaranteed or

unlikely to contain a goal state (or a utility maximizing state or cost minimizing path) Note:

An algorithm that uses a fringe node ordering heuristic to generate a goal state faster, but is still ready to generate all state space states if necessary to find goal state i.e., an algorithm that does no pruning is not a heuristic search algorithm

Motivation: exhaustive search algorithms do not scale up, neither theoretically (exponential worst case time or space complexity) nor empirically (experimentally measured average case time or space complexity)

Heuristic search algorithms do scale up to very large problem instances, in some cases by giving up completeness and/or optimality

New data structure: heuristic function h(s) estimates the cost of the path from a fringe state s to a goal state

Best-First Global SearchBest-First Global Search

Keep all expanded states on the fringe just as exhaustive breadth-first search and uniform-cost search

Define an evaluation function f(s) that maps each state onto a number Expand the fringe in order of decreasing f(s) values Variations:

Greedy Global Search (also called Greedy Best-First Search) defines f(s) = h(s)

A* defines f(s) = g(s) + h(s) where g(s) is the real cost from the initial state to the state s, i.e., the value used to choose the state to expand in uniform cost search

Greedy Local Search: ExampleGreedy Local Search: Example

Greedy Local Search CharacteristicsGreedy Local Search Characteristics

Strengths: simple Weaknesses:

Not optimal because it relies only on estimated cost from current state to goal while ignoring confirmed cost from initial state to current state ex, misses better path through Riminicu and Pitesti

Incomplete can enter in loop between two states that seem heuristically closer to the goal but

are in fact farther away ex, from Iasi to Fagaras, it oscillates indefinitely between Iasi and Meant because the

only road from either one to Fagaras goes through Valsui which in straight line is farther away to Fagaras than both

A* ExampleA* Example

h(s): straight-line distance to Bucharest:

75 + 374374449449

140 + 253253

393393118 + 329329447447

220

239239 + 178178

417417

220 + 193193

413413

366

317317 + 9898

415415

336 + 160160496496

455

418

A* Search CharacteristicsA* Search Characteristics

Strengths: Graph A* search is complete and Tree A* search is complete and optimal if h(s) is an

admissible heuristic, i.e., if it never overestimates the real cost to a goal Graph A* search if optimal if h(s) admissible and in addition a monotonic (or consistent)

heuristic h(s) is monotonic iff it satisfies the triangle inequality, i.e., s,s’ stateSpace (a actions, s’ = result(a,s)) h(s) cost(a) + h(s´)

A* is optimally efficient,i.e., no other optimal algorithm will expand fewer nodes than A* using the same heuristic function h(s)

Weakness: Runs out of memory for large problem instance because it keeps all generated nodes in

memory Why? Worst-case space complexity = O(bd), Unless n, |h(n) – c*(n)| O(log c*(n)) But very few practical heuristics verify this property

A* Search CharacteristicsA* Search Characteristics

A* explores generates a contour around the best path The better the heuristic estimates the real cost, the narrower the contour Extreme cases:

Perfect estimates, A* only generates nodes on the best path Useless estimates, A* generate all the nodes in the worst-case, i.e., degenerates into

uniform-cost search

Recursive Best-First Search (RBFS)Recursive Best-First Search (RBFS)

Fringe limited to siblings of nodes along the path from the root to the current node

For high branching factors, this is far smaller than A*’s fringe which keeps all generated nodes

At each step: Expand node n with lowest f(n) to generate successor(n) = {n1, ..., ni} Store at n:

A pointer to node n’ with the second lowest f(n’) on the previous fringe Its cost estimate f(n’)

Whenever f(n’) f(nm) where f(nm) = min{f(n1), ..., f(ni)}: Update f(n) with f(nm) Backtrack to f(n’)

RBFS: ExampleRBFS: Example

RBFS: CharacteristicsRBFS: Characteristics

Complete and optimal for admissible heuristics Space complexity O(bd) Time complexity hard to characterize

In hard problem instance, can loose a lot of time swinging from one side of the tree to the other, regenerating over and over node that it had erased in the previous swing to

that direction in such cases A* is faster

Heuristic Function DesignHeuristic Function Design

Desirable properties: Monotonic, Admissible, i.e., s stateSpace, h(s) c*(s), where c*(s) is the real cost of s High precision, i.e., as close as possible to c*

Precision measure: effective branching factor b*(h) 1, N = average number of nodes generated by A* over a sample set of runs using h as

heuristic Obtained by solving the equation: b*(h) + (b*(h))2 + ... + (b*(h))d = N The closer b*(h) gets to 1, the more precise h is

h2 dominates h1 iff: s stateSpace, h1(s) h2(s),i.e., if b*(h1) closer to 1 than b*(h2)

hd(n) = max{h1(n), .., hk(n)} always dominates h1(n), .., hk(n) General heuristic function design principle:

Estimated cost = actual cost of simplified problem computed using exhaustive search as a cheap pre-processing stage

Problem can be simplified by: Constraint relaxation on its actions and/or goal state, which guarantees admissibility

and monotonicity Decomposition in independent sub-problems

Heuristic Function Design: Heuristic Function Design: Constraint Relaxation ExampleConstraint Relaxation Example

Constraints:1. Tile cannot move diagonally 2. Tile cannot move in occupied location3. Tile cannot move directly to non-

neighboring locations

Relaxed problem 1: Ignore all constraints

h1: number of tiles out of place Relaxed problem 2:

Ignore only constraint 2

h2: sum over all tiles t of the Manhattan distance between t’s current and goal positions

h2 dominates h1

Heuristic Function Design:Heuristic Function Design:Disjoint Pattern DatabasesDisjoint Pattern Databases

Preprocessing for one problem class amortized over many run of different instances of this class

For each possible sub-problem instance: Use backward search from the goal to compute its cost, counting only the cost of the actions involving the entities of the sub-problem ex, moving tiles 1-4 for sub-problem 1, tiles 5-8 for sub-problem 2 store this cost in a disjoint pattern database

During a given full problem run: Divide it into sub-problems Look up their respective costs in the database Use the sum of these costs as heuristic

Only work for domains where most actions involve only a small subsets of entities

Sub-problems: A: move tiles {1-4} B: move tiles {5-8}

Local Heuristic SearchLocal Heuristic Search

Search algorithms for complete state formulated problems for which the solution is only a target state that satisfies a goal or maximizes a utility function and not a path from the current initial state to that target state;

Only keeps a very limited fringe in memory, often only the direct neighbors of the current node, or even only the current node.

Best local heuristic search algorithms are stochastic algorithms implement an agent with non-deterministic behavior

Far more scalable than global heuristic search, though generally neither complete nor optimal

Frequently used for multi-criteria optimization Easily adaptable to online search and continuous space search In continuous space search, the radius of the local neighborhood

generated by the successor function is a key parameter of success

Ridge

Basic Hill-Climbing (HC)Basic Hill-Climbing (HC)

Fringe limited to immediate neighbors in state space, no explicit search tree Always move to neighbor node which most improves heuristic function value (steepest uphill neighbor)

until no neighbor improves such function value This is greedy local search Strengths: simple, very space scalable, works without modification for partial information and online search Weaknesses: incomplete, not optimal, "an amnesic climbing the Everest on a foggy day"

Hill-Climbing VariationsHill-Climbing Variations

HC with sideway moves: When estimated utility of all neighboring states are equal to that of current node move

to one of them instead of stopping Avoid getting stuck in shoulders, but number of allowed consecutive sideways moves

must be bounded to avoid infinite loop in flat local maxima In 8-Queens problem HC w/ up to 100 consecutive sideway moves solves 94% of

problem instances instead of a mere 14% for basic HC Stochastic HC: instead of systematically picking steepest uphill move,

stochastically chose among uphill moves following a probability distribution favoring steeper moves Slower convergence, but better result for various landscape classes

First-Choice HC: variation of stochastic HC that randomly generates random neighbors until an uphill one is found and chosen, instead of systematically generating all neighbors before choosing among them Efficient for state spaces with very high branching factor

HC with random restart: repeatedly perform multiple HC from different randomly chosen initial state to "pull out" from local maxima, shoulders and ridges Find solutions to 3.106-Queen problem in less than a minute

Other Local Search AlgorithmsOther Local Search Algorithms

Random Walk: randomly move among neighbor states Complete but extremely inefficient

Simulated Annealing: combines characteristics of HC with RW to find best compromise between completeness and efficiency Generates one random neighbor state If it is uphill always pick it Otherwise, pick it with a probability that

Is lower for the steeper downhill moves Decreases over time

Local beam search: Maintain a current fringe of k nodes Form the new fringe by expanding at once the k successors state with highest

utility from this entire fringe to form the new fringe Stochastic beam search:

At each step, pick successors semi-randomly with nodes with higher utility having a higher probability to be picked

It is a form of genetic search with asexual reproduction

On-Line SearchOn-Line Search

At each cycle sense-reason-act the search-based reasoning component has access to the following information: actions(State):Set(Action), which returns the set of actions that can be

executed in the current environment state goalTest(State):Boolean, which defines whether the current state matches the

agent’s goal costFunction(CurrentState,Action,NextState), which defines the cost

associated with the arc labeled by Action from the node CurrentState to the node NextState as opposed to off-line search, this can only be called after the execution of Action for an online search agent has no successor function i.e, it has no model to predict the consequences of its actions it can only sense these consequences of one action after having chosen and

executed it

On-Line Search Problem ExampleOn-Line Search Problem Example

Initial state: agent in S = [1,1] Unique goal state: agent in G = [3,3] Path optimization problem: agent must discover lowest

cost path from S to G Sensor: provides agent position [X,Y] in the maze Deterministic actions: GoRight, GoLeft, GoUp, GoDown Uniform cost: all actions share unitary cost Admissible heuristic: Manhattan distance from agent’s

current position to G But no action model:

e.g., agent does not know that result(GoRight,[1,1]) = [1,2]), that result(GoLeft,[1,2]) = [1,2]), etc.

On-Line Search Performance MetricOn-Line Search Performance Metric

Competitive ratio of an online search algorithm: For a specific problem instance: Cost of action

sequence found by algorithm / Minimum cost of all possible action sequences

For a problem class: Average cost of action sequences found by the algorithm on a representative sample of all problem instances / Average minimum cost of all possible action sequences over the same sample

For problems with some irreversible action or some unbounded cost path the competitive ratio of any online algorithm is infinite

An environment is safely explorable if at least one of the goal state is reachable from every reachable state

Adapting Off-Line AlgorithmsAdapting Off-Line Algorithmsto On-Line Searchto On-Line Search

Which off-line search algorithm easiest to adapt to online case? Local ones Why? Because they do not not involve jumps (teletransportation) from one

node (environment state) to a distant node (environment state) in the search tree or graph (state space) which is possible for off-line search but not for on-line search

Best base candidates: Hill-Climbing, First-Choice Hill-Climbing, Simulated Annealing Depth-First Search (and its specialization chronological bracktracking search)

But isn’t Depth-First Search a global algorithm? While it works on partial-state formulations like Breadth-First Search, Uniform Cost

Search, A* or even RBFS it has a more local flavor than those algorithms; Similarly, algorithms using random restarts, neighbor beams or state

populations while they work on full-state formulations have a less local flavor than those a single current state at a time and no restart

On-Line Depth First SearchOn-Line Depth First Search

Learns successor function as it explores previously unknown environment After executing each action, adds one mapping to successor function Upon reaching a new, non-goal state:

Randomly choose among untried executable actions for that state Upon reaching a dead-end state:

i.e., a state for which all executable actions have been tried without reaching a goal state

executes chronological backtracking by performing the inverse action of the one that led to the dead-end

After executional backtracking: avoids going back to the identified dead-ends by keeping for each state the list of the predecessor states from which it has not backtracked

In the worst case, tries all pairs (state,action) exactly twice Optimal for exploration (i.e., agent goal is to learn successor function) Unbounded competitive ratio for exploitation (i.e., agent goal is to find as fast as

possible a state or path that satisfies a given characteristic) Because it can go off to an arbitrarily long excursion in a wrong direction

On-line iterative deepening search avoids going astray indefinitely

On-Line Depth First Search ExampleOn-Line Depth First Search Example

G

S, A

G

A

S

G

S, A

G

S A


G

S A

G

A

S

A G

S

A G

S


A G

S

G

A

S

G

S A

G

S A


G

S A

G

A

S

G

S

In which Environments can a State Space In which Environments can a State Space Searching Agent (S3A) Act Successfully?Searching Agent (S3A) Act Successfully?

Fully observable? Partially observable? Sensorless? Deterministic or stochastic? Discrete or continuous? Episodic or non-episodic? Static? sequential? concurrent synchronous or concurrent asynchronous? Small or large? Lowly or highly diverse?

ontologies reasoning components agents simulations agents solving problems by environment state...

Documents

search treeeach search

current state

initial state

generated state

navigating state space

action sequencein search

searchtreesthe state

order of state generation