Intelligent Agents

Module 3
Part 1
Solving Problems by Searching

• Reflex agent is simple

• base their actions on
• a direct mapping from states to actions
• but cannot work well in environments
• which this mapping would be too large to store
• and would take too long to learn

• Hence, goal-based agent is used

Agents

• Agent
• Problem-solving agents
• Agent may need to plan ahead
• Uses Search Algorithms
• To consider a sequence of actions that form a path to a goal state
• Consider Atomic representations
• Planning agents
• Consider Factored or Structured representations instead

• Search Algorithms
• Informed algorithms
• Agent can estimate how far it is from the goal
• Uninformed algorithms
• No such estimate is available.

3
Example : Partial map of Romania

4
Problem-solving agents

• Goal formulation
• A set of states, where the goal is satisfied
• Reaching from initial state -> goal state
• Actions are required
• Example Goal: Initial State(Arad) -> Goal State(Bucharest)
• Problem formulation
• process of deciding what actions and states to consider
• Example:
• Traveling from one city to an adjacent city
• Arad -> Sibiu

5
Problem-solving agents

• Search
• Many ways to achieve the same goal
• Simulate multiple sequences to reach the goal
• Solution
• Sequence of actions that reaches the goal
• Possibility of NO solutions
• Example: Arad -> Sibiu -> Fagaras -> Bucharest

• Execution
• Execute the actions in the solution
• One at a time

6
Problem-solving agents

• Property of a fully observable, deterministic, known environment

• The solution to any problem is a fixed sequence of actions
• Solution is guaranteed to lead to the goal

• Open-loop system
• Agent ignores the percepts
• Breaks the loop between agent and environment

• Closed-loop system
• Agent monitors the percepts
• Chance of the model being incorrect
• Or, the environment is nondeterministic

7
Problem-solving agents

• function Simple-Problem-Sol ving-Agent( percept) returns an action

• static:
• seq, an action sequence, initially empty
• state, some description of the current world state
• goal, a goal, initially null
• problem, a problem formulation
• state ← Update-State(state, percept)
• if seq is empty then
• goal ← Formulat e - G oa l (state)
• problem ← Formulat e - P roblem (state, goal)
• seq ← Search( problem)
• action ← Recommendation (seq, state)
• seq ← Remainder(seq, state)

• return action
8
Example: Romania

• On holiday in Romania
• currently in Arad.
• Flight leaves tomorrow from Bucharest

• Formulate goal:
• be in Bucharest
• Formulate problem:
• states: various cities
• actions: drive between cities
• Find solution:
• sequence of cities
• e.g., Arad, Sibiu, Fagaras, Bucharest

9
Problem types

• Deterministic, fully observable ⇒ single-state problem

• Agent knows exactly which state it will be in; solution is a sequence

• Non-observable ⇒ conformant problem

• Agent may have no idea where it is; solution (if any) is a sequence

• Nondeterministic and/or partially observable ⇒ contingency

problem
• percepts provide new information about current state
• solution is a contingent plan or a policy
• often interleave search, execution

• Unknown state space ⇒ exploration problem (“online”)

10
Example: vacuum world

• Single-state, start in #5.

• Solution??
• [Right, Suck]

• Conformant, start in {1, 2, 3, 4, 5, 6, 7, 8}

• e.g., Right goes to {2, 4, 6, 8}.
• Solution??
• [Right, Suck, Left, Suck]

• Contingency, start in #5
• Local sensing: dirt, location only.
• Solution??
• [Right, if dirt then Suck]

11
Single-state problem formulation

• State space
• A set of possible states that the environment can be in
• Initial state
• the agent starts in this state.
• For example: Arad
• A set of one or more goal states
• sometimes the goal is defined by a property that applies to many states
• specify an IS-GOAL() method for a problem
• The actions available to the agent ⇒ ACTIONS(s)
• returns a set of actions that can be executed in s
• Given a state s
• example: ACTIONS(Arad) = {ToSibiu, ToTimisoara, ToZerind}

12
Single-state problem formulation

• A transition model ⇒ RESULT(s, a)

• Describes what each action does
• returns the state that results from doing action a in state s.
• Example: RESULT(Arad, ToZerind) = Zerind
• An action cost function ⇒ ACTION-COST(s,a,s′)
• gives the cost of applying action a in state s to reach state s′
• Example: ACTION-COST(Arad, ToZerind, Zerind) = 75

• Path
• Formed by a sequence of actions
• Solution
• path from the initial state to a goal state
• An optimal solution has the lowest path cost among all solutions

13
 • State-space graph
Example : Revisited • The vertices are states
• The directed edges are actions

14
Abstraction in Problem Formulation

• Real world is absurdly complex

• state space must be abstracted for problem solving

• (Abstract) state
• set of real states
• (Abstract) action
• complex combination of real actions
• e.g., “Arad → Zerind” represents a complex set of possible routes, detours, rest
stops, etc.
• For guaranteed realizability,
• any real state “in Arad” must get to some real state “in Zerind”
• (Abstract) solution
• set of real paths that are solutions in the real world
• Each abstract action should be “easier”
• than the original problem!

15
Example: vacuum world state space graph

• states: (integer) dirt and robot locations (ignore dirt amounts etc.)
• actions: Left, Right, Suck
• goal states: no dirt
• path cost: 1 per action
16
Example: The 8-puzzle

• states: integer locations of tiles (ignore intermediate positions)

• actions: move blank left, right, up, down (ignore unjamming etc.)
• goal states: (Given above)
• path cost: 1 per move
17
Example: The 8-puzzle

18
Example: robotic assembly

R R

R R

• states: coordinates of robot joint angles; parts of the object to be assembled

• actions: continuous motions of robot joints
• goal states: complete assembly (with no robot included!)
• path cost: time to execute
19
Search Algorithms
Module 3
Part 2

20
Search algorithms

• Takes a search problem as input

• Returns a solution, or an indication of failure.
• Search tree
• Superimpose various paths from the initial state (root node)
• from state-space graph
• find a path that reaches a goal state
• Node
• a data structure constituting part of a search tree
• corresponds to a State
• a representation of a physical configuration
• States do not have: Children, Depth, Path cost
• Includes parent, children, depth, path cost g(x)
• Edges
• correspond to actions

21
Implementation: states vs. nodes

• EXPAND function
• Creates new nodes (child node or successor node)
• considering available ACTIONS for the present state
• using the RESULT function for destination
• Create the corresponding states
• using the SUCCESSOR function
• Fills in the various fields
• Sets Parent node to the present state

22
Tree search algorithms

• Basic idea:
• offline and simulated exploration of state space
• by generating successors of already-explored states (expanding states)

function Tree-Search(problem, strategy) returns a solution, or failure

initialize the search tree using the initial state of problem
loop do
if there are no candidates for expansion then return failure
choose a leaf node for expansion according to strategy
if the node contains a goal state then return the corresponding solution
else expand the node and add the resulting nodes to the search tree
end

23
Tree search example
Arad

Sibiu Timisoara Zerind

Arad Fagaras Oradea RimnicuVilcea Arad Lugoj Arad Oradea

24
Tree search example
Arad

Sibiu Timisoara Zerind

Arad Fagaras Oradea Rimnicu Vilcea Arad Lugoj Arad Oradea

25
Tree search example
Arad

Sibiu Timisoara Zerind

Arad Fagaras Oradea Rimnicu Vilcea Arad Lugoj Arad Oradea

26
General tree searching
function Tree-Search( problem, frontier) returns a solution, or failure
frontier ← Insert (Make-Node(Initial-Stat e [problem]),frontier)
loop do
if frontier is empty then return failure
node ← Remove-Fro n t (frontier)
if Goal-Test(problem,State(node)) then return node
frontier ← Insert A l l (Expand(node,problem),frontier)
end
function Expand( node, problem) returns a set of nodes
successors ← the empty set
for each action, result in Successor-Fn(problem,State[node]) do
s ← a new Node
Parent-Node[s]← node
Action [s]← action
St at e [s] ← result
Path-Cost[s] ← Path-Cost[node] + Action-Cost(node,action,s)
Depth[s] ← Depth[node] + 1
add s to successors
return successors 27
Search Data Structures and Methods

The node data structure

• node.STATE
• the state to which the node corresponds
• node.PARENT
• the node in the tree that generated this node
• node.ACTION
• the action that was applied to the parent’s state to generate this node
• node.PATH-COST
• the total cost of the path from the initial state to this node.
• In mathematical formulas, we will call this as g(node)

28
Search Data Structures and Methods

• The frontier data structure

• Use of queue
• Operations on frontier
• IS-EMPTY(frontier)
• returns true only if there are no nodes in the frontier.
• POP(frontier)
• removes the top node from the frontier and returns it.
• TOP(frontier)
• returns (but does not remove) the top node of the frontier.
• ADD(node, frontier)
• inserts node into its proper place in the queue.
• Choice of queue
• FIFO / LIFO / priority queue
• Priority queue
• first pops the node with the minimum cost
• according to some evaluation function, f.
• It is used in best-first search.
29
Best-first Search
based on evaluation function f(n)

30
Search strategies

• A strategy is defined by picking the order of node expansion

• Strategies are evaluated along the following dimensions:

• Completeness
• does it always find a solution if one exists?
• Time Complexity
• number of nodes generated/expanded
• Space Complexity
• maximum number of nodes in memory
• Cost Optimality
• does it always find a least-cost solution?

• Time and space complexity are measured in terms of

• b: maximum branching factor of the search tree
• d: depth of the least-cost solution
• m: maximum depth of the state space (may be ∞)

31
Uninformed Search Strategies
Module 3
Part 3

32
News
Break

Source: https://www.bbc.com/ 33
Uninformed search strategies

• Use only the information available in the problem definition

• Also known as blind searching

• Examples:
1. Breadth-first search
2. Uniform-cost search
3. Depth-first search
4. Depth-limited search
5. Iterative deepening search

34
Breadth-First Search

• Recall from Data Structures the basic algorithm for a breadth-first

search
• on a graph or tree

1. Expand the shallowest unexpanded node

2. Place all new successors at the end of a FIFO queue

35
Breadth-First Search

36
Properties of Breadth-First Search

• Complete
• Yes, if b (max branching factor) is finite
• Time
• 1 + b + b2 + … + bd = O(bd)
• exponential in d
• Space
• O(bd)
• Keeps every node in memory
• This is the big problem
• If an agent generates nodes at 10 MB/sec → produce 860 MB in 24 hours
• Optimal
• Yes (if cost is 1 per step)
• not optimal in general

37
Lessons From Breadth First Search

• The memory requirements are a bigger problem

• than is execution time

• Exponential-complexity search problems

• cannot be solved by uniformed methods for any
• Except the smallest instances

38
Uniform-cost search (AI)
Dijkstra’s algorithm (Theo.CompSc.)

• Evaluation function
• the cost of the path from the root to the current node
• BEST-FIRST-SEARCH
• evaluation function => PATH-COST

• Example:

39
Uniform-cost search / Dijkstra’s algorithm

• Complete
• Yes if the cost is greater than some threshold
• step cost >= ε
• Time
• Complexity cannot be determined easily by d or d
• Let C* be the cost of the optimal solution
• O(bceil(C*/ ε)) or O(b1+floor(C*/ ε))
• Space
• O(bceil(C*/ ε)) or O(b1+floor(C*/ ε))
• Optimal
• Yes, Nodes are expanded in increasing order

40
Depth-First Search (DFS)

• Recall from Data Structures the basic algorithm for a depth-first

search
• on a graph or tree

1. Expand the deepest unexpanded node

2. Unexplored successors are placed on a stack until fully
explored

41
Depth-First Search:
example binary tree

42
Depth-First Search:
example binary tree

43
Depth-First Search:
example binary tree

44
Depth-First Search

• Complete
• Yes: in finite state spaces
• No: fails in
• infinite-depth state spaces
• spaces with loops
• Modify to avoid repeated spaces along path
• Time
• O(bm)
• Not great if m (max depth) is much larger than d
• But if the solutions are dense, this may be faster than breadth-first search
• Space
• O(bm)
• linear space
• Optimal
• No

45
Depth-Limited Search

• A variation of depth-first search

• that uses a depth limit (l )

• Alleviates the problem of unbounded trees

• Search to a predetermined depth l (“ell”)
• Nodes at depth l (assumed to) have no successors

• Same as depth-first search if l = ∞

• Can terminate for failure and cutoff

46
Depth-Limited Search
Recursive implementation:
function Depth-Limited-Search( problem,limit) returns soln/fail/cutoff
Recursive-DLS(Make-Node(Initial-Stat e [problem]),problem,limit)

function Recursive-DLS(node, problem,limit) returns soln/fail/cutoff

cutoff-occurred? ← false
if Goal-Test(problem,State[node]) then return node
else if Depth[node] = limit then return cutoff
else for each successor in Expand(node,problem) do
result ← Recursive-DLS(successor,problem,limit)
if result = cutoff then cutoff-occurred? ← true
else if result /= failure then return result
if cutoff-occurred? then return cutoff else return failure

47
Depth-Limited Search

• Complete
• Yes if l < d
• Time
• O(bl)
• Space
• O(bl)
• Optimal
• No if l > d

48
Iterative Deepening Search (IDS)

• Iterative deepening depth-first search

• Uses depth-first search

• Finds the best depth limit

• Gradually increases the depth limit
• 0, 1, 2, … until a goal is found

49
Iterative Deepening Search

50
IDS: Example

51
IDS: Example

52
Iterative Deepening Search

• Complete
• Yes
• Time
• (d ) b 1 + (d − 1)b 2 + (d − 2)b 3 + . . . + b d = O(b d )
• Space
• O(bd)
• Optimal
• far-right if step cost = 1
• Can be modified to explore uniform cost tree

• Numerical comparison for b = 10 and d = 5 ,

• solution at far-right leaf
• N (IDS) = 50 + 400 + 3,000 + 20,000 + 100,000 = 123,450
• N (BFS) = 10 + 100 + 1,000 + 10,000 + 100,000 + 999, 990 = 1,111,100

53
Iterative Deepening Search: Take-away

• Faster than BFS

• even though IDS generates repeated states

• BFS generates nodes up to level d+1

• IDS only generates nodes up to level d

• In general, iterative deepening search is the preferred uninformed

search method
• when there is a large search space, and
• the depth of the solution is not known

• BFS can be modified to apply goal test

• when a node is generated

54
Bidirectional Search

• Previous Algorithms
• start at an initial state
• can reach any one of multiple possible goal states
• Bidirectional search
• Simultaneously searches
• forward from the initial state, and
• backwards from the goal state(s),
• Hopes that the two searches will meet

• Motivation
• (bd/2 +bd/2) is much less than (bd)

55
Bidirectional Best-First Search

56
Bidirectional Best-First Search

57
Comparison of Uninformed Search Strategies

1. complete if b is finite, and the state space either has a solution or is finite
2. complete if all action costs are ≥ ε > 0
3. cost-optimal if action costs are all identical
4. if both directions are breadth-first or uniform-cost.

58
Repeated States

• Failure to detect repeated states can turn a

• path to be infinite -> loop forever
• linear problem into an exponential one!

59
Avoiding repeated states

• Algorithms that forget their history

• are doomed to repeat it
• Three ways to deal with this possibility
1. Do not return to the state it just came from
• Refuse generation of any successor same as its parent state
2. Do not create paths with cycles
• Refuse generation of any successor same as its ancestor states
3. Do not generate any generated state
• Not only its ancestor states, but also all other expanded states have to be checked
against
• We then define a data structure
• closed list
• a set storing every expanded node so far
• If the current node matches a node on the closed list, discard it.

60
Avoiding repeated states: Graph search

function Graph-Search( problem, frontier) returns a solution, or failure

closed ← an empty set
frontier ← Insert (Make-Node(Initial-Stat e [problem]),frontier)
loop do
if frontier is empty then return failure
node ← Remove-Fro n t (frontier)
if Goal-Test(problem,State[node]) then return node
if State[node] is not in closed then
add State[node] to closed
frontier ← Insert A l l (Expand(node,problem),frontier)
end

61
Graph search

• Using Depth-first search for Graph Search

• Completeness
• for finite state spaces
• Time complexity
• Bounded by the size of the state space
• i.e. the number of vertices and edges, |V | + |E|
• Space complexity
• Bounded by the size of the state space
• i.e. the number of vertices and edges, |V | + |E|

62
Informed (Heuristic) Search
Strategies
Module 3
Part 4

63
Informed (Heuristic) Search

• Informed
• use problem-specific knowledge
• Which search strategies?
• Best-first search and its variants
• Heuristic functions?
• “A rule of thumb, simplification, or educated guess that reduces or limits
the search for solutions in domains that are difficult and poorly
understood.”
• h(n) = estimated cost of the cheapest path from node n to goal
node.
• If n is goal then h(n)=0

64
Greedy Search

• Evaluation function h(n) (heuristic)=

• estimate of cost from n to the closest goal
• hSLD=
• straight-line distance heuristic.
• can NOT be computed from the problem description itself

• In the next example f(n)=h(n)

• Expand node that is closest to goal
• = Greedy best-first search

65
Example: Romania map with hSLD

66
Example: Arad to Bucharest

Arad
366

67
Example: Arad to Bucharest

Arad

Sibiu Timisoara Zerind

253 329 374

68
 Example: Arad to Bucharest

Arad

Sibiu Timisoara Zerind

329 374

Arad Fagaras Oradea Rimnicu Vilcea

366 176 380 193

69
 Example: Arad to Bucharest

Arad

Sibiu Timisoara Zerind

329 374

Arad Fagaras Oradea Rimnicu Vilcea

366 380 193

Sibiu Bucharest
253 0

70
Properties of greedy search

• Complete?
• No–can get stuck in loops
• e.g., Iasi → Neamt → Iasi → Neamt →
• Complete in finite space with
repeated-state checking
• Time?
• O(bm),
• but a good heuristic can give
dramatic improvement
• Space?
• O(bm)—keeps all nodes in memory
• Optimal?
• No

71
A* search

• Best-known form of best-first search

• Idea
• avoid expanding paths that are already expensive.

• Evaluation function f(n)=g(n) + h(n)

• g(n) =
• the cost (so far) to reach the node.
• h(n)
• estimated cost to get from the node to the goal.
• f(n)
• estimated total cost of path through n to goal.

72
A* search Example

Arad
366=0+366

73
A* search Example

Arad

Sibiu Timisoara Zerind

393=140+253 447=118+329 449=75+374

74
 A* search Example

Arad

Sibiu Timisoara Zerind

447=118+329 449=75+374

Arad Fagaras Oradea Rimnicu Vilcea

646=280+366 415=239+176 671=291+380 413=220+193

75
 A* search Example

Arad

Sibiu Timisoara Zerind

447=118+329 449=75+374

Arad Fagaras Oradea Rimnicu Vilcea

646=280+366 415=239+176 671=291+380

Craiova Pitesti Sibiu

526=366+160 417=317+100 553=300+253

76
 A* search Example

Arad

Sibiu Timisoara Zerind

447=118+329 449=75+374

Arad Fagaras Oradea Rimnicu Vilcea

646=280+366 671=291+380

Sibiu Bucharest Craiova Pitesti Sibiu

591=338+253 450=450+0 526=366+160 417=317+100 553=300+253

77
 A* search Example

Arad

Sibiu Timisoara Zerind

447=118+329 449=75+374

Arad Fagaras Oradea Rimnicu Vilcea

646=280+366 671=291+380

Sibiu Bucharest Craiova Pitesti Sibiu

591=338+253 450=450+0 526=366+160 553=300+253

Bucharest Craiova Rimnicu Vilcea

418=418+0 615=455+160 607=414+193

78
Consistency

• A heuristic h(n) is consistent if,

• for every node n and every successor n′ of n
• generated by an action a,
• we have: h(n) ≤ c(n,a,n′)+ h(n′)

• Triangle inequality:

79
Optimality of A ∗ (standard proof )

• Suppose some suboptimal goal G2 has been generated and is

in the queue.
• Let n be an unexpanded node on a shortest path to an
optimal goal G1. Start

G G2

f (G 2 ) = g(G 2 ) since h(G 2 ) = 0

> g(G 1 ) since G 2 is suboptimal
≥ f (n) since h is admissible
• Since f (G 2 ) > f (n), A∗ will never select G 2 for expansion 80
Optimality of A ∗ (proof by contradiction )

• Suppose the optimal path has cost C*,

• Assume: algorithm returns a path with (suboptimal) cost C>C*
• g*(n) = cost of the optimal path from the start to n
• h*(n) = cost of the optimal path from n to the nearest goal

• a contradiction,
• so, the assumption must be wrong
81
Optimality of A∗
(Search Contours)

• A* expands nodes in order of increasing f()

value

• Contours can be drawn in state space

• Uniform-cost search adds circles.
• Possible only with g() cost
• F-contours are gradually added by A*:

1) Expanded all nodes with f(n)<C*

2) Some nodes on the goal contour are
expanded ( where f(n)=C*).
3) No nodes with f(n)>C* are expanded

82
Properties of A∗

• Complete?
• Yes, unless there are infinitely many nodes with f ≤ f (G)
• Time?
• Exponential in [relative error in h X length of soln.]
• Space?
• Keeps all nodes in memory
• Optimal?
• Yes—cannot expand f i + 1 until f i is finished

83
Admissibility

• A heuristic is admissible if it never overestimates the

cost to reach the goal
• Are optimistic

Formally:
1. h(n) <= h*(n) where h*(n) is the true cost from n
2. h(n) >= 0 so h(G)=0 for any goal G.

e.g. hSLD(n) never overestimates the actual road distance

84
Admissible heuristics

• E.g., for the 8-puzzle:

• h1(n) = number of misplaced tiles
• h2(n) = total Manhattan distance
(i.e., no. of squares from desired location of each tile)

• h1(S) = 8
• h2(S) = 3+1+2+2+2+3+3+2 = 18
Dominance

• If h2(n) ≥ h1(n) for all n (both admissible),

• then h2 dominates h1 and is better for search

• Typical search costs:

• d=12 IDS = 3,644,035 nodes
A*(h1) = 227 nodes
A*(h2) = 73 nodes
• d=24 IDS = too many nodes
A*(h1) = 39,135 nodes
A*(h2) = 1,641 nodes

• Given any admissible heuristics ha, hb, h(n) = max(ha(n), hb(n))

• is also admissible and dominates ha, hb

86
Dealing with hard problems

• For large problems, A* often requires

• too much space.
• Two variations conserve memory: IDA* and SMA*
• IDA* (iterative deepening A* )
• uses successive iteration with growing limits on f.
• For example,
• A* but don’t consider any node n where f(n) >10
• A* but don’t consider any node n where f(n) >20
• A* but don’t consider any node n where f(n) >30, ...
• SMA* (Simplified Memory-Bounded A*)
• uses a queue of restricted size to limit memory use.
• throws away the “oldest” worst solution.
• Back it up to its parent, in case
Relaxed problems

• Relaxed problem
• A problem with fewer restrictions on the actions
• The cost of an optimal solution to a relaxed problem
• is an admissible heuristic for the original problem

• If the rules of the 8-puzzle are relaxed so that a tile can

move anywhere,
• then h1(n) gives the shortest solution

• If the rules are relaxed so that a tile can move to any

adjacent square,
• then h2(n) gives the shortest solution

88
Relaxed problems (2)

• Well-known example: travelling salesperson problem (TSP)

• Find the shortest tour visiting all cities exactly once

• Minimum spanning tree can be computed in O(n2)

• and is a lower bound on the shortest (open) tour

89
Take-away

• A problem consists of five parts:

• the initial state,
• a set of actions,
• a transition model describing the results of those actions,
• a set of goal states,
• an action cost function.

• Uninformed search methods have access only to the problem

definition.
• Algorithms build a search tree in an attempt to find a solution.

• Informed search methods have access to a heuristic function h(n)

• estimates the cost of a solution from n.

90
Search in Complex Environments
Module 3
Part 5

91
Local Search and Optimization Problems

• In many optimization problems, path is irrelevant

• the goal state itself is the solution

• Then state space = set of “complete” configurations

• find optimal configuration, e.g., TSP
• or, find configuration satisfying constraints, e.g., timetable

• In such cases, one can use iterative improvement algorithms

• keep a single “current” state, try to improve it

92
Local Search and Optimization Problems

• Local search algorithms

• can also solve optimization problems
• Optimization problems
• the aim is to find the best state according to an objective function

• If elevation corresponds to an objective function

• aim is to find the highest peak
• a global maximum
• we call the process hill climbing
• If elevation corresponds to cost
• then the aim is to find the lowest valley
• a global minimum
• we call it gradient descent

93
Local Search and Optimization Problems

• Local search algorithms operate

• by searching from a start state to neighboring states,
• without keeping track of
• the paths
• the set of states that have been reached.

• They are not systematic

• they might never explore a portion of the search space
• where a solution actually resides.

94
Local Search and Optimization Problems

• Local search algorithms have 2 key advantages

• They use very little memory
• They can find reasonable solutions
• in large or infinite (continuous) state spaces.

• Some examples of local search algorithms are

• Hill-climbing
• Simulated annealing
• Local beam search

95
Example: Travelling Salesperson Problem

• Start with any complete tour, perform pairwise exchanges

• Variants of this approach get within 1% of optimal very quickly

with thou- sands of cities

96
Example: n-queens

• Put n queens on an n × n board

• with no two queens on the same row, column, or diagonal
• Move a queen to reduce number of conflicts

• Almost always solves n-queens problems almost

instantaneously
• for very large n, e.g., n = 1million

97
Hill-climbing (or gradient ascent/descent)

• “Like climbing Everest in thick fog with amnesia”

function Hill-Climbing( problem) returns a state that is a local maximum

inputs: problem, a problem
local variables: current, a node
neighbor, a node
current ← Make-Node(Initial-Stat e [problem])
loop do
neighbor ← a highest-valued successor of current
if Value[neighbor] ≤ Value[current] then return State[current]
current ← neighbor
end

98
 Hill climbing example f(n) = -(number of tiles out of place)

2 8 3 1 2 3
start 1 6 4 h = -4 goal 8 4 h=0
7 5 7 6 5

-5 -5 -2

2 8 3 1 2 3
1 4 h = -3 8 4 h = -1
7 6 5 7 6 5

-3 -4
2 3 2 3
1 8 4 1 8 4 h = -2
7 6 5 7 6 5
h = -3 -4
Hill-climbing search: 8-queens problem

• h = number of pairs of queens that are attacking each other, either

directly or indirectly
• h = 17 for the state (right side)
• h = 1 for the state (left side)
100
Exploring the Landscape
local maximum
• Local Maxima
• peaks that aren’t the highest plateau
point in the space

• Plateaus
• the space has a broad flat region
• gives the search algorithm no
direction (random walk)
ridge

• Ridges
• flat like a plateau
• but with drop-offs to the sides
• steps to the North, East, South
and West may go down,
• but a step to the NW may go up.

Image from: http://classes.yale.edu/fractals/CA/GA/Fitness/Fitness.html

Hill-climbing remidies

• Useful to consider state space landscape

• Possible remedies
• Random-restart hill climbing: overcomes local maxima—trivially complete

• Random sideways moves: (i) escape from shoulders (ii)loop on flat maxima
102
Simulated Annealing

• Annealing
• thermal process
• obtains low energy states of a solid in a heat bath
• The process contains two steps:
• Increase the temperature of the heat bath to a maximum value
• at which the solid melts
• Decrease carefully the temperature of the heat bath
• Until the particles arrange themselves in the ground state of the solid
• Ground state is a minimum energy state of the solid

• The ground state of the solid is obtained only if

1. maximum temperature is high enough
2. the cooling is done slowly

103
Simulated Annealing
• Idea: escape local maxima by allowing some “bad” moves
• but gradually decrease their size and frequency

function Simulated-Annealing( problem, schedule) returns a solution state

inputs: problem, a problem
schedule, a mapping from time to “temperature”
local variables: current, a node
next, a node
T, a “temperature” controlling prob. of downward steps
current ← Make-Node(Initial-Stat e [problem])
for t ← 1 to ∞ do
T ← schedule[t]
if T = 0 then return current
next ← a randomly selected successor of current
∆ E ← Value[next] – Value[current]
if ∆ E > 0 then current ← next
else current ← next only with probability e∆ E/T
104
Properties of simulated annealing

• At fixed “temperature” T , state occupation probability reaches

Boltzman distribution
E(x)
p(x) = αe kT
• T decreased slowly enough =⇒ always reach best state x∗
• because for small T

• Is this necessarily an interesting guarantee?

• Devised by Metropolis et al., 1953, for physical process modelling
• Widely used in VLSI layout, airline scheduling, etc.

105
Local beam search

• Idea:
• keep k states instead of 1
• choose top k of all their successors
• Not the same as k searches run in parallel!

• Searches that find good states

• recruit other searches to join them
• Problem:
• quite often, all k states end up on same local hill
• Idea:
• choose k successors randomly,
• biased towards good ones
• Observe the close analogy to natural selection!

106
Summary

• Local search methods

• keep only a small number of states in memory
• that are useful for optimization

• Nondeterministic environments
• agents can apply AND–OR search to generate contingency plans that
• reach the goal regardless of which outcomes occur during execution.

• Belief-state is the set of possible states that the agent is in for

partially observable environments.

• Standard search algorithms can be applied directly to belief-state

space to solve sensorless problems.

107
See you next day!!!

10
8