Chapter 4 (3rd Edition)

Local Search and Optimization

Outline

• Local search techniques and optimization

– Hill-climbing
– Gradient methods
– Simulated annealing
– Genetic algorithms
– Issues with local search

Local Search 2
Local search and optimization

• Previously: systematic exploration of search space.

– Path to goal is solution to problem

• YET, for some problems path is irrelevant.

– E.g 8-queens

• Different algorithms can be used

– Local search

Local Search 3
Local search and optimization

• Local search
– Keep track of single current state
– Move only to neighboring states
– Ignore paths

• Advantages:
– Use very little memory
– Can often find reasonable solutions in large or infinite (continuous)
state spaces.

• “Pure optimization” problems

– All states have an objective function
– Goal is to find state with max (or min) objective value
– Does not quite fit into path-cost/goal-state formulation
– Local search can do quite well on these problems.

Local Search 4
“Landscape” of search

Local Search 5
Hill-climbing search

function HILL-CLIMBING( problem) return a state that is a local maximum

input: problem, a problem

local variables: current, a node.
neighbor, a node.

current  MAKE-NODE(INITIAL-STATE[problem])
loop do
neighbor  a highest valued successor of current
if VALUE [neighbor] ≤ VALUE[current] then return STATE[current]
current  neighbor

Local Search 6
Hill-climbing search

• “a loop that continuously moves in the direction of increasing

value”
– terminates when a peak is reached
– Aka greedy local search

• Value can be either

– Objective function value
– Heuristic function value (minimized)

• Hill climbing does not look ahead of the immediate neighbors of the
current state.

• Can randomly choose among the set of best successors, if multiple

have the best value

• Characterized as “trying to find the top of Mount Everest while in a

thick fog”

Local Search 7
Hill climbing and local maxima

• When local maxima exist, hill climbing is suboptimal

• Simple (often effective) solution

– Multiple random restarts

Local Search 8
Hill-climbing example

• 8-queens problem, complete-state formulation

– All 8 queens on the board in some configuration

• Successor function:
– move a single queen to another square in the same column.

• Example of a heuristic function h(n):

– the number of pairs of queens that are attacking each other
(directly or indirectly)
– (so we want to minimize this)

Local Search 9
Hill-climbing example

Current state: h=17

Shown is the h-value for each possible successor in each column

Local Search 10
A local minimum for 8-queens

A local minimum in the 8-queens state space (h=1)

Local Search 11
Other drawbacks

• Ridge = sequence of local maxima difficult for greedy algorithms to

navigate

• Plateau = an area of the state space where the evaluation function is

flat.

Local Search 12
Performance of hill-climbing on 8-queens

• Randomly generated 8-queens starting states…

• 14% the time it solves the problem

• 86% of the time it get stuck at a local minimum

• However…
– Takes only 4 steps on average when it succeeds
– And 3 on average when it gets stuck
– (for a state space with ~17 million states)

Local Search 13
Possible solution…sideways moves

• If no downhill (uphill) moves, allow sideways moves in hope

that algorithm can escape
– Need to place a limit on the possible number of sideways moves to
avoid infinite loops

• For 8-queens
– Now allow sideways moves with a limit of 100
– Raises percentage of problem instances solved from 14 to 94%

– However….
• 21 steps for every successful solution
• 64 for each failure

Local Search 14
Hill-climbing variations

• Stochastic hill-climbing
– Random selection among the uphill moves.
– The selection probability can vary with the steepness of the
uphill move.

• First-choice hill-climbing
– stochastic hill climbing by generating successors randomly
until a better one is found
– Useful when there are a very large number of successors

• Random-restart hill-climbing
– Tries to avoid getting stuck in local maxima .

Local Search 15
Hill-climbing with random restarts

• Different variations
– For each restart: run until termination vs. run for a fixed time
– Run a fixed number of restarts or run indefinitely

• Analysis
– Say each search has probability p of success
• E.g., for 8-queens, p = 0.14 with no sideways moves

– Expected number of restarts?

– Expected number of steps taken?

Local Search 16
Expected number of restarts

• Probability of Success = p
• Number of restarts = 1 / p
• This means 1 successful iteration after (1/p – 1) failed
iterations
• Let avg. number of steps in a failure iteration = f
and avg. number of steps in a successful iteration = s

Therefore, expected number of steps in random-restart hill

climbing = 1 * s + (1/p – 1) f

So for 8-queens, p = 14%, s = 4, f = 3,

Expected no of moves = 1 * 4 + (1/0.14 -1) * 3 = 22

With sideways moves, p = 94%, s = 21, f = 64

Expected no of moves = 1 * 21 + (1/0.94 -1) * 64 = 25

Local Search 17
Local beam search

• Keep track of k states instead of one

– Initially: k randomly selected states
– Next: determine all successors of k states
– If any of successors is goal  finished
– Else select k best from successors and repeat.

• Major difference with random-restart search

– Information is shared among k search threads.

• Can suffer from lack of diversity.

– Stochastic beam search
• choose k successors proportional to state quality.

Local Search 18
 Gradient Descent

Assume we have some cost-function: C (x1,..., xn )

and we want minimize over continuous variables X1,X2,..,Xn


1. Compute the gradient : C (x1,..., xn ) i
xi
2. Take a small step downhill in the direction of the gradient:

xi  x 'i xi   C (x1,..., xn ) i
xi
3. Check if C (x1,.., x 'i ,.., xn )  C (x1,.., xi ,.., xn )

4. If true then accept move, if not reject.

5. Repeat.

Local Search 19
Learning as optimization

• Many machine learning problems can be cast as optimization

• Example:
– Training data D = {(x1,c1),………(xn, cn)}
where xi = feature or attribute vector
and ci = class label (say binary-valued)

– We have a model (a function or classifier) that maps from x to c

e.g., sign( w. x’ )  {-1, +1}

– We can measure the error E(w) for any setting of the weights w,
and given a training data set D

– Optimization problem: find the weight vector that minimizes E(w)

Local Search 20
Learning a minimum error decision boundary

Minimum Error
7 Decision Boundary

5
FEATURE 2

0
0 1 2 3 4 5 6 7 8
FEATURE 1

Local Search 21
Search using Simulated Annealing
• Simulated Annealing = hill-climbing with non-deterministic search

• Basic ideas:
– like hill-climbing identify the quality of the local improvements
– instead of picking the best move, pick one randomly
– say the change in objective function is 
– if  is positive, then move to that state
– otherwise:
• move to this state with probability proportional to 
• thus: worse moves (very large negative ) are executed less
often
– however, there is always a chance of escaping from local maxima
– over time, make it less likely to accept locally bad moves

Local Search 22
Physical Interpretation of Simulated Annealing

• A Physical Analogy:
• imagine letting a ball roll downhill on the function surface
– this is like hill-climbing (for minimization)
• now imagine shaking the surface, while the ball rolls, gradually
reducing the amount of shaking
– this is like simulated annealing

• Annealing = physical process of cooling a liquid or metal until

particles achieve a certain frozen crystal state
• simulated annealing:
– free variables are like particles
– seek “low energy” (high quality) configuration
– get this by slowly reducing temperature T, which particles
move around randomly

Local Search 23
Simulated annealing

function SIMULATED-ANNEALING( problem, schedule) return a solution state

input: problem, a problem
schedule, a mapping from time to temperature
local variables: current, a node.
next, a node.
T, a “temperature” controlling the probability of downward
steps

current  MAKE-NODE(INITIAL-STATE[problem])
for t  1 to ∞ do
T  schedule[t]
if T = 0 then return current
next  a randomly selected successor of current
∆E  VALUE[next] - VALUE[current]
if ∆E > 0 then current  next
else current  next only with probability e∆E /T

Local Search 24
More Details on Simulated Annealing

– Lets say there are 3 moves available, with changes in the objective
function of d1 = -0.1, d2 = 0.5, d3 = -5. (Let T = 1).
– pick a move randomly:
• if d2 is picked, move there.
• if d1 or d3 are picked, probability of move = exp(d/T)
• move 1: prob1 = exp(-0.1) = 0.9,
– i.e., 90% of the time we will accept this move
• move 3: prob3 = exp(-5) = 0.05
– i.e., 5% of the time we will accept this move

– T = “temperature” parameter
• high T => probability of “locally bad” move is higher
• low T => probability of “locally bad” move is lower
• typically, T is decreased as the algorithm runs longer
– i.e., there is a “temperature schedule”

Local Search 25
Simulated Annealing in Practice

– method proposed in 1983 by IBM researchers for solving VLSI

layout problems (Kirkpatrick et al, Science, 220:671-680, 1983).
• theoretically will always find the global optimum (the best
solution)

– useful for some problems, but can be very slow

– slowness comes about because T must be decreased very
gradually to retain optimality
• In practice how do we decide the rate at which to decrease T?
(this is a practical problem with this method)

Local Search 26
Genetic algorithms
• Different approach to other search algorithms
– A successor state is generated by combining two parent states

• A state is represented as a string over a finite alphabet (e.g. binary)

– 8-queens
• State = position of 8 queens each in a column
=> 8 x log(8) bits = 24 bits (for binary representation)

• Start with k randomly generated states (population)

•
– Higher values for better states.

• Produce the next generation of states by “simulated evolution”

– Random selection
– Crossover
– Random mutation

– Opposite to heuristic function, e.g.,

# non-attacking pairs in 8-queens
• Evaluation function (fitness function).
–
Local Search 27
Genetic algorithms

• Fitness function: number of non-attacking pairs of queens (min = 0, max = 8 ×

7/2 = 28)
• 24/(24+23+20+11) = 31%
• 23/(24+23+20+11) = 29% etc

•
•
•states for
4 2 pairs of 2 states randomly New states Random
8-queens selected based on fitness. after crossover mutation
problem Random crossover points applied
selected

Local Search 28
Genetic algorithms

Has the effect of “jumping” to a completely different new

part of the search space (quite non-local)

Local Search 29
Genetic algorithm pseudocode

function GENETIC_ALGORITHM( population, FITNESS-FN) return an individual

input: population, a set of individuals
FITNESS-FN, a function which determines the quality of the individual
repeat
new_population  empty set
loop for i from 1 to SIZE(population) do
x  RANDOM_SELECTION(population, FITNESS_FN)
y  RANDOM_SELECTION(population, FITNESS_FN)
child  REPRODUCE(x,y)
if (small random probability) then child  MUTATE(child )
add child to new_population
population  new_population
until some individual is fit enough or enough time has elapsed
return the best individual

Local Search 30
Comments on genetic algorithms

• Positive points
– Random exploration can find solutions that local search can’t
• (via crossover primarily)
– Appealing connection to human evolution
• E.g., see related area of genetic programming

• Negative points
– Large number of “tunable” parameters
• Difficult to replicate performance from one problem to another

– Lack of good empirical studies comparing to simpler methods

– Useful on some (small?) set of problems but no convincing

evidence that GAs are better than hill-climbing w/random restarts
in general

Local Search 31
Vacuum-cleaner world

• Percepts: location and state of the environment, e.g.,

[A,Dirty], [A,Clean], [B,Dirty]

• Actions: Left, Right, Clean, NoOp

Local Search 32
Searching with non-deterministic action

Local Search 33
Erratic Vaccum world

In the erratic vacuum world, the Clean action

works as follows:
•When applied to a dirty square the action cleans the square
and sometimes cleans up dirt in an adjacent square, too.

•When applied to a clean square the action sometimes

deposits dirt on the carpet.

•For example, in the erratic vacuum world, the Clean action

in state 1 leads to a state in the set {5, 7}—the dirt in the
right-hand square may or may not be vacuumed up.

Local Search 34
Contingency Plan

• Hence, the solution to a problem is not a

sequence but a contingency plan (also known
as a strategy)
• that specifies what to do depending on what
percepts are received.

• if we start in state 1, there is no single

sequence of actions that solves the problem.
Instead, we need a contingency plan such as
the following:

• Clean, if State = 5 then [Right, Clean]

Local Search 35
Contingency Plan

• Solutions for nondeterministic problems can

contain nested if–then–else statements;
this means that they are trees rather than
sequences.

• Many problems in the real, physical world are

contingency problems because exact
prediction is impossible.

• For this reason, many people keep their eyes

open while walking around or driving

Local Search 36
AND-OR Search tree

Local Search 37
Searching with no Observation

• Sensorless problem: agent’s percepts provide

no information at all
• Sensorless agent has no hope of solving a
problem if it has no idea what state it is in

• Its initial state could be any element of the set

{1, 2, 3, 4, 5, 6, 7, 8}
• To solve sensorless problems, we search in the
space of belief states rather than physical
states.

Local Search 38
Searching with no Observation

• Its initial state could be any element of the set

{1, 2, 3, 4, 5, 6, 7, 8}
• Right => {2,4,6,8}
• Right, Suck => {4, 8}
• [Right,Suck,Left,Suck] is guaranteed to move it
to state 7

Local Search 39
Summary

• Local search techniques and optimization

– Hill-climbing
– Gradient methods
– Simulated annealing
– Genetic algorithms
– Searching with non-deterministic action
– Searching with no observation

Local Search 40