Design and Analysis of Algorithms

Backtracking, Branch-and-Bound

Hemantha T D
Assistant Professor

Department of ISE Acharya Institute of Technology

 Introduction
2
• There are problems that are difficult to solve algorithmically.
• Backtracking and Branch-and-Bound—that often make it possible to solve
at least some large instances of difficult combinatorial problems.
• improvement over exhaustive search
• These construct candidate solutions one component at a time and evaluate
the partially constructed solutions: if no potential values of the remaining
components can lead to a solution, the remaining components are not
generated at all.
• Both backtracking and branch-and-bound are based on the construction of
a state-space tree

Department of ISE Acharya Institute of Technology

2
 Introduction cont’d…
3
• terminate a node as soon as it can be guaranteed that no solution to the
problem can be obtained by considering choices that correspond to the
node’s descendants.
• Branch-and-bound is applicable only to optimization problems.
• it is based on computing a bound on possible values of the problem’s
objective function.
• Backtracking applies to non-optimization problems.
• Backtracking constructs tree in DFS order
• Branch-and-bound constructs tree in BFS order

Department of ISE Acharya Institute of Technology

3
 Backtracking
4
• The problems that cannot be solved in polynomial time they can be solved
by exhaustive search technique.
• Backtracking is an intelligent variation of exhaustive search , which
constructs solution one component at a time and evaluates such partially
constructed candidates.
• If a partially constructed solution can be developed further without
violating the problem’s constraints, it is done by taking the first remaining
legitimate option for the next component.
• If there is no legitimate option for the next component, no alternatives for
any remaining component need to be considered.
• the algorithm backtracks to replace the last component of the
partially constructed solution with its next option
Department of ISE Acharya Institute of Technology

4
 Backtracking cont’d…
5
• In state space tree the root represents an initial state before the search for
a solution begins. The nodes of the first level in the tree represent the
choices made for the first component of a solution, the nodes of the second
level represent the choices for the second component, and so on.
• A node is said to be promising if it corresponds to a partially constructed
solution that may still lead to a complete solution
• Otherwise its called nonpromising
• Leaves represent either nonpromising dead ends or complete solutions
found by the algorithm.
• statespace tree for a backtracking algorithm is constructed in the manner
of depth first search(DFS).

Department of ISE Acharya Institute of Technology

5
 n-Queens Problem
6
• n-queens problem asks us to place n queens on an n × n chessboard so that
no two queens attack each other by being in the same row or in the same
column or on the same diagonal.
n=4
n=1 n=2 n=3 1 2 3 4

Q 1  queen 1
2  queen 2
Solution no Solution
3  queen 3
no Solution
4  queen 4

Department of ISE Acharya Institute of Technology

6
N-Queen’s Problem
7
Q Q

x x x x x
Q Q Q
Q Q Q

x x x x x x x
dead node Q
Q
Q
Q
Q
Q

x x x x x x
dead node Q
Q
Q
Department of ISE Acharya Institute of Technology
Q
7 Solution
n-Queens Problem
8

Solution 1 Solution 2

Q Q
Q Q
Q Q
Q Q

Department of ISE Acharya Institute of Technology

8
 Hamiltonian Circuit Problem
9
• Hamiltonian Cycle of a given graph with ‘n’ vertices is a path of n+1
vertices in which except first and last vertex all other vertices are distinct.
• Ex: If graph has n vertices- v1,v2, . . ., vn
Then Hamiltonian cycle at v1 is: v1, v2,v3, . . ., vn-1, vn, v1

Department of ISE Acharya Institute of Technology

9
Hamiltonian Circuit Example
10 a

c f

d e
e
Hamiltonian Cycle is
a, b, f, e, c, d, a
e d f c
dead node dead node
d
f
dead node
Department of ISE Acharya Institute of Technology a
10 Solution
Hamiltonian Circuit Example a
11
b

d e

g
f g
Hamiltonian Cycle is
d a, b, e, g, d, f, c, a
e f
c g
dead node
dead node
f
e c
dead node dead node
c

Department of ISE Acharya Institute of Technology

a Solution
11
 Subset-Sum Problem
• subset-sum problem ask us to find a subset of a given set
A = {a1, . . . , an} of n positive integers whose sum is equal to a given positive
integer d.
Ex: A= {1, 2, 5, 6, 8} and d = 9
{1, 2, 6} and {1, 8}.
• we will assume that a1< a2 < . . . < an.
• The state-space tree can be constructed as a binary tree
• The root of the tree represents the starting point, with no decisions about
the given elements made as yet.
• Its left and right children represent, respectively, inclusion and exclusion
of a1 in a set being sought.
• Similarly, going to the left from a node of the first level corresponds to
inclusion of a2 while going to the right corresponds to its exclusion, and
so on. Department of ISE Acharya Institute of Technology

12
 Subset-Sum Problem cont’d…
• a path from the root to a node on the ith level of the tree indicates which
of the first i numbers have been included in the subsets represented by
that node.
• We record the value of s, the sum of these numbers, in the node. If s is
equal to d, we have a solution to the problem. We can either report this
result and stop or, if all the solutions need to be found, continue by
backtracking to the node’s parent.
• If s is not equal to d, we can terminate the node as nonpromising if either
of the following two inequalities holds:

Department of ISE Acharya Institute of Technology

13
Subset-Sum Problem
S={3, 5, 6, 7} d=15
0
14
3 0

8 3 5 0

14 8 9 3 11 5 6 0
X X X
14+7>15 11+7>15
9+7>15
15 8 10 3
12 5 13 6 7 0
Solution

Solution set is:

{3, 5, 7}
Department of ISE Acharya Institute of Technology
14
 Backtracking-General Remarks
• An output of a backtracking algorithm can be thought of as an n-tuple
(x1, x2, . . . , xn)
• Depending on the problem, all solution tuples can be of the same length
( n-queens and Hamiltonian circuit) and of different lengths (subset-
sum).
• backtracking algorithm generates a state-space tree
• nodes represent partially constructed tuples with the first i coordinates
defined by the earlier actions of the algorithm.
• If such a tuple (x1, x2, . . . , xi) is not a solution, the algorithm finds the next
element in Si+1 that is consistent with the values of (x1, x2, . . . , xi) and the
problem’s constraints, and adds it to the tuple as its (i + 1)st coordinate.
• If such an element does not exist, the algorithm backtracks to consider
the next value of xi, and so on.
Department of ISE Acharya Institute of Technology

15
 Backtracking-General Remarks
ALGORITHM Backtrack(X[1..i])
//Gives a template of a generic backtracking algorithm
//Input: X[1..i] specifies first i promising components of a solution
//Output: All the tuples representing the problem’s solutions

if X[1. . .i] is a solution write X[1. . .i]

else
for each element x ∈ Si+1 consistent with X[1..i] and the constraints do
X[i + 1]←x
Backtrack(X[1. . .i + 1])

Department of ISE Acharya Institute of Technology

16
 Branch-and-Bound
• An optimization problem seeks to minimize or maximize some objective
function usually subject to some constraints
• A feasible solution is a point in the problem’s search space that satisfies
all the problem’s constraints
• An optimal solution is a feasible solution with the best value of the
objective function
• branch-and-bound requires two additional items:
1. a way to provide, for every node of a state-space tree, a bound on
the best value of the objective function on any solution that can be
obtained by adding further components to the partially constructed
solution represented by the node
2. the value of the best solution seen so far

Department of ISE Acharya Institute of Technology

17
 Branch-and-Bound cont’d…
• we can compare a node’s bound value with the value of the best solution
seen so far.
• If the bound value is not better than the value of the best solution seen so
far the node is nonpromising and can be terminated/pruned
• we terminate a search path at the current node in a state-space tree of a
branch-and-bound algorithm for any one of the following three reasons:
1. The value of the node’s bound is not better than the value of the best
solution seen so far.
2. The node represents no feasible solutions because the constraints of the
problem are already violated.
3. The subset of feasible solutions represented by the node consists of a
single point

Department of ISE Acharya Institute of Technology

18
 Assignment Problem
• Assigning ‘n’ people to ‘n’ jobs so that the total cost of the assignment
is as small as possible.
• An instance of the assignment problem is specified by an n × n cost matrix
C

• select one element in each row of the matrix so that no two selected
elements are in the same column and their sum is the smallest possible

• To compute lower bound select smallest elements in each of the matrix’s

rows.
lb= 2+3+1+4=10
Department of ISE Acharya Institute of Technology

19
Assignment Problem cont’d…
start
lb=2+3+1+4=10

a1 a2 a3 a4

lb=9+3+1+4=17 lb=2+3+1+4=10 lb=7+4+5+4=20 lb=8+3+1+6=18
X X X
b1 b3 b4
lb=2+6+1+4=13 lb=2+3+5+4=14 lb=2+7+1+7=17
X X
Optimal Assignment is:
c3, d4 c4, d3 Person a Job2
cost=2+6+1+4=13 cost=2+6+8+9=25
Inferior Solution
Person b Job1
Solution
Cost=2+6+1+4=13 Person c Job3
Department of ISE Acharya Institute of Technology
Person d Job4
20
 Knapsack Problem
• given n items of known weights wi and values vi , i = 1, 2, . . . , n, and a
knapsack of capacity W, find the most valuable subset of the items that fit
in the knapsack.
• order the items of a given instance in descending order by their value-to-
weight ratios.
• v1/w1 ≥ v2/w2 ≥ . . . ≥ vn/wn.
• state-space tree for this problem is constructed as a binary tree
• Each node on the ith level of this tree, 0 ≤ i ≤ n, represents all the subsets
of n items that include a particular selection made from the first i ordered
items.
• a branch going to the left indicates the inclusion of the next item, and a
branch going to the right indicates its exclusion.

Department of ISE Acharya Institute of Technology

21
 Knapsack Problem cont’d…
• We record the total weight w and the total value v of this selection in the
node, along with some upper bound ub on the value of any subset that
can be obtained by adding zero or more items to this selection.
• upper bound ub is calculated as add v, the total value of the items already
selected, the product of the remaining capacity of the knapsack W − w
and the best per unit payoff among the remaining items, which is
vi+1/wi+1
ub = v + (W − w)(vi+1/wi+1)

Department of ISE Acharya Institute of Technology

22
Knapsack Problem Example
Item Weight Value Value/
Item Weight Value
1 7 42 Weight
2 3 12 1 7 42 6
3 4 40 2 3 12 4
4 5 25 3 4 40 10
Capacity W=10 4 5 25 5

Arrange the items in decreasing order of value/weight ratio

Value/
Item Weight Value
Weight
3 4 40 10
1 7 42 6
4 5 25 5
2 3 12 4
Capacity W=10
Department of ISE Acharya Institute of Technology

23
 Knapsack Problem Example cont’d…
w=0, v=0
ub = v + (W − w)(vi+1/wi+1) ub=100 0+10*10

40+6*6
w=4, v=40 w=0, v=0
Valu
ub=76 ub=60 0+10*6 Ite Weig Val
e/W
X m ht ue
eight
w=11 w=4, v=40 3 4 40 10
ub=70 40+6*5 1 7 42 6
X 4 5 25 5
Not feasible 2 3 12 4
w=9, v=65 w=4, v=40 Capacity W=10
65+1*4 ub=69 ub=64 40+6*4
X
Optimal Solution: {item3, item4}
w=12 w=9, v=65
Total weight=4+5=9
value=65 Total Profit= 40+25=65
X Optimal Solution
Not feasible
Department of ISE Acharya Institute of Technology

24
 Traveling Salesman Problem
• We will be able to apply the branch-and-bound technique to instances of
the traveling salesman problem if we come up with a reasonable lower
bound on tour lengths.
• lower bound can be obtained by finding the smallest element in the
intercity distance matrix D and multiplying it by the number of cities n
• For each city i, 1≤ i ≤ n, find the sum si of the distances from city i to the
two nearest cities; compute the sum s of these n numbers, divide the
result by 2, and, if all the distances are integers, round up the result to the
nearest integer:

• For any subset of tours that must include particular edges of a given
graph, we can modify lower bound in above statement accordingly
Department of ISE Acharya Institute of Technology

25
TSP Example a
lb=14

a, b a, c a, d a, e
lb=14 lb=14 lb=16 lb=19
X X X
a, b, c a, b, d a, b, e
lb=16 lb=16 lb=19
X

a, b, c, d, (e,a) a, b, c, e, (d,a) a, b, d, c, (e,a) a, b, d, e, (c,a)

length=24 length=19 length=24 length=16
l=3+ 6+4+3+8 = 24 l=3+6+2+3+5 = 19 l=3+7+4+2+8 = 24 l=3+7+3+2+1=16
Optimal Solution
Optimal Tour: abdeca

Department of ISE Acharya Institute of Technology

26
Graph Colouring Problem
• Let G be a graph and m be a positive integer, it asks us to find whether
nodes of graph G can be colored in such a way that no two adjacent nodes
have same color yet only m colors used

• The problem has two version

1. m-colorability decision problem
2. m-colorability optimization problem
• m is called chromatic number
• minimum number of colors required such that no two adjacent
nodes have same color
Department of ISE Acharya Institute of Technology

27
Graph Colouring Problem cont’d…
• If d is the degree of given graph G, then it can be colored with
(d+1) colors

• A map and its planar graph representation

• A graph is said to be planar iff it can be drawn in a plane such that no two
edges cross each other

Department of ISE Acharya Institute of Technology

28
 Graph Colouring Example n=4, m=3
1 2 {Red, Green, Blue}
0

4 3 1 1 1

2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

Department of ISE Acharya Institute of Technology

29
P, NP and NP-Complete Problems

Polynomial Time Exponential Time

Linear Search n Knapsack problem 2n
Binary Search logn Travelling Salesman 2n
Merge Sort nlogn Problem
Floyds Algorithm n3 Sum of Subsets 2n
Bubble Sort n2 Graph Coloring 2n
Hamiltonian Cycle 2n

• Polynomial-time algorithm is an algorithm whose execution time is either

given by a polynomial on the size of the input, or can be bounded by such
a polynomial. O(nk)
• Tractable problems : problems that can be solved by a polynomial-time
are called tractable problems

Department of ISE Acharya Institute of Technology

30
P, NP and NP-Complete Problems cont’d…
• Deterministic Algorithm- an algorithm which, given a particular input,
will always produce the same output, with the underlying machine
always passing through the same sequence of states.
• Nondeterministic algorithm- an algorithm that, even for the same input,
can exhibit different behaviors on different runs

1. Try to relate these problems

2. Why don’t you write non deterministic algorithms
Department of ISE Acharya Institute of Technology

31
P, NP and NP-Complete Problems cont’d…
• choice(X) : chooses any value randomly from the set X.
• failure( ) : denotes the unsuccessful solution.
• success( ) : Solution is successful and current thread terminates.
• Problem Statement : Search an element x on A[1:n] where n>=1, on successful
search return j if a[j] is equals to x otherwise return 0.
Algorithm Nsearch(A,n,key)
{
j= choice(A, n)
if(A[j]==key) then
{
write(j);
success();
}
write(0);
failure();
}
Department of ISE Acharya Institute of Technology

32
P, NP and NP-Complete Problems cont’d…
• Nondeterministic Polynomial(NP) Algorithms- Problem which can't be
solved in polynomial time. But NP problems are checkable/verifiable in
polynomial time

• To relate these problem we need some base problem

• CNF-Satisfiability problem
• (XvYvZ) ᶺ (XvYvZ) ᶺ (XvYvZ)
• For what values of X Y Z this satisfies

Department of ISE Acharya Institute of Technology

33
P, NP and NP-Complete Problems cont’d…
• Reduction- If we can convert one instance of a problem A into problem B (NP problem) then
it means that A is reducible to B.

• If problem A is NP-Hard then Problem B is also NP-Hard

• Reduction has transitivity property.
• NP-hard-- Now suppose we found that A is reducible to B, then it means that B is at least as
hard as A.
• NP-Complete -- The group of problems which are both in NP and NP-hard are known as NP-
Complete problem. Department of ISE Acharya Institute of Technology

34
P, NP and NP-Complete Problems cont’d…
• Now suppose we have a NP-Complete problem R and it is reducible to Q
then Q is at least as hard as R and since R is an NP-hard problem.
therefore Q will also be at least NP-hard , it may be NP-complete also.

Department of ISE Acharya Institute of Technology

35
36

Department of ISE Acharya Institute of Technology

36