Limitations of Algorithmic Power

Module 5 - Chapter 1

Mrs. Madhu Nagaraj

Assistant Professor
Dept of CSE-Data Science
ATMECE
 DecisionTrees
• Many important algorithms, especially those for sorting and searching, work by
comparing items of their inputs. We can study the performance of such
algorithms with a device called a decision tree.

• As an example, Figure next presents a decision tree of an algorithm for finding a

minimum of three numbers. Each internal node of a binary decision tree
represents a key comparison indicated in the node.

• Each leaf represents a possible outcome of the algorithm’s run on some input of
size n.

2
3
• An important point is that the number of leaves must be at least as
large as the number of possible outcomes.
• The algorithm’s work on a particular input of size n can be traced
by a path from the root to a leaf in its decision tree, and the number
of comparisons made by the algorithm on such a run is equal to the
length of this path.
• Hence, the number of comparisons in the worst case is equal to the
height of the algorithm’s decision tree.

4
Decision Trees for Sorting
• Most sorting algorithms are comparison based, i.e., they work by comparing
elements in a list to be sorted. By studying properties of decision trees for such
• algorithms, we can derive important lower bounds on their time efficiencies.

• Consider, as an example, a three-element list a, b, c of orderable items such as

• real numbers or strings

• In general, the number of possible outcomes for sorting an arbitrary n-element

list is equal to n!
• In other words, about n log2 n comparisons are necessary in the worst case to sort
an arbitrary n-element list by any comparison-based sorting algorithm.
5
6
7
Decision Trees for Searching a Sorted Array
decision trees can be used for establishing lower bounds on the number of key
comparisons in searching a sorted array of n keys:
A[0] < A[1] < ... < A[n − 1]. The principal algorithm for this problem is binary
search.

. Since the minimum height h of a ternary tree with l leaves is [ log3 l]

worst-case comparisons: we get the following lower bound on the number of
Cworst(n) ≥ [log3(2n + 1)]
P, NP, and NP-Complete Problems
Depending on the complexity of various problems of algorithms, the problems are classified
into 2 classes:
Computational
Complexity Problem

P-Class NP-Class

NP-Hard NP-Complete
P, NP, and NP-Complete Problems
We say that an algorithm solves a problem in polynomial time.
if its worst-case time efficiency belongs to O(p(n)) where p(n) is a polynomial of
the problem’s input size n.
Problems that can be solved in polynomial time are called tractable, and
problems that cannot be solved in polynomial time are called intractable.
Deterministic algorithm Given a particular input, will always produce the same
output, with the underlying machine always passing through the same sequence of
states. 
State machine: a state describes what a machine is doing at a particular instant in
time. State machines pass in a discrete manner from one state to another.

 Non-deterministic Algorithms- If it uses external state other than the input, such as
user input, a global variable, a hardware timer value, a random value, or stored disk
data. If it is timing-sensitive, e.g. if it has multiple processors writing to the same
data at the same time.

In measureing the complexity of an algorithm, we use input length as the parameter.

An algorithm A is of polynomial complexity if there exists a polynomial p(), such
that the computing time of A is O(p(n)) for every input of size n.
1. Problems with solution time is bound by a polynomial of a small degree
• Most searching and sorting algorithms
• Also called tractable algorithms
• For example, ordered search (O(lg n)), polynomial evaluation (O(n)),
sorting (O(n log n))

2. Problems with best known algorithms not bound by a polynomial

•Hard, or intractable, problems
•Traveling salesperson (O(n22n)), knapsack (O(2n/2))
•None of the problems in this group has been solved by any polynomial time
algorithm
P (Polynomial) Time Problems- P is the set of all decision problems solvable by
deterministic algorithms in polynomial time.

NP (Polynomial) Time Problems - NP is set of all decision problems solvable by

nondeterministic algorithms in polynomial time.
Since deterministic algorithms are just a special case of nondeterministic ones, we
conclude that P ⊆NP, what we dont know in CS is whether P=NP or P≠NP.
• No one has been able to device an algorithm which is bounded by the
polynomial of small degree for the problems belonging to the second
group.

• The theory of the NP-Completeness does not provide any method of obtaining
polynomial time algorithms for the problems of the second group. “Many
of the problems for which there is no polynomial time algorithm
available are computationally related”.
Two Classes
1. NP-Complete- have the property that it can be solved in polynomial time if all
other NP-Complete problems can be solved in polynomial time.
2.NP-Hard- if it can be solved in polynomial time then all NP-Complete can be solved
in polynomial time.
 Chapter 9: NP H&C Problems

NP-Hard

NP-Complete

8
 Chapter 9: NP H&C Problems

P vs NP problems
–The problems in class P can be solved in O(Nk) time, for
some constant k (polynomial time)
–The problems in class NP can be verified in polynomial time. If we are given
a certificate of a solution, we can verify that the certificate is correct in
polynomial time in the size of input to the problem.
–Some polynomial-time solvable problems look very similar to NP-
complete problems.
• An algorithm on such a non-deterministic machine
consists of two phases:

the first of which consists of a guess about the solution,

which is generated in a non-deterministic way,
while the second consists of a deterministic algorithm that
verifies or rejects the guess as a valid solution to the
problem.
In order to explain the verifier-based definition of NP, let us consider the
subset sum problem:
Assume that we are given some integers, such as {−7, −3,−2, 5, 8}, and we
wish to know whether some of these integers sum up to zero.
The answer is "yes", since the subset of integers {−3, −2, 5} corresponds to
the sum(−3) + (−2) + 5 = 0.
The task of deciding whether such a subset with sum zero exists is called
the subset sum problem.
• A problem is said to be in NP if there exists a verifier V for the problem.
Given any instance I of problem P, where the answer is "yes", there must
exist a certificate (also called a witness) W such that, given the ordered pair
(I,W) as input, V returns the answer "yes" in polynomial time.
• Furthermore, if the answer to I is "no", the verifier will return "no" with
input (I,W) for all possible W.
• Note that V could return the answer "No" even if the answer to I is "yes",
if W is not a valid witness.
 Chapter 9: NP H&C Problems

An algorithm A is of polynomial complexity is there exist a polynomial p( ) such that

the computing time of A is O(p(n)).

Definition: P is a set of all decision problems solvable by a deterministic algorithm

in polynomial time.
NP is the set of all decision problems solvable by
a nondeterministic algorithm in polynomial time.
 P ⊆ NP
The most famous unsolved problem in Computer Science is
?
whether P=NP or P ≠ NP  P =NP

Cook’s theorem: Satisfiability is in P if P = NP

Definition. Let L1 and L2 be problems. L1 reduces to L2(L1L2) iff there is a

way to solve L1 by deterministic polynomial time algorithm that solve L2 in
polynomial time.
 Chapter 9: NP H&C Problems

if we have a polynomial time algorithm for L2 then we can solve

L1 in polynomial time.
Definition. A problem L is NP-Hard if and only if
satisfiability reduces to L. ( satisfiabilityL).

Definition. A problem L is NP-Complete if and only if L is NP-Hard

and L  NP.

Halting problem: An example of NP-Hard decision problem which

is not NP-Complete.
COPING WITH LIMITATIONS OF ALGORITHMIC
POWER
Module 5 - Chapter 2
Mrs. Madhu Nagaraj
Assistant Professor
Dept of CSE-Data Science
ATMECE
Techniques to solve large instance of difficult Combinatorial problems.
1. Backtracking
2. Branch and Bound

Both techniques follow Brute Force Strategy

• Brute Force Approach : Try all possible solution and pick desired one.
• Both are based on the construction of State Space Tree
• Both technique terminate if there is no guarenteed solution by considering descendant
nodes.

Backtracking technique resembles a depth-first – search in a directed graph. The graph

concerned here is usually a tree, the aim of backtracking is to search the state space tree
systematically. The aim of the search is to find solutions to some problems.
State Space Tree
• In state space tree, root represents an initial state before the search for a solution
begins.
• The nodes of the first level in the tree represent the choice made for the first
component of a solution, the nodes of the second level represent the choices for
the second components, and so on.
• A node in a state space tree is said to be promising if it corresponds to a partially
constructed solution that may lead to a complete solution; otherwise a node is said
to be non promising.
 Horizontal Attack:
• Row wise attacking is avoided by placing 1st queen in 1st row,
2nd queen in 2nd row and so on.
• By placing ith queen in ith row, horizontal attacking can be
avoided

ATME College of
Engineer ing
 Vertical Attack:
(i, x[i]) means the position of ith queen in row i and column x[i]
(k, x[k]) means the position of kth queen in row k and column x[k]
If ith & kth queen are in same column then
X[i] == x[k] --------------- (1)
Hence indicate that queens attack vertically

(1,1) & (4,1) x[i] ==

Q1 x[k] to be avoided

Q4 Depar t ment ATME College of

of CSE Engineer ing
 Diagonal Attack:
Top left corner to bottom right corner: The difference between
row value and column value is same.
(1,3) & (2,4) |i - x[i]| = |k
- x[k]|-------(2) to be
avoided

1,1 1,2 1,3 1,4

2,1 2,2 2,3 2,4
3,1 3,2 3,3 3,4
4,1 4,2 4,3 4,4
Depar t ment ATME College of
of CSE Engineer ing
•Top right corner to bottom left corner: The difference between row value and column
value is same.

• (1,3) & (3,1) i + x[i] = k

+ x[k] ------(3) to be avoided
• Using eqn. (2) and (3)
• i–k = x[i] - x[k]
-----------------(4)
• i–k = - x[i] + x[k]
----------------(5)

• |i – k| = |x[i] - x[k]| indicates queens attack diagonally.

• X[i] == x[k] || abs(i –k) = abs(x[i] – x[k]) two queens attack each other and
cannot be placed.
State Space Tree for 4
Queens

Depar t ment ATME College of

of CSE Engineer ing
Two Solutions of 4 Queen Problem

Depar t ment
of CSE
 Sum of subsets

Depar t ment
of CSE
Depar t ment
of CSE
 Sum of Subsets
Find a subset of a given set S= {S1, S2, S3, S4, ------ Sn}
Of n +ve integers whose sum is equal to given +ve integer d subject
to the constrains

1. Implicit: All Xi values should be distinct and should belong to the set S
2. Explicit: optimal solution be ∑𝑘 𝑆𝑖 = d
𝑖=1

Xi of the solution vector is either 1 or 0 depending on weather the weight Wi is

included or not.

Depar t ment
of CSE
 For a node at level i, the left child corresponds to Xi = 1
and the right child to Xi = 0
The bounding function X[X1, X2, X3, -----Xn] = true iff
𝑊𝑖 𝑋𝑖 + ∑𝑛 𝑊𝑖 ≥ 𝑑
∑𝑖
𝑘= 𝑖=𝑘+1
1

X1, X2, -----Xk cannot lead to an promising node if this

condition is not satisfied.
 0, 1, 21

X1=1 X1=0

3, 2, 18 0, 2, 18 State space tree

X2=1 X2=0

8, 3, 13 3, 3, 13

X3=1 X3=0 X3=1 X3=0

14, 4, 7 8, 4, 7 9, 4, 7 3, 4, 7

X4=1
15, 5, 0

Solution
Sum of subsets
Depar t ment
of CSE
Depar t ment ATME Colleg e of Eng ineer ing
of CSE
Branch and Bound
The term Branch means the way in which we search the state space tree and
Bound means assigning bounding function at each node. This bounding
function is used to prevent the expansion of nodes that cannot possibly
lead to an answer node.

Knapsack Problem
Knapsack Problem: Given n items of known weights wi and values vi, i=1, 2, . . . ,
n,and a knapsack of capacity W, find the most valuable subset of the items that fit in
the knapsack. It is convenient to order the items of a given instance in descending
order by their value-to-weight ratios. Then the first item gives the best payoff per
weight unit and the last one gives the worst payoff per weight unit
•First arrange in V/W in decreasing order
•Since it is a maximization problem. The upper bound is calculated
using the function

i =0 , v=0, w=0 v i+1/wi+1 = 10

Ub = 0 + (10) 10 = 100
 w=0. v=0

Ub = 100

With item 1 Without item 1

i =1, w=4. v=40, v i+1/wi+1 = 6 i =1, w=0. v=0, v i+1/wi+1 = 6

Ub = v + (W-w)(v i+1/wi+1 ) Ub = v + (W-w)(v i+1/wi+1 )

= 40 + 6. 6 = 0 + 10 . 6
= 76 = 60

With item 2 With out item 2

i =2, w=7. v=42, v i+1/wi+1 = 5 i =2, w=0+4. v=40+0, v i+1/wi+1 =
5
Ub = v + (W-w)(v i+1/wi+1 )
= 42 + (10 – 11) . 5 Ub = v + (W-w)(v i+1/wi+1 )
= Not Feasible = 40 + 6. 5
= 70
 w=0. v=0

Ub = 100

With item 3 Without item 3

i =3, w=5+4. v=40+25, v i+1/wi+1 = 4 i =3, w=4+0. v=40+0, v i+1/wi+1 = 4

Ub = v + (W-w)(v i+1/wi+1 ) Ub = v + (W-w)(v i+1/wi+1 )

= 65 + 1. 4 = 40 + 6 . 4
= 69 = 64

With item 4 With out item 4

i =4, w=9+3 =12. i =4, w=0+9. v=65+0, v i+1/wi+1 = 1

Ub = v + (W-w)(v i+1/wi+1 ) Ub = v + (W-w)(v i+1/wi+1 )

= Not Feasible = 65 + 1. 1
= 66
 w=0. v=0

Ub = 100

With item 1 Without item 2

i =1, w=4. v=40, v =6 i =1, w=0. v=0, v =6
i+1/wi+1 i+1/wi+1

= 76 = 60

With item 2 With out item 2

i =2, w=7. v=42, v i+1/wi+1 = 5 i =2, w=0+4. v=40+0, v =5
i+1/wi+1

= Not Feasible = 70

With item 3 Without item 3

i =3, w=5+4. v=40+25, v =4 i =3, w=4+0. v=40+0, v =4
i+1/wi+1 i+1/wi+1

= 69 = 64

With item 4 With out item 4

i =4, w=0+9. v=65+0, v =0
i =4, w=9+3 =12. i+1/wi+1

Not Feasible = 65