Algorithms and Their Complexity

with an Introduction to NP Completeness

Abhijat Vichare
[email protected]

For the students of

CCoEW, Pune.

September 2022
 Plan

1. Some interesting problems

2. Background ideas
3. Algorithm usability – the intuition behind “problem hardness”
4. P, NP, NP Completeness – the ideas
 Some interesting problems

1. Sat, the Boolean satisfiability problem

Find values that make a Boolean expression true.
2. Euler, the Königsberg bridges problem
3. TSP, the traveling salesman problem
Rudrat, or Hamilton, cycle problem.
4. 2D Match, or the pair matching problem
Find right pairings of sets of boys and girls given the “likes”
info.
5. 3D Match, or the triplet matching problem
Find right triples of boys, girls and pets given the “likes” info.
6. Shortest and Longest paths in a graph.
 Some interesting problems
Sat, the Boolean satisfiability problem

The problem
Given a Boolean formula in CNF
(x ∨ y ) ∧ (¬y ∨ ¬z) ∧ (z ∨ x)
Find an assignment of values to variables that make it true
OR
Report that none exists.

Note
• 2Sat: Disjunction of two variables, e.g. (x ∨ y )
• 3Sat: Disjunction of three variables, e.g. (x ∨ y ∨ ¬z)
 Some interesting problems
Sat, the Boolean satisfiability problem

Find an assignment for, or report that none exists!

(¬x ∨ y ∨ ¬z) ∧ (x ∨ ¬y ∨ z) ∧ (x ∨ y ∨ z) ∧ (¬x ∨ ¬y )

Solution

Find independent vertices that are not directly connected, else fail.
 Some interesting problems
Euler, the Königsberg bridges problem

The Seven Bridges of Konigsberg

Problem:

Find a path that

will visit each bridge
once.
aka Eulerian tour
 Some interesting problems
TSP, the traveling salesman problem

The Traveling Salesman

Problem:

Find a path that

will visit each city once
within the budget b or
report if no such path
exists.
aka Rudrat cycle OR
Hamilton cycle
 Some interesting problems
2D Match, or the pair matching problem

2D Pairing

Problem: Boy 1 Girl 1

Boy 2 Girl 2
Find a pairing such
that the likes relation- Boy 3 Girl 3
ship is satisfied
Boy 4 Girl 4
 Some interesting problems
3D Match, or the triplet matching problem

3D Pairing

Boy 1 Girl 1

Boy 2 Girl 2

Problem: Boy 3 Girl 3

Boy 4 Girl 4

Find a pairing such

that the likes relation-
ship is satisfied Pet 1

Pet 2

Pet 3
 Some interesting problems
Shortest and Longest paths in a graph.
 Background ideas
What does the Theory of Computation tell us?

Computability ≡ . . .
 Background ideas
What does the Theory of Computation tell us?

Computability ≡ . . .
• . . . Mechanical Evaluation
 Background ideas
What does the Theory of Computation tell us?

Computability ≡ . . .
• . . . Mechanical Evaluation
• Mechanical: A machine – “non-intelligent” – can do it
Machines view: Turing machines
• Evaluation: “Step-by-step” process to reach the result value
the Function view: λ calculus, Partial recursive functions
 Background ideas
What does the Theory of Computation tell us?

Computability ≡ . . .
• . . . Mechanical Evaluation
• Mechanical: A machine – “non-intelligent” – can do it
Machines view: Turing machines
• Evaluation: “Step-by-step” process to reach the result value
the Function view: λ calculus, Partial recursive functions
• . . . Algorithm
 Background ideas
What does the Theory of Computation tell us?

Computability ≡ . . .
• . . . Mechanical Evaluation
• Mechanical: A machine – “non-intelligent” – can do it
Machines view: Turing machines
• Evaluation: “Step-by-step” process to reach the result value
the Function view: λ calculus, Partial recursive functions
• . . . Algorithm

In essence:
Computation theory tells us what is an algorithm, if it exists!
 Background ideas
Some terminology

A problem?
 Background ideas
Some terminology

A problem?

Add two (positive) integers.

 Background ideas
Some terminology

Instance of a problem?
 Background ideas
Some terminology

Instance of a problem?

4109837 + 8781239854
 Background ideas
Some terminology

• Algorithms describe solution process of a problem!

• They are applied to specific instances.
• They must work for all instances.
 Background ideas
Some questions

Can a problem have more than one solution?

 Background ideas
Some questions

Can a problem have more than one solution?

True
 Background ideas
Some questions

More than one algorithm?

 Background ideas
Some questions

More than one algorithm?

True
 Background ideas
Some questions

Solution ≡ Algorithm: True or False?

 Background ideas
Some questions

Solution ≡ Algorithm: True or False?

False
 Background ideas
Some questions

Solution ≡ Algorithm: True or False?

False
Algorithm: Effective procedure to solve (yield a solution) a
class of problems

Solution: A set of one or more values that satisfy an instance

from the class of problems
 Background ideas
Some questions

Solution ≡ Algorithm: True or False?

False
Algorithm: Effective procedure to solve (yield a solution) a
class of problems

Solution: A set of one or more values that satisfy an instance

from the class of problems

If Solution 6≡ Algorithm, then a sensible question is:

How “quickly” does an algorithm yield the solution?
 Background ideas
Comparing two algorithms of the same problem

• “Subjective” versus “Objective” comparison

• Problem: Sorting (positive integers – simple, but illustrative)
Algorithms: Quick sort, Bubble sort
• The Ω, O and Θ notations
(Lower, Upper and Tight bounds)
 Background ideas
Comparing two algorithms of the same problem

“Subjective” versus “Objective” comparison

• Comparing speed of two people
reaching point X from point Y .
• Exact speed = rate of a step × number of steps
• When is exact speed necessary? When not? Why?

Tip for algorithm analysis

Develop skill to identify relevant parameters in a problem.
 Background ideas
Comparing two algorithms of the same problem

Comparing Quick sort & Bubble sort

Some statements/questions?
• What operations are relevant for each?
• N ln N vs. N·(N−1)
2
 Background ideas
Comparing two algorithms of the same problem

The Ω, O and Θ notations

Do we need exact functions?
• N ln N vs. N·(N−1)
2 ? OR
2
• N log N vs. N ?

Tip
Study the shapes of various functions!
 Background ideas
What are Ω, O and Θ?
200
x2+2x+1
x2

150
y = x2 + 2x + 1, x2

100

50

0
0 1 2 3 4 5 6 7 8 9 10
x

Figure: Ω(x 2 )
 Background ideas
What are Ω, O and Θ?
200
2x2
180 x2+2x+1

160

140
y = x2 + 2x + 1, x2

120

100

80

60

40

20

0
0 1 2 3 4 5 6 7 8 9 10
x

Figure: O(x 2 )
 Background ideas
What are Ω, O and Θ?
200
2x2
x2+2x+12
x

150
y = x2 + 2x + 1, x2

100

50

0
0 1 2 3 4 5 6 7 8 9 10
x

Figure: Θ(x 2 )
 What intuitively is “algorithm hardness”?

First: “measure” an algorithm

Count the number of operations,
Express them as a function of the input size, and
Find suitable O/Ω/Θ form.
 What intuitively is “algorithm hardness”?

Second: classify
Use the O/Ω/Θ form to label an algorithm as
• logarthmic (O(logn)),
• polylogarithmic (O(nlogn)),
• linear (O(n)),
• polynomial (O(nk )),
• exponential (O(an ))
• etc.
and
Compare with other algorithms using plots of graphs earlier.
 What intuitively is “algorithm hardness”?

Third: intractable problem

Exponential algorithms are intractable!
 Recap

• Some problems with few ideas about their solution process.

• What is an algorithm?
• Behaviour of some functions
• Intuitively: How “useful” is an algorithm?
Or inversely, how “difficult” is it?
 Two Problem Types
TSP, the traveling salesman problem
The Traveling Salesman – as a search problem
Given n vertices 1, . . . , n and all n(n − 1)/2 distances between
them, and a budget b, find a tour, i.e. a cycle, that passes through
each vertex exactly once with a total cost b or less. Report that no
such tour exists if you cannot find one.
 Two Problem Types
TSP, the traveling salesman problem
The Traveling Salesman – as a search problem
Given n vertices 1, . . . , n and all n(n − 1)/2 distances between
them, and a budget b, find a tour, i.e. a cycle, that passes through
each vertex exactly once with a total cost b or less. Report that no
such tour exists if you cannot find one.

The Traveling Salesman – as an optimization problem

Given n vertices 1, . . . , n and all n(n − 1)/2 distances between
them, find the shortest tour that passes through each vertex
exactly once. Else Report that no such tour exists.
 Two Problem Types
TSP, the traveling salesman problem
The Traveling Salesman – as a search problem
Given n vertices 1, . . . , n and all n(n − 1)/2 distances between
them, and a budget b, find a tour, i.e. a cycle, that passes through
each vertex exactly once with a total cost b or less. Report that no
such tour exists if you cannot find one.

The Traveling Salesman – as an optimization problem

Given n vertices 1, . . . , n and all n(n − 1)/2 distances between
them, find the shortest tour that passes through each vertex
exactly once. Else Report that no such tour exists.

Any difference between the two?

• Not really. One version can be reduced to the other!
• Hence: Any algorithm that solves one can also solve the other!
 Key ideas in Complexity theory

• Polynomial time, P & Nondeterministic Polynomial time, NP

• Verification of a proposed solution to a problem
• Reduction, i.e. conversion, of a problem to another
 Classes P and NP

Class P
• Set of functions computable in polynomial time on a DTM
• Has nice closure properties, i.e.
Combinations of polynomial time algorithms is also polynomial

Class NP
• Set of functions verifiable in polynomial time on a DTM
(They may be computable in polynomial time on a NDTM)
• Not much known, despite efforts
• A subset (proper or improper?) of problems reduce to each
other, and are called NP Complete
 Key ideas in Complexity theory
What makes algorithms P class?

Consider: Kruskal algorithm for MST

Instead of brute force search of the search space . . .
• . . . Kruskal does a smart search, and . . .
• . . . reduces the search effort.
 Key ideas in Complexity theory
What makes algorithms P class?

Consider: Kruskal algorithm for MST

Instead of brute force search of the search space . . .
• . . . Kruskal does a smart search, and . . .
• . . . reduces the search effort.

This is what makes . . .

Kruskal a P class algorithm!
 Key ideas in Complexity theory
What makes algorithms NP class?

Searching the search space . . .

• We do not have some smart ways to search the solutions
effectively!
• We then try to use a Non Deterministic TM to search.
 Key ideas in Complexity theory
What makes algorithms NP class?

Searching the search space . . .

• We do not have some smart ways to search the solutions
effectively!
• We then try to use a Non Deterministic TM to search.

However if we have a solution proposal, then . . .

the proposal can be verified in polynomial time.
 Key ideas in Complexity theory
Verifying a proposed solution

Given a proposed solution S to an instance

• S can be verified in O(nk ), k ∈ N when on a DTM.
• Compare: Euler tour vs. Rudrat path/tour
• Compare: Rudrat path vs All possible paths!
• All possible paths is harder, and ∈ NP-Hard
 Key ideas in Complexity theory

Given a proposed solution S to an instance

• S can be verified in O(nk ), k ∈ N when on a DTM.
• Compare: Euler tour vs. Rudrat path/tour
• Compare: Rudrat path vs All possible paths!
• All possible paths is harder, and ∈ NP-Hard
 Key ideas in Complexity theory
Reduction, i.e. conversion, of a problem to another

Reduction: The idea

Problem A can be reduced to problem B, if there exists an
algorithm F that transforms information in B to information in A,
and there exists an algorithm G that transforms information in A
to information in B.
 Key ideas in Complexity theory
Reduction, i.e. conversion, of a problem to another
Reduction: The idea
Problem A can be reduced to problem B, if there exists an
algorithm F that transforms information in B to information in A,
and there exists an algorithm G that transforms information in A
to information in B.
 Key ideas in Complexity theory
Reduction, i.e. conversion, of a problem to another

Reduction: The idea

Problem A can be reduced to problem B, if there exists an
algorithm F that transforms information in B to information in A,
and there exists an algorithm G that transforms information in A
to information in B.

Questions!
• Does F always exist?
• What is the complexity of F ?
• Is this same as Turing reduction?
 Key ideas in Complexity theory
Reduction: Examples

Reduction: Converting Sat to 3-Sat

A n-Sat expression has clauses like

(a1 ∨ a2 ∨ a3 ∨ a4 ∨ a5 ∨ . . .)
Introduce new variables yi and rewrite as:

(a1 ∨ a2 ∨ y1 ) ∧ (¬y1 ∨ a3 ∨ y2 ) ∧ (¬y2 ∨ a3 ∨ y3 ) . . .)

 Key ideas in Complexity theory
Reduction: Examples

Reduction: Converting 3-Sat to Independent Set

(¬x ∨ y ∨ ¬z) ∧ (x ∨ ¬y ∨ z) ∧ (x ∨ y ∨ z) ∧ (¬x ∨ ¬y )
 Key ideas in Complexity theory
The set of NP-Complete Problems

The idea
(P ∈ NP-Complete) ≡ (P ∈ NP) ∧ (∃Q ∈ NP : P reduces to Q)
 Revisiting our problems

The sets P and NP

P NP
2-Sat 3-Sat
Shortest path Longest path
Euler tour Rudrat (Hamiltonian) cycle
Bipartite matching Tripartite matching
 Revisiting our problems

The sets P and NP

P NP
2-Sat 3-Sat
Shortest path Longest path
Euler tour Rudrat (Hamiltonian) cycle
Bipartite matching Tripartite matching

Note
• ∀A ∈ P: A finds solution in polynomial time.
• ∀A ∈ NP: A verifies solution in polynomial time.
 Key ideas in Complexity theory
What we know and do not know

For problems in class NP, we know that:

• they are O(nk ), k ∈ I+ when on an NDTM.
• their solutions are verified in O(nk ), k ∈ I+ when on a DTM.
• NDTM ≡ DTM – computation wise, and
• Known algorithms are superpolynomial!
 Key ideas in Complexity theory
What we know and do not know

For problems in class NP, we know that:

• they are O(nk ), k ∈ I+ when on an NDTM.
• their solutions are verified in O(nk ), k ∈ I+ when on a DTM.
• NDTM ≡ DTM – computation wise, and
• Known algorithms are superpolynomial!

For problems in class NP, we do not know if:

• Is NDTM ≡ DTM – complexity wise?
• A ∈ NP ⇒ A is intractable
• Given algorithm A, all instances are difficult!
• Do faster, i.e. polynomial or better, algorithms exist?
 P = NP?

P = NP (some experts believe this)

• ∀A ∈ NP Polynomial algorithms will eventually be found.
• Applications like cryptography would need other techniques.
 P = NP?

P = NP (some experts believe this)

• ∀A ∈ NP Polynomial algorithms will eventually be found.
• Applications like cryptography would need other techniques.

P 6= NP (most experts believe this)

• ∃A ∈ NP that would be harder to compute but easy to verify.
• ∃A ∈ NP s.t. A 6∈ P ∧ A 6∈ NP-Complete.
(Graph isomorphism ∈ NP|NP-Complete|NP-Intermediate?)
• Matches real world experience!
(hard to solve, easy to check)
Key ideas in Complexity theory
The Complexity classes Venn Diagram
(from the Wikipedia)
Thank You