Algorithms NP-Completeness and Approximation Algorithms

Professor Chien-Mo James Li Graduate Institute of Electronics Engineering National Taiwan University
Algorithms NTUEE

Outline
NP-completeness, CH34
Polynomial time verification NP-completeness and reducibility NP-completeness proofs NP-complete problems Approximation Algorithms, CH35

Algorithms

NTUEE

Hamiltonian Cycles (HC)

Hamiltonian cycle:
Simple cycle that contains every vertex in undirected graph Hamiltonian-cycle problem: Given a graph G, does it have a Hamiltonian cycle? Yes or no HC problem instance: A particular HC problem with a given graph Example: Fig 34.2 (a) yes, HC exists (b) no

Algorithms

NTUEE

Stephen Cook (1939~, USA)

A renowned American-Canadian computer
scientist and mathematician who made major contributions to the fields of complexity theory Currently a University Professor at the University of Toronto Cook formalized the notion of NPcompleteness in his seminal 1971 paper "The Complexity of Theorem Proving Procedures", where he proved the existence of an NPcomplete problem by showing that the satisfiability problem is NP-complete In 1982, Cook received the Turing award for his contributions to complexity theory.

Algorithms

NTUEE

Complexity Classes P, NP
Complexity class P:
Set of problems that can be solved in polynomial time O(nk) for some constant k Polynomial time: O(n), O(n2), O(n3), O(n lg n) .... Problems in P are considered tractable Not polynomial time: O(2n), O(nn), O(n!) .... considered intractable Closed under addition, multiplication, composition, complement Complexity class NP (Nondeterministic Polynomial): Set of problems can be verified in polynomial time we know P NP, but P = NP ?
Think of a non-deterministic computer as a computer that magically guesses a solution, then has to verify that it is correct. If a solution exists, computer always guesses it. One way to imagine it: a parallel computer that can freely spawn an infinite number of processes.
Algorithms NTUEE

HC NP
Is HC NP?
Given a set of vertices Check if the tour visits every vertex exactly once Check whether each edge along the cycle actually exists Check if the first vertex = last vertex All can be done in O(n) time so HC NP

Algorithms

NTUEE

The P = NP Question
P = NP? is one of biggest open questions in computer science
Most theorists believe P NP Complexity class NP-complete (NPC): The set of problems that can be verified in polynomial time but not sure if it can be solved in polynomial time NPC is at crux of P = NP question If any NPC problem can be solved in polynomial time, then all problems in NP can be solved in polynomial time a US$1 million reward to anyone who has a formal proof !

NP P=NP=NPC
Algorithms

P
NTUEE

NPC 7

not likely

more likely

Outline
NP-completeness, CH34
polynomial time verification NP-completeness and reducibility NP-completeness proofs NP-complete problems Approximation Algorithms, CH35

Algorithms

NTUEE

Decision Problems
Decision problems: find answers to yes/no questions Examples of decision problems:

Minimum Spanning Tree Given a graph G=(V, E) and a bound K, is there a spanning tree with a cost at most K? Traveling salesman Problem (TSP): Given cities, distance between each pair of cities, and a bound B, is there a route that starts and ends at a given city, visits every city exactly once, and has total distance at most B ?

Algorithms

NTUEE

Polynomial-time Reduction
Let L1 and L2 be two decision problems
Suppose algorithm A2 can solve L2. Can we use A2 to solve L1? Polynomial-time reduction from L1 to L2: L1 P L2 Function f reduces input for L1 into an input for L2 reduced input is a yes input for L2 iff the original input is a yes input for L1 f is computable in polynomial time L1 is no more than a polynomial factor harder than L2

yes
Algorithms

yes
NTUEE

no

no

10

Lemma 34.3
(Lemma 34.3) If L1 P L2 then L2 P implies L1 P Fig 34.5

F is a reduction algorithm that computes the reduction function f from L1 to L2 in polynomial time A2 is a polynomial time algorithm that decides L2 A1 decides whether xL1 by using F to transform input x into f(x)

Algorithms

NTUEE

11

FFT
it takes more time to compute L1 than L2 why L1 P L2 ? not L1 P L2 ?

Algorithms

NTUEE

12

Example: HC P TSP (step 1)

Define a reduction function f for HC P TSP Given an arbitrary HC instance G = (V, E) with n vertices

Create a set of n cities labeled with names in V Assign distance between u and v

Set bound B = n f can be computed in O(V2) time.

Algorithms

NTUEE

13

HC P TSP (Step 2)
G has an HC iff reduced instance has a TSP with distance B

If x is a solution to HC , then f(x) must be a solution to TSP Suppose the HC is x = <v1, v2, , vn, v1>. Then, x is also a tour in the transformed TSP instance The distance of the tour h is n = B since there are n consecutive edges in E, and so has distance 1 in f(x) Thus, f(x) is a solution to TSP f(x) has a TSP tour with distance B

Algorithms

NTUEE

14

HC P TSP (step 2, contd)

G has an HC iff the reduced instance has a TSP with distance B

If f(x) is a solution to TSP, then x must be a solution to HC Suppose there is a TSP tour with distance n = B Let it be <v1, v2, , vn, v1> Since distance of the tour n and there are n edges in the TSP tour, the tour contains only edges in E Thus, <v1, v2, , vn, v1> is a Hamiltonian cycle

Algorithms

NTUEE

15

NP-Completeness
A decision problem L is NPC if both conditions are true
1. 2.

L NP L' P L for every L' NP

A problem is called NP-hard if only condition 2 is true

*not in exam

(Theorem 34.4) Suppose L NPC

If L P, then there exists a polynomial-time algorithm for every L' NP (i.e., P = NP) If L P, then there exists no polynomial-time algorithm for any L' NPC (i.e., P NP) NPC is at crux of deciding whether P = NP

?? tractable NP-complete problems intractable

Algorithms

NTUEE

16

Combinational Circuit

Inverter x x x AND xy xy OR xy x+y XOR xy xy+xy XNOR x y xy xy+xy if and only if implication xy x+xy

Algorithms

NTUEE

17

Circuit-SAT
Circuit-Satisfiability Problem (Circuit-SAT):
Given a combinational circuit C of AND, OR, and NOT gates Is there an assignment of Boolean values to the inputs that makes output of C = 1? A circuit is satisfiable if there exists a set of Boolean input values that makes output of the circuit =1 Circuit (a) is satisfiable since x1, x2, x3= 1, 1, 0 makes output =1 how about (b)?

=1?

Algorithms

NTUEE

18

Circuit-SAT is NPC
To prove Circuit-SAT is NPC, we need two conditions
1. 2.

L NP L' P L for every L' NP

Condition 1: (Lemma 34.5) Circuit-SAT belongs to NP

given inputs, takes only polynomial time to verify if output is 1 Condition 2: (Lemma 34.6) Circuit-SAT is NP-hard Informal proof see textbook Based on two lemmas, Circuit Satisfiability is NP-complete our first NPC problem will prove other NPC problems based on this one

Algorithms

NTUEE

19

Outline
NP-completeness, CH34
polynomial time verification NP-completeness and reducibility NP-completeness proofs NP-complete problems Approximation Algorithms, CH35

Algorithms

NTUEE

20

How to Prove NP-Completeness?

Five steps for proving that L is NP-complete:
1. 2. 3.

4. 5.

Prove L NP Select a known NP-complete problem L' Construct a reduction function f that maps every instance of L' to an instance of L Prove that x L' iff f(x) L for all input x Prove that f is done in polynomial time

So far, thousands of problems has been proven NPC

http://en.wikipedia.org/wiki/List_of_NP-complete_problems

Algorithms

NTUEE

21

Formula Satisfiability (SAT)

A Boolean formula composed of
n Boolean variables x1, x2, x3, ... xn m Boolean connectives: AND, OR, NOT, implication, if and only if parentheses A satisfying assignment of a set of values for variables that evaluates = 1 Example Boolean formula = (( x1 x2 ) (( x1 x3 ) x4 )) x2 satisfying assignment x1=0, x2=0, x3=1, x4=1

= ((0 0) ((0 1) 1)) 0

= (1 (1 1)) 1 = (1 0) 1 =1
Algorithms NTUEE

22

Prove SAT NPC (1)

step1: prove SAT NP
Given a satisfying assignment, a Boolean formula can be evaluated in polynomial time step 2: prove SAT NP-hard Show that Circuit-SAT P SAT Given circuit C , reduction function f maps C to a formula AND of output with the conjunction of clauses each clause describe the operation of a gate in C f can be done in polynomial time = x10 ( x4 x3 ) ( x5 ( x1 x2 ))

( x6 x4 ) ( x7 ( x1 x2 x4 )) ( x8 ( x5 x6 )) ( x9 ( x6 x7 )) ( x10 ( x7 x8 x9 )) 144424443
clause

Algorithms

NTUEE

23

Prove SAT NPC (2)

step 2: (contd) :
prove if x satisfies C, then f(x) satisfies x satisfies C, so every clause =1 in the whole formula = 1 prove if f(x) satisfies , then x satisfies C f(x) satisfies , so every clause =1 in circuit output=1, so x satisfies circuit C Q.E.D. = x10 ( x4 x3 )

( x5 ( x1 x2 )) ( x6 x4 ) ( x7 ( x1 x2 x4 )) ( x8 ( x5 x6 )) ( x9 ( x6 x7 )) ( x10 ( x7 x8 x9 )) 144424443
clause

Algorithms

NTUEE

24

3-CNF Satisfiability
Conjunctive normal form (CNF) if a Boolean formula is expressed as
AND of clauses 3-CNF means each clauses has exactly three distinct literals literal is an occurrence of a variable or its negation Example: 3-CNF, 4 clauses, 12 literals

( x1 x1 x2 ) ( x3 x2 { x4 ) ( x1 x3 x4 ) ( x1 x3 x5 ) { 14 4 244 3
literal literal clause

Exercise: Prove 3CNF-SAT is NPC

Algorithms

NTUEE

25

Outline
NP-completeness, CH34
polynomial time verification NP-completeness and reducibility NP-completeness proofs NP-complete problems Approximation Algorithms, CH35

Algorithms

NTUEE

26

Well-known NPC Problems

Fig 34.13: all proof ultimately follow by reduction from Circuit-SAT Circuit -SAT Formula SAT 3-CNF SAT Clique (decision) Vertex Cover Ham. Circuit
Algorithms

Subset Sum

Set Cover *not in textbook

NTUEE

TSP (decision)

27

Max. Clique Problem

Clique is a subset of vertices such that
each pair of vertices in a clique are connected clique is a complete subgraph Maximum clique = find clique of maximum size in the graph size of clique = number of vertices in clique Maximum clique problem (optimization version) Given an undirected graph G= (V,E) find maximum clique Clique problem (decision version) Given an undirected graph G= (V,E), and size k Does G has a clique of size k ? Example: maximum clique is {v0, v1, v2, v4} size= 4

Algorithms

NTUEE

28

Clique NPC (1)

step 1: prove CLIQUE NP

given a set V of k vertices for each pair of vertices u,v V, check if there is an edge (u,v) can be done in polynomial time

Algorithms

NTUEE

29

Clique NPC (2)

step 2: prove CLIQUENP-hard by reduction 3CNF-SAT P CLIQUE

given a 3CNF of k clauses: C1 C2 ... Ck construct a graph G such that each clause Cr is represented by a triple of 3 vertices vr1 vr2 vr3 an edge between vri and vsj of difference clauses (r s) their literals are consistent e.g. x and x are not consistent 3 3

Fig. 34.14
= ( x1 x2 x3 )
(x1 x2 x3 ) ( x1 x2 x3 )

f f

No edge

Algorithms

NTUEE

30

Clique NPC (3)

step 2: (contd) 3CNF-SAT P CLIQUE

x satisfy f(x) is a clique of size k x satisfy so at least on literal =1 on each side correspond to a clique of size k in G example: x2=0, x3=1 f(x) is a clique of size k x satisfy f(x) is a clique so it contains exactly one vertex in every triple assign 1 to each literal in the clique without conflict thus, x satisfies

= ( x1 x2 x3 )
(x1 x2 x3 ) ( x1 x2 x3 )

Algorithms

NTUEE

31

Vertex-cover Problem
A vertex cover of undirected graph G=(V,E) is a subset V V
if (u,v) E, then u V or v V or both that means, every edges is covered by at least one vertex Cover size = number of vertices in V Minimum vertex-cover problem: find a vertex cover of minimum size Exercises: prove minimum vertex cover is NPC show that CLIQUE P VERTEX-COVER example: Fig 34.15 clique V = {u,v,y,x} Vertex cover of G =V-V={w, z}

f G
Algorithms NTUEE

G
32

Subset-sum Problem
Given a finite set of positive integers S, and a target integer t Find a subset S of S whose elements sum to t example:
S={1, 2, 7, 14, 49, 98, 343, 686, 2409, 16808, 17206, 117705, 117993} t = 138457 S={1, 2, 7, 98, 343, 686, 2409, 17206, 117705} Exercise: Prove Subset-Sum is NPC Hint: 3CNF-SAT P SUBSET-SUM

Algorithms

NTUEE

33

Outline
NP-completeness, CH34 Approximation Algorithms, CH35

Approximate Vertex-cover 35.1 Approximate TSP 35.2 Approximate Set-Covering 35.3

Algorithms

NTUEE

34

Decision Problems vs. Optimization Problems

Decision problems: find answers to yes/no questions
MST: Given a graph G=(V, E) and a bound K, is there a spanning tree with a cost at most K? TSP: Given cities, distance between each pair of cities, and a bound B, is there a route that starts and ends at a given city, visits every city exactly once, and has total distance at most B ? Optimization problems: find a legal answer to minimize cost MST: Given a graph G=(V, E), find the cost of a minimum spanning tree of G TSP: Given a set of cities and that distance between each pair of cities, find the distance of a minimum route starts and ends at a given city and visits every city exactly once

Algorithms

NTUEE

35

How to Solve Opt Problems?

Could apply binary search on decision problems to obtain solutions
to optimization problems Assume a cost C, decide if solution exist If no, increase cost; if yes, reduce cost Repeat until we get satisfactory answers Many optimization version of NPC problems are NP-hard No known polynomial time algorithm But we can still find some alternative solutions

Algorithms

NTUEE

36

Approximation Algorithms
In practice, near optimal solutions are good enough
approximation algorithms returns near-optimal solutions often in polynomial time Ratio bound, (n) 1 C* is optimal solution; C is our solution C C* max C* , C (n) if (n) = constant, we call it (n)-approximation algorithm Relative error bound, (n) (n)-1 | C C* | (n) C*

if (n) = constant, we call it (1+ ) approximation algorithm

Algorithms

NTUEE

37

Approx. Alg. for Vertex Cover

Complexity O(V+E) Example: Fig 35.1
pick edge (b,c) pick edge (e,f) pick edge (d,g) Size = 6 optimal solution = {b,d,e} Size = 3 C= E= G.E while E let (u,v) be an arbitrary edge of E C = C{u,v} remove from E every edge incident on either u or v return C
NTUEE

APPROX-VERTEX-COVER(G)

1 2 4 5 6 7 8

Algorithms

38

Ratio Bound = 2
(Theorem 35.1) APPROX-VERTEX-COVER is 2-approximation
algorithm Proof: Let A = set of edges picked ; C = set of vertices picked |C| = 2|A| Let C* is optimal cover no edges in A share an endpoint, so no two edges in A are covered by the same vertex from C* |A||C*| so |C|2|C*|

Algorithms

NTUEE

39

Outline
NP-completeness, CH34 Approximation Algorithms, CH35

Approximate Vertex-cover 35.1 Approximate TSP 35.2 Approximate Set-Covering 35.3

Algorithms

NTUEE

40

Traveling Salesman Problem

TSP: Given an undirected graph G=(V,E) that has non-negative
integer cost c(u,v) associated with each edge, find a Hamiltonian cycle with minimum cost Let A = set of edges chosen
c( A) =
( u , v ) A

c(u, v)

triangle inequality

c(u , w) c(u , v) + c(v, w)

Algorithms

NTUEE

41

Approx-TSP Tour with Triangle Inequality

G.V

MST root= a preorder walk

abchdefg approx. solution cost = 19 optimal solution cost = 14.7 H
Algorithms NTUEE

H*

42

Ratio Bound = 2
(Theorem 35.2) APPROX-TSP is a polynomial time 2-approximation
algorithm for TSP with triangle inequality Proof: Let T = MST Let H* be the optimal tour. W H* is formed by some tree plus an edge c(T) c(H*) Let W = a full walk along T e.g. a, b, c, b, h, b, a, d, e, f, e, g, e, d, a since W traverse every edge of T twice H c(W) 2 c(T) H = removed from W all but first visit to each vertex e.g. a, b, c, h, d, e, f, g triangle inequality: removing vertex does not increase cost c(H) c(W) so, c(H) 2 c(H*)
Algorithms NTUEE

43

General TSP without Triangle Inequality (* not in exam)

(Theorem 35.3) If PNP, there is no polynomial time -approximation
algorithm for the general TSP problem (1) Proof: by contradiction Suppose there is a polynomial time -approximation for TSP Suppose we want to find an HC in graph G =(V,E) Let G(V,E) be a complete graph, where E contains all edges if (u, v ) E 1 c (u, v ) = V + 1 otherwise apply the -approximation algorithm for TSP, we are guaranteed to find the HC in polynomial time because the other edges are too costly to choose ( V + 1) + ( V - 1) = V + V > V
then we get a polynomial time HC algorithm!

this is wrong since we already proved HC is NPC

Algorithms

NTUEE

44

Outline
NP-completeness, CH34 Approximation Algorithms, CH35

Approximate Vertex-cover 35.1 Approximate TSP 35.2 Approximate Set-Covering 35.3

Algorithms

NTUEE

45

Set-Covering Problem
Given a finite set X, and a family F of subsets of X subset S F covers its elements
X=U S
SF

Set covering problem: find a minimum number of subset C F that

cover all X
X=U S
SC

Optimization problem
minimum size of C Decision problem exist a covering C size k? exercise: prove set covering is NPC (Hint: use vertex covering) Applications of Set-Covering find minimum of people (C) that perform all tasks (X) minimum 2-level combinational circuit design
NTUEE

Algorithms

46

Greedy-Set-Cover
Each iteration chooses the subset that covers most elements

running time O(|X||F| min(|X|,|F|))

example: |X|=12, |F|=6

greedy solution {S1, S4, S5, S6} size =4 optimal solution C={S3, S4, S5} size=3 (Corollary 35.5) GREEDY-SET-COVER is a polynomial time (ln|X|+1)approximation algorithm
NTUEE

Algorithms

47

SOP Minimization * not in exam

Two-level AND-OR Combinational Circuit Sum of product (SOP)
f = acd + bcd + bcd + acd Disjunctive normal form (DNF) f=(ac d)(bc d)(bc d)(ac d) Very important EDA problem: Given f, find minimize number of gates if gates are the same, minimize number of inputs

Algorithms

NTUEE

48

Quine-McCluskey Algorithm
Invented in 1956, one of the earliest EDA algorithms Two-step process

1. Generate all Prime Implicants (PI) PI = product term that cannot be combined with other terms 2. Select minimum number of PI
set-covering

Algorithms

NTUEE

49

Prime Implicant Generation (1/5)

f(abcd)=m(4,5,6,8,9,10,13)+d(0,7,15)

at least ONE of the following terms must be covered abcd + abcd + abcd + abcd + abcd + abcd + abcd called On-set abcd+abcd+abcd may or may not be covered ab a called Dont-care set cd 00 01 11 10 None of the others can be covered called Off-set 00 X 1 0 1
01

0 0 0

1 X 1 b

1 X 0

1 d 0 1

11

c
10

Algorithms

NTUEE

50

Prime Implicant Generation (2/5)

zero 1 one 1

two 1

three 1 four 1
Algorithms NTUEE

51

Prime Implicant Generation (3/5)

Algorithms NTUEE

- are dont cares

52

Prime Implicant Generation (4/5)

Algorithms

NTUEE

7 Prime Implicants: 0-00 = a'c'd' 100- = ab'c' 1-01 = ac'd -1-1 = bd -000 = b'c'd' 10-0 = ab'd' 01- - = a'b

53

Karnaugh Map
PI on Karnaugh map: circles that can not be grow larger

see logic design textbook

Algorithms

NTUEE

54

PI Table

Algorithms

NTUEE

55

Essential Rows and Distinguished Column

essential rows

distinguished columns 56

Algorithms

NTUEE

Select Essential Rows

Algorithms

NTUEE

57

Final Result

Algorithms

NTUEE

58

FFT
Can Quine-McCluskey find optimal solution for all cases?
obviously not ... set-covering is NP-hard Give an example where PI table has neither essential rows nor distinguished columns Quine-McCluskey has trouble finding optimal solution

Algorithms

NTUEE

59

Coping with NPC, NP-hard problems

Approximation algorithms:
Guarantee to be a fixed percentage away from the optimum. Exponential algorithms/Branch and Bound/Exhaustive search: Feasible only when the problem size is small. Local search: Simulated annealing (hill climbing), genetic algorithms, etc. Randomized algorithms: Make use of a randomizer (random # generator) for operation. Pseudo-polynomial time algorithms: E.g., dynamic programming for the 0-1 Knapsack problem. Probabilistic algorithms: Assume some probabilistic distribution of the instances. Restriction: Work on some special cases of the original problem. E.g., the maximum independent set problem in circle graphs. Heuristics: No formal guarantee of performance.
NTUEE

Algorithms

60

Reading
CH 34 CH35 REF: Roth, Fundamentals of Logic Design Cengage Learning, CH6

Algorithms

NTUEE

61