Greedy Algorithms

Greed is good.
(Some of the time)
Outline
 Elements of greedy algorithm
 Greedy choice property
 Optimal substructures
 Minimum spanning tree
 Kruskal’s algorithm
 Prim’s algorithm
 Huffman code
 Activity selection

Ravirajsinh Chauhan Greedy Algorithms 2

 Optimization Problem
 Optimization problem
 A problem in which the optimization function is to be
optimized (usually minimized or maximized) subject to
some constraints

 A feasible solution
 a solution that satisfies the constraints
 An optimal solution
 a feasible solution for which the optimization function has
the best possible value
 In general, finding an optimal solution is computationally
hard

Ravirajsinh Chauhan Greedy Algorithms 3

 The Greedy Method

 Solve a problem by making a sequence of

decisions

 Decisions are made one by one in some order

 Each decision is made using a greedy criterion

 At each stage we make a decision that appears to be the
best at the time

 A decision, once made, is (usually) not changed

later
Ravirajsinh Chauhan Greedy Algorithms 4
Introduction
 Concepts
 Choosing the best possible choice at each step.
 This decision leads to the best over all solution.

 Greedy algorithms do not always yield

optimal solutions.

Ravirajsinh Chauhan Greedy Algorithms 5

 Elements of
Greedy Algorithms
Greedy-choice property
 A globally optimal solution is derived from a
locally optimal (greedy) choice.

 When choices are considered, the choice that

looks best in the current problem is chosen,
without considering results from
subproblems.

Ravirajsinh Chauhan Greedy Algorithms 7

Optimal substructures
 A problem has optimal substructure if an
optimal solution to the problem is composed
of optimal solutions to subproblems.

 This property is important for both greedy

algorithms and dynamic programming.

Ravirajsinh Chauhan Greedy Algorithms 8

Greedy Algorithm v.s. Dynamic Programming
Greedy algorithm Dynamic programming
 The best choice is made at  A choice is made at each
each step and after that the step.
subproblem is solved.
 The choice made by a  The choice made at each
greedy algorithm may step usually depends on the
solutions to subproblems.
depend on choices so far,
but it cannot depend on any
future choices or on the
solutions to subproblems.
 A greedy strategy usually  Dynamic-programming
progresses in a top-down problems are often solved in
fashion, making one greedy a bottom-up manner.
choice after another,
reducing each given problem
instance to a smaller one.

Ravirajsinh Chauhan Greedy Algorithms 9

Steps in Design Greedy Algorithms
 Determine the optimal substructure of the problem.
 Develop a recursive solution.
 Prove that at any stage of the recursion, one of the
optimal choices is the greedy choice. Thus, it is
always safe to make the greedy choice.
 Show that all but one of the subproblems induced by
having made the greedy choice are empty.
 Develop a recursive algorithm that implements the
greedy strategy.
 Convert the recursive algorithm to an iterative
algorithm.
Ravirajsinh Chauhan Greedy Algorithms 10
Shortcuts Design
 Form the optimization problem so that, after a
choice is made and there is only one subproblem left
to be solved.
 Prove that there is always an optimal solution to the
original problem that makes the greedy choice, so
that the greedy choice is always safe.

 Demonstrate that, having made the greedy choice,

what remains is a subproblem with the property that
if we combine an optimal solution to the subproblem
with the greedy choice we have made, we arrive at
an optimal solution to the original problem.

Ravirajsinh Chauhan Greedy Algorithms 11

Minimum Spanning Tree
Definitions and Representation
 An undirected graph G is a pair (V,E), where V is a
finite set of points called vertices and E is a finite set
of edges.
 An edge e ∈ E is an unordered pair (u,v), where u,v
∈ V.
 In a directed graph, the edge e is an ordered pair
(u,v). An edge (u,v) is incident from vertex u and is
incident to vertex v.
 A path from a vertex v to a vertex u is a sequence
<v0,v1,v2,…,vk> of vertices where v0 = v, vk = u, and
(vi, vi+1) ∈ E for I = 0, 1,…, k-1.
 The length of a path is defined as the number of
edges in the path.
Ravirajsinh Chauhan Greedy Algorithms 13
Definitions and Representation

a) An undirected graph and (b) a directed

graph.

Ravirajsinh Chauhan Greedy Algorithms 14

Definitions and Representation
 An undirected graph is connected if every
pair of vertices is connected by a path.
 A forest is an acyclic graph, and a tree is a
connected acyclic graph.
 A graph that has weights associated with
each edge is called a weighted graph.

Ravirajsinh Chauhan Greedy Algorithms 15

Definitions and Representation
 Graphs can be represented by their adjacency
matrix or an edge (or vertex) list.

 Adjacency matrices have a value ai,j = 1 if nodes i

and j share an edge; 0 otherwise. In case of a
weighted graph, ai,j = wi,j, the weight of the edge.

 The adjacency list representation of a graph G =

(V,E) consists of an array Adj[1..|V|] of lists. Each
list Adj[v] is a list of all vertices adjacent to v.

 For a grapn with n nodes, adjacency matrices take

Θ(n2) space and adjacency list takes Θ(|E|) space.
Ravirajsinh Chauhan Greedy Algorithms 16
Definitions and Representation

An undirected graph and its adjacency matrix representation.

An undirected graph and its adjacency list

Ravirajsinh Chauhan
representation.
Greedy Algorithms 17
Minimum Spanning Tree
 A spanning tree of an undirected graph G is a
sub graph of G that is a tree containing all
the vertices of G.

 In a weighted graph, the weight of a sub

graph is the sum of the weights of the edges
in the sub graph.

 A minimum spanning tree (MST) for a

weighted undirected graph is a spanning tree
with minimum weight.
Ravirajsinh Chauhan Greedy Algorithms 18
Minimum Spanning Tree

An undirected graph and its minimum spanning tree.

Ravirajsinh Chauhan Greedy Algorithms 19

Definitions
 Let G = (V, E) be an undirected graph.
T is a minimum spanning tree of G if T ⊆ E is an
acyclic subset that connects all of the vertices and whose
total weight w(T ) =  w(u, v) is minimized.
(u,v)∈T

 Let A be a subset of some minimum spanning tree.

An edge (u, v) is a safe edge for A if A  {(u, v)} is also
a subset of a minimum spanning tree.

Ravirajsinh Chauhan Greedy Algorithms 20

 Kruskal’s Algorithm

Work with edges, rather than nodes

Two steps:
– Sort edges by increasing edge weight
– Select the first |V| – 1 edges that do
not generate a cycle

Ravirajsinh Chauhan Greedy Algorithms 21

 Walk-Through
Consider an undirected, weight
3
F graph
10 C
A 4
4
3
8
6
5
4
B D
4
H 1
2
3
G 3
E

Ravirajsinh Chauhan Greedy Algorithms 22

 Sort the edges by increasing edge
3
F weight
10 C
A 4
4
3 edge dv edge dv
8
6 (D,E) 1 (B,E) 4
5
4
B D (D,G) 2 (B,F) 4
4
H 1
(E,G) 3 (B,H) 4
2
3 (C,D) 3 (A,H) 5
G 3
E (G,H) 3 (D,F) 6
(C,F) 3 (A,B) 8
(B,C) 4 (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 23

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8 (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3 (B,H) 4
4
H 1
(C,D) 3 (A,H) 5
2
3 (G,H) 3 (D,F) 6
G 3
E (C,F) 3 (A,B) 8
(B,C) 4 (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 24

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3 (B,H) 4
4
H 1
(C,D) 3 (A,H) 5
2
3 (G,H) 3 (D,F) 6
G 3
E (C,F) 3 (A,B) 8
(B,C) 4 (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 25

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3 (A,H) 5
2
3 (G,H) 3 (D,F) 6
G 3
E (C,F) 3 (A,B) 8
(B,C) 4 (A,F) 10

Accepting edge (E,G) would create a

cycle

Ravirajsinh Chauhan Greedy Algorithms 26

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3 (D,F) 6
G 3
E (C,F) 3 (A,B) 8
(B,C) 4 (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 27

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3 (A,B) 8
(B,C) 4 (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 28

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3  (A,B) 8
(B,C) 4 (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 29

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3  (A,B) 8
(B,C) 4  (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 30

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4 
8  (B,F) 4
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3  (A,B) 8
(B,C) 4  (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 31

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4 
8  (B,F) 4 
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3  (A,B) 8
(B,C) 4  (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 32

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4 
8  (B,F) 4 
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4 
4
H 1
(C,D) 3  (A,H) 5
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3  (A,B) 8
(B,C) 4  (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 33

 Select first |V|–1 edges which do
not generate a cycle
3
10
F C edge dv edge dv

A 4
4
3 (D,E) 1  (B,E) 4 
8  (B,F) 4 
6 (D,G) 2
5
4
B D (E,G) 3  (B,H) 4 
4
H 1
(C,D) 3  (A,H) 5 
2
3 (G,H) 3  (D,F) 6
G 3
E (C,F) 3  (A,B) 8
(B,C) 4  (A,F) 10

Ravirajsinh Chauhan Greedy Algorithms 34

 Select first |V|–1 edges which do
not generate a cycle
3
F C edge dv edge dv

A 4
3 (D,E) 1  (B,E) 4 
(D,G) 2  (B,F) 4 
5
B D (E,G) 3  (B,H) 4 
H 1
(C,D) 3  (A,H) 5 

}
2
3 (G,H) 3  (D,F) 6
G E (C,F) 3  (A,B) 8
not
consider
ed
(B,C) 4  (A,F) 10

Done
Total Cost =  dv =
21

Ravirajsinh Chauhan Greedy Algorithms 35

Kruskal’s Algorithm
 Concept
 Build a forest of minimum spanning trees.
 Repeatedly connect the trees to create a subset of a
minimum spanning tree, until all nodes are covered.
 In connecting two trees, choose the edge of the least
weight.

 Let C1 and C2 denote the two trees that are

connected by (u, v). Since (u, v) must be a light
edge connecting C1 to some other tree, we need to
prove that (u, v) is a safe edge for C1.
 Kruskal’s algorithm is a greedy algorithm, because
at each step it adds to the forest an edge of least
possible weight.

Ravirajsinh Chauhan Greedy Algorithms 36

Example of Kruskal’s Algorithm

12 8

2 9 1 9

5 3 11

8 7 3 5

1 10

Ravirajsinh Chauhan Greedy Algorithms 37

Kruskal’s Algorithm
MST-KRUSKAL(G,w)
A=
for each vertex v in V [G]
do MAKE-SET(v)
sort the edges of E in nondecreasing order by weight w
for each edge (u, v) in E, taken in nondecreasing order
by weight
do if FIND-SET(u)  FIND-SET(v)
then A = A  {(u, v)}
UNION(u, v)
return A
Ravirajsinh Chauhan Greedy Algorithms 38
Kruskal’s Algorithm: Complexity
MST-KRUSKAL(G,w)
A=
for each vertex v in V [G] O(v)
do MAKE-SET(v) O(e lg e)
sort the edges of E in nondecreasing order by weight w
for each edge (u, v) in E, taken in nondecreasing order
by weight
do if FIND-SET(u)  FIND-SET(v)
then A = A  {(u, v)}
UNION(u, v) O((v+e) lg v)
return A
e  v ; O(e lg v)
2
Ravirajsinh Chauhan Greedy Algorithms 39
Set Operations
1 2
 Given the Member array
3
 Make-Set(v)
member[v] = v
member = [1,2,3] ; {1} {2} {3}
Make-Set runs in constant running time for a single set.

 Find-Set(v)
Return member[v]

Find-Set runs in constant time. find-set(2) = 2

 Union(u,v)
for i=1 to n
do if member[i] = u then member[i]=v

Scan through the member array and update old members to be the new set.
Running time O(n), length of member array.
Union(2,3)
member = [1,3,3] ; {1} {2 3}

Ravirajsinh Chauhan Greedy Algorithms 40

Kruskal’s Example
a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

• A={ }, Make each element its own set. {a} {b} {c} {d} {e} {f} {g}
{h}
• Sort edges.
• Look at smallest edge first: {c} and {f} not in same set, add it to A,
union together.
• Now get {a} {b} {c f} {d} {e} {g} {h}
Ravirajsinh Chauhan Greedy Algorithms 41
Kruskal Example
Keep going, checking next smallest edge.
Had: {a} {b} {c f} {d} {e} {g} {h}
{e} ≠ {h}, add edge.
a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

Now get {a} {b} {c f} {d} {e h} {g}

Ravirajsinh Chauhan Greedy Algorithms 42

Kruskal Example
Keep going, checking next smallest edge.
Had: {a} {b} {c f} {d} {e h} {g}
{a} ≠ {c f}, add edge.
a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

Now get {b} {a c f} {d} {e h} {g}

Ravirajsinh Chauhan Greedy Algorithms 43

Kruskal’s Example
Keep going, checking next smallest edge.
Had {b} {a c f} {d} {e h} {g}
{b}  {a c f}, add edge.
a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

Now get {a b c f} {d} {e h} {g}

Ravirajsinh Chauhan Greedy Algorithms 44

Kruskal’s Example
Keep going, checking next smallest edge.
Had {a b c f} {d} {e h} {g}
{a b c f} = {a b c f}, dont add it!
a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

Ravirajsinh Chauhan Greedy Algorithms 45

Kruskal’s Example
Keep going, checking next smallest edge.
Had {a b c f} {d} {e h} {g}
{a b c f} = {e h}, add it.
a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

Now get {a b c f e h} {d}{g}

Ravirajsinh Chauhan Greedy Algorithms 46

Kruskal’s Example
Keep going, checking next smallest edge.
Had {a b c f e h} {d}{g}
{d}  {a b c e f h}, add it.

a
6 4
5 9
b c d

14
2
10
e f g
15
3
h 8

Now get {a b c d e f h} {g}

Ravirajsinh Chauhan Greedy Algorithms 47

Kruskal’s Example
Keep going, check next two smallest edges.
Had {a b c d e f h} {g}
{a b c d e f h} = {a b c d e f h}, don’t add it.

a
6 4
5 9
b c d

14
2
10
e f g
15
3 8
h

Ravirajsinh Chauhan Greedy Algorithms 48

Kruskal’s Example

Do add the last one:

Had {a b c d e f h} {g}

a
6 4
5 9
b c d

14
2
10
e f g
15
3 8
h

Ravirajsinh Chauhan Greedy Algorithms 49

Prim’s Algorithm
 Prim’s algorithm has the property that the edges in the
set A always form a single tree.
 The tree starts from an arbitrary root vertex r.
 Grow the tree until it spans all the vertices in V.
 At each step, a light edge is added to the tree A that connects A
to an isolated vertex of GA = (V, A).

Ravirajsinh Chauhan Greedy Algorithms 50

Walk-Through
2
Initialize
3
F array
10 C
A 7
4
3 K dv pv
8  
18 A F
4
9
B D B F  
10
H 25 C F  
2
3 D F  
G 7
E E F  
F F  
G F  
H F  

Ravirajsinh Chauhan Greedy Algorithms 51

 2
Start with any node,
3
F say D
10 C
A 7
4
3 K dv pv
8
18 A
4
9
B D B
10
H 25 C
2
3 
G E
D T 0
7 E
F
G
H

Ravirajsinh Chauhan Greedy Algorithms 52

 2 Update distances of
adjacent, unselected
3 nodes
10
F C
A 7
4
3
8
18 K dv pv
4
9
B D A
10
H 25 B
2
3 C 3 D
G 7
E D T 0 
E 25 D
F 18 D
G 2 D
H

Ravirajsinh Chauhan Greedy Algorithms 53

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A
8
18 B
4
9
B D C 3 D
10
H 25
D T 0 
2
3 E 25 D
G 7
E F 18 D
G T 2 D
H

Ravirajsinh Chauhan Greedy Algorithms 54

 2 Update distances of
adjacent, unselected
3 nodes
10
F C
A 7
4
3
8 K dv pv
18
4
9
B D A
10
H 25
B
2
3 C 3 D
G 7
E D T 0 
E 7 G
F 18 D
G T 2 D
H 3 G

Ravirajsinh Chauhan Greedy Algorithms 55

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A
8
18 B
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E 7 G
G 7
E F 18 D
G T 2 D
H 3 G

Ravirajsinh Chauhan Greedy Algorithms 56

 2 Update distances of
adjacent, unselected
3 nodes
10
F C
A 7
4
3 K dv pv
8
18 A
4
9
B D B 4 C
10
H 25 C T 3 D
2
3 D T 0 
G 7
E E 7 G
F 3 C
G T 2 D
H 3 G

Ravirajsinh Chauhan Greedy Algorithms 57

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A
8
18 B 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E 7 G
G 7
E F T 3 C
G T 2 D
H 3 G

Ravirajsinh Chauhan Greedy Algorithms 58

 2 Update distances of
adjacent, unselected
3 nodes
10
F C
A 7
4
3 K dv pv
8
18 A 10 F
4
9
B D B 4 C
10
H 25 C T 3 D
2
3 D T 0 
G 7
E E 2 F
F T 3 C
G T 2 D
H 3 G

Ravirajsinh Chauhan Greedy Algorithms 59

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A 10 F
8
18 B 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E T 2 F
G 7
E F T 3 C
G T 2 D
H 3 G

Ravirajsinh Chauhan Greedy Algorithms 60

 2 Update distances of
adjacent, unselected
3 nodes
10
F C K dv pv

A 7
4
3 A 10 F
8
18 B 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E T 2 F
G 7
E F T 3 C
G T 2 D
H 3 G
Table entries
unchanged

Ravirajsinh Chauhan Greedy Algorithms 61

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A 10 F
8
18 B 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E T 2 F
G 7 E F T 3 C
G T 2 D
H T 3 G

Ravirajsinh Chauhan Greedy Algorithms 62

 2 Update distances of
adjacent, unselected
3 nodes
10
F C
A 7
4
3
K dv pv
8 A 4 H
18
4
9
B D B 4 C
H 10
C T 3 D
2 25
3
D T 0 
G 7
E E T 2 F
F T 3 C
G T 2 D
H T 3 G

Ravirajsinh Chauhan Greedy Algorithms 63

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A T 4 H
8
18 B 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E T 2 F
G 7
E F T 3 C
G T 2 D
H T 3 G

Ravirajsinh Chauhan Greedy Algorithms 64

 2 Update distances of
adjacent, unselected
3 nodes
10
F C K dv pv

A 7
4
3 A T 4 H
8
18 B 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E T 2 F
G 7
E F T 3 C
G T 2 D
H T 3 G
Table entries
unchanged

Ravirajsinh Chauhan Greedy Algorithms 65

 2 Select node with
minimum distance
3
10
F C K dv pv

A 7
4
3 A T 4 H
8
18 B T 4 C
4
9
B D C T 3 D
10
H 25
D T 0 
2
3 E T 2 F
G 7
E F T 3 C
G T 2 D
H T 3 G

Ravirajsinh Chauhan Greedy Algorithms 66

 2 Cost of Minimum
Spanning Tree =  dv =
3 21
F C K dv pv

A 4
3 A T 4 H
B T 4 C
4
B D C T 3 D
H D T 0 
2
3 E T 2 F
G E F T 3 C
G T 2 D
H T 3 G

Done

Ravirajsinh Chauhan Greedy Algorithms 67

Example of Prim’s Algorithm

12 8

2 9 1 9

5 3 11

8 7 3 5

1 10

Ravirajsinh Chauhan Greedy Algorithms 68

Prim’s Algorithm
PRIM(G,w, r)
for each u in V[G]
do key[u] = ∞
π[u] = NIL
key[r] = 0
Q = V[G]
while Q  
do u = EXTRACT-MIN(Q)
for each v in Adj[u]
do if v in Q and w(u, v) < key[v]
then π[v] = u
key[v] = w(u, v)

Ravirajsinh Chauhan Greedy Algorithms 69

Execution of Prim’s Algorithm

12 8
0 12
9
3 8

2 9 1 9

5 2 3 3
1 11

8 7 3 5

5 1 1
7 10 10
5

Ravirajsinh Chauhan Greedy Algorithms 70

Prim’s Algorithm:Complexity
PRIM(G,w, r)
for each u in V[G] Also build min-heap
do key[u] = ∞
O(v)
π[u] = NIL
key[r] = 0
Q = V[G] O(e lg v)
while Q  
do u = EXTRACT-MIN(Q) O(lg v) O(v lg v)
for each v in Adj[u]
do if v in Q and w(u, v) < key[v]
then π[v] = u
key[v] = w(u, v) O(lg v)

Ravirajsinh Chauhan Greedy Algorithms 71

Prim’s MST Algorithm
 Also greedy, like Kruskal’s
 Will find a MST but may differ from Prim’s if multiple MST’s are
possible
MST-Prim(G,w,r) ; Graph G, weights w, root r
Q  V[G]
for each vertex u  Q do key[u]   ; infinite “distance”
key[r]  0
P[r]  NIL
while Q<>NIL do
u  Extract-Min(Q) ; remove closest node
; Update children of u so they have a parent and a min key val
; the key is the weight between node and parent
for each v Adj[u] do
if v Q & w(u,v)<key[v] then
P[v]  u
key[v]  w(u,v)
Ravirajsinh Chauhan Greedy Algorithms 72
Prim’s Example
Example: Graph given earlier.
Q={ (e,0) (a,  ) (b,  ) (c,  ) (d,  ) (f,  ) (g,  ) (h,  ) }

inf
a
6 4
inf inf inf
5 9
b c d

14
2
10 inf
e f g
0/nil 15
3 inf
8
h

inf

Extract min, vertex e. Update neighbor if in Q and weight < key.

Ravirajsinh Chauhan Greedy Algorithms 73

Prim’s Example
inf
a
6 4
14/e inf inf
5 9
b c d

14
2
10 inf
e f g
0/nil 15
3 inf
h 8

3/e
Q={ (a,  ) (b,14) (c,  ) (d,  ) (f,  ) (g,  ) (h,3) }

Extract min, vertex h. Update neighbor if in Q and weight < key

Ravirajsinh Chauhan Greedy Algorithms 74

Prim’s Algorithm
inf
a
6 4
10/h inf inf
5 9
b c d

14
2
10 inf
e f g
0/nil 15
3 8/h
h 8

3/e

Q={ (a,  ) (b,10) (c,  ) (d,  ) (f,8) (g,  ) }

Extract min, vertex f. Update neighbor if in Q and weight < key

Ravirajsinh Chauhan Greedy Algorithms 75

Prim’s Algorithm
inf
a
6 4
10/h 2/f inf
5 9
b c d

14
2
10 15/f
e f g
0/nil 15
3 8/h
h 8

3/e
Q={ (a,  ) (b,10) (c, 2) (d,  ) (g,15) }

Extract min, vertex c. Update neighbor if in Q and weight < key

Ravirajsinh Chauhan Greedy Algorithms 76

Prim’s Algorithm
4/c
a
6 4
5/c 2/f 9/c
5 9
b c d

14
2
10 15/f
e f g
0/nil 15
3 8/h
h 8

3/e
Q={ (a,4) (b,5) (d,9) (g,15) }

Extract min, vertex a. No keys are smaller than edges from a (4>2 on edge ac, 6>5 on edge ab) so nothing
done.

Q={ (b,5) (d,9) (g,15) }

Extract min, vertex b.

Same case, no keys are smaller than edges, so nothing is done.

Same for extracting d and g, and we are done.
Ravirajsinh Chauhan Greedy Algorithms 77
Prim’s Algorithm
Get spanning tree by connecting nodes with their parents:

4/c
a
6 4
5/c 2/f 9/c
5 9
b c d

14
2
10 15/f
e f g
0/nil 15
3 8/h
8
h

3/e

Ravirajsinh Chauhan Greedy Algorithms 78

Runtime for Prim’s Algorithm
MST-Prim(G,w,r)
Q  V[G]
; Graph G, weights w, root r O(V) if using a heap
for each vertex u  Q do key[u]   ; infinite “distance”
key[r]  0
P[r]  NIL O(lgV) if using a heap
while Q<>NIL do
u  Extract-Min(Q) ; remove closest node
O(V) ; Update children of u so they have a parent and a min key val
; the key is the weight between node and parent O(E) over entire while(Q<>NIL) loo
for each v Adj[u] do
if v Q & w(u,v)<key[v] then
P[v]  u O(lgV) to update if using a heap!
key[v]  w(u,v)

The inner loop takes O(E lg V) for the heap update inside the O(E) loop.
This is over all executions, so it is not multiplied by O(V) for the while loop
(this is included in the O(E) runtime through all edges.

The Extract-Min requires O(V lg V) time.

O(lg V) for the Extract-Min and O(V) for the while loop.

Total runtime is then O(V lg V) + O(E lg V) which is O(E lg V)

in a connected graph
(a connected
Ravirajsinh Chauhangraph will always have at least V-1Algorithms
Greedy edges). 79
Huffman Code
Problem
 Find an optimal prefix code representing a
set of characters in a file, where each
character has a frequency of occurrences.
 Prefix code
 Codes in which no codeword is also a prefix of
some other codeword.
 {0, 110, 101, 111, 1000, 1001} is a prefix code.
 {0, 110, 101, 111, 1010, 0111} is not prefix code
 Optimality
 The code yields a file with the minimum number
of bits.

Ravirajsinh Chauhan Greedy Algorithms 81

Creating Huffman Code: Example
100
1

64
0 1
0

34 30
1

0 1 0 15

0 1

ก: 36 ข: 17 ค: 17 ง: 15 จ: 10 ฉ: 5
0 100 101 110 1110 1111

Ravirajsinh Chauhan Greedy Algorithms 82

Optimal Code
 An optimal code for a file is always
represented by a full binary tree, in which
every nonleaf node has two children.

 If C is the alphabet from which the

characters are drawn and all character
frequencies are positive, then the tree for an
optimal prefix code has exactly |C| leaves,
one for each letter of the alphabet, and
exactly |C|−1 internal nodes.

Ravirajsinh Chauhan Greedy Algorithms 83

Full Binary Trees for Prefix Code
Tree for 1 letter Tree for 2 letters
Tree for 4 letters
1 A
1 2
B
Tree for 3 letters
C A
B 4 3 1 2
3 A C
1 2
4 B
A 3 A
1 B 1 2
2 3
Ravirajsinh Chauhan Greedy Algorithms 84
Full Binary Trees for Prefix Code
Tree for 4 letters Tree for 5 letters

B B
C A C A
4 3 1 2 D 3 1 2
5 4

C
D
4 B
5 B
3 A
C A
1 2
4 3 1 2

Ravirajsinh Chauhan Greedy Algorithms 85

Creating Huffman Code: Example
กขคงจฉ: 100

ขคงจฉ: 64
0 0 1

ขค: 34 งจฉ: 30
1

0 1 0 จฉ: 15
0 1

ก: 36 ข: 17 ค: 17 ง: 15 จ: 10 ฉ: 5
0 100 101 110 1110 1111
Ravirajsinh Chauhan Greedy Algorithms 86
Theorem
 A full binary tree for an optimal prefix code for C letters
has exactly C leaves, and exactly C−1 internal nodes.
Proof by induction.
Basis:
C=1. If there is one letter, the binary tree requires only 1
leaf node, and 0 internal node.
Induction hypotheses:
For C< n, the full binary tree for an optimal prefix code
for C letters has exactly C leaves, and exactly C−1
internal nodes.

Ravirajsinh Chauhan Greedy Algorithms 87

Theorem
 Induction Step:
Let T be a full binary tree for an optimal prefix code for
C+1 letters.
To create a full binary tree or optimal prefix code, we
can take a full binary tree for an optimal prefix code for
C letters, and add another leaf node L by either
 adding a new root node R and put L and the old root of T as its
children, or
 replacing a leaf node N of T by an internal node and put L and
N as its children.
In either case, the number of leaf nodes of T is C+1 and
the number of internal nodes is C.

Ravirajsinh Chauhan Greedy Algorithms 88

Creating Huffman Code: Algorithms
HUFFMAN(C) Use min-heap for Q
n = |C| O(n) to build min-heap
Q=C
for i = 1 to n − 1 O(n lg n)
do allocate a new node z
z.left = x = EXTRACT-MIN(Q)
z.right = y = EXTRACT-MIN(Q)
z.f =x.f + y.f O(lg n)
INSERT(Q, z)
►Return the root of the tree.
return EXTRACT-MIN(Q)
Ravirajsinh Chauhan Greedy Algorithms 89
Greedy-choice property of Huffman Code
Let C be an alphabet in which each character c  C has
frequency f [c], and
x and y be two characters in C having the lowest
frequencies.
 Then, there exists an optimal prefix code for C in which
the codewords for x and y have the same length and
differ only in the last bit.

Ravirajsinh Chauhan Greedy Algorithms 90

Proof
 The idea of the proof is to :
 take the tree T representing an arbitrary optimal prefix code
 modify it to make a tree representing another optimal prefix
code such that the characters x and y appear as sibling leaves of
maximum depth in the new tree.
 If we can do this, then their codewords will have the
same length and differ only in the last bit.

Ravirajsinh Chauhan Greedy Algorithms 91

Proof
Let a and b be two characters that are sibling leaves of
maximum depth in T .

Assume that f [a] ≤ f [b] and f [x] ≤ f [y].

Since f [x] and f [y] are the two lowest leaf frequencies, in
order, and f [a] and f [b] are two arbitrary frequencies, in
order, we have f [x] ≤ f [a] and f [y] ≤ f [b].

Ravirajsinh Chauhan Greedy Algorithms 92

Proof
Exchange the positions of a and x in T to produce T’ .
B(T) − B(T’ ) =  f(c) dT (c) −  f(c)dT’ (c)
cC cC

= f [x] dT (x) + f [a] dT (a) − f [x] dT’ (x) − f [a] dT’ (a)
= f [x] dT (x) + f [a] dT (a) − f [x] dT (a) − f [a] dT (x)
= ( f [a] − f [x])( dT (a) − dT (x)) ≥ 0

T T’
x a

y y
a b x b
Ravirajsinh Chauhan Greedy Algorithms 93
Proof
Then, exchange the positions of b
and y in T’ to produce T’’. T’
a
Similarly, it does not increase the
cost, and so B(T’ ) − B(T’’ )  0. y
Therefore, B(T’’ ) ≤ B(T).
x b
Since T is optimal, B(T) ≤ B(T’’ ).
Then, B(T’’ ) = B(T).
T’’
Thus, T’’ is an optimal tree in which a
x and y appear as sibling leaves of
maximum depth. b
x y
Ravirajsinh Chauhan Greedy Algorithms 94
Optimal-substructure Property
Let C be a given alphabet with frequency f [c] defined for
each character c  C,
x and y be two characters in C with minimum frequency,
C’ be the alphabet C with characters x, y removed and
(new) character z added, so that C’ = C − {x, y}  {z};
define f for C’ as for C, except that f [z] = f [x] + f [y], and
T be any tree representing an optimal prefix code for the
alphabet C ’.
 Then the tree T , obtained from T ’ by replacing the leaf
node for z with an internal node having x and y as
children, represents an optimal prefix code for the
alphabet C.
Ravirajsinh Chauhan Greedy Algorithms 95
Proof
Show that the cost B(T) can be expressed in terms of the
cost B(T’) by considering the component costs.
For each c  C − {x, y}, we have
 dT (c) = dT ’ (c).
 f [c] dT (c) = f [c] dT ’ (c).
Since dT (x) = dT (y) = dT ’(z) + 1, we have
f [x]dT (x) + f [y]dT (y) = ( f [x] + f [y])( dT’ (z) + 1)
= f [z] dT ’ (z) + ( f [x] + f [y])
Thus, B(T) = B(T ’) + f [x] + f [y].
That is, B(T ’) = B(T) − f [x] − f [y] .

Ravirajsinh Chauhan Greedy Algorithms 96

Proof
We now prove by contradiction.

Suppose T does not represent an optimal prefix code for C.

Then there exists a tree T ’’ such that B(T ’’) < B(T).

Without loss of generality, T ’’ has x and y as siblings.

Ravirajsinh Chauhan Greedy Algorithms 97

Proof
Let T ’’’ be the tree T ’’ with the common parent of x and
y replaced by a leaf z with frequency f [z] = f [x] + f [y].

Then, B(T ’’’) = B(T ’’) − f [x] − f [y]

< B(T) − f [x] − f [y] = B(T ’).

We reach a contradiction to the assumption that T ’

represents an optimal prefix code for C’.

Thus, T must represent an optimal prefix code for the

alphabet C.

Ravirajsinh Chauhan Greedy Algorithms 98

Interval Scheduling
Problem definition
 Let S be {a1, a2, . . . , an} of n proposed activities that
wish to use the same resource.
 Only one activity can use the resource at a time.
 Each activity ai has a start time si and a finish time fi,
where 0 ≤ si < fi < ∞.
 If selected, activity ai takes place during the half-open
time interval [si , fi ).
 Activities ai and aj are compatible if the intervals [si , fi )
and [s j , f j ) do not overlap .
 The activity-selection problem is to select a largest
subset of mutually compatible activities.
Ravirajsinh Chauhan Greedy Algorithms 100
Example

A1
A2
A3

Ravirajsinh Chauhan Greedy Algorithms 101

Subproblems
 Si j = {ak  S : fi ≤ sk < fk ≤ s j }
 the subset of activities in S that can start after activity ai
finishes and finish before activity aj starts.

 Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞.

 Then S = S0,n+1, and 0 ≤ i, j ≤ n + 1.
Ravirajsinh Chauhan Greedy Algorithms 102
Optimal Solution
 Let Ai j be an optimal solution to Si j .
 Let c[i, j ] be the number of activities in a maximum-size
subset of mutually compatible activities in Si j.
 c[i, j ] = 0 for i ≥ j (i.e. Si j = )
ak

 c[i, j ] = c[i, k] + c[k, j ] + 1 if ak be an activity in Ai j.

Then,
 c[i, j ] =  0 if Si j = 
max {c[i, k] + c[k, j ] + 1} if Si j  
 i<k<j
akSi j
Ravirajsinh Chauhan Greedy Algorithms 103
Theorem
 Consider any nonempty subproblem Si j , and let am be
the activity in Si j with the earliest finish time:
fm = min { fk : ak ∈ Si j } .
Then,
 Activity am is used in some maximum-size subset of
mutually compatible activities of Si j .
 The subproblem Sim is empty, so that choosing am leaves
the subproblem Smj as the only one that may be
nonempty.

Ravirajsinh Chauhan Greedy Algorithms 104

Proof: First part
 Let Ai j be a maximum-size subset of mutually
compatible activities of Si j , and Ai j is sorted in
monotonically increasing order of finish time.
 Let ak be the first activity in Ai j.
 If ak = am, am is used in some maximum-size subset of
mutually compatible activities of Si j.
 If ak  am, we construct the subset A’i j = Ai j − {ak} 
{am}.
S

Aij ak
A’ij am

Ravirajsinh Chauhan Greedy Algorithms 105

Proof: First part
 The activities in A’i j are disjoint, since the activities in Aij
are, ak is the first activity in Ai j to finish, and fm ≤ fk .
 Noting that A’i j has the same number of activities as Ai j ,
we see that A’i j is a maximum-size subset of mutually
compatible activities of Si j that includes am.

Aij ak
A’ij am

Ravirajsinh Chauhan Greedy Algorithms 106

Proof: Second part
Let Sim be nonempty.
 Then, there is an activity ak such that fi ≤ sk <fk ≤ sm< fm.
 Then, ak is also in Si j and it has an earlier finish time than
am, which contradicts our choice of am.
 We conclude that Sim is empty.

ai

am
S
ak
aj

Ravirajsinh Chauhan Greedy Algorithms 107

Greedy Solution

Ai j = {ak}  Akj

where ak is the activity which finishes earliest

among activities in Si j.

Ravirajsinh Chauhan Greedy Algorithms 108

Recursive Algorithms
ACTIVITY-SELECTOR(s, f, i, n)
m=i+1
while m ≤ n and s[m] < f [i]
► Find the first activity in Si,n+1.
do m= m+1
if m ≤ n
then return {am} 
ACTIVITY-SELECTOR(s, f,m, n)
else return 

Ravirajsinh Chauhan Greedy Algorithms 109

Iterative Algorithm
GREEDY-ACTIVITY-SELECTOR(s, f )
n = length[s]
A = {a1}
i=1
for m = 2 to n
do if s[m ] ≥ f[i]
then A = A  {am}
i=m
return A

Ravirajsinh Chauhan Greedy Algorithms 110