CSE 431/531: Algorithm Analysis and Design (Fall 2022)

Greedy Algorithms

Lecturer: Shi Li
Department of Computer Science and Engineering
University at Buffalo
Def. In an optimization problem, our goal of is to find a valid
solution with the minimum cost (or maximum value).

Trivial Algorithm for an Optimization Problem

Enumerate all valid solutions, compare them and output the best one.

However, trivial algorithm often runs in exponential time, as the

number of potential solutions is often exponentially large.
f (n) is a polynomial if f (n) = O(nk ) for some constant k > 0.
convention: polynomial time = efficient

Goals of algorithm design

1 Design efficient algorithms to solve problems
2 Design more efficient algorithms to solve problems

2/80
Common Paradigms for Algorithm Design

Greedy Algorithms
Divide and Conquer
Dynamic Programming
Greedy algorithms are often for optimization problems.
They often run in polynomial time due to their simplicity.

3/80
Greedy Algorithm
Build up the solutions in steps
At each step, make an irrevocable decision using a “reasonable”
strategy

Analysis of Greedy Algorithm

Prove that the reasonable strategy is “safe” (key)
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem (usually
easy)

Def. A strategy is safe: there is always an optimum solution that

agrees with the decision made according to the strategy.

4/80
Outline

1 Toy Example: Box Packing

2 Interval Scheduling

3 Offline Caching
Heap: Concrete Data Structure for Priority Queue

4 Data Compression and Huffman Code

5 Summary

5/80
Box Packing
Input: n boxes of capacities c1 , c2 , · · · , cn
m items of sizes s1 , s2 , · · · , sm
Can put at most 1 item in a box
Item j can be put into box i if sj ≤ ci
Output: A way to put as many items as possible in the boxes.

Example:
Box capacities: 60, 40, 25, 15, 12
Item sizes: 45, 42, 20, 19, 16
Can put 3 items in boxes: 45 → 60, 20 → 40, 19 → 25

6/80
Greedy Algorithm
Build up the solutions in steps
At each step, make an irrevocable decision using a “reasonable”
strategy

Designing a Reasonable Strategy for Box Packing

Q: Take box 1. Which item should we put in box 1?
A: The item of the largest size that can be put into the box.

7/80
Analysis of Greedy Algorithm
Prove that the reasonable strategy is “safe”
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem

Lemma The strategy that put into box 1 the largest item it can
hold is “safe”: There is an optimum solution in which box 1 contains
the largest item it can hold.

Intuition: putting the item gives us the easiest residual problem.

formal proof via exchanging argument:

8/80
Lemma There is an optimum solution in which box 1 contains the
largest item it can hold.

Proof.
Let j = largest item that box 1 can hold.
Take any optimum solution S. If j is put into Box 1 in S, done.
Otherwise, assume this is what happens in S:
box 1
S 0: · · ·· · ·

item j 0 item j

sj ′ ≤ sj , and swapping gives another solution S ′

S ′ is also an optimum solution. In S ′ , j is put into Box 1.

9/80
 Notice that the exchanging operation is only for the sake of
analysis; it is not a part of the algorithm.

Analysis of Greedy Algorithm

Prove that the reasonable strategy is “safe”
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem

Trivial: we decided to put Item j into Box 1, and the remaining

instance is obtained by removing Item j and Box 1.

10/80
Generic Greedy Algorithm
1: while the instance is non-trivial do
2: make the choice using the greedy strategy
3: reduce the instance

Greedy Algorithm for Box Packing

1: T ← {1, 2, 3, · · · , m}
2: for i ← 1 to n do
3: if some item in T can be put into box i then
4: j ← the largest item in T that can be put into box i
5: print(“put item j in box i”)
6: T ← T \ {j}

11/80
Generic Greedy Algorithm
1: while the instance is non-trivial do
2: make the choice using the greedy strategy
3: reduce the instance

Lemma Generic algorithm is correct if and only if the greedy

strategy is safe.

Greedy strategy is safe: we will not miss the optimum solution

Greedy stretegy is not safe: we will miss the optimum solution for
some instance, since the choices we made are irrevocable.

12/80
Greedy Algorithm
Build up the solutions in steps
At each step, make an irrevocable decision using a “reasonable”
strategy

Analysis of Greedy Algorithm

Prove that the reasonable strategy is “safe”
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem

Def. A strategy is “safe” if there is always an optimum solution that

is “consistent” with the decision made according to the strategy.

13/80
Exchange argument: Proof of Safety of a Strategy

let S be an arbitrary optimum solution.

if S is consistent with the greedy choice, done.
otherwise, show that it can be modified to another optimum
solution S ′ that is consistent with the choice.

The procedure is not a part of the algorithm.

14/80
Outline

1 Toy Example: Box Packing

2 Interval Scheduling

3 Offline Caching
Heap: Concrete Data Structure for Priority Queue

4 Data Compression and Huffman Code

5 Summary

15/80
Interval Scheduling
Input: n jobs, job i with start time si and finish time fi
i and j are compatible if [si , fi ) and [sj , fj ) are disjoint
Output: A maximum-size subset of mutually compatible jobs

0 1 2 3 4 5 6 7 8 9

16/80
Greedy Algorithm for Interval Scheduling
Which of the following strategies are safe?
Schedule the job with the smallest size? No!
0 1 2 3 4 5 6 7 8 9

17/80
Greedy Algorithm for Interval Scheduling
Which of the following strategies are safe?
Schedule the job with the smallest size? No!
Schedule the job conflicting with smallest number of other jobs?
No!
0 1 2 3 4 5 6 7 8 9

18/80
Greedy Algorithm for Interval Scheduling
Which of the following strategies are safe?
Schedule the job with the smallest size? No!
Schedule the job conflicting with smallest number of other jobs?
No!
Schedule the job with the earliest finish time? Yes!
0 1 2 3 4 5 6 7 8 9

19/80
Greedy Algorithm for Interval Scheduling

Lemma It is safe to schedule the job j with the earliest finish time:
There is an optimum solution where the job j with the earliest finish
time is scheduled.

Proof.
Take an arbitrary optimum solution S
If it contains j, done
Otherwise, replace the first job in S with j to obtain another
optimum schedule S ′ .

S:

j:

S 0:
20/80
Greedy Algorithm for Interval Scheduling

Lemma It is safe to schedule the job j with the earliest finish time:
There is an optimum solution where the job j with the earliest finish
time is scheduled.

What is the remaining task after we decided to schedule j?

Is it another instance of interval scheduling problem? Yes!
0 1 2 3 4 5 6 7 8 9

21/80
Greedy Algorithm for Interval Scheduling

Schedule(s, f, n)
1: A ← {1, 2, · · · , n}, S ← ∅
2: while A ̸= ∅ do
3: j ← arg minj ′ ∈A fj ′
4: S ← S ∪ {j}; A ← {j ′ ∈ A : sj ′ ≥ fj }
5: return S

0 1 2 3 4 5 6 7 8 9

22/80
Greedy Algorithm for Interval Scheduling

Schedule(s, f, n)
1: A ← {1, 2, · · · , n}, S ← ∅
2: while A ̸= ∅ do
3: j ← arg minj ′ ∈A fj ′
4: S ← S ∪ {j}; A ← {j ′ ∈ A : sj ′ ≥ fj }
5: return S

Running time of algorithm?

Naive implementation: O(n2 ) time
Clever implementation: O(n lg n) time

23/80
Clever Implementation of Greedy Algorithm

Schedule(s, f, n)
1: sort jobs according to f values
2: t ← 0, S ← ∅
3: for every j ∈ [n] according to non-decreasing order of fj do
4: if sj ≥ t then
t
5: S ← S ∪ {j}
6: t ← fj
7: return S 0 1 2 3 4 5 6 7 8 9
2 5 9
4 8
3 7
1 6

24/80
Outline

1 Toy Example: Box Packing

2 Interval Scheduling

3 Offline Caching
Heap: Concrete Data Structure for Priority Queue

4 Data Compression and Huffman Code

5 Summary

25/80
Offline Caching
cache
page
Cache that can store k pages sequence
Sequence of page requests 1 1
Cache miss happens if 5 1 5
requested page not in cache. 4 5 4
1
We need bring the page into
cache, and evict some existing 2 1 2 4
page if necessary. 5 1 2 5
Cache hit happens if requested 3 1 2 3
page already in cache.
2 1 2 3
Goal: minimize the number of
cache misses. 1 1 2 3
misses = 6

26/80
A Better Solution for Example

cache cache
page
sequence
1 1 1
5 1 5 1 5
4 1 5 4 1 5 4
2 1 2 4 1 5 2
5 1 2 5 1 5 2
3 1 2 3 1 3 2
2 1 2 3 1 3 2
1 1 2 3 1 3 2
misses = 6 misses = 5

27/80
Offline Caching Problem
Input: k : the size of cache
We use [n] for {1, 2, 3, · · · , n}.
n : number of pages
ρ1 , ρ2 , ρ3 , · · · , ρT ∈ [n]: sequence of requests
Output: i1 , i2 , i3 , · · · , iT ∈ {hit, empty} ∪ [n]: indices of pages to
evict (“hit” means evicting no page, “empty” means
evicting empty page)

Offline Caching: we know the whole sequence ahead of time.

Online Caching: we have to make decisions on the fly, before
seeing future requests.

Q: Which one is more realistic?

A: Online caching
28/80
 Offline Caching: we know the whole sequence ahead of time.
Online Caching: we have to make decisions on the fly, before
seeing future requests.

Q: Which one is more realistic?

A: Online caching

Q: Why do we study the offline caching problem?

A: Use the offline solution as a benchmark to measure the

“competitive ratio” of online algorithms

29/80
Offline Caching: Potential Greedy Algorithms

FIFO(First-In-First-Out): always evict the first page in cache

LRU(Least-Recently-Used): Evict page whose most recent access
was earliest
LFU(Least-Frequently-Used): Evict page that was least frequently
requested

All the above algorithms are not optimum!

Indeed all the algorithms are “online”, i.e, the decisions can be
made without knowing future requests. Online algorithms can not
be optimum.

30/80
FIFO is not optimum

FIFO Furthest-in-Future
requests
1 1 1
2 1 2 1 2
3 1 2 3 1 2 3
4 4 2 3 1 4 3
1 4 1 3 1 4 3
misses = 5 misses = 4

31/80
Optimum Offline Caching

Furthest-in-Future (FF)
Algorithm: every time, evict the page that is not requested until
furthest in the future, if we need to evict one.
The algorithm is not an online algorithm, since the decision at a
step depends on the request sequence in the future.

32/80
Furthest-in-Future (FF)

FIFO Furthest-in-Future
requests
1 1 1
2 1 2 1 2
3 1 2 3 1 2 3
4 4 2 3 1 4 3
1 4 1 3 1 4 3
misses = 5 misses = 4

33/80
Example

requests
1 5 4 2 5 3 2 4 3 1 5 3

1 1 1 2 2 2 2 2 2 1 5 5

5 5 5 5 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4

34/80
Recall: Designing and Analyzing Greedy Algorithms

Greedy Algorithm
Build up the solutions in steps
At each step, make an irrevocable decision using a “reasonable”
strategy

Analysis of Greedy Algorithm

Prove that the reasonable strategy is “safe” (key)
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem (usually
easy)

35/80
Offline Caching Problem
Input: k : the size of cache
n : number of pages
ρ1 , ρ2 , ρ3 , · · · , ρT ∈ [n]: sequence of requests
p1 , p2 , · · · , pk ∈ {empty} ∪ [n]: initial set of pages in cache
Output: i1 , i2 , i3 , · · · , it ∈ {hit, empty} ∪ [n]
empty stands for an empty page
“hit” means evicting no pages

36/80
Analysis of Greedy Algorithm
Prove that the reasonable strategy is “safe” (key)
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem (usually
easy)

Lemma Assume at time 1 a page fault happens and there are no

empty pages in the cache. Let p∗ be the page in cache that is not
requested until furthest in the future. There is an optimum solution
in which p∗ is evicted at time 1.

37/80
 4 2 3

1 1
S: 2 4
3 3

Proof.
1 S: any optimum solution
2 p∗ : page in cache not requested until furthest in the future.
In the example, p∗ = 3.
3 Assume S evicts some p′ ̸= p∗ at time 1; otherwise done.
In the example, p′ = 2.
38/80
 4 5 4 6 2 3

1 1 5 5 5
S: 2 4 4 4 4
3 3 3 3 6

1 1 5 5 5
S0 : 2 4 4 4 4
3 2 2 2 6

Proof.
4 Create S ′ . S ′ evicts p∗ (=3) instead of p′ (=2) at time 1.
5 After time 1, cache status of S and that of S ′ differ by only 1
page. S ′ contains p′ (=2) and S contains p∗ (=3).
6 From now on, S ′ will “copy” S.

39/80
 4 5 4 ··· 2 3

1 1 5 5 ··· 6
S: 2 4 4 4 8
3 3 3 3 ··· 3

1 1 5 5 ··· 6
S0 : 2 4 4 4 8
3 2 2 2 ··· 2

Proof.
7 If S evicted the page p∗ , S ′ will evict the page p′ . Then, the
cache status of S and that of S ′ will be the same. S and S ′ will
be exactly the same from now on.
8 Assume S did not evict p∗ (=3) before we see p′ (=2).

40/80
 4 5 4 ··· 2 3

1 1 5 5 ··· 6 2
S: 2 4 4 4 8 8
3 3 3 3 ··· 3 3

1 1 5 5 ··· 6 6
S0 : 2 4 4 4 8 8
3 2 2 2 ··· 2 2

Proof.
9 If S evicts p∗ (=3) for p′ (=2), then S won’t be optimum. Assume
otherwise.
10 So far, S ′ has 1 less page-miss than S does.
11 The status of S ′ and that of S only differ by 1 page.

41/80
 4 5 4 ··· 2 3

1 1 5 5 ··· 6 2
S: 2 4 4 4 8 8
3 3 3 3 ··· 3 3

1 1 5 5 ··· 6 6
S0 : 2 4 4 4 8 8
3 2 2 2 ··· 2 2

Proof.
12 We can then guarantee that S ′ make at most the same number of
page-misses as S does.
Idea: if S has a page-hit and S ′ has a page-miss, we use the
opportunity to make the status of S ′ the same as that of S.

42/80
 Thus, we have shown how to create another solution S ′ with the
same number of page-misses as that of the optimum solution S.
Thus, we proved

Lemma Assume at time 1 a page fault happens and there are no

empty pages in the cache. Let p∗ be the page in cache that is not
requested until furthest in the future. It is safe to evict p∗ at time 1.

Theorem The furthest-in-future strategy is optimum.

43/80
1: for t ← 1 to T do
2: if ρt is in cache then do nothing
3: else if there is an empty page in cache then
4: evict the empty page and load ρt in cache
5: else
6: p∗ ← page in cache that is not used furthest in the future
7: evict p∗ and load ρt in cache

44/80
Q: How can we make the algorithm as fast as possible?

A:
The running time can be made to be O(n + T log k).
For each page p, use a linked list (or an array with dynamic size)
to store the time steps in which p is requested.
We can find the next time a page is requested easily.
Use a priority queue data structure to hold all the pages in cache,
so that we can easily find the page that is requested furthest in
the future.

45/80
time 0 1 2 3 4 5 6 7 8 9 10 11 12
pages P1 P5 P4 P2 P5 P3 P2 P4 P3 P1 P5 P3

1 10 priority queue
P1:

pages priority
P2: 4 7 values
P5 ∞
P3: 6 9 12
P3 ∞
P4: 3 8 P4 ∞

P5: 2 5 11
46/80
 1: for every p ← 1 to n do
2: times[p] ← array of times in which p is requested, in
increasing order ▷ put ∞ at the end of array
3: pointer[p] ← 1
4: Q ← empty priority queue
5: for every t ← 1 to T do
6: pointer[ρt ] ← pointer[ρt ] + 1
7: if ρt ∈ Q then
8: Q.increase-key(ρt , times[ρt , pointer[ρt ]]), print “hit”,
continue
9: if Q.size() < k then
10: print “load ρt to an empty page ”
11: else
12: p ← Q.extract-max(), print “evict p and load ρt ”
13: Q.insert(ρt , times[ρt , pointer[ρt ]]) ▷ add ρt to Q with key
value times[ρt , pointer[ρt ]]
47/80
Outline

1 Toy Example: Box Packing

2 Interval Scheduling

3 Offline Caching
Heap: Concrete Data Structure for Priority Queue

4 Data Compression and Huffman Code

5 Summary

48/80
 Let V be a ground set of size n.

Def. A priority queue is an abstract data structure that maintains a

set U ⊆ V of elements, each with an associated key value, and
supports the following operations:
insert(v, key value): insert an element v ∈ V \ U , with associated
key value key value.
decrease key(v, new key value): decrease the key value of an
element v ∈ U to new key value
extract min(): return and remove the element in U with the
smallest key value
···

49/80
Simple Implementations for Priority Queue

n = size of ground set V

data structures insert extract min decrease key

array O(1) O(n) O(1)
sorted array O(n) O(1) O(n)
heap O(lg n) O(lg n) O(lg n)

50/80
Heap

The elements in a heap is organized using a complete binary tree:

1
Nodes are indexed as {1, 2, 3, · · · , s}
2 3 Parent of node i: ⌊i/2⌋
Left child of node i: 2i
4 5 6 7
Right child of node i: 2i + 1
8 9 10

51/80
Heap
A heap H contains the following fields
s: size of U (number of elements in the heap)
A[i], 1 ≤ i ≤ s: the element at node i of the tree
p[v], v ∈ U : the index of node containing v
key[v], v ∈ U : the key value of element v

1
f
s=5
2 3 A = (‘f ’, ‘g’, ‘c’, ‘e’, ‘b’)
g c
p[‘f ’] = 1, p[‘g’] = 2, p[‘c’] = 3,
p[‘e’] = 4, p[‘b’] = 5
4 5
e b
52/80
Heap
The following heap property is satisfied:
for any two nodes i, j such that i is the parent of j, we have
key[A[i]] ≤ key[A[j]].
2

4 5

10 9 7 11

15 17 20 17 15 8 16 23

21 16 19

A heap. Numbers in the circles denote key values of elements.

53/80
insert(v, key value)

3 5

4 9 7 11

15 10 20 17 15 8 16 23

21 16 19 17

54/80
 heapify-up(i)
insert(v, key value) 1: while i > 1 do
1: s←s+1 2: j ← ⌊i/2⌋
2: A[s] ← v 3: if key[A[i]] < key[A[j]] then
3: p[v] ← s 4: swap A[i] and A[j]
4: key[v] ← key value 5: p[A[i]] ← i, p[A[j]] ← j
5: heapify up(s) 6: i←j
7: else break

55/80
extract min()

17
3

17
3
4 5

17
10
4 9 7 11

15 10
17 20 17 15 8 16 23

21 16 19

56/80
extract min() heapify-down(i)
1: ret ← A[1] 1: while 2i ≤ s do
2: A[1] ← A[s] 2: if 2i = s or
3: p[A[1]] ← 1 key[A[2i]] ≤ key[A[2i + 1]] then
4: s←s−1 3: j ← 2i
5: if s ≥ 1 then 4: else
6: heapify down(1) 5: j ← 2i + 1
7: return ret 6: if key[A[j]] < key[A[i]] then
7: swap A[i] and A[j]
decrease key(v, key value)
8: p[A[i]] ← i, p[A[j]] ← j
1: key[v] ← key value 9: i←j
2: heapify-up(p[v]) 10: else break

57/80
 Running time of heapify up and heapify down: O(lg n)
Running time of insert, exact min and decrease key: O(lg n)
data structures insert extract min decrease key
array O(1) O(n) O(1)
sorted array O(n) O(1) O(n)
heap O(lg n) O(lg n) O(lg n)

58/80
Two Definitions Needed to Prove that the
Procedures Maintain Heap Property

Def. We say that H is almost a heap except that key[A[i]] is too

small if we can increase key[A[i]] to make H a heap.

Def. We say that H is almost a heap except that key[A[i]] is too

big if we can decrease key[A[i]] to make H a heap.

59/80
Outline

1 Toy Example: Box Packing

2 Interval Scheduling

3 Offline Caching
Heap: Concrete Data Structure for Priority Queue

4 Data Compression and Huffman Code

5 Summary

60/80
Encoding Letters Using Bits

8 letters a, b, c, d, e, f, g, h in a language
need to encode a message using bits
idea: use 3 bits per letter
a b c d e f g h
000 001 010 011 100 101 110 111

deacf g → 011100000010101110

Q: Can we have a better encoding scheme?

Seems unlikely: must use 3 bits per letter

Q: What if some letters appear more frequently than the others?

61/80
Q: If some letters appear more frequently than the others, can we
have a better encoding scheme?

A: Using variable-length encoding scheme might be more efficient.

Idea
using fewer bits for letters that are more frequently used, and
more bits for letters that are less frequently used.

62/80
Q: What is the issue with the following encoding scheme?
a: 0 b: 1 c: 00

A: Can not guarantee a unique decoding. For example, 00 can be

decoded to aa or c.

Solution
Use prefix codes to guarantee a unique decoding.

63/80
Prefix Codes

Def. A prefix code for a set S of letters is a function γ : S → {0, 1}∗

such that for two distinct x, y ∈ S, γ(x) is not a prefix of γ(y).

0 1

a b c d
0 1 0 1
001 0000 0001 100
h e
e f g h 0 1 0 1

a d
11 1010 1011 01 0 1 0 1

b c f g

64/80
Prefix Codes Guarantee Unique Decoding

Reason: there is only one way to cut the first code.

0 1

a b c d
0 1 0 1
001 0000 0001 100
h e
e f g h 0 1 0 1

a d
11 1010 1011 01 0 1 0 1

b c f g

0001/001/100/0000/01/01/11/1010/0001/001/
cadbhhefca

65/80
 Properties of Encoding Tree
0 1
Rooted binary tree
Left edges labelled 0 and right
0 1 0 1 edges labelled 1
h e A leaf corresponds to a code
0 1 0 1
for some letter
a d If coding scheme is not
0 1 0 1
wasteful: a non-leaf has exactly
b c f g two children

Best Prefix Codes

Input: frequencies of letters in a message
Output: prefix coding scheme with the shortest encoding for the
message

66/80
example
letters a b c d e
frequencies 18 3 4 6 10
scheme 1 length 2 3 3 2 2 total = 89
scheme 2 length 1 3 3 3 3 total = 87
scheme 3 length 1 4 4 3 2 total = 84

a
a

e
a d e
d
b c b c d e
b c

scheme 1 scheme 2 scheme 3

67/80
 Example Input: (a: 18, b: 3, c: 4, d: 6, e: 10)

Q: What types of decisions should we make?

Can we directly give a code for some letter?

Hard to design a strategy; residual problem is complicated.
Can we partition the letters into left and right sub-trees?
Not clear how to design the greedy algorithm

A: We can choose two letters and make them brothers in the tree.

68/80
Which Two Letters Can Be Safely Put Together
As Brothers?
Focus on the “structure” of the optimum encoding tree
There are two deepest leaves that are brothers

best to put the two least

frenquent symbols here!

Lemma It is safe to make the two least frequent letters brothers.

69/80
Lemma There is an optimum encoding tree, where the two least
frequent letters are brothers.

So we can irrevocably decide to make the two least frequent

letters brothers.

Q: Is the residual problem another instance of the best prefix codes

problem?

A: Yes, though it is not immediate to see why.

70/80
 fx : the frequency of the letter x in the support.
x1 and x2 : the two letters we decided to put together.
dx the depth of letter x in our output encoding tree.
X
f x dx
x∈S
X
= fx dx + fx1 dx1 + fx2 dx2
encoding tree for x∈S\{x1 ,x2 }
X
S \ {x1 , x2 } ∪ {x0 } = fx dx + (fx1 + fx2 )dx1
x∈S\{x1 ,x2 }
0
X
x = fx dx + fx′ (dx′ + 1)
x∈S\{x1 ,x2 }
x1 x2 X
= f x d x + f x′
x∈S\{x1 ,x2 }∪{x′ }
Def: fx′ = fx1 + fx2
71/80
In order to minimize X
f x dx ,
x∈S

we need to minimize
X
f x dx ,
x∈S\{x1 ,x2 }∪{x′ }

subject to that d is the depth function for an encoding tree of

S \ {x1 , x2 }.

This is exactly the best prefix codes problem, with letters

S \ {x1 , x2 } ∪ {x′ } and frequency vector f !

72/80
Example

75 A : 00
B : 10
0 1 C : 010
D : 011
47 28
E : 110
1 0 F : 111
1

0
20 13

0 1 0 1
27 15 11 9 8 5
A B C D E F

73/80
Def. The codes given the greedy algorithm is called the Huffman
codes.

Huffman(S, f )
1: while |S| > 1 do
2: let x1 , x2 be the two letters with the smallest f values
3: introduce a new letter x′ and let fx′ = fx1 + fx2
4: let x1 and x2 be the two children of x′
5: S ← S \ {x1 , x2 } ∪ {x′ }
6: return the tree constructed

74/80
Algorithm using Priority Queue

Huffman(S, f )
1: Q ← build-priority-queue(S)
2: while Q.size > 1 do
3: x1 ← Q.extract-min()
4: x2 ← Q.extract-min()
5: introduce a new letter x′ and let fx′ = fx1 + fx2
6: let x1 and x2 be the two children of x′
7: Q.insert(x′ , fx′ )
8: return the tree constructed

75/80
Outline

1 Toy Example: Box Packing

2 Interval Scheduling

3 Offline Caching
Heap: Concrete Data Structure for Priority Queue

4 Data Compression and Huffman Code

5 Summary

76/80
Summary for Greedy Algorithms

Greedy Algorithm
Build up the solutions in steps
At each step, make an irrevocable decision using a “reasonable”
strategy

Interval scheduling problem: schedule the job j ∗ with the earliest

deadline
Offline Caching: evict the page that is used furthest in the future
Huffman codes: make the two least frequent letters brothers

77/80
Summary for Greedy Algorithms

Analysis of Greedy Algorithm

Prove that the reasonable strategy is “safe” (key)
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem (usually
easy)

Def. A strategy is “safe” if there is always an optimum solution

that “agrees with” the decision made according to the strategy.

78/80
Proving a Strategy is Safe

Take an arbitrary optimum solution S

If S agrees with the decision made according to the strategy, done
So assume S does not agree with decision
Change S slightly to another optimum solution S ′ that agrees
with the decision
Interval scheduling problem: exchange j ∗ with the first job in an
optimal solution
Offline caching: a complicated “copying” algorithm
Huffman codes: move the two least frequent letters to the deepest
leaves.

79/80
Summary for Greedy Algorithms

Analysis of Greedy Algorithm

Prove that the reasonable strategy is “safe” (key)
Show that the remaining task after applying the strategy is to
solve a (many) smaller instance(s) of the same problem (usually
easy)

Interval scheduling problem: remove j ∗ and the jobs it conflicts

with
Offline caching: trivial
Huffman codes: merge two letters into one

80/80