Outline Dynamic Programming Greedy Algorithms Amortized

Outline Analysis
Dynamic Programming Greedy Algorithms

1 Dynamic Programming
Rod cutting problem
Elements of dynamic programming
Longest common subsequence
Optimal binary search trees

2 Greedy Algorithms
An activity-selection problem
Chapter 4 Elements of the greedy strategy
Advanced Algorithm Design and Analysis Techniques Huffman Codes

3 Amortized Analysis
Yonas Y. Aggregate analysis
Algorithm Analysis and Design
School of Electrical and Computer Engineering
1 2
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Dynamic Programming

When developing a dynamic-programming algorithm, we follow a

Not a specific algorithm, but a technique (like
sequence of four steps:
divide-and-conquer).

Developed back in the day when ”programming” meant 1 Characterize the structure of an optimal solution.
”tabular method” (like linear programming). Doesn’t really
refer to computer programming.
2 Recursively define the value of an optimal solution.

Dynamic programming applies when the subproblems

3 Compute the value of an optimal solution in a bottom-up
overlap-that is, when subproblems share subsubproblems. fashion.

Used for optimization problems: 4 Construct an optimal solution from computed information.

Find a solution with the optimal value.

Minimization or maximization.

3 4
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

Rod cutting
Sterling Enterprises buys long steel rods and cuts them into shorter The rod-cutting problem is the following.
rods, which it then sells.
Given a rod of length n inches and a table of prices pi for i =
Each cut is free. 1, 2,..., determine the maximum revenue rn obtainable by
The management of sterling enterprises wants to know the cutting up the rod and selling the pieces.
best way to cut up the rods.
We assume that we know, for i = 1, 2,. . . , the price pi in birr Consider the case when n = 4.
that sterling enterprises charges for a rod of length i inches.
Rod lengths are always an integral number of inches. Figure 4.1 shows all the ways to cut up a rod of 4 inches in
length, including the way with no cuts at all.
length i 1 2 3 4 5 6 7 8 9 10
price pi 1 5 8 9 10 17 17 20 24 30 We see that cutting a 4-inch rod into two 2-inch pieces
produces revenue p2 + p2 = 5 + 5 = 10, which is optimal.
Table: 4.1 A sample price table for rods. Each rod of length i inches
earns the company pi birrs of revenue.
5 6
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

If an optimal solution cuts the rod into k pieces, for some

1 ≤ k ≤ n, then an optimal decomposition

n = i1 + i2 + . . . + ik

of the rod into pieces of lengths i1 , i2 , . . . , ik provides maximum

corresponding revenue

rn = pi1 + pi2 + . . . + pik .

Figure: 4.1 The 8 possible ways of cutting up a rod of length 4. Above
each piece is the value of that piece price.
Generally, we can frame the values rn for n ≥ 1 in terms of optimal
revenues from shorter rods:
We can cut up a rod of length n in 2n−1 different ways, since we
have an independent option of cutting, or not cutting, at distance rn = max (pn , r1 + rn−1 , r2 + rn−2 , . . . , rn−1 + r1 ).
i inches from the left end, for i = 1, 2,..., n-1.

7 8
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

Recursive top-down implementation

Note that to solve the original problem of size n, we solve smaller
problems of the same type, but of smaller sizes.
In a related, but slightly simpler, way to arrange a recursive
Once we make the first cut, we may consider the two pieces structure for the rod cutting problem:
as independent instances of the rod-cutting problem.
the rod is cut into a first piece of length i cut off the left-hand
The overall optimal solution incorporates optimal solutions to end, and then a right-hand remainder of length n - i.
the two related subproblems, maximizing revenue from each of
those two pieces. Only the remainder, and not the first piece, may be further
divided.
We say that the rod-cutting problem exhibits optimal
substructure: optimal solutions to a problem incorporate
optimal solutions to related subproblems, which we may solve rn = max (pi , rn-i ).
1≤i≤n
independently.

9 10
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

Figure 4.2 illustrates what happens for n = 4.

The following procedure implements the computation implicit CUT-ROD(p, n) calls CUT-ROD(p, n - i) for i = 1, 2,. . . , n.
above in a straightforward, top-down, recursive manner.
CUT-ROD(p, n) calls CUT-ROD(p, j) for each j = 0, 1,. . . , n
- 1.
Algorithm 1 CUT-ROD(p, n)
1: if n == 0 then
2: return 0
3: end if
4: q = −∞
5: for i = 1 to n do
6: q = max (q, p[i] + CUT-ROD(p, n - i))
7: end for
8: return q

Figure: 4.2 The recursion tree showing recursive calls resulting from a call
CUT-ROD(p, n) for n = 4.
11 12
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

Using dynamic programming for optimal rod cutting

To analyze the running time of CUT-ROD,

Since the recursive solution is inefficient =⇒ we arrange for
Let T(n) denote the total number of calls made to CUT-ROD. each subproblem to be solved only once, saving its solution.

The count includes the initial call at its root. Dynamic programming thus uses additional memory to save
computation time =⇒ time-memory trade-off.
Thus, T(0) = 1 and
A dynamic-programming approach runs in polynomial time
T(n) = 1 + n-1
P
j=0 T(j), when the number of distinct subproblems involved is
which gives an exponential running time of 2n . polynomial in the input size.

We can solve each such subproblem in polynomial time.

13 14
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

There are usually two equivalent ways to implement a

dynamic-programming approach. Here is the the pseudocode for the top-down CUT-ROD procedure,
with memoization added:
1 The first approach is top-down with memoization.
In this approach, we write the procedure recursively in a natural
manner, but modified to save the result of each subproblem. Algorithm 2 MEMOIZED-CUT-ROD(p, n)
1: let r[0 .. n] be a new array
2 The second approach is the bottom-up method. 2: for i = 1 to n do
3: r[i] = −∞
This approach typically depends on some natural notion of the
4: end for
”size” of a subproblem, such that solving any particular
5: return MEMOIZED-CUT-ROD-AUX(p, n, r)
subproblem depends only on solving ”smaller” subproblems.

These two approaches yield algorithms with the same asymptotic

running time.

15 16
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

The bottom-up version is even simpler:

Algorithm 3 MEMOIZED-CUT-ROD-AUX(p, n, r)
1: if r[n] ≤ 0 then
2: return r[n] Algorithm 4 BOTTOM-UP-CUT-ROD(p, n)
3: end if 1: let r[0 .. n] be a new array
4: if n == 0 then 2: r[0] = 0
5: return q = 0 3: for j = 1 to n do
6: else 4: q = −∞
7: q = −∞ 5: for i = 1 to j do
8: for i = 1 to n do 6: q = max (q, p[i] + r[j-i])
9: q = max (q, p[i] + MEMOIZED-CUT-ROD-AUX(p, n-i, 7: end for
r)) 8: r[j] = q
10: end for 9: end for
11: end if 10: return r[n]
12: r[n] = q
13: return q
The running time of the above procedures is Θ(n2 ).
17 18
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

Subproblem graphs

The subproblem graph for the problem embodies =⇒ the set of

subproblems involved and how subproblems depend on one
another.

Figure 4.3 shows the subproblem graph for the rod-cutting problem Figure: 4.3 The subproblem graph for the rod-cutting problem with n = 4.
with n = 4.
The size of the subproblem graph G = (V, E) can help us
It is a directed graph, containing one vertex for each distinct determine the running time of the dynamic programming
subproblem. algorithm.

Since we solve each subproblem just once, the running time is

the sum of the times needed to solve each subproblem.
19 20
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Rod cutting problem

Reconstructing a solution
Algorithm 5 EXTENDED-BOTTOM-UP-CUT-ROD(p, n)
1: let r[0 .. n] be a new array
2: r[0] = 0
3: for j = 1 to n do
Our dynamic-programming solutions to the rod-cutting problem 4: q = −∞
return the value of an optimal solution. 5: for i = 1 to j do
6: if q < p[i] + r[j-i] then
They do not return an actual solution: a list of piece sizes. 7: q = p[i] + r[j-i]
8: s[j] = i
We can record not only the optimal value computed for each
9: end if
subproblem, but also a choice that led to the optimal value.
10: end for
11: r[j] = q
12: end for
13: return r and s

21 22
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Rod cutting problem Elements of dynamic programming

Elements of dynamic programming

In our rod-cutting example, the call

EXTENDED-BOTTOM-UP-CUT-ROD(p, 10) would return the
When should we look for a dynamic-programming solution to a
following arrays:
problem?

i 0 1 2 3 4 5 6 7 8 9 10 The two key ingredients:

r[i] 0 1 5 8 10 13 17 18 22 25 30
s[i] 0 1 2 3 2 2 6 1 2 3 10 Optimal substructure

Overlapping subproblems

23 24
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of dynamic programming Elements of dynamic programming

Optimal substructure

Show that the solutions to the subproblems used within the

A problem exhibits optimal substructure if an optimal solution to optimal solution must themselves be optimal.
the problem contains within it optimal solutions to subproblems.
Suppose that one of the subproblem solutions is not optimal.
Show that a solution to a problem consists of making a Cut it out.
choice, which leaves one or subproblems to solve.
Paste in an optimal solution.
Suppose that you are given this last choice that leads to an Get a better solution to the original problem.
optimal solution.

Given this choice, determine which subproblems arise and how That was optimal substructure procedure.
to characterize the resulting space of subproblems.

25 26
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of dynamic programming Elements of dynamic programming

Optimal substructure varies across problem domains:

How to characterize the space of subproblems?

1 How many subproblems are used in an optimal solution.

2 How many choices in determining which subproblem(s) to use.

Keep the space as simple as possible.

Expand it as necessary. Informally, running time depends on (# of subproblems overall) ×

(# of choices).

Rod-cutting problem: Θ(n) subproblems overall, and at most

n choices to examine for each ⇒ O(n2 ) running time.

27 28
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of dynamic programming Elements of dynamic programming

Subtleties

Dynamic programming uses optimal substructure bottom up.

Be careful not to assume that optimal substructure applies when it
Find optimal solutions to subproblems.
does not.
Choose which to use in optimal solution to the problem.
Consider the following two problems in which we are given a
directed graph G = (V, E) and vertices u, v ∈ V.
Greedy algorithms work top down:

First make a choice that looks best, then solve the resulting V is a set of vertices.
subproblem.
E is a set of edges.

29 30
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of dynamic programming Elements of dynamic programming

Let’s find a path (sequence of connected edges) from vertex u to

vertex v.

Shortest path: find path u v with fewest edges. Must be Suppose p is shortest path u v.
simple (no cycles). Let w be any vertex on p.

Longest simple path: find simple path u v with most Let p1 be the portion of p, u w.
edges. Could repeatedly traverse a cycle to make an arbitrarily Then p1 is a shortest u w.
long path.
Same argument applies to p2 .

Therefore, we can find shortest path u v by considering all

intermediate vertices w, then fnding shortest paths u w and w
v (optimal substructure).
31 32
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of dynamic programming Elements of dynamic programming

Does longest path have optimal substructure?

Not only isn’t there optimal substructure, but we can’t even

assemble a legal solution from solutions to subproblems.

Combine longest simple paths:

Consider q → r → t = longest path q t. Are its subpaths q→s→t→r→q→s→t

longest paths?

NO! In fact, this problem is NP-complete (so it probably has no optimal

substructure to find).
Subpath q r is q → r.
Longest simple path q r is q → s → t → r.
Subpath r t is r → t.
Longest simple path r t is r → q → s → t.
33 34
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of dynamic programming Elements of dynamic programming

Overlapping subproblems
Alternative example: memoization

These occur when a recursive algorithm revisits the same problem ”Store, don’t recompute”.
over and over.
Make a table indexed by subproblem.
Good divide-and-conquer algorithms usually generate a brand new
problem at each stage of recursion. When solving a subproblem:

Example: Merge sort Lookup in table.

If answer is there, use it.
Else, compute answer, then store it.

In dynamic programming, we go one step further. We

determine in what order we’d want to access the table, and fill
it in that way.

35 36
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Longest common subsequence Longest common subsequence

Longest common subsequence Example 1:

Problem: Given 2 sequences, X = hx1 , ..., xm i and Y = hy1 , ..., yn i.

Find a subsequence common to both whose length is longest.

A subsequence doesn’t have to be consecutive, but it has to

be in order.

Biological applications =⇒ To compare the DNA of two (or Example 2: For X = hA, B, C , B, D, A, Bi and
more) different organisms. Y = hB, D, C , A, B, Ai.

The sequence hB, C , Ai is a CS.

The sequences hB, C , B, Ai and hB, D, A, Bi are LCS.

37 38
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Longest common subsequence Longest common subsequence

Brute-force approach Optimal substructure

Notation: ith prefix
Xi = prefix hx1 , ..., xi i

For every subsequence of X, check whether it’s a subsequence of Y. Yi = prefix hy1 , ..., yi i

Time: Θ(n2m ) For example, if X = hA, B, C, B, D, A, Bi, then X4 =

hA, B, C, Bi and X0 is the empty sequence.
2m subsequences of X to check.
Theorem:
Each subsequence takes Θ(n) time to check: scan Y for first Let Z = hz1 , ..., zk i be any LCS of X and Y.
letter, from there scan for second, and so on.
1 If xm = yn , then zk = xm = yn and Zk−1 is an LCS of Xm−1 and
Yn−1 .
2 If xm 6= yn , then zk 6= xm ⇒ Z is an LCS of Xm−1 and Y.

39
3 If xm 6= yn , then zk 6= yn ⇒ Z is an LCS of X and Yn−1 . 40
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Longest common subsequence Longest common subsequence

A recursive solution Compute length of optimal solution

LCS problem involves establishing a recurrence for the value of an

optimal solution. Procedure LCS-LENGTH takes two sequences X = hx1 , ..., xm i and Y
= hy1 , ..., yn i as inputs.
c[i, j] to be the length of an LCS of the sequences Xi and
Yj . The LCS problem has only Θ(mn) distinct subproblems.

0,
 if i = 0 or j = 0, Store the c[i, j] values in a table c[0..m, 0..n], and
c[i,j] = c[i-1,j-1]+1, if i,j > 0 and xi = yj , computes the entries in row-major order.

max(c[i,j-1],c[i-1,j]), if i,j > 0 and xi 6= yj .


41 42
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Longest common subsequence Longest common subsequence

Algorithm 6 LCS-LENGTH(X, Y)
1: m = X.length
2: n = Y.length
3: let b[1..m, 1..n] and c[0..m, 0..n] be new tables
4: for i = 1 to m do
5: c[i, 0] = 0
6: end for
7: for j = 1 to n do
8: c[0, j] = 0
9: end for
10: for i = 1 to m do
11: for j = 1 to n do
12: if xi == yj then
13: c[i, j] = c[i-1, j-1] + 1
14: b[i, j] = ” - ”
15: else if c[i-1, j] ≥ c[i, j-1] then
16: c[i, j] = c[i-1, j]
17: b[i, j] = ” ↑ ” Figure: 4.4 The c and b tables computed by LCS-LENGTH on the
18: else c[i, j] = c[i, j-1] sequences X = hA, B, C, B, D, A, Bi and Y = hB, D, C, A, B, Ai.
19: b[i, j] = ” ← ”
20: end if
21: end for The running time of the procedure is Θ(mn), since each table entry
22: end for
23: return c and b 43
takes Θ(1) time to compute. 44
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Longest common subsequence Optimal binary search trees

Constructing an LCS Optimal binary search trees

The b table returned by LCS-LENGTH enables us to quickly
construct an LCS of X = hx1 , ..., xm i and Y = hy1 , ..., yn i.
Suppose that we are designing a program to translate text from
Algorithm 7 PRINT-LCS(b, X, i, j English to Amharic.
1: if i == i or j==0 then
2: return
3: end if We could perform these lookup operations by building a
4: if b[i, j] == ” - ” then binary search tree with n English words as keys and their
5: PRINT-LCS(b, X, i-1, j-1) Amharic equivalents as satellite data.
6: print xi
7: else if b[i, j] == ” ↑ ” then Because we will search the tree for each individual word in the
8: PRINT-LCS(b, X, i-1, j) text, we want the total time spent searching to be as low as
9: else possible.
10: PRINT-LCS(b, X, i, j-1)
11: end if
The procedure takes time O(m + n), since it decrements at least
one of i and j in each recursive call. 45 46
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Optimal binary search trees Optimal binary search trees

Given sequence K = hk1 , k2 , ..., kn i of n distinct keys, sorted

(k1 < k2 < · · · < kn ).
We want words that occur frequently in the text to be placed Want to build a binary search tree from the keys.
nearer the root.
For ki , have probability pi that a search is for ki .
Moreover, some words in the text might have no Amharic
For each dummy key di , we have a probability qi that a
translation, and such words would not appear in the binary
search will correspond to di .
search tree at all.
Each key ki is an internal node, and each dummy key di is a
Minimize the number of nodes visited in all searches, given leaf.
that we know how often each word occurs?
Want best search time (BST) with minimum expected search
cost.

Actual cost = # of items examined.

47 48
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Optimal binary search trees Optimal binary search trees

The probablity table

i 0 1 2 3 4 5
pi 0.15 0.10 0.05 0.10 0.20
qi 0.05 0.10 0.05 0.05 0.05 0.10

Every search is either successful (finding some key ki ) or

unsuccessful (finding some dummy key di ), and so we have

n
X n
X
pi + qi = 1
Figure: 4.5 Two binary search trees for a set of n = 5 keys. i=1 i=0

49 50
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Optimal binary search trees Optimal binary search trees

From figure 4.5, we can calculate the expected search cost node by
node:

Let us assume that the actual cost of a search equals the number For the first binary search tree
of nodes examined, i.e., the depth of the node found by the search node depth probablity contribution
in T, plus 1. k1 1 0.15 0.30
k2 0 0.10 0.10
Then the expected cost of a search in T is:
k3 2 0.05 0.15
n
X n
X k4 1 0.10 0.20
E[search cost in T] = (depthT (ki ) + 1) · pi + (depthT (di ) + 1) · qi k5 2 0.20 0.60
i=1 i=0
n n d0 2 0.05 0.15
X X
=1+ depthT (ki ) · pi + depthT (di ) · qi d1 2 0.10 0.30
i=1 i=0 d2 3 0.05 0.20
d3 3 0.05 0.20
d4 3 0.05 0.20
d5 3 0.10 0.40
Total 2.80
51 52
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Optimal binary search trees Optimal binary search trees

Use optimal substructure (Recursive procedure) to construct an

Following a similar procedure the expected cost of the second optimal solution to the problem from optimal solutions to
figure ⇒ 2.75. subproblems:

Given keys ki , ..., kj and dummy keys di−1 , ..., dj . (the

Observations: problem).
Optimal BST might not have smallest height.
One of them, kr , where i ≤ r ≤ j, must be the root.
Optimal BST might not have highest-probability key at root.
Left subtree of kr contains keys ki , ..., kr−1 (and dummy keys
The lowest expected cost of any binary search tree with k5 at di−1 , ..., dr−1 ).
the root is 2.85.
Right subtree of kr contains kr+1 , ..., kj (and dummy keys
dr , ..., dj ).

53 54
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Optimal binary search trees

Greedy Algorithms

If, we examine all candidate roots kr , for i ≤ r ≤ j, and

Similar to dynamic programming.
We determine all optimal BSTs containing ki , ..., kr−1 and
containing kr+1 , ..., kj , Used for optimization problems.

Then we’re guaranteed to find an optimal BST for ki , ..., kj . Idea: When we have a choice to make, make the one that
looks best right now.

Computing of the optimal solution using dynamic programming Make a locally optimal choice in hope of getting a globally
requires to store intermediate solutions which can be used to solve optimal solution.
the bigger subproblems.
Greedy algorithms don’t always yield an optimal solution.
But sometimes they do.

55 56
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

An activity-selection problem
Note: Could have many other objectives:
Suppose we have a set S = {a1 , a2 , ..., an } of n activities require
exclusive use of a common resource. Schedule room for longest time.

For example, scheduling the use of a classroom. Maximize income rental fees.

ai needs resource during period [si , fi ), which is a half-open For example, consider the following set S of activities for the
interval, where si = start time and fi = finish time. activity selection problem:

i 1 2 3 4 5 6 7 8 9
Goal: Select the largest possible set of nonoverlapping (mutually
si 1 2 4 1 5 8 9 11 13
compatible) activities.
fi 3 5 7 8 9 10 11 14 16
We assume that the activities are sorted in monotonically
increasing order of finish time.

57 58
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

We shall solve this problem in several steps.

Dynamic-programming solution, consider several choices when

determining which subproblems to use in an optimal solution.

Consider only one choice-the greedy choice-and that when we

make the greedy choice, only one subproblem remains.

Maximum-size mutually compatible set: {a1 , a3 , a6 , a8 }.

Not unique: also {a2 , a5 , a7 , a9 }.

59 60
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

The optimal substructure

Let Aik = Aij ∩ Sik and Akj = Aij ∩ Skj .

Sij = {ak ∈ S : fi ≤ sk < fk ≤ sj } Aik contains the activities in Aij that finish before ak starts
and Akj contains the activities in Aij that start after ak
activities that start after ai finishes and finish before aj starts. finishes.

Activities in Sij are compatible with Thus,

all activities that finish by fi , and Aij = Aik ∪ {ak } ∪ Akj
all activities that start no earlier than sj . and so the maximum-size set Aij of mutually compatible activities
in Sij consists of |Aij | = |Aik | + |Akj | + 1 activities.
By including ak in an optimal solution, we are left with two
subproblems: Thus, we might solve the activity-selection problem by
Activities, Sik and Skj . dynamic programming.

61 62
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

Making the greedy choice

If we denote the size of an optimal solution for the set Sij by What if we could choose an activity to add to our optimal solution
c[i, j], then we would have the recurrence
without having to first solve all the subproblems?
c[i, j] = c[i, k] + c[k, j] + 1.
That could save us from having to consider all the choices
inherent in recurrence.
If we did not know that an optimal solution for the set Sij
includes activity ak , we would have to examine all activities in In fact, for the activity-selection problem, we need consider
Sij to find which one to choose, so that only one choice: the greedy choice.

0 if Sij = ∅
c[i, j] = Now we can solve the problem Sij top down,
 max {c[i, k] + c[k, j] + 1}, if Sij 6= ∅
ak ∈Sij
Choose am ∈ Sij with earliest finish time: the greedy choice.

Then solve Smj .

63 64
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

A recursive greedy algorithm

What are the subproblems?
The procedure RECURSIVE-ACTIVITY-SELECTOR takes:
Original problem is S0,n+1 , suppose our first choice is am1 .
The start and finish times of the activities, represented as
Then next subproblem is Sm1 ,n+1 , suppose next choice is am2 . arrays s and f,
Next subproblem is Sm2 ,n+1 , and so on.
Index k that defines the subproblem Sk to be solved and,

Each subproblem is Smi ,n+1 , i.e., the last activities to finish. And Size n of the original problem.
the subproblems chosen have finish times that increase.
It returns a maximum-size set of mutually compatible
Therefore, we can consider each activity just once, in activities in Sk .
monotonically increasing order of finish time.
Assumes activities are already sorted by monotonically increasing
finish time. (If not, then sort in O(n lg n) time.)

65 66
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

The initial call, which solves the entire problem, is

RECURSIVE-ACTIVITY- SELECTOR(s, f, 0, n).

Algorithm 8 RECURSIVE-ACTIVITY-SELECTOR(s, f, k, n)
1: m = k + 1 Example: Consider the following set S of activities:
2: while m ≤ n and s[m] < f[k] do . find the first activity in Sk
to finish i 1 2 3 4 5 6 7 8 9 10 11
3: m=m+1
si 1 3 0 5 3 5 6 8 8 2 12
4: end while
5: if m ≤ n then
fi 4 5 6 7 9 9 10 11 12 14 16
6: return {am } ∪ RECURSIVE-ACTIVITY-SELECTOR(s, f, m, n)
7: else return ∅
8: end if

Assuming that the activities have already been sorted by finish

times, the running time of the call
RECURSIVE-ACTIVITY-SELECTOR(s, f, k, n) is Θ(n).
67 68
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

An activity-selection problem An activity-selection problem

An iterative greedy algorithm

Algorithm 9 GREEDY-ACTIVITY-SELECTOR(s, f)
1: n = s.length
2: A = {a1 }
3: k=1
4: for m = 2 to n do
5: if s[m] ≥ f[k] then
6: A = A ∪ {am }
7: k=m
8: end if
9: end for
10: return A

Like the recursive version, GREEDY-ACTIVITY-SELECTOR

Figure: 4.6 The operation of RECURSIVE-ACTIVITY-SELECTOR on the 11 schedules a set of n activities in Θ(n) time.
activities given above. 69 70
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of the greedy strategy Elements of the greedy strategy

Elements of the greedy strategy

Generally, we design greedy algorithms according to the following

How can we tell whether a greedy algorithm will solve a particular
sequence of steps:
optimization problem?
1 Cast the optimization problem as one in which we make a
choice and are left with one subproblem to solve. Greedy-choice property

2 Prove that there is always an optimal solution to the original Optimal substructure
problem that makes the greedy choice, so that the greedy
choice is always safe.

3 Demonstrate optimal substructure.

71 72
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of the greedy strategy Elements of the greedy strategy

Greedy-choice property Greedy vs. dynamic programming

A globally optimal solution can be arrived at by making a locally 0-1 knapsack problem:
optimal (greedy) choice.
n items.
Greedy:
Item i is worth vi birr, weighs wi pounds.
Make a choice at each step. Find a most valuable subset of items with total weight ≤ W.
Make the choice before solving the subproblems.
Have to either take an item or not take it-can’t take part of it.
Solve top-down.

Fractional knapsack problem:

Optimal substructure
Like the 0-1 knapsack problem, but can take fraction of an
Just show that optimal solution to subproblem and greedy choice
item.
⇒ optimal solution to problem.
73 74
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Elements of the greedy strategy Elements of the greedy strategy

Both have optimal substructure.

The fractional knapsack problem has the greedy-choice property.

To solve the fractional problem, rank items by

value/weight: vi /wi .

Sorting the items by value per pound, the greedy algorithm

runs in O(n lg n) time.

Greedy doesn’t work for the 0-1 knapsack problem. Might get
empty space, which lowers the average value per pound of the Figure: 4.7 An example showing that the greedy strategy does not work
items taken. for the 0-1 knapsack problem.

75 76
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Huffman Codes Huffman Codes

Huffman Codes
Represent via a binary character code ⇒ unique binary string
Huffman codes compress data very effectively. (codeword).
Savings of 20% to 90% are typical, depending on the
1 Fixed-length code:
characteristics of the data being compressed.
3 bits to represent 6 characters =⇒ 300,000 bits to code
Based on how often each character occurs(i.e frequency).
the entire file.
Example: Suppose we have a 100,000 - character data file. 2 Variable-length code:
Frequent characters =⇒ short codewords.
a b c d d f
Frequency (in thousands) 45 13 12 16 9 5 From Table 4.1 the code requires
Fixed-length codeword 000 001 010 011 100 101 (45· 1 + 1· 3 + 12·3 + 16· 3 + 19·4 + 5·4)·1000 = 224,000
Variable-length codeword 0 101 100 111 1101 1100 bits
to represent the file, a savings of approximately 25%.
Table: 4.2 A Character-coding problem.
77 78
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Huffman Codes Huffman Codes

Prefix Codes

Achieves the optimal data compression among any character code.

A binary tree whose leaves are the given characters provides one
Encoding a binary character code: Concatenate the code representation.
words.

For example: abc =⇒ 0 · 101 · 100 = 0101100. Binary codeword for a character =⇒ simple path from the
root to that character.
They simplify decoding.
0 means ”go to the left child” and 1 means ”go to the right
Decoding Procedure: child.”
Identify the initial codeword,
Translate it back to the original character, and
Repeat the decoding process on the remainder of the encoded
file.

79 80
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Huffman Codes Huffman Codes

Constructing a Huffman code

Greedy algorithm that constructs an optimal prefix code called a

Huffman code.

C is a set of n characters and that each character c ∈ C is an

object with an attribute c·freq giving its frequency.

Figure: 4.8 Trees corresponding to the coding schemes in Table 4.2. Builds the tree T corresponding to the optimal code in a
An optimal code for a file is always represented by a full bottom-up manner.
binary tree, in which every nonleaf node has two children.
The number of bits required to encode a file is thus Performs a sequence of |C| - 1 ”merging” operations to
X create the final tree.
B(T ) = c.freq · dT (c),
c∈C
which we define as the cost of the tree T. 81 82
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Huffman Codes Huffman Codes

Algorithm 10 HUFFMAN(C)
1: n = |C|
2: Q = C
3: for i = 1 to n-1 do
4: allocate a new node z
5: z.left = x = EXTRACT-MIN(Q)
6: z.right = y = EXTRACT-MIN(Q)
7: z.freq = x.freq + y.freq
8: INSERT(Q, z)
9: end for
10: return EXTRACT-MIN(Q) . return the root of the tree

Figure: 4.9 The steps of Huffmans algorithm for the frequencies given in
83 Figure 4.2. 84
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Huffman Codes

Amortized Analysis
To analyze the running time of Huffman’s algorithm:

Assume that Q is implemented as a binary min-heap. Analyze a sequence of operations on a data structure.
For a set C of n characters, we can initialize Q in line 2 in Goal: Show that although some individual operations may be
O(n) time using the BUILD-MIN-HEAP. expensive, on average the cost per operation is small.

The for loop in lines 3-8 executes exactly n - 1 times, and

Average in this context does not mean that we’re averaging over a
since each heap operation requires time O(lg n), the loop
distribution of inputs.
contributes O(n lg n) to the running time.
No probability is involved.
Thus, the total running time of HUFFMAN on a set of n characters We’re talking about average cost in the worst case.
is O(n lg n).

85 86
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Aggregate analysis Aggregate analysis

Aggregate analysis Stack operations

PUSH(S, x): O(1) each ⇒ O(n) for any sequence of n

operations.

POP(S): O(1) each ⇒ O(n) for any sequence of n

In aggregate analysis, we show that for all n, a sequence of n operations.
operations takes worst-case time T(n) in total.
Now let’s add the stack operation MULTIPOP(S, k), which
In the worst case, the average cost, or amortized cost, per removes the k top objects of stack S.
operation is therefore T(n)/n.
Algorithm 11 MULTIPOP(S, k)
1: while not STACK-EMPTY(S) and k ¿ 0 do
2: POP(S)
3: k = k-1
4: end while

87 88
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Aggregate analysis Aggregate analysis

Considering a sequence of n PUSH, POP, MULTIPOP operations:

Worst-case cost of MULTIPOP is O(n), since the stack size is

at most n.

Figure: 4.10 The action of MULTIPOP on a stack S. The top 4 objects Have n operations.
are popped by MULTIPOP(S, 4), whose result is shown in (b). The next
operation is MULTIPOP(S, 7)-shown in (c). Therefore, worst-case cost of sequence is O(n2 ).

Running time of MULTIPOP: Although this analysis is correct, the worst-case cost of each
operation individually, is not tight.
Linear in # of POP operations.
# of iterations of while loop is min(s, k), where s = # of
objects on stack.
Therefore, total cost = min(s, k). 89 90
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Aggregate analysis Aggregate analysis

Observation Incrementing a binary counter

Each object can be popped only once per time that it’s
pushed.
Have ≤ n PUSHes ⇒ ≤ n POPs, including those in Consider the problem of implementing a k-bit binary counter
MULTIPOP. A[0..k - 1] of bits, where A[0] is the least significant bit and
A[k - 1] is the most significant bit.
Therefore, total cost = O(n).
Average over the n operations ⇒ O(1) per operation on Counts upward from 0.
average. k-1
A[i] · 2i .
P
Value of counter is
i=0
Again, notice no probability.
Initially, counter value is 0, so A[0..k - 1] = 0.
Showed worst-case O(n) cost for sequence.
Therefore, O(1) per operation on average.
This technique is called aggregate analysis. 91 92
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Aggregate analysis Aggregate analysis

Algorithm 12 INCREMENT(A)
1: i=0
2: while i < A.length and A[i]==1 do
3: A[i]=0
4: i = i+1
5: end while
6: if i < A.length then
7: A[i]=1
8: end if
Figure: 4.11 An 8-bit binary counter as its value goes from 0 to 16 by a
sequence of 16 INCREMENT operations.
Cost of INCREMENT = Θ(# of bits flipped).
Analysis: Each call could flip k bits, so n INCREMENTs takes
93 O(nk) time. 94
Outline Dynamic Programming Greedy Algorithms Amortized
Outline Analysis
Dynamic Programming Greedy Algorithms

Aggregate analysis Aggregate analysis

Observation
Not every bit flips every time.
bit flips how often times in n INCREMENTs
0 every time n
1 1/2 the time bn/2c End of Chapter 4
2 1/4 the time bn/4c
.
.
i 1/2i the time bn/2i c
.
Questions?
.
i≥ k never 0
k-1 ∞
i
b1/2i c = 2n.
P P
Therefore, total # of flips = bn/2 c < n
i=0 i=0
As a result, n INCREMENTs costs O(n).
Average cost per operation = O(1). 95 96