Design and Analysis of Algorithm

Tanveer Ahmed Siddiqui

Department of Computer Science

COMSATS University, Islamabad
 Recap and Today Covered

Algorithm Design and Analysis Process

Understand the problem

Decide on : algorithm
design techniques etc.

Design an algorithm

Prove correctness

Analyze efficiency etc

Code the algorithm

Department of Computer Science
 Reading Material

Read Chapter 8
Dynamic Programming

Department of Computer Science

 Objectives

 How to Design Algorithm using Dynamic

Programming

Department of Computer Science

New designing technique build from the idea of divide and conquer

 Perspective #1: Divide & Conquer with Memory

Table
 Solve the sub-problems one by one, smallest first,
possibly by recursion.
 Store the solutions to sub-problems in a table and
 Reuse the solutions of subproblem from table to
solve larger sub-problems
 Repeat above steps until the top (original)
problem instance is solved.

Department of Computer Science

 Procedure

 Characterize the Optimal Substructure Property for the

given optimization problem.
 Objective Function: What should be optimize
 Devise a recursive formula for the objective function.
 Compute the value of an optimal solution in a bottom-up
fashion.
 replace its recursive calls with table lookups. That is
instead of returning a value, record it in a table entry
 We devise “look-up template”

 Use base case of divide-and-conquer to fill in

start of table
 devise for-loops that fill the table using “look-
up template”
Department of Computer Science
Chain Matrix Multiplication: Motivation

Problem: Given a sequence

of matrices, find the most
efficient way to multiply
these matrices together.

A1 = Translation along x axis

A1 · (A2 . (A3 . A4)))
A2 = Translation along y axis
(A1 · ((A2 . A3). A4))
A3 = Rotation
((A1 · A2). (A3 . A4))
A4 = Scaling
((A1 · (A2 . A3)). A4)
(((A1 · A2). A3). A4)
Statement: The chain-matrix multiplication problem can be stated as
below:
Given a chain of [A1, A2, . . . , An] of n matrices where for i = 1,
2, . . . , n, matrix Ai has dimension pi-1 x pi, find the order of
multiplication which
Department of Computer Scienceminimizes the number of scalar multiplications.
Chain Matrix Multiplication: Motivation
 Problem: Given a chain A1, A2, …, An of matrices. The dimension of
Ai is Pi-1 X Pi. How do we fully add parenthesis to the chain so that
the total number of scalar multiplication is minimized?

 Brute force: Try all possible combinations

A 1 A2 A 3 … … An
=> ( (A1 … Ak) (Ak+1… An) ) (Split at k)

Number of possibilities
to parenthesize n 1 if n  1
matrices 
P (=>
n)   n 1
 P ( k ) P ( n  k ) if n  2
 k 1
=> P( n )  C ( n  1)  1  2( n  1)   ( 4 )
n

n  n  1  n 3/ 2
C(n) - Catalan number
Department of Computer Science 8
Chain Matrix Multiplication: Motivation

Problem: Given a sequence

of matrices, find the most
efficient way to multiply
these matrices together.

A1 = Translation along x axis

A1 · (A2 . (A3 . A4)))
A2 = Translation along y axis
(A1 · ((A2 . A3). A4))
A3 = Rotation
((A1 · A2). (A3 . A4))
A4 = Scaling
((A1 · (A2 . A3)). A4)
(((A1 · A2). A3). A4) The problem is not actually to perform
the multiplications, but merely to
In order to find best decide in which order to perform the
multiplication order, what multiplications.
should we minimize?
Department of Computer Science
Divide and Conquer: Recursive Algorithm

Recurrence Relation: After kth matrix, create two

sub-lists, one with k and other with n - k
matrices i.e.
(A1 A2A3A4A5 . . . Ak) (Ak+1Ak+2…An)

Department of Computer Science

Divide and Conquer

 Let cost(i, j) be the minimum cost of computing

 A1 A2A3A4A5 . . . Ak.Ak+1Ak+2…An

 What is the cost of breaking the product at Ak?

(A1 A2A3A4A5 . . . Ak) (Ak+1Ak+2…An)
 It is cost(i, k) plus cost(k + 1, j) plus the cost of
multiplying an ri−1×rk matrix by an rk×rj matrix.
 Therefore, the cost of breaking the product at Ak
is cost(i, k) + cost(k + 1, j) + ri−1rkrj
 What is the base case?
 Observe that if i = j then the sequence contains only
one matrix, and so the cost is 0. (There is nothing to
Department of Computer Science
 Procedure

 Characterize the Optimal Substructure Property for the

given optimization problem.
 Devise a recursive formula for the objective
function.
Cost (i, j) = 0 i=j
Cost (i, j) = cost(i, k) + cost(k + 1, j) + ri−1rkrj otherwise

Department of Computer Science

 A simple recursive (top-down) solution
using the formula for m[i,j] we could solve the problem:
RECURSIVE-MATRIX-CHAIN(p, i, j)
1. if i=j then return 0
2. m[i, j] = 
3. for k ← 1 to j − 1
4. q ← RECURSIVE-MATRIX-CHAIN (p, i , k)
5. + RECURSIVE-MATRIX-CHAIN (p, k+1 ,
j)
6. + p[i-1] p[k] p[j]
7. if q < m[i, j] then m[i, j] ← q
T (1)  1
8. return m[i, j]
Complexity: n 1
T (n)  1   (T (k )  T (n  k )  1) for n > 1
k 1
Department of Computer Science 13
 Running Time of The Algorithm
n 1

T(n) has lower bound T ( n)  1   (T (k )  T (n  k )  1)

k 1
of 2n, too slow
n 1
 2 T (i )  n
i 1
n 1
 2 2i 1  n
i 1
n2
 2 2 i  n
i 0

 2( 2 n 1  1)  n
 2n  2  n
 2n

Department of Computer Science

Divide and Conquer

The problem is, the algorithm solves

the same sub problems over and over
again!
Department of Computer Science
 Complexity of recursive solution
n 1 n 1
T ( n )  1   (T ( k )  T ( n  k )  1)  n  2  T (i )
k 1 i 1
 We will use the substitution method – we guess a solution and
then prove by using mathematical induction that it is correct.
 We will prove that T(n) = (2n). Specifically, that T(n) ≥ 2n-1 for
all n ≥1.
 Induction Base: T(1) ≥1=20
 Induction Assumption: assume T(k) ≥ 2k-1 for all 1 ≤ k < n
 Induction Step:
n 1 n2
T (n)  n  2 2i 1  n  2 2i 
i 1 i 0
n 1 n 1
 n  2(2  1)  n  2  2  2
n

Department of Computer Science 16

 Memoization
 A top-down variation of dynamic programming
 Idea: remember the solution to subproblems as they are
solved in the simple recursive algorithm
Initialize all m elements to  ; call Lookup-Chain(p, i, j)
LOOKUP-CHAIN(p,i,j) 3
Takes O(n ) time
if m[i,j] <  then return m[i,j]
if i =j then m[i,j]  0 Requires (n2) space
else
for k ← 1 to j − 1
q ← LOOKUP-CHAIN (p, i , k)
+ LOOKUP-CHAIN (p, k+1 , j) + p[i-1] p[k] p[j]
if q < m[i, j] then m[i, j] ← q
return m[i, j]
 The memoized algorithm has the advantage of solving
only those problems that are needed
 Dynamic programming is considered better when all
subproblems must be calculated, because there is no
overhead
Department for recursion
of Computer Science 17
 Procedure

 Compute the value of an optimal solution in a bottom-

up fashion.
 replace its recursive calls with table lookups. That is
instead of returning a value, record it in a table
entry.

 We devise “look-up template”


0 if i  j
minm[i, k ]  m[k  1, j ]  pi 1 pk p j  if i  j
m[i, j ]  

 ik  j

Department of Computer Science

 Procedure

 Compute the value of an optimal solution in a bottom-

up fashion.
 We devise “look-up template”

m[1,1] m[1,2] m[1,3] m[1,4]

m[2,2] m[2,3] m[2,4]
m[3,3] m[3,4]
m[4,4]

Department of Computer Science

 Procedure

 We devise “look-up template”

 Use base case of divide-and-conquer to fill
in start of table.
1
2
3
4

 devise for-loops that fill the table using

“look-up template”
Department of Computer Science
 Procedure

 We devise “look-up template”

 Use base case of divide-and-conquer to fill
in start of table.

Department of Computer Science

 Procedure

 We devise “look-up template”

 devise for-loops that fill the table using
“look-up template”.
 Compute entries of m[] in increasing order

of difference between parameters(j-i).

1 5 8 10
2 6 9
3 7
4

Department of Computer Science

Filling in the First Super diagonal

Department of Computer Science

Filling in the Second Super diagonal

Department of Computer Science

Filling in the dth Super diagonal

Department of Computer Science

Filling in the dth Super diagonal

Analysis:

Department of Computer Science

 Step 3: Compute the value of an optimal
solution buttom-up
Input: n ; an array p[0…n] containing matrix dimensions
State: m[1..n, 1..n] for storing m[i, j]
s[1..n, 1..n] for storing the optimal k that was used to calculate m[i, j]
Result: Minimum-cost table m and split table s
Cost of chains of length 1
MATRIX-CHAIN-ORDER(p, n)
for i ← 1 to n
m[i, i] ← 0 Use recursive formula to
for l ← 2 to n compute the cost of chains of
for i ← 1 to n-l+1 length l ≥2
j ← i+l-1
m[i, j] ← 
for k ← i to j-1
q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]
if q < m[i, j]
m[i, j] ← q Takes O(n 3
) time
s[i, j] ← k Requires (n2) space
return m and s
Department of Computer Science 27
Computational Cost
n n n n i
T ( n)  n    ( j  i)   k
i 1 j i 1 i 1 k 1

(n  i )(n  i  1)
n
T ( n)  n  
i 1 2
1 n 2
T (n)  n   (n  2ni  i 2  n  i )
2 i 1
1 n 2 n n n n
T (n)  n  ( n   2ni   i 2   n   i )
2 i 1 i 1 i 1 i 1 i 1

Department of Computer Science

Computational Cost

1 n 2 n n n n
T (n)  n  ( n   2ni   i 2   n   i )
2 i 1 i 1 i 1 i 1 i 1

1 2 n n n n n
T ( n)  n  ( n  1  2n  i   i 2  n  1   i )
2 i 1 i 1 i 1 i 1 i 1

1 2 n(n  1) n(n  1)(2n  1) n(n  1)

T (n)  n  (n .n  2n.   n.n  )
2 2 6 2
Department of Computer Science
Computational Cost

1 2 n(n  1) n(n  1)(2n  1) n(n  1)

T (n)  n  (n .n  2n.   n.n  )
2 2 6 2

1 3 n(n  1)(2n  1) n(n  1)

T (n)  n  (n  n (n  1) 
2
n 
2
)
2 6 2

1
T (n)  n  (6n 3  6n 3  6n 2  2n 3  3n 2  n  6n 2  3n 2  3n)
12

1 1 1
T (n)  (12n  2n  2n)  (10n  2n )  (5n  n 3 )
3 3

12 12 6
Department of Computer Science
 Obtaining the optimal multiplication order

j
i 2 3 4 A1 30×1
1 1 1 1
A2 1×40
2 2 3 A3 40×10

3 3 A4 10×25
PRINT(s, 1, 4)
PRINT (s, 1, 1) PRINT (s, 2, 4)

PRINT (s, 2, 3) PRINT (s, 4, 4)

Output: (A1((A2A3)A4))

PRINT (s, 2, 2) PRINT (s, 3, 3)

Department of Computer Science 31

 Constructing an optimal solution

 Each entry s[i, j ]=k shows where to split the product Ai

Ai+1 … Aj for the minimum cost:
A1 … An = ( (A1 … As[i, n]) (As[i, n]+1… An) )
 To print the solution invoke the following function with ( s,
1, n) as the parameter:
PRINT-OPTIMAL-PARENS(s, i, j)
1. if i=j then print “A”i
2. else print “(”
3. PRINT-OPTIMAL-PARENS(s, i, s[i, j])
4. PRINT-OPTIMAL-PARENS(s, s[i, j]+1, j)
5. print “)”

Department of Computer Science 32

Example: Dynamic Programming
 Problem: Compute optimal multiplication order
for a series of matrices given below
A1 A2 A3 A4
. . .
10  100 100  5 5  50 50  20

P0 = 10 m[1,1] m[1,2] m[1,3] m[1,4]

P1 = 100 m[2,2] m[2,3] m[2,4]
P2 = 5 m[3,3] m[3,4]
P3 = 50 m[4,4]
P4 = 20
Department of Computer Science
Main Diagonal

m[i, i ]  0, i  1,...,4
m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )
ik  j

Main Diagonal 0
 m[1, 1] = 0 0
 m[2, 2] = 0 0
 m[3, 3] = 0 0
 m[4, 4] = 0

Department of Computer Science

Computing m[1, 2], m[2, 3], m[3, 4]

m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )

ik  j

m[1,2]  min (m[1, k ]  m[k  1,2]  p0 . pk . p2 )

1 k  2

m[1,2]  min (m[1,1]  m[2,2]  p0 . p1. p2 )

0 5000
m[1, 2] = 0 + 0 + 10 . 100 . 5
0
= 5000
0
s[1, 2] = k = 1
Department of Computer Science
0
Computing m[2, 3]

m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )

ik  j

m[2,3]  min (m[2, k ]  m[k  1,3]  p1. pk . p3 )

2 k 3

m[2,3]  min (m[2,2]  m[3,3]  p1. p2 . p3)

0 5000
m[2, 3] = 0 + 0 + 100 . 5 . 50
0 25000
= 25000
0
s[2, 3] = k = 2
Department of Computer Science
0
Computing m[3, 4]

m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )

ik  j

m[3,4]  min (m[3, k ]  m[k  1,4]  p2 . pk . p4 )

3 k  4

m[3,4]  min (m[3,3]  m[4,4]  p2 . p3 . p4 )

0 5000
m[3, 4] = 0 + 0 + 5 . 50 . 20
0 25000
= 5000
0 5000
s[3, 4] = k = 3
Department of Computer Science
0
Computing m[1, 3], m[2, 4]

m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )

ik  j

m[1,3]  min (m[1, k ]  m[k  1,3]  p0 . pk . p3 )

1 k 3

m[1,3]  min (m[1,1]  m[2,3]  p0 . p1. p3 ,

m[1,2]  m[3,3]  p0 . p2 . p3 ))
m[1, 3] = min(0+25000+10.100.50, 5000+0+10.5.50)
= min(75000, 2500) = 2500
s[1, 3] = k = 2
Department of Computer Science
Computing m[2, 4]

m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )

ik  j

m[2,4]  min (m[2, k ]  m[k  1,4]  p1. pk . p4 )

2 k  4

m[2,4]  min (m[2,2]  m[3,4]  p1. p2 . p4 ,

m[2,3]  m[4,4]  p1. p3 . p4 ))

m[2, 4] = min(0+5000+100.5.20, 25000+0+100.50.20)
= min(15000, 35000) = 15000
s[2, 4] = k = 2
Department of Computer Science
Computing m[1, 4]
m[i, j ]  min (m[i, k ]  m[k  1, j ]  pi 1. pk . p j )
ik  j

m[1,4]  min (m[1, k ]  m[k  1,4]  p0 . pk . p4 )

1 k  4

m[1,4]  min (m[1,1]  m[2,4]  p0 . p1. p4 ,

m[1,2]  m[3,4]  p0 . p2 . p4 , m[1,3]  m[4,4]  p0 . p3 . p4 )

m[1, 4] = min(0+15000+10.100.20, 5000+5000+
10.5.20, 2500+0+10.50.20)
= min(35000, 11000, 35000) = 11000
s[1, 4] = k = 2
Department of Computer Science
Final Cost Matrix and Its Order of Computation

 Final Cost Matrix 0 5000 2500 11000

0 25000 15000
0 5000
0

1 5 8 10
 Order of Computation 2 6 9
3 7
4

Department of Computer Science

K,s Values Leading Minimum m[i, j]

0 1 2 2
0 2 2
0 3
0
For, m(1, 4) 2
k=2
10x20
((A1 . A2) . (A3 . A4)) 1
3
10 x 5 5 x 20

A1 A2 A3 A4
10 x 100 100 x 5 5 x 50 50 x 20
Department of Computer Science
Representing Order using Binary Tree
 The above computation shows that the minimum
cost for multiplying those four matrices is 11000.
 The optimal order for multiplication is
((A1 . A2) . (A3 . A4))
2
For, m(1, 4) 10x20
k=2 1
3
10 x 5 5 x 20

A1 A2 A3 A4
10 x 100 100 x 5 5 x 50 50 x 20

Department of Computer Science

 Similar
Problems
Department of Computer Science
 Meaning full learning
 Some typical optimization problems like:
 Matrix Chain Multiplication
 Optimal Binary Search Tree
 Optimal polygon triangulation
 Even though, every problem has its own nature and
description; however after changing certain constraints,
they become equivalent to each other.

Department of Computer Science 45

 Similarity between MCM and OPT

 Optimal polygon triangulation

 A polygon can be divided intro triangles in a number
of ways
 If there is a weight
associated with
each triangle,
 Find the triangulation of minimum (maximum)
weight
 Any weight, e.g. length of perimeter, will do!
 This problem maps to matrix chain
multiplication

Department of Computer Science

 Mapping Problems to related ones
 Polygon triangulation / matrix chain
multiplication

Department of Computer Science

 Similarity between OBST and MCM

MCM OBST

Same optimal substructure.

Department of Computer Science 48

 Similarity between OBST and MCM

Recurrence relation of OBST

00 ifif jjii11

ee[[ii, ,jj]]
min {e[i, r  1]  e[r  1, j ]  w(i, j )} ififii jj
r  j {e[i , r  1]  e[ r  1, j ]  w(i , j )}
i min
 ir  j
Recurrence relation of OBST


0 if i  j
minm[i, k ]  m[k  1, j ]  pi 1 pk p j  if i  j
m[i, j ]  

 ik  j

Department of Computer Science 49

 Similarity between OBST and MCM

Filling Pattern of OBST Filling Pattern of MCM

Department of Computer Science 50

 Similarity between OBST and MCM

OBST

MCM
Department of Computer Science 51
 CONCLUSION

Department of Computer Science

 What we have learnt?

 Define a set of sub problems that can lead to the

solution of the original problem.
 Think: what information would make solving this
problem easier?
 Try to find a relation between the sub problems
you found above, and the current problem you're
trying to solve.
 Write the recursion solution of the original problem
from the solutions of sub problems.
 Build the solution lookup table in a bottom-up
fashion, until reaching the original problem.
 Initialize the lookup table.
 Fill the other terms through iteration by using the recurrence
relation.
Department of Computer Science 53
 What we have learnt?

 In a nutshell, dynamic programming is recursion

without repetition.
 Dynamic programming algorithms store the
solutions of intermediate subproblems, often but
not always in some kind of array or table.
 Many algorithms students make the mistake of
focusing on the table (because tables are easy and
familiar) instead of the much more important (and
difficult) task of finding a correct recurrence.
 As long as we memoize the correct recurrence, an explicit table
isn’t really necessary, but if the recursion is incorrect, nothing
works

Department of Computer Science 54