Matrix Chain Problem

Problem:
Given a sequence of matrices A1 A2…An , and dimensions p0 p1…pn
where Ai is of dimension pi-1 x pi , determine multiplication
sequence that minimizes the number of operations.

Note:
This algorithm does not perform the multiplication, it just figures out
the best order in which to perform the multiplication.
 Matrix Chain Multiplication
Matrix Chain Problem (Divide and Conquer)

A1..4

A1..1 A2..4 A1..2 A3..4 A1..3 A4..4

A2..2 A3..4 A2..3 A4..4 A1..1 A2..2 A3..3 A4..4 A1..1 A2..3 A1..2 A3..3

A3..3 A4..4 A2..2 A3..3 A2..2 A3..3 A1..1 A2..2

A divide-and-conquer approach usually generates brand new problems (i.e., subproblems are
independent) at each step of recursion. In solving problems with overlapping subproblems, A
Divide And Conquer algorithm does redundant work (i.e., Repeatedly solves common
subproblems)
Since subproblems overlap (as shown in the following tree), we don’t use recursion.
Matrix Chain Multiplication

O(n3) – Dynamic Programming

Dynamic Programing – General Technique

For dynamic programming to be applicable, an optimization problem

must have:

1. Overlapping Subproblems

2. Optimal substructure
Dynamic Programing – General Technique

Overlapping Subproblems

When a recursive algorithm revisits the same problem over and over again,
we say that the optimization problem has overlapping subproblems
(i.e., subproblems share subsubproblems).

The subproblems are not independent, but instead they overlap

(hence, should be constructed bottom-up).

In contrast , a divide-and-conquer approach is suitable usually generates

brand new problems (i.e., subproblems are independent) at each step of
recursion.
Dynamic Programing – General Technique

Optimal Substructure

A problem exhibits optimal substructure if an optimal solution to the

problem contains within it optimal solutions to subproblems.

Whenever a problem exhibits optimal substructure, it is a good clue that

dynamic programming might apply.

Dynamic programming uses optimal substructure in a bottom-up fashion.

Dynamic programming algorithms take advantage of overlapping subproblems by

solving each subproblem once and then storing the solution in a table where it
can be looked up when needed. Since subproblems overlap, we don’t use
recursion. Instead, we construct optimal subproblems “bottom-up.”
Dynamic Programing – Principle of Optimality

Optimization Problems

Principle of optimality
In an optimal sequence of decisions or choices, each subsequence must be
optimal
or
the optimal solution to the problem contains within it the optimal solution
to subproblems
 Matrix Chain Multiplication
Dynamic Programming (Dynamic Programming)

1. Establish a recursive property that gives the optimal

solution to an instance of the problem.

2. Compute the value of an optimal solution in a bottom-up

fashion.

3. Construct an optimal solution in a bottom-up fashion.

 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

Solve it Optimally Solve it Optimally

A1 A2 A3 A4 A5 - - - - - - An k=1 A1..1 A2..n

(Left Subproblem) (Right Subproblem)

We need to cut the chain A1,, An somewhere in the middle,

i.e., pick some k, where 1  k < n
All Possible Cuttings

A1 A2 A3 A4 A5 - - - - - - An
p0xp1 p1xp2 p2xp3 p3xp4 p4xp5 - - - - - Pn-1xpn
 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

Solve it Optimally Solve it Optimally

A1 A2 A3 A4 A5 - - - - - - An k=1 A1..2 A3..n

(Left Subproblem) (Right Subproblem)
A1 A2 A3 A4 A5 - - - - - - An k=2

A1 A2 A3 A4 A5 - - - - - - An
p0xp1 p1xp2 p2xp3 p3xp4 p4xp5 - - - - - Pn-1xpn
 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

Solve it Optimally Solve it Optimally

A1 A2 A3 A4 A5 - - - - - - An k=1 A1..n-1 An..n

(Left Subproblem) (Right Subproblem)
A1 A2 A3 A4 A5 - - - - - - An k=2

A1 A2 A3 A4 A5 - - - - - - An k=3

-- -- -- -- -- -- -- --

-- -- -- -- -- -- -- --

A1 A2 A3 A4 - - - - - -
An-1 An k = n-1

A1 A2 A3 A4 A5 - - - - - - An
p0xp1 p1xp2 p2xp3 p3xp4 p4xp5 - - - - - Pn-1xpn
 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

Min. of these
A1 A2 A3 A4 A5 - - - - - - An k=1

A1 A2 A3 A4 A5 - - - - - - An k=2

A1 A2 A3 A4 A5 - - - - - - An k=3

-- -- -- -- -- -- -- --

-- -- -- -- -- -- -- --

A1 A2 A3 A4 - - - - - -
An-1 An k = n-1

A1..n = ( A1..k )( Ak+1..n ) 1 ≤ k < n

A1 A2 A3 A4 A5 - - - - - - An
Generalization: p0xp1 p1xp2 p2xp3 p3xp4 p4xp5 - - - - - Pn-1xpn

Ai..j = ( Ai..k )( Ak+1..j ) i ≤ k < j

 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

Ai..j = ( Ai..k ) ( Ak+1..j ) i ≤ k < j

opt. mul (Ai..j) = min ( mul (Ai..k) + mul (Ak+1..j) + # of mul ( Ai..k ) · ( Ak+1..j ) )
i≤k<j
Let
m[i, j] = minimum # of multiplications needed to compute Ai..j (Ai Ai+1 Aj)
m[i,k] = minimum # of scalar multiplications needed to compute Ai..k
m[k+1,j] = minimum # scalar multiplications needed to compute Ak+1..j

mul ( Ai..k ) · ( Ak+1..j ) = pi-1 x pk x pj

pi-1xpk pkxpj
k=3
A1 A2 A3 A4 A5 - - - - - - An
p0xp1 p1xp2 p2xp3 p3xp4 p4xp5 - - - - - Pn-1xpn
 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

Ai..j = ( Ai..k ) ( Ak+1..j ) i ≤ k < j

opt. mul (Ai..j) = min ( mul (Ai..k) + mul (Ak+1..j) + # of mul ( Ai..k ) · ( Ak+1..j ) )
i≤k<j

m[i, j] = minimum # of multiplications needed to compute Ai..j (Ai Ai+1 Aj)

m[i,k] = minimum # of scalar multiplications needed to compute Ai..k
m[k+1,j] = minimum # scalar multiplications needed to compute Ak+1..j
mul ( Ai..k ) · ( Ak+1..j ) = pi-1 x pk x pj
pi-1xpk pkxpj
Recurrence Relation
m[i, j] = min ( m[i,k] + m[k+1,j] + pi-1 x pk x pj )
i>j
i≤k<j
Recursive Definition
m[i, j] = 0 i=j
 Matrix Chain Multiplication
1. Establish a Recursive Property (Dynamic Programming)

m[i, j] = min ( m[i,k] + m[k+1,j] + pi-1 x pk x pj ) i > j

i≤k<j
m[i, j] = 0 i = j : (contains 1 matrix) and (i>j invalid)

To help us keep track of how to constrct an optimal solution, we define

s[ i,j] = value of k at which we can split the product Ai...j to obtain

an optimal paranthesization.

s[i, j] = k
 Matrix Chain Multiplication
Subproblem Graph – Top Down (Dynamic Programming)

A1..4

A1..1 A2..4 A1..2 A3..4 A1..3 A4..4

A2..2 A3..4 A2..3 A4..4 A1..1 A2..2 A3..3 A4..4 A1..1 A2..3 A1..2 A3..3

A3..3 A4..4 A2..2 A3..3 A2..2 A3..3 A1..1 A2..2

A1 A2 A3 A4
p0xp1 p1xp2 p2xp3 p3xp4
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)

A1..4

A1..3 A2..4

A1..2 A2..3 A3..4

A1..1 A2..2 A3..3 A4..4

 Dynamic Programming Matrix Multiplications Brute Force/D&C
(A1 . A2) (A2 . A3)(A3 . A4) 15 1. (A1(A2(A3A4)))
Scalar Multiplications 2. (A1((A2A3)A4))
A1..3 = (A1) . (A2 A3) 3. ((A1A2)(A3A4))
= (A1 A2) . (A3) ≈ 4n 4. ((A1(A2A3))A4)
Depends on Order of Matrices 5. (((A1A2)A3)A4)
A2..4 = (A2) . (A3 A4)
= (A2 A3) . (A4) A1..4

A1..4 = (A1) . (A2..4)

= (A1 A2) . (A3 A4)
= (A1..3) . (A4) A1..3 A2..4
Matrix Multiplications
10
Scalar Multiplications A1..2 A2..3 A3..4
n3
Depends on Order of Matrices
A1..1 A2..2 A3..3 A4..4
Note: Difference becomes significant in case of large # matrices
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)

Example:
matrix dimension
A1 30 x 35
A2 35 x 15
A3 15 x 5
A4 5 x 10
A5 10 x 20
A6 20 x 25
Order Matrix P :
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25

19
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25 2
3

m[ i, j ] = 0 for i=j 4
5
m[1,1] = 0 6
m[2,2] = 0
m[3,3] = 0 1 2 3 4 5 6
m[4,4] = 0 S 1
m[5,5] = 0
m[6,6] = 0 2
3
4
5
6
20
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25 2 0

3 0

m[ i, j ] = 0 for i=j 4 0

5 0

m[1,1] = 0 6 0
m[2,2] = 0
m[3,3] = 0 1 2 3 4 5 6
m[4,4] = 0 S 1
m[5,5] = 0
m[6,6] = 0 2
3
4
5
6
21
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25 2 0

3 0

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0

i≤k<j 0
5
m[1,2] 6 0

m[2,3] 1 2 3 4 5 6
m[3,4] S 1

m[4,5] 2
3
m[5,6]
4
5
6
22
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25 2 0

3 0

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0

i≤k<j 0
5
m[1,2] = min { m[1, 1] + m[2, 2] + p0 p1 p2 } 6 0
1≤k<2
= min { 0 + 0 + 30x35x15} 1 2 3 4 5 6
S 1
= min ( 15750 ) = 15750 S[1,2] = 1
2
m[2, 3] = min { m[2, 2] + m[3, 3] + p1 p2 p3 } 3
2≤k<3
4
= min { 0 + 0 + 35x15x5} 5
S[2,3] = 2 6
= min ( 2625 ) = 2625
23
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25 2 0 2625

3 0

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0

i≤k<j 0
5
m[1,2] = min { m[1, 1] + m[2, 2] + p0 p1 p2 } 6 0
1≤k<2
= min { 0 + 0 + 30x35x15} 1 2 3 4 5 6
S 1 1
= min ( 15750 ) = 15750 S[1,2] = 1
2 2
m[2, 3] = min { m[2, 2] + m[3, 3] + p1 p2 p3 } 3
2≤k<3
4
= min { 0 + 0 + 35x15x5} 5
= min ( 2625 ) = 2625 S[2,3] = 2 6
24
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750
0 1 2 3 4 5 6
P 30 35 15 5 10 20 25 2 0 2625

3 0

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0

i≤k<j 0
5
m[3, 4] = min { m[3, 3] +m[4, 4] + p2 p3 p4 } 6 0
3≤k<4
= min { 0 + 0 + 15x5x10 } 1 2 3 4 5 6
= min ( 750 ) = 750 S[3,4] = 3 1
S 1
m[4, 5] = min { m[4, 4] + m[5, 5] + p3 p4 p5} 2 2
4≤k<5
= min { 0 + 0 + 5x10x20} 3
= min ( 1000 ) = 1000 S[4,5] = 4
4
m[5, 6] = min { m[5, 5] + m[6, 6] + p4 p5 p6} 5
5≤k<6
= min { 0 + 0 + 10x20x25} 6
= min ( 5000 ) = 5000 S[5,6] = 5 25
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750
0 1 2 3 4 5 6
2 0 2625
P 30 35 15 5 10 20 25
3 0 750

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0 1000

i≤k<j 0
5 5000

l = 24] = min { m[3, 3] +m[4, 4] + p p p }

m[3, 2 3 4 6 0
3≤k<4
= min { 0 + 0 + 15x5x10 } 1 2 3 4 5 6
= min ( 750 ) = 750 S[3,4] = 3 1
S 1
m[4, 5] = min { m[4, 4] + m[5, 5] + p3 p4 p5} 2 2
4≤k<5
= min { 0 + 0 + 5x10x20} 3 3
= min ( 1000 ) = 1000 S[4,5] = 4 4
4
m[5, 6] = min { m[5, 5] + m[6, 6] + p4 p5 p6} 5 5
5≤k<6
= min { 0 + 0 + 10x20x25} 6
= min ( 5000 ) = 5000 S[5,6] = 4 26
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750
0 1 2 3 4 5 6
2625
P 30 35 15 5 10 20 25 2 0

3 0 750

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0 1000

i≤k<j 0
5 5000

m[1, 3] = min { m[1, 1] +m[2, 3] + p0 p1 p3 , 6 0

1≤k<3 m[1, 2] +m[3, 3] + p0 p2 p3 }
1 2 3 4 5 6
= min { 0 + 2625 + 30x35x5 ,
15750 + 0 + 30x15x5 } S 1 1

2
= min ( 7875 , 18000 ) = 7875 S[1,3] = 1 2
m[2, 4] = min { m[2, 2] + m[3, 4] + p1 p2 p4 , 3 3
2 ≤ k < 4 m [2, 3]+m[4, 4] + p1 p3 p4 } 4 4
= min { 0 + 750 + 35x15x10 , 5 5
2625 + 0 + 35x5x10 }
6
= min ( 6000, 4375 ) = 4375 S[1,3] = 3 27
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750 7875
0 1 2 3 4 5 6
2625
P 30 35 15 5 10 20 25 2 0 4375

3 0 750

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0 1000

i≤k<j 0
5 5000

m[1, 3] = min { m[1, 1] +m[2, 3] + p0 p1 p3 , 6 0

1≤k<3 m[1, 2] +m[3, 3] + p0 p2 p3 }
1 2 3 4 5 6
= min { 0 + 2625 + 30x35x5 ,
15750 + 0 + 30x15x5 } S 1 1 1

2 3
= min ( 7875 , 18000 ) = 7875 S[1,3] = 1 2
m[2, 4] = min { m[2, 2] + m[3, 4] + p1 p2 p4 , 3 3
2 ≤ k < 4 m [2, 3]+m[4, 4] + p1 p3 p4 } 4 4
= min { 0 + 750 + 35x15x10 , 5 5
2625 + 0 + 35x5x10 }
6
= min ( 6000, 4375 ) = 4375 S[1,3] = 3 28
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750 7875
0 1 2 3 4 5 6
2625
P 30 35 15 5 10 20 25 2 0 4375

3 0 750

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0 1000

i≤k<j 0
5 5000

m[3, 5]=min { m[3, 3] +m[4, 5] + p2 p3 p5 , 6 0

3≤k<5 m[3, 4] +m[5, 5] + p2 p4 p5 }
1 2 3 4 5 6
= min { 0 + 1000 + 15x5x20 , 1 1
750 + 0 +15x10x20 } S 1
= min ( 2500 , 3750 ) = 2500 S[3,5] = 3 2 2 3

3 3
m[4, 6] = min { m[4, 4] + m[5, 6] + p3 p4 p6 ,
4 4
4 ≤ k < 6 m[4, 5]+m[6, 6] + p3 p5 p6 }
5 5
= min { 0 + 5000 + 5x10x25 ,
1000 + 0 + 5x20x25} 6
= min ( 6250, 3500) = 3500 S[4,6] = 5 29
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
m 1 0 15750 7875
0 1 2 3 4 5 6
2625
P 30 35 15 5 10 20 25 2 0 4375

3 0 750 2500

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0 1000 3500

i≤k<j 0
5 5000

m[3, 5]=min { m[3, 3] +m[4, 5] + p2 p3 p5 , 6 0

3≤k<5 m[3, 4] +m[5, 5] + p2 p4 p5 }
1 2 3 4 5 6
= min { 0 + 1000 + 15x5x20 , 1 1
750 + 0 +15x10x20 } S 1
= min ( 2500 , 3750 ) = 2500 S[3,5] = 3 2 2 3

3 3 3
m[4, 6] = min { m[4, 4] + m[5, 6] + p3 p4 p6 ,
4 4 5
4 ≤ k < 6 m[4, 5]+m[6, 6] + p3 p5 p6 }
5 5
= min { 0 + 5000 + 5x10x25 ,
1000 + 0 + 5x20x25} 6
= min ( 6250, 3500) = 3500 S[4,6] = 5 30
 Matrix Chain Multiplication
2. Compute Optimal Value (DP-Bottom Up)
1 2 3 4 5 6
0 1 2 3 4 5 6 m 1 0 15750 7875 9375 11875 15125

P 30 35 15 5 10 20 25 2 0 2625
4375 7125 10500

3 0 750 2500 5375

m[i, j] = min { m[i, k] + m[k+1, j] + pi-1 pk pj } 4 0 1000 3500

i≤k<j 0
5 5000

m[1, 6] = min { m[1, 1] + m[2, 6] + p0 p1 p6 , 6 0

1 ≤ k < 6 m[1, 2] + m[3, 6] + p0 p2 p6 ,
m[1,3] + m[4,6] + p0 p3 p6 , 1 2 3 4 5 6
m[1, 4] + m[5, 6] + p0 p4Sp6 ,1 1 1 3 3 3
m[1, 5] + m[6, 6] + p0 p5 p6 }
2 2 3 3 3
=min { 0 + 10500 + 30x35x25 , 3 3 3 3
15750+ 5375 + 30x15x25 ,
7875 + 3500 + 30x5x25 , 4 4 5
9375+ 5000 + 30x10x25 , 5
11875+ 0 + 30x20x25 } 5
6
=min { 36750 , 32375 , 15125S[1,6]
, 21875,
= 326875} 31
= 15125
 Matrix Chain Multiplication
3. Optimal Solution (DP-Bottom Up)

( A1 A 2 A 3 A4 A 5 A6 )

( A1 A 2 A 3 ) ( A4 A 5 A6 )

1 2 3 4 5 6
S 1 1 1 3 3 3

2 2 3 3 3
3 3 3 3

4 4 5

5 5

6
32
 Matrix Chain Multiplication
3. Optimal Solution (DP-Bottom Up)

( A1 A 2 A 3 A4 A 5 A6 )

( A1 A 2 A 3 ) ( A4 A 5 A6 )

( ( A1 ( A 2 A 3 ) ) ( A4 A 5 A6 )
1 2 3 4 5 6
S 1 1 1 3 3 3

2 2 3 3 3
3 3

4 4 5

5 5

6
33
 Matrix Chain Multiplication
3. Optimal Solution (DP-Bottom Up)

( A1 A 2 A 3 A4 A 5 A6 )

( A1 A 2 A 3 ) ( A4 A 5 A6 )

( ( A1 ( A 2 A 3 ) ) ( A4 A 5 A6 )
1 2 3 4 5 6
S 1 1 1 3 3 3
( ( A1 ( A 2 A 3 ) ) ( ( A4 A 5 ) A6 )
2 2 3 3 3
3 3

4 4 5

5 5

6
34
 Matrix Chain Multiplication
3. Optimal Solution (DP-Bottom Up)

( A1 A 2 A 3 A4 A 5 A6 ) OUTPUT:
( ( A1 (A2 A3 ) ) ( ( A4 A5 ) A6 ) )
( A1 A 2 A 3 ) ( A4 A 5 A6 )

( ( A1 ( A 2 A 3 ) ) ( A4 A 5 A6 )
1 2 3 4 5 6
S 1 1 1 3 3 3
( ( A1 ( A 2 A 3 ) ) ( ( A4 A 5 ) A6 )
2 2 3 3 3
3 3

4 4 5

5 5

6
35