CS412: Lecture #17

Mridul Aanjaneya
March 19, 2015

Solving linear systems of equations

Consider a lower triangular matrix L:
 
l11 0 0 ... 0
 l21 l22 0 ... 0 
 
L =  l31 l32 l33 ... 0
 

 .. .. 
 . . 
ln1 ... ... ... lnn

A procedure similar to that for upper triangular systems can be followed to

solve Lx = b. The overall complexity is again O(n2 ).

Algorithm 1 Forward substitution for Lx = b:

1: for j = 1 . . . n do
2: if ljj = 0 then
3: return. . matrix is singular
4: end if
5: xj ← bj /ljj
6: for i = j + 1 . . . n do
7: bi ← bi − lij xj
8: end for
9: end for

The forward and backward substitution processes can be used to solve a

non-triangular system by virtue of the following factorization property.

Theorem 1. If A is an n × n matrix, it can be (generally) written as a product:

A = LU

where L is a lower triangular matrix and U is an upper triangular matrix.

Furthermore, it is possible to construct L such that all diagonal elements lii = 1.

1
Algorithm 2 LU factorization by Gaussian Elimination
1: for k = 1 . . . n − 1 do
2: if akk = 0 then
3: return.
4: end if
5: for i = k + 1 . . . n do
6: aik ← aik /akk
7: end for
8: for j = k + 1 . . . n do
9: for i = k + 1 . . . n do
10: aij ← aij − aik akj
11: end for
12: end for
13: end for

Note that this algorithm executes in-place, i.e., the matrix A is replaced by
its LU factorization, in compact form. More specifically, this algorithm produces
a factorization A = LU , where:
 
1  
 l21 u11 u12 u13 ... u1n
1 O 

 l31 l32
  u22 u23 ... u2n 
1   
L =  l41 l42 l43 1

,

U =
 u33 ... u3n 


 .. .. ..
  .. 
..   O . un−1,n 
 . . . . 
unn
ln1 nn2 ln3 ... ln,n−1 1

After the in-place factorization algorithm completes, A is replaced by the fol-

lowing “‘compacted” encoding of L and U together:
 
u11 u12 u13 ... u1n
 l21 u22 u23 ... u2n 
 
A =  l31 l32 u33 ... u3n 


 .. .. .. .. .. 
 . . . . . 
ln1 ln2 ... ln−1,n unn

Here is a slightly different algorithm for Gaussian Elimination via elimination

matrices. Define the basis vector ek as:
 
0
 .. 
 . 
 
 0 
 
 1
ek =  

 0 
 
 .
 ..


0

2
where the 1 is in the kth row and the length of ek is n. In order to perform
Gaussian Elimination on the kth column ak of A, we define the n×n elimination
matrix Mk = I − mk eTk where

 
0
 .. 

 . 

1  0 
mk = · 
akk 
 ak+1,k


...
 
 
an,k
Mk adds multiples of row k to the rows > k in order to create 0’s. As an
example, for ak = (2, 4, −2)T

    
1 0 0 2 2
M1 ak =  −2 1 0  4  =  0 
1 0 1 −2 0
Similarly,

    
1 0 0 2 2
M2 ak =  0 1 0  4  =  4 
0 1/2 1 −2 0

The inverse of an elimination matrix is defined as Lk = Mk−1 = I + mk eTk . For

example,

   
1 0 0 1 0 0
L1 = M1−1 = 2 1 0  , and L2 = M2−1 =  0 1 0 
−1 0 1 0 −1/2 1
The algorithm now proceeds as follows. Consider the example:

    
2 4 −2 x1 2
 4 9 −3   x2  =  8 
−2 −3 7 x3 10
First, we eliminate the lower triangular portion of A one column at a time using
Mk to get U = Mn−1 . . . M1 A. Note that we also carry out the operations on
b to get a new system of equations M − 2M1 Ax = M2 M1 b or U x = M2 M1 b
which can be solved for via back substitution.

    
1 0 0 2 4 −2 2 4 −2
M1 A =  −2 1 0  4 9 −3  =  0 1 1 
1 0 1 −2 −3 7 0 1 5

3
     
1 0 0 2 2
M1 b =  −2 1 0  8  =  4 
1 0 1 10 12

    
1 0 0 2 4 −2 2 4 −2
M2 M 1 A =  0 1 0  0 1 1 = 0 1 1 
0 −1 1 0 1 5 0 0 4

    
1 0 0 2 2
M2 M1 b =  0 1 0  4  =  4 
0 −1 1 12 8

Finally, solve the following system via back substitution.

    
2 4 −2 x 2
 0 1 1  y  =  4 
0 0 4 z 8

Note that using the fact that the L matrices are inverses of the M matrices allows
us to write LU = (L1 . . . Ln−1 )(Mn−1 . . . M1 A) = A where L = L1 . . . Ln−1 can
be formed trivially from the Mk to obtain:

    
1 0 0 1 0 0 1 0 0
L = L1 L2 =  2 1 0  0 1 0 = 2 1 0 
−1 0 1 0 1 1 −1 1 1

And thus, although we never needed it to solve the equations, the LU factor-
ization of A is

    
2 4 −2 1 0 0 2 4 −2
A= 4 9 −3  =  2 1 0  0 1 1  = LU
−2 −3 7 −1 1 1 0 0 4

Existence/Uniqueness and pivoting

When computing the LU factorization, the algorithm will halt if the diagonal
element akk = 0. This can be avoided by swapping rows of A prior to computing
the LU factorization. This is done to always select the largest akk from the
equations that follow. As an example, consider the matrix A and the action of
the permutation matrix P on it.

4
      
1 2 5 −1 1 0 0 0 1 2 5 −1
 0 0 3 1   0 0 0 1   0 −6 0 3 
A= ,P =  ,PA =  
 0 4 1 −2   0 0 1 0   0 4 1 −2 
0 −6 0 3 0 1 0 0 0 0 3 1

This is pivoting: the pivot akk is selected to be non-zero. In this process, we

can guarantee uniqueness and existence of LU :
Theorem 2. If P is a permutation matrix such that all pivots in the Gaussian
Elimination of P A are non-zero, and lii = 1, then the LU factorization exists
and is unique!

P A = LU