MAT 461/561: Introduction, 3.

1 Gaussian Elimination

James V. Lambers

January 22, 2020

Introduction
What does this course cover?

• Solving systems of linear equations Ax = b where A is a n × n invertible matrix (unique

solution) by direct methods (Ch. 3, Gaussian elimination and variants), and iterative methods
(Ch. 5)

• Solving systems of nonlinear equations by fixed-point iteration, Newton’s method (Ch. 11)

• Initial value problems for ODEs (time-stepping methods) (Ch. 12)

• Two-point boundary value problems for ODEs (Ch. 13)

Gaussian Elimination
Let A be a n × n invertible matrix, and let b be a n-dimensional column vector. Problem: find the
unique x such that Ax = b (that is, x = A−1 b)
Although we can use methods from solving such systems “by hand” on a computer, it’s not exactly
the same:

• Efficiency matters–must minimize arithmetic operations. Therefore: NEVER EVER EVER

EVER EVER EVER explicitly compute A−1 because it requires 4× as many arithmetic
operations, and is generally less accurate

• When using Gaussian elimination, the reason for interchanging rows is different: by hand,
goal is to avoid dividing by zero. On a computer, goal is to reduce rounding error

• A computer doesn’t care if the numbers are nice!

• Remember, the criteria for a numerical algorithm are: accuracy, efficiency, and robustness
(even if the problem is ill-conditioned)

To design an algorithm for solving Ax = b in the general case, we proceed as follows:

• For which cases is the problem “easy” to solve?

• How can we reduce the general case to one of these easy cases?

1
Triangular Systems
The following systems of equations are relatively easy to solve:

Diagonal Systems
A diagonal system Ax = b is a system in which all of the equations are independent of one another:

a11 x1 = b1
a22 x2 = b2
..
.
ann xn = bn

Note that each unknown xi is involved in only one equation, so we can just compute xi = bi /aii .
This system is called a diagonal system because A is a diagonal matrix:

a11
 

A=
 .. 
. 
ann

Solving this system requires only n floating-point operations.

Upper Triangular Systems

What if the matrix A is not diagonal, but instead has nonzero entries only above the main diagonal?
Example:

2x1 − 3x2 + 4x3 = 5

−6x2 + 7x3 = −8
9x3 = 10.

In this system,  
2 −3 4
A= 0 −6 7 
 
0 0 9
This is an upper triangular matrix. How can we proceed to solve this system? First, we solve the
last equation to get x3 = 10/9. Next, we can substitute x3 into the second equation, which isolates
x2 :
−8 − 7x3
x2 =
−6
Finally, we can substitute x3 and x2 into the first equation:

5 − (−3x2 ) − 4x3
x1 =
2
This process is called back substitution. It requires n2 floating-point operations.

2
Lower Triangular Systems
Similar to upper triangular systems, a matrix is lower triangular if all entries above the main
diagonal are zero. In this case, the first equation can be solved immediately. The algorithm for
this kind of system is forward substitution. It requires O(n2 ) floating-point operations.

Row Operations
Given algorithms for forward and back substitution (solving triangular systems), how we can solve
a general system Ax = b, where A is not triangular? By performing elementary row operations
on the system that do not change the solution, but lead to the system having a triangular structure:

• Multiplying a row by a constant

• Interchanging rows of the system

• Adding or subtracting a multiple of one row from another

Goal: use these row operations to reduce a general invertible matrix A to upper triangular form.
How elimination works: to eliminate aij using row j, where i > j, we need to find the right multiple
of row j to subtract from row i. This means that for k = 1, . . . , n, we perform the following:

aik = aik − mij ajk

where mij is the multiplier needed to ensure aij = 0. So what should mij be? To ensure aij = 0:

aij = aij − mij ajj = 0

has the solution mij = aij /ajj .

Really, we only need to work on columns j + 1, . . . , n. Why? In columns 1, . . . , j − 1, the relevant
entries in row i and j are already zero. In column j, there is no need to explicitly update aij
because we know it will become zero.

Operation Counts
A key measure (but not the only measure) of efficiency of numerical methods is the number of
“flops” (floating-point operation) performed by an algorithm, in terms of some measure of the
problem size (for Ax = b, the problem size is just n, where A is n × n).
Generally, the algorithm complexity of a numerical algorithm is expressed as O(np ) for some p.
How efficient (or inefficient) are the algorithms we have seen?

• Diagonal systems: n flops

• Back substitution: n2 flops

• Forward substitution: n2 flops, but n2 − n if diagonal assumed to be 1

• Gaussian elimination: O(n3 ) flops, more precisely: 32 n3 + lower-order terms

Note: power of n in complexity is most important, but not all-important!

3
Caveats: number of flops isn’t everything!
• Not all flops are created equal! Multiplication, division MUCH more expensive than addition,
subtraction. For a floating-point number stored in p binary bits (see Ch. 2), addition and
subtraction circuits work in O(p) operations, but is O(p2 ) for multiplication and division, due
to distribution.
• All other operations take time! In particularly, moving items around memory (transpose of
a matrix–bad!)
Example: Fourier transform, FFT reduces O(n2 ) work to O(n log n), HUGE difference

How do we get the flops?

Gaussian elimination:
Outer loop: j = 1, . . . , n − 1 (n − 1 iterations)
Middle loop: i = j + 1, . . . , n (n − j iterations)
1 flop
Inner loop: k = j + 1, . . . , n (n − j iterations)
2 flops
2 flops
Inner loop: 2(n − j) flops total
Body of middle loop: 2(n − j) + 3 flops total
Entire middle loop: 2(n − j)2 + 3(n − j) flops total
Total operation count:
n−1
X
2(n − j)2 + 3(n − j)
j=1
Problem: this expression is not in closed form!
n
X n(n + 1)(2n + 1)
j2 =
j=1
6

n−1
X
total = 2(n − j)2 + 3(n − j)
j=1
n−1
X
= 2n2 − 4jn + 2j 2 + 3n − 3j
j=1
n−1
X n−1
X n−1
X n−1
X n−1
X
2 2
= 2n 1 − 4n j+2 j + 3n 1−3 j
j=1 j=1 j=1 j=1 j=1
n−1
X n−1
X n−1
X
= 2n2 (n − 1) − 4n j+2 j 2 + 3n(n − 1) − 3 j
j=1 j=1 j=1
n(n − 1) n(n − 1)(2n − 1) n(n − 1)
= 2n2 (n − 1) − 4n +2 + 3n(n − 1) − 3
2 6 2
3 2
2n − 3n + n 3 3
= + 3n2 − 3n − n2 − n
3 2 2
2 3
= n + lower-order terms
3