Complexity of Matrix Operations

Brian Van Koten

2024/04/19

Goals
We assess the computational cost of matrix operations such as products, back
substitution, and Gaussian elimination. We assume that the number of floating-
point operations (FLOPS) required to perform a given operation is proportional
to the computational cost. Other factors such as memory usage are as important
in some cases.
We compare the cost of direct and iterative methods for the solution of linear
systems. We conclude that iterative methods may be much more efficient than
direct methods for large sparse systems.

Dot Product
For a, b ∈ RN computing the dot product

N
X
a·b= ai bi
i=1

requires N multiplications and N − 1 additions, for a total of

2N − 1

floating-point operations (FLOPS).

Big O Notation
Let f : N → R and g : N → [0, ∞). We write

f (N ) = O(g(N )) as N → ∞

1
iff for some N0 ∈ N and some constant C > 0 we have

|f (N )| ≤ Cg(N ) for all N ≥ N0 .

Example: Complexity of Dot Product

Let

f (N ) = 2N − 1

be the FLOPS required to compute the dot product.

Observe that

f (N ) = O(N ).

Matrix Vector Product

Let M ∈ RM ×N and v ∈ RN .
To compute each entry in M v, we take a dot product:

vi = Mi: · v.

We take one such dot product for each row of M , and computing each dot
product requires 2N − 1 operations. Thus, we require

M (2N − 1)

operations to compute M v.
If M ∈ RN ×N is square, then the number of operations is

N (2N − 1) = O(N 2 ).

Back Substitution
To solve an upper-triangular system

2
 A11 x1 + A12 x2 + · · · + A1N xN = b1
A22 x2 + · · · + A2N xN = b2
..
.
AN N x N = bN ,

we solve the equations in sequence starting with the last.

It costs 1 floating-point division to solve the last equation for xn .
The second-to-last equation is

AN −1,N −1 xN −1 + AN −1,N xN = bN −1 .

Solving the second-to-last equation to calculate xn−1 costs 1 division, 1 multipli-

cation, and 1 subtraction, since we have

xN −1 = A−1
N −1,N −1 (bN −1 − AN −1,N xN ).

In general, solving the N − k’th equation for xN −k costs 1 division, k multipli-

cations, and k subtractions, since

xN −k = A−1
N −k,N −k (bN −k −AN −k+1 xN −k+1 −AN −k+2 xN −k+2 −. . . AN −k,N xN ).

Therefore, the total cost of back substitution is

N −1
X N (N − 1)
2k + 1 = 2 +N = O(N 2 ).
2
k=0 | {z }
sum of integers k from zero to N − 1

Gaussian Elimination with Partial Pivoting

One can show that the cost to compute the P LU decomposition by Gaussian
elimination with partial pivoting is

2 3
PLU FLOPS ∼ N
3
for a matrix of size N × N .

3
Householder QR Factorization
The cost to compute the QR-factorization by the Householder method is

4 3
QR FLOPS ∼ N .
3

Thus, P LU and QR both cost O(N 3 ).

But note that QR-factorization requires approximately twice as many operations!

Sparse vs. Dense Matrices

Review of Sparsity
Recall the matrix
 
2 −1 0 0 ... 0 0
−1 2 −1 0 ... 0 0
 
A = k 0
 −1 2 −1 ... 0 0  ∈ RN ×N
 .. .. .. .. .. .. .. 
 . . . . . . .
0 0 0 0 ... −1 2

from our mass-spring system.

Most of the entries are zero.
That is, A is sparse.
Matrices with mostly nonzero entries are called dense.

FLOPS for Sparse Multiplication

Our operation counts above assume dense matrices.
If most of the entries of a matrix are zero, then you don’t have to do all those
multiplications. You know that when a zero entry multiplies some possibly
nonzero element of a vector, the result must be zero.
Therefore, for the mass-spring matrix A, the cost to compute Ax is

N
X −1
2 × (2 multiplications + 1 addition ) + (3 multiplications + 2 additions ) = O(N ).
| {z }
k=2
1’st and N ’th rows | {z }
2’d through N − 1’st rows

This is a lower order of complexity than the dense matrix-vector product!

4
FLOPS for Sparse PLU and QR
It can be very difficult to take advantage of sparsity when computing the QR or
PLU decomposition of a matrix.
This is essentially because the inverse of a sparse matrix need not be sparse, and
the factors Q, R, L, and U need not be sparse.
Matrix factorizations usually cost O(N 3 ) for sparse matrices unless you do
something very clever.
There are a exceptions for sparse matrices of special forms.
For example, the tridiagonal matrix algorithm computes solutions of tridiagonal
systems (such as the mass-spring) at O(n) cost.

Complexity of Iterative Methods

Iterative methods may have a much lower cost for sparse systems than dense.

Complexity of One Step

At each step in a classical iteration, one computes a single matrix-vector product
and a vector sum:

xn+1 = xn + P (b − Axn ) = (I − P A)xn + P b.

The cost is at most O(N 2 ), but may be as low as O(N ) if P A is sparse.

For this reason, iterative methods can be much more efficient than QR-
factorization or Gaussian elimination for sparse systems.
However, to really see the benefits of iterative methods, you need more so-
phisticated ideas than Gauss-Seidel, e.g. preconditioned conjugate gradient or
multigrid methods.
One can show that for an important class of physics problems with structure
similar to the mass-spring system (including many where A is not tridiagonal)
the total cost of computing the solution by a multigrid method is O(N )!
This should strike you as an amazing improvement over the O(N 3 ) cost of
Gaussian elimination.

Overall Complexity of Iterative Methods

Suppose that we wish to calculate the solution x to a certain precision ϵ.
To make things concrete, suppose that we wish to calculate an approximate
solution x̃ of Ax = b so that

5
 ∥x̃ − x∥∞ ≤ ϵ.

Recall our error estimate for classical iterations: If ∥I − P A∥∞ < 1, then the
iteration is convergent and

∥xm − x∥∞ ≤ ∥I − P A∥m

∞ ∥x0 − x∥∞ .
| {z } | {z }
error after m steps initial error

To estimate the number of steps required to acheive precision ϵ, we can find the
smallest m so that

∥I − P A∥m
∞ ∥x0 − x∥∞ ≤ ϵ.

Taking ln on both sides, we see

m ln(∥I − P A∥∞ ) + ln(∥x0 − x∥∞ ) ≤ ln(ϵ).

Since we assume ∥I − P A∥∞ < 1, we have ln(∥I − P A∥∞ ) < 0, and therefore
the above equation is equivalent with

ln(ϵ/∥x0 − x∥∞ )
m≥ .
ln(∥I − P A∥∞ )

Thus, one estimate of the total complexity would be

ln(ϵ/∥x0 − x∥∞ )
× FLOPS for one step.
ln(∥I − P A∥∞ )

Note that the complexity of an iterative method is highly dependent on the

contraction rate ∥I − P A∥∞ , the quality of the initial guess x0 for the solution,
and the sparsity of P and A.
If P A is sparse, the cost of one step may be as low as O(N ).
This does not necessarily imply that the total cost is O(N ), since ∥I − P A∥∞
may depend on N for any given family of matrices.
On the homework, you will investigate the dependence of ∥I − P A∥∞ on N for
Jacobi and Gauss–Seidel methods applied to the mass-spring system.
There are no useful estimates of ∥I − P A∥∞ for arbitrary matrices A and P .
One has to analyze the efficiency of iterative methods for particular families of
matrices, such as the mass-spring system.