LINEAR ALGEBRA AND VECTOR ANALYSIS

MATH 22B

Unit 7: Gram-Schmidt

Lecture
7.1. For vectors in the linear space Rn , the dot product is defined as v · w = i vi wi .
P
More generally, in the linear space M (n, m) there is a natural dot product Pv·w =
tr(v T w), where tr is the trace, the sum of the diagonal entries. It is the sum i,j vij wij .
√
The dot product allows to compute length |v| = v · v and angles α between two
vectors defined by the equation v · w = |v||w| cos(α). If the relation v · w = 0 holds,
the vectors v and w are called orthogonal.

7.2. A collection of pairwise orthogonal vectors {v1 , v2 , · · · , vn } in Rn is linearly inde-

pendent because a1 v1 + · · · an vn = 0 implies that vk · (a1 v1 + · · · an vn ) = ak vk · vk =
ak |vk |2 = 0 and so ak = 0. A collection of n orthogonal vectors therefore automatically
forms a basis.

7.3. Definition. A basis is called orthonormal if all vectors have length 1 and
are orthogonal. Why do we like to have an orthogonal basis? One reason is that
an orthogonal basis looks like the standard basis. Another reason is that rotations
preserving a space V or orthogonal projections onto a space V are easier to describe
if an orthogonal basis is known on V . Let’s look at projections as we will need them
to produce an orthonormal basis. Remember that the projection of a vector x onto
a unit vector v is (v · x)v. We can now give the matrix of a projection onto a space V
if we know an orthonormal basis in V :

Lemma: If B = {v1 , v2 , · · · , vn } is an orthonormal basis in V , then the

projection P onto V satisfies P x = (v1 · x)v1 + · · · + (vn · x)vn .

Proof. By Pythagoras, (x − P x) · x = |x|2 − (v1 · x)2 − · · · − (vn · x)2 = 0, so that x − P x

is perpendicular to x.

Let Q be the matrix containing the basis vk as columns. We can rewrite the result as
P = QQT . We write Q because it is not a n × n matrix like S. The matrix Q contains
the basis of the subspace V and not the basis of the entire space. We will see next week
a more general formula for P which also holds if the vectors are not perpendicular.
 Linear Algebra and Vector Analysis

7.4. Let v1 , . . . , vn be a basis in V . Let w1 = v1 and u1 = w1 /|w1 |. The Gram-

Schmidt process recursively constructs from the already constructed orthonormal
set u1 , . . . , ui−1 which spans a linear space Vi−1 the new vector wi = (vi − projVi−1 (vi ))
which is orthogonal to Vi−1 , and then normalizes wi to get ui = wi /|wi |. Each vector
wi is orthonormal to the linear space Vi−1 . The vectors {u1 , . . . , un } form then an
orthonormal basis in V .
7.5. The formulas can be written as
v1 = |w1 |u1 = r11 u1
...
vi = (u1 · vi )u1 + · · · + (ui−1 · vi )ui−1 + |wi |ui = r1i u1 + · · · + rii ui
...
vn = (u1 · vn )u1 + · · · + (un−1 · vn )un−1 + |wn |un = r1n u1 + · · · + rnn un .
In matrix form this means
    
| | · | | | · | r11 r12 · r1n
A =  v1 v2 · vn  =  u1 u2 · un   0 r22 · r2n  = QR ,
| | · | | | · | 0 0 · rnn
where A and Q are m × n matrices and R is a n × n matrix with
rij = vj · ui , for i < j and vii = |wi |
We have just seen:
Theorem: A matrix A with linearly independent columns vi can be de-
composed as A = QR, where Q has orthonormal column vectors and
where R is an upper triangular square matrix with the same number of
columns than A. The matrix Q has the orthonormal vectors ui in the
columns.

7.6. The recursive process was stated first by Erhard Schmidt (1876-1959) in 1907.
The essence of the formula was already in a 1883 paper by J.P.Gram in 1883 which
Schmidt mentions in a footnote. The process seems to already have been anticipated
by Laplace (1749-1827) and was also used by Cauchy (1789-1857) in 1836.

Figure 1.

Examples
     
2 1 1
7.7. Problem. Use Gram-Schmidt on {v1 =  0 , v2 =
  3 , v3 =
  2 }.
0 0 5
 
1
v1
Solution. 1. w1 = |v1 | = 0 , u1 = w1 .

0
    
0 0
w2
2. w2 = (v2 − projV1 (v2 )) = v2 − (u1 · v2 )u1 =  3 . u2 = |w2 |
=  1 .
0 0
   
0 0
w3
3. w3 = (v3 − projV2 (v3 )) = v3 − (u1 · v3 )u1 − (u2 · v3 )u2 = 0 , u3 =

|w3 |
=  0 .
5 1

7.8. From v1 = |v1 |u1 , v2 = (u1 ·v2 )u1 +|w2 |u2 , and v3 = (u1 ·v3 )u1 +(u2 ·v3 )u2 +|w3 |u3 ,
we get the QR decomposition
    
2 1 1 1 0 0 2 1 1
A =  0 3 2  =  0 1 0  0 3 2  = QR .
0 0 5 0 0 1 0 0 5
7.9. One reason why we are interested in orthogonality is that in statistics, “orthog-
onal” means “uncorrelated”. Data are often  alsoarranged in matrices  as relational
1 1 0 3
databases. Let’s take the matrices v1 = and v2 = . They span
1 1 3 0
a two dimensional plane in the linear space M (2, 2) of 2 × 2 matrices. We want to
have an orthogonal set of vectors in that plane. Now, how do we do that? We can
use Gram-Schmidt in the same way as with vectors in Rn . One possibility is to write
 T  T
the matrices as vectors like v1 = 1 1 1 1 and v2 = 0 3 3 0 and pro-
ceed with vectors. But we can also remain within matrices and do the Gram-Schmidt
procedure in M (2, 2). Let us do that. The first step is to normalize the first vector.
1 1
We get u1 = /2. The second step is to produce w2 = v2 − (u1 · v2 )u1 =
1 1
     
0 3 1 1 −3/2 3/2
−3 /2 = . Now, we have to normalize this to get
3 0 1 1 3/2 −3/2
 
−1/2 1/2
u2 = w2 /|w2 | = w2 /3 = . Now, B = {u1 , u2 } is an orthonormal basis
1/2 −1/2
in the space X spanned by {v1 , v2 }.

7.10.
Theorem: If S T = S −1 the map T : A → S −1 AS is an orthogonal
transformation from M (n, n) → M (n, n).

Proof. The dot product between A1 = S −1 AS and B1 = S −1 BS is equal to the one

between A and B:
tr(AT1 B1 ) = tr((S −1 AS)T S −1 BS) = tr((S −1 AT SS −1 BS) = tr(S −1 AT BS) = tr(AT B) .

We have used in the last step that similar matrices always have the same trace. We
prove this later. For 2 × 2 matrices we can check it by brute force:

A={{a , b } ,{ c , d } } ; S={{p , q } ,{ r , s } } ;
Simplify [ Tr [ Inverse [ S ] . A. S]==Tr [A ] ]
 
 Linear Algebra and Vector Analysis

Homework
This homework is due on Tuesday, 2/20/2019.
Problem
   7.1:  Perform
 the Gram-Schmidt process on the three vectors
5 2 1
 5   2   −1 
 5  ,  4  ,  0 } and then write down the QR decomposition.
{     
5 2 0

Problem 7.2: a) Find an orthonormal basis of the plane x + y + z = 0

and form the projection matrix P = QQT .
b) Find an orthonormal basis of the hyper plane x1 + x2 + x3 + x4 + x5 = 0
in R5 .

Problem 7.3: a) Produce an orthonormal basis of the kernel of

 
1 −1 1 −1 1
A= .
1 1 −1 1 1
b) Write down an orthonormal basis for the image of A.

Problem 7.4: Find the QR factorization of the following matrices

   
  0 −3 0 1  
2 4  0 0 0   2  12 5
A= ,B = 
  ,C =   ,D =
  .
1 1 7 0 0  3 −5 12
0 0 4 4

Problem 7.5: Find the QR factorization of the following matrices (D

was the quaternion matrix you have derived earlier)
 
1 1  
1 1 −1  1 2
A=   ,
2  1 −1  0 5
1 1
 
4 0 0 0  
 0 4 0 0  5 12
B=  0 0 4 0  , C = 12 −5 .

0 0 0 4
 
a −b −c −d
 b a −d c 
D= 
 c d a −b 
d −c b a

Oliver Knill, [email protected], Math 22b, Harvard College, Spring 2019