Advanced Linear Algebra

MTH 706

Virtual University of Pakistan

Knowledge beyond the boundaries
 TABLE OF CONTENTS :

Lesson 1: Introduction and Overview .................................................................................... 2

Lesson 2: Introduction to Matrices ...................................................................................... 10
Lesson 3: Systems of Linear Equations ............................................................................... 23
Lesson 4: Row Reduction and Echelon Forms .................................................................... 36
Lesson 5: Null Spaces, Column Spaces, and Linear Transformations ................................ 46
Lesson 6: Linearly Independent Sets; Bases ........................................................................ 69
Lesson 7: Dimension of a Vector Space .............................................................................. 85
Lesson 8: Rank ................................................................................................................... 103
Lesson 9: Solution of Linear System of Equations (Jacobi Method) ............................... 121
Lesson 10: Solution of Linear System of Equations (Gauss–Seidel Iteration Method) .... 128
Lesson 11: Solution of Linear System of Equations (Relaxation Method) ....................... 136
Lesson 12: Norms of Vectors and Matrices, Matrix Norms and Distances....................... 142
Lesson 13: Error Bounds and Iterative Refinement .......................................................... 143
Lesson 14: Eigenvalues and Eigenvectors ......................................................................... 144
Lesson 15: The Characteristic Equation ............................................................................ 154
Lesson 16: Diagonalization................................................................................................ 166
Lesson 17: Inner Product ................................................................................................... 176
Lesson 18: Orthogonal and Orthonormal set ..................................................................... 183
Lesson 19: Orthogonal Decomposition ............................................................................. 200
Lesson 20: Orthogonal basis, Gram-Schmidt Process, Orthonormal basis ....................... 208
Lesson 21: Least Square Solution ...................................................................................... 217
Lesson 22-15: Eigen Value Problems (Power Method) .................................................... 226
Lesson 23-16: Eigen Value Problems (Jacobi’s Method) ................................................. 233
Lesson 24-17: Eigen Value Problems (continued) ........................................................... 235
Lesson 25: Inner Product Space ......................................................................................... 241
Lesson 26: Application of inner product spaces ................................................................ 251
Lesson 27: Householder’s Method and QR Algorithum ................................................. 262
Lesson 28: Singular Value Decomposition ....................................................................... 263
Lesson 29: Fixed Points for Functions of Several Variables ............................................ 264
Lesson 30: Newton’s Method ............................................................................................ 265
Lesson 31: Quasi-Newton Method .................................................................................... 266
1-Introduction and Overview VU

Lecture 1

Introduction and Overview

What is Algebra?

History
Algebra is named in honor of Mohammed Ibn-e- Musa al-Khowârizmî. Around 825, he
wrote a book entitled Hisb al-jabr u'l muqubalah, ("the science of reduction and
cancellation"). His book, Al-jabr, presented rules for solving equations.
Algebra is a branch of Mathematics that uses mathematical statements to describe
relationships between things that vary over time. These variables include things like the
relationship between supply of an object and its price. When we use a mathematical
statement to describe a relationship, we often use letters to represent the quantity that
varies, since it is not a fixed amount. These letters and symbols are referred to as
variables.
Algebra is a part of mathematics in which unknown quantities are found with the help of
relations between the unknown and known.
In algebra, letters are sometimes used in place of numbers.
The mathematical statements that describe relationships are expressed using algebraic
terms, expressions, or equations (mathematical statements containing letters or symbols
to represent numbers). Before we use algebra to find information about these kinds of
relationships, it is important to first introduce some basic terminology.

Algebraic Term

The basic unit of an algebraic expression is a term. In general, a term is either a product
of a number and with one or more variables.

For example 4x is an algebraic term in which 4 is coefficient and x is said to be variable.

Study of Algebra
Today, algebra is the study of the properties of operations on numbers. Algebra
generalizes arithmetic by using symbols, usually letters, to represent numbers or

______________________________________________________________________
©Virtual University Of Pakistan 2
1-Introduction and Overview VU

unknown quantities. Algebra is a problem-solving tool. It is like a tractor, which is a

farmer's tool. Algebra is a mathematician's tool for solving problems. Algebra has
applications to every human endeavor. From art to medicine to zoology, algebra can be a
tool. People who say that they will never use algebra are people who do not know about
algebra. Learning algebra is a bit like learning to read and write. If you truly learn
algebra, you will use it. Knowledge of algebra can give you more power to solve
problems and accomplish what you want in life. Algebra is a mathematicians’ shorthand!

Algebraic Expressions

An expression is a collection of numbers, variables, and +ve sign or –ve sign, of

operations that must make mathematical and logical behaviour.

For example 8 x 2 + 9 x − 1 is an algebraic expression.

What is Linear Algebra?

One of the most important problems in mathematics is that of solving systems of linear
equations. It turns out that such problems arise frequently in applications of mathematics
in the physical sciences, social sciences, and engineering. Stated in its simplest terms, the
world is not linear, but the only problems that we know how to solve are the linear ones.
What this often means is that only recasting them as linear systems can solve non-linear
problems. A comprehensive study of linear systems leads to a rich, formal structure to
analytic geometry and solutions to 2x2 and 3x3 systems of linear equations learned in
previous classes.
It is exactly what the name suggests. Simply put, it is the algebra of systems of linear
equations. While you could solve a system of, say, five linear equations involving five
unknowns, it might not take a finite amount of time. With linear algebra we develop
techniques to solve m linear equations and n unknowns, or show when no solution exists.
We can even describe situations where an infinite number of solutions exist, and describe
them geometrically.
Linear algebra is the study of linear sets of equations and their transformation properties.

______________________________________________________________________
©Virtual University Of Pakistan 3
1-Introduction and Overview VU

Linear algebra, sometimes disguised as matrix theory, considers sets and functions, which
preserve linear structure. In practice this includes a very wide portion of mathematics!
Thus linear algebra includes axiomatic treatments, computational matters, algebraic
structures, and even parts of geometry; moreover, it provides tools used for analyzing
differential equations, statistical processes, and even physical phenomena.
Linear Algebra consists of studying matrix calculus. It formalizes and gives geometrical
interpretation of the resolution of equation systems. It creates a formal link between
matrix calculus and the use of linear and quadratic transformations. It develops the idea
of trying to solve and analyze systems of linear equations.
Applications of Linear algebra
Linear algebra makes it possible to work with large arrays of data. It has many
applications in many diverse fields, such as
• Computer Graphics,
• Electronics,
• Chemistry,
• Biology,
• Differential Equations,
• Economics,
• Business,
• Psychology,
• Engineering,
• Analytic Geometry,
• Chaos Theory,
• Cryptography,
• Fractal Geometry,
• Game Theory,
• Graph Theory,
• Linear Programming,
• Operations Research

______________________________________________________________________
©Virtual University Of Pakistan 4
1-Introduction and Overview VU

It is very important that the theory of linear algebra is first understood, the concepts are
cleared and then computation work is started. Some of you might want to just use the
computer, and skip the theory and proofs, but if you don’t understand the theory, then it
can be very hard to appreciate and interpret computer results.
Why using Linear Algebra?
Linear Algebra allows for formalizing and solving many typical problems in different
engineering topics. It is generally the case that (input or output) data from an experiment
is given in a discrete form (discrete measurements). Linear Algebra is then useful for
solving problems in such applications in topics such as Physics, Fluid Dynamics, Signal
Processing and, more generally Numerical Analysis.
Linear algebra is not like algebra. It is mathematics of linear spaces and linear functions.
So we have to know the term "linear" a lot. Since the concept of linearity is fundamental
to any type of mathematical analysis, this subject lays the foundation for many branches
of mathematics.
Objects of study in linear algebra
Linear algebra merits study at least because of its ubiquity in mathematics and its
applications. The broadest range of applications is through the concept of vector spaces
and their transformations. These are the central objects of study in linear algebra

1. The solutions of homogeneous systems of linear equations form paradigm

examples of vector spaces. Of course they do not provide the only examples.
2. The vectors of physics, such as force, as the language suggests, also provide
paradigmatic examples.
3. Binary code is another example of a vector space, a point of view that finds
application in computer sciences.
4. Solutions to specific systems of differential equations also form vector spaces.
5. Statistics makes extensive use of linear algebra.
6. Signal processing makes use of linear algebra.
7. Vector spaces also appear in number theory in several places, including the
study of field extensions.

______________________________________________________________________
©Virtual University Of Pakistan 5
1-Introduction and Overview VU

8. Linear algebra is part of and motivates much abstract algebra. Vector spaces
form the basis from which the important algebraic notion of module has been
abstracted.
9. Vector spaces appear in the study of differential geometry through the tangent
bundle of a manifold.
10. Many mathematical models, especially discrete ones, use matrices to represent
critical relationships and processes. This is especially true in engineering as
well as in economics and other social sciences.

There are two principal aspects of linear algebra: theoretical and computational. A major
part of mastering the subject consists in learning how these two aspects are related and
how to move from one to the other.

Many computations are similar to each other and therefore can be confusing without
reasonable level of grasp of their theoretical context and significance. It will be very
tempting to draw false conclusions.

On the other hand, while many statements are easier to express elegantly and to
understand from a purely theoretical point of view, to apply them to concrete problems
you will need to “get your hands dirty”. Once you have understood the theory sufficiently
and appreciate the methods of computation, you will be well placed to use software
effectively, where possible, to handle large or complex calculations.

______________________________________________________________________
©Virtual University Of Pakistan 6
1-Introduction and Overview VU

Course Segments

The course is covered in 45 Lectures spanning over six major segments, which are given
below;

1. Linear Equations
2. Matrix Algebra
3. Determinants
4. Vector spaces
5. Eigen values and Eigenvectors, and
6. Orthogonal sets

Course Objectives
The main purpose of the course is to introduce the concept of linear algebra, to explain
the underline theory, the computational techniques and then try to apply them on real life
problems. Major course objectives are as under;

• To master techniques for solving systems of linear equations

• To introduce matrix algebra as a generalization of the single-variable algebra of
high school.
• To build on the background in Euclidean space and formalize it with vector space
theory.
• To develop an appreciation for how linear methods are used in a variety of
applications.
• To relate linear methods to other areas of mathematics such as calculus and,
differential equations.

______________________________________________________________________
©Virtual University Of Pakistan 7
1-Introduction and Overview VU

Recommended Books and Supported Material

I am indebted to several authors whose books I have freely used to prepare the lectures
that follow. The lectures are based on the material taken from the books mentioned
below.

1. Linear Algebra and its Applications (3rd Edition) by David C. Lay.

2. Contemporary Linear Algebra by Howard Anton and Robert C. Busby.
3. Introductory Linear Algebra (8th Edition) by Howard Anton and Chris Rorres.
4. Introduction to Linear Algebra (3rd Edition) by L. W. Johnson, R.D. Riess and
J.T. Arnold.
5. Linear Algebra (3rd Edition) by S. H. Friedberg, A.J. Insel and L.E. Spence.
6. Introductory Linear Algebra with Applications (6th Edition) by B. Kolman.

I have taken the structure of the course as proposed in the book of David C. Lay. I would
be following this book. I suggest that the students should purchase this book, which is
easily available in the market and also does not cost much. For further study and
supplement, students can consult any of the above mentioned books.
I strongly suggest that the students should also browse on the Internet; there is plenty of
supporting material available. In particular, I would suggest the website of David C. Lay;
www.laylinalgebra.com, where the entire material, study guide, transparencies are readily
available. Another very useful website is www.wiley.com/college/anton, which contains a
variety of useful material including the data sets. A number of other books are also
available in the market and on the internet with free access.
I will try to keep the treatment simple and straight. The lectures will be presented in
simple Urdu and easy English. These lectures are supported by the handouts in the form
of lecture notes. The theory will be explained with the help of examples. There will be
enough exercises to practice with. Students are advised to go through the course on daily
basis and do the exercises regularly.

______________________________________________________________________
©Virtual University Of Pakistan 8
1-Introduction and Overview VU

Schedule and Assessment

The course will be spread over 45 lectures. Lectures one and two will be introductory and
the Lecture 45 will be the summary. The first two lectures will lay the foundations and
would provide the overview of the course. These are important from the conceptual point
of view. I suggest that these two lectures should be viewed again and again.

The course will be interesting and enjoyable, if the student will follow it regularly and
completes the exercises as they come along. To follow the tradition of a semester system
or of a term system, there will be a series of assignments (Max eight assignments) and a
mid term exam. Finally there will be terminal examination.

The assignments have weights and therefore they have to be taken seriously.

______________________________________________________________________
©Virtual University Of Pakistan 9
2-Introduction to Matrices VU

Lecture 2
Background

Introduction to Matrices

Matrix A matrix is a collection of numbers or functions arranged into rows and columns.

Matrices are denoted by capital letters A, B,  , Y , Z . The numbers or functions are called
elements of the matrix. The elements of a matrix are denoted by small letters a, b,  , y, z .

Rows and Columns The horizontal and vertical lines in a matrix are, respectively, called the
rows and columns of the matrix.

Order of a Matrix The size (or dimension) of matrix is called as order of matrix. Order of
matrix is based on the number of rows and number of columns. It can be written as r × c ; r
means no. of row and c means no. of columns.

If a matrix has m rows and n columns then we say that the size or order of the matrix
is m × n . If A is a matrix having m rows and n columns then the matrix can be written as
 a11 a12  a1n 
a 
 21 a22  a2 n 
A=     
 
    
  amn 
 am1 am 2
The element, or entry, in the ith row and jth column of a m × n matrix A is written as aij

 2 −1 3 
For example: The matrix A =   has two rows and three columns. So order of A
0 4 6
will be 2 × 3

Square Matrix A matrix with equal number of rows and columns is called square matrix.
 4 7 −8 
 
For Example The matrix A =  9 3 5  has three rows and three columns. So it is a
 1 −1 2 
 
square matrix of order 3.

Equality of matrices

The two matrices will be equal if they must have

©Virtual University Of Pakistan 10

2-Introduction to Matrices VU

a) The same dimensions (i.e. same number of rows and columns)

b) Corresponding elements must be equal.

 4 7 −8   4 7 −8 
   
Example The matrices A =  9 3 5  and B =  9 3 5  equal matrices
   1 −1 2 
 1 −1 2   

(i.e A = B) because they both have same orders and same corresponding elements.

Column Matrix A column matrix X is any matrix having n rows and only one column.
Thus the column matrix X can be written as
 b11 
 
b 
 21 
X =  b31  = [bi1 ]n ×1
 
  
 
 
 bn1 
A column matrix is also called a column vector or simply a vector.

Multiple of matrix A multiple of a matrix A by a nonzero constant k is defined to be

 ka11 ka12  ka1n 

 
 ka 21 ka 22  ka 2n 
kA =   = [kaij ]m × n
     
 
ka m1 ka m 2  ka mn 

Notice that the product kA is same as the product Ak . Therefore, we can write kA = Ak .

It implies that if we multiply a matrix by a constant k, then each element of the matrix is to
be multiplied by k.
Example 1

 2 − 3 10 − 15
   
(a) 5⋅  4 − 1 = 20 −5
   
1 / 5 6   1 30 

©Virtual University Of Pakistan 11

2-Introduction to Matrices VU

1   e 
t
   
(b) t 
e ⋅ − 2 = − 2e t
   
 4   t 
 4e 

Since we know that kA = Ak . Therefore, we can write

− 3t
2 2e − 3t  2 − 3t
e ⋅  =  =  e
5 5e − 3t  5

Addition of Matrices Only matrices of the same order may be added by adding
corresponding elements.
If A = [aij ] and B = [bij ] are two m × n matrices then A + B = [aij + bij ]
Obviously order of the matrix A + B is m × n

Example 2 Consider the following two matrices of order 3× 3

 2 −1 3  4 7 − 8
   
A= 0 4 
6 , B= 9  3 5 
   
− 6 − 5  −1 2 
 10 1
Since the given matrices have same orders, therefore, these matrices can be added and their
sum is given by

 2+4 −1+ 7 3 + (−8)   6 6 − 5

   
A+ B =  0+9 4+3 6+5  = 9 7 11 
   
− 6 +1 10 + (−1) − 5 + 2   − 5 − 3 
 9

Example 3 Write the following single column matrix as the sum of three column vectors
 3t 2 − 2e t 
 2 
 t + 7t 
 5t 
 
Solution

©Virtual University Of Pakistan 12

2-Introduction to Matrices VU

 3t 2 − 2et   3t 2   
  0   − 2e  0   −2 
t
    3

 t 2 + 7t =  t 2  +  7t  +  0 =
 1  t 2  7  t  0  et
         +  + 
 0 5  0 
 5t   0   5t   0       
   

Difference of Matrices The difference of two matrices A and B of same order m × n is

defined to be the matrix A − B = A + (− B)
The matrix − B is obtained by multiplying the matrix B with − 1 . So that − B = ( − 1 ) B

Multiplication of Matrices We can multiply two matrices if and only if, the number of
columns in the first matrix equals the number of rows in the second matrix.
Otherwise, the product of two matrices is not possible.
OR
If the order of the matrix A is m × n then to make the product AB possible order of the
matrix B must be n × p . Then the order of the product matrix AB is m × p . Thus
Am × n ⋅ Bn × p = C m × p

If the matrices A and B are given by

 a11 a12  a1n   b11 b12  b1 p 
   
 a 21 a 22  a 2n  b21 b22  b2 p 
A= , B =  
           
   
a m1 a m 2  a mn  bn1 bn 2  bnp 

Then
 a11 a12  a1n   b11 b12  b1 p 
  
 a 21 a 22  a 2n  b21 b22  b2 p 
AB =   
          
  
a m1 a m 2  a mn  bn1 bn 2  bnp 


 a11b11 + a12b21 +  + a1nbn1  a11b1 p + a12b2 p +  + a1nbnp 

 a b + a b + + a b  a b + a b +  + a b 
= 
21 11 22 21 2 n n1 21 1 p 22 2 p 2 n np
    
 
 am1b11 + am 2b21 +  + amnbn1  am1b1 p + am 2b2 p +  + amnbnp 

©Virtual University Of Pakistan 13

2-Introduction to Matrices VU

 n 
=  ∑ aik bkj 
 
 k =1  n× p
Example 4 If possible, find the products AB and BA , when
4 7 9 − 2
(a) A= , B =  
 3 5 6 8 
   

5 8
   − 4 − 3
(b) A = 1 0 ,
 B= 
   2 0 
2 7  
 

Solution (a) The matrices A and B are square matrices of order 2. Therefore, both of the
products AB and BA are possible.

 4 7  9 − 2   4 ⋅ 9 + 7 ⋅ 6 4 ⋅ (−2) + 7 ⋅ 8   78 48 
AB =   = = 
 3 5  6 8   3 ⋅ 9 + 5 ⋅ 6 3 ⋅ (−2) + 5 ⋅ 8   57 34 
      

 9 − 2  4 7   9 ⋅ 4 + (−2) ⋅ 3 9 ⋅ 7 + (−2) ⋅ 5   30 53 
Similarly BA =   = = 
 6 8  3 5   6 ⋅ 4 + 8 ⋅ 3  
6 ⋅ 7 + 8 ⋅ 5   48 82 
   
Note From above example it is clear that generally a matrix multiplication is not
commutative i.e. AB ≠ BA .

(b) The product AB is possible as the number of columns in the matrix A and the number of
rows in B is 2. However, the product BA is not possible because the number of column in the
matrix B and the number of rows in A is not same.

5 8
   −4 −3 
AB =  1 0   
 2 7 2 0 
 
 5 ⋅ (−4) + 8 ⋅ 2 5 ⋅ (−3) + 8 ⋅ 0   −4 −15 
   
=  1 ⋅ (−4) + 0 ⋅ 2 1 ⋅ (−3) + 0 ⋅ 0  =  −4 −3 
 2 ⋅ (−4) + 7 ⋅ 2 2 ⋅ (−3) + 7 ⋅ 0   6 −6 
 
 78 48   30 53 
AB =   , BA =  
 57 34   48 82 
   

Clearly AB ≠ BA.

©Virtual University Of Pakistan 14

2-Introduction to Matrices VU

− 4 − 15 
 
AB =  − 4 −3 
 
 6
 − 6 

However, the product BA is not possible.

Example 5
 2 − 1 3  − 3   2 ⋅ (−3) + (−1) ⋅ 6 + 3 ⋅ 4   0 
      
(a)  0 4 5  6  =  0 ⋅ (−3) + 4 ⋅ 6 + 5 ⋅ 6  =  44 
      
 1 − 7 9  4   1 ⋅ (−3) + (−7) ⋅ 6 + 9 ⋅ 4   − 9 
      

 − 4 2  x   − 4 x + 2 y 
(b)    =  
 3 8  y   3 x + 8 y 
    

Multiplicative Identity For a given any integer n , the n × n matrix

1 0 0  0
 
0 1 0  0
 
I = 0 0 1  0
 

     
 
0 0 0  1

is called the multiplicative identity matrix. If A is a matrix of order n × n , then it can be

verified that I ⋅ A = A ⋅ I = A
1 0 0
1 0  
Example I =   , I =  0 1 0  are identity matrices of orders 2 x 2 and 3 x 3
0 1 0 0 1
 
9 − 2
respectively and If B =   then we can easily prove that BI = IB = B
6 8 
 

©Virtual University Of Pakistan 15

 2-Introduction to Matrices VU

Zero Matrix or Null matrix A matrix whose all entries are zero is called zero matrix or null
matrix and it is denoted by O .
0 0
0 0 0  
For example O=   ; O=   ; O = 0 0

0 0 0  
    0 0
 
and so on. If A and O are the matrices of same orders, then A + O = O + A = A

Associative Law The matrix multiplication is associative. This means that if A, B and
C are m × p , p × r and r × n matrices, then A( BC ) = ( AB)C
The result is a m × n matrix. This result can be verified by taking any three matrices which
are confirmable for multiplication.

Distributive Law If B and C are matrices of order r × n and A is a matrix of order m × r ,

then the distributive law states that
A( B + C ) = AB + AC
Furthermore, if the product ( A + B)C is defined, then
( A + B )C = AC + BC
Remarks
It is important to note that some rules arithmetic for real numbers  do not carry over the
matrix arithmetic.
For example, ∀a, b, c and d ∈ 
i) if ab = cd and a ≠ 0 , then b = c (Law of Cancellation)
ii) if ab = 0 , then least one of the factors a or b (or both) are zero.
However the following examples shows that the corresponding results are not true in case of
matrices.
Example
0 1  1 1  2 5 1 7 
= Let A =  , B =  , C   and D =   , then one can easily check that
0 2  3 4  3 4 0 0 
3 4
= AC
AB =   . But B ≠ C .
6 8 
0 0 
Similarly neither A nor B are zero matrices but AD =  
0 0 
1 0 
But if D is diagonal say D =   , then AD ≠ DA .
0 7 
Determinant of a Matrix Associated with every square matrix A of constants, there is a
number called the determinant of the matrix, which is denoted by det( A) or A . There is a
special way to find the determinant of a given matrix.

©Virtual University Of Pakistan 16

2-Introduction to Matrices VU

 3 6 2
 
Example 6 Find the determinant of the following matrix A = 2 5 1 

 
 −1 2 4
 
Solution The determinant of the matrix A is given by
3 6 2

det( A) = 2 5 1

−1 2 4
We expand the det( A) by first row, we obtain
3 6 2
5 1 2 1 2 5
det( A) = 2 5 1 =3 -6 +2
2 4 −1 4 −1 2
−1 2 4
or det( A) = 3(20 - 2) - 6(8 + 1) + 2(4 + 5) = 18

Transpose of a Matrix The transpose of m × n matrix A is denoted by Atr and it is

obtained by interchanging rows of A into its columns. In other words, rows of A become the
columns of Atr . Clearly Atr is n × m matrix.

 a11 a21  am1 

 a11 a12  a1n  a
a  a22  am 2 
 a  a2n  A =  12
tr
If A= 21 22
, then      
       
   a1n a2 n  amn 
 am1 am 2  amn 

Since order of the matrix A is m × n , the order of the transpose matrix Atr is n × m .

Properties of the Transpose

The following properties are valid for the transpose;

• The transpose of the transpose of a matrix is the matrix itself:`

• The transpose of a matrix times a scalar (k) is equal to the constant times the
transpose of the matrix: (kA)T = kAT
• The transpose of the sum of two matrices is equivalent to the sum of their
transposes: ( A + B)T =AT + BT
• The transpose of the product of two matrices is equivalent to the product of their
transposes in reversed order: ( AB)T = BT AT
• The same is true for the product of multiple matrices: ( ABC )T = C T BT AT

©Virtual University Of Pakistan 17

2-Introduction to Matrices VU

 3 6 2
   3 2 −1
   
Example 7 (a) The transpose of matrix A = 2 5 1 is A =  6 5 2 
T
 
 −1 2 4 2 1 4 
   
5
 
(b) If X =  0  , then X T = [5 0 3]
 
 3
 

Multiplicative Inverse Suppose that A is a square matrix of order n × n . If there exists an

n × n matrix B such that AB = BA = I , then B is said to be the multiplicative inverse of the
matrix A and is denoted by B = A −1 .
1 4 
For example: If A =   then the matrix B =  5 −2  is multiplicative inverse of A
 2 10   
   −1 1/ 2 
 1 4   5 −2   1 0 
because AB =   =  =I
 2 10   −1 1/ 2   0 1 
Similarly we can check that BA = I

Singular and Non-Singular Matrices A square matrix A is said to be a non-singular

matrix if det( A) ≠ 0 , otherwise the square matrix A is said to be singular. Thus for a
singular matrix A we must have det( A) = 0

2 3 −1 
Example: A = 1 1 0 
 2 −3 5 

A = 2(5 − 0) − 3(5 − 0) − 1(−3 − 2)

= 10 − 15 + 5 = 0

which means that A is singular.

Minor of an element of a matrix

Let A be a square matrix of order n x n. Then minor M ij of the element aij ∈ A is the
determinant of (n − 1) × (n − 1) matrix obtained by deleting the ith row and jth column
from A .

©Virtual University Of Pakistan 18

2-Introduction to Matrices VU

2 3 −1 
Example If A = 1 1 0  is a square matrix. The Minor of 3 ∈ A is denoted by
 2 −3 5 
1 0
M 12 and is defined to be M 12 = = 5-0 = 5
2 5
Cofactor of an element of a matrix

Let A be a non singular matrix of order n × n and let C ij denote the cofactor (signed minor)
of the corresponding entry aij ∈ A , then it is defined to be Cij = (−1) i + j M ij
2 3 −1 
Example If A = 1 1 0  is a square matrix. The cofactor of 3 ∈ A is denoted by
 2 −3 5 
1 0
C12 and is defined to be C12 = = (−1)1+ 2 = - (5 - 0) = -5
2 5

Theorem If A is a square matrix of order n × n then the matrix has a multiplicative inverse
A −1 if and only if the matrix A is non-singular.

1
Theorem Then inverse of the matrix A is given by A −1 = (Cij ) tr
det( A)

1. For further reference we take n = 2 so that A is a 2 × 2 non-singular matrix given by

 a11 a12 
A= 
a 
 21 a 22 
Therefore C11 = a 22 , C12 = −a 21 , C 21 = −a12 and C 22 = a11 . So that

tr
−1 1  a 22 − a 21  1  a 22 − a12 
A = =
det( A)  − a12 a11  det( A)  − a 21 a11 

 a11 a12 a13 


2. For a 3 × 3 non-singular matrix A= a21

 a22 a23 
a a33 
 31 a32

©Virtual University Of Pakistan 19

2-Introduction to Matrices VU

a 22 a 23 a 21 a 23 a 21 a 22
C11 = , C12 = − , C 13 = and so on.
a32 a33 a31 a33 a31 a32

 C11 C21 C31 

1  
Therefore, inverse of the matrix A is given by A−1 =
det A 
C12 C22 C32  .
 C13 C23 C33 

1 4 
Example 8 Find, if possible, the multiplicative inverse for the matrix A =  .
 2 10 
 
1 4
Solution The matrix A is non-singular because det( A) = = 10 - 8 = 2
2 10
1  10 − 4   5 − 2 
Therefore, A −1 exists and is given by A −1 =  =
2  − 2 1   − 1 1 / 2 
 1 4  5 − 2   5 − 4 − 2 + 2   1 0 
Check AA −1 =   = = =I
 2 10  − 1 1 / 2  10 − 10 − 4 + 5   0 1 
      

 5 − 2  1 4   5 − 4 20 − 20   1 0 
AA −1 =   = = =I
 − 1 1 / 2  2 10   − 1 + 1 − 4 + 5   0 1 
      

Example 9 Find, if possible, the multiplicative inverse of the following matrix

 2 2
A= 
 3 3
 
Solution The matrix is singular because
2 2
det( A) = = 2⋅3− 2⋅3 = 0
3 3
Therefore, the multiplicative inverse A −1 of the matrix does not exist.

Example 10 Find the multiplicative inverse for the following matrix

 2 2 0
A=  −2 1 1  .
 3 0 1
 

©Virtual University Of Pakistan 20

2-Introduction to Matrices VU

2 2 0
Solution Since det( A) = − 2 1 1 = 2(1 − 0) − 2(−2 − 3) + 0(0 − 3) = 12 ≠ 0

3 0 1
Therefore, the given matrix is non singular. So, the multiplicative inverse A −1 of the matrix
A exists. The cofactors corresponding to the entries in each row are

1 1 −2 1 −2 1
C11 = = 1, C12 = − = 5, C13 = = −3
0 1 3 1 3 0
2 0 2 0 2 2
C 21 = − = −2, C 22 = = 2, C 23 = − =6
0 1 3 1 3 0
2 0 2 0 2 2
C31 = = 2, C32 = − = −2, C33 = =6
1 1 −2 1 −2 1
 1 − 2 2   1 / 12 − 1 / 6 1 / 6 
1 
−1
  
Hence A =  5 2 − 2  =  5 / 12 1 / 6 − 1 / 6 
12 
− 3 6 6   − 1 / 4 1 / 2 1 / 2 
We can also verify that A ⋅ A −1 = A −1 ⋅ A = I

Derivative of a Matrix of functions

Suppose that
A(t ) =  aij (t ) 
m× n
is a matrix whose entries are functions those are differentiable in a common interval, then
derivative of the matrix A(t ) is a matrix whose entries are derivatives of the corresponding
entries of the matrix A(t ) . Thus
dA  daij 
= 
dt  dt 
m× n
The derivative of a matrix is also denoted by A′(t ).

Integral of a Matrix of Functions

(
Suppose that A(t ) = aij (t ) )m× n is a matrix whose entries are functions those are continuous
on a common interval containing t , then integral of the matrix A(t ) is a matrix whose entries
are integrals of the corresponding entries of the matrix A(t ) . Thus
t
 t 
∫ A(s)ds =  ∫t0 aij (s)ds m×n
t0

©Virtual University Of Pakistan 21

2-Introduction to Matrices VU

 sin 2t 
 
Example 11 Find the derivative and the integral of the following matrix X (t ) =  e3t 
 8t − 1 
 
 
Solution The derivative and integral of the given matrix are, respectively, given by
t 
d   ∫ sin 2 sds 
 (sin 2t )   2 cos 2t  0 
 dt    t  t   −1/ 2 cos 2t + 1/ 2 
 d 3t     
X ′(t ) = (e ) = 3e 3t
and ∫ X=
( s )ds  ∫ e =3s
ds  1/ 3e3t − 1/ 3 
 dt     0   2
d   8  − 
0 t   4t t 
 (8t − 1)     
 dt   ∫ 8s − 1ds 
0 
Exercise
Write the given sum as a single column matrix
 2  − 1  3t 
     
1. 3t  t  + (t − 1) − t  − 2 4 
 − 1  3   − 5t 
     
 1 − 3 4   t   −t   2 
2.
 2 5 − 1   2t − 1  +  1  −  8 
      
 0 − 4 − 2   −t   4   − 6 
      
Determine whether the given matrix is singular or non-singular. If singular, find A− 1 .
 3 2 1
 
3. A =  4 1 0 
 −2 5 −1
 
 4 1 −1 
 
= 4. A  6 2 −3 
 
 −2 −1 2 
dX
Find
dt
1 
5. X =  2 sin 2t − 4 cos 2t 
 
 − 3 sin 2t + 5 cos 2t 
 
e 4 t
cos π t 
2 t
6. If A ( t ) = 
 2t 3t 2 − 1 
 then find (a) ∫ A(t )dt , (b) ∫ A(s)ds.
  0 0

2
 6t 2
7. Find the integral ∫ B(t )dt if B ( t ) =  
1 1/ t 4t 

©Virtual University Of Pakistan 22

3-System of Linear Equations VU

Lecture 3

Systems of Linear Equations

In this lecture we will discuss some ways in which systems of linear equations arise, how
to solve them, and how their solutions can be interpreted geometrically.

Linear Equations
We know that the equation of a straight line is written as = y mx + c , where m is the
slope of line(Tan of the angle of line with x-axis) and c is the y-intercept(the distance at
which the straight line meets y-axis from origin).
Thus a line in R2 (2-dimensions) can be represented by an equation of the form
a1 x + a2 y =
b (where a 1 , a 2 not both zero). Similarly a plane in R3 (3-dimensional space)
can be represented by an equation of the form a1 x + a2 y + a3 z = b (where a 1 , a 2 , a 3 not
all zero).

A linear equation in n variables x1, x2 ,, xn can be expressed in the form

a1 x1 + a2 x2 +  + an xn =
b (hyper plane in  n ) --------(1)

where a1, a2 ,, an and b are constants and the “a’s” are not all zero.

Homogeneous Linear equation

In the special case if b = 0, Equation (1) has the form a1 x1 + a2 x2 +  + an xn =

0 (2)
This equation is called homogeneous linear equation.

Note A linear equation does not involve any products or square roots of variables. All
variables occur only to the first power and do not appear, as arguments of trigonometric,
logarithmic, or exponential functions.

Examples of Linear Equations

(1) The equations

2 x1 + 3 x2 +=
2 x3 and =
x2 2 ( )
5 + x1 + 2 x3 are both linear
(2) The following equations are also linear
= x + 3y 7 x1 −=
2 x2 − 3 x3 + x4 0
1
2 x − y + 3 z =−1 x1 + x2 +  + xn =1

(3) The equations 3 x1 − 2 x2 = x1 x2 and x2 = 4 x1 − 6

are not linear because of the presence of x1 x2 in the first equation and x1 in the second.

©Virtual University Of Pakistan 23

3-System of Linear Equations VU

System of Linear Equations

A finite set of linear equations is called a system of linear equations or linear system. The
variables in a linear system are called the unknowns.

For example,
4 x1 − x2 + 3 x3 =−1
3 x1 + x2 + 9 x3 =−4
is a linear system of two equations in three unknowns x 1 , x 2 , and x 3 .

General System of Linear Equations

A general linear system of m equations in n-unknowns x1, x2 ,, xn can be written as
a11 x1 + a12 x2 +  + a1n xn =
b1
a21 x1 + a22 x2 +  + a2 n xn =
b2
(3)
   
am1 x1 + am 2 x2 +  + amn xn =
bm

Solution of a System of Linear Equations

A solution of a linear system in the unknowns x1, x2 ,, xn is a sequence of n numbers
s1 , s2 , , sn such that when substituted for x1, x2 ,, xn respectively, makes every
equation in the system a true statement. The set of all such solutions {s1 , s2 , , sn } of a
linear system is called its solution set.

Linear System with Two Unknowns

When two lines intersect in R2, we get system of linear equations with two unknowns

a1 x + b1 y =
c1
For example, consider the linear system
a2 x + b2 y =
c2

The graphs of these equations are straight lines in the xy-plane, so a solution (x, y) of this
system is infact a point of intersection of these lines.

Note that there are three possibilities for a pair of straight lines in xy-plane:

1. The lines may be parallel and distinct, in which case there is no intersection and
consequently no solution.
2. The lines may intersect at only one point, in which case the system has exactly
one solution.
3. The lines may coincide, in which case there are infinitely many points of
intersection (the points on the common line) and consequently infinitely many
solutions.

©Virtual University Of Pakistan 24

3-System of Linear Equations VU

Consistent and inconsistent system

A linear system is said to be consistent if it has at least one solution and it is called
inconsistent if it has no solutions.

Thus, a consistent linear system of two equations in two unknowns has either one
solution or infinitely many solutions – there is no other possibility.

Example consider the system of linear equations in two variables

x1 − 2 x2 = −1, − x1 + 3 x2 =3
Solve the equation simultaneously:
Adding both equations we get x2 = 2, Put x2 = 2 in any one of the above equation we
get x1 = 3 . So the solution is the single point (3, 2). See the graph of this linear system

x2

x1
l2 3

l1 (a)

This system has exactly one solution

See the graphs to the following linear systems:

(a) x1 − 2 x2 =
−1 (b) x1 − 2 x2 =
−1
− x1 + 2 x2 =3 − x1 + 2 x2 =1

x2 x2

2
2

x1
l2 3 3
l1
l1 (a) (b)

(a) No solution. (b) Infinitely many solutions.

©Virtual University Of Pakistan 25

3-System of Linear Equations VU

Linear System with Three Unknowns

Consider r a linear system of three equations in three unknowns:

a1 x + b1 y + c1 z =
d1
a2 x + b2 y + c1 z =
d2
a3 x + b3 y + c3 z =
d3

In this case, the graph of each equation is a plane, so the solutions of the system, If any
correspond to points where all three planes intersect; and again we see that there are only
three possibilities – no solutions, one solution, or infinitely many solutions as shown in
figure.

Theorem 1 Every system of linear equations has zero, one or infinitely many solutions;
there are no other possibilities.

x− y =1
Example 1 Solve the linear system
2x + y =6

Solution
7
Adding both equations, we get x = . Putting this value of x in 1st equation, we
3
4 7 4
get y = . Thus, the system has the unique solution= x = ,y .
3 3 3

Geometrically, this means that the lines represented by the equations in the system
7 4
intersect at a single point  ,  and thus has a unique solution.
3 3

x+ y = 4
Example 2 Solve the linear system
3x + 3 y =
6
Solution
Multiply first equation by 3 and then subtract the second equation from this. We obtain
0=6
This equation is contradictory.

©Virtual University Of Pakistan 26

3-System of Linear Equations VU

Geometrically, this means that the lines corresponding to the equations in the original
system are parallel and distinct. So the given system has no solution.

4x − 2 y =1
Example 3 Solve the linear system
16 x − 8 y =
4

Solution

Multiply the first equation by -4 and then add in second equation.

−16 x + 8 y =
−4
16 x − 8 y =4
0 =0
Thus, the solutions of the system are those values of x and y that satisfy the single
equation 4 x − 2 y =
1

Geometrically, this means the lines corresponding to the two equations in the original
system coincide and thus the system has infinitely many solutions.

Parametric Representation

It is very convenient to describe the solution set in this case is to express it

parametrically. We can do this by letting y = t and solving for x in terms of t, or by
letting x = t and solving for y in terms of t.

The first approach yields the following parametric equations (by taking y=t in the
equation 4 x − 2 y =
1)

4 x − 2=
t 1, y= t
1 1
x=+ t, y =
t
4 2

We can now obtain some solutions of the above system by substituting some numerical
values for the parameter.
1 3
Example For t = 0 the solution is ( , 0). For t = 1, the solution is ( ,1) and for t = −1
4 4
1
the solution is (− , −1) etc.
4

x − y + 2z =
5
Example 4 Solve the linear system 2x − 2 y + 4z =10
3x − 3 y + 6 z =
15

©Virtual University Of Pakistan 27

3-System of Linear Equations VU

Solution
Since the second and third equations are multiples of the first.

Geometrically, this means that the three planes coincide and those values of x, y and z
that satisfy the equation x − y + 2 z =
5 automatically satisfy all three equations.

We can express the solution set parametrically as

x = 5 + t1 − 2t2 , y = t1 , z = t2
Some solutions can be obtained by choosing some numerical values for the parameters.

For example if we take y= t1= 2 and z= t2= 3 then

x = 5 + t1 − 2t2
= 5 + 2 − 2(3)
=1
Put these values of x, y, and z in any equation of linear system to verify

x − y + 2z = 5
1 − 2 + 2(3) =5
1− 2 + 6 = 5
5=5

Hence x = 1, y = 2, z = 3 is the solution of the system. Verified.

Matrix Notation

The essential information of a linear system can be recorded compactly in a rectangular

array called a matrix.

x1 − 2 x2 + x3 =
0
Given the system 2 x2 − 8 x3 =8
−4 x1 + 5 x2 + 9 x3 =
−9
 1 −2 1 
With the coefficients of each variable aligned in columns, the matrix  0 2 −8
 −4 5 9 
is called the coefficient matrix (or matrix of coefficients) of the system.

An augmented matrix of a system consists of the coefficient matrix with an added column
containing the constants from the right sides of the equations. It is always denoted by A b

©Virtual University Of Pakistan 28

3-System of Linear Equations VU

 1 −2 1 0 
A b =  0 2 −8 8 
 −4 5 9 −9 

Solving a Linear System

In order to solve a linear system, we use a number of methods. 1st of them is given
below.

Successive elimination method In this method the x1 term in the first equation of a
system is used to eliminate the x1 terms in the other equations. Then we use the x2 term
in the second equation to eliminate the x2 terms in the other equations, and so on, until
we finally obtain a very simple equivalent system of equations.

x1 − 2 x2 + x3 =
0
Example 5 Solve 2 x2 − 8 x3 =8
−4 x1 + 5 x2 + 9 x3 =
−9

Solution We perform the elimination procedure with and without matrix notation,
and place the results side by side for comparison:
x1 − 2 x2 + x3 = 0  1 −2 1 0 
2 x2 − 8 x3 = 8  0 2 −8 8 
 
−4 x1 + 5 x2 + 9 x3 =
−9  −4 5 9 −9 

To eliminate the x1 term from third equation add 4 times equation 1 to equation 3,
4 x1 − 8 x2 + 4 x3 =
0
−4 x1 + 5 x2 + 9 x3 =
−9
−3 x2 + 13 x3 =
−9

The result of the calculation is written in place of the original third equation:
x1 − 2 x2 + x3 = 0 1 −2 1 0 
2 x2 − 8 x3 = 8 0 2 −8 8 
 
−3 x2 + 13 x3 = −9 0 −3 13 −9 

Next, multiply equation 2 by ½ in order to obtain 1 as the coefficient for x2

©Virtual University Of Pakistan 29

3-System of Linear Equations VU

x1 − 2 x2 + x3 =
0 1 −2 1 0 
x2 − 4 x3 =
4 0 1 −4 4 
 
−3 x2 + 13 x3 =
−9 0 −3 13 −9 

To eliminate the x2 term from third equation add 3 times equation 2 to equation 3,

The new system has a triangular form

x1 − 2 x2 + x3 =0 1 −2 1 0 
 
x2 − 4 x3 =
4 0 1 −4 4 
x3 = 3 0 0 1 3 

Now using 3rd equation eliminate the x 3 term from first and second equation i.e. multiply
3rd equation with 4 and add in second equation. Then subtract the third equation from first
equation we get

x1 − 2 x2 =
−3 1 −2 0 −3
x2 = 16 0 1 0 16 
 
x3 = 3 0 0 1 3 

Adding 2 times equation 2 to equation 1, we obtain the result

 x1 = 29 1 0 0 29 
 0 1 0 16 
 x2 = 16  
x = 3 0 0 1 3 
 3

This completes the solution.

Our work indicates that the only solution of the original system is (29, 16, 3).

To verify that (29, 16, 3) is a solution, substitute these values into the left side of the
original system for x 1 , x 2 and x 3 and after computing, we get

(29) – 2(16) + (3) = 29 – 32 + 3 = 0

2(16) – 8(3) = 32 – 24 = 8
–4(29) + 5(16) + 9(3) = –116 + 80 + 27 = –9

The results agree with the right side of the original system, so (29, 16, 3) is a solution of
the system.

©Virtual University Of Pakistan 30

3-System of Linear Equations VU

This example illustrates how operations on equations in a linear system correspond to

operations on the appropriate rows of the augmented matrix. The three basic operations
listed earlier correspond to the following operations on the augmented matrix.

Elementary Row Operations

1. (Replacement) Replace one row by the sum of itself and a nonzero multiple of
another row.
2. (Interchange) Interchange two rows.
3. (Scaling) Multiply all entries in a row by a nonzero constant.

Row equivalent matrices

A matrix B is said to be row equivalent to a matrix A of the same order if B can be

obtained from A by performing a finite sequence of elementary row operations of A.
If A and B are row equivalent matrices, then we write this expression mathematically as
A  B.
 1 −2 1 0  1 −2 1 0 
For example  0 2 −8 8   0 2 −8 8  are row equivalent matrices
 −4 5 9 −9  0 −3 13 −9 
because we add 4 times of 1st row in 3rd row in 1st matrix.

Note If the augmented matrices of two linear systems are row equivalent, then the two
systems have the same solution set.

Row operations are extremely easy to perform, but they have to be learnt and practice.

Two Fundamental Questions

1. Is the system consistent; that is, does at least one solution exist?
2. If a solution exists is it the only one; that is, is the solution unique?

We try to answer these questions via row operations on the augmented matrix.

Example 6 Determine if the following system of linear equations is consistent

x1 − 2 x2 + x3 =0
2 x2 − 8 x3 =8
−4 x1 + 5 x2 + 9 x3 =
−9

Solution

First obtain the triangular matrix by removing x 1 and x 2 term from third equation and
removing x 2 from second equation.

©Virtual University Of Pakistan 31

3-System of Linear Equations VU

First divide the second equation by 2 we get

x1 − 2 x2 + x3 =
0  1 −2 1 0 
x2 − 4 x3 =4  0 1 −4 4 
 
−4 x1 + 5 x2 + 9 x3 =
−9  −4 5 9 −9 

Now multiply equation 1 with 4 and add in equation 3 to eliminate x 1

from third equation.

x1 − 2 x2 + x3 =
0 1 −2 1 0 
x2 − 4 x3 =4 0 1 −4 4 
 
− 3 x2 + 13 x3 =
−9 0 −3 13 −9 

Now multiply equation 2 with 3 and add in equation 3 to eliminate x 2

from third equation.

x1 − 2 x2 + x3 =
0 1 −2 1 0 
 
x2 − 4 x3 =
4 0 1 −4 4 
x3 = 3 0 0 1 3 

Put value of x 3 in second equation we get

x2 − 4(3) =4
x2 = 16

Now put these values of x 2 and x 3 in first equation we get

x1 − 2(16) + 3 =0

x1 = 29

So a solution exists and the system is consistent and has a unique solution.

Example 7 Solve if the following system of linear equations is consistent.

x2 − 4 x3 =8
2 x1 − 3 x2 + 2 x3 =
1
5 x1 − 8 x2 + 7 x3 =
1

©Virtual University Of Pakistan 32

3-System of Linear Equations VU

Solution The augmented matrix is

 0 1 −4 8
 2 −3 2 1
 
 5 −8 7 1

To obtain x 1 in the first equation, interchange rows 1 and 2:

 2 −3 2 1
 0 1 −4 8
 
 5 −8 7 1

To eliminate the 5x 1 term in the third equation, add –5/2 times row 1 to row 3:
 2 −3 2 1 
0 1 −4 8 

 0 −1/ 2 2 −3 / 2 

Next, use the x 2 term in the second equation to eliminate the –(1/2) x 2 term from the
third equation. Add ½ times row 2 to row 3:
 2 −3 2 1 
 0 1 −4 8 
 
 0 0 0 5 / 2 

The augmented matrix is in triangular form.

To interpret it correctly, go back to equation notation:

2 x1 − 3 x2 + 2 x3 =
1
x2 − 4 x3 =
8
0 = 2.5
There are no values of x 1 , x 2 , x 3 that will satisfy because the equation 0 = 2.5 is never
true.
Hence original system is inconsistent (i.e., has no solution).

©Virtual University Of Pakistan 33

3-System of Linear Equations VU

Exercises

1. State in words the next elementary “row” operation that should be performed on the
system in order to solve it. (More than one answer is possible in (a).)

a. x1 + 4 x2 − 2 x3 + 8 x4 =
12 b. x1 − 3 x2 + 5 x3 − 2 x4 =0
x2 − 7 x3 + 2 x4 =
−4 x2 + 8 x3 =
−4
5 x3 − x4 =7 2 x3 = 7
x3 + 3 x4 =
−5 x4 = 1

2. The augmented matrix of a linear system has been transformed by row operations into
the form below. Determine if the system is consistent.

1 5 2 −6 
0 4 −7 2 
 
0 0 5 0 

3. Is (3, 4, –2) a solution of the following system?

5 x1 − x2 + 2 x3 =7
−2 x1 + 6 x2 + 9 x3 =0
−7 x1 + 5 x2 − 3 x3 =
−7

4. For what values of h and k is the following system consistent?

2 x1 − x2 =
h
−6 x1 + 3 x2 =
k

Solve the systems in the exercises given below;

x2 + 5 x3 =
−4 x1 − 5 x2 + 4 x3 =
−3
5. x1 + 4 x2 + 3 x3 =
−2 6. 2 x1 − 7 x2 + 3 x3 =
−2
2 x1 + 7 x2 + x3 =
−1 2 x1 − x2 − 7 x3 =
1

x1 + 2 x2 =
4 2 x1 − 4 x3 =
−10
7. x1 − 3 x2 − 3 x3 =
2 8. x2 + 3 x3 =
2
x2 + x3 =
0 3 x1 + 5 x2 + 8 x3 =
−6

©Virtual University Of Pakistan 34

3-System of Linear Equations VU

Determine the value(s) of h such that the matrix is augmented matrix of a consistent
linear system.

 1 −3 h   1 h −2 
9.   10.  
 −2 6 −5  −4 2 10 

Find an equation involving g, h, and that makes the augmented matrix correspond to a
consistent system.

 1 −4 7 g   2 5 −3 g 
11.  0 3 −5 h  12.  4 7 −4 h 
 −2 5 −9 k   −6 −3 1 k 

Find the elementary row operations that transform the first matrix into the second, and
then find the reverse row operation that transforms the second matrix into first.

1 3 −1 1 3 −1  0 5 −3  1 5 −2 

13. 0 2 −4  , 0 1 −2  14. 1 5 −2  ,  0 5 −3
0 −3 4  0 −3 4   2 1 8   2 1 8 

1 3 −1 5  1 3 −1 5 
15. 0 1 −4 2  , 0 1 −4 2 
0 2 −5 −1 0 0 3 −5

©Virtual University Of Pakistan 35

4-Row Reduction and Echelon Forms VU

Lecture 4

Row Reduction and Echelon Forms

To analyze system of linear equations, we shall discuss how to refine the row reduction
algorithm. While applying the algorithm to any matrix, we begin by introducing a non
zero row or column (i.e. contains at least one nonzero entry) in a matrix,

Echelon form of a matrix

A rectangular matrix is in echelon form (or row echelon form) if it has the following three
properties:

1. All nonzero rows are above any rows of all zeros

2. Each leading entry of a row is in a column to the right of the leading entry of the
row above it.
3. All entries in a column below a leading entry are zero.

Reduced Echelon Form of a matrix

If a matrix in echelon form satisfies the following additional conditions, then it is in

reduced echelon form (or reduced row echelon form):

4. The leading entry in each nonzero row is 1.

5. Each leading 1 is the only nonzero entry in its column.

Examples of Echelon Matrix form

The following matrices are in echelon form. The leading entries (  ) may have any
nonzero value; the started entries (*) may have any values (including zero).

 2 −3 2 1 
1.  0 1 −4 8 

 0 0 0 5 / 2 

©Virtual University Of Pakistan 36

4-Row Reduction and Echelon Forms VU

0  * * * * * * * *
  * * * 0
0  * *   0 0  * * * * * *
2.   3. 0 0 0 0  * * * * *
0 0 0 0   
  0 0 0 0 0  * * * *
0 0 0 0  0 0 0 0 0 0 0 0  *
1 4 −3 7  1 1 0 
4. 0 1 6 2  5. 0 1 0 
0 0 1 5  0 0 0 

0 1 2 6 0 
6. 0 0 1 −1 0 
0 0 0 0 1 

Examples of Reduced Echelon Form

The following matrices are in reduced echelon form because the leading entries are 1’s,
and there are 0’s below and above each leading 1.
1 0 0 29 
1. 0 1 0 16 
0 0 1 1 
0 1 * 0 0 0 * * 0 *
1 * 0 *
0 *
0 1 * *   0 0 1 0 0 * * 0
2.  3. 0 0 0 0 1 0 * * 0 *
0 0 0 0  
  0 0 0 0 0 1 * * 0 *
0 0 0 0
0 0 0 0 0 0 0 0 1 *
0 1 −2 0 1 
1 0 0 4  1 0 0  0 0 0 1 3
4. 0 1 0 7 
 5. 0 1 0 
 6. 
0 0 0 0 0
0 0 1 −1 0 0 1   
0 0 0 0 0

Note A matrix may be row reduced into more than one matrix in echelon form, using
different sequences of row operations. However, the reduced echelon form obtained from
a matrix, is unique.

Theorem 1 (Uniqueness of the Reduced Echelon Form) Each matrix is row equivalent
to one and only one reduced echelon matrix.

©Virtual University Of Pakistan 37

4-Row Reduction and Echelon Forms VU

Pivot Positions
A pivot position in a matrix A is a location in A that corresponds to a leading entry in an
echelon form of A.

Note When row operations on a matrix produce an echelon form, further row operations
to obtain the reduced echelon form do not change the positions of the leading entries.

Pivot column

A pivot column is a column of A that contains a pivot position.

Example 2 Reduce the matrix A below to echelon form, and locate the pivot columns
 0 −3 −6 4 9 
 −1 −2 −1 3 1 
A= 
 −2 −3 0 3 −1
 
1 4 5 −9 −7 

Solution Leading entry in first column of above matrix is zero which is the pivot
position. A nonzero entry, or pivot, must be placed in this position. So interchange first
and last row.

1 ↵Pivot 4 5 −9 −7 
 
 −1 −2 −1 3 1 
 −2 −3 0 3 −1 
 
 0 −3 −6 4 9 

Pivot Column

Since all entries in a column below a leading entry should be zero. For this add row 1 in
row 2, and multiply row 1 by 2 and add in row 3.
Pivot
 1 4 5 −9 −7 
 0 −6 −6 
 2 4 R1 + R2
 0 5 10 −15 −15
  2 R1 + R3
 0 − 3 − 6 4 9 
Next pivot column

Add –5/2 times row 2 to row 3, and add 3/2 times row 2 to row 4.

©Virtual University Of Pakistan 38

4-Row Reduction and Echelon Forms VU

 1 4 5 −9 −7  5
 − R2 + R3
 0 2 4 −6 −6  2
 0 0 0 0 0 3
  R2 + R4
 0 0 0 −5 0  2

Interchange rows 3 and 4, we can produce a leading entry in column 4.

Pivot
 1 4 5 −9 −7   * * * *
 0 2 4 
−6 −6  0  * * * 
 General form 
 0 0 0 −5 0  0 0 0  *
   
 0 0 0 0 0 0 0 0 0 0
Pivot column

This is in echelon form and thus columns 1, 2, and 4 of A are pivot columns.

Pivot positions
 0 −3 −6 4 9 
 −1 −2 −1 3 1 
 
 −2 −3 0 3 −1
 
1 4 5 −9 −7 
Pivot columns

Pivot element

A pivot is a nonzero number in a pivot position that is used as needed to create zeros via
row operations

The Row Reduction Algorithm consists of four steps, and it produces a matrix in
echelon form. A fifth step produces a matrix in reduced echelon form.

The algorithm is explained by an example.

Example 3 Apply elementary row operations to transform the following matrix first
into echelon form and then into reduced echelon form.

0 3 −6 6 4 −5 
3 −7 8 −5 8 9 

 3 −9 12 −9 6 15 

©Virtual University Of Pakistan 39

4-Row Reduction and Echelon Forms VU

Solution

STEP 1 Begin with the leftmost nonzero column. This is a pivot column. The pivot
position is at the top.

0 3 −6 6 4 −5 
3 −7 8 −5 8 9 

 3 −9 12 −9 6 15 
Pivot column

STEP 2 Select a nonzero entry in the pivot column as a pivot. If necessary, interchange
rows to move this entry into the pivot position

Interchange rows 1 and 3. (We could have interchanged rows 1 and 2 instead.)

Pivot
3 −9 12 −9 6 15 
3 −7 8 −5 8 9 

 0 3 −6 6 4 −5 

STEP 3 Use row replacement operations to create zeros in all positions below the pivot

Subtract Row 1 from Row 2. i.e. R2 − R1

Pivot
 3 −9 12 −9 6 15 
 0 2 −4 4 2 −6 

 0 3 −6 6 4 −5

STEP 4 Cover (or ignore) the row containing the pivot position and cover all rows, if
any, above it. Apply steps 1 –3 to the sub-matrix, which remains. Repeat the process until
there are no more nonzero rows to modify.

With row 1 covered, step 1 shows that column 2 is the next pivot column; for step 2,
we’ll select as a pivot the “top” entry in that column.

©Virtual University Of Pakistan 40

4-Row Reduction and Echelon Forms VU

Pivot
3 −9 12 −9 6 15 
0 2 −4 4 2 −6 

 0 3 −6 6 4 −5
Next pivot column

According to step 3 “All entries in a column below a leading entry are zero”. For this
subtract 3/2 time R 2 from R 3

3 −9 12 −9 6 15 
0 2 −4 4 2 −6  R3 − R2
3
 2
 0 0 0 0 1 4 

When we cover the row containing the second pivot position for step 4, we are left with a
new sub matrix having only one row:

3 −9 12 −9 6 15 
0 2 −4 4 2 −6 

 0 0 0 0 1 4 
Pivot
This is the Echelon form of the matrix.
To change it in reduced echelon form we need to do one more step:

STEP 5 Make the leading entry in each nonzero row 1. Make all other entries of that
column to 0.

Divide first Row by 3 and 2nd Row by 2

1 −3 4 −3 2 5
0 1 −2 2 1 −3
1
R2 ,
1
R1
 2 3
 0 0 0 0 1 4 

Multiply second row by 3 and then add in first row.

1 0 −2 3 5 −4 
0 1 −2 2 1 −3 3R2 + R1

 0 0 0 0 1 4 

Subtract row 3 from row 2, and multiply row 3 by 5 and then subtract it from first row

©Virtual University Of Pakistan 41

4-Row Reduction and Echelon Forms VU

 1 0 −2 3 0 −24 
 0  R2 − R3
 1 − 2 2 0 − 7  R1 − 5 R3
 0 0 0 0 1 4 
This is the matrix is in reduced echelon form.

Solutions of Linear Systems

When this algorithm is applied to the augmented matrix of the system it gives solution set
of linear system.
Suppose, for example, that the augmented matrix of a linear system has been changed
into the equivalent reduced echelon form
1 0 −5 1 
0 1 1 4 
 
0 0 0 0 

There are three variables because the augmented matrix has four columns. The associated
system of equations is
x1 −5 x3 = 1
x2 + x3 =
4 (1)
0=0 which means x 3 is free

The variables x 1 and x 2 corresponding to pivot columns in the above matrix are called
basic variables. The other variable, x 3 is called a free variable.

Whenever a system is consistent, the solution set can be described explicitly by solving
the reduced system of equations for the basic variables in terms of the free variables. This
operation is possible because the reduced echelon form places each basic variable in one
and only one equation.

In (4), we can solve the first equation for x 1 and the second for x 2 . (The third equation is
ignored; it offers no restriction on the variables.)
x1 = 1 + 5 x3
x2= 4 − x3 (2)
x3 is free

By saying that x 3 is “free”, we mean that we are free to choose any value for x 3 . When
x 3 = 0, the solution is (1, 4, 0); when x 3 = 1, the solution is (6, 3, 1 etc).

Note The solution in (2) is called a general solution of the system because it gives an
explicit description of all solutions.

©Virtual University Of Pakistan 42

4-Row Reduction and Echelon Forms VU

Example 4 Find the general solution of the linear system whose augmented matrix has
 1 6 2 −5 −2 −4 
been reduced to  0 0 2 −8 −1 3 
 
 0 0 0 0 1 7 

Solution The matrix is in echelon form, but we want the reduced echelon form
before solving for the basic variables. The symbol “~” before a matrix indicates that the
matrix is row equivalent to the preceding matrix.

1 6 2 −5 −2 −4 
0 0 2 −8 −1 3 

 0 0 0 0 1 7 
By R1 + 2 R3 and R2 + R3 We get

1 6 2 −5 0 10 
  0 0 2 −8 0 10 
 0 0 0 0 1 7 
1
By R2 we get
2

1 6 2 −5 0 10 
  0 0 1 −4 0 5 
 0 0 0 0 1 7 

By R1 − 2 R2 we get

1 6 0 3 0 0
  0 0 1 −4 0 5 
 0 0 0 0 1 7 

The matrix is now in reduced echelon form.

The associated system of linear equations now is
x1 + 6 x2 + 3 x4 = 0
x3 − 4 x4 =
5 (6)
x5 = 7

The pivot columns of the matrix are 1, 3 and 5, so the basic variables are x 1 , x 3 , and x 5 .
The remaining variables, x 2 and x 4 , must be free.

©Virtual University Of Pakistan 43

4-Row Reduction and Echelon Forms VU

Solving for the basic variables, we obtain the general solution:

x1 = -6x 2 –3x 4
x2 is free
x3 = 5 + 4x 4 (7)
x4 is free
x5 =7

Note that the value of x 5 is already fixed by the third equation in system (6).

Exercise

1. Find the general solution of the linear system whose augmented matrix is

1 −3 −5 0 
0 1 1 3
 

2. Find the general solution of the system

x1 − 2 x2 − x3 + 3 x4 =
0
−2 x1 + 4 x2 + 5 x3 − 5 x4 =
3
3 x1 − 6 x2 − 6 x3 + 8 x4 =
2

Find the general solutions of the systems whose augmented matrices are given in
Exercises 3-12

1 0 2 5   1 −3 0 −5
3. 2 0 3 6 4.  −3 7 0 9 
   

0 3 6 9 1 3 −3 7 
5.  −1 1 −2 −1 6. 3 9 −4 1 
   

 1 2 −7  1 2 4
   
7.  −1 −1 1  8.  −2 −3 −5 
2 1 5  2 1 −1 
   

©Virtual University Of Pakistan 44

4-Row Reduction and Echelon Forms VU

1 0 −9 0 4
 2 −4 3   
  0 1 3 0 −1 
9.  −6 12 −9  10.
0 0 0 1 −7 
 4 −8 6   
 
0 0 0 0 1

 1 −2 0 0 7 −3  1 0 −5 0 −8 3
   
0 1 0 0 −3 1 0 1 4 −1 0 6
11. 12.
0 0 0 1 5 −4  0 0 0 0 1 0
   
0 0 0 0 0 0 0 0 0 0 0 0

Determine the value(s) of h such that the matrix is the augmented matrix of a consistent
linear system.

1 4 2  1 h 3
13.   14.  
 −3 h −1  2 8 1

Choose h and k such that the system has (a) no solution, (b) a unique solution, and (c)
many solutions. Give separate answer for each part.

15. x 1 + hx 2 = 1 16. x 1 - 3x 2 = 1
2x 1 + 3x 2 = k 2x 1 + hx 2 = k

©Virtual University Of Pakistan 45

05-Null Spaces, Column Spaces and Linear Transformation VU

Lecture 05

Null Spaces, Column Spaces, and Linear Transformations

Subspaces arise in as set of all solutions to a system of homogenous linear equations as

the set of all linear combinations of certain specified vectors. In this lecture, we compare
and contrast these two descriptions of subspaces, allowing us to practice using the
concept of a subspace. In applications of linear algebra, subspaces of Rn usually arise in
one of two ways:
 as the set of all solutions to a system of homogeneous linear equations or
 as the set of all linear combinations of certain specified vectors.
Our work here will provide us with a deeper understanding of the relationships between
the solutions of a linear system of equations and properties of its coefficient matrix.

Null Space of a Matrix:

Consider the following system of homogeneous equations:

x1 − 3 x2 − 2 x3 =
0
(1)
−5 x1 + 9 x2 + x3 =
0
In matrix form, this system is written as Ax = 0, where
 1 -3 -2 
A=   (2)
 -5 9 1 
Recall that the set of all x that satisfy (1) is called the solution set of the system (1). Often
it is convenient to relate this set directly to the matrix A and the equation Ax = 0. We call
the set of x that satisfy Ax = 0 the null space of the matrix A. The reason for this name is
that if matrix A is viewed as a linear operator that maps points of some vector space V
into itself, it can be viewed as mapping all the elements of this solution space of AX = 0
into the null element "0". Thus the null space N of A is that subspace of all vectors in V
which are imaged into the null element “0" by the matrix A.

NULL SPACE

Definition The null space of an m × n matrix A, written as Nul A, is the set of all
solutions to the homogeneous equation Ax = 0. In set notation,
Nul A = {x: x is in Rn and Ax = 0}
OR
Nul ( A=
) {x / ∀x ∈  , Ax= 0}

A more dynamic description of Nul A is the set of all x in Rn that are mapped into the
zero vector of Rm via the linear transformation x → Ax , where A is a matrix of
transformation. See Figure1

©Virtual University Of Pakistan 46

05-Null Spaces, Column Spaces and Linear Transformation VU

Nul A
0
0
Rm
Rn

Figure 1

5
and let u =  3  . Determine if u ∈ Nul A .
 1 -3 -2 
Example 1: Let A =  
-5 9 1    -2 
Solution: To test if u satisfies Au = 0, simply compute
5
 1 -3 -2     5 - 9 + 4  0 
Au =  3 =   =   . Thus u is in Nul A.
-5 9 1    -25 + 27 - 2  0 
-2 

Example: Determine the null space of the following matrix:

 4 0 
A= 
 −8 20 

Solution: To find the null space of A we need to solve the following system of
equations:
 4 0   x1   0 
   =  
 −8 20   x2   0 
 4 x1 + 0 x2   0 
⇒  = 
 −8 x1 + 20 x2   0 
⇒ 4 x1 + 0=
x2 0 ⇒=
x1 0
and ⇒ − 8 x1 + 20=
x2 0 ⇒=
x2 0

We can find Null space of a matrix with two ways i.e. with matrices or with system of
linear equations. We have given this in both matrix form and (here first we convert the
matrix into system of equations) equation form. In equation form it is easy to see that by
solving these equations together the only solution is x=
1 x=
2 0 . In terms of vectors from

 2 the solution consists of the single vector {0} and hence the null space of A is {0} .

©Virtual University Of Pakistan 47

05-Null Spaces, Column Spaces and Linear Transformation VU

Activity: Determine the null space of the following matrices:

0 0 0
 
1. 0 = 0 0 0
0 0 0
 
 1 −5 
2. M = 
 −5 25 

In earlier (previous) lectures, we developed the technique of elementary row operations

to solve a linear system. We know that performing elementary row operations on an
augmented matrix does not change the solution set of the corresponding linear system
Ax=0. Therefore, we can say that it does not change the null space of A. We state this
result as a theorem:

Theorem 1: Elementary row operations do not change the null space of a matrix.
Or
Null space N(A) of a matrix A can not be changed (always same) by changing the matrix
with elementary row operations.

Example: Determine the null space of the following matrix using the elementary row
operations: (Taking the matrix from the above Example)

 4 0 
A= 
 −8 20 
Solution: First we transform the matrix to the reduced row echelon form:
 4 0 1 0 1
  ~  R1
 −8 20   −8 20  4
1 0
~  R2 + 8 R1
0 20 
1 0 1
~  R2
0 1 20

which corresponds to the system

x1 =0
x2 = 0
Since every column in the coefficient part of the matrix has a leading entry that means
our system has the trivial solution only:
x1 = 0
x2 = 0
This means the null space consists only of the zero vector.

We can observe and compare both the above examples which show the same result.

©Virtual University Of Pakistan 48

05-Null Spaces, Column Spaces and Linear Transformation VU

Theorem 2: The null space of an m × n matrix A is a subspace of Rn. Equivalently, the

set of all solutions to a system Ax = 0 of m homogeneous linear equations in n unknowns
is a subspace of Rn.

Or simply, the null space is the space of all the vectors of a Matrix A of any order those
are mapped (assign) onto zero vector in the space Rn (i.e. Ax = 0).
Proof: We know that the subspace of A consists of all the solution to the system
Ax = 0 . First, we should point out that the zero vector, 0, in Rn will be a solution to this
system and so we know that the null space is not empty. This is a good thing since a
vector space (subspace or not) must contain at least one element.
Now we know that the null space is not empty. Consider u, v be two any vectors
(elements) (in) from the null space and let c be any scalar. We just need to show that the
sum (u+v) and scalar multiple (c.u) of these are also in the null space.

Certainly Nul A is a subset of Rn because A has n columns. To show that Nul(A) is the
subspace, we have to check three conditions whether they are satisfied or not. If Nul(A)
satisfies the all three condition, we say Nul(A) is a subspace otherwise not.
First, zero vector “0” must be in the space and subspace. If zero vector does not in the
space we can not say that is a vector space (generally, we use space for vector space).
And we know that zero vector maps on zero vector so 0 is in Nul(A). Now choose any
vectors u, v from Null space and using definition of Null space (i.e. Ax=0)
Au = 0 and Av = 0

Now the other two conditions are vector addition and scalar multiplication. For this we
proceed as follow:
Let start with vector addition:
To show that u + v is in Nul A, we must show that A (u + v) = 0. Using the property of
matrix multiplication, we find that
A (u + v) = Au + Av = 0 + 0 = 0
Thus u + v is in Nul A, and Nul A is closed under vector addition.
For Matrix multiplication, consider any scalar , say c,
A (cu) = c (Au) = c (0) = 0
which shows that cu is in Nul A. Thus Nul A is a subspace of Rn.

Example 2: The set H, of all vectors in R4 whose coordinates a, b, c, d satisfy the

equations
a – 2b + 5c = d
c–a=b
is a subspace of R4.
Solution: Since a – 2b + 5c = d
c–a=b
By rearranging the equations, we get

a - 2b + 5c - d = 0
-a - b + c =0

©Virtual University Of Pakistan 49

05-Null Spaces, Column Spaces and Linear Transformation VU

We see that H is the set of all solutions of the above system of homogeneous linear
equations.
Therefore from the Theorem 2, H is a subspace of R4.

It is important that the linear equations defining the set H are homogeneous. Otherwise,
the set of solutions will definitely not be a subspace (because the zero-vector (origin) is
not a solution of a non- homogeneous system), geometrically means that a line that not
passes through origin can not be a subspace, because subspace must hold the zero vector
(origin). Also, in some cases, the set of solutions could be empty. In this case, we can not
find any solution of a system of linear equations, geometrically says that lines are parallel
or not intersecting.
If the null space having more than one vector, geometrically means that the lines intersect
more than one point and must passes through origin (zero vector) .

An Explicit Description of Nul A:

There is no obvious relation between vectors in Nul A and the entries in A. We say that
Nul A is defined implicitly, because it is defined by a condition that must be checked. No
explicit list or description of the elements in Nul A is given. However, when we solve the
equation Ax = 0, we obtain an explicit description of Nul A.

Example 3: Find a spanning set for the null space of the matrix
- 3 6 -1 1 - 7 
A =  1 - 2 2 3 - 1 
 2 - 4 5 8 - 4 

Solution: The first step is to find the general solution of Ax = 0 in terms of free
variables.
After transforming the augmented matrix [A 0] to the reduced row echelon form and we
get;
1 -2 0 -1 3 0 
0 0 1 2 -2 0 
 
0 0 0 0 0 0 
which corresponds to the system
x1 - 2x2 - x4 + 3x5 = 0
x3 + 2x4 - 2x5 = 0
0=0
The general solution is
x1 = 2x2 + x4 - 3x5
x2 = free variable
x3 = - 2x4 + 2x5
x4 = free variable
x5 = free variable

©Virtual University Of Pakistan 50

05-Null Spaces, Column Spaces and Linear Transformation VU

Next, decompose the vector giving the general solution into a linear combination of
vectors where the weights are the free variables. That is,

 x1   2 x2 + x4 - 3 x5  2 1 -3

x   x2       
 2   1  0 0
 x3  =  -2 x4 + 2 x5  = x2  0  + x4 -2  + x5  2 
         
 x4   x4  0 1 0
 x5   x5   0   0   1 
↑ ↑ ↑
u v w
= x2 u + x4 v + x5 w (3)
Every linear combination of u, v and w is an element of Nul A. Thus {u, v, w} is a
spanning set for Nul A.

Two points should be made about the solution in Example 3 that apply to all problems of
this type. We will use these facts later.
1. The spanning set produced by the method in Example 3 is automatically linearly
independent because the free variables are the weights on the spanning vectors.
For instance, look at the 2nd, 4th and 5th entries in the solution vector in (3) and
note that x2 u + x4 v + x5 w can be 0 only if the weights x 2 , x 4 and x 5 are all zero.
2. When Nul A contains nonzero vector, the number of vectors in the spanning set
for Nul A equals the number of free variables in the equation Ax = 0.

 1 -3 2 2 1 
 0 3 6 0 -3
 
Example 4: Find a spanning set for the null space of A =  2 -3 -2 4 4  .
 
 3 -6 0 6 5 
-2 9 2 -4 -5
Solution: The null space of A is the solution space of the homogeneous system
x1 - 3x2 + 2x3 + 2x4 + x5 = 0
0x1 + 3x2 +6x3 +0x4 - 3x5 = 0
2x1 - 3x2 - 2x3 + 4x4 + 4x5 = 0
3x1 - 6x2 +0x3 +6x4 + 5x5 = 0
-2x1 + 9x2 + 2x3 - 4x4 - 5x5 = 0

©Virtual University Of Pakistan 51

05-Null Spaces, Column Spaces and Linear Transformation VU

1 -3 2 2 1 0
0 3 6 0 -3 0 

2 -3 -2 4 4 0
 
3 -6 0 6 5 0
-2 9 2 -4 -5 0 
1 -3 2 2 1 0
0 3 6 0 -3 0  - 2R1 + R3

0 3 -6 0 2 0  -3R1 + R4
 
0 3 -6 0 2 0  2R1 + R5
 0 3 6 0 -3 0 
1 -3 2 2 1 0
0 1 2 0 -1 0 

0 3 -6 0 2 0  (1/3)R2
 
0 3 -6 0 2 0
 0 3 6 0 -3 0 
1 -3 2 2 1 0
 0  - 3R2 + R3
0 1 2 0 -1
0 0 -12 0 5 0  -3R2 + R4
 
0 0 -12 0 5 0  -3R2 + R5
0 0 0 0 0 0 
1 -3 2 2 1 0
0 1 2 0 -1 0 

0 0 1 0 - 5/12 0  (-1/12)R3
 
0 0 -12 0 5 0
 0 0 0 0 0 0 
1 -3 2 2 1 0
0 1 2 0 -1 0 

0 0 1 0 - 5/12 0  12R3 + R4
 
0 0 0 0 0 0
0 0 0 0 0 0 
1 -3 0 2 11/ 6 0
 0 
0 1 0 0 -1/6
- 2R3 + R2
0 0 1 0 - 5/12 0
  -2R3 + R1
0 0 0 0 0 0
 0 0 0 0 0 0 

©Virtual University Of Pakistan 52

05-Null Spaces, Column Spaces and Linear Transformation VU

1 0 0 2 4/3 0
0 1 0 0 -1/6 0 

0 0 1 0 - 5/12 0  3R2 + R1
 
0 0 0 0 0 0
0 0 0 0 0 0 

The reduced row echelon form of the augmented matrix corresponds to the system

1 x1 + 2 x4 +(4/3) x5 = 0
1 x2 + (-1/6) x5 = 0
1 x3 + (-5/12) x5 = 0 .
0=0
0=0

No equation of this system has a form zero = nonzero; Therefore, the system is
consistent. The system has infinitely many solutions:

x1 = -2 x4 +(-4/3) x5 x2 = +(1/6) x5 x3 = +(5/12) x5

x4 = arbitrary x5 = arbitrary

The solution can be written in the vector form:

c4 = (-2,0,0,1,0) c5 = (-4/3,1/6,5/12,0,1)

Therefore {(-2,0,0,1,0), (-4/3,1/6,5/12,0,1)} is a spanning set for Null space of A.

Activity: Find an explicit description of Nul A where:

3 5 5 3 9 
1. A= 
5 1 1 0 3 
 4 1 −1 0 1
 
 −1 − 1 2 −3 1
2. A=
 1 1 −2 0 −1
 
 0 0 1 1 1

The Column Space of a Matrix: Another important subspace associated with a matrix
is its column space. Unlike the null space, the column space is defined explicitly via
linear combinations.

©Virtual University Of Pakistan 53

05-Null Spaces, Column Spaces and Linear Transformation VU

Definition: (Column Space): The column space of an m × n matrix A, written as Col A,

is the set of all linear combinations of the columns of A. If A = [a 1 … a n ], then
Col A = Span {a 1 ,… , a n }

Since Span {a 1 , …, a n } is a subspace, by Theorem of lecture 20 i.e. if v1 ,..., v p are in a

vector space V , then Span {v1 ,..., v p } is a subspace of V .
The column space of a matrix is that subspace spanned by the columns of the matrix
(columns viewed as vectors). It is that space defined by all linear combinations of the
column of the matrix.

Example, in the given matrix,

1 1 3
 
 2 1 4
A=
3 1 5
 
 4 1 6
The column space ColA is all the linear combination of the first (1, 2, 3, 4), the second (1,
1, 1, 1) and the third column ( 3, 4, 5, 6). That is, ColA = { a·(1, 2, 3, 4) + b·(1, 1, 1, 1) +
c·( 3, 4, 5, 6) }. In general, the column space ColA contains all the linear
combinations of columns of A.

The next theorem follows from the definition of Col A and the fact that the columns of A
are in Rm.

Theorem 3: The column space of an m × n matrix A is a subspace of Rm.

Note that a typical vector in Col A can be written as Ax for some x because the notation
Ax stands for a linear combination of the columns of A. That is,
Col A = {b: b = Ax for some x in Rn}
The notation Ax for vectors in Col A also shows that Col A is the range of the linear
transformation x → Ax.
  6a - b  
  
Example 6: Find a matrix A such that W = Col A. W =  a + b : a,b in R 
 
  -7 a  
   
Solution: First, write W as a set of linear combinations.
 6 -1    6  -1 
     
W = a  1  +b  1  : a,b in R  = Span   1  ,  1  
 
   
 -7   0    -7   0  
 6 -1
Second, use the vectors in the spanning set as the columns of A. Let A =  1 1  .
-7 0 
Then W = Col A, as desired.

©Virtual University Of Pakistan 54

05-Null Spaces, Column Spaces and Linear Transformation VU

x3 x2

0 W

x1
We know that the columns of A span Rm if and only if the equation Ax = b has a
solution for each b. We can restate this fact as follows:
The column space of an m × n matrix A is all of Rm if and only if the equation Ax = b has
a solution for each b in Rm.

Theorem 4: A system of linear equations Ax = b is consistent if and only if b in the

column space of A.

Example 6: A vector b in the column space of A. Let Ax = b is the linear system

-1 3 2   x1   1 
 1 2 -3  x  = -9  . Show that b is in the column space of A, and express b as a
  2  
 2 1 -2   x3   -3
linear combination of the column vectors of A.
Solution: Augmented Matrix is given by
-1 3 2 1 
1 2 -3 -9 

 2 1 -2 -3 
1 -3 -2 -1  -1R1
0 5 -1 - 8 -1R1 + R2

0 7 2 -1  -2R1 + R3
1 -3 -2 -1 
0 1 -1/5 - 8/5 
1/5R2
 -7R2 + R3
0 0 17/5 51/5
1 -3 0 5  (5/17)R3
0 1 0 -1 (1/5)R3 + R2

0 0 1 3  2R3 + R1

©Virtual University Of Pakistan 55

 05-Null Spaces, Column Spaces and Linear Transformation VU

1 0 0 2
0 1 0 -1 3R2 + R1

0 0 1 3 
⇒ x1 = 2,x2 = -1,x3 = 3 . Since the system is consistent, b is in the column space of A.

-1  3  2 1

Moreover, 2  1  -  2  + 3  -3 = -9 
    
 2  1  -2   -3

Example: Determine whether b is in the column space of A and if so, express b as a

linear combination of the column vectors of A:
1 1 2  −1
   
= A = 1 0 1 : b  0 
 2 1 3  2
   

Solution:
The coefficient matrix Ax = b is:
 1 1 2   x1   −1
    
 1 0 1   x2  =  0 
 2 1 3 x   2 
  3   
The augmented matrix for the linear system that corresponds to the matrix
equation Ax = b is:
 1 1 2 −1 
 
1 0 1 0 
2 1 3 2 

We reduce this matrix to the Reduced Row Echelon Form:
 1 1 2 −1  1 1 2 −1 
   
1 0 1 0  ~  0 −1 −1 1  R2 + ( −1) R1
2 1 3 2  2 1 3 2 
 
1 1 2 −1 
 
~  0 −1 −1 1  R3 + ( −2 ) R1
 0 −1 − 1 4 
 
1 1 2 −1 
 
~ 0 1 1 −1  ( −1) R2
 0 −1 − 1 4 
 

©Virtual University Of Pakistan 56

 05-Null Spaces, Column Spaces and Linear Transformation VU

1 1 2 −1 
 
~ 0 1 1 −1  R3 + R2
0 0 0 3 

1 1 2 −1 
  1
~ 0 1 1 −1  R3
0 3
 0 0 1 
1 1 2 −1 
 
~ 0 1 1 0  R2 + R3
0 0 0 1 

1 1 2 0
 
~ 0 1 1 0  R1 + R3
0 0 0 1 

1 0 1 0
 
~ 0 1 1 0  R1 + ( −1) R2
0 0 0 1 


The new system for the equation Ax = b is

x1 + x3 = 0
x2 + x3 =
0
0= 1
Equation 0 = 1 cannot be solved, therefore, the system has no solution (i.e. the system is
inconsistent).
Since the equation Ax = b has no solution, therefore b is not in the column space of A.

Activity: Determine whether b is in the column space of A and if so, express b as a

linear combination of the column vectors of A:

1.
 1 −1 2  5 
   
=A =9 3 1 ; b 1
1 1 1  0
   

 1 −1 1 1 
   
2. =A  1 1 −1 =; b  2
 −1 −1 
−1  3
  

©Virtual University Of Pakistan 57

 05-Null Spaces, Column Spaces and Linear Transformation VU

 1 1 − 2 1 1 
   
0 2 0 1  2
=3. A = ; b
 1 1 1 − 3 3
   
 0 2 2 1  4

Theorem 5: If x 0 denotes any single solution of a consistent linear system Ax=b and if
v1 , v 2 , v 3 ,..., vk form the solution space of the homogeneous system Ax=0, then every
solution of Ax=b can be expressed in the form x = x0 + c1v1 + c2v 2 + ... + ck vk and,
conversely, for all choices of scalars c1 , c2 , c3 ,..., ck , the vector x is a solution of Ax=b.

General and Particular Solutions: The vector x 0 is called a particular solution of Ax=b
.The expression x 0 + c 1 v 1 +c 2 v 2 + . . . +c k v k is called the general solution of Ax=b , and
the expression c 1 v 1 +c 2 v 2 + . . . +c k v k is called the general solution of Ax=0.

Example 7: Find the vector form of the general solution of the given linear system
Ax = b; then use that result to find the vector form of the general solution of Ax=0.
x1 + 3x2 - 2x3 + 2x5 =0
2x1 +6x2 - 5x3 - 2x4 + 4x5 - 3x6 = -1
5x3 +10x4 +15x6 = 5
2x1 +6x2 + 8x4 + 4x5 +18x6 = 6
Solution: We solve the non-homogeneous linear system. The augmented matrix of this
system is given by
1 3 -2 0 2 0 0
2 6 -5 -2 4 -3 -1

0 0 5 10 0 15 5
 
2 6 0 8 4 18 6
1 3 -2 0 2 0 0 
0 0 -1 -2 0 -3 -1  -2R1 + R2

0 0 5 10 0 15 5  -2R1 + R4
 
0 0 4 8 0 18 6 
1 3 -2 0 2 0 0
0 0 1 2 0 3 1 
 - 1R2
0 0 5 10 0 15 5
 
0 0 4 8 0 18 6

©Virtual University Of Pakistan 58

05-Null Spaces, Column Spaces and Linear Transformation VU

1 3 -2 0 2 0 0
0 0 1 2 0 3 1  -5R2 + R3

0 0 0 0 0 0 0  -4R2 + R4
 
0 0 0 0 0 6 2
1 3 -2 0 2 0 0
0 0 1 2 0 3 1 
 R34
0 0 0 0 0 6 2
 
0 0 0 0 0 0 0

1 3 -2 0 2 0 0 
0 0 1 2 0 3 1 
 (1/6)R3
0 0 0 0 0 1 1/3 
 
0 0 0 0 0 0 0 
1 3 -2 0 2 0 0 
0 0 1 2 0 0 0 
 - 3R3 + R2
0 0 0 0 0 1 1/3 
 
0 0 0 0 0 0 0 
1 3 0 4 2 0 0 
0 0 1 2 0 0 0 
 2R2 + R1
0 0 0 0 0 1 1/3 
 
0 0 0 0 0 0 0 

The reduced row echelon form of the augmented matrix corresponds to the system

1 x1 + 3 x2 + 4 x4 + 2 x5 =0
1 x3 + 2 x4 =0
1 x6 = (1/3)
0=0

No equation of this system has a form zero = nonzero; Therefore, the system is
consistent. The system has infinitely many solutions:

x1 = -3 x2 - 4 x4 - 2 x5 x2 = r x3 = -2 x4
x4 = s x5 = t x6 = 1/3

x1 = -3r - 4s - 2t x2 = r x3 = -2s
1
x4 = s x5 = t x6 =
3

©Virtual University Of Pakistan 59

05-Null Spaces, Column Spaces and Linear Transformation VU

This result can be written in vector form as

-3r - 4 s - 2t   0 
 x1    0 -3 -4  -2 
x   r    1 0 0
 2  -2 s  0      
 x3    =  +r  0  -2   0 
 =  s    0   + s  +t   (A)
 x4    0 0 1 0
 x5   t 0 0 1
  
   1  1       
 x6    0 0 0
 3   3 
which is the general solution of the given system. The vector x 0 in (A) is a particular
-3 -4  -2 
1 0 0
     
0 -2   0 
solution of the given system; the linear combination r   + s   + t   in (A) is the
0 1 0
0 0 1
     
0 0 0
general solution of the homogeneous system.

Activity:
1. Suppose that x1 = −1, x2 =
2, x3 = 4, x4 =−3 is a solution of a non-homogenous
linear system Ax = b and that the solution set of the homogenous system Ax = 0
is given by this formula:
x1 =−3r + 4 s ,
x2 = r − s ,
x3 = r ,
x4 = s
(a) Find the vector form of the general solution of Ax = 0 .
(b) Find the vector form of the general solution of Ax = 0 .

Find the vector form of the general solution of the following linear system Ax = b; then
use that result to find the vector form of the general solution of Ax=0:
x1 − 2 x2 =
1
2.
3 x1 − 9 x2 =
2

x1 + 2 x2 − 3 x3 + x4 =3
− 3 x1 − x2 + 3 x3 + x4 =
−1
3.
− x1 + 3 x2 − x3 + 2 x4 =2
4 x1 − 5 x2 − 3 x4 =
−5

©Virtual University Of Pakistan 60

05-Null Spaces, Column Spaces and Linear Transformation VU

The Contrast between Nul A and Col A:

It is natural to wonder how the null space and column space of a matrix are related. In
fact, the two spaces are quite dissimilar. Nevertheless, a surprising connection between
the null space and column space will emerge later.

 2 4 -2 1 
Example 8: Let A = -2 -5 7 3
 3 7 -8 6 
(a) If the column space of A is a subspace of Rk, what is k?
(b) If the null space of A is a subspace of Rk, what is k?

Solution:

(a) The columns of A each have three entries, so Col A is a subspace of Rk, where k = 3.
(b) A vector x such that Ax is defined must have four entries, so Nul A is a subspace of
Rk, where k = 4.

When a matrix is not square, as in Example 8, the vectors in Nul A and Col A live in
entirely different “universes”. For example, we have discussed no algebraic operations
that connect vectors in R3 with vectors in R4. Thus we are not likely to find any relation
between individual vectors in Nul A and Col A.

 2 4 -2 1 
Example 9: If A = -2 -5 7 3 , find a nonzero vector in Col A and a nonzero
 3 7 -8 6 
vector in Nul A
2
Solution: It is easy to find a vector in Col A. Any column of A will do, say, -2  . To
 3 
find a nonzero vector in Nul A, we have to do some work. We row reduce the augmented
1 0 9 0 0 
matrix [A 0] to obtain [ A 0 ] ~ 0 1 -5 0 0  . Thus if x satisfies Ax = 0,
0 0 0 1 0 
then x1 = -9x3 , x2 = 5x3 , x4 = 0 , and x 3 is free. Assigning a nonzero value to x 3 (say), x 3 =
1, we obtain a vector in Nul A, namely, x = (-9, 5, 1, 0).

3
 2 4 -2 1  -2  3
 
Example 10: With A = -2 -5 7 3 , let u =   and v = -1 .
 -1
 3 7 -8 6     3 
 
0
(a) Determine if u is in Nul A. Could u be in Col A?

©Virtual University Of Pakistan 61

05-Null Spaces, Column Spaces and Linear Transformation VU

(b) Determine if v is in Col A. Could v be in Nul A?

Solution: (a) An explicit description of Nul A is not needed here. Simply compute the
product
3
 2 4 -2 1     0  0 
Au = -2 -5 7 3   = -3 ≠ 0 
-2
 -1
 3 7 -8 6     3  0 
0
Obviously, u is not a solution of Ax = 0, so u is not in Nul A.
Also, with four entries, u could not possibly be in Col A, since Col A is a subspace of R3.
(b) Reduce [A v] to an echelon form:
 2 4 -2 1 3   2 4 -2 1 3 
[ A v ] = -2 -5 7 3 -1   0 1 -5 -4 2 
 3 7 -8 6 3   0 0 0 17 1 
At this point, it is clear that the equation Ax = v is consistent, so v is in Col A. With only
three entries, v could not possibly be in Nul A, since Nul A is a subspace of R4.

The following table summarizes what we have learned about Nul A and Col A.

1. Nul A is a subspace of Rn. 1. Col A is a subspace of Rm.

2. Nul A is implicitly defined; i.e. we 2. Col A is explicitly defined; that is,
are given only a condition (Ax = 0) we are told how to build vectors in
that vectors in Nul A must satisfy. Col A.
3. It takes time to find vectors in Nul 3. It is easy to find vectors in Col A
A. Row operations on [A 0] are The columns of A are displayed;
required. others are formed from them.
4. There is no obvious relation 4. There is an obvious relation
between Nul A and the entries in A. between Col A and the entries in
A, since each column of A is in Col
A.
5. A typical vector v in Nul A has the 5. A typical vector v in Col A has the
property that Av = 0. property that the equation Ax = v
is consistent.
6. Given a specific vector v, it is easy 6. Given a specific vector v, it may
to tell if v is in Nul A. Just compute take time to tell if v is in Col A.
Av. Row operations on [A v] are
required.
7. Nul A = {0} if and only if the 7. Col A.= Rm if and only if the
equation Ax = 0 has only the trivial equation Ax = b has a solution for
solution. every b in Rm.
8. Nul A = {0} if and only if the linear 8. Col A = Rm if and only if the linear
transformation x → Ax is one-to- transformation x → Ax maps Rn
one. onto Rm.

©Virtual University Of Pakistan 62

05-Null Spaces, Column Spaces and Linear Transformation VU

Kernel and Range of A Linear Transformation:

Subspaces of vector spaces other than Rn are often described in terms of a linear
transformation instead of a matrix. To make this precise, we generalize the definition
given earlier in Segment I.

Definition: A linear transformation T from a vector space V into a vector space W is a

rule that assigns to each vector x in V a unique vector T (x) in W, such that
(i) T (u + v) = T (u) + T (v) for all u, v in V, and
(ii) T (cu) = c T (u) for all u in V and all scalars c.

The kernel (or null space) of such a T is the set of all u in V such that T (u) = 0 (the zero
vector in W). The range of T is the set of all vectors in W of the form T (x) for some x in
V. If T happens to arise as a matrix transformation, say, T (x) = Ax for some matrix A –
then the kernel and the range of T are just the null space and the column space of A, as
defined earlier. So if T(x) = Ax, col A = range of T.

Definition: If T : V → W is a linear transformation, then the set of vectors in V that T

maps into 0 is called the kernel of T; it is denoted by ker(T). The set of all vectors in W
that are images under T of at least one vector in V is called the range of T; it is denoted
by R(T).

Example: If TA :  n →  m is multiplication by the m × n matrix A, then from the

above definition; the kernel of TA is the null space of A and the range of TA is the column
space of A.

Remarks: The kernel of T is a subspace of V and the range of T is a subspace of W.

Range
Domain
0
0 W
V’
Kernel

Kernel is a Range is a
subspace of V subspace of W

Figure 2 Subspaces associated with a linear transformation.

In applications, a subspace usually arises as either the kernel or the range of an

appropriate linear transformation. For instance, the set of all solutions of a homogeneous
linear differential equation turns out to be the kernel of a linear transformation. Typically,
such a linear transformation is described in terms of one or more derivatives of a

©Virtual University Of Pakistan 63

05-Null Spaces, Column Spaces and Linear Transformation VU

function. To explain this in any detail would take us too far a field at this point. So we
present only two examples. The first explains why the operation of differentiation is a
linear transformation.

Example 11: Let V be the vector space of all real-valued functions f defined on an
interval [a, b] with the property that they are differentiable and their derivatives are
continuous functions on [a, b]. Let W be the vector space of all continuous functions on
[a, b] and let D : V → W be the transformation that changes f in V into its
derivative f ′ . In calculus, two simple differentiation rules are
D( f + g= ) D( f ) + D ( g ) and D(cf =
) cD( f )
That is, D is a linear transformation. It can be shown that the kernel of D is the set of
constant functions of [a, b] and the range of D is the set W of all continuous functions on
[a, b].

Example 12: The differential equation y ′′ + wy = 0 (4)

where w is a constant, is used to describe a variety of physical systems, such as the
vibration of a weighted spring, the movement of a pendulum and the voltage in an
inductance – capacitance electrical circuit. The set of solutions of (4) is precisely the
kernel of the linear transformation that maps a function y = f (t ) into the
function f ′′(t ) + wf (t ) . Finding an explicit description of this vector space is a problem in
differential equations.

a  
  
Example 13: Let W =  b : a - 3b - c = 0  . Show that W is a subspace of R3 in
 
 c  
  
different ways.
Solution: First method: W is a subspace of R3 by Theorem 2 because W is the set of all
solutions to a system of homogeneous linear equations (where the system has only one
equation). Equivalently, W is the null space of the 1x3 matrix A = [1 - 3 - 1].
Second method: Solve the equation a – 3b – c = 0 for the leading variable a in terms of
the free variables b and c.
3b + c 
Any solution has the form  b  , where b and c are arbitrary, and
 c 
3b+ c  3 1 
 b  = b 1  + c  0 
     
 c  0  1 
↑ ↑
v1 v2

©Virtual University Of Pakistan 64

05-Null Spaces, Column Spaces and Linear Transformation VU

This calculation shows that W = Span{v 1 , v 2 }. Thus W is a subspace of R3 by Theorem

i.e. if v1 ,..., v p are in a vector space V , then Span {v1 ,..., v p } is a subspace of V . We could
also solve the equation a – 3b – c = 0 for b or c and get alternative descriptions of W as a
set of linear combinations of two vectors.

 7 -3 5  2 7
Example 14: Let A = -4 1 -5 , v =  1  ,and W =  6 
   
 -5 2 -4  -1 -3
Suppose you know that the equations Ax = v and Ax = w are both consistent. What can
you say about the equation Ax = v + w?
Solution: Both v and w are in Col A. Since Col A is a vector space, v + w must be in Col
A. That is, the equation Ax = v + w is consistent.

Activity:
1. Let V and W be any two vector spaces. The mapping T : V → W such that T (v) =
0 for every v in V is a linear transformation called the zero transformation. Find
the kernel and range of the zero transformation.

2. Let V be any vector space. The mapping I : V → V defined by I(v) = v is called

the identity operator on V. Find the kernel and range of the identity operator.

©Virtual University Of Pakistan 65

05-Null Spaces, Column Spaces and Linear Transformation VU

Exercises:

5  5 21 19 
1. Determine if w = -3 is in Nul A, where A= 13 23 2  .
 
   
 2   8 14 1 

In exercises 2 and 3, find an explicit description of Nul A, by listing vectors that span the
null space.
1 -2 0 4 0 
1 3 5 0 
2.   3. 0 0 1 -9 0 

0 1 4 -2   
0 0 0 0 1 

In exercises 4-7, either use an appropriate theorem to show that the given set, W is a
vector space, or find a specific example to the contrary.

 a  
a     
     b  a - 2b = 4c 
4.  b : a + b + c = 2  5.  : 
 
 c     c  2a = c + 3d 
     d  

 b - 2d 
 5+d  -a + 2b 
6.   :b,d real 7.  a - 2b  : a,b real
b + 3d   
   3a - 6b 
 d 

In exercises 8 and 9, find A such that the given set is Col A.

  2 s + 3t    b - c  
     
  r + s - 2t     2b + c + d  
8.  : r,s,t real  9.  :b, c, d real 
  4r + s     5c - 4d  
  3r - s - t     d

 

For the matrices in exercises 10-13, (a) find k such that Nul A is a subspace of Rk, and
(b) find k such that Col A is a subspace of Rk.

©Virtual University Of Pakistan 66

 05-Null Spaces, Column Spaces and Linear Transformation VU

 2 -6   7 -2 0 
 -1 3  -2 0 -5
10. A =   11. A =  
-4 12   0 -5 7 
   
 3 -9   -5 7 -2 

 4 5 -2 6 0 
12. A=   13. A= [1 -3 9 0 -5]
1 1 0 1 0 

-6 12  2
=
14. Let A =  and w   . Determine if w is in Col A. Is w in Nul A?
 -3 6  1 

-8 -2 -9  2
15. Let A =
=   1  . Determine if w is in Col A. Is w in Nul A?
 6 4 8  and w  
 4 0 4  -2 

 p(0) 
16. Define T: P 2 → R2 by T (p) =   . For instance, if p (t) = 3 + 5t + 7t2, then
 p(1) 
3
T( p ) =   .
15
a. Show that T is a linear transformation.
b. Find a polynomial p in P 2 that spans the kernel of T, and describe the range of T.

 p(0) 
17. Define a linear transformation T: P 2 → R2 by T (p) =   . Find polynomials p 1
 p(1) 
and p 2 in P 2 that span the kernel of T, and describe the range of T.

18. Let M 2x2 be the vector space of all 2x2 matrices, and define T: M 2x2 → M 2x2 by
a b 
T (A) = A + AT, where A =  .
c d 
(a) Show that T is a linear transformation.
(b) Let B be any element of M 2x2 such that BT=B. Find an A in M 2x2 such that T (A) = B.
(c) Show that the range of T is the set of B in M 2x2 with the property that BT=B.
(d) Describe the kernel of T.

19. Determine whether w is in the column space of A, the null space of A, or both, where

©Virtual University Of Pakistan 67

05-Null Spaces, Column Spaces and Linear Transformation VU

1 7 6 -4 1  1   -8 5 -2 0 
1  -5 -1 0 -2  2  -5 2 1 -2 
(a) w =
=  ,A   (b) w =
=  ,A  
 -1  9 -11 7 -3 1  10 -8 6 -3
       
-3 19 -9 7 1  0  3 -2 1 0 

20. Let a 1 , …, a 5 denote the columns of the matrix A, where

5 1 2 2 0 
3 3 2 -1 -12 
A= , B = [ a1 a2 a4 ]
8 4 4 -5 12 
 
2 1 1 0 -2 
(a) Explain why a 3 and a 5 are in the column space of B
(b) Find a set of vectors that spans Nul A
(c) Let T: R5 → R4 be defined by T (x) = Ax. Explain why T is neither one-to-one nor
onto.

©Virtual University Of Pakistan 68

06- Linear Independence Sets; Bases VU

Lecture 06

Linearly Independent Sets; Bases

First we revise some definitions and theorems from the Vector Space:

Definition: Let V be an arbitrary nonempty set of objects on which two operations are
defined, addition and multiplication by scalars.

If the following axioms are satisfied by all objects u, v, w in V and all scalars l and
m, then we call V a vector space.

Axioms of Vector Space:

For any set of vectors u, v, w in V and scalars l, m, n:
1. u + v is in V
2. u+v=v+u
3. u + (v + w) = (u + v) + w
4. There exist a zero vector 0 such that
0+u=u+0=u
5. There exist a vector – u in V such that
-u + u = 0 = u + (-u)
6. (l u) is in V
7. l (u + v)= l u + l v
8. m (n u) = (m n) u = n (m u)
9. (l +m) u= I u+ m u
10. 1u = u where 1 is the multiplicative identity

Definition: A subset W of a vector space V is called a subspace of V if W itself is a

vector space under the addition and scalar multiplication defined on V.

Theorem: If W is a set of one or more vectors from a vector space V, then W is subspace
of V if and only if the following conditions hold:

(a) If u and v are vectors in W, then u + v is in W

(b) If k is any scalar and u is any vector in W, then k u is in W.

Definition; The null space of an m x n matrix A (Nul A) is the set of all solutions of the
hom equation Ax = 0
Nul A = {x: x is in Rn and Ax = 0}

Definition: The column space of an m x n matrix A (Col A) is the set of all linear
combinations of the columns of A.
If A = [a 1 … a n ],
then
Col A = Span { a 1 ,… , a n }

©Virtual University Of Pakistan 69

06- Linear Independence Sets; Bases VU

Since we know that a set of vectors S = {v1 , v2 , v3 ,...v p } spans a given vector space V if
every vector in V is expressible as a linear combination of the vectors in S. In general
there may be more than one way to express a vector in V as linear combination of vectors
in a spanning set. We shall study conditions under which each vector in V is expressible
as a linear combination of the spanning vectors in exactly one way. Spanning sets with
this property play a fundamental role in the study of vector spaces.
In this Lecture, we shall identify and study the subspace H as “efficiently” as possible.
The key idea is that of linear independence, defined as in Rn.

Definition: An indexed set of vectors {v 1 ,…, v p } in V is said to be linearly

independent if the vector equation
c1v1 + c2 v 2 + ...+ c p v p = 0 (1)
has only the trivial solution, i.e. c 1 = 0, … , c p = 0.
The set {v 1 ,…,v p } is said to be linearly dependent if (1) has a nontrivial solution, that is,
if there are some weights, c 1 ,…,c p , not all zero, such that (1) holds. In such a case, (1) is
called a linear dependence relation among v 1 , … , v p . Alternatively, to say that the v’s
are linearly dependent is to say that the zero vector 0 can be expressed as a nontrivial
linear combination of the v’s.

If the trivial solution is the only solution to this equation then the vectors in the set are
called linearly independent and the set is called a linearly independent set. If there is
another solution then the vectors in the set are called linearly dependent and the set is
called a linearly dependent set.

Just as in Rn, a set containing a single vector v is linearly independent if and only if v ≠ 0 .
Also, a set of two vectors is linearly dependent if and only if one of the vectors is a
multiple of the other. And any set containing the zero-vector is linearly dependent.

Determining whether a set of vectors a1 , a2 , a3 ,...an is linearly independent is easy when

one of the vectors is 0: if, say, a1 = 0 , then we have a simple solution to
x1a1 + x2 a2 + x3 a3 + ... + xn an =given
0 by choosing x1 to be any nonzero value and putting
all the other x’s equal to 0. Consequently, if a set of vectors contains the zero vector, it
must always be linearly dependent. Equivalently, any set of linearly independent vectors
cannot contain the zero vector.

Another situation in which it is easy to determine linear independence is when there are
more vectors in the set than entries in the vectors. If n > m, then the n vectors
a1 , a2 , a3 ,...an in Rm are columns of an m × n matrix A. The vector equation
x1a1 + x2 a2 + x3 a3 + ... + xn an =0 is equivalent to the matrix equation Ax = 0 whose
corresponding linear system has more variables than equations. Thus there must be at
least one free variable in the solution, meaning that there are nontrivial solutions

©Virtual University Of Pakistan 70

06- Linear Independence Sets; Bases VU

0 If n > m, then the set {a1 , a2 , a3 ,...an } of vectors in Rm

to x1a1 + x2 a2 + x3 a3 + ... + xn an =:
must be linearly dependent.
When n is small we have a clear geometric picture of the relation amongst linearly
independent vectors. For instance, the case n = 1 produces the equation x1a1 = 0 , and as
long as a1 ≠ 0 , we only have the trivial solution x1 = 0 . A single nonzero vector always
forms a linearly independent set.
When n = 2, the equation takes the form x1a1 + x2 a2 = 0 . If this were a linear dependence
relation, then one of the x’s, say x1 , would have to be nonzero. Then we could solve the
equation for a1 and obtain a relation indicating that a1 is a scalar multiple of a2 .
Conversely, if one of the vectors is a scalar multiple of the other, we can express this in
the form x1a1 + x2 a2 = 0 . Thus, a set of two nonzero vectors is linearly dependent if and
only if they are scalar multiples of each other.

Example: (linearly independent set)

Show that the following vectors are linearly independent:

 −2   2 0 
=v1 = 1  , v2 = 
 1  , v3 0 
 
 1   − 2  1 
Solution: Let there exist scalars c1 , c2 , c3 in R such that
c1v1 + c3v2 + c3v3 =
0
Therefore,
 −2   2 0 
⇒ c1  1  + c2  1  + c3 0  =
    0
 1   − 2  1 
 −2c1   2c2  0 
⇒  c  +  c  + 0  =
 1  2   0
 c1   −2c2  c3 
 −2c1 + 2c2  0 
⇒  =  
c1 + c2  0
c1 − 2c2 + c3  0 
The above can be written as:

=
−2c1 + 2c2 0 ........(1) ⇒
= −c1 + c2 0........(4) (dividing by 2 on both sides of (1))
c1 + c2 =
0 .......(2)
c1 − 2c2 + c3 =
0 ......(3)

©Virtual University Of Pakistan 71

06- Linear Independence Sets; Bases VU

Solving (2) and (4) implies :

c1 + c2 =
0
Solving (3) implies :
−c1 + c2 =0 Solving (2) implies :
0 + 0 + c3 =0
__________ c1 + 0 =0
⇒ c3 = 0
0 + 2 c2 =
0 ⇒ c1 = 0
⇒ c2 =
0
⇒ c1 = c2 = c3 = 0 ; scalars c1 , c2 , c3 ∈ R are all zero
∴ The system has trivial solution.
Hence the given vectors v1 , v2 , v3 are linearly independent.

Example: (linearly dependent set)

{2, − 1, 0,3} , v2 =
If v1 = {1, 2,5, − 1} and v3 =
{7, − 1,5,8} , then the set of vectors
S = {v1 , v2 , v3 } is linearly dependent, since 3v1 + v2 − v3 =
0

Example; (linearly dependent set)

The polynomials p1 = − x + 1, p2 = −2 x 2 + 3 x + 5, and p3 =
− x 2 + 3 x + 1 form a linearly
dependent set in p2 since 3 p1 − p2 + 2 p3 = 0.

Note: The linearly independent or linearly dependent sets can also be determined using
the Echelon Form or the Reduced Row Echelon Form methods.

Theorem 1: An indexed set { v 1 , … , v p } of two or more vectors, with v1 ≠ 0 , is

linearly dependent if and only if some v j (with j > 1 ) is a linear combination of the
preceding vectors, v 1 , … , v j-1 .

The main difference between linear dependence in Rn and in a general vector space is that
when the vectors are not n – tuples, the homogeneous equation (1) usually cannot be
written as a system of n linear equations. That is, the vectors cannot be made into the
columns of a matrix A in order to study the equation Ax = 0. We must rely instead on the
definition of linear dependence and on Theorem 1.

Example 1: Let p 1 (t) = 1, p 2 (t) = t and p 3 (t) = 4 – t. Then { p 1 , p 2 , p 3 } is linearly

dependent in P because p 3 = 4p 1 – p 2 .

Example 2: The set {Sin t, Cos t} is linearly independent in C [0, 1] because Sin t and
Cos t are not multiples of one another as vectors in C [0, 1]. That is, there is no scalar c
such that Cos t = c. Sin t for all t in [0, 1]. (Look at the graphs of Sin t and Cos t.)
However, {Sin t Cos t, Sin 2t} is linearly dependent because of he identity:
Sin 2t = 2 Sin t Cos t, for all t.

©Virtual University Of Pakistan 72

 06- Linear Independence Sets; Bases VU

Useful results:
• A set containing the zero vector is linearly dependent.
• A set of two vectors is linearly dependent if and only if one is a multiple of the
other.
• A set containing one nonzeoro vector is linearly independent. i.e. consider the set
containing one nonzeoro vector {v1} so {v1} is linearly independent when v1 ≠ 0 .
• A set of two vectors is linearly independent if and only if neither of the vectors is
a multiple of the other.

Activity: Determine whether the following sets of vectors are linearly independent or
linearly dependent:

=1. i(1,=
0, 0, 0 ) , j (= 0,1, 0, 0 ) , k ( 0, 0, 0,1) in  4.
2.
v1 =( 2, 0, − 1) , v2 = ( −3, − 2, − 5) , v3 = ( −6,1, − 1) , v4 =( −7, 0, 2 ) in 
3
.
=3. i (1,=0, 0,..., 0 ) , j (= 0,1, 0,..., 0 ) , k ( 0, 0, 0,...,1) in m
.
4. 3 x 2 + 3 x + 1, 4 x 2 + x, 3 x 2 + 6 x + 5, − x 2 + 2 x + 7 in p2

Definition: Let H be a subspace of a vector space V. An indexed set of vectors B =

{b 1 ,…, b p } in V is a basis for H if

(i) B is a linearly independent set, and

(ii) the subspace spanned by B coincides with H; that is,
H = Span {b 1 ,...,b p }
The definition of a basis applies to the case when H = V, because any vector space is a
subspace of itself. Thus a basis of V is a linearly independent set that spans V. Observe
that when H ≠ V , condition (ii) includes the requirement that each of the vectors b 1 ,...,b p
must belong to H, because Span { b 1 ,...,b p } contains b 1 ,…,b p , as we saw in lecture 21.

Example 3: Let A be an invertible n × n matrix – say, A = [a 1 … a n]. Then the columns

of A form a basis for Rn because they are linearly independent and they span Rn, by the
Invertible Matrix Theorem.

Example 4: Let e 1 ,…, e n be the columns of the n × n identity matrix, In . That is,

1  0  0 
0  1  0 
e1 =   , e2 =   , ... en =  
     
     
0  0  1 

The set {e 1 , …, e n } is called the standard basis for R n (Fig. 1).

©Virtual University Of Pakistan 73

06- Linear Independence Sets; Bases VU

x3

e3

e2
x2
e1

x1

Figure 1 - The standard basis for R3

3 -4  -2 

Example 5: Let v1 =  0  , v 2 =  1  , and v 3 =  1  . Determine if {v 1 , v 2 , v 3 } is a basis
   
-6   7   5 
for R3.
Solution: Since there are exactly three vectors here in R3, we can use one of any
methods to determine whether they are basis for  3 or not. For this, let solve with help of
matrices. First form a matrix of vectors i.e. matrix A = [v 1 v 2 v 3]. If this matrix is
invertible (i.e. |A| ≠ 0 determinant should be non zero).
For instance, a simple computation shows that det A = 6 ≠ 0 . Thus A is invertible. As in
example 3, the columns of A form a basis for R3.

Example 6: Let S = {1, t, t2, …, tn}. Verify that S is a basis for P n . This basis is called
the standard basis for P n .

Solution: Certainly S spans P n . To show that S is linearly independent, suppose that

c 0 ,…, c n satisfy
c 0 .1 + c 1 t + c 2 t2 + ….. + c n tn = 0 (t) (2)
This equality means that the polynomial on the left has the same values as the zero
polynomial on the right. A fundamental theorem in algebra says that the only polynomial
in P n with more than n zeros is the zero polynomial. That is, (2) holds for all t only if
c 0 = …= c n = 0. This proves that S is linearly independent and hence is a basis for P n .
See Figure 2.

©Virtual University Of Pakistan 74

06- Linear Independence Sets; Bases VU

y=t
2
y=t
y=1

Figure 2 – The standard basis for P 2

Problems involving linear independence and spanning in P n are handled best by a

technique to be discussed later.

Example 7: Check whether the set of vectors {(2, -3, 1), (4, 1, 1), (0, -7, 1)} is basis for
R3?
Solution: The set S = {v 1 , v 2 , v 3 } of vectors in R3 spans V = R3 if
c 1 v 1 + c 2 v 2 + c 3 v 3 = d 1w 1 + d 2w 2 + d3 w3 (*)
with w 1 = (1,0,0), w 2 = (0,1,0) , w 3 = (0,0,1) has at least one solution for every set of
values of the coefficients d 1 , d 2 , d 3 . Otherwise (i.e., if no solution exists for at least some
values of d 1 , d 2 , d 3 ), S does not span V. With our vectors v 1 , v 2 , v 3 , (*) becomes
c 1 (2,-3,1) + c 2 (4,1,1) + c 3 (0,-7,1) = d 1 (1,0,0) + d 2 (0,1,0) + d 3 (0,0,1)
Rearranging the left hand side yields
2 c1 + 4 c2 +0 c3 = 1 d1 +0 d 2 +0 d 3
-3 c1 +1 c2 -7 c3 = 0 d1 +1 d 2 +0 d 3 (A)
1 c1 +1 c2 +1 c3 = 0 d1 +0 d 2 +1 d 3
 2 4 0  c1   d1 
⇒ -3 1 -7  c2  =
d 
 2
 1 1 1  c3   d 3 
 

2 4 0
We now find the determinant of coefficient matrix -3 1 -7  to determine whether the
 1 1 1 
system is consistent (so that S spans V), or inconsistent (S does not span V).

©Virtual University Of Pakistan 75

06- Linear Independence Sets; Bases VU

2 4 0
Now det -3 1 -7  = 2(8) – 4(4) +0 = 0
 1 1 1 
Therefore, the system (A) is inconsistent, and, consequently, the set S does not span the
space V.

Example 8: Check whether the set of vectors

{-4 + 1 t + 3 t2 , 6 + 5 t + 2 t2 , 8 + 4 t + 1 t2} is a basis for P2?
Solution The set S = {p1 (t), p2 (t), p3 (t)} of vectors in P2 spans V = P2 if
c1 p1 (t) + c2 p2 (t) + c3 p3 (t) = d1 q1 (t) + d2 q2 (t) + d3 q3 (t) (*)
with q1(t) = 1 + 0 t + 0 t 2 , q2(t) = 0 + 1 t + 0 t 2 , q3(t) = 0 + 0 t + 1 t 2 has at least
one solution for every set of values of the coefficients d1, d2, d3. Otherwise (i.e., if no
solution exists for at least some values of d1, d2, d3), S does not span V. With our vectors
p1(t), p2(t), p3(t), (*) becomes:
c1 (-4 + 1 t + 3 t2) + c2 (6 + 5 t + 2 t2) + c3 (8 + 4 t + 1 t2) =
d1 (1 + 0 t + 0 t 2 ) + d2 (0 + 1 t + 0 t 2 ) + d3 (0 + 0 t + 1 t 2 )
Rearranging the left hand side yields
(-4 c1 +6 c2 +8 c3)1 + (1 c1 +5 c2 +4 c3) t + (3 c1 +2 c2 +1 c3) t2 =
(1 d1 +0 d2 +0 d3)1 + (0 d1 +1 d2 +0 d3) t + (0 d1 +0 d2 +1 d3) t2
In order for the equality above to hold for all values of t, the coefficients corresponding to
the same power of t on both sides of the equation must be equal. This yields the
following system of equations:
-4 c1 +6 c2 + 8 c3 = 1 d1 +0 d 2 +0 d 3
1 c1 + 5 c2 + 4 c3 = 0 d1 +1 d 2 +0 d 3 (A)
3 c1 + 2 c2 +1 c3 = 0 d1 +0 d 2 +1 d 3
- 4 6 8  c1   d1 
⇒  1 5 4  c2  = d 
 2
 3 2 1  c3   d 3 

- 4 6 8 
We now find the determinant of coefficient matrix  1 5 4  to determine whether the
 3 2 1 
system is consistent (so that S spans V), or inconsistent (S does not span V).
- 4 6 8 
Now det  1 5 4  = -26 ≠ 0. Therefore, the system (A) is consistent, and,
 3 2 1 
consequently, the set S spans the space V.

The set S = {p 1 (t), p 2 (t), p 3 (t)} of vectors in P 2 is linearly independent if the only
solution of
c 1 p 1 (t) + c 2 p 2 (t) + c 3 p 3 (t) = 0 (**)

©Virtual University Of Pakistan 76

06- Linear Independence Sets; Bases VU

is c 1 , c 2 , c 3 = 0. In this case, the set S forms a basis for span S. Otherwise (i.e., if a
solution with at least some nonzero values exists), S is linearly dependent. With our
vectors p 1 (t), p 2 (t), p 3 (t), (2) becomes: c 1 (-4 + 1 t + 3 t2) + c 2 (6 + 5 t + 2 t2) + c 3 (8 +
4 t + 1 t2) = 0 Rearranging the left hand side yields
(-4 c 1 +6 c 2 +8 c 3 )1 + (1 c 1 +5 c 2 +4 c 3 ) t + (3 c 1 +2 c 2 +1 c 3 ) t2 = 0
This yields the following homogeneous system of equations:
-4 c1 +6 c2 + 8 c3 = 0 - 4 6 8  c1  0 
1 c1 + 5 c2 + 4 c3 = 0 ⇒  1 5 4  c2  = 0 
 
3 c1 + 2 c2 +1 c3 = 0 
 3 2 1   
 c3  0 
- 4 6 8 
As det  1 5 4  = -26 ≠ 0. Therefore the set S = {p 1 (t), p 2 (t), p 3 (t)} is linearly
 3 2 1 
independent. Consequently, the set S forms a basis for span S.

 1 0   0 1   0 0   0 0  
Example 9: The set S =   , , ,   is a basis for the vector
  0 0   0 0  1 0   0 1  
space V of all 2 x 2 matrices.
Solution: To verify that S is linearly independent, we form a linear combination of the
vectors in S and set it equal to zero:
1 0  0 1  0 0  0 0  0 0 
c1   + c2   + c3   + c4  =  
0 0  0 0  1 0  0 1  0 0 
 c1 c2  0 0 
This gives c c  = 0 0  , which implies that c 1 = c 2 = c 3 = c 4 = 0. Hence S is
 3 4  
linearly independent.
a b 
To verify that S spans V we take any vector   in V and we must find scalars c 1 , c 2 ,
c d 
c 3 , and c 4 such that
1 0  0 1  0 0  0 0   a b  c c  a b 
c1   + c2   + c3   + c4   =  ⇒  1 2=  
0 0  0 0  1 0  0 1   c d  c3 c4   c d 
We find that c1 = a, c2 = b, c3 = c, and c4 = d so that S spans V.

The basis S in this example is called the standard basis for M22. More generally, the
standard basis for Mmn consists of mn different matrices with a single 1 and zeros for the
remaining entries

Example 10: Show that the set of vectors

 3 6  0 -1 0 -8  1 0 
 
-6  -1 0  -12 -4  -1 2  
, , ,
 3

is a basis for the vector space V of all 2 x 2 matrices (i.e. M 22 ).

©Virtual University Of Pakistan 77

06- Linear Independence Sets; Bases VU

Solution: The set S = {v 1 , v 2 , v 3 , v 4 } of vectors in M 22 spans V = M 22 if

c1 v1 + c2 v2 + c3 v3 + c4 v4 = d 1 w 1 + d2 w2 + d 3 w 3 + d 4 w4
(*)
1 0 0 1 0 0 0 0
w1 =   , w2 =   , w3 =   , w4 = 
1 
with
0 0 0 0 1 0 0
has at least one solution for every set of values of the coefficients d 1 , d 2 , d 3 , d 4 .
Otherwise (i.e., if no solution exists for at least some values of d 1 , d 2 , d 3 , d 4 ), S does not
span V. With our vectors v 1 , v 2 , v 3 , v 4 , (*) becomes:
3 6 0 -1 0 -8  1 0
c1   + c2   + c3   + c4 
3 -6  -1 0 -12 -4  -1 2 

1 0 0 1 0 0 0 0
= d1  + d2 + d3  + d4 
0 0  0


0 1

0 0 1 

Rearranging the left hand side yields

3 c1 +0 c2 +0 c3 +1c4 6 c1 -1c2 - 8 c3 +0 c4 
3 c -1c -12c -1c -6 c +0 c - 4 c +2c  =
 1 2 3 4 1 2 3 4

1 d1 +0 d 2 +0 d 3 +0 d 4 0 d1 +1 d 2 +0 d 3 +0 d 4 
0 d +0 d +1 d +0 d 0 d1 +0 d 2 +0 d 3 +1 d 4 
 1 2 3 4

The matrix equation above is equivalent to the following system of equations

3 c1 + 0 c2 + 0 c3 +1 c4 = 1 d1 +0 d 2 +0 d 3 +0 d 4
6 c1 - 1 c2 - 8 c3 +0 c4 = 0 d1 +1 d 2 +0 d 3 +0 d 4
3 c1 - 1 c2 - 12 c3 - 1 c4 = 0 d1 +0 d 2 +1 d 3 +0 d 4
-6 c1 +0 c2 - 4 c3 + 2 c4 = 0 d1 +0 d 2 +0 d 3 +1 d 4

 3 0 0 1   c1   d1 
 6 -1 -8 0  c   d 
⇒  2 =  2
 3 -1 -12 -1  c3   d 3 
    
-6 0 -4 2  c4   d 4 

3 0 0 1
 6 -1 -8 0 
We now find the determinant of coefficient matrix A =   to determine
 3 -1 -12 -1
 
-6 0 -4 2 
whether the system is consistent (so that S spans V), or inconsistent (S does not span V).

©Virtual University Of Pakistan 78

06- Linear Independence Sets; Bases VU

Now det (A) = 48 ≠ 0. Therefore, the system (A) is consistent, and, consequently, the set
S spans the space V.
Now, the set S = {v 1 , v 2 , v 3 , v 4 } of vectors in M 22 is linearly independent if the only
solution of c 1 v 1 + c 2 v 2 + c 3 v 3 + c 4 v 4 = 0 is c 1 , c 2 , c 3 , c 4 = 0. In this case the set S
forms a basis for span S. Otherwise (i.e., if a solution with at least some nonzero values
exists), S is linearly dependent. With our vectors v 1 , v 2 , v 3 , v 4 , we have
3 6 0 -1 0 -8  1 0  0 0
c1  + c + c + c =  0 
-6  -1 0  -12 -4  -1 2  0
2 3 4
3   
Rearranging the left hand side yields
3 c1 +0 c2 +0 c3 +1 c4 6 c1 - 1 c2 - 8 c3 +0 c4  0 0
3 c - 1 c - 12 c - 1 c 
- 6 c1 +0 c2 - 4 c3 + 2 c4  0
= 
0 
 1 2 3 4

The matrix equation above is equivalent to the following homogeneous equation.

 3 0 0 1   c1  0 
 6 -1 -8 0  c  0 
  2 =  
 3 -1 -12 -1  c3  0 
    
-6 0 -4 2  c4  0 

As det (A) = 48 ≠ 0

Therefore the set S = {v 1 , v 2 , v 3 , v 4 } is linearly independent. Consequently, the set S

forms a basis for span S.

1 -3 - 4 
Example 11: Let v1 = -2  , v 2 =  5  , v 3 =  5  , and H = Span{v1 , v 2 , v 3 }.
   
 -3  7   6 
Note that v 3 = 5v 1 + 3v 2 and show that Span {v 1 , v 2 , v 3 } = Span {v 1 , v 2 }. Then find a
basis for the subspace H.

Solution:

Every vector in Span {v 1 , v 2 } belongs to H because

c1 v1 + c2 v2 = c1 v1 + c2 v2 + 0 v3

©Virtual University Of Pakistan 79

06- Linear Independence Sets; Bases VU

x3
x2

v3

v1
v2

x1

Now let x be any vector in H – say, x = c1v1 + c2v2 + c3v3. Since v3 = 5v1 + 3v2, we may
substitute
x = c 1 v 1 + c 2 v 2 + c 3 (5v 1 + 3v 2 )
= (c 1 + 5c 3 ) v 1 + (c 2 + 3c 3 ) v 2
Thus x is in Span {v 1 , v 2 }, so every vector in H already belongs to Span {v 1 , v 2 }. We
conclude that H and Span {v 1 , v 2 } are actually the same set of vectors. It follows that
{v 1 , v 2 } is a basis of H since {v 1 , v 2 } is obviously linearly independent.

Activity: Show that the following set of vectors is basis for 

3
:

1.
=v1 (1,=
0, 0 ) , v2 (=
0, 2, 1) , v1 ( 3, 0, 1)

2.
= v1 (1,=
2, 3) , v2 (=
0, 1, 1) , v1 ( 0, 1, 3)

The Spanning Set Theorem:

As we will see, a basis is an “efficient” spanning set that contains no unnecessary vectors.
In fact, a basis can be constructed from a spanning set by discarding unneeded vectors.

Theorem 2: (The Spanning Set Theorem) Let S = {v 1 , … , v p } be a set in V and let

H = Span {v 1 , …, v p }.
a. If one of the vectors in S – say, v k – is a linear combination of the
remaining vectors in S, then the set formed from S by removing v k still
spans H.
b. If H ≠ {0} , some subset of S is a basis for H.

©Virtual University Of Pakistan 80

06- Linear Independence Sets; Bases VU

Since we know that span is the set of all linear combinations of some set of vectors and
basis is a set of linearly independent vectors whose span is the entire vector space. The
spanning set is a set of vectors whose span is the entire vector space. "The Spanning set
theorem" is that a spanning set of vectors always contains a subset that is a basis.

Remark: Let V = Rm and let S = {v 1 , v 2 ,…, v n } be a set of nonzero vectors in V.

Procedure:
The procedure for finding a subset of S that is a basis for W = span S is as follows:
Step 1 Write the Equation,
c 1 v 1 + c 2 v 2 + …+ c n v n =0 (3)
Step 2 Construct the augmented matrix associated with the homogeneous system of
Equation (1) and transforms it to reduced row echelon form.
Step 3 The vectors corresponding to the columns containing the leading 1’s form a basis
for W = span S.
Thus if S = {v 1 , v 2 ,…, v 6 } and the leading 1’s occur in columns 1, 3, and 4, then { v 1 , v 3 , v 4 } is
a basis for span S.

Note In step 2 of the procedure above, it is sufficient to transform the augmented matrix to row
echelon form.

Example 12: Let S = {v 1 , v 2 , v 3 , v 4 , v 5 } be a set of vectors in R4, where

v 1 = (1,2,-2,1), v 2 = (-3,0,-4,3), v 3 = (2,1,1,-1), v 4 = (-3,3,-9,6), and v 5 = (9,3,7,-6).
Find a subset of S that is a basis for W = span S.
Solution: Step 1 Form Equation (3),
c 1 (1,2, -2,1) + c 2 (-3,0,-4,3) + c 3 (2,1,1,-1)+ c 4 (-3,3,-9,6) + c 5 (9,3.7,-6) = (0,0,0,0).
Step 2 Equating corresponding components, we obtain the homogeneous system
c1 - 3c2 + 2c3 - 3c4 + 9c5 = 0
2c1 + c3 + 3c4 + 3c5 = 0
-2c1 - 4c2 + c3 - 9c4 + 7c5 = 0
c1 + 3c2 - c3 + 6c4 - 6c5 = 0
The reduced row echelon form of the associated augmented matrix is
1 0 ½ 3/2 3/2 : 0 
0 1 -1/2 3/2 -5/2 : 0 

0 0 0 0 0 : 0
 
0 0 0 0 0 : 0
Step 3 The leading 1’s appear in columns 1 and 2, so {v 1 , v 2 } is a basis for W = span S.

©Virtual University Of Pakistan 81

06- Linear Independence Sets; Bases VU

Two Views of a Basis When the Spanning Set Theorem is used, the deletion of
vectors from a spanning set must stop when the set becomes linearly independent. If
an additional vector is deleted, it will not be a linear combination of the remaining
vectors and hence the smaller set will no longer span V. Thus a basis is a spanning set
that is as small as possible.
A basis is also a linearly independent set that is as large as possible. If S is a basis for V,
and if S is enlarged by one vector – say, w – from V, then the new set cannot be linearly
independent, because S spans V, and w is therefore a linear combination of the elements
in S.

Example 13: The following three sets in R3 show how a linearly independent set can be
enlarged to a basis and how further enlargement destroys the linear independence of the
set. Also, a spanning set can be shrunk to a basis, but further shrinking destroys the
spanning property.

 1   2    1   2   4    1   2   4   7  
                    
 0 ,  3   0 ,  3  , 5    0  ,  3  , 5  , 8  
 0 0    0 0  6    0 0  6  9  
                    

Linearly independent A basis Spans R3 but is

but does not span R3 for R3 linearly dependent

1  0   s  
      
Example 14: Let v1 = 0  , v 2 = 1  , and H =  s : s in R  . then every vector in H is a
 
0  0   0  
  
s 1  0
linear combination of v 1 and v 2 because=  s  s 0 + s 1  . Is {v , v } a basis for H?
      1 2

0 0 0

Solution: Neither v 1 nor v 2 is in H, so {v 1 , v 2 } cannot a basis for H. In fact, {v 1 , v 2 } is a
basis for the plane of all vectors of the form (c 1 , c 2 , 0), but H is only a line.

Activity: Find a Basis for the subspace W in 

3
spanned by the following sets of
vectors:

=
1. v1 (1,=
0, 2 ) , v2 (=
3, 2,1) , v3 (1,
= 0, 6 ) , v4 ( 3, 2,1)

=
2. v1 (1,=
2, 2 ) , v2 ( 3,
= 2,1) , v3 (1,1,
= 7 ) , v4 ( 7 , 6, 4 )

©Virtual University Of Pakistan 82

 06- Linear Independence Sets; Bases VU

Exercises:

Determine which set in exercises 1-4 are bases for R2 or R3. Of the sets that are not bases,
determine which one are linearly independent and which ones span R2 or R3. Justify your
answers.

 1   3   -3  1  -2  0   0 
     
1. 0 , 2 , -5
       
2. -3 , 9 , 0 , -3
             
-2  -4   1   0   0  0   5 

 1  -4  1  0 3 0 

3.  2  ,  -5  4.  -4  ,  3  ,  -5 ,  2 
           
-3  6   3  -1  4  -2 

5. Find a basis for the set of vectors in R3 in the plane x + 2y + z = 0.

6. Find a basis for the set of vectors in R2 on the line y = 5x.

7. Suppose R4 = Span {v 1 , v 2 , v 3 , v 4 }. Explain why {v 1 , v 2 , v 3 , v 4 } is a basis for R4.

8. Explain why the following sets of vectors are not bases for the indicated vector spaces.
(Solve this problem by inspection).
(a) u 1 = (1, 2), u 2 = (0, 3), u 3 = (2, 7) for R2
(b) u 1 = (-1, 3, 2), u 2 = (6, 1, 1) for R3
(c) p 1 = 1 + x + x2, p 2 = x – 1 for P 2
1 1  6 0 3 0  5 1  7 1 
=
(d) A =  ,B  =  ,C =  , D =  ,E   for M 22
 2 3   -1 4  1 7   4 2   2 9 

9. Which of the following sets of vectors are bases for R2?

(a) (2, 1), (3, 0) (b) (4, 1), (-7, -8) (c) (0, 0), (1, 3) (d) (3, 9), (-4, -12)

10. Let V be the space spanned by v 1 = Cos2 x, v 2 = Sin2x, v 3 = cos 2x.

(a) Show that S = {v 1 , v 2 , v 3 } is not a basis for V (b) Find a basis for V

In exercises 11-13, determine a basis for the solution space of the system.

x1 + x2 - x3 = 0 2x1 + x2 + 3x3 = 0
11. -2x1 - x2 + 2x3 = 0 12. x1 + 5x3 = 0
- x1 + x3 = 0 x2 + x3 = 0

©Virtual University Of Pakistan 83

06- Linear Independence Sets; Bases VU

x+ y+ z = 0
3x + 2y - 2z = 0
13.
4x + 3y - z = 0
6x + 5y + z = 0

14. Determine bases for the following subspace of R3

(a) the plane 3x – 2y + 5z = 0 (b) the plane x – y = 0
(c) the line x = 2t, y = -t, z = 4t (d) all vectors of the form (a, b, c), where b = a + c

15. Find a standard basis vector that can be added to the set {v 1 , v 2 } to produce a basis
for R3.
(a) v 1 = (-1, 2, 3), v 2 = (1, -2, -2) (b) v 1 = (1, -1, 0), v 2 = (3, 1, -2)

16. Find a standard basis vector that can be added to the set {v 1 , v 2 } to produce a basis
for R4.
v 1 = (1, -4, 2, -3), v 2 = (-3, 8, -4, 6)

©Virtual University Of Pakistan 84

07-Dimension of a Vector Space VU

Lecture 07

Dimension of a Vector Space

In this lecture, we will focus over the dimension of the vector spaces. The
dimension of a vector space V is the cardinality or the number of vectors in the basis B of
the given vector space. If the basis B has n (say) elements then this number n (called the
dimension) is an intrinsic property of the space V. That is it does not depend on the
particular choice of basis rather, all the bases of V will have the same cardinality. Thus,
we can say that the dimension of a vector space is always unique. The discussion of
dimension will give additional insight into properties of bases.
The first theorem generalizes a well-known result about the vector space Rn.
Note:
A vector space V with a basis B containing n vectors is isomorphic to Rn i.e., there
exist a one-to-one linear transformation from V to Rn.

Theorem 1: If a vector space V has a basis B = {b 1 , …, b n }, then any set in V containing

more than n vectors must be linearly dependent.

Theorem 2: If a vector space V has a basis of n vectors, then every basis of V must
consist of exactly n vectors.

Finite and infinite dimensional vector spaces:

If the vector space V is spanned or
generated by a finite set, then V is said to be finite-dimensional, and the dimension of V,
written as dim V, is the number of vectors in a basis for V. If V is not spanned by a finite
set, then V is said to be infinite-dimensional. That is, if we are unable to find a finite set
that can generate the whole vector space, then such a vector space is called infinite
dimensional.

Note:

(1) The dimension of the zero vector space {0} is defined to be zero.
(2) Every finite dimensional vector space contains a basis.

Example 1: The n dimensional set of real numbers Rn, set of polynomials of order n Pn,
and set of matrices of order m × n Mmn are all finite- dimensional vector spaces.
However, the vector spaces F (- ∞ , ∞ ), C (- ∞ , ∞ ), and Cm (- ∞ , ∞ ) are infinite-
dimensional.

Example 2:
(a) Any pair of non-parallel vectors a, b in the xy-plane, which are necessarily linearly
independent, can be regarded as a basis of the subspace R2. In particular the set of unit
vectors {i, j} forms a basis for R2. Therefore, dim (R2) = 2.

_______________________________________________________________________________________
@Virtual University Of Pakistan 85
07-Dimension of a Vector Space VU

Any set of three non coplanar vectors {a, b, c} in ordinary (physical) space, which will be
necessarily linearly independent, spans the space R3. Therefore any set of such vectors forms a
basis for R3. In particular the set of unit vectors {i, j, k} forms a basis of R3. This basis is called
standard basis for R3. Therefore dim (R3) = 3.

The set of vectors {e 1 , e 2 , …, e n } where

e 1 = (1, 0, 0, 0, …, 0),
e 2 = (0, 1, 0, 0, …, 0),
e 3 = (0, 0, 1, 0, …, 0),
…
…
…
e n = (0, 0, 0, 0, …, 1)
is linearly independent.
Moreover, any vector x = (x 1 , x 2 , …, x n ) in Rn can be expressed as a linear combination of these
vectors as
x = x 1 e 1 + x 2 e 2 + x 3 e 3 +…+ x n e n .
Hence, the set {e 1 , e 2 , … , e n } forms a basis for Rn. It is called the standard basis of Rn, therefore
dim (Rn) = n. Any other set of n linearly independent vectors in Rn will form a non-standard
basis.

(b) The set B = {1, x, x2, … ,xn} forms a basis for the vector space P n of polynomials of degree
< n. It is called the standard basis with dim (P n ) = n + 1.

(c) The set of 2 x 2 matrices with real entries (elements) {u 1 , u 2 , u 3 , u 4 } where

1 0  0 1  0 0 0 0 
u1 =   , u2 =   , u3 =   , u4 =  
0 0  0 0  1 0  0 1 
is a linearly independent and every 2 x 2 matrix with real entries can be expressed as their linear
combination. Therefore, they form a basis for the vector space M 2X2 . This basis is called the
standard basis for M 2X2 with dim (M 2X2 ) = 4.

Note:

(1) dim (Rn) = n { The standard basis has n vectors}.

(2) dim (P n ) = n + 1 { The standard basis has n+1 vectors}.

(3) dim (M m× n ) = mn { The standard basis has mn vectors.}

Example 3: Let W be the subspace of the set of all (2 x 2) matrices defined by

a b 
W = {A =   : 2a – b + 3c + d = 0}.
c d 
Determine the dimension of W.
Solution: The algebraic specification for W can be rewritten as d = -2a + b – 3c.

_______________________________________________________________________________________
@Virtual University Of Pakistan 86
07-Dimension of a Vector Space VU

a b 
Now A=  
c d 
Substituting the value of d, it becomes

a b 
A=  
 c -2a + b - 3c 
This can be written as
a 0  0 b  0 0 
A=   +  + 
 0 -2a  0 b   c - 3c 
1 0  0 1  0 0 
=a   + b  + c 
0 -2  0 1   1 - 3
= a A 1 + bA 2 + cA 3
1 0  0 1 0 0 
where A1 =   , A2 =   , and A 3 =  
0 −2  0 1 1 −3
The matrix A is in W if and only if A = aA 1 + bA 2 + cA 3 , so {A 1 , A 2 , A 3 } is a spanning set for
W. Now, check if this set is a basis for W or not. We will see whether {A 1 , A 2 , A 3 } is linearly
independent or not. {A 1 , A 2 , A 3 } is said to be linearly independent if
aA1 + bA 2 + cA 3 =0 ⇒ a=b=c=0 i.e.,
1 0 0 1 0 0  0 0 
a  +b  +c =  
0 -2  0 1  1 - 3  0 0 
a 0  0 b  0 0  0 0 
0  + 0 + =
b   c - 3c  0 0 

 -2a 
a b  0 0
c =
 −2a + b − 3c  0 0 

Equating the elements, we get

=
a 0,= b 0,= c 0
This implies {A 1 , A 2 , A 3 } is a linearly independent set that spans W. Hence, it’s the basis of W
with dim( W)= 3.

3 -1
Example 4: Let H = Span {v 1 , v 2 }, where v1 =  6  and v 2 =  0  . Then H is the plane
 
 2   1 
studied in Example 10 of lecture 23. A basis for H is {v 1 , v 2 }, since v 1 and v 2 are not
multiples and hence are linearly independent. Thus, dim H = 2.

_______________________________________________________________________________________
@Virtual University Of Pakistan 87
07-Dimension of a Vector Space VU

3v2

x=2v1+3v2
2v2
v2

0
v1
2v1

A coordinate system on a plane H in R3

Example 5: Find the dimension of the subspace

  a - 3b + 6c  
  
  5a + 4 d  
H= : a,b,c,d ∈ R 

 b - 2c - d  
   
 5 d  
Solution: The representative vector of H can be written as
 a − 3b + 6c  1   −3 6 0
 5a + 4d  5  0 0  
  =a   + b   + c   + d  4 
 b − 2c − d  0  1  −2   −1
         
 5d  0  0 0 5
Now, it is easy to see that H is the set of all linear combinations of the vectors

1   -3 6 0

5 0 0 4
v1 =   , v2 =   , v3 =   , v4 =  
0 1  -2   -1
       
0 0 0 5

Clearly, v1 ≠ 0 , v 2 is not a multiple of v 1 , but v 3 is a multiple of v 2 . By the Spanning Set

Theorem, we may discard v 3 and still have a set that spans H. Finally; v 4 is not a linear

_______________________________________________________________________________________
@Virtual University Of Pakistan 88
07-Dimension of a Vector Space VU

combination of v 1 and v 2 . So {v 1 , v 2 , v 4 } is linearly independent and hence is a basis for

H. Thus dim H = 3.

Example 6: The subspaces of R3 can be classified by various dimensions as shown in

Fig. 1.
0-dimensional subspaces:
The only 0-dimensional subspace of R3 is zero space.

1-dimensional subspaces:
1-dimensional subspaces include any subspace spanned by a single non-zero
vector. Such subspaces are lines through the origin.

2-dimensional subspaces:
Any subspace spanned by two linearly independent vectors. Such subspaces are
planes through the origin.

3-dimensional subspaces:
The only 3-dimensional subspace is R3 itself. Any three linearly independent
vectors in R3 span all of R3, by the Invertible Matrix Theorem.

x3
x3
3 dim

0 dim

x2 x1
x1 2 dim
1 dim

Figure 1 – Sample subspaces of R3

Bases for Nul A and Col A:

We already know how to find vectors that span the null
space of a matrix A. The discussion in Lecture 21 pointed out that our method always
produces a linearly independent set. Thus the method produces a basis for Nul A.

_______________________________________________________________________________________
@Virtual University Of Pakistan 89
07-Dimension of a Vector Space VU

2 2 -1 0 1 
-1 -1 2 -3 1 
Example 7: Find a basis for the null space of A =  .
1 1 -2 0 -1
 
0 0 1 1 1
Solution: The null space of A is the solution space of homogeneous system
2x1 + 2x2 - x3 + x5 = 0
- x1 - x2 + 2x3 - 3x4 + x5 = 0
x1 + x2 - 2x3 - x5 = 0
x3 + x4 + x5 = 0
The most appropriate way to solve this system is to reduce its augmented matrix into
reduced echelon form.
2 2 -1 0 1 0
-1 -1 2 -3 1 0 
 R4  R2 , R3  R1
1 1 -2 0 -1 0
 
0 0 1 1 1 0

1 1 -2 0 -1 0
0 0 1 1 1 0 
 R3 − 2 R1 , R3 − 3R2
2 2 -1 0 1 0
 
-1 -1 2 -3 1 0

1 1 -2 0 -1 0
0 0 1 1 1 0 
 R3 − 3R2
0 0 3 0 3 0
 
-1 -1 2 -3 1 0

1 1 -2 0 -1 0
0 0 1 1 1 0  1
 − R3
0 0 0 -3 0 0 3
 
-1 -1 2 -3 1 0

1 1 -2 0 -1 0
0 0 1 1 1 0 
 R4 + R1
0 0 0 1 0 0
 
-1 -1 2 -3 1 0

_______________________________________________________________________________________
@Virtual University Of Pakistan 90
07-Dimension of a Vector Space VU

1 1 -2 0 -1 0
0 0 1 1 1 0 
 R4 + 3R3
0 0 0 1 0 0
 
0 0 0 -3 0 0

1 1 -2 0 -1 0
0 0 1 1 1 0 
 R1 + 2 R2
0 0 0 1 0 0
 
0 0 0 0 0 0
1 1 0 2 1 0
0 0 1 1 1 0 
 R2 − R3 , R1 − 2 R3
0 0 0 1 0 0
 
0 0 0 0 0 0

1 1 0 0 1 0
0 0 1 0 1 0 
 
0 0 0 1 0 0
 
0 0 0 0 0 0
Thus, the reduced row echelon form of the augmented matrix is
1 1 0 0 1 0
0 0 1 0 1 0 

0 0 0 1 0 0
 
0 0 0 0 0 0
which corresponds to the system
1x1 +1 x2 + 1 x5 = 0
1 x3 + 1 x5 = 0
1 x4 =0
0=0
No equation of this system has a form zero = nonzero. Therefore, the system is
consistent. Since the number of unknowns is more than the number of equations, we will
assign some arbitrary value to some variables. This will lead to infinite many solutions of
the system.

_______________________________________________________________________________________
@Virtual University Of Pakistan 91
07-Dimension of a Vector Space VU

x1 = - 1x2 -1x5
x2 = s
x3 = - 1x5
x4 = 0
x5 = t
The general solution of the given system is
x1 = - s - t , x2 = s , x3 = - t , x4 = 0 , x5 = t
Therefore, the solution vector can be written as
 x1  -s - t  -s  -t  -1 -1
x   s   s  0  1 0 
 2          
 x3  =  -t  =  0  + -t  = s  0  +t -1
           
 x4   0   0   0  0  0 
 x5   t   0   t   0   1 
-1 -1
1  0 
   
which shows that the vectors v1 = 0  and v 2 = -1 span the solution space .Since they
   
0  0 
0  1 
are also linearly independent,{v1,v2} is a basis for Nul A.

The next two examples describe a simple algorithm for finding a basis for the column
space.

1 4 0 2 0 
0 0 1 -1 0 
Example 8: Find a basis for Col B, where B = [ b1 b2 , ..., b5 ] =  
0 0 0 0 1 
 
0 0 0 0 0 
Solution Each non-pivot column of B is a linear combination of the pivot columns. In
fact, b 2 = 4b 1 and b 4 = 2b 1 – b 3 . By the Spanning Set Theorem, we may discard b 2 and
b 4 and {b 1 , b 3 , b 5 } will still span Col B. Let
 1   0   0  
      
 0 1 0 
S = { b1 , b3 , b5 } =    ,   ,   
  0   0  1  
      
 0  0  0   
Since b 1 ≠ 0 and no vector in S is a linear combination of the vectors that precede it, S is
linearly independent. Thus S is a basis for Col B.

What about a matrix A that is not in reduced echelon form? Recall that any
linear dependence relationship among the columns of A can be expressed in the form Ax

_______________________________________________________________________________________
@Virtual University Of Pakistan 92
07-Dimension of a Vector Space VU

= 0, where x is a column of weights. (If some columns are not involved in a particular
dependence relation, then their weights are zero.) When A is row reduced to a matrix B,
the columns of B are often totally different from the columns of A. However, the
equations Ax = 0 and Bx = 0 have exactly the same set of solutions. That is, the columns
of A have exactly the same linear dependence relationships as the columns of B.

Elementary row operations on a matrix do not affect the linear dependence relations
among the columns of the matrix.

Example 9: It can be shown that the matrix

1 4 0 2 -1
 3 12 1 5 5 
A = [ a1 a2 ... a5 ] =  
2 8 1 3 2 
 
 5 20 2 8 8 
is row equivalent to the matrix B in Example 8. Find a basis for Col A.
Solution: In Example 8, we have seen that b2 = 4 b1 and b4 = 2b1 - b3
so we can expect that a 2 = 4 a1 and a4 = 2a1 - a3 . This is indeed the case.
Thus, we may discard a 2 and a 4 while selecting a minimal spanning set for Col A. In fact,
{a 1 , a 3 , a 5 } must be linearly independent because any linear dependence relationship
among a 1 , a 3 , a 5 would imply a linear dependence relationship among b 1 , b 3 , b 5 . But we
know that {b 1 , b 3 , b 5 } is a linearly independent set. Thus {a 1 , a 3 , a 5 } is a basis for Col A.
The columns we have used for this basis are the pivot columns of A.

Examples 8 and 9 illustrate the following useful fact.

Theorem 3: The pivot columns of a matrix A form a basis for Col A.

Proof: The general proof uses the arguments discussed above. Let B be the reduced
echelon form of A. The set of pivot columns of B is linearly independent, for no vector in
the set is a linear combination of the vectors that precede it. Since A is row equivalent to
B, the pivot columns of A are linearly independent too, because any linear dependence
relation among the columns of A corresponds to a linear dependence relation among the
columns of B. For this same reason, every non-pivot column of A is a linear combination
of the pivot columns of A. Thus the non-pivot columns of A may be discarded from the
spanning set for Col A, by the Spanning Set Theorem. This leaves the pivot columns of A
as a basis for Col A.

Note: Be careful to use pivot columns of A itself for the basis of Col A. The columns of
an echelon form B are often not in the column space of A. For instance, the columns of
the B in Example 8 all have zeros in their last entries, so they cannot span the column
space of the A in Example 9.

_______________________________________________________________________________________
@Virtual University Of Pakistan 93
07-Dimension of a Vector Space VU

1 -2 
Example 10: Let v1 = -2  and v 2 =  7  . Determine if {v 1 , v 2 } is a basis for R3. Is {v 1 ,
 
 3  -9 
2
v 2 } a basis for R ?
 1 -2  1 -2 
Solution Let A = [v 1 v 2]. Row operations show that A = -2 7   0 3  . Not every
 3 -9  0 0 
row of A contains a pivot position. So the columns of A do not span R3, by Theorem 4 in
Lecture 6. Hence {v 1 , v 2 } is not a basis for R3. Since v 1 and v 2 are not in R2, they cannot
possibly be a basis for R2. However, since v 1 and v 2 are obviously linearly independent,
they are a basis for a subspace of R3, namely, Span {v 1 , v 2 }.

1 6 2 -4 

Example 11: Let v1 = -3 , v 2 =  2  , v 3 = -2  , v4 =  -8 . Find a basis for the subspace
 4  -1  3   9 
W spanned by {v 1 , v 2 , v 3 , v 4 }.
Solution: Let A be the matrix whose column space is the space spanned by {v 1 , v 2 , v 3 ,
v 4 },
 1 6 2 -4 
A = -3 2 -2 -8
 4 -1 3 9 
Reduce the matrix A into its echelon form in order to find its pivot columns.
 1 6 2 -4 
A = -3 2 -2 -8
 4 -1 3 9 
1 6 2 -4 
 0 20 4 -20  by R2 + 3R1 , R3 − 4 R1
0 -25 -5 25 
1 6 2 -4 
 0 5 1 -5 by R2 , − R3 , R3 − R2
1 1
4 5
0 0 0 0 
The first two columns of A are the pivot columns and hence form a basis of Col A = W.
Hence {v 1 , v 2 } is a basis for W.
Note that the reduced echelon form of A is not needed in order to locate the pivot
columns.

_______________________________________________________________________________________
@Virtual University Of Pakistan 94
07-Dimension of a Vector Space VU

Procedure:
Basis and Linear Combinations
Given a set of vectors S = {v 1 , v 2 , …,v k } in Rn, the following procedure produces a subset
of these vectors that form a basis for span (S) and expresses those vectors of S that are
not in the basis as linear combinations of the basis vector.
Step1: Form the matrix A having v 1 , v 2 ,..., v k as its column vectors.
Step2: Reduce the matrix A to its reduced row echelon form R, and let
w 1 , w 2 ,…, w k be the column vectors of R.
Step3: Identify the columns that contain the leading entries i.e., 1’s in R. The
corresponding column vectors of A are the basis vectors for span (S).
Step4: Express each column vector of R that does not contain a leading entry as
a linear combination of preceding column vector that do contain leading entries
(we will be able to do this by inspection). This yields a set of dependency
equations involving the column vectors of R. The corresponding equations for the
column vectors of A express the vectors which are not in the basis as linear
combinations of basis vectors.

Example 12: Basis and Linear Combinations

(a) Find a subset of the vectors v 1 = (1, -2, 0, 3), v 2 = (2, -4, 0, 6), v 3 = (-1, 1, 2, 0) and
v 4 = (0, -1, 2, 3) that form a basis for the space spanned by these vectors.
(b) Express each vector not in the basis as a linear combination of the basis vectors.
Solution: (a) We begin by constructing a matrix that has v 1 , v 2 , v 3 , v 4 as its column
vectors
 1 2 -1 0 
-2 -4 1 -1
 
0 0 2 2
 
 3 6 0 3
↑ ↑ ↑ ↑ (A)
v1 v 2 v 3 v 4
Finding a basis for column space of this matrix can solve the first part of our problem.
Transforming Matrix to Reduced Row Echelon Form:
1 2 -1 0
-2 -4 1 -1

0 0 2 2
 
3 6 0 3
1 2 -1 0 
0 0 -1 -1  2 R1 + R2

0 0 2 2  -3R1 + R4
 
0 0 3 3 

_______________________________________________________________________________________
@Virtual University Of Pakistan 95
07-Dimension of a Vector Space VU

1 2 -1 0 
0 0 1 1 
  - 1R2
0 0 2 2 
 
0 0 3 3 
1 2 -1 0
0 0 1 1  -2R2 + R3

0 0 0 0  -3R2 + R4
 
0 0 0 0
1 2 0 1
0 0 1 1 
 R2 + R1
0 0 0 0
 
0 0 0 0
Labeling the column vectors of the resulting matrix as w 1 , w 2 , w 3 and w 4 yields
1 2 0 1
0 0 1 1 

0 0 0 0
  (B)
0 0 0 0
↑ ↑ ↑ ↑
w1 w2 w3 w4
The leading entries occur in column 1 and 3 so {w 1 , w 3 } is a basis for the column space
of (B) and consequently {v 1 , v 3 } is the basis for column space of (A).
(b) We shall start by expressing w 2 and w 4 as linear combinations of the basis vector w 1
and w 3 . The simplest way of doing this is to express w 2 and w 4 in term of basis vectors
with smaller subscripts. Thus we shall express w 2 as a linear combination of w 1 , and we
shall express w 4 as a linear combination of w 1 and w 3. By inspection of (B), these linear
combinations are w 2 = 2w 1 and w 4 = w 1 + w 3 . We call them the dependency equations.
The corresponding relationship of (A) are v 3 = 2v 1 and v 5 = v 1 + v 3 .

Example 13: Basis and Linear Combinations

(a) Find a subset of the vectors v 1 = (1, -1, 5, 2), v 2 = (-2, 3, 1, 0), v 3 = (4, -5, 9, 4),
v 4 = (0, 4, 2, -3) and v 5 = (-7, 18, 2, -8) that form a basis for the space spanned by these
vectors.
(b) Express each vector not in the basis as a linear combination of the basis vectors
Solution: (a) We begin by constructing a matrix that has v 1 , v 2 , ... , v 5 as its column
vectors

_______________________________________________________________________________________
@Virtual University Of Pakistan 96
07-Dimension of a Vector Space VU

1 -2 4 0 -7 
-1 3 -5 4 18 
 
5 1 9 2 2 
  (A)
2 0 4 -3 -8 
↑ ↑ ↑ ↑ ↑
v1 v2 v3 v4 v5
Finding a basis for column space of this matrix can solve the first part of our problem.
Transforming Matrix to Reduced Row Echelon Form:
1 -2 4 0 -7 
-1 3 -5 4 18 

5 1 9 2 2 
 
2 0 4 -3 -8 
1 -2 4 0 -7 
0 1 -1 4 11  R1 + R2
  -5R1 + R3
0 11 -11 2 37 
  -2R1 + R4
0 4 -4 -3 6 
1 -2 4 0 -7 
0 1 -1 4 11  -11R2 + R3

0 0 0 - 42 - 84  -4R2 + R4
 
0 0 0 -19 - 38 
1 -2 4 0 -7 
0 1 -1 4 11 
 (-1/42)R3
0 0 0 1 2 
 
0 0 0 -19 - 38 
1 -2 4 0 -7 
0 1 -1 4 11 
 19R3 + R4
0 0 0 1 2 
 
0 0 0 0 0 
1 -2 4 0 -7 
0 1 -1 0 3 
 (-4)R3 + R2
0 0 0 1 2 
 
0 0 0 0 0 
1 0 2 0 -1 
0 1 -1 0 3 
 2R2 + R1
0 0 0 1 2 
 
0 0 0 0 0 

_______________________________________________________________________________________
@Virtual University Of Pakistan 97
07-Dimension of a Vector Space VU

Denoting the column vectors of the resulting matrix by w 1 , w 2 , w 3 , w 4 , and w 5 yields

1 0 2 0 -1 
0 1 -1 0 3 

0 0 0 1 2 
  (B)
0 0 0 0 0 
↑ ↑ ↑ ↑ ↑
w1 w2 w3 w4 w5
The leading entries occur in columns 1,2 and 4 so that {w1, w2, w4} is a basis for the
column space of (B) and consequently {v1, v2, v4} is the basis for column space of (A).
(b) We shall start by expressing w 3 and w 5 as linear combinations of the basis vector w 1 ,
w 2 , w 4 . The simplest way of doing this is to express w 3 and w 5 in term of basis vectors
with smaller subscripts. Thus we shall express w 3 as a linear combination of w 1 and w 2 ,
and we shall express w 5 as a linear combination of w 1 , w 2 , and w 4. By inspection of (B),
these linear combination are w 3 = 2w 1 – w 2 and w 5 = -w 1 + 3w 2 + 2w 4 .
The corresponding relationship of (A) are v 3 = 2v 1 – v 2 and v 5 = -v 1 + 3v 2 + 2v 4 .

Example 14: Basis and Linear Combinations

(a) Find a subset of the vectors v 1 = (1, -2, 0, 3), v 2 = (2, -5, -3, 6), v 3 = (0, 1, 3, 0),
v 4 = (2, -1, 4, -7) and v 5 = (5 , -8, 1, 2) that form a basis for the space spanned by these
vectors.
(b) Express each vector not in the basis as a linear combination of the basis vectors.
Solution: (a) We begin by constructing a matrix that has v 1 , v 2 , ... , v 5 as its column
vectors
1 2 0 2 5
-2 -5 1 -1 -8
 
 0 -3 3 4 1 
 
 3 6 0 -7 2 
↑ ↑ ↑ ↑ ↑ (A)
v1 v 2 v 3 v 4 v5
Finding a basis for column space of this matrix can solve the first part of our problem.
Reducing the matrix to reduced-row echelon form and denoting the column vectors of the
resulting matrix by w 1 , w 2 , w 3 , w 4 , and w 5 yields
1 0 2 0 1
0 1 -1 0 1 

 0 0 0 1 1
  (B)
0 0 0 0 0
↑ ↑ ↑ ↑ ↑
w1 w2 w3 w 4 w5
The leading entries occur in columns 1, 2 and 4 so {w 1 , w 2 , w 4 } is a basis for the column
space of (B) and consequently {v 1 , v 2 , v 4 } is the basis for column space of (A).

_______________________________________________________________________________________
@Virtual University Of Pakistan 98
07-Dimension of a Vector Space VU

(b) Dependency equations are w 3 = 2w 1 – w 2 and w 5 = w 1 + w 2 + w 4

The corresponding relationship of (A) are v 3 = 2v 1 – v 2 and v 5 = v 1 + v 2 + v 4

Subspaces of a Finite-Dimensional Space: The next theorem is a natural counterpart to

the Spanning Set Theorem.

Theorem 5: Let H be a subspace of a finite-dimensional vector space V. Any linearly

independent set in H can be expanded, if necessary, to a basis for H. Also, H is finite-
dimensional and dim H ≤ dim V .

When the dimension of a vector space or subspace is known, the search for a basis is
simplified by the next theorem. It says that if a set has the right number of elements, then
one has only to show either that the set is linearly independent or that it spans the space.
The theorem is of critical importance in numerous applied problems (involving
differential equations or difference equations, for example) where linear independence is
much easier to verify than spanning.

Theorem 5 (The Basis Theorem): Let V be a p-dimensional vector space, p> 1. Any
linearly independent set of exactly p elements in V is automatically a basis for V. Any set
of exactly p elements that spans V is automatically a basis for V.

The Dimensions of Nul A and Col A: Since the pivot columns of a matrix A form a
basis for Col A, we know the dimension of Col A as soon as we know the pivot columns.
The dimension of Nul A might seem to require more work, since finding a basis for Nul
A usually takes more time than a basis for Col A. Yet, there is a shortcut.

Let A be an m × n matrix, and suppose that the equation Ax = 0 has k free variables.
From lecture 21, we know that the standard method of finding a spanning set for Nul A
will produce exactly k linearly independent vectors say, u 1 , … , u k , one for each free
variable. So {u 1 , … , u k } is a basis for Nul A, and the number of free variables determines
the size of the basis. Let us summarize these facts for future reference.

The dimension of Nul A is the number of free variables in the equation Ax = 0, and the
dimension of Col A is the number of pivot columns in A.

Example 15: Find the dimensions of the null space and column space of
-3 6 -1 1 -7 
A =  1 -2 2 3 -1
 2 -4 5 8 -4 
Solution: Row reduce the augmented matrix [A 0] to echelon form and obtain
1 -2 2 3 -1 0 
0 0 1 2 -2 0 
 
0 0 0 0 0 0 

_______________________________________________________________________________________
@Virtual University Of Pakistan 99
07-Dimension of a Vector Space VU

Writing it in equations form, we get

x1 − 2 x2 + 2 x3 + 3 x4 − x5 =
0
x3 + 2 x4 − 2 x5 =
0
Since the number of unknowns is more than the number of equations, we will introduce
free variables here (say) x 2 , x 4 and x 5 . Hence the dimension of Nul A is 3. Also dim Col
A is 2 because A has two pivot columns.

Example 16: Decide whether each statement is true or false, and give a reason for each
answer. Here V is a non-zero finite-dimensional vector space.
1. If dim V = p and if S is a linearly dependent subset of V, then S contains more than
p vectors.
2. If S spans V and if T is a subset of V that contains more vectors than S, then T is
linearly dependent.
Solution:
1. False. Consider the set {0}.
2. True. By the Spanning Set Theorem, S contains a basis for V; call that basis S ′ .
Then T will contain more vectors than S ′ . By Theorem 1, T is linearly dependent.

_______________________________________________________________________________________
@Virtual University Of Pakistan 100
07-Dimension of a Vector Space VU

Exercises:

For each subspace in exercises 1-6, (a) find a basis and (b) state the dimension.

  2c  
  s − 2t     
   a − b  
1.  s + t : s, t in R  2.  : a, b, c in R 
   
 3t    b − 3c 
     a + 2b  

  a − 4b − 2c    3a + 6b − c  
     
 2a + 5b − 4c    6a − 2b − 2c  
3.   : a, b, c in R  4.  : a, b, c in R 

 − a + 2c   
 −9a + 5b + 3c  
  −3a + 7b + 6c     −3a + b + c  

5. {(a, b, c): a – 3b + c = 0, b – 2c = 0, 2b – c = 0}

6 {(a, b, c, d): a - 3b + c = 0}

7. Find the dimension of the subspace H of R2 spanned by

 2   −4   −3
 −5 , 10  , 6 
     

8. Find the dimension of the subspace spanned by the given vectors.

1  3 9   −7 
0  , 1  ,  4  ,  −3 
       
 2  1   −2  1 

Determine the dimensions of Nul A and Col A for the matrices shown in exercises 9 to
12.

1 −6 9 0 −2  1 3 −4 2 −1 6 
0 1 2 −4 5  0 0 1 −3 7 0 
9. A =  10. A = 
0 0 0 5 1 0 0 0 1 4 −3
   
0 0 0 0 0 0 0 0 0 0 0

1 −1 0 
1 0 9 5 
11. A =   12. A = 0 4 7 
0 0 1 −4  0 0 5 

_______________________________________________________________________________________
@Virtual University Of Pakistan 101
07-Dimension of a Vector Space VU

13. The first four Hermite polynomials are 1, 2t, -2 + 4t2, and -12t + 8t3. These
polynomials arise naturally in the study of certain important differential equations in
mathematical physics. Show that the first four Hermite polynomials form a basis of P 3 .

14. Let B be the basis of P 3 consisting of the Hermite polynomials in exercise 13, and let
p (t) = 7 – 12 t – 8 t2 + 12 t3. Find the coordinate vector of p relative to B.

15. Extend the following vectors to a basis for R5:

 −9  9  6 
 −7  4  7 
     
v1 = 8  , v 2 = 1  , v 3 =  −8 
     
 −5  6  5 
7   −7   −7 

_______________________________________________________________________________________
@Virtual University Of Pakistan 102
08- Rank VU

Lecture 08
Rank

With the help of vector space concepts, for a matrix several interesting and useful
relationships in matrix rows and columns have been discussed.
For instance, imagine placing 2000 random numbers into a 40 x 50 matrix A and then
determining both the maximum number of linearly independent columns in A and the
maximum number of linearly independent columns in AT (rows in A). Remarkably, the
two numbers are the same. Their common value is called the rank of the matrix. To
explain why, we need to examine the subspace spanned by the subspace spanned by the
rows of A.

The Row Space: If A is an m × n matrix, each row of A has n entries and thus can be
identified with a vector in Rn. The set of all linear combinations of the row vectors is
called the row space of A and is denoted by Row A. Each row has n entries, so Row A is
a subspace of Rn. Since the rows of A are identified with the columns of AT, we could
also write Col AT in place of Row A.
-2 -5 8 0 -17  r1 = (-2,-5,8,0,-17)
 1 3 -5 1 5  r2 = (1,3,-5,1,5)
Example 1: Let A =   and
 3 11 -19 7 1  r3 = (3,11,-19,7,1)
 
 1 7 -13 5 -3  r4 = (1,7,-13,5,-3)
5
The row space of A is the subspace of R spanned by {r 1 , r 2 , r 3 , r 4 }. That is, Row A =
Span {r 1 , r 2 , r 3 , r 4 }. Naturally, we write row vectors horizontally; however, they could
also be written as column vectors
Example: Let
r1 = (2,1,0)
2 1 0  r2 = (3,-1,4)
A=   and
3 -1 4 

That is Row A=Span {r 1 , r 2 }.

We could use the Spanning Set Theorem to shrink the spanning set to a
basis.
Some times row operation on a matrix will not give us the required information but row
reducing certainly worthwhile, as the next theorem shows

Theorem 1: If two matrices A and B are row equivalent, then their row spaces are the
same. If B is in echelon form, the nonzero rows of B form a basis for the row space of A
as well as B.

Theorem 2: If A and B are row equivalent matrices, then

©Virtual University Of Pakistan 103

08- Rank VU

(a) A given set of column vectors of A is linearly independent if and only if the
corresponding column vectors of B are linearly independent.
(b) A given set of column vector of A forms a basis for the column space of A if and only
if the corresponding column vector of B forms a basis for the column space of B.

Example 2: (Bases for Row and Column Spaces)

 1 -3 4 -2 5 4 
 2 -6 9 -1 8 2 
Find the bases for the row and column spaces of A =  .
 2 -6 9 -1 9 7 
 
-1 3 -4 2 -5 -4 
Solution: We can find a basis for the row space of A by finding a basis for the row
space of any row-echelon form of A.
1 -3 4 -2 5 4
2 -6 9 -1 8 2 
Now 
2 -6 9 -1 9 7
 
-1 3 -4 2 -5 - 4
1 -3 4 -2 5 4 
0 0 1 3 -2 - 6 
-2 R1 + R2
 -2 R1 + R3
0 0 1 3 -1 -1 
  R1 + R4
0 0 0 0 0 0 
1 -3 4 -2 5 4 
0 0 1 3 -2 - 6 
 - 1R2 + R3
0 0 0 0 1 5 
 
0 0 0 0 0 0 
1 -3 4 -2 5 4 
0 0 1 3 -2 -6 
Row-echelon form of A: R =  
0 0 0 0 1 5 
 
0 0 0 0 0 0 
Here Theorem 1 implies that that the non zero rows are the basis vectors of the matrix.
So these bases vectors are
r1 = [1 -3 4 -2 5 4]
r2 = [ 0 0 1 3 -2 -6]
r3 = [ 0 0 0 0 1 5]
A and R may have different column spaces, we cannot find a basis for the column space
of A directly from the column vectors of R. however, it follows from the theorem (2b) if

©Virtual University Of Pakistan 104

08- Rank VU

we can find a set of column vectors of R that forms a basis for the column space of R,
then the corresponding column vectors of A will form a basis for the column space of A.

The first, third, and fifth columns of R contains the leading 1’s of the row vectors, so
1  4 5
0  1  -2 
c1′ =   c′3 =   c5′ =  
0  0 1
     
0  0 0
form a basis for the column space of R, thus the corresponding column vectors of A
1 4 5
2 9 8
namely, c1 =   c3 =   c5 =  
2 9 9
     
-1 -4  -5
form a basis for the column space of A.

Example:
The matrix

1 -2 5 0 3 
0 1 3 0 0 
R=  
0 0 0 1 0 
 
0 0 0 0 0 
is in row-echelon form.
The vectors
r1 = [1 -2 5 0 3 ]
r2 = [ 0 1 3 0 0 ]
r3 = [ 0 0 0 1 0 ]
form a basis for the row space of R, and the vectors
1  -2  0 
0  1  0 
c1 =   , c2 =   , c3 =  
0  0  1 
     
0  0  0 
form a basis for the column space of R.

©Virtual University Of Pakistan 105

08- Rank VU

Example 3: (Basis for a Vector Space using Row Operation)

Find bases for the space spanned by the vectors
v1 = (1,-2,0,0,3) v 2 = (2,-5,-3,-2,6)
v 3 = (0,5,15,10,0) v4 = (2,6,18,8,6)
Solution: The space spanned by these vectors is the row space of the matrix
1 -2 0 0 3
 2 -5 -3 -2 6 
 
 0 5 15 10 0 
 
 2 6 18 8 6 

Transforming Matrix to Row Echelon Form:

1 -2 0 0 3
2 -5 -3 -2 6 

0 5 15 10 0
 
2 6 18 8 6
1 -2 0 0 3
0 1 3 2 0 
(-2)R1 + R2
 (-2)R1 + R4
0 5 15 10 0
  (-1)R2
0 10 18 8 0
1 -2 0 0 3
0 1 3 2 0  (-5)R2 + R3

0 0 0 0 0  (-10)R2 + R4
 
0 0 -12 -12 0
1 -2 0 0 3
0 1 3 2 0 
 R34
0 0 -12 -12 0
 
0 0 0 0 0
1 -2 0 0 3
0 1 3 2 0 
 (-1/12)R3
0 0 1 1 0
 
0 0 0 0 0

©Virtual University Of Pakistan 106

08- Rank VU

1 -2 0 0 3
0 1 3 2 0 
Therefore, R=  
0 0 1 1 0 
 
0 0 0 0 0 
The non-zero row vectors in this matrix are
w1 = (1,-2,0,0,3), w 2 = (0,1,3,2,0), w 3 = (0,0,1,1,0)
These vectors form a basis for the row space and consequently form a basis for the
subspace of R5 spanned by v 1 , v 2 , v 3 .

Example 4: (Basis for the Row Space of a Matrix)

1 -2 0 0 3
 2 -5 -3 -2 6 
Find a basis for the row space of A =   consisting entirely of row
 0 5 15 10 0 
 
 2 6 18 8 6 
vectors from A.
Solution: We find AT; then we will use the method of example (2) to find a basis for the
column space of AT; and then we will transpose again to convert column vectors back to
row vectors. Transposing A yields
1 2 0 2
-2 -5 5 6 
 
AT =  0 -3 15 18
 
 0 -2 10 8 
 3 6 0 6 
Transforming Matrix to Row Echelon Form:
1 2 0 2
-2 -5 5 6 

0 -3 15 18 
 
0 -2 10 8
 3 6 0 6 
1 2 0 2 
0 -1 5 10 
 2 R1 + R2
0 -3 15 18 
  (-3)R1 + R5
0 -2 10 8 
 0 0 0 0 

©Virtual University Of Pakistan 107

08- Rank VU

1 2 0 2 
0 1 -5 -10 

0 -3 15 18  (-1)R2
 
0 -2 10 8 
 0 0 0 0 
1 2 0 2 
0 1 -5 -10 
 (3)R2 + R3
0 0 0 -12 
  (2)R2 + R4
0 0 0 -12 
 0 0 0 0 
1 2 0 2 
0 1 -5 -10 

0 0 0 1  (-1/12)R3
 
0 0 0 -12 
 0 0 0 0 
1 2 0 2 
0 1 -5 -10 

0 0 0 1  12 R3 + R4
 
0 0 0 0 
 0 0 0 0 
1 2 0 2 
0 1 -5 -10 

Now R = 0 0 0 1 
 
0 0 0 0 
0 0 0 0 
The first, second and fourth columns contain the leading 1’s, so the corresponding
column vectors in AT form a basis for the column space of AT; these are
1 2 2
-2   -5 6
     
c1 =  0  , c2 =  -3 and c4 = 18
     
 0 -2
  8 
 3  6   6 
Transposing again and adjusting the notation appropriately yields the basis vectors
r1 = [1 -2 0 0 3] , r2 = [ 2 -5 -3 -2 6] and r4 = [ 2 6 18 8 6]
for the row space of A.

©Virtual University Of Pakistan 108

08- Rank VU

The following example shows how one sequence of row operations on A leads to bases
for the three spaces: Row A, Col A, and Nul A.

Example 5: Find bases for the row space, the column space and the null space of the
matrix

-2 -5 8 0 -17 
 1 3 -5 1 5 
A=  
 3 11 -19 7 1 
 
 1 7 -13 5 -3 

Solution: To find bases for the row space and the column space, row reduce A to an
1 3 -5 1 5 
0 1 -2 2 -7 
echelon form: A  B =  
0 0 0 -4 20 
 
0 0 0 0 0 
By Theorem (1), the first three rows of B form a basis for the row space of A (as well as
the row space of B). Thus Basis for Row A:
{(1, 3, -5, 1, 5), (0, 1, -2, 2, -7), (0, 0, 0, -4, 20)}
For the column space, observe from B that the pivots are in columns 1, 2 and 4. Hence
columns 1, 2 and 4 of A (not B) form a basis for Col A:
  -2   -5   0  
      
 1 3 1 
Basis for Col A :    ,   ,   
  3  11  7  
      
 1   7  5  
Any echelon form of A provides (in its nonzero rows) a basis for Row A and also
identifies the pivot columns of A for Col A. However, for Nul A, we need the reduced
echelon form. Further row operations on B yield
1 0 1 0 1 
0 1 -2 0 3 
A BC= 
0 0 0 1 -5
 
0 0 0 0 0 
The equation Ax = 0 is equivalent to Cx = 0, that is,
x1 + x3 + x5 = 0
x2 - 2x3 + 3x5 = 0
x4 - 5x5 = 0

©Virtual University Of Pakistan 109

08- Rank VU

So x 1 = -x 3 – x 5 , x 2 = 2x 3 – 3x 5 , x 4 = 5x 5 , with x 3 and x 5 free variables. The usual

calculations (discussed in lecture 21) show that
 -1  -1 
    
  2  -3 
Basis for Nul A :   1  ,  0  
 0   5  
    
  0   1  

Observe that, unlike the bases for Col A, the bases for Row A and Nul A have no simple
connection with the entries in A itself.

Note:
1. Although the first three rows of B in Example (5) are linearly independent, it is wrong
to conclude that the first three rows of A are linearly independent. (In fact, the third
row of A is 2 times the first row plus 7 times the second row).
2. Row operations do not preserve the linear dependence relations among the rows of a
matrix.

Definition: The rank of A is the dimension of the column space of A.

Since Row A is the same as Col AT, the dimension of the row space of A is the rank of
AT. The dimension of the null space is sometimes called the nullity of A.

Theorem 3: (The Rank Theorem) The dimensions of the column space and the row
space of an m × n matrix A are equal. This common dimension, the rank of A, also equals
the number of pivot positions in A and satisfies the equation
rank A + dim Nul A = n

Example 6:
(a) If A is a 7 × 9 matrix with a two – dimensional null space, what is the rank of A?
(b). Could a 6 × 9 matrix have a two – dimensional null space?
Solution:
(a) Since A has 9 columns, (rank A) + 2 = 9 and hence rank A = 7.
(b) No, If a 6 × 9 matrix, call it B, had a two – dimensional null space, it would have to
have rank 7, by the Rank Theorem. But the columns of B are vectors in R6 and so the
dimension of Col B cannot exceed 6; that is, rank B cannot exceed 6.

The next example provides a nice way to visualize the subspaces we have been studying.
Later on, we will learn that Row A and Nul A have only the zero vector in common and
are actually “perpendicular” to each other. The same fact will apply to Row AT (= Col A)
and Nul AT. So the figure in Example (7) creates a good mental image for the general
case.

©Virtual University Of Pakistan 110

08- Rank VU

 3 0 −1
Example 7:= Let A  3 0 −1 . It is readily checked that Nul A is the x 2 – axis, Row A
 4 0 5 
is the x 1 x 3 – plane, Col A is the plane whose equation is x 1 – x 2 = 0 and Nul AT is the set
of all multiples of (1, -1, 0). Figure 1 shows Nul A and Row A in the domain of the linear
transformation x → Ax; the range of this mapping, Col A, is shown in a separate copy of
R3, along with Nul AT.
x3 x3

0 0
Nul A Nul AT x2

x2 Col A
Row A
x1
x1
3
R
R3
Figure 1 – Subspaces associated with a matrix A

Applications to Systems of Equations:

The Rank Theorem is a powerful tool for processing information about systems of
linear equations. The next example simulates the way a real-life problem using linear
equations might be stated, without explicit mention of linear algebra terms such as
matrix, subspace and dimension.

Example 8: A scientist has found two solutions to a homogeneous system of 40

equations in 42 variables. The two solutions are not multiples and all other solutions can
be constructed by adding together appropriate multiples of these two solutions. Can the
scientist be certain that an associated non-homogeneous system (with the same
coefficients) has a solution?
Solution: Yes. Let A be the 40 × 42 coefficient matrix of the system. The given
information implies that the two solutions are linearly independent and span Nul A. So
dim Nul A = 2. By the Rank Theorem, dim Col A = 42 – 2 = 40. Since R40 is the only
subspace of R40 whose dimension is 40, Col A must be all of R40. This means that every
non-homogeneous equation Ax = b has a solution.

©Virtual University Of Pakistan 111

08- Rank VU

 -1 2 0 4 5 -3
 3 -7 2 0 1 4 
Example 9: Find the rank and nullity of the matrix A =  .
 2 -5 2 4 6 1
 
 4 -9 2 -4 -4 7 
Verify that values obtained verify the dimension theorem.
 -1 2 0 4 5 -3 
3 -7 2 0 1 4 
Solution 
2 -5 2 4 6 1 
 
4 -9 2 -4 -4 7 
1 -2 0 -4 -5 3
3 -7 2 0 1 4 
 (-1)R1
2 -5 2 4 6 1
 
4 -9 2 -4 -4 7
1 -2 0 -4 -5 3 
0 -1 2 12 16 - 5 
(-3)R1 + R2
 (-2)R1 + R3
0 -1 2 12 16 -5 
  (-4)R1 + R4
0 -1 2 12 16 -5 
1 -2 0 -4 -5 3 
0 1 -2 -12 -16 5 
 (-1)R2
0 -1 2 12 16 -5 
 
0 -1 2 12 16 -5 
1 -2 0 -4 -5 3
0 1 -2 -12 -16 5  R2 + R3

0 0 0 0 0 0  R2 + R4
 
0 0 0 0 0 0
1 0 -4 - 28 - 37 13 
0 1 -2 -12 -16 5 
 2R2 + R1
0 0 0 0 0 0 
 
0 0 0 0 0 0 
The reduced row-echelon form of A is
 1 0 -4 -28 -37 13 
 0 1 -2 -12 -16 5 
 (1)
 0 0 0 0 0 0 
 
 0 0 0 0 0 0 

©Virtual University Of Pakistan 112

08- Rank VU

The corresponding system of equations will be

x1 - 4x3 - 28x4 - 37x5 + 13x6 = 0
x2 - 2x3 - 12x4 - 16x5 + 5x6 = 0
or, on solving for the leading variables,
x1 = 4x3 - 28x4 + 37x5 - 13x6
(2)
x2 = 2x3 +12x4 + 16x5 - 5x6
it follows that the general solution of the system is
x1 = 4r + 28s + 37t - 13u
x2 = 2r + 12s + 16t - 5u
x3 = r
x4 = s
x5 = t
x6 = u
 x1  4   28  37  -13 
x   2  12  16   -5 
 2        
 x3  1  0   0   0 
or equivalently,   = r   + s   +t   +u   (3)
 x4  0   1   0   0 
 x5  0   0   1   0 
         
 x6  0   0   0   1 
The four vectors on the right side of (3) form a basis for the solution space, so
 -1 2 0 4 5 -3
 3 -7 2 0 1 4 
nullity (A) = 4. The matrix A =   has 6 columns,
 2 -5 2 4 6 1 
 
 4 -9 2 -4 -4 7 
so rank(A) + nullity(A) = 2 + 4 = 6 = n

©Virtual University Of Pakistan 113

08- Rank VU

Example 10: Find the rank and nullity of the matrix; then verify that the values obtained
 1 -3 2 2 1 
 0 3 6 0 -3
 
satisfy the dimension theorem A =  2 -3 -2 4 4 
 
 3 -6 0 6 5 
-2 9 2 -4 -5
Solution: Transforming Matrix to the Reduced Row Echelon Form:
1 -3 2 2 1 
0 3 6 0 - 3 

2 -3 -2 4 4 
 
3 -6 0 6 5 
 - 2 9 2 -4 - 5 
1 -3 2 2 1 
0 3 6 0 - 3  (-2)R1 + R3

0 3 -6 0 2  (-3)R1 + R4
 
0 3 -6 0 2  2R1 + R5
 0 3 6 0 - 3 
1 -3 2 2 1 
0 1 2 0 -1 

0 3 -6 0 2  (1/3)R2
 
0 3 -6 0 2 
 0 3 6 0 - 3 
1 -3 2 2 1 
0 1 2 0 -1  (-3) R2 + R3

0 0 -12 0 5  (-3) R2 + R4
 
0 0 -12 0 5  (-3)R2 + R5
 0 0 0 0 0 
1 -3 2 2 1 
0 1 2 0 -1 
 
0 0 1 0 - 5/12  (-1/12)R3
 
0 0 -12 0 5 
 0 0 0 0 0 

©Virtual University Of Pakistan 114

08- Rank VU

1 -3 2 2 1 
0 1 2 0 -1 
 
0 0 1 0 - 5/12  12R3 + R4
 
0 0 0 0 0 
 0 0 0 0 0 
1 -3 0 2 11/6 
0 1 0 0 -1/6 
 (-2 ) R3 + R2
0 0 1 0 - 5/12 
  (-2 ) R3 + R1
0 0 0 0 0 
 0 0 0 0 0 
1 0 0 2 4/3 
0 1 0 0 -1/6 

0 0 1 0 - 5/12  (3) R2 + R1 (1)
 
0 0 0 0 0 
 0 0 0 0 0 
Since there are three nonzero rows (or equivalently, three leading 1’s) the row space and
column space are both three dimensional so rank (A) = 3.
To find the nullity of A, we find the dimension of the solution space of the linear system
Ax = 0. The system can be solved by reducing the augmented matrix to reduced row
echelon form. The resulting matrix will be identical to (1), except with an additional last
column of zeros, and the corresponding system of equations will be
4
x1 + 0x2 + 0x3 + 2x4 + x5 = 0
3
1
0x1 + x2 + 0x3 + 0x4 - x5 = 0
6
5
0x1 + 0x2 + x3 + 0x4 - x5 = 0
12

The system has infinitely many solutions:

x 1 = -2 x 4 +(-4/3) x 5 x 2 = (1/6) x 5

x 3 = (5/12) x 5 x4 = s

x5 = t

The solution can be written in the vector form:

c 4 = (-2, 0, 0, 1, 0) c 5 = (-4/3, 1/6, 5/12,0,1)

©Virtual University Of Pakistan 115

08- Rank VU

Therefore the null space has a basis formed by the set

{(-2, 0, 0, 1, 0), (-4/3, 1/6, 5/12,0,1)}

The nullity of the matrix is 2. Now Rank (A) + nullity (A) = 3 + 2 =5 = n

Theorem 4: If A is an m x n, matrix, then

(a) rank (A) = the number of leading variables in the solution of Ax = 0
(b) nullity (A) = the number of parameters in the general solution of Ax = 0

Example 11: Find the number of parameters in the solution set of Ax = 0 if A is a 5 × 7

matrix of rank 3.
Solution: nullity (A) = n – rank (A) = 7-3 =4
Thus, there are four parameters.

Example: Find the number of parameters in the solution set of Ax = 0 if A is a 4 × 4

matrix of rank 0.
Solution nullity (A) = n – rank (A) = 4-0 =4
Thus, there are four parameters.

Theorem 5: If A is any matrix, then rank (A) = rank (AT)

Four fundamental matrix spaces:

If we consider a matrix A and its transpose AT together, then there are six
vectors spaces of interest:
Row space of A row space of AT
Column space of A column space of AT
Null space of A null space of AT
However, transposing a matrix converts row vectors into column vectors and column
vectors into row vectors, so that, except for a difference in notation, the row space of AT
is the same as the column space of A and the column space of AT is the same as row
space of of A.
This leaves four vector spaces of interest:
Row space of A column space of A
Null space of A null space of AT
These are known as the fundamental matrix spaces associated with A, if A is an m x n
matrix, then the row space of A and null space of A are subspaces of Rn and the column
space of A and the null space of AT are subspaces of Rm.

Suppose now that A is an m x n matrix of rank r, it follows from theorem (5) that AT is an
n x m matrix of rank r . Applying theorem (3) on A and AT yields
Nullity (A)=n-r, nullity (AT)=m-r
From which we deduce the following table relating the dimensions of the four
fundamental spaces of an m x n matrix A of rank r.

©Virtual University Of Pakistan 116

08- Rank VU

Fundamental space Dimension

Row space of A r
Column space of A r
Null space of A n-r
Null space of AT m-r

Example 12: If A is a 7 x 4 matrix, then the rank of A is at most 4 and, consequently,

the seven row vectors must be linearly dependent. If A is a 4 x 7 matrix, then again the
rank of A is at most 4 and, consequently, the seven column vectors must be linearly
dependent.

Rank and the Invertible Matrix Theorem: The various vector space concepts
associated with a matrix provide several more statements for the Invertible Matrix
Theorem. We list only the new statements here, but we reference them so they follow the
statements in the original Invertible Matrix Theorem in lecture 13.

Theorem 6: The Invertible Matrix Theorem (Continued)

Let A be an n x n matrix. Then the following statements are each equivalent to the
statement that A is an invertible matrix.
m. The columns of A form a basis of Rn.
n. Col A = Rn.
o. dim Col A = n
p. rank A = n
q. Nul A = {0}
r. dim Nul A = 0

Proof: Statement (m) is logically equivalent to statements (e) and (h) regarding linear
independence and spanning. The other statements above are linked into the theorem by
the following chain of almost trivial implications:
( g ) ⇒ ( n ) ⇒ (o ) ⇒ ( p ) ⇒ ( r ) ⇒ ( q ) ⇒ ( d )
Only the implication (p) ⇒ (r) bears comment. It follows from the Rank Theorem
because A is n × n . Statements (d) and (g) are already known to be equivalent, so the
chain is a circle of implications.

We have refrained from adding to the Invertible Matrix Theorem obvious statements
about the row space of A, because the row space is the column space of AT. Recall from
(1) of the Invertible Matrix Theorem that A is invertible if and only if AT is invertible.
Hence every statement in the Invertible Matrix Theorem can also be stated for AT.

Numerical Note:
Many algorithms discussed in these lectures are useful for understanding
concepts and making simple computations by hand. However, the algorithms are often
unsuitable for large-scale problems in real life.
Rank determination is a good example. It would seem easy to reduce a matrix to echelon
form and count the pivots. But unless exact arithmetic is performed on a matrix whose
entries are specified exactly, row operations can change the apparent rank of a matrix.

©Virtual University Of Pakistan 117

08- Rank VU

5 7 
For instance, if the value of x in the matrix   is not stored exactly as 7 in a
5 x 
computer, then the rank may be 1 or 2, depending on whether the computer treats x – 7 as
zero.
In practical applications, the effective rank of a matrix A is often determined from the
singular value decomposition of A.

Example 13: The matrices below are row equivalent

 2 -1 1 -6 8  1 -2 -4 3 -2 
 1 -2 -4 3 -2  0 3 9 -12 12 
A=   , B=  
-7 8 10 3 -10  0 0 0 0 0
   
 4 -5 -7 0 4  0 0 0 0 0
1. Find rank A and dim Nul A.
2. Find bases for Col A and Row A.
3. What is the next step to perform if one wants to find a basis for Nul A?
4. How many pivot columns are in a row echelon form of AT?
Solution:
1. A has two pivot columns, so rank A = 2. Since A has 5 columns altogether, dim
Nul A = 5 – 2 = 3.
2. The pivot columns of A are the first two columns. So a basis for Col A is
  2   -1 
    
 1 -2 
{ a1 , a2 }=    ,   
 -7   8  
   
  4   -5 
The nonzero rows of B form a basis for Row A, namely {(1, –2, –4, 3, –2), (0, 3,
9, –12, 12)}. In this particular example, it happens that any two rows of A form a
basis for the row space, because the row space is two-dimensional and none of the
rows of A is a multiple of another row. In general, the nonzero rows of an echelon
form of A should be used as a basis for Row A, not the rows of A itself.
3. For Nul A, the next step is to perform row operations on B to obtain the reduced
echelon form of A.
4. Rank AT = rank A, by the Rank Theorem, because Col AT = Row A. So AT has
two pivot positions.

Exercises:

In exercises 1 to 4, assume that the matrix A is row equivalent to B. Without calculations,

list rank A and dim Nul A. Then find bases for Col A, Row A, and Nul A.

 1 −4 9 −7  1 0 −1 5 

1. A =  0 −2 5 −6 
 −1 2 −4 1  , B =
 
 5 −6 10 7  0 0 0 0 

©Virtual University Of Pakistan 118

08- Rank VU

 1 −3 4 −1 9  1 −3 0 −7 
5
 −2 6 −6 −1 −10   2 −3 8 
2. A =   , B = 0 0
 −3 9 −6 −6 −3  0 0 0 0 5
   
 3 −9 4 9 0  0 0 0 0 0

 2 −3 6 2 5   2 −3 6 2 5
 −2 3 −3 −3 −4  0 0 3 −1 1 
3. A = ,B 
 4 −6 9 5 9  0 0 0 1 3
   
 −2 3 3 −4 1  0 0 0 0 0

1 1 −3 7 9 −9  1 1 −3 7 9 −9 
1 2 −4 10 13 −12  0 1 −1 3 4 −3
  
4. A =1 −1 −1 1 1 −3  , B 0 0 0 1 −1 −2 
   
1 −3 1 −5 −7 3  0 0 0 0 0 0
1 −2 0 0 −5 −4  0 0 0 0 0 0 

5. If a 3 x 8 matrix A has rank 3, find dim Nul A, dim Row A, and rank AT.

6. If a 6 x 3 matrix A has rank 3, find dim Nul A, dim Row A, and rank AT.

7. Suppose that a 4 x 7 matrix A has four pivot columns. Is Col A = R4? Is Nul A = R3?
Explain your answers.

8. Suppose that a 5 x 6 matrix A has four pivot columns. What is dim Nul A? Is Col A =
R4? Why or why not?

9. If the null space of a 5 x 6 matrix A is 4-dimensional, what is the dimension of the

column space of A?

10. If the null space of a 7 x 6 matrix A is 5-dimensional, what is the dimension of the
column space of A?

11. If the null space of an 8 x 5 matrix A is 2-dimensional, what is the dimension of the
row space of A?

12. If the null space of a 5 x 6 matrix A is 4-dimensional, what is the dimension of the
row space of A?

13. If A is a 7 x 5 matrix, what is the largest possible rank of A? If A is a 5 x 7 matrix,

what is the largest possible rank of A? Explain your answers.

©Virtual University Of Pakistan 119

08- Rank VU

14. If A is a 4 x 3 matrix, what is the largest possible dimension of the row space of A? If
A is a 3 x 4 matrix, what is the largest possible dimension of the row space of A? Explain.

15. If A is a 6 x 8 matrix, what is the smallest possible dimension of Nul A?

16. If A is a 6 x 4 matrix, what is the smallest possible dimension of Nul A?

©Virtual University Of Pakistan 120

09- Sol. Of Linear System VU

Lecture 09
Solution of Linear System of Equations and Matrix Inversion

Jacobi’s Method
This is an iterative method, where initial approximate solution to a given system of
equations is assumed and is improved towards the exact solution in an iterative way.

In general, when the coefficient matrix of the system of equations is a sparse matrix
(many elements are zero), iterative methods have definite advantage over direct methods
in respect of economy of computer memory
Such sparse matrices arise in computing the numerical solution of partial differential
equations
Let us consider
a11 x1 + a12 x2 +  + a1n xn = b1 
a21 x1 + a22 x2 +  + a2 n xn = b2 

    
an1 x1 + an 2 x2 +  + ann xn = b1 
In this method, we assume that the coefficient matrix [A] is strictly diagonally dominant,
that is, in each row of [A] the modulus of the diagonal element exceeds the sum of the
off-diagonal elements.
We also assume that the diagonal element do not vanish. If any diagonal element
vanishes, the equations can always be rearranged to satisfy this condition.
Now the above system of equations can be written as
b a a 
x1 = 1 − 12 x2 −  − 1n xn 
a11 a11 a11

b2 a21 a2 n 
x2 = − x1 −  − xn 
a22 a22 a22 
    

bn an1 an ( n −1) 
xn = − x1 −  − xn −1 
ann ann ann 
We shall take this solution vector ( x1 , x2 ,..., xn )T as a first approximation to the exact
solution of system. For convenience, let us denote the first approximation vector by
( x1(1) , x2(1) ,..., xn(1) ) got after taking as an initial starting vector.

Substituting this first approximation in the right-hand side of system, we obtain the
second approximation to the given system in the form

© Copyright Virtual University of Pakistan 121

 09- Sol. Of Linear System VU

b1 a12 (1) a 
x1(2) = − x2 −  − 1n xn(1) 
a11 a11 a11

b2 a21 (1) a2 n (1) 
x2 =
(2)
− x1 −  − xn 
a22 a22 a22 
    

bn an1 (1) an ( n −1) (1) 
xn =
(2)
− x1 −  − xn −1 
ann ann ann 
This second approximation is substituted into the right-hand side of Equations and obtain
the third approximation and so on.
This process is repeated and (r+1)th approximation is calculated
b a a 
x1( r +1) = 1 − 12 x2( r ) −  − 1n xn( r ) 
a11 a11 a11

( r +1) b2 a21 ( r ) a2 n ( r ) 
x2 = − x1 −  − xn 
a22 a22 a22 
    

( r +1) bn an1 ( r ) an ( n −1) ( r ) 
xn = − x1 −  − xn −1 
ann ann ann 
Briefly, we can rewrite these Equations as
n a
bi
xi( r +=
1)
− ∑ ij x (jr ) ,
aii j =1 aii
j ≠i

= r 1,=2,..., i 1, 2,..., n
It is also known as method of simultaneous displacements,
since no element of xi( r +1) is used in this iteration until every element is computed.

A sufficient condition for convergence of the iterative solution to the exact solution is
n
aii > ∑ aij , i=
1, 2,..., n When this condition (diagonal dominance) is true, Jacobi’s
j =1
j ≠1

method converges

Example
Find the solution to the following system of equations using Jacobi’s iterative method for
the first five iterations:
83 x + 11 y − 4 z = 95
7 x + 52 y + 13 z =104
3 x + 8 y + 29 z = 71
Solution

© Copyright Virtual University of Pakistan 122

09- Sol. Of Linear System VU

95 11 4 
x= − y+ z 
83 83 83

104 7 13 
y= − x − z
52 52 52 
71 3 8 
z = − x− y
29 29 29 
Taking the initial starting of solution vector as (0, 0, 0)T , from Eq. ,we have the first
approximation as

 x (1)   1.1446 
 (1)   
 y  =  2.0000 
 (1)   
 z   2.4483 
Now, using Eq. ,the second approximation is computed from the equations

1.1446 − 0.1325 y (1) + 0.0482 z (1) 

x (2) =

y (2) =
2.0 − 0.1346 x (1) − 0.25 z (1) 
(1) 
z =
(2)
2.4483 − 0.1035 x − 0.2759 y 
(1)

Making use of the last two equations we get the second approximation as
 x (2)   0.9976 
 (2)   
 y  =  1.2339 
 z (2)   1.7424 
   

Similar procedure yields the third, fourth and fifth approximations to the required
solution and they are tabulated as below;

Variables

Iteration number r x y z

1 1.1446 2.0000 2.4483

2 0.9976 1.2339 1.7424

3 1.0651 1.4301 2.0046

© Copyright Virtual University of Pakistan 123

09- Sol. Of Linear System VU

4 1.0517 1.3555 1.9435

5 1.0587 1.3726 1.9655

Example
Solve the system by jacobi’s iterative method
8x − 3 y + 2 z = 20
4 x + 11 y − z = 33
6 x + 3 y + 12 z =35
(Perform only four iterations)
Solution
Consider the given system as

© Copyright Virtual University of Pakistan 124

 09- Sol. Of Linear System VU

8x − 3 y + 2 z = 20
4 x + 11 y − z = 33
6 x + 3 y + 12 z =35
the system is diagonally do min ant
1
x= [ 20 + 3 y − 2 z ]
8
1
=
y [33 − 4 x + z ]
11
1
=
z [35 − 6 x − 3 y ]
12
we start with an initial aproximation x=
0 y=
0 z=
0 0
substituting these
first iteration
1
=
x1 [ 20 + 3(0) − 2(0)] = 2.5
8
1
=
y1 [33 − 4(0) + 0=] 3
11
1
z1= [35 − 6(0) − 3(0)]= 2.916667
12
Second iteration
1
x2 = [ 20 + 3(3) − 2(2.9166667) ] = 2.895833
8
1
y2 = [33 − 4(2.5) + 2.9166667 ] = 2.3560606
11
1
z2 = [35 − 6(2.5) − 3(3)] = 0.9166666
12

third iteration
1
x3 = [ 20 + 3(2.3560606) − 2(0.9166666)] = 3.1543561
8
1
y3 = [33 − 4(2.8958333) + 0.9166666] = 2.030303
11
1
z3 =− [35 6(2.8958333) − 3(2.3560606)] = 0.8797348
12

© Copyright Virtual University of Pakistan 125

 09- Sol. Of Linear System VU

fourth iteration
1
x4 = [ 20 + 3(2.030303) − 2(0.8797348)] = 3.0419299
8
1
y4 = [33 − 4(3.1543561) + 0.8797348] = 1.9329373
11
1
z4 = [35 − 6(3.1543561) − 3(2.030303)] = 0.8319128
12

Example
Solve the system by jacobi’s iterative method
3 x + 4 y + 15 z =54.8
x + 12 y + 3 z =39.66
10 x + y − 2 z = 7.74
(Perform only four iterations)
Solution

Consider the given system as

3 x + 4 y + 15 z =54.8
x + 12 y + 3 z = 39.66
10 x + y − 2 z = 7.74
the system is not diagonally do min ant we rearrange the system
10 x + y − 2 z = 7.74
x + 12 y + 3 z = 39.66
3 x + 4 y + 15 z =54.8
1
=x [7.74 − y + 2 z ]
10
1
=y [39.66 − x − 3z ]
12
1
=z [54.8 − 3x − 4 y ]
15

© Copyright Virtual University of Pakistan 126

 09- Sol. Of Linear System VU

we start with an initial aproximation x=

0 y=
0 z=
0 0
substituting these
first iteration
1
=x1 [7.74 − (0) + 2(0)
= ] 0.774
10
1
=y1 [39.66 − (0) − 3(0)
= ] 1.1383333
12
1
=z1 [54.8 − 3(0) − 4(0)=] 3.6533333
15
Second iteration
1
=x2 [7.74 − 1.1383333 + 2(3.6533333)] = 1.3908333
10
1
y=
2 [39.66 − 0.774 − 3(3.6533333)=] 2.3271667
12
1
z2 = [54.8 − 3(0.774) − 4(1.1383333) ] = 3.1949778
15

third iteration
1
x3 = [ 7.74 − 2.3271667 + 2(3.1949778) ] = 1.1802789
10
1
y3 = [39.66 − 1.3908333 − 3(3.1949778) ] =2.3903528
12
1
z3 = [54.8 − 3(1.3908333) − 4(2.3271667)] = 2.7545889
15
fourth iteration

1
x4 = [ 7.74 − 2.5179962 + 2(2.7798501) ] = 1.0781704
10
1
y4 = [39.66 − 1.1802789 − 3(2.7545889) ] =2.51779962
12
1
z4 = [54.8 − 3(1.1802789) − 4(2.3903528)] = 2.7798501
15

© Copyright Virtual University of Pakistan 127

10- Sol. Of Linear System VU

Lecture 10
Solution of Linear System of Equations and Matrix Inversion
Gauss–Seidel Iteration Method

It is another well-known iterative method for solving a system of linear equations of the
form
a11 x1 + a12 x2 +  + a1n xn = b1 
a21 x1 + a22 x2 +  + a2 n xn = b2 

    
an1 x1 + an 2 x2 +  + ann xn = bn 
In Jacobi’s method, the (r + 1)th approximation to the above system is given by
Equations
b a a 
x1( r +1) = 1 − 12 x2( r ) −  − 1n xn( r ) 
a11 a11 a11

( r +1) b2 a21 ( r ) a2 n ( r ) 
x2 = − x1 −  − xn 
a22 a22 a22 
    

b a a 
xn( r +1) = n − n1 x1( r ) −  − n ( n −1) xn( r−)1 
ann ann ann 
Here we can observe that no element of xi( r +1) replaces xi( r ) entirely for the next cycle of
computation.
In Gauss-Seidel method, the corresponding elements of xi( r +1) replaces those of
xi( r ) as soon as they become available.
Hence, it is called the method of successive displacements. For illustration consider
a11 x1 + a12 x2 +  + a1n xn = b1 
a21 x1 + a22 x2 +  + a2 n xn = b2 

    
an1 x1 + an 2 x2 +  + ann xn = bn 
In Gauss-Seidel iteration, the (r + 1)th approximation or iteration is computed from:
b a a 
x1( r +1) = 1 − 12 x2( r ) −  − 1n xn( r ) 
a11 a11 a11

b a a 
x2( r +1) = 2 − 21 x1( r +1) −  − 2 n xn( r ) 
a22 a22 a22 
    

( r +1) bn an1 ( r +1) an ( n −1) ( r +1) 
xn = − x1 −  − xn −1 
ann ann ann 
Thus, the general procedure can be written in the following compact form

© Copyright Virtual University of Pakistan 128

10- Sol. Of Linear System VU

bi i −1 aij ( r +1) n a
xi( r +1) = − ∑ x j − ∑ ij x (jr ) for all i = 1, 2,..., n and r = 1, 2,...
aii j = 1 aii j = i +1 aii

To describe system in the first equation, we substitute the r-th approximation into the
right-hand side and denote the result by x1( r +1) . In the second equation, we substitute
( x1( r +1) , x3( r ) ,..., xn( r ) ) and denote the result by x2( r +1)
In the third equation, we substitute ( x1( r +1) , x2( r +1) , x4( r ) ,..., xn( r ) ) and denote the result by
x3( r +1) , and so on. This process is continued till we arrive at the desired result. For
illustration, we consider the following example :

Note
The difference between jacobi’s method and gauss Seidel method is that in jacobi’s
method the approximation calculated are used in the next iteration for next
approximation but in Gauss-seidel method the new approximation calculated is
instantly replaced by the previous one.

Example
Find the solution of the following system of equations using Gauss-Seidel method and
perform the first five iterations:
4 x1 − x2 − x3 =
2
− x1 + 4 x2 − x4 =2
− x1 + 4 x3 − x4 =
1
− x2 − x3 + 4 x4 =
1

Solution
The given system of equations can be rewritten as

x1 =0.5 + 0.25 x2 + 0.25 x3 

x2 =0.5 + 0.25 x1 + 0.25 x4 

x3 =0.25 + 0.25 x1 + 0.25 x4 
x4 =0.25 + 0.25 x2 + 0.25 x3 
Taking x= 2 x=3 x=
4 0 on the right-hand side of the first equation of the system , we get
x1(1) = 0.5. Taking x= 3 x=
4 0 and the current value of x1 , we get from the 2nd equation
of the system
x2(1) = 0.5 + (0.25)(0.5) + 0 = 0.625

Further, we take x4 = 0 and the current value of x1 we obtain from the third equation of
the system

x3(1) =
0.25 + (0.25)(0.5) + 0
= 0.375

© Copyright Virtual University of Pakistan 129

10- Sol. Of Linear System VU

Now, using the current values of x2 and x3 the fourth equation of system gives
=
x4(1) 0.25 + (0.25)(0.625)
+(0.25)(0.375) =
0.5

The Gauss-Seidel iterations for the given set of equations can be written as
x1( r +1) =
0.5 + 0.25 x2( r ) + 0.25 x3( r )
x2( r +1) =
0.5 + 0.25 x1( r +1) + 0.25 x4( r )
x3( r +1) =
0.25 + 0.25 x1( r +1) + 0.25 x4( r )
x4( r +1) =
0.25 + 0.25 x2( r +1) + 0.25 x3( r +1)
Now, by Gauss-Seidel procedure, the 2nd and subsequent approximations can be
obtained and the sequence of the first five approximations are tabulated as below:

Variables

Iteration x1 x2 x3 x4
number r

1 0.5 0.625 0.375 0.5

2 0.75 0.8125 0.5625 0.59375

3 0.84375 0.85938 0.60938 0.61719

4 0.86719 0.87110 0.62110 0.62305

5 0.87305 0.87402 0.62402 0.62451

Example
Solve the system by Gauss-Seidel iterative method
8x − 3 y + 2 z = 20
4 x + 11 y − z = 33
6 x + 3 y + 12 z =35
(Perform only four iterations)
Solution
Consider the given system as

© Copyright Virtual University of Pakistan 130

10- Sol. Of Linear System VU

8x − 3 y + 2 z = 20
4 x + 11 y − z = 33
6 x + 3 y + 12 z =35
the system is diagonally do min ant
1
x= [ 20 + 3 y − 2 z ]
8
1
=
y [33 − 4 x + z ]
11
1
=
z [35 − 6 x − 3 y ]
12
we start with an initial aproximation x=
0 y=
0 z=
0 0
substituting these
first iteration
1
=
x1 [ 20 + 3(0) − 2(0)] =2.5
8
1
=
y1 [33 − 4(2.5) + 0=] 2.0909091
11
1
z1 = [35 − 6(2.5) − 3(2.0909091) ] = 1.1439394
12
Second iteration
1 1
x2= [ 20 + 3 y1 − z1 ]= [ 20 + 3(2.0909091) − 2(1.1439394)]= 2.9981061
8 8
1 1
y=2 [33 − 4 x2 + z1=] [33 − 4(2.9981061) + 1.1439394=] 2.0137741
11 11
1 1
z2= [35 − 6 x2 − 3 y2 =] [35 − 6(2.9981061) − 3(2.0137741)=] 0.9141701
12 12

third iteration
1
x3 = [ 20 + 3(2.0137741) − 2(0.9141701)] = 3.0266228
8
1
y3 = [33 − 4(3.0266228) + 0.9141701] = 1.9825163
11
1
z3 =− [35 6(3.0266228) − 3(1.9825163)] = 0.9077262
12

© Copyright Virtual University of Pakistan 131

 10- Sol. Of Linear System VU

fourth iteration
1
x4 = [ 20 + 3(1.9825163) − 2(0.9077262)] = 3.0165121
8
1
y4 = [33 − 4(3.0165121) + 0.9077262] = 1.9856071
11
1
z4 =− [35 6(3.0165121) − 3(1.9856071)] = 0.8319128
12

Example

Solve the system by suing Gauss-seidel iteration method

28 x + 4 y − z = 32
x + 3 y + 10 z = 24
2 x + 17 y + 4 z =35

Solution
28 x + 4 y − z =32
x + 3 y + 10 z =
24
2 x + 17 y + 4 z =
35

the given system is diagonally do min ant so we will make it diagonaaly do min ant by
iterchanaginhg the equations

28 x + 4 y − z = 32
2 x + 17 y + 4 z =35
x + 3 y + 10 z = 24

hence we can apply Gauss − Seidel method

from the above equations

1
=
x [32 − 4 y + z ]
28
1
=y [35 − 2 x − 4 z ]
17
1
=z [24 − x − 3 y ]
10

© Copyright Virtual University of Pakistan 132

 10- Sol. Of Linear System VU

First approximation
putting y = z = 0
1
=x1 = [32] 1.1428571
28
puting x = 1.1428571 , z = 0
1
y1 = [35 − 2(1.1428571) − 4(0)] = 1.9243697
17
=
putting =
x 1.1428571 , y 1.9243697
1
z1 =[24 − 1.1428571 − 3(1.9243697)] = 1.7084034
10
Second iteration
1
x2 = [32 − 4(1.9243697) + 1.7084034] =0.9289615
28
1
y2 =− [35 2(0.9289615) − 4(1.7084034)] = 1.5475567
17
1
z2 =[24 − 0.9289615 − 3(1.5475567)] = 1.8408368
10
third iteration
1
x3 = [32 − 4(1.5475567) + 1.8428368] =
0.9875932
28
1
y3 =− [35 2(0.9875932) − 4(1.8428368)] = 1.5090274
17
1
z3 =[24 − 0.9875932 − 3(1.5090274)] = 1.8485325
10
fourth iteration
1
x4 = [32 − 4(1.5090274) + 1.8485325] =
0.9933008
28
1
y4 =− [35 2(0.9933008) − 4(1.8428368)] = 1.5070158
17
1
z4 = [24 − 0.9933008 − 3(1.5070158)] = 1.8485652
10

Example
Using Gauss-Seidel iteration method, solve the system of the equation.
10 x − 2 y − z − w = 3
−2 x + 10 y − z − w = 15
− x − y + 10 z − 2 w =27
− x − y − 2 z + 10 w =−9

© Copyright Virtual University of Pakistan 133

 10- Sol. Of Linear System VU

(Perform only four iterations)

Solution
10 x − 2 y − z − w = 3
−2 x + 10 y − z − w = 15
− x − y + 10 z − 2 w =27
− x − y − 2 z + 10 w =−9
it is diagonally do min anat and we may write eqaution as
1
=
x [3 + 2 y + z + w]
10
1
=y [15 + 2 x + z + w]
10
1
=z [27 + x + y + 2 w]
10
1
w= [−9 + x + y + 2 z ]
10
first approximation
putting y =z =w =0 on RHS of (1) , we get
x1 = 0.3
1
y1 = [15 + 2(0.3)] =1.56
10
putting= x 0.3,= y 1.56 and= w 0
1
z=
1 [27 + 0.3 + 1.56]
= 2.886
10
putting x= 0.3, y= 1.56 and z= 2.886
1
w1 = [−9 + 0.3 + 1.56 + 2(2.886)] =−0.1368
10
sec ond iteration
1
x2 = [3 + 2(1.56) + 2.886 − 0.1368] =0.88692
10
1
y2 = [15 + 2(0.88692) + 2.886 − 0.1368] = 1.952304
10
1
z2 = [27 + 0.88692 + 1.952304 + 2(−0.1368)] =2.9565624
10
1
w2 = [−9 + 0.88692 + 1.952304 + 2(2.9565624)] =−0.0247651
10
third iteration
1
x3 = [3 + 2(1.952304) + 2.9565624 − 0.0.0247651] = 0.9836405
10

© Copyright Virtual University of Pakistan 134

10- Sol. Of Linear System VU

1
y3 = [15 + 2(0.9836405) + 2.9565624 − 0.0247651] = 1.9899087
10
1
=z3 [27 + 0.9836405 + 1.9899087 + 2(−0.0247651)]= 2.9924019
10
1
w3 = [−9 + 0.983405 + 1.9899087 + 2(2.9924019)] =−0.0041647
10
fourth iteration
1
x4 = [3 + 2(1.9899087) + 2.9924019 − 0.0041647] = 0.9968054
10
1
y4 = [15 + 2(0.9968054) + 2.9924019 − 0.0041647] = 1.9981848
10
1
=z4 [27 + 0.9968054 + 1.9981848 + 2(−0.0041647)]= 2.9986661
10
1
w4 = [−9 + 0.9968054 + 1.9981848 + 2(2.9986661)] =−0.0007677
10
Note
When to stop the iterative processes ,we stop the iterative process when we get the
required accuracy means if your are asked that find the accurate up to four places of
decimal then we will simply perform up to that iteration after which we will get the
required accuracy. If we calculate the root of the equation and its consecutive values are
1.895326125, 1.916366125, 1.919356325, 1.919326355, 1.919327145, 1.919327128
Here the accuracy up to seven places of decimal is achieved so if you are asked to acquire
the accuracy up to six places of decimal then we will stop here .
But in the solved examples only some iteration are carried out and accuracy is not
considered here.

© Copyright Virtual University of Pakistan 135

11- Sol. Of Linear System VU

Lecture 11
Solution of Linear System of Equations and Matrix Inversion

Relaxation Method
This is also an iterative method and is due to Southwell.To explain the details, consider
again the system of equations
a11 x1 + a12 x2 +  + a1n xn = b1 
a21 x1 + a22 x2 +  + a2 n xn = b2 

    
an1 x1 + an 2 x2 +  + ann xn = bn 
Let
X ( p ) = ( x1( p ) , x2( p ) ,..., xn( p ) )T
be the solution vector obtained iteratively after p-th iteration. If Ri( p ) denotes the
residual of the i-th equation of system given above , that is of ai1 x1 + ai 2 x2 +  + ain xn =
bi
defined by
Ri( p ) = bi − ai1 x1( p ) − ai 2 x2( p ) −  − ain xn( p )
we can improve the solution vector successively by reducing the largest residual to zero
at that iteration. This is the basic idea of relaxation method.

To achieve the fast convergence of the procedure, we take all terms to one side and then
reorder the equations so that the largest negative coefficients in the equations appear on
the diagonal.

Now, if at any iteration, Ri is the largest residual in magnitude, then we give an

increment to xi ; aii being the coefficient of xi

Ri
dxi =
aii
In other words, we change xi . to ( xi + dxi ) to relax Ri that is to reduce Ri to zero.
Example

Solve the system of equations

6 x1 − 3 x2 + x3 =
11
2 x1 + x2 − 8 x3 =−15
x1 − 7 x2 + x3 =10
by the relaxation method, starting with the vector (0, 0, 0).

Solution

© Copyright Virtual University of Pakistan 136

11- Sol. Of Linear System VU

At first, we transfer all the terms to the right-hand side and reorder the equations, so that
the largest coefficients in the equations appear on the diagonal.
Thus, we get
0 = 11 − 6 x1 + 3 x2 − x3 

0 = 10 − x1 + 7 x2 − x3 
0= −15 − 2 x1 − x2 + 8 x3 

after interchanging the 2nd and 3rd equations.

Starting with the initial solution vector (0, 0, 0), that is taking x=
1 x=
2 x=
3 0,

we find the residuals R1 = 11, R2 = 10, R3 = −15

of which the largest residual in magnitude is R3, i.e. the 3rd equation has more error and
needs immediate attention for improvement.

Thus, we introduce a change, dx3in x3 which is obtained from the formula

R 15
dx3 = − 3 == 1.875
a33 8
Similarly, we find the new residuals of large magnitude and relax it to zero, and so on.
We shall continue this process, until all the residuals are zero or very small.
Iteration Residuals Maximum Difference Variables

number R1 R2 R3 Ri dxi x1 x2 x3

0 11 10 -15 -15 1.875 0 0 0

1 9.125 8.125 0 9.125 1.5288 0 0 1.875

2 0.0478 6.5962 - 6.5962 -0.9423 1.5288 0 1.875

3.0576

© Copyright Virtual University of Pakistan 137

11- Sol. Of Linear System VU

Iteration Residuals Maximum Difference Variables

number R1 R2 R3 Ri dxi x1 x2 x3

0 11 10 -15 -15 15/8 0 0 0

=1.875

1 9.125 8.125 0 9.125 -9.125/(-6) 0 0 1.875

=1.5288

2 0.0478 6.5962 -3.0576 6.5962 -6.5962/7 1.5288 0 1.875

=-0.9423

3 -2.8747 0.0001 -2.1153 -2.8747 2.8747/(-6) 1.0497 -0.9423 1.875

=-0.4791

4 -0.0031 0.4792 -1.1571 -1.1571 1.1571/8 1.0497 -0.9423 1.875

=0.1446

Iteration Residuals Maximum Difference Variables

number R1 R2 R3 Ri dxi x1 x2 x3

5 -0.1447 0.3346 0.0003 0.3346 -.3346/7 1.0497 -0.9423 2.0196

=-0.0478

6 0.2881 0.0000 0.0475 0.2881 -.2881/(-6) 1.0497 -0.9901 2.0196

=0.0480

7 -0.0001 0.048 0.1435 0.1435 =-0.0179 1.0017 -0.9901 2.0196

8 0.0178 0.0659 0.0003 - - 1.0017 -0.9901 2.0017

© Copyright Virtual University of Pakistan 138

11- Sol. Of Linear System VU

At this stage, we observe that all the residuals R1, R2 and R3 are small enough and
therefore we may take the corresponding values of xi at this iteration as the solution.
Hence, the numerical solution is given by
x1 =1.0017, x2 = −0.9901, x3 = 2.0017,
The exact solution is
x1 =1.0, x2 =−1.0, x3 = 2.0

Example

Solve by relaxation method, the equation

10 x − 2 y − 2 z =6
− x − 10 y − 2 z =7
− x − y + 10 z = 8

Solution

The residual r1 , r2 , r3 are given by

r1 =6 − 10 x + 2 y + 2 z
r2 = 7 + x − 10 y + 2 z
r3 = 8 + x + y − 10 z
The operation table is as follows

x y z r1 r2 r3
1 0 0 -10 1 1 L1
0 1 0 2 -10 1 L2
0 0 1 2 2 -10 L3

The relaxation table is as follows

x y z r1 r2 r3
0 0 0 6 7 8 L4
0 0 1 8 9 -2 L5=L4+L3
0 1 0 10 -1 -1 L6=L5+L2
1 0 0 0 0 0 L7=L6+L1

Explanation

© Copyright Virtual University of Pakistan 139

11- Sol. Of Linear System VU

(1) In L4 ,the largest residual is 8.to reduce it, To reduce it ,we give an increment of
8 8
= = 0.8 ≅ 1
c3 10
the resulting residulas are obtained by
L4 + (1) L3 , i.e line L5

(2) In line L5 the largest residual is 9

9 9
Increment= = = 0.9 ≅ 1
b2 10
The resulting residuals (= L6 ) = L5 + 1.L2
(3) In line L6 ,the largest residual is 10
10 10
Increment = = ≅1
a1 10
The resulting residuals (= L6 ) = L5 + 1.L2
Exact solution is arrived and it is x=1,y=1,z=1

Example
Solve the system by relaxation method, the equations

9x − y + 2z = 7
x + 10 y − 2 z = 15
2 x − 2 y − 13 z = −17
Solution
The residuals r1 , r2 , r3 are given by
9x − y + 2z = 9
x + 10 y − 2 z = 15
2 x − 2 y − 13 z =−17
here
r1 = 9 − 9 x + y − 2 z
r2 = 15 − x − 10 y + 2 z
r3 =−17 − 2 x + 2 y + 13 z

Operation table

x y z r1 r2 r3
1 0 0 -9 -1 -2
0 1 0 1 -10 2
0 0 1 -2 2 13

Relaxation table is

© Copyright Virtual University of Pakistan 140

11- Sol. Of Linear System VU

x y z r1 r2 r3
0 0 0 9 15 -17
0 0 1 7 17 -4
0 1 0 8 7 -2
0.89 0 0 -0.01 6.11 -3.78
0 0.61 0 0.6 0.01 -2.56
0 0 0.19 0.22 0.39 -0.09
0 0.039 0 0.259 0 -0.012
0.028 0 0 0.007 -0.028 -0.068
0 0 0.00523 -0.00346 -1.01754 -0.00001

Then x=0.89+0.028=0.918;y=1+0.61+0.039=1.694
And z=1+0.19+0.00523=1.19523
Now substituting the values of x,y,z in (1) ,we get
r1=9-9(0.918)+1.649-2(1.19523)=-0.00346
r2=15-0.918-10(1.649)+2(1.19523)=-0.1754
r3=-17-2(0.918) +2(1.649) +13(1.19523) =-0.00001
Which is agreement with the final residuals.

© Copyright Virtual University of Pakistan 141

12-Vector and Matrix Norms VU

Lecture 12

Norms of Vectors and Matrices, Matrix Norms and Distances

PPT’s slides are available in VULMS/downloads

© Copyright Virtual University of Pakistan 142

13-Error bounds and iterative refinment VU

Lecture 13

Error Bounds and Iterative Refinement

PPT’s slides are available in VULMS/downloads

© Copyright Virtual University of Pakistan 143

14- Eigenvalues and Eigenvectors VU

Lecture 14

Eigenvalues and Eigenvectors

In this lecture we will discuss linear equations of the form Ax = x and, more generally,
equations of the form Ax = λ x , where λ is a scalar. Such equations arise in a wide
variety of important applications and will be a recurring theme in the rest of this course.

Fixed Points:
A fixed point of an n × n matrix A is a vector x in Rn such that Ax = x. Every square
matrix A has at least one fixed point, namely x = 0. We call this the trivial fixed point of
A.
The general procedure for finding the fixed points of a matrix A is to rewrite the equation
Ax = x as Ax = Ix or, alternatively, as
(I – A)x = 0 (1)
Since this can be viewed as a homogeneous linear system of n equations in n unknowns
with coefficient matrix I – A, we see that the set of fixed points of an n × n matrix is a
subspace of Rn that can be obtained by solving (1).

The following theorem will be useful for ascertaining the nontrivial fixed points of a
matrix.

Theorem 1:
If A is an n x n matrix, then the following statements are equivalent.
(a) A has nontrivial fixed points.
(b) I – A is singular.
(c) det(I – A) = 0.

Example 1:
In each part, determine whether the matrix has nontrivial fixed points; and, if so, graph
the subspace of fixed points in an xy-coordinate system.
3 6  0 2 
= (a ) A =  (b) A  
1 2  0 1 

Solution:
(a) The matrix has only the trivial fixed point since.

 1 0   3 6   −2 −6 
(I −=A)  − =   
 0 1   1 2   −1 −1 
 −2 −6 
det( I − A) = det  = (−1)(−2) − (−1)(−6) =
− 4≠ 0
 −1 −1 

(b) The matrix has nontrivial fixed points since

©Virtual University Of Pakistan 144

14- Eigenvalues and Eigenvectors VU

 1 0   0 2   1 −2 
( I −=A)  − =   
0 1 0 1 0 0 
 1 −2 
=
det( I − A) det =  0
0 0 

The fixed points x =(x, y) are the solutions of the linear system (I – A)x=0, which we can
express in component form as
1 -2   x  0 
0 0   y  = 0 
    
A general solution of this system is
x = 2t, y = t (2)

which are parametric equations of the line y = 12 x . It follows from the corresponding
vector form of this line that the fixed points are
 x   2t   2 
= x =  =  t  (3)
 y   t  1 
0 2   2t   2t 
As a check, =Ax   =  =  x
0 1   t   t 
so every vector of form (3) is a fixed point of A.

y = 12 x
(2, 1) •

Figure 1

Eigenvalues and Eigenvectors:

In a fixed point problem one looks for nonzero vectors that satisfy the equation Ax = x.
One might also consider whether there are nonzero vectors that satisfy such equations as
Ax = 2x, Ax = –3x, Ax = 2 x
or, more generally, equations of the form Ax = λ x in which λ is a scalar.

Definition: If A is an n x n matrix, then a scalar λ is called an eigenvalue of A if there

is a nonzero vector x such that Ax = λ x . If λ is an eigenvalue of A, then every nonzero
vector x such that Ax = λ x is called an eigenvector of A corresponding to λ .

©Virtual University Of Pakistan 145

 14- Eigenvalues and Eigenvectors VU

Example 2:
1 6  6 3
=
Let A =  , u =  and v -2  . Are u and v eigenvectors of A?
5 2  -5  
Solution:
1 6   6  -24  6
=
Au  =
    = - 4=-5 - 4u
5 2  -5  20   
1 6   3   -9  3
=
Av   =    ≠λ 
5 2  -2  11 -2 
Thus u is an eigenvector corresponding to an eigenvalue – 4, but v is not an eigenvector
of A, because Av is not a multiple of v.

Example 3:
1 6 
Show that 7 is an eigenvalue of A =   , find the corresponding eigenvectors.
5 2 
Solution:
The scalar 7 is an eigenvalue of A if and only if the equation
Ax = 7x (A)
has a nontrivial solution. But (A) is equivalent to Ax – 7x = 0, or
(A – 7I) x = 0 (B)
To solve this homogeneous equation, form the matrix
1 6  7 0  -6 6 
= A - 7 I = -   
5 2   0 7   5 -5
The columns of A – 7I are obviously linearly dependent, so (B) has nontrivial solutions.
Thus 7 is an eigenvalue of A. To find the corresponding eigenvectors, use row operations:
 −6 6 0 
 5 −5 0 
 
1 −1 0
~ (−1R1 − R2 )
5 −5 0 
1 −1 0
~ ( R2 − 5 R1 )
0 0 0 
1
The general solution has the form x 2   . Each vector of this form with x2 ≠ 0 is an
1
eigenvector corresponding to λ = 7.

The equivalence of equations (A) and (B) obviously holds for any λ in place of λ = 7.
Thus λ is an eigenvalue of A if and only if the equation
(A - λ I)x = 0 (C)
has a nontrivial solution.

©Virtual University Of Pakistan 146

14- Eigenvalues and Eigenvectors VU

Eigen space:
The set of all solutions of (A - λ I)x = 0 is just the null space of the matrix A - λ I. So
this set is a subspace of Rn and is called the eigenspace of A corresponding to λ . The
eigenspace consists of the zero vector and all the eigenvectors corresponding to λ .
Example 3 shows that for matrix A in Example 2, the eigenspace corresponding to λ = 7
consists of all multiples of (1, 1), which is the line through (1, 1) and the origin. From
Example 2, one can check that the eigenspace corresponding to λ = -4 is the line through
(6, -5). These eigenspaces are shown in Fig. 1, along with eigenvectors (1, 1) and (3/2, -
5/4) and the geometric action of the transformation x → Ax on each eigenspace.

 4 -1 6 
Example 4: Let A =  2 1 6  .
 2 -1 8 
Find a basis for the corresponding eigenspace where eigen value of matrix is 2.

©Virtual University Of Pakistan 147

14- Eigenvalues and Eigenvectors VU

 4 -1 6   2 0 0   2 -1 6 
=
Solution: Form A - 2 I =2 1 6  -  0 2 0   2 -1 6  and row reduce the
 
 2 -1 8   0 0 2   2 -1 6 
2 -1 6 0  augmented matrix for (A – 2I) x = 0:
2 -1 6 0 

 2 -1 6 0 
2 -1 6 0 
~  0 0 0 0  R2 − R1
 2 −1 6 0 

 2 -1 6 0 
~  0 0 0 0  R3 − R1
 0 0 0 0 
At this point we are confident that 2 is indeed an eigenvalue of A because the equation
(A – 2I) x = 0 has free variables. The general solution is
2 x1 − x2 + 6 x3 =
0........(a )
Let=
x2 t ,=
x3 s then
2 x1 = t − 6 s
=x1 ( 1 2 ) t − 3s
then
 x1  t / 2 − 3s  t / 2   −3s  1/ 2   −3
 x  = t         
 2   = t  + 0  = t 1  + s 0 
 x3   s  0   s  0  1 

By back substitution the general solution is

 x1  1 2  -3
=     
 x2  x2  1  + x3  0  , x2 and x3 free
 x3   0   1 
The eigenspace, shown in Fig. 2, is a two – dimensional subspace of R3. A basis is
  1   -3 
    
  2  ,  0   is a basis.
0  1  
    

©Virtual University Of Pakistan 148

14- Eigenvalues and Eigenvectors VU

The most direct way of finding the eigenvalues of an n × n matrix A is to rewrite the
equation Ax = λ x as Ax = λ Ix , or equivalently, as
(λ I - A) x = 0 (4)
and then try to determine those values of λ , if any, for which this system has nontrivial
solutions. Since (4) have nontrivial solutions if and only if the coefficient matrix λ I - A is
singular, we see that the eigenvalues of A are the solutions of the equation
det(λ I - A) = 0 (5)
Equation (5) is known as characteristic equation. Also, if λ is an eigenvalue of A, then
equation (4) has a nonzero solution space, which we call the eigenspace of A
corresponding to λ . It is the nonzero vectors in the eigenspace of A corresponding to λ
that are the eigenvectors of A corresponding to λ .
The above discussion is summarized by the following theorem.

Theorem: If A is an n × n matrix and λ is a scalar, then the following statements are

equivalent.
(i) λ is an eigenvalue of A.
(ii) λ is a solution of the equation det(λ I - A) = 0 .
(iii) The linear system (λ I - A) x = 0 has nontrivial solutions.

Eigenvalues of Triangular Matrices: If A is an n × n triangular matrix with diagonal

entries a11, a22, …, ann, then λ I - A is a triangular matrix with diagonal entries
λ - a11 , λ - a22 , , λ - ann . Thus, the characteristic polynomial of A is
det(λ I - A) = (λ - a11 )(λ - a22 )  (λ - ann )
which implies that the eigenvalues of A are
= λ1 a= 11 , λ2 =
a22 ,  , λn ann
Thus, we have the following theorem.

©Virtual University Of Pakistan 149

 14- Eigenvalues and Eigenvectors VU

Theorem: If A is a triangular matrix (upper triangular, lower triangular, or diagonal)

then the eigenvalues of A are the entries on the main diagonal of A.

Example 5: (Eigenvalues of Triangular Matrices)

 12 0 0 0
-1 - 2 0 0 
By inspection, the characteristic polynomial of the matrix A =  3
is
 7 85 6 0
4 
 9 -4 3 6
p (λ ) (λ - 12 )(λ + 23 )(λ - 6) 2 . So the distinct eigenvalues of A are λ = 12 ,
= λ = − 23 , and
λ = 6.

Eigenvalues of Powers of a Matrix: Once the eigenvalues and eigenvectors of a matrix

A are found, it is a simple matter to find the eigenvalues and eigenvectors of any positive
integer power of A. For example, if λ is an eigenvalue of A and x is a corresponding
eigenvector, then= A2 x A= ( Ax) A= (λ x) λ= ( Ax) λ=(λ x) λ 2 x , which shows that λ 2 is
an eigenvalue of A2 and x is a corresponding eigenvector. In general we have the
following result.

Theorem: If λ is an eigenvalue of a matrix A and x is a corresponding eigenvector, and

if k is any positive integer, then λ k is an eigenvalue of Ak and x is a corresponding
eigenvector.
Some problems that use this theorem are given in the exercises.

A Unifying Theorem: Since λ is an eigenvalue of a square matrix A if and only if

there is a nonzero vector x such that Ax = λ x, it follows that λ = 0 is an eigenvalue of A
if and only if there is a nonzero vector x such that Ax = 0. However, this is true if and
only if det(A) = 0, so we list the following

Theorem: If A is an n × n matrix, then the following statements are equivalent.

(a) The reduced row echelon form of A is I n .
(b) A is expressible as a product of elementary matrices.
(c) A is invertible.
(d) Ax = 0 has only the trivial solution.
(e) Ax = b is consistent for every vector b in Rn.
(f) Ax = b has exactly one solution for every vector b in Rn
(g) The column vectors of A are linearly independent.
(h) The row vectors of A are linearly independent.
(i) det(A) ≠ 0.
(j) λ = 0 is not an eigenvalue of A.

©Virtual University Of Pakistan 150

 14- Eigenvalues and Eigenvectors VU

Example 6:
 6 -3 1 
(1) Is 5 an eigenvalue of A =  3 0 5  ?
 2 2 6 
(2) If x is an eigenvector for A corresponding to λ , what is A3x?

Solution:
(1) The number 5 is an eigenvalue of A if and only if the equation (A- λ I) x = 0 has a
nontrivial solution. Form
 6 -3 1  5 0 0  1 -3 1
= A - 5 I = 3 0 5  - 0 5 0   3 -5 5
 
 2 2 6  0 0 5   2 2 1
and row reduce the augmented matrix:
1 -3 1 0 
 3 -5 5 0 
 
 2 2 1 0 
1 -3 1 0 
~  0 4 2 0  R2 − 3R1
 2 2 1 0 
1 -3 1 0 
~ 0 4 2 0  R3 − 2 R1
0 8 -1 0 
1 -3 1 0 
~ 0 4 2 0  R3 − 2 R2
0 0 -5 0 
At this point it is clear that the homogeneous system has no free variables. Thus A – 5I is
an invertible matrix, which means that 5 is not an eigenvalue of A.

(2). If x is an eigenvector for A corresponding to λ , then Ax = λ x and so

=
A 2
x A(λ =x ) λ= Ax λ 2 x
Again = A3 x A( A =2
x ) A(λ=2
x ) λ=
2
Ax λ 3 x. The general pattern, Ak x = λ k x , is
proved by induction.

Exercises:

3 2 
1. Is λ = 2 an eigenvalue of  ?
3 8 

©Virtual University Of Pakistan 151

 14- Eigenvalues and Eigenvectors VU

 −1 + 2  2 1
2. Is   an eigenvector of 1 4  ? If so, find the eigenvalue.
 1   

4  3 7 9
3. Is  −3 an eigenvector of
 
   −4 −5 1  ? If so, find the eigenvalue.
1   2 4 4 

1 3 6 7 
4. Is  −2  an eigenvector of
3 3 7  ? If so, find the eigenvalue.
   
1  5 6 5 

 3 0 −1
 
5. Is λ = 4 an eigenvalue of 2 3 1 ? If so, find one corresponding eigenvector.
 
 −3 4 5 

1 2 2 
 
6. Is λ = 3 an eigenvalue of 3 −2 1 ? If so, find one corresponding eigenvector.
 
0 1 1 

In exercises 7 to 12, find a basis for the eigenspace corresponding to each listed
eigenvalue.

 4 −2  7 4
=7. A =
−  , λ 10 =8. A =
− −  , λ 1.5
 3 9   3 1

 4 0 1  1 0 −1
 
9. A = −2 1 0 , λ =1, 2,3
 
10. A = 1 −3 0 , λ = −2
   
 −2 0 1   4 −13 1 

3 0 2 0
4 2 3 1
 −1 1 −3 , λ = 3 3 1 0 
11. A = 12. A =  ,λ=4
  0 1 1 0
 2 4 9   
0 0 0 4

Find the eigenvalues of the matrices in Exercises 13 and 14.

©Virtual University Of Pakistan 152

14- Eigenvalues and Eigenvectors VU

0 0 0  4 0 0 

13. 0 2 5
 
14. 0 0 0

   
0 0 −1 1 0 −3

1 2 3
 
15. For A = 1 2 3 , find one eigenvalue, with no calculation. Justify your answer.
 
1 2 3

16. Without calculation, find one eigenvalue and two linearly independent vectors of
5 5 5 
A = 5 5 5 . Justify your answer.
5 5 5

©Virtual University Of Pakistan 153

15-The Characteristic Equation VU

Lecture 15

The Characteristic Equation

The Characteristic equation contains useful information about the eigenvalues of a square
matrix A. It is defined as
det( A − λ I ) =
0,
Where λ is the eigenvalue and I is the identity matrix. We will solve the Characteristic
equation (also called the characteristic polynomial) to work out the eigenvalues of the
given square matrix A.

2 3 
Example 1: Find the eigenvalues of A =  .
 3 -6 
Solution: In order to find the eigenvalues of the given matrix, we must solve the matrix
equation
(A- λI)x = 0
for the scalar λ such that it has a nontrivial solution (since the matrix is non singular).
By the Invertible Matrix Theorem, this problem is equivalent to finding all λ such that
the matrix A - λ I is not invertible, where
 2 3  λ 0   2 - λ 3 
= A - λ I =  -   -6 - λ 
.
 3 -6   0 λ   3

By definition, this matrix A - λ I fails to be invertible precisely when its determinant is

zero. Thus, the eigenvalues of A are the solutions of the equation
2 - λ 3 
= det( A - λ I ) det=  3 -6 - λ 
0.

a b 
Recall that det   = ad - bc
c d 
So det( A - λ I ) = (2 - λ )(-6 - λ ) - (3)(3)
= -12 + 6λ - 2λ + λ 2 - 9
= λ 2 + 4λ - 21
λ 2 + 4λ - 21 = 0,
(λ - 3)(λ + 7) = 0,
so the eigenvalues of A are 3 and –7.

1 5 0 
Example 2: Compute det A for A =  2 4 -1
 0 -2 0 

__________________________________________________________________________________
@Virtual University Of Pakistan 154
15-The Characteristic Equation VU

Solution:
Firstly, we will reduce the given matrix in echelon form by applying elementary row
operations
by R2 − 2 R1
1 5 0 
A = 0 -6 -1 ,
0 -2 0 
by R2 ↔ R3
1 5 0 
 0 -2 0  ,
0 -6 -1
by R3 − 3R2
1 5 0 
 0 -2 0  ,
0 0 -1
which is an upper triangular matrix. Therefore,
det A = (1)(−2)(−1)
= 2.

Theorem 1: Properties of Determinants

Let A and B be two matrices of order n then

(a) A is invertible if and only if det A ≠ 0.

(b) det AB = (det A)(det B).
(c) det AT = det A.
(d) If A is triangular, then det A is the product of the entries on the main diagonal of A.
(e) A row replacement operation on A does not change the determinant.
(f) A row interchange changes the sign of the determinant.
(g) A row scaling also scales the determinant by the same scalar factor.

Note: These Properties will be helpful in using the characteristic equation to find
eigenvalues of a matrix A.

Example 3: (a) Find the eigenvalues and corresponding eigenvectors of the matrix
1 3 
A= 
4 2
(b) Graph the eigenspaces of A in an xy-coordinate system.

Solution: (a) The eigenvalues will be worked out by solving the characteristic equation
of A. Since

__________________________________________________________________________________
@Virtual University Of Pakistan 155
15-The Characteristic Equation VU

1 0  1 3  λ - 1 -3 
= λ I - A λ= 0 1  -  4 2   -4 λ - 2  .
     
The characteristic equation det(λ I - A) = 0 becomes
λ - 1 -3
= 0.
-4 λ - 2
Expanding and simplifying the determinant, it yields
λ 2 - 3λ - 10 = 0,
or
(λ + 2)(λ - 5) = 0. (1)
Thus, the eigenvalues of A are λ = −2 and λ = 5 .

Now, to work out the eigenspaces corresponding to these eigenvalues, we will solve the
system
λ - 1 -3   x  0 
 -4 λ - 2   y  = 0  (2)
    
for λ = −2 and λ = 5 . Here are the computations for the two cases.

(i) Case λ = -2
In this case Eq. (2) becomes
 −3 −3  x  0 
 −4 −4   y  = 0  ,
    
which can be written as
−3 x − 3 y = 0,
−4 x − 4 y =0 ⇒ x =− y.
In parametric form,
x = – t, y = t . (3)
Thus, the eigenvectors corresponding to λ = −2 are the nonzero vectors of the form
 x  -t  -1
= x =  =  t  . (4)
 y  t   1 
It can be verified as
1 3  -t   2t  -t 
= =
 4 2   t  -2t  -2= t  -2 x
      
Thus,
Ax = λ x
(ii) Case λ = 5
In this case Eq. (2) becomes
 4 -3  x  0 
-4 3   y  = 0  ,
    
which can be written as

__________________________________________________________________________________
@Virtual University Of Pakistan 156
15-The Characteristic Equation VU

4x − 3y =
0
3
−4 x + 3 y = 0 ⇒ x = y.
4
In parametric form,
3
=x =t , y t. (5)
4
Thus, the eigenvectors corresponding to λ = 5 are the nonzero vectors of the form
 x   34 t   34 
= x =   = t 1  . (6)
 y  t   
It can be verified as
1 3   34 t   154 t   34 t 
=Ax  =
    = 5 =  5 x.
 4 2   t   5t  t 

(b) The eigenspaces corresponding to λ = −2 and λ = 5 can be sketched from the

parametric equations (3) and (5) as shown in figure 1(a).

y
( λ = −2 ) ( λ = 5)
y = –x y = 43 x

Figure 1(a)

It can also be drawn using the vector equations (4) and (6) as shown in Figure 1(b). When
an eigenvector x in the eigenspace for λ = 5 is multiplied by A, the resulting vector has
the same direction as x but the length is increased by a factor of 5 and when an
eigenvector x in the eigenspace for λ = −2 is multiplied by A, the resulting vector is
oppositely directed to x and the length is increased by a factor of 2. In both cases,
multiplying an eigenvector by A produces a vector in the same eigenspace.

__________________________________________________________________________________
@Virtual University Of Pakistan 157
15-The Characteristic Equation VU

y
( λ = −2 ) ( λ = 5)
5x

x
x x

–2x

Figure 1(b)

Eigenvalues of an n×n matrix:

Eigen values of an n × n matrix can be found in
the similar fashion. However, for the higher values of n, it is more convenient to work
them out using various available mathematical software. Here is an example for a 3 × 3
matrix.

 0 -1 0 
Example 4: Find the eigen values of the matrix A =  0 0 1 
- 4 -17 8 
Solution:
λ 0 0  0 −1 0
det(λ I - A) det 0
= λ 0  − 0 0 1
0 0 λ   −4 − 17 8 
λ 1 0
= 0 λ -1 (7)
4 17 λ -8
= λ 3 - 8 λ 2 + 17 λ - 4,
which yields the characteristic equation
λ 3 - 8 λ 2 + 17 λ - 4 =
0 (8)

To solve this equation, firstly, we will look for integer solutions. This can be done by
using the fact that if a polynomial equation has integer coefficients, then its integer
solutions, if any, must be divisors of the constant term of the given polynomial. Thus, the
only possible integer solutions of Eq.(8) are the divisors of –4, namely ± 1, ±2, and ± 4 .
Substituting these values successively into Eq. (8) yields that λ =4 is an integer solution.
This implies that λ – 4 is a factor of Eq.(7), Thus, dividing the polynomial by λ – 4 and
rewriting Eq.(8), we get
(λ - 4 )(λ 2 - 4 λ + 1) =
0.

__________________________________________________________________________________
@Virtual University Of Pakistan 158
15-The Characteristic Equation VU

Now, the remaining solutions of the characteristic equation satisfy the quadratic equation
λ 2 - 4λ + 1 =0.
Solving the above equation by the quadratic formula, we get the eigenvalues of A as
λ= 4, λ= 2 + 3, λ= 2− 3
5 -2 6 -1
0 3 -8 0 
Example 5 Find the characteristic equation of A =  
0 0 5 4 
 
0 0 0 1 

Solution: Clearly, the given matrix is an upper triangular matrix. Forming A - λ I , we get
5 - λ -2 6 -1 
 0 3 - λ -8 0 
det( A - λ I ) = det 
 0 0 5-λ 4 
 
 0 0 0 1- λ 
Now using the fact that determinant of a triangular matrix is equal to product of its
diagonal elements, the characteristic equation becomes
(5 - λ ) 2 (3 - λ )(1- λ ) = 0.
Expanding the product, we can also write it as
λ 4 -14λ 3 + 68λ 2 -130λ + 75 = 0.
Here, the eigenvalue 5 is said to have multiplicity 2 because ( λ - 5) occurs two times as a
factor of the characteristic polynomial. In general, the (algebraic) multiplicity of an
eigenvalue λ is its multiplicity as a root of the characteristic equation.

Note:
From the above mentioned examples, it can be easily observed that if A is an n × n matrix,
then det (A – λ I) is a polynomial of degree n called the characteristic polynomial of A.

Example 6 The characteristic polynomial of a 6 × 6 matrix is λ 6 - 4λ 5 -12λ 4 . Find the

eigenvalues and their multiplicities.
Solution:
In order to find the eigenvalues, we will factorize the polynomial as
λ 6 - 4λ 5 -12λ 4
= λ 4 (λ 2 - 4λ -12)
= λ 4 (λ - 6)(λ + 2)
The eigenvalues are 0 (multiplicity 4), 6 (multiplicity 1) and – 2 (multiplicity 1).We
could also list the eigenvalues in Example 6 as 0, 0, 0, 0, 6 and –2, so that the eigenvalues
are repeated according to their multiplicities

Activity:

__________________________________________________________________________________
@Virtual University Of Pakistan 159
15-The Characteristic Equation VU

Work out the eigenvalues and eigenvectors for the following square matrix.
 5 8 16 
A =  4 1 8  .
 −4 −4 −11
Similarity:
Let A and B be two n x n matrices, A is said to be similar to B if there exist
an invertible matrix P such that
P -1AP = B,
or equivalently,
A = PBP -1.
-1
Replacing Q by P , we have
Q -1BQ = A.
So B is also similar to A. Thus, we can say that A and B are similar.

Similarity transformation:
The act of changing A into P -1AP is called a similarity
transformation.

The following theorem illustrates use of the characteristic polynomial and it provides the
foundation for several iterative methods that approximate eigenvalues.

Theorem 2:
If n x n matrices A and B are similar, then they have the same
characteristic polynomial and hence the same eigenvalues (with the same multiplicities).

Proof: If B = P -1AP, then

= B - λ I P= -1
AP - λ I P -1 AP = - λ P -1 P P -1 (=
AP - λ P) P -1 ( A - λ I ) P
Using the multiplicative property (b) of Theorem 1, we compute
det( B - λ I ) = det  P -1 ( A - λ I ) P 
= det( P -1 ).det( A - λ I ).det( P) (A)
Since
det (P -1). det(P) = det(P -1P)
= det I
= 1,
Eq. (A) implies that
det( B - λ I ) = det( A - λ I ).
Hence, both the matrices have the same characteristic polynomials and therefore, same
eigenvalues.

Note: It must be clear that Similarity and row equivalence are two different concepts. ( If
A is row equivalent to B, then B = EA for some invertible matrix E.) Row operations on
a matrix usually change its eigenvalues.

__________________________________________________________________________________
@Virtual University Of Pakistan 160
 15-The Characteristic Equation VU

Application to Dynamical Systems:

Dynamical system is the one which evolves with the passage
of time. Eigenvalues and eigenvectors play a vital role in the evaluation of a dynamical
system. Let’s consider an example of a dynamical system.

.95 .03
Example 7: Let A =   . Analyze the long term behavior of the dynamical
.05 .97 
0.6 
system defined by x k+1 = Ax k (k = 0, 1, 2, …), with x0 =   .
0.4 
Solution: The first step is to find the eigenvalues of A and a basis for each eigenspace.
The characteristic equation for A is
= 0 det( A − λ I )
0.95 - λ 0.03 
= =  0.05 (.95 - λ )(.97 - λ ) - (.03)(.05)
0.97 - λ 
0 det

= λ 2 -1.92λ + .92
By the quadratic formula
1.92 ± (1.92) 2 - 4(.92) 1.92 ± .0064 1.92 ± .08
=λ = = = 1 or .92
2 2 2
Firstly, the eigenvectors will be found as given below.
Ax = λ x,
( Ax − λ x) =
0,
( A − λI )x =
0.
For λ = 1

 0.95 0.03   1 0    x1 
 −    =0,
 0.05 0.97   0 1    x2 
 −0.05 0.03   x1 
    = 0,
 0.05 −0.03   x2 
which can be written as
−0.05 x1 + 0.03 x2 = 0
0.03 3
0.05 x1 − 0.03 x2 =0 ⇒ x1 =x2 or x1 = x2 .
0.05 5
In parametric form, it becomes
3
= x1 = t and x2 t.
5

For λ = 0.92

__________________________________________________________________________________
@Virtual University Of Pakistan 161
 15-The Characteristic Equation VU

 0.95 0.03   0.92 0    x1 

 −    = 0,
 0.05 0.97   0 0.92    x2 
 0.03 0.03   x1 
    = 0.
 0.05 0.05   x2 
It can be written as
0.03 x1 + 0.03 x2 =0
0.05 x1 + 0.05 x2 =⇒
0 x1 =− x2
In parametric form, it becomes
x1 = t and x2 = −t

Thus, the eigenvectors corresponding to λ = 1 and λ = .92 are multiples of

 3 1
=v1 =  and v 2   respectively.
5  -1
The next step is to write the given x 0 in terms of v 1 and v 2 . This can be done because
{v 1 , v 2 } is obviously a basis for R2. So there exists weights c 1 and c 2 such that
c 
x0 = c1v1 + c2 v 2 =[v1 v 2 ]  1  (1)
c2 
−1
 c1  −1 3 1  .60 
In fact, =c  [ =
v 1 v 2 ] x0 5 -1 .40 
 2    
Here,
−1
3 1  1 3 1  1  −1 −1
5 −1 = 3 1 Adj 5 −1 = − 8  −5 3 
     
5 −1
Therefore,

 c1  1  -1 -1 .60  .125

=c  -8= -5 3  .40  .225 (2)
 2     

Because v 1 and v 2 in Eq.(1) are eigenvectors of A, with Av 1 = v 1 and Av 2 = (.92) v 2 , x k

can be computed as
x 1 = Ax 0 = c 1 Av 1 + c 2 Av 2 (Using linearity of x → Ax )
= c 1 v 1 + c 2 (.92)v 2 (v 1 and v 2 are eigenvectors.)
x 2 = Ax 1 = c 1 Av 1 + c 2 (.92)Av 2 = c 1 v 1 + c 2 (.92)2 v 2.
Continuing in the same way, we get the general equation as
x k = c 1 v 1 + c 2 (.92)kv 2 (k = 0, 1, 2, …).
Using c 1 and c 2 from Eq.(2),
 3 1
xk = .125   + .225(.92) k   (k = 0,1, 2,...) (3)
5  -1
This explicit formula for x k gives the solution of the difference equation x k+1 = Ax k .

__________________________________________________________________________________
@Virtual University Of Pakistan 162
15-The Characteristic Equation VU

.375
As k → ∞ , (.92)k tends to zero and x k tends to   = .125v1 .
.625

1 - 4 
Example 8: Find the characteristic equation and eigenvalues of A =  .
4 2 
Solution: The characteristic equation is
1- λ - 4 
= =
0 det( A - λ I ) det  
 4 2-λ
= (1- λ )(2 - λ ) - (-4)(4),
= λ 2 - 3λ + 18,
which is a quadratic equation whose roots are given as

3 ± (-3) 2 - 4(18)
λ=
2
3 ± -63
=
2
Thus, we see that the characteristic equation has no real roots, so A has no real
eigenvalues. A is acting on the real vector space R2 and there is no non-zero vector v in
R2 such that Av = λ v for some scalar λ .

__________________________________________________________________________________
@Virtual University Of Pakistan 163
15-The Characteristic Equation VU

Exercises:
Find the characteristic polynomial and the eigenvalues of matrices in exercises 1 to 12.

3 −2   5 −3
1.   2.  
1 −1  −4 3 

 2 1  3 −4 
3.   4.  
 −1 4  4 8 

 5 3 7 −2 
5.   6.  
 −4 4  2 3 

1 0 −1 0 3 1 

7. 2 3 −1
 
8. 3 0 2

   
 0 6 0  1 2 0 

 4 0 0  −1 0 1 

9. 5 3 2
 
10. −3 4 1

   
 −2 0 2   0 0 2 

 6 −2 0  5 −2 3 

11. −2 9 0
 
12. 0 1 0 
  
 5 8 3 6 7 −2 

For the matrices in exercises 13 to 15, list the eigenvalues, repeated according to their
multiplicities.

 4 −7 0 2  5 0 0 0
 0 3 −4 6  8 −4 0 0 
13.   14. 
 0 0 3 −8 0 7 1 0
   
0 0 0 1  1 −5 2 1

__________________________________________________________________________________
@Virtual University Of Pakistan 164
15-The Characteristic Equation VU

3 0 0 0 0
 −5 1 0 0 0 

15.  3 8 0 0 0
 
 0 −7 2 1 0
 −4 1 9 −2 3

16. It can be shown that the algebraic multiplicity of an eigenvalue λ is always greater
than or equal to the dimension of the eigenspace corresponding to λ . Find h in the matrix
A below such that the eigenspace for λ =5 is two-dimensional:

5 −2 6 −1
0 3 h 0 
A=
0 0 5 4
 
0 0 0 1

__________________________________________________________________________________
@Virtual University Of Pakistan 165
16-Diagonalization VU

Lecture 16

Diagonalization
Diagonalization is a process of transforming a vector A to the form A = PDP-1 for some
invertible matrix P and a diagonal matrix D. In this lecture, the factorization enables us to
compute Ak quickly for large values of k which is a fundamental idea in several
applications of linear algebra. Later, the factorization will be used to analyze (and
decouple) dynamical systems.
The “D” in the factorization stands for diagonal. Powers of such a D are trivial to
compute.

5 0   5 0   5 0   52 0
=
Example 1: If D  =  , then D 2 =     and
0 3 0 3 0 3  0 32 
 5 0   52 0  53 0 
= D 3 =  2  3
0 3  0 3  0 3 
 5k 0 
=
In general, D k
 k
for k ≥ 1
0 3 
The next example shows that if A = PDP-1 for some invertible P and diagonal D, then it
is quite easy to compute Ak.

 7 2
Example 2: Let A =  k -1
 . Find a formula for A , given that A = PDP , where
 - 4 1 
1 1  5 0 
= P =  and D  
-1 -2  0 3
Solution: The standard formula for the inverse of a 2 × 2 matrix yields
2 1
P -1 =  
-1 -1
By associative property of matrix multiplication,
−1
= A2 ( PDP -1 )(= PDP -1 ) PD ( P -1= =
P ) DP -1 PDIDP PDDP -1

1
where I is the identity matrix.
 1 1   52 0   2 1 
= PD= 2 -1
P -1 -2   2 
   0 3  -1 -1
Again, =A3 ( PDP= -1
P -1 )=
) A2 ( PD  =
P D 2 P -1 PDD 2 -1
P PD 3 P -1
1

 1 1   5k 0 2 1
= =
Ak PD k -1
P -1 -2    ,
Thus, in general, for k ≥ 1,   0 3k  -1 -1

__________________________________________________________________________________
@Virtual University Of Pakistan 166
16-Diagonalization VU

 5k 3k   2 1 
= k  ,
 −5 −2.3k  -1 -1
 2.5k - 3k 5k - 3k 
= k k .
 2.3 - 2.5 2.3k - 5k 
Activity:
Work out C 4 , given that C = PDP −1 where
1 0  2 0
= P =  , D 
3 1  0 1 

Remarks:
A square matrix A is said to be diagonalizable if A is similar to a diagonal matrix, that is,
if A = PDP-1 for some invertible matrix P and some diagonal matrix D. The next theorem
gives a characterization of diagonalizable matrices and tells how to construct a suitable
factorization.

Theorem 1: The Diagonalization Theorem

An n x n matrix A is diagonalizable if and only if A has n linearly independent
eigenvectors.

In fact, A = PDP-1, with D a diagonal matrix, if and only if the columns of P are n
linearly independent eigenvectors of A. In this case, the diagonal entries of D are
eigenvalues of A that correspond, respectively, to the eigenvectors in P.

In other words, A is diagonalizable if and only if there are enough eigenvectors to form a
basis of Rn. We call such a basis an eigenvector basis.

Proof: First, observe that if P is any n × n matrix with columns v 1 , … , v n and if D is any
diagonal matrix with diagonal entries λ1 ,...., λn then
= [v1 v2 ... vn ]
AP A= [ Av1 Av 2 ... Avn ] , (1)
λ1 0  0 
0 λ  0 
=
while PD P=  2  [λ v λ v  λ v ] (2)
   1 1 2 2 n n

 
 0 0  λn 
Suppose now that A is diagonalizable and A = PDP-1. Then right-multiplying this
relation by P, we have AP = PD. In this case, (1) and (2) imply that
[ Av1 Av2  Avn ] = [λ1v1 λ2v2  λn vn ] (3)
Equating columns, we find that
= Av1 λ= 1v 1 , Av 2 λ2 v 2 ,=
 , Avn λn vn (4)

Since P is invertible, its columns v 1 ,…, v n must be linearly independent. Also, since these
columns are nonzero, Eq.(4) shows that λ1 ,....., λn are eigenvalues and v 1 , …, v n are

__________________________________________________________________________________
@Virtual University Of Pakistan 167
16-Diagonalization VU

corresponding eigenvectors. This argument proves the “only if” parts of the first and
second statements along with the third statement, of the theorem.

Finally, given any n eigenvectors v 1 , …, v n use them to construct the columns of P and
use corresponding eigenvalues λ1 ,....., λn to construct D. By Eqs. (1) – (3). AP =PD. This
is true without any condition on the eigenvectors. If, in fact, the eigenvectors are linearly
independent, then P is invertible (by the Invertible Matrix Theorem), and AP = PD
implies that A = PDP-1.

Diagonalizing Matrices
1 3 3
Example 3: Diagonalize the following matrix, if possible A = -3 -5 -3
 3 3 1 
Solution: To diagonalize the given matrix, we need to find an invertible matrix P and a
diagonal matrix D such that A = PDP-1 which can be done in following four steps.

Step 1: Find the eigenvalues of A.

The characteristic equation becomes
0= det( A − λ I ) =−λ 3 − 3λ 2 + 4
= −(λ − 1)(λ + 2) 2
The eigenvalues are λ = 1 and λ = -2 (multiplicity 2)
Step 2: Find three linearly independent eigenvectors of A. Since A is a 3 × 3 matrix and
we have obtained three eigen values, we need three eigen vectors. This is the critical step.
If it fails, then above Theorem says that A cannot be diagonalized. Now we will produce
basis for these eigen values.

Basis vector for λ = 1:

( A − λI )x =
0
 0 3 3   x1  0 
 −3 −6 −3  x  =  
   2  0  .
 3 3 0   x3  0 
After applying few row operations on the matrix ( A − λ I ) , we get
0 1 1   x1  0 
3 3 0   x  = 0  ,
  2  
0 0 0   x3  0 
which can be written as
x2 + x3 =
0
3 x1 + 3 x2 =0
In parametric form, it becomes
x1 = t , x2 =
−t , x3 =
t

__________________________________________________________________________________
@Virtual University Of Pakistan 168
16-Diagonalization VU

1
Thus, the basis vector for λ = 1 is v1 =  −1
 1 

Basis vector for λ = -2

( A − λI )x =
0
 3 3 3   x1  0 
 −3 −3 −3  x  =  
   2  0  ,
 3 3 3   x3  0 
which can be written as
3 x1 + 3 x2 + 3 x3 =0
−3 x1 − 3 x2 − 3 x3 =
0
3 x1 + 3 x2 + 3 x3 =
0
In parametric form, it becomes
x1 =− s − t , x2 =s, x3 =t
Now,
 x1   − s − t   − s   −t 
=     s  +  0 ,
 x2   s=     
 x3   t   0   t 
 −1  −1
= s  1  + t  0  ,
 0   1 
 −1  −1
= x2  1  + x3  0  .
 
 0   1 
 −1  −1
Thus, the basis for λ = −2 is v2 =  1  and v3 =  0 
 
 0   1 

We can check that {v 1 , v 2 , v 3 } is a linearly independent set.

Step 3: Check that {v 1 , v 2 , v 3 } is a linearly independent set.
Construct P from the vectors in step 2. The order of the vectors is not important. Using
 1 -1 -1
the order chosen in step= 2, form P [v= 1 v 2 v 3 ] -1 1 0 
 1 0 1 

__________________________________________________________________________________
@Virtual University Of Pakistan 169
 16-Diagonalization VU

Step 4: Form D from the corresponding eigen values. For this purpose, the order of the
eigen values must match the order chosen for the columns of P. Use the eigen value λ = -
2 twice, once for each of the eigenvectors corresponding to λ = -2:

1 0 0 
D = 0 -2 0 
0 0 -2 
Now, we need to check do P and D really work. To avoid computing P -1, simply verify
that AP = PD. This is equivalent to A = PDP -1 when P is invertible.We compute

 1 3 3   1 -1 -1 1 2 2
=AP =-3 -5 -3 -1 1 0  -1
 -2 0 
 3 3 1   1 0 1   1 0 -2 
 1 -1 -1 1 0 0   1 2 2
=PD =
-1 1 0  0 -2 0  -1 -2 0 
 1 0 1  0 0 -2   1 0 -2 

Example 4: Diagonalize the following matrix, if possible.

2 4 3
A = -4 -6 -3
 3 3 1 
Solution: The characteristic equation of A turns out to be exactly the same as that in
example 3 i.e.,
0 = det( A - λ I )
2−λ 4 3
= −4 −6 − λ −3
3 3 1− λ
= -λ 3 - 3λ 2 + 4
= -(λ -1)(λ + 2) 2
The eigen values are λ = 1 and λ = -2 (multiplicity 2). However, when we look for eigen
vectors, we find that each eigen space is only one – dimensional.
1
Basis= for λ 1:= v1 -1
 1 
-1
=
Basis for λ -2=
: v 2  1 
 0 

__________________________________________________________________________________
@Virtual University Of Pakistan 170
16-Diagonalization VU

There are no other eigen values and every eigen vector of A is a multiple of either v 1 or
v 2 . Hence it is impossible to form a basis of R3 using eigenvectors of A. By above
Theorem, A is not diagonalizable.

Theorem 2: An n x n matrix with n distinct eigenvalues is diagonalizable.

The condition in Theorem 2 is sufficient but not necessary i.e., it is not necessary for an n
x n matrix to have n distinct eigen values in order to be diagonalizable. Example 3 serves
as a counter example of this case where the 3 x 3 matrix is diagonalizable even though it
has only two distinct eigen values.

Example 5: Determine if the following matrix is diagonalizable.

5 -8 1 
A = 0 0 7 
0 0 -2 
Solution : In the light of Theorem 2, the answer is quite obvious. Since the matrix is
triangular, its eigen values are obviously 5, 0, and –2.Since A is a 3 x 3 matrix with
three distinct eigen values, A is diagonalizable.

Matrices Whose Eigenvalues Are Not Distinct:

If an n x n matrix A has n distinct eigen values, with corresponding eigen vectors v1 ,..., v
n and if P = [v1 … vn] , then P is automatically invertible because its columns are linearly
independent , by Theorem 2 of lecture 28. When A is diagonalizable but has fewer than n
distinct eigen values, it is still possible to build P in a way that makes P automatically
invertible, as shown in the next theorem.

Theorem 3: Let A be an n x n matrix whose distinct eigen values are λ1 ,..., λ p .

a. For 1 ≤ k ≤ p, the dimension of the eigen space for λ k is less than or equal to the
multiplicity of the eigen value λ k
b. The matrix A is diagonalizable if and only if the sum of the dimensions of the
distinct eigen spaces is equal to n, and this happens if and only if the dimension of
the eigen space for each of λ k equals the multiplicity of λ k .
c. If A is diagonalizable and B k is basis for the eigen space corresponding to λ k for
each k, then the total collection of vectors in the sets B 1 , ..., B p form an
eigenvector basis for Rn .

Example 6: Diagonalize the following matrix, if possible.

__________________________________________________________________________________
@Virtual University Of Pakistan 171
 16-Diagonalization VU

5 0 0 0
0 5 0 0
A= 
 1 4 -3 0 
 
-1 -2 0 -3
Solution: Since A is triangular matrix, the eigenvalues are 5 and –3, each with
multiplicity 2. Using the method of lecture 28, we find a basis for each eigen space.
-8  -16 
4  4 
=
Basis for λ 5=: v1   and= v2  
1  0 
   
0  1 
0  0 
0  0 
=
Basis for λ -3=: v3   and
= v4  
1  0 
   
0  1 
The set {v 1 ,…,v 4 }is linearly independent, by Theorem 3. So the matrix P =[v 1 …v 4 ] is
invertible, and A=PDP -1 , where

-8 -16 0 0  5 0 0 0
 4 4 0 0 0 5 0 0 
=P =  and D 
 1 0 1 0 0 0 -3 0 
   
 0 1 0 1 0 0 0 -3

Example 7:
 4 -3
(1) Compute A8 where A =  
 2 -1
 -3 12   3 2
=
(2) Let A  =  1 1
, v = , and v 2 1  . Suppose you are told that v 1 and
-2 7     
v 2 are eigenvectors of A. Use this information to diagonalize A.
(3) Let A be a 4 x 4 matrix with eigenvalues 5, 3, and -2, and suppose that you
know the eigenspace for λ =3 is two-dimensional. Do you have enough
information to determine if A is diagonalizable?
Solution:
Here, det (A- λ I)= λ 2 -3 λ +2=( λ -2)( λ -1).

__________________________________________________________________________________
@Virtual University Of Pakistan 172
 16-Diagonalization VU

The eigen values are 2 and 1, and corresponding eigenvectors are

3 1
=v1 =  , and v 2 1 . Next, form
2 
 3 1 2 0  1 -1
= P =  , D   ,=and P −1 -2 3 
 2 1 0 1  
 3 1  28 0   1 -1
Since A = PDP , A = PD P = 
–1 8 8
 8
 2 1  0 1  -2 3 
 3 1  256 0   1 -1
=   
 2 1  0 1  -2 3 
766 -765
= 
510 -509 
 -3 12  3  3
=
(2)Here, Av1 -2 7  =  =  1.v1 , and
  1 1
 -3 12   2  6 
=
Av 2 -2 7  =  =  3.v2
  1  3
Clearly, v 1 and v 2 are eigenvectors for the eigenvalues 1 and 3 , respectively. Thus
3 2  1 0 
=
A = PDP -1
, where P =
 and D  
1 1  0 3
(3) Yes A is diagonalizable. There is a basis {v 1 , v 2 } for the eigen space corresponding
to λ =3. Moreover, there will be at least one eigenvector for λ =5 and one for λ =-2 say
v 3 and v 4 . Then {v 1 , …., v 4 } is linearly independent and A is diagonalizable , by
Theorem 3. There can be no additional eigen vectors that are linearly independent from
v 1 to v 4 because the vectors are all in R4 .Hence the eigenspaces for λ =5 and λ =-2 are
both one–dimensional.

__________________________________________________________________________________
@Virtual University Of Pakistan 173
16-Diagonalization VU

Exercise:

In exercises 1 and 2, let A = PDP-1 and compute A4.

5 7  2 0  2 −3 1 0 
1. P =   ,D =   2. P =   ,D =  
2 3 0 1  −3 5  0 1/2 

In exercises 3 and 4, use the factorization A = PDP-1 to compute Ak, where k represents
an arbitrary positive integer.

 a 0  1 0   a 0   1 0 
3. 
−  = 3 1   0 b   −3 1 
 3( a b ) b     

 −2 12  3 4   2 0   −1 4 
4.  =   
 −1 5  1 1   0 1   1 −3

In exercises 5 and 6, the matrix A is factored in the form PDP-1. Use the Diagonalization
Theorem to find the eigenvalues of A and a basis for each eigenspace.

 2 2 1  1 1 2  5 0 0  1 / 4 1 / 2 1/ 4 

5. 1 3 1 =
 1 0 −1   0 1 0  1 / 4 1 / 2 −3 / 4 
     
1 2 2  1 −1 0  0 0 1  1 / 4 −1 / 2 1 / 4 

 4 0 −2   −2 0 −1 5 0 0   0 0 1 
   
6. 2 5 4 = 0 1 2 0 5 0 2 1 4
 
     
 0 0 5   1 0 0  0 0 4   −1 0 −2 

Diagonalize the matrices in exercises 7 to 18, if possible.

3 −1  2 3
7.   8.  
1 5   4 1

 −1 4 −2  4 2 2

9. −3 4 0
 
10. 2 4 2

   
 −3 1 3   2 2 4 

__________________________________________________________________________________
@Virtual University Of Pakistan 174
16-Diagonalization VU

 2 2 −1  4 0 −2 

11. 1 3 −1 
12. 2 5 4

  
 −1 −2 2   0 0 5 

7 4 16   0 −4 −6 

13. 2 5 8  
14. −1 0 −3

  
 −2 −2 −5  1 2 5 

4 0 0  −7 −16 4 

15. 1 4 0
 
16. 6 13 −2 
  
 0 0 5  12 16 1 

5 −3 0 9 4 0 0 0
0 3 1 −2  0 4 0 0 
17.  18. 
0 0 2 0 0 0 2 0
   
0 0 0 2 1 0 0 2

__________________________________________________________________________________
@Virtual University Of Pakistan 175
17- Inner Product VU

Lecture 17

Inner Product

If u and v are vectors in R n , then we regard u and v as n × 1 matrices. The transpose

u t is a 1× n matrix, and the matrix product u t v is a 1×1 matrix which we write as a
single real number (a scalar) without brackets.
The number u t v is called the inner product of u and v . And often is written as u.v
This inner product is also referred to as a dot product.
u1  v1 
u  v 
 2  2
.  . 
If u =   and v =  
.  . 
.  . 
   
un  vn 

Then the inner product of u and v is

v1 
v 
 2
. 
u1 u2 . . . un   = u1v1 + u2 v2 + ... + un vn
. 
. 
 
vn 

Example 1
 2  3
Compute u.v and v.u when u =   
 −5  and v =
 2 .
 −1   −3 
Solution

 2  3
u=   2
 −5  and v = 
 −1   −3 
u t = [ 2 − 5 − 1]
 3
u.v= u v= [ 2 − 5 − 1]  2 = 2(3) + (−5)(2) + (−1)(−3)
t

 −3 
=6 − 10 + 3 =−1

©Virtual University Of Pakistan 176

 17- Inner Product VU

=vt [3 2 − 3]
 2
v.u= v u= [3 2 − 3]  −5 = 3(2) + (2)(−5) + (−3)(−1)
t

 −1 
=6 − 10 + 3 =−1

Theorem
Let u , v and w be vectors in R n , and let c be a scalar. Then
a. u.v = v.u
b. (u + v).w =u.w + v.w
c. (cu= ).v c=
(u.v) u.(cv)
d. u.u ≥ 0 and u.u = 0 if and only if u = 0

Observation

(c1u1 + c2u2 + ....c p u p ).w

= c1 (u1.w) + c2 (u2 .w) + ...c p (u p .w)

Length or Norm
The length or Norm of v is the nonnegative scalar v defined by
v= v.v= v12 + v12 + ... + vn 2
v = v.v
2

Note: For any scalar c , cv = c v

Unit vector
A vector whose length is 1 is called a unit vector. If we divide a non-zero vector
v by its length v , we obtain a unit vector u as
v
u=
v
1
The length of u is =u = v 1
v

Definition
The process of creating the unit vector u from v is sometimes called normalizing v ,
and we say that u is in the same direction as v . In this case “ u ” is called the
normalized vector.

Example 2
Let v = (1, 2, 2, 0) in R 4 . Find a unit vector u in the same direction as v .
Solution
The length of v is given by

©Virtual University Of Pakistan 177

17- Inner Product VU

v = v.v = v12 + v2 2 + v32 + v4 2

So,
v = 12 + 22 + 22 + 02 = 1+ 4 + 4 + 0 = 9= 3
The unit vector u in the direction of v is given as

 1
 3
 1  
   2 
1 1  2 
=u = v = 3
v 3  2  
   2
 0  3
 
 0
To check that u = 1
1 −2 2 1 4 4
u = u.u = ( ) 2 + ( ) 2 + ( ) 2 + (0) 2 = + + +0= 1
3 3 3 9 9 9

Example 3
2
Let W be the subspace of R 2 spanned by X = ( ,1) . Find a unit vector Z that is a basis
3
for W.

Solution

W consists of all multiples of x, as in Fig. 2(a). Any nonzero vector in W is a basis

for W. To simplify the calculation, x is scaled to eliminate fractions. That is, multiply
x by 3 to get
2
y= 
3
Now compute y = 22 + 32 = 13, y = 13, and normalize y to get
2

1 2  2 13 
=z =    
13  3   3 13 

See Fig. 2(b). Another unit vector is (−2 13 , −3 13).

©Virtual University Of Pakistan 178

17- Inner Product VU

Figure 2 Normalizing a vector to produce a unit vector.

Definition
For u and v vectors in R n , the distance between u and v , written as dist ( u , v ), is the
length of the vector u − v . That is
dist (u , v=
) u −v

Example 4
Compute the distance between the vectors u = (7, 1) and v = (3, 2)

Solution
 7   3   7 − 3  4 
u − v=   −  =  =  
1   2  1 − 2   −1 

dist ( u , v ) = u − v= (4) 2 + (−1)=

2
16 +=
1 17

Law of Parallelogram of vectors

The vectors, u , v and u − v are shown in the fig. below. When the vector u − v is
added to v , the result is u . Notice that the parallelogram in the fig. below shows that
the distance from u to v is the same as the distance of u − v to o .

©Virtual University Of Pakistan 179

17- Inner Product VU

Example 5
If u = (u1 , u2 , u3 ) and v = (v1 , v2 , v3 ) , then
dist (u , v) = u − v = (u − v).(u − v)
= (u1 − v1 ) 2 + (u2 − v2 ) 2 + (u3 − v3 ) 2
Definition
Two vectors in u and v in R n are orthogonal (to each other) if u.v = 0

Note
The zero vector is orthogonal to every vector in R n because 0 .v = 0 for all v in R n .
t

The Pythagorean Theorem

u+v = u + v
2 2 2
Two vectors u and v are orthogonal if and only if

Orthogonal Complements
The set of all vectors z that are orthogonal to w in W is called the orthogonal
complement of W and is denoted by w⊥
Example 6
Let W be a plane through the origin in R3, and let L be the line through the origin
and perpendicular to W. If z and w are nonzero, z is on L, and w is in W, then the line
segment from 0 to z is perpendicular to the line segment from 0 to w; that is, z . w = 0.
So each vector on L is orthogonal to every w in W. In fact, L consists of all vectors
that are orthogonal to the w’s in W, and W consists of all vectors orthogonal to the z’s
in L. That is,
L = W ⊥ and W = L⊥

©Virtual University Of Pakistan 180

17- Inner Product VU

Remarks

The following two facts about W ⊥ , with W a subspace of Rn, are needed later in the
segment.
(1) A vector x is in W ⊥ if and only if x is orthogonal to every vector in a set that
spans W.
(2) W ⊥ is a subspace of Rn.

Theorem 3
Let A be m x n matrix. Then the orthogonal complement of the row space of A is the
null space of A, and the orthogonal complement of the column space of A is the null
space of AT: (Row A) ⊥ = Nul A, (Col A) ⊥ = Nul AT
Proof
The row-column rule for computing Ax shows that if x is in Nul A, then x is
orthogonal to each row of A (with the rows treated as vectors in Rn). Since the rows of
A span the row space, x is orthogonal to Row A. Conversely, if x is orthogonal to
Row A, then x is certainly orthogonal to each row of A, and hence Ax = 0. This
proves the first statement. The second statement follows from the first by replacing A
with AT and using the fact that Col A = Row AT.

Angles in R2 and R3
If u and v are nonzero vectors in either R2 or R3, then there is a nice connection
between their inner product and the angle ϑ between the two line segments from the
origin to the points identified with u and v. The formula is
u ⋅ v =u v cos ϑ (2)
2
To verify this formula for vectors in R , consider the triangle shown in Fig. 7, with
sides of length u , v , and u − v . By the law of cosines,
u − v = u + v − 2 u v cos ϑ
2 2 2

which can be rearranged to produce

1 2
u v cos= ϑ u + v − u−v 
2 2

2  
1
= u12 + u22 + v12 + v22 − (u1 − v1 ) 2 − (u2 − v2 ) 2  =u1v1 + u2 v2 =u ⋅ v
2
(u 1 , u 2 )

u u−v
ϑ
v (v 1 , v 2 )

Figure 7 The angle between two vectors.

©Virtual University Of Pakistan 181

 17- Inner Product VU

Example 7
Find the angle between the vectors u=
(1, −1, 2), v=
(2,1, 0)

Solution
u.v = (1)(2) + (−1)(1) + (2)(0) = 2 − 1 + 0 = 1
And
u= (1) 2 + (−1) 2 + (2) 2 = 1 + 1 + 4= 6
v= (2) 2 + (1) 2 + (0) 2 = 4 + 1= 5

Angle between the two vectors is given by

u⋅v
cos ϑ =
u v
Putting the values, we get

1 1
cos θ
= =
6 5 30
1
cos θ =
30
−1 1
=θ cos= 79.48°
30
Exercises

Q.1
1   −3
when u =
Compute u.v and v.u= 5 and v  
 1
3  5
Q.2
Let v = (2,1, 0,3) in R 4 . Find a unit vector u in the direction opposite to that of v .

Q.3
1 3 5
Let W be the subspace of R 3 spanned by X = ( , , ) . Find a unit vector Z that is a
2 2 2
basis for W.

Q.4
Compute the distance between the vectors u = (1, 5, 7) and v = (2, 3, 5).

Q.5
=
Find the angle between the vectors =
u (2,1,3), v (1, 0, 2) .

©Virtual University Of Pakistan 182

 18-Orthogonal and Orthonormal Sets VU

Lecture 18
Orthogonal and Orthonormal sets
Objectives

The objectives of the lecture are to learn about:

• Orthogonal Set.
• Orthogonal Basis.
• Unique representation of a vector as a linear combination of Basis vectors.
• Orthogonal Projection.
• Decomposition of a vector into sum of two vectors.
• Orthonormal Set.
• Orthonormal Basis.
• Some examples to verify the definitions and the statements of the theorems.

Orthogonal Set

{ }
Let S = u1 , u2 ,..., u p be the set of non-zero vectors in R n , is said to be an orthogonal set
if all vectors in S are mutually orthogonal. That is
O ∉ S and ui . u j = o ∀ i ≠ j , i, j = 1, 2,..., p.

Example

Show that S = {u1 , u2 , u3 } is an orthogonal set. Where

 −1 
 3  −1  2
 
=u1 1=  ,u
 2
 2  and u =
  3  −2  .
1   1   7
 
 2
Solution

To show that S is orthogonal, we show that each vector in S is orthogonal to other. That
is
ui . u j = o ∀ i ≠ j , i, j = 1, 2,3.
For= i 1,= j 2
3  −1 
u1 . u2 = 1  .  2 
1   1 
=− 3 + 2 + 1 =0
Which implies u1 is orthogonal to u2 .

© Virtual University Of Pakistan 183

18-Orthogonal and Orthonormal Sets VU

For= i 1,= j 3
 −1 
 3  2 
u= 1  .  − 2 
1 . u3    
1   7 
 
 2 
−3 7
= −2+
2 2
= 0.

Which implies u1 is orthogonal to u3 .

For= i 2,= j 3
 −1 
 −1  2 
 
u2 . u3  2  .  − 2 
=
 
 1   7 
 2 
1 7
= −4+
2 2
= 0.

Which implies u2 is orthogonal to u3 .

Thus S = {u1 , u2 , u3 } is an orthogonal set.

Theorem
{ }
Suppose that S = u1 , u2 ,..., u p is an orthogonal set of non-zero vectors in R n and
W = Span {u1 , u2 ,..., u p } . Then S is linearly independent set and a basis for W .

Proof

Suppose
0= c1u1 + c2u2 + ... + c p u p .
Where c1 , c2 ..., c p are scalars.
u=
1 .0 u1 .(c1u1 + c2u2 + ... + c p u p )
=
0 u1 .(c1u1 ) + u1 .(c2u2 ) + ... + u1 .(c p u p )
= c1 (u1 . u1 ) + c2 (u1 . u2 ) + ... + c p (u1 . u p )
= c1 (u1 . u1 )

© Virtual University Of Pakistan 184

18-Orthogonal and Orthonormal Sets VU

Since S is orthogonal set, so, u1 . u2 + ... + u1 . u p =0 but u1 . u1 > 0.

Therefore c1 = 0 . Similarly, it can be shown that c2= c3= ...= c p= 0
Therefore by definition S = {u1 , u2 ,..., u p } is linearly independent set and by definition of
basis is a basis for subspace W .

Example

If S = {u1 , u2 } is an orthogonal set of non-zero vector in R 2 . Show that S is linearly

independent set. Where
 3  −1
u1 =   and u2 =   .
1   3

Solution

To show that S = {u1 , u2 } is linearly independent set, we show that the following vector
equation
c1u1 + c2u2 =
0.
has only the trivial solution. i.e. c=
1 c=
2 0.
c1u1 + c2u2 =
0
3  −1  0 
c1   + c2   =
 
1   3 0 
3c1   −c2  0 
c  +  3c  = 
 1   2  0 
3c1 − c2 =
0
c1 + 3c2 =
0
Solve them simultaneously, gives
c=
1 c=
2 0.
Therefore if S is an orthogonal set then it is linearly independent.

Orthogonal basis

Let S = {u1 , u2 ,..., u p } be a basis for a subspace W of R n , is also an orthogonal basis if S

is an orthogonal set.

Theorem
If S = {u1 , u2 ,..., u p } is an orthogonal basis for a subspace W of R n . Then each y in W can
be uniquely expressed as a linear combination of u1 , u2 ,..., u p . That is
y= c1u1 + c2u2 + ... + c p u p .

© Virtual University Of Pakistan 185

 18-Orthogonal and Orthonormal Sets VU

Where
y .u j
cj =
u j .u j

Proof

y . u=1 (c1u1 + c2u2 + ... + c p u p ). u1

= (c1u1 ). u1 + (c2u2 ). u1 + ... + (c p u p ). u1
= c1 (u1 . u1 ) + c2 (u1 . u2 ) + ... + c p (u1 . u p )
= c1 (u1 . u1 ).
Since S is orthogonal set, so, u1 . u2 + ... + u1 . u p =0 but u1 . u1 > 0.
Hence
y . u1 y . u2 y .u p
c1 = and =
similarly c2 = ,... c p .
u1 . u1 u2 . u2 u p .u p

Example

The set S = {u1 , u2 , u3 } as in first example is an orthogonal basis for R 3 . Express y as a

linear combination of the vectors in S . Where
=y [6 1 − 8]
T

Solution

We want to write
y =c 1 u1 + c 2 u2 + c 3 u3
Where c 1 , c 2 and c 3 are to be determined.
By the above theorem
y . u1
c1 =
u1 . u1
 6  3
 1 . 1 
  
 −8  1  11
= = = 1
 3  3 11
 1. 1 
  
 1   1 

© Virtual University Of Pakistan 186

18-Orthogonal and Orthonormal Sets VU

y . u2
c2 =
u2 . u2
 6   − 1
 1  .  2
   
 −8  1  −12
= = = −2
 − 1  − 1 6
 2 .  2
   
 1   1 
And
y . u3
c3 =
u3 . u3
 −1 
 6  2 
 1  .  −2 
   
 −8  7 
 
=  2  = −33 = − 2
 −1   −1  33 / 2
 2  2
   
 −2  .  −2 
 7  7
   
 2  2

Hence
y= u1 − 2u2 − 2u3 .

Example

The set S = {u1 , u2 , u3 } is an orthogonal basis for R 3 . Write y as a linear combination of

the vectors in S . Where
3   1 1  0 
y=        
7  , u1 = 1  and u3 = 0 
 −1  , u2 =
 4   0  0  1 

Solution

We want to write
y =c 1 u1 + c 2 u2 + c 3 u3

© Virtual University Of Pakistan 187

 18-Orthogonal and Orthonormal Sets VU

Where c 1 , c 2 and c 3 are to be determined.

By the above theorem
3   1 
 7  .  −1 
   
4  0 3−7 + 0
=     =
y . u1
c1 = = −2
u1 . u1  1   1  1+1+ 0
 −1  .  −1 
   
 0   0 
3   1 
7  .  1 
   
y . u2  4   0  3+ 7 + 0
=c2 = = = 5
u2 . u2  1  1 1+1
 1 . 1 
   
 0   0 

And
3   0 
7  .  0
   
y . u3  4   1  4
c3= = = = 4
u3 . u3  0  0 1
 0 .  0
   
 1   1 
Hence
y= − 2u1 + 5u2 + 4u3 .

Exercise

The set S = {u1 , u2 , u3 } is an orthogonal basis for R 3 . Write y as a linear combination of

the vectors in S . where
1 8 
5 3
2  2    
y=  1  , u 
= −1  , u 
=
2
and u = 16 
  1   2
5 3
3
3   0     
 −1 8 
   3 

© Virtual University Of Pakistan 188

18-Orthogonal and Orthonormal Sets VU

An Orthogonal Projection (Decomposition of a vector into the sum of two vectors

Decomposition of a non- zero vector y ∈ R n into the sum of two vectors in such a way,
one is multiple of u ∈ R n and the other orthogonal to u . That is
=y y∧ + z
Where y ∧ = α u for some scalar α and z is orthogonal to u .

− y∧ z= y − y ∧
u

y
y∧ = αu

z= y − y ∧

− y∧

In the above figure a vector y is decomposed into two vectors z= y − y ∧ and y ∧ = α u.

Clearly it can be seen that z= y − y ∧ is orthogonal to u and y ∧ = α u is a multiple of u .
Since z= y − y ∧ is orthogonal to u.
Therefore
z .u = 0
( y − y ∧ ). u = 0
( y − α u ). u = 0
y . u − α (u . u ) =0
y .u
⇒α =
u .u
And
z= y − y ∧
y .u
= y− u
u .u

© Virtual University Of Pakistan 189

18-Orthogonal and Orthonormal Sets VU

Hence
y .u
y∧ = u , which is an orthogonal projection of y onto u .
u .u
And
z= y − y ∧
y .u is a component of y
= y− u
u .u

Example

7  4
Let y =   and u =   .
6 2
Find the orthogonal projection of y onto u. Then write y as a sum of two orthogonal
vectors, one in span {u} and one orthogonal to u.

Solution

Compute
7   4
y ⋅ u=   ⋅  = 40
6 2
4 4
u ⋅ u=   ⋅  = 20
2 2
y ⋅u 40  4  8 
The orthogonal projection of y onto u is=yˆ = u = u 2=
 2   4  and the
u ⋅u 20    
     
7 8 − 1
component of y orthogonal to u is y − yˆ=   −  =  
6 4  2 
7  8   −1
= +
The sum of these two vectors is y. That is,  6   4   2 
y yˆ ( y − yˆ )

This decomposition of y is illustrated in Fig. 3. Note: If the calculations above are

correct, then { yˆ , y − yˆ } will be an orthogonal set. As a check, compute
8   −1
yˆ ⋅ ( y − yˆ ) =  ⋅   =−8 + 8 =0
4  2 

© Virtual University Of Pakistan 190

18-Orthogonal and Orthonormal Sets VU

x2

•y

L = Span {u}
y − ˆy • ŷ
•2
•u

x1
1 8

Figure 3 The orthogonal projection of y on to a line through the origin.

Example

Find the distance in figure below from y to L.

x2

•y

L = Span {u}
y − ˆy • ŷ
•2
•u

x1
1 8

Solution

The distance from y to L is the length of the perpendicular line segment from y to the
orthogonal projection ŷ .
The length equals the length of y − yˆ .

This distance is
y − yˆ = (−1) 2 + 22 = 5

© Virtual University Of Pakistan 191

 18-Orthogonal and Orthonormal Sets VU

Example
Decompose y = (-3,-4) into two vectors ŷ and z, where ŷ is a multiple of u =
(-3, 1) and z is orthogonal to u. Also prove that y ∧ . z = 0
Solution
It is very much clear that y ∧ is an orthogonal projection of y onto u and it is calculated by
applying the following formula
y. u
y∧ = u
u. u
 −3   −3  3
 −4  .  1   −3
9 − 4  −3 1  −3 − 2 
=      = =   = 
 −3  −3  1  9 + 1  1  2  1  1
 1. 1  2 
  

 
 
   3
∧  −3   − 3   −3 + 3/ 2  − 2 
z =y − y =  − =  = 
 −4   2   −4 − 1/ 2   − 9 
1  2 
 2 

So,
 3  3
∧
− 2  − 2 
=y =  and z  
 1 − 9 
 2   2 

Now

© Virtual University Of Pakistan 192

18-Orthogonal and Orthonormal Sets VU

 3  3
∧
− 2  − 2 
y .z =  .  
 1 − 9 
 2   2 
9 9
= −
4 4
=0

∧
Therefore, y is orthogonal to z

Exercise

Find the orthogonal projection of a vector y = (-3, 2) onto u = (2, 1). Also prove that
=y y ∧ + z , where y ∧ a multiple of u and z is is an orthogonal to u.

Orthonormal Set

Let S = {u1 , u2 ,..., u p } be the set of non-zero vectors in R n , is said to be an orthonormal set
if S is an orthogonal set of unit vectors.

Example

Show that S = {u1 , u2 , u3 } is an orthonormal set. Where

 2   1 
 5  0  5
   −1 and u =  0  .
u1 =  0  , u2 =   3  
 −1   0   2 
   
 5   5 

Solution

To show that S is an orthonormal set, we show that it is an orthogonal set of unit

vectors.
It can be easily prove that S is an orthogonal set because
ui . u j = 0 ∀ i ≠ j , i, j = 1, 2,3.
Furthermore

© Virtual University Of Pakistan 193

18-Orthogonal and Orthonormal Sets VU

 2   1 
 5  0  5
   −1 u =  
u1 =  0  , u2 =   3  0 .
 −1   0   2 
   
 5   5 
 2   2 
 5  5
   
u1 . u1 =  0  .  0 
 −1   −1 
   
 5   5 
4 1
= +0+
5 5
=1
 0  0
u2 . u2 =   
 −1 .  −1
 0   0 
= 0 +1+ 0
=1

And
 1   1 
 5  5
   
u3 . u3 =  0  .  0 
 2   2 
   
 5   5 
1 4
= +
5 5
=1
Hence
S = {u1 , u2 , u3 } is an orthonormal set.

Orthonormal basis

{ }
Let S = u1 , u2 ,..., u p be a basis for a subspace W of R n , is also an orthonormal basis if
S is an orthonormal set.

Example

© Virtual University Of Pakistan 194

 18-Orthogonal and Orthonormal Sets VU

Show that S = {u1 , u2 , u3 } is an orthonormal basis of R 3 , where

 −1 
 3    −1 
 11  6  
  66 
   2  
 1     −4 
=u1 = , u2 6 and u3 =  .
11   
 66 
   1 
 1    7 
6 
 11     66 
 

Solution

To show that S = {u1 , u2 , u3 } is an orthonormal basis, it is sufficient to show that it is an

orthogonal set of unit vectors. That is
ui . u j = 0 ∀ i ≠ j , i, j = 1, 2,3.
And
ui . u j = 1∀ i = j , i, j = 1, 2,3.
Clearly it can be seen that
u1 . u2 = 0,
u1 . u3 = 0
And
u2 . u3 = 0.
Furthermore
u1 . u1 = 1,
u2 . u2 = 1
And
u3 . u3 = 1.
Hence S is an orthonormal basis of R 3 .

Theorem
A m × n matrix U has orthonormal columns if and only if U t U = I
Proof
Keep in mind that in an if and only if statement, one part depends on the other,
so, each part is proved separately. That is, we consider one part and then prove the other
part with the help of that assumed part.
Before proving both sides of the statements, we have to do some extra work which is
necessary for the better understanding.
Let u1 , u2 ,..., um be the columns of U. Then U can be written in matrix form as
U = [u1 u2 u3 ... um ]
Taking transpose, it becomes

© Virtual University Of Pakistan 195

18-Orthogonal and Orthonormal Sets VU

u1t 
 t 
u2 
u t 
 3
Ut =. 
 
. 
. 
 t 
um 
u1t  u1t u1 u1t u2 u1t u3 ... u1t um 
 t  t t t t

u2  u2u1 u2u2 u2u3 ... u2um 
u t  u t u u t u u t u ... u t u 
 3  3 1 3 2 3 3 3 m 
U t U =  .  [u1 u2 u3 . . . um ] =  . . . . 
   
.  . . . . 
.  . . . . 
 t  t 
um   umu1 umu2 umu3 ... umum 
t t t
 
As u .v = v u t

Therefore
u1.u1 u1.u2 u1.u3 ... u1.um 
u .u u .u u .u ... u .u 
 2 1 2 2 2 3 2 m 
u3 .u1 u3 .u2 u3 .u3 ... u3 .um 
 
UtU =  . . . . 
. . . . 
 
. . . . 
 u .u u .u u .u ... u .u 
 m 1 m 2 m 3 m m

Now, we come to prove the theorem.

First suppose that U t U = I , and we prove that columns of U are orthonormal.
Since, we assume that

© Virtual University Of Pakistan 196

18-Orthogonal and Orthonormal Sets VU

u1.u1 u1.u2 u1.u3 ... u1.um 

u .u u .u u .u ... u .u 
 2 1 2 2 2 3 2 m 
u3 .u1 u3 .u2 u3 .u3 ... u3 .um 
 
UtU =  . . . . 
. . . . 
 
. . . . 
 u .u u .u u .u ... u .u 
 m 1 m 2 m 3 m m

1 0 0 ... 0 
0 1 0 ... 0 
 
0 0 1 ... 0 
 
= . . . . 
. . . . 
 
. . . . 
0 0 0 ... 
 1 

Clearly, it can be seen that

ui . u j =0 for i ≠ j i, j =1, 2,...m
and
u=
i .u j 1=
for i j =
i, j 1, 2,...m

Therefore, columns of U are orthonormal.

Next suppose that the columns of U are orthonormal and we will show that U t U = I .
Since we assume that columns of U are orthonormal, so, we can write
ui . u j =0 for i ≠ j i, j =1, 2,...m
and
u=
i . u j 1=
for i j =
i, j 1, 2,...m
u1.u1 u1.u2 u1.u3 ... u1.um 
u .u u .u u .u ... u .u 
 2 1 2 2 2 3 2 m 
u3 .u1 u3 .u2 u3 .u3 ... u3 .um 
 
Hence, U t U =  . . . . 
. . . . 
 
. . . . 
 u .u u .u u .u ... u .u 
 m 1 m 2 m 3 m m

© Virtual University Of Pakistan 197

18-Orthogonal and Orthonormal Sets VU

1 0 0 ... 0
0 1 0 ... 0 

0 0 1 ... 0
 
= . . . . 
. . . . 
 
. . . . 
0 1 
 0 0 ...

That is
U t U = I.
Which is our required result.

Exercise
Prove that the following matrices have orthonormal columns using above theorem.
1 1 1
(1) 1 − 1
2 
2 −2 1 
(2) 1 2 2 

 2 1 − 2 
cos θ sin θ 
(3)  − sin θ cos θ 


Solution (1)
Let
1 1 1
U = 1 − 1
2  
1 1 1
Ut = 1 − 1
2 
Then
1 1 1 1 1
UtU = 1
2  − 1 
1 − 1
1 0
= =
0 1
I

UtU =I
Therefore, by the above theorem, U has orthonormal columns.

(2) And (3) are left for reader.

© Virtual University Of Pakistan 198

18-Orthogonal and Orthonormal Sets VU

Theorem

Let U be an m × n matrix with orthonormal columns, and let x and y be in R n . Then

a) Ux = x
b) (Ux).(Uy ) = x. y
=
c) (Ux ).(Uy ) 0= iff x. y 0

Example

1/ 2 2/3 
1    2
Let U = 1/ 2 − 2 / 3  and X =  
2
0 1/ 3 3 

Verify that Ux = x

Solution

Notice that U has orthonormal columns and

1/ 2 2/3 
1/ 2 1/ 2 0    1 0 
U TU  =  1/ 2 − 2 / 3  
− 2 / 3 1/ 3  0 1
2 / 3
0 1/ 3


1/ 2 2/3   3
1    2  
Ux = 1/ 2 − 2 / 3   = −1
2 
1/ 3    1 
3
0 
Ux = 9 + 1 + 1= 11
x = 2+9 = 11

© Virtual University Of Pakistan 199

19-Orthogonal Decomposition VU

Lecture 19
Orthogonal Decomposition
Objectives
The objectives of the lecture are to learn about:
• Orthogonal Decomposition Theorem.
• Best Approximation Theorem.

Orthogonal Projection

The orthogonal projection of a point in R 2 onto a line through the origin has an important
analogue in R n .
That is given a vector Y and a subspace W in R n , there is a vector ŷ in W such that

1) ŷ is the unique vector in W for which y − yˆ is orthogonal to W, and

2) ŷ is the unique vector in W closest to y.

0 ŷ

We observe that whenever a vector y is written as a linear combination of vectors

u1 , u2 ,..., un in a basis of R n , the terms in the sum for y can be grouped into two parts so
that y can be written as y= z1 + z2 , where z1 is a linear combination of some of the ui ’s,
and z2 is a linear combination of the rest of the ui ' s . This idea is particularly useful when
{u1 , u2 ,..., un } is an orthogonal basis.

Example 1

Let {u1 , u2 ,..., u5 } be an orthogonal basis for R 5 and let y= c1u1 + c2 u2 + ... + c5 u5 .
Consider the subspace W= Span{u 1 , u 2 } and write y as the sum of a vector z1 in W and a
vector z2 in W ⊥ .

Solution

©Virtual University Of Pakistan 200

19-Orthogonal Decomposition VU

Write
y = c1u1 + c2u2 + c3u3 + c4u4 + c5u5
  
z1 z2

where = z1 c1u1 + c2u2 is in Span of {u 1 , u 2 }, and z2 = c3u3 + c4u4 + c5u5 is in Span of {u 3 ,

u 4 , u 5 }.
To show that z 2 is in W ⊥ , it suffices to show that z 2 is orthogonal to the vectors in the
basis {u 1 , u 2 } for W. Using properties of the inner product, compute
z2 ⋅ u1 = (c3u3 + c4u4 + c5u5 ) ⋅ u1 = c3u3 ⋅ u1 + c4u4 ⋅ u1 + c5u5 ⋅ u1 = 0
since u 1 is orthogonal to u 3 , u 4 , and u 5 , a similar calculation shows that z2 ⋅ u2 = 0
Thus, z2 is in W ⊥ .

Orthogonal decomposition theorem

Let W be a subspace of R n , then each y in R n can be written uniquely in the form
y= yˆ + z
∧ ⊥
Where y ∈W and z ∈W
Furthermore, if {u , u ,..., u }
1 2 p
is any orthogonal basis for W , then
y. u j
y ∧ = c1u1 + c2u2 + ... + cnun , where c j =
u j .u j
Proof

z = y - yˆ y
• •

• •
0 yˆ = projwY

Fig: Orthogonal projection of y on to W.

Firstly, we show that y ∧ ∈W , z ∈ W ⊥ . Then we will show that =

y y ∧ + z can be
represented in a unique way.

Suppose W is a subspace of R n and let {u , u ,..., u }

1 2 p be an orthogonal basis for
W.
y. u j
y ∧ c1u1 + c2u2 ...c p u p where c j =
As =
u j .u j

©Virtual University Of Pakistan 201

19-Orthogonal Decomposition VU

Since {u1 , u2 ,..., u p } is the basis for W and y ∧ is written as a linear combination of
these basis vectors. Therefore, by definition of basis y ∧ ∈W .
Now, we will show that z =y − y ∧ ∈ W ⊥ . For this it is sufficient to show that z ⊥ u j for
each j = 1, 2,... p.
Let u1 ∈W be an arbitrary vector.
z . u1= ( y − y ∧ ). u1
= y . u1 − y ∧ . u1
= y . u1 − (c1u1 + c2u2 + ... + c p u p ). u1
=
y . u1 − c1 (u1 . u1 ) − c2 (u2 . u1 ) − ...c p (u p . u1 )
=
y . u1 − c1 (u1 . u1 ) where u j . u1 =
0, j =
2,3,... p
y . u1
= y . u1 − (u1 . u1 )
u1 . u1
= y . u1 − y . u1
=0
Therefore, z ⊥ u1.
Since u1 is an arbitrary vector, therefore z ⊥ u j for j = 1, 2,... p.
Hence by definition of W ⊥ , z ∈W ⊥
Now, we must show that = y y ∧ + z is unique by contradiction.
y y ∧ + z and =
Let = y y1∧ + z1 , where y ∧ , y1∧ ∈ W and z , z1 ∈ W ⊥ ,
also z ≠ z1 and y ∧ ≠ y1∧ . Since above representations for y are equal, that is
y ∧ + z = y1∧ + z1
⇒ y ∧ − y1∧ =z1 − z
Let
s y ∧ − y1∧
=
Then
s= z1 − z
Since W is a subspace, therefore, by closure property
s = y ∧ − y1∧ ∈ W
Furthermore, W ⊥ is also a subspace, therefore by closure property
s = z1 − z ∈ w⊥
Since
s ∈ W and s ∈ W ⊥ . Therefore by definition s ⊥ s
That is s. s=0
Therefore
s = y ∧ − y1∧ = 0
⇒ y∧ =
y1∧

©Virtual University Of Pakistan 202

19-Orthogonal Decomposition VU

Also
z1 = z
This shows that representation is unique.

Example

 2 − 2 1 
Let u1 =  5  , u2 =  1  , and y =  2 
   
 −1  1   3

Observe that {u1 , u2 } is an orthogonal basis for W=span {u1 , u2 } , write y as the sum of a
vector in W and a vector orthogonal to W.

Solution

Since y ∧ ∈ W , therefore y ∧ can be written as following:

y ∧ c1u1 + c2u2
=
y.u1 y.u2
= u1 + u2
u1.u1 u2 .u2
 2  − 2  2  − 2
9   3   9   15  
= 5 + 1 = 5 + 1
30   6   30   30  
 −1  1   −1  1 
 − 2 / 5  −2 
= = 2  10 
1
5 
 1/ 5  1 
 1  − 2 / 5 7/5 
y − y =  2  −  2  =
∧  0 
 
 3   1/ 5   14 / 5
1 
= 7 / 5  0 
 2 
Above theorem ensures that y − y ∧ is in W ⊥ .
0 and ( y − y ∧ ).u2 =
You can also verify by ( y − y ∧ ).u1 = 0.

The desired decomposition of y is

©Virtual University Of Pakistan 203

 19-Orthogonal Decomposition VU

 1  − 2 / 5  7 / 5 
y =
= 2   2 + 0 
   
 3   1/ 5   14 / 5

Example
1  4
Let W=span {u1 , u2 } , where u1 =  −3 and u2 =  2 
 

 2   1 
 2
Decompose y =  −2  into two vectors; one in W and one in W ⊥ . Also verify that these
 5 
two vectors are orthogonal.
Solution

Let y ∧ ∈ W and z =y − y ∧ ∈ W ⊥ .
Since y ∧ ∈ W , therefore y ∧ can be written as following:
y ∧ c1u1 + c2u2
=
y.u1 y.u2
= u1 + u2
u1.u1 u2 .u2
1  4
9  3
=  −3 +  2 
7 7
 2   1 
 3
y =  −3
∧

 3
Now
 2   3
z = y − y =  −2  −  −3
∧

 5   3
 −1 
z =  1 
 2 
Now we show that z ⊥ y ∧ , i.e. z. y ∧ = 0

©Virtual University Of Pakistan 204

19-Orthogonal Decomposition VU

 −1   3
=
z. y ∧ 1  .  −3
   
 2   3
=0
Therefore z ⊥ y ∧ .
Exercise
1  3
Let W=span {u1 , u2 } , where u1 =  0  and u2 =  1 
 
 −3  1 
6 
Write y =  −8 as a sum of two vectors; one in W and one in W ⊥ . Also verify that these

12 
two vectors are orthogonal.

Best Approximation Theorem

Let W is a finite dimensional subspace of an inner product space V and y is

any vector in V. The best approximation to y from W is then Pr ojwy , i.e for every w (that
is not Pr ojwy ) in W, we have
y − Pr ojwy < y − w .

Example
1  4  2
Let W=span {u1 , u2 } , where u1 =  −3 , u2 =  2  and y =
   
 −2  . Then using above
 2   1   5 
theorem, find the distance from y to W.

Solution
Using above theorem the distance from y to W is calculated using the following formula
y − Pr ojwy =y − y∧
Since, we have already calculated
 2   3  −1 
y − y =  −2  −  −3 =  1 
∧

 5   3  2 

So y − y∧ =6
Example

©Virtual University Of Pakistan 205

19-Orthogonal Decomposition VU

The distance from a point y in R n to a subspace W is defined as the distance from y to the
nearest point in W.
Find the distance from y to W =span {u1 , u2 } , where

 −1 5 1

y=    2
 −5 , u1 =
 −2  , u2 =
 
10   1   −1

By the Best Approximation Theorem, the distance from y to W is y − yˆ , where

ŷ = proj w y. Since, {u 1 , u 2 } is an orthogonal basis for W, we have
5  1   −1
15 −21 1  7   
yˆ = u1 + u2 =  −2  −  2  = −8
30 6 2 2
 1   −1  4 
 −1  −1 0 
y − yˆ =  −5 −  −8 = 3
 
10   4  6 
y − yˆ = 32 + 62 = 45
2

The distance from y to W is 45 = 3 5.

Theorem

If {u1 , u2 ,..., u p } is an orthonormal basis for a subspace W of R n , then

Pr ojw y =( y.u1 )u1 + ( y.u2 )u2 + ...( y.u p )u p
If U = [u1 u2 .... u p ]
then=
Pr ojwy UU T y ∀y in R n

Example

 −7   −1  −9 
Let u1  =
= 1  , u2 = 
 1 , y
1
 
 4   −2   6 

and W =span {u1 , u2 } . Use the fact that u 1 and u 2 are orthogonal to compute Pr ojw y .

Solution

©Virtual University Of Pakistan 206

19-Orthogonal Decomposition VU

y.u1 y.u2
Pr=
ojw y u1 + u2
u1.u1 u2 .u2
88 −2
= u1 + u2
66 6

 −7   − 1  − 9 
4  1   
= 1  −  1  = 1  =y
3 3
 4   2   6 

In this case,y happens to be a linear combination of u 1 and u 2 . So y is in W. The closest

point in W to y is y itself.

©Virtual University Of Pakistan 207

 20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

Lecture 20

Orthogonal basis, Gram-Schmidt Process, Orthonormal basis

Example 1

Let W = Span {x 1 , x 2 }, where

3 1 
= 
x1 = 6  and x2  2 

0   2 

Find the orthogonal basis {v 1 , v 2 } for W.

Solution

Let P be a projection of x 2 on to x 1 . The component of x 2 orthogonal to x 1 is x 2 – P,

which is in W as it is formed from x 2 and a multiple of x 1 .

Let v 1 = x 1 and compute

1  3 0 
v2 =x2 − P = x2 −
x2 .v1   15    
v1 =  2  − 6  =  0 
v1.v1 45
 2  0   2 

Thus, {v 1 , v 2 } is an orthogonal set of nonzero vectors in W , dim W = 2 and {v 1 , v 2 } is a

basis of W.

Example 2

For the given basis of a subspace W = Span {x 1 , x 2 },

0 5
=x1 = 4  and x2 6
 
 2   −7 

Find the orthogonal basis {v 1 , v 2 } for W.

Solution

Set v 1 = x 1 and compute

©Virtual University Of Pakistan 208

20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

5 0
v2 = x2 −
x2 .v1
v1 =−   10  
v1.v1  6  20  4 
-7   2 
5 0  5 
  1   
= 6  −  4  = 4 
2
-7   2   −8

0  5  
 
Thus, an orthogonal basis for W is   4  ,  4  
   
  2   −8 
    

Theorem

Given a basis {x 1 , …, x p } for a subspace W of R n . Define

x2 .v1
v=
1 x1 v=
2 x2 − v1
v1.v1
x3 .v1 x .v
v3 =
x3 − v1 − 3 2 v2
v1.v1 v2 .v2

x p .v1 x p .v2 x p .v p -1
vp = xp − v1 − v2 − ... − v p -1
v1.v1 v2 .v2 v p -1.v p -1

Then {v 1 , …,v p } is an orthogonal basis for W.

In addition

Span {v 1 , …, v k }= Span {x 1 ,…, x k } for 1 ≤ k ≤ p

Example 3

The following vectors {x 1 , x 2 , x 3 } are linearly independent

1 0  0 
1 1  0 
=x1 =, x = , x  
1 2 1  3 1 
    
1 1  1 

Construct an orthogonal basis for W by Gram-Schmidt Process.

©Virtual University Of Pakistan 209

20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

Solution

To construct orthogonal basis we have to perform the following steps.

Step 1 Let v 1 = x 1

Step 2

x2 .v1
Let v2= x2 − v1
v1.v1

0  1  −3 / 4   −3
1      
  3 1  1/ 4   1 
Since v2 =x1 = − = =
1  4 1  1/ 4   1 
      
1  1  1/ 4   1 

Step 3

0   1  −3  0  
0   1    
 x3 .v1 x3 .v2     2   2  1   2 / 3 
v3 =
x3 −  v1 + v2  = − + =
 v1.v1 v2 .v2  1   4 1 12  1   2 / 3 
       
1   1  1   2 / 3 

0   0   0 
0   2 / 3  −2 / 3
 −
v3 = = 
1   2 / 3  1/ 3 
     
1   2 / 3  1/ 3 

Thus, {v 1 , v 2, v 3 } is an orthogonal set of nonzero vectors in W.

Example 4

Find an orthogonal basis for the column space of the following matrix by Gram-Schmidt
Process.

©Virtual University Of Pakistan 210

20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

 −1 6 6
3 − 8 3 

1 − 2 6 
 
1 − 4 − 3

Solution
Name the columns of the above matrix as x 1 , x 2 , x 3 and perform the Gram-Schmidt
Process on these vectors.
 −1 6  6 
3   −8  3 
= 
x1 =  ,x =  ,x  
1  2  −2  3 6 
     
1   −4   −3

Set v 1 = x 1

x2 .v1
v=
2 x2 − v1
v1.v1
6   −1 3 
 −8  3  1 
=  − (−3)  =   
 −2  1  1 
     
 −4  1   −1

x3 .v1 x .v
v3 =
x3 − v1 − 3 2 v2
v1.v1 v2 .v2
6   −1 3   −1
3  3     −1
1 5 1
=  −   −   = 
6  2 1  2 1  3 
       
 −3 1   −1  −1
  −1 3   −1 
      
 3  1   −1 
Thus, orthogonal basis is  , , 
 1  1  3  
 1   −1  −1 

©Virtual University Of Pakistan 211

 20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

Example 5

Find the orthonormal basis of the subspace spanned by the following vectors.

3 0
=x1 =6  , x2 0
 
0   2 

Solution

Since from example # 1, we have

3  0 
=v1 =6  , v20 
 
0   2 
Orthonormal Basis

3 1/ 5  0 
1    
v2 0 
1 1
=u1 = v1 = 6 2 / 5 =
 , u2 =
v1 45    0  v2
0  1 
 

Example 6

Find the orthonormal basis of the subspace spanned by the following vectors.

2 4
   −1
x1 =
 −5 and x 2 =
 
 1   2 

Solution

Firstly we find v 1 and v 2 by Gram-Schmidt Process as

x2 .v1
v=
2 x2= x2 − v1
v1.v1
Set v1 = x 1 4 2 4 2
  15     1  
v2 = −1 −  −5 = −1 −  −5
30 2
 2   1   2   1 

©Virtual University Of Pakistan 212

20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

 4   1  6 
=  −1 −  −5 / 2  =  3
 2   1/ 2   3
1 1
Now = v2 = and since=
v1 30
54 3 6

Thus , the orthonormal basis for W is

  2 / 30   2 / 6  
 v1 v 2       
 , =   −5 / 30  , 1/ 6  
 v1 v 2     
  1/ 30  1/ 6  

Theorem

If A is an m x n matrix with linearly independent columns, then A can be factored as

A = QR, where Q is an m x n matrix whose columns form an orthonormal basis for Col A
and R is an n x n upper triangular invertible matrix with positive entries on its diagonal.

Example 7
1 2 5
 −1 1 − 4 

Find a QR factorization of matrix A = −1 4 −3
 
1 −4 7
1 2 1 

Solution

Firstly find the orthonormal basis by applying Gram Schmidt process on the columns of
A. We get the following matrix Q.
1/ 5 1/ 2 1/ 2 
 
 −1/ 5 0 0 
 
Q =  −1/ 5 1/ 2 1/ 2 
 
1/ 5 − 1/ 2 1/ 2 
1/ 5 1/ 2 − 1/ 2 


©Virtual University Of Pakistan 213

 20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

 5 − 5 4 5
 
Now R = Q
T
A = 0 6 −2
0 0 4 

Verify that A=QR.

Theorem

If {u1 , ..., u p } is an orthonormal basis for a subspace W of R n , then

projw y = ( y ⋅ u1 )u1 + ( y ⋅ u2 )u2 +  + ( y ⋅ u p )u p
If U = [u1 u2 ... u p ],
=
then proj w
y UU T y ∀ y in R n

The Orthogonal Decomposition Theorem

Let W be a subspace of R n Then each y in R n can be written uniquely in the form

y= yˆ + z
where W ⊥ is in W and z is in

In fact, if {u 1 , …, u p } is any orthogonal basis of W, then

y ⋅ u1 y ⋅up
=yˆ u1 +  + up
u1 ⋅ u1 up ⋅up
and z = y – ŷ . The vector ŷ is called the orthogonal projection of y onto W and is
often written as proj w y.

Best Approximation Theorem

Let W be a subspace of R n , y is any vector in R n and ŷ the orthogonal projection of y

onto W. Then ŷ is the closest point in W to y, in the sense that

for all v in W distinct from ŷ .

y − yˆ < y − v

The vector y ^ in this theorem is called the best approximation to y by elements of W.

©Virtual University Of Pakistan 214

 20-Othogonal basis, Gram-Schmidt process, Orthonormal basis VU

Exercise 1

Let W = Span {x1, x2}, where

1  1/ 3 
=x =1 and x  1/ 3 
 
1 2
1  −2 / 3

Construct an orthonormal basis for W.

Exercise 2

Find an orthogonal basis for the column space of the following matrix by Gram-Schmidt
Process.

3 −5 1
1 1 1 

 −1 −5 − 2
 
3 −7 8 

Exercise 3

Find a QR factorization of

1 3 5
 1 
 −1 −3
A = 0 2 3
 
1 5 2
1 5 8 

©Virtual University Of Pakistan 215

21-Least Square Solution VU

Lecture 21
Least Square Solution
Best Approximation Theorem

Let W be a subspace of R n , y be any vector in R n and ŷ the orthogonal projection of y

onto W. Then ŷ is the closest point in W to y, in the sense that y − yˆ < y − v
for all v in W distinct from ŷ

The vector ŷ in this theorem is called the best approximation to y by elements of W.

Least-squares solution

The most important aspect of the least-squares problem is that no matter what “x” we
select, the vector Ax will necessarily be in the column space Col A. So we seek an x that
makes Ax the closest point in Col A to b. Of course, if b happens to be in Col A, then b is
Ax for some x and such an x is a “least-squares solution.”

Solution of the General Least-Squares Problem

Given A and b as above, apply the Best Approximation Theorem stated above to the
subspace Col A. Let bˆ = projCol Ab
Since b̂ is in the column space of A, the equation Ax = bˆ is consistent, and there is an x̂
in Rn such that
Axˆ = bˆ (1)
Since b̂ is the closest point in Col A to b, a vector x̂ is a least-squares solution of Ax = b
if and only if x̂ satisfies Axˆ = bˆ . Such an x̂ in Rn is a list of weights that will build b̂ out
of the columns of A.

Normal equations for x̂

Suppose that x̂ satisfies Axˆ = bˆ. By the Orthogonal Decomposition Theorem the
projection b̂ has the property that b − bˆ is orthogonal to Col A, so b − Axˆ is orthogonal
0, and aTj (b − Axˆ ) =
to each column of A. If a j is any column of A, then a j ⋅ (b − Axˆ ) = 0.
Since each a Tj is a row of AT,
AT (b − Axˆ ) =
0 (2)
AT b − AT Axˆ =0
A Axˆ = A b
T T
(3)
The matrix equation (3) represents a system of linear equations commonly referred to as
the normal equations for x̂.

© Virtual University Of Pakistan 217

21-Least Square Solution VU

Since set of least-squares solutions is nonempty and any such x̂ satisfies the normal
equations. Conversely, suppose that x̂ satisfies AT Axˆ = AT b. Then it satisfy that b − Axˆ
is orthogonal to the rows of AT and hence is orthogonal to the columns of A. Since the
columns of A span Col A, the vector b − Axˆ is orthogonal to all of Col A. Hence the
equation b = Axˆ + (b − Axˆ ) is a decomposition of b into the sum of a vector in Col A and
a vector orthogonal to Col A. By the uniqueness of the orthogonal decomposition, Axˆ
must be the orthogonal projection of b onto Col A. That is, Axˆ = bˆ and x̂ is a least-
squares solution.

Definition

If A is m x n and b is in R n , a least-squares solution of Ax = b is an I x̂ in R n such that

b − Axˆ ≤ b − Ax ∀ x ∈ Rn

Theorem

The set of least-squares solutions of Ax = b coincides with the nonempty set of solutions
of the normal equations
AT Axˆ = AT b
Example 1

Find the least squares solution and its error from the following matrices,

4 0 2
= 
A =0 2  , b 0
 
1 1  11

Solution

Firstly we find
4 0
 4 0 1   17 1 and
=A A =  0 T
2   
 0 2 1 1 1   1 5
 
2
 4 0 1   19 
=A b =
T
0  
 0 2 1 11 11
 

17 1  x1  19 
Then the equation AT Axˆ = AT b becomes    =  
 1 5  x2  11

© Virtual University Of Pakistan 218

21-Least Square Solution VU

Row operations can be used to solve this system, but since ATA is invertible and 2 × 2 , it
1  5 −1
is probably faster to compute ( AT A) −1 = 
84  −1 17 

1  5 −1 19  1  84  1 
) −1 AT b
Therefore, xˆ = ( AT A= = =
84  −1 17  11 84 168  2 
4 0 2
 
Now again as A=  0 2  , b =  0 
1 1  11
4 0 4
1 
Then =Axˆ =0 2    4
 
1 1   
2
 3 
 2   4   −2 
b − Axˆ = 0  − 4 =  
Hence      −4 
11  3   8 
So b − Axˆ = (−2) 2 + (−4) 2 + 82 = 84
The least-squares error is 84. For any x in R2, the distance between b and the vector Ax
is at least 84.

Example 2

Find the general least-squares solution of Ax = b in the form of a free variable with

1 1 0 0   −3
1 1 0 0   −1
   
1 0 1 0  0
=A =  ,b  
1 0 1 0  2
1 0 0 1  5
   
1 0 0 1  1

Solution

© Virtual University Of Pakistan 219

 21-Least Square Solution VU

1 1 0 0 
 
1 1 1 1 1 1  1 1 0 0  6 2 2 2
1 1 0 0 0 0  1 0 1 0   0 
=Firstly we find, A A =
T    2 2 0
and
0 0 1 1 0 0  1 0 1 0  2 0 2 0
   
0 0 0 0 1 1  1 0 0 1  2 0 0 2
 
1 0 0 1 
 −3
 
1 1 1 1 1 1   −1  4 
1 1 0 0 0 0   0   −4 
A b =
T    
0 0 1 1 0 0   2   2 
   
0 0 0 0 1 1   5   6 
 
1
Then augmented matrix for AT Axˆ = AT b is

6 2 2 2 4  1 0 0 1 3
2 2 0 0 −4  0 1 0 −1 −5
 
2 0 2 0 2  0 0 1 −1 −2 
   
2 0 0 2 6  0 0 0 0 0

The general solution is x1 =3 − x4 , x2 =−5 + x4 , x3 =−2 + x4 , and x 4 is free.

So the general least-squares solution of Ax = b has the form
3  −1
 −5 1
=xˆ   + x4  
 −2  1
   
0 1

Theorem

The matrix AT A is invertible iff the columns of A are linearly independent. In this case,
the equation Ax = b has only one least-squares solution x̂ , and it is given by
xˆ = ( AT A) −1 AT b
Example 3
Find the least squares solution to the following system of equations.
2 4 6 0 
1 − 3 0   x1   
= A =   x 2  , b 1 
7 1 4    −2 
   x3   
1 0 5 4 
Solution

© Virtual University Of Pakistan 220

 21-Least Square Solution VU

As
2 1 7 1
=  0 
A 4 − 3
T
1
6 0 4 5
2 4 6
2 1 7 1  55 12 45 
 4 − 3 1 0  1 − 3 0  
AT A =
  7 1 =12 26 28
4
6 0 4 5    45 28 77 
1 0 5 
Now

0 
2 1 7 1    −9 
 4 − 3 1 0  1  =
AT b =  
   −2   −5 
6 0 4 5   12 
4 

As AT Axˆ = AT b

55 12 45   x1   −9 
12 26 28  x 2  =  −5 
  
 45 28 77   x3  12 

 x1   −.676 
 x2 =  −.776 
   
 x3  .834 

Example 4

Compute the least square error for the solution of the following equation
2 4 6 0 
1 − 3 0   x1  1 
=A =   
x2 , b  
7 1 4    −2 
   x3   
1 0 5 4 
.

Solution

© Virtual University Of Pakistan 221

 21-Least Square Solution VU

2 4 6 2 4 6
1 − 3 0   x1  1 − 3  −.676 
  x2 0  
−.776 

=Ax  = 
7 1 4   7 1 4
   x3    .834 
1 0 5  1 0 5 
0.548 
 1.652 
Ax =  
 −2.172 
 
3.494 

As least square error

∈= b − Ax

is as small as possible, or in other words is smaller than all other possible choices of x.

0  0.548   −0.548
1  1.652  0.652 
b − Axˆ=   −  =  
 −2   −2.172  0.172 
As      
 4  3.494  1.494 

∈ =∈12 + ∈2 2 + ∈32 + ∈4 2
2

Thus, least square error is

∈= b − Ax =
(−0.548) 2 + (0.652) 2 + (0.172) 2 + (1.494) 2

=0.3003+0.4251+.02958+2.23=2.987

Theorem

Given an m x n matrix A with linearly independent columns.Let A = QR be a QR

factorization of A ,then for each b in Rm, the equation Ax = b has a unique least-squares
solution, given by
xˆ = R -1QT b

Example 1

© Virtual University Of Pakistan 222

21-Least Square Solution VU

1 3 5 3 
1 1 0  5 
=
Find the least square solution for A = ,b  
1 1 2 7 
   
1 3 3  −3

Solution

First of all we find QR factorization of the given matrix A. For this we have to find out
orthonormal basis for the column space of A by applying Gram-Schmidt Process, we get
the matrix of orthonormal basis Q,

1
2 12 12
1
2 −1 2 −1 2 
Q= And
1
2 −1 2 1 2 
 
2 1 2 −1 2
1
1 3 5
1 2 1 2 1 2  2 4 5
0  
12

R = Q A = 1 2 −1 2 −1 2 1 2 
T  1 1
= 0 2 3
  1 1 2
1 2 −1 2 1 2 −1 2     0 0 2 
1 3 3 

3
1 2 1 2 12 1 2    6 
1 2 −1 2 −1 2 1 2   5  =
QT b =  
Then    7   −6 
1 2 −1 2 1 2 −1 2     4 
 −3

 2 4 5   x1  6
The least-squares solution x̂ satisfies Rxˆ = Q b; that is,  0 2 3   x2  =
T  
 −6 
 0 0 2   x3   4 

10 
This equation is solved easily and yields xˆ =  −6  .
 2 
Example 2

Find the least squares solution Rxˆ = Q b

T
to the given matrices,

© Virtual University Of Pakistan 223

21-Least Square Solution VU

3 1 1 
= 
A =6 2  , b 2
 
0 2  3 
Solution
First of all we find QR factorization of the given matrix A. Thus, we
have to make
Orthonormal basis by applying Gram Schmidt process on the columns of A,

Let v 1 = x 1
1  3 0 
v1 =  2  − 6  =  0 
x2 .v1 15
v2 =x2 − P = x2 −
v1.v1 45
 2  0   2 

Thus , the orthonormal basis are

 1  0  
 v1 v2       
 ,  =   2  , 0  
 v1 v2    0  1  
    

Thus

1 0
 1 0
0  and QT 
2
=Q =
1
2
0 0
0 1 
3 1
1 0 
Now R = Q A =  2 
T 2
1 
6
0 0
0 2 
1 
15 5 1 2 0    5
=  = And Q b =
T

2  1   3
2
0 0 0 3 

Thus, least squares solution of Rxˆ = QT b is

15 5 5 
0  xˆ =  
 2 3 
1 
xˆ =  
1.8

© Virtual University Of Pakistan 224

 21-Least Square Solution VU

Exercise 1

Find a least-squares solution of Ax = b for

1 −6   −1
1 −2  2
=A = , b  
1 1  1
   
1 7  6

Exercise 2

Find the least-squares solution and its error of Ax = b for

1 3 5  3
1 1 0  5
=A = , b  
1 1 2  7
   
1 3 3   −3

Exercise 3

Find the least squares solution Rxˆ = QT b to the given matrices,

 2 1  −5

A=  8
 −2 0 , b =
 
 2 3   1 

© Virtual University Of Pakistan 225

22- Eigen Value Problems VU

Lecture 22
Eigen Value Problems

Let [A] be an n x n square matrix. Suppose, there exists a scalar and a vector

X = ( x1 x2  xn )T
such that
[ A]( X ) = λ ( X )
d ax
(e ) = a (e ax )
dx
d2
2
(sin ax) = −a 2 (sin ax)
dx
Then λ is the eigen value and X is the corresponding eigenvector of the matrix [A].
We can also write it as [ A − λ I ]( X ) =
(O)
This represents a set of n homogeneous equations possessing non-trivial solution,
provided
A − λI = 0
This determinant, on expansion, gives an n-th degree polynomial which is called
characteristic polynomial of [A], which has n roots. Corresponding to each root, we can
solve these equations in principle, and determine a vector called eigenvector.
Finding the roots of the characteristic equation is laborious. Hence, we look for better
methods suitable from the point of view of computation. Depending upon the type of
matrix [A] and on what one is looking for, various numerical methods are available.

Power Method and Jacobi’s Method

Note!
We shall consider only real and real-symmetric matrices and discuss power and Jacobi’s
methods

Power Method

To compute the largest eigen value and the corresponding eigenvector of the system
[ A]( X ) = λ ( X )
where [A] is a real, symmetric or un-symmetric matrix, the power method is widely used
in practice.

Procedure
Step 1: Choose the initial vector such that the largest element is unity.

Step 2: The normalized vector v (0) is pre-multiplied by the matrix [A].

Step 3:The resultant vector is again normalized.

© Copyright Virtual University of Pakistan 226

 22- Eigen Value Problems VU

Step 4: This process of iteration is continued and the new normalized vector is repeatedly
pre-multiplied by the matrix [A] until the required accuracy is obtained.
At this point, the result looks like
= A]v ( k −1) qk v ( k )
u (k ) [ =
Here, qk is the desired largest eigen value and v ( k ) is the corresponding eigenvector.

Example
Find the eigen value of largest modulus, and the associated eigenvector of the matrix by
power method
2 3 2
[ A] =  4 3 5 
 3 2 9 
Solution
We choose an initial vector υ (0)
as (1,1,1)T .
Then, compute first iteration
 2 3 2  1  7 
   
=u (1)
A]v  4 3 5  1 12 
[= (0) 

 3 2 9  1 14 
Now we normalize the resultant vector to get
 12 
6
=
u (1) 14=  7  q1v
(1)

1
 
The second iteration gives,
 2 3 2   12   397 
   
= A]v (1)  4 3 5   76   677 
u (2) [ =
   171 
 3 2 9   1   14 
 0.456140 
 
= 12.2143
=  0.783626  q2 v
(2)

 1.0 
 
Continuing this procedure, the third and subsequent iterations are given in the following
slides

 2 3 2   0.456140 
=
u [=
(3)
A]v  4 3 5   0.783626 
(2)
  
 3 2 9   1.0 

© Copyright Virtual University of Pakistan 227

 22- Eigen Value Problems VU

 5.263158   0.44096 
   
= = 9.175438  11.935672 =
 0.776874  q3v
(3)

11.935672   1.0 
   
 5.18814 
 
=
u (4)
[=
A]v (3)
 9.07006 
11.86036 
 
 0.437435 
 
= 11.8636
=  0.764737  q4 v
(4)

 1.0 
 
 5.16908 
 
=
u (5)
[=
A]v (4)
 9.04395 
11.84178 
 
 0.436512 
 
= 11.84178
=  0.763732  q5v
(5)

 1.0 
 

After rounding-off, the largest eigen value and the corresponding eigenvector as accurate
to two decimals are
 0.44 
λ = 11.84 ( X ) =  0.76 
 1.00 
 
Example
Find the first three iterations of the power method of the given matrix

 7 6 −3
 −12 −20 24 
 
 −6 −12 16 
Solution

© Copyright Virtual University of Pakistan 228

22- Eigen Value Problems VU

 7 6 −3
 −12 −20 24 
 
 −6 −12 16 
we choose finitial vector as v (0) =(1,1,1)t
first iteration
 7 6 −3 1  7 + 6 − 3  10 
u =[ A] v = −12 −20 24  1 = −12 − 20 + 24  = −8
(1) 
(0)

 −6 −12 16  1  −6 − 12 + 16   −2 
10   1 
by diagonali sin g  −8 u= 10  −0.8=
  (1)  q v (1)
 1

 −2   −0.2 
sec ond iteration
 7 6 −3  1   7 − 4.8 + 0.6   2.8 

u = [ A] v =  −12 −20
(2) (1)
24   −0.8 =
 −12 + 16 − 4.8 =  −0.8
   
 −6 −12 16   −0.2 
 −6 + 9.6 − 3.2   0.4 
 2.8   1 
by diagonali sin g  −0.8 u= 2.8  −0.2857
(1)
=  q v (2)
 2

 0.4   0.1428 

third iteration
 7 6 −3  1   7 − 1.7142 − 0.4284   4.8574 
u = [ A] v =  −12 −20 24   −0.2857  =
(3) 
(2)  −12 + 5.714 + 3.4272  =
 
 −2.8588
 
 −6 −12 16   0.1428   −6 + 3.4284 + 2.2848  −0.2868
now daigonali sin g
 4.8574   1 
 −2.8588 now normali sin g 4.8574  −0.5885
 
 −0.2868  −0.0590 

Example
Find the first three iteration of the power method applied on the following matrices

 1 −1 0 
 −2 4 −2  use x 0 = (−1, 2,1)t
 
 0 −1 2 

© Copyright Virtual University of Pakistan 229

 22- Eigen Value Problems VU

Solution

 1 −1 0 
 −2 4 −2  USE x (0) = (−1, 2,1)t
 
 0 −1 2 
1st iterations
 1 −1 0   −1  −1 − 2 + 0   −3
u = [ Α ] x =  −2 4 −2   2  =  2 + 8 − 2  =  8 
(1) (0)

 0 −1 2   1   0 − 2 + 2   0 
now we normalize the resul tan t vector to get
 −3 
 −3 8
=   
=
u (1)
8 8 =1  q1 x (1)
 0  0
 
 

 −3 
 −3   − 1 + 0 
 1 −1 0   8   8   −1.375
    6
u = [ Α ] x =  −2 4 −2   1  =
(2) (1)
+ 4 + 0 =  4.75 

8 
 0 −1 2   0     −1 
   −1  
  
 
 −1.375  −0.28947 
=  4.75  4.75 
 
=
(2)
u 1 
 −1   −0.2152 
 1 −1 0   −0.28947   −1.28947   −0.25789 
u (3)
= [ Α] x (2)
=  −2 4 −2   1 =

 4.99998  = 4.99998 
   1 

 0 −1 2   −0.2152   −1.42104   −0.28420 

Exercise
Find the largest eigen value and the corresponding eigen vector by power method after
fourth iteration starting with the initial vector υ (0) = (0, 0,1)T

© Copyright Virtual University of Pakistan 230

 22- Eigen Value Problems VU

1 −3 2 
=[ A]  4 4 −1
 6 3 5 

Let
λ1 , λ2 , , λn
be the distinct eigen values of an n x n matrix [A], such that λ1 > λ2 >  > λn and
suppose v1 , v2 ,  , vn are the corresponding eigen vectors
Power method is applicable if the above eigen values are real and distinct, and hence, the
corresponding eigenvectors are linearly independent.
Then, any eigenvector v in the space spanned by the eigenvectors v1 , v2 ,  , vn
can be written as their linear combination v = c1v1 + c2 v2 +  + cn vn

Pre-multiplying by A and substituting

=Av1 λ=
1v1 , Av2 λ2 v2 , =
 Avn λn vn

We get
 λ λ 
Av λ1  c1v1 + c2 2 v2 +  + cn n vn 
=
 λ1 λ1 
Again, pre-multiplying by A and simplifying, we obtain
  λ2 
2
 λn  
2

=
A 2
v λ1 c1v1 + c2   v2 +  + cn   vn 
2

  λ1   λ1  
Similarly, we have

  λ2 
r
 λn  
r

=
A r
v λ c1v1 + c2   v2 +  + cn   vn 
r

 λ1   λ1  
1

and
  λ2 
r +1
 λn 
r +1

=A v (λ1 ) c1v1 + c2   v2 +  + cn   vn 
r +1 r +1

  λ1   λ1  
Now, the eigen value λ1

can be computed as the limit of the ratio of the corresponding components of Ar v

and Ar +1v.
That is,

© Copyright Virtual University of Pakistan 231

22- Eigen Value Problems VU

r +1
λ1r +1 ( A v) p
λ1
= = Lt , =p 1, 2,  , n
λ1 r r →∞ ( Ar v )
p

Here, the index p stands for the p-th component in the corresponding vector
Sometimes, we may be interested in finding the least eigen value and the corresponding
eigenvector.
In that case, we proceed as follows.
We note that [ A]( X ) = λ ( X ).
Pre-multiplying by [ A−1 ] , we get
[ A−=
1
A−1 ]λ ( X ) λ[ A−1 ]( X )
][ A]( X ) [ =
Which can be rewritten as
1
[ A−1 ]( X ) = ( X )
λ
which shows that the inverse matrix has a set of eigen values which are the reciprocals of
the eigen values of [A].
Thus, for finding the eigen value of the least magnitude of the matrix [A], we have to
apply power method to the inverse of [A].

© Copyright Virtual University of Pakistan 232

 23- Jacobi’s Method VU

Lecture 23
Jacobi’s Method

Definition
An n x n matrix [A] is said to be orthogonal if
[ A]T [ A] = [ I ],
i.e.[ A]T = [ A]−1
In order to compute all the eigen values and the corresponding eigenvectors of a real symmetric matrix,
Jacobi’s method is highly recommended. It is based on an important property from matrix theory, which
states
that, if [A] is an n x n real symmetric matrix, its eigen values are real, and there exists an orthogonal matrix
[S] such that the diagonal matrix D is
[ S −1 ][ A][ S ]

This digitalization can be carried out by applying a series of orthogonal transformations

S1 , S 2 ,..., S n ,
Let A be an n x n real symmetric matrix. Suppose aij be numerically the largest element amongst the off-
diagonal elements of A. We construct an orthogonal matrix S1 defined as

sij = − sin θ , s ji =
sin θ ,
= =
sii cos θ , s jj cos θ
While each of the remaining off-diagonal elements are zero, the remaining diagonal elements are assumed
to be unity. Thus, we construct S1 as under
i-th column j -th column
↓ ↓
1 0  0  0  0
0 1  0  0  0 

    
 
0 0  cos θ  − sin θ  0  ← i-th row
S1 = 
    
 
0 0  sin θ  cos θ  0  ← j -th row
    
 
0 0  0  0  1 
Where cos θ , − sin θ ,sin θ and cos ϑ are inserted in (i, i ), (i, j ), ( j , i ), ( j , j ) − th positions respectively,
and elsewhere it is identical with a unit matrix.
Now, we compute
−1
=D1 S= 1 AS1 S1T AS1
Since S1 is an orthogonal matrix, such that .After the transformation, the elements at the position (i , j), (j ,
i) get annihilated, that is dij and dji reduce to zero, which is seen as follows:

© Copyright Virtual University of Pakistan 233

23- Jacobi’s Method VU

 dii dij 
d d jj 
 ji
 cos θ sin θ   aii aij  cos θ − sin θ 
=  a jj   sin θ cos θ 
a
 − sin θ cos θ   aij
 aii cos 2 θ sin θ cos θ + a jj sin 2 θ (a jj − aii ) sin θ cos θ + aij cos 2θ 
 
(a ji − aii ) sin θ cos θ + aij cos 2θ aii sin 2 θ + a jj cos 2 θ − 2aij sin θ cos θ 
Therefore, d ij = 0 only if,

a jj − aii
aij cos 2θ + sin 2θ =
0
2

That is if
2aij
tan 2θ =
aii − a jj
Thus, we choose θ such that the above equation is satisfied, thereby, the pair of off-diagonal elements dij
and dji reduces to zero.However, though it creates a new pair of zeros, it also introduces non-zero
contributions at formerly zero positions.
Also, the above equation gives four values of , but to get the least possible rotation, we choose

© Copyright Virtual University of Pakistan 234

 24- Example VU

Lecture 24

Example
Find all the eigen values and the corresponding eigen vectors of the matrix by Jacobi’s
method
 1 2 2 
 
A= 2 3 2
 
 2 2 1 
Solution
The given matrix is real and symmetric. The largest off-diagonal element is found to be
a=
13 a=
31 2.
Now, we compute
2aij 2a13 4
tan 2θ = = = = ∞
aii − a jj a11 − a33 0
This gives, θ =π 4
Thus, we construct an orthogonal matrix Si as
cos π4 0 − sin π4   2 0 − 2 
1 1

 
=S1 = 0 1 0   0 1 0 
 sin π4 0 cos π4   12 0 1 
 2 

The first rotation gives,

D1 = S1−1 AS1
 12 0 − 1
 1 2 2   12 0 − 12 
 
2
 
=0 1 0  2 3 2  0 1 0 
1 1   1 1 
 2 0 2  2 2 1   2 0 2 

3 2 0 
=  2 3 0 
 0 0 −1
We observe that the elements d13 and d31 got annihilated. To make sure that calculations
are correct up to this step, we see that the sum of the diagonal elements of D1 is same as
the sum of the diagonal elements of the original matrix A.
As a second step, we choose the largest off-diagonal element of D1 and is found to be
d=
12 d=
21 2, and compute
2d12 4
tan 2θ = = = ∞
d11 − d 22 0
This again gives θ = π 4
Thus, we construct the second rotation matrix as

© Copyright Virtual University of Pakistan 235

24- Example VU

 12 − 12 0 
 
S 2 =  12 1
2
0
0 0 1 

At the end of the second rotation, we get
D2 = S 2−1 D1S 2
 12 1
2
0   3 2 0   12 − 1
2
0
   
=  − 12 1
2
0   2 3 0   12 1
2
0
 0 1   0 0 −1  0 1 
 0 0
5 0 0 
= 0 1 0 
0 0 −1
This turned out to be a diagonal matrix, so we stop the computation. From here, we
notice that the eigen values of the given matrix are 5,1 and –1. The eigenvectors are the
column vectors of S = S1S 2
Therefore
 12 0 − 12   12 − 12 0 
  
S =  0 1 0   12 1
2
0 
1 0 1 
0 1 
 2 2  0

 1
− 12 − 1
 
2 2

= 1
2
1
2
0 
 

1
2 − 12 1
2 


Example
Find all the eigen values of the matrix by Jacobi’s method.
 2 −1 0 
A=  −1 2 −1
 
 0 −1 2 
Solution
Here all the off-diagonal elements are of the same order of magnitude. Therefore, we can
choose any one of them. Suppose, we choose a12 as the largest element and compute
−1
tan 2θ = = ∞
0
Which gives, θ = π 4.

Then cos= θ sin= θ 1 2

and we construct an orthogonal matrix S1 such that

© Copyright Virtual University of Pakistan 236

 24- Example VU

 12 − 12 0 
 
S =  12 1
2
0
0 0 1 

The first rotation gives
D1 = S1−1 AS1
 12 − 12 0   2 −1 0   12 − 1
2
0
 1   
=
− 2
1
2
0   −1 2 −1  12 1
2
0
 0 1   0 −1 2   0 1 
 0 0
 1 0 − 12 
 
=  0 3 − 12 
 1 
 − 2 − 2
1
2 
Now, we choose d13 = −1 2
As the largest element of D1 and compute
2d13 − 2
tan 2θ =
=
d11 − d33 1 − 2
θ = 27o 22′41′′ .
Now we construct another orthogonal matrix S2, such that
 0.888 0 −0.459 
S 2 =  0 1 0 
0.459 0 0.888 
At the end of second rotation, we obtain
0.634 −0.325 0 
=D2 S= −1  −0.628
2 D1 S 2  0.325 3
 0 −0.628 2.365 
Now, the numerically largest off-diagonal element of D2 is found to be d 23 = −0.628 and
compute.
−2 × 0.628
tan 2θ =
3 − 2.365
θ = −31 35′24′′.
o

Thus, the orthogonal matrix is

1 0 0 
S3 = 0 0.852 0.524 

0 −0.524 0.852 
At the end of third rotation, we get

© Copyright Virtual University of Pakistan 237

 24- Example VU

 0.634 −0.277 0 
= D3 S= −1 
D2 S3 0.277 3.386 0 
3

 0 0 1.979 
To reduce D3 to a diagonal form, some more rotations are required. However, we may
take 0.634, 3.386 and 1.979 as eigen values of the given matrix.

Example

Using Jacobi’s method, find the eigenvalues and eigenvectors of the following matrix,
1 1/ 2 1/ 3
1/ 2 1/ 3 1/ 4 

1/ 3 1/ 4 1/ 5 
Solution:
The given matrix is real and symmetric. The l arg est off − diagonal element is found to be
1
a=
12 a=
21
2
Now we comute
1
2 
=
tan 2θ
2aij
=
2a12
= = 2 3
aii − a jj a11 − a22 1 − 1 2
3
3
tan −1  
=θ =  2  28.155
2

Thus we construct an orthogonal matrix S1 as

cos 28.155 − sin 28.155 0  0.882 −0.472 0 
S1 =  0.472 0.882 0 
 sin 28.155 cos 28.155 0   
 0 0 1   0 0 1 
The first rotation gives, D1 = S1−1 AS1
 0.882 0.472 0  1 1/ 2 1/ 3 0.882 −0.472 0 
=  −0.472 0.882 0  1/ 2 1/ 3 1/ 4  0.472 0.882 0 
 0 0 1  1/ 3 1/ 4 1/ 5   0 0 1 
1.268 0.000 0.412 
= 0.000 0.066 0.063
0.412 0.063 0.200 

We see that sum of the diagonal elements of D 1

=1.53

© Copyright Virtual University of Pakistan 238

 24- Example VU

And the sum of the diagonal elements of A = 1.53

This means that our question is going right.

As a second step we choose the largest of off-diagonal element of D 1 , which is d 13 = d 31

= 0.412, and comput
2d13 2 ( 0.412 )
tan 2θ =
= = 0.772
d11 − d33 1.268 − 0.200
tan −1 ( 0.772 )
=θ = 18.834
2

Thus we construct an orthogonal matrix S 2 as

cos18.834 0 − sin18.834  0.946 0 −0.323
S2 = 0 1 0   0 1 0 
  
 sin18.834 0 cos18.834   0.323 0 0.946 

Thus the rotation gives,

D2 = S 2−1 D1S 2
 0.946 0 0.323 1.268 0.000 0.412  0.946 0 −0.323
= 0 1 0  0.000 0.066 0.063  0 1 0 
 −0.323 0 0.946  0.412 0.063 0.200   0.323 0 0.946 
 1.408 0.020 −0.001
=  0.020 0.066 0.060 
 −0.001 0.060 0.059 
We again see that sum of the diagonal elements of D 2 =1.53
Also the sum of the diagonal elements of A = 1.53
This means that our question is going right.
Hence the eigenvalues are 1.408 , .066 and .059 and the corresponding eigenvectors are
the columns of S.Where
S = S 1S 2
0.882 −0.472 0  0.946 0 −0.323
= 0.472 0.882 0   0 1 0 
 0 0 1   0.323 0 0.946 
.8343 − .472 − .2848 

= .446 .88 − .1524 
.323 0 .946 

© Copyright Virtual University of Pakistan 239

24- Example VU

© Copyright Virtual University of Pakistan 240

25-Inner Product Space VU

Lecture 25
Inner Product Space
Inner Product Space

In mathematics, an inner product space is a vector space with the additional structure
called an inner product. This additional structure associates each pair of vectors in the
space with a scalar quantity known as the inner product of the vectors. Inner products
allow the rigorous introduction of intuitive geometrical notions such as the length of a
vector or the angle between two vectors. They also provide the means of defining
orthogonality between vectors (zero inner product). Inner product spaces generalize
Euclidean spaces (in which the inner product is the dot product, also known as the scalar
product) to vector spaces of any (possibly infinite) dimension, and are studied in
functional analysis.

Definition
An inner product on a vector space V is a function that to each pair of vectors
u and v associates a real number 〈u , v〉 and satisfies the following axioms,
For all u , v , w in V and all scalars C:
1) 〈 u , v〉 = 〈 v, u 〉
2) 〈u + v, w〉 = 〈u , w〉 + 〈 v, w〉
3) 〈 cu , v〉 = c 〈u , v〉
4) 〈u , u 〉 ≥ 0 and= 〈u , u 〉 0 =
iff u 0

A vector space with an inner product is called inner product space.

Example 1
Fix any two positive numbers say 4 & 5 and for vectors u = u1 , u2 and
v = v1 , v2 in R 2 set
u=
, v 4u1v1 + 5u2 v2
Show that it defines an inner product.

Solution
Certainly Axiom 1 is satisfied, because
u , v = 4u 1 v 1 +5u 2 v 2 = 4v 1 u 1 + 5v 2 u 2 = v ,u .
If w = (w 1 , w 2 ), then
u + v, w = 4(u1 + v1 ) w1 + 5(u2 + v2 ) w2
= 4u1w1 + 5u2 w2 + 4v1w1 + 5v2 w2 = u , w + v, w
This verifies Axiom 2.
For Axiom 3, we have cu, v = 4(cu1 )v1 + 5(cu2 )v2 = c(4u1v1 + 5u2 v2 ) = c u, v
For Axiom 4, note that u , u = 4u12 + 5u22 ≥ 0, and 4u12 + 5u22 =
0 only if u 1 = u 2 = 0, that
is, if u = 0. Also, 0, 0 = 0.

©Virtual University Of Pakistan 241

25-Inner Product Space VU

So u , v = 4u 1 v 1 +5u 2 v 2 = 4v 1 u 1 + 5v 2 u 2 defines an inner product on R2.

Example 2
Let A be symmetric, positive definite n × n matrix and let u and v be vectors in ℜn .
Show that u , v = u t Av defines and inner product.
Solution
We check that
u=
, v u= t
Av u= . Av Av.u

( v A=
)u
t
= A=
t
v.u t
v=
t
Au v, u
Also
u , v + w = u t A ( v + w )= u t Av + u t Aw
= u, v + u, w
And
=
cu , v ( cu
= ) Av
t
( u t Av ) c u, v
c=
Finally since A is positive definite
u, u = u t Au > 0 for all u ≠ 0
So u=
, u u=
t
Au 0 iff=u 0
So u , v = u t Av is an inner product space.

Example 3
Let t 0 , …, t n be distinct real numbers. For p and q in P n , define
= p, q p (t0 )q (t0 ) + p (t1 )q (t1 ) +  + p (tn )q (tn )
Show that it defines inner product.

Solution
Certainly Axiom 1 is satisfied, because
=p, q p (t0 )q (t0 ) + p (t1 )q (t1 ) +  + p (tn )q (tn )
= q (t0 ) p (t0 ) + q (t1 ) p (t1 ) +  + q (tn )=
p (tn ) q, p
If r = r (t0 ) + r (t1 ) +  + r (tn ) , then
p + q, r= [ p(t0 ) + q(t0 )] r (t0 ) + [ p(t1 ) + q(t1 )] r (t1 ) +  + [ p(tn ) + q(tn )] r (tn )
= [ p(t0 )r (t0 ) + p(t1 )r (t1 ) + ... + p(tn )r (tn )] + [ q(t0 )r (t0 ) + q(t1 )r (t1 ) + ... + q(tn )r (tn )]
= p, r + q, r
This verifies Axiom 2.
For Axiom 3, we have
= cp, q [cp(t0 )] q(t0 ) + [cp(t1 )] q(t1 ) +  + [cp(tn )] q(tn )
= c [ p (t0 )q (t0 ) + p (t1 )q (t1 ) +  + p (t=
n ) q (t n ) ] c p, q
For Axiom 4, note that
p=
, p [ p (t0 )]2 + [ p (t1 )]2 +  + [ p (tn )]2 ≥ 0

©Virtual University Of Pakistan 242

25-Inner Product Space VU

Also, 0, 0 =0. (We still use a boldface zero for the zero polynomial, the zero vector in
P n .) If p, p =0, then p must vanish at n + 1 points: t 0 , …, t n . This is possible only if p is
the zero polynomial, because the degree of p is less than n + 1. Thus
= p, q p (t0 )q (t0 ) + p (t1 )q (t1 ) +  + p (tn )q (tn ) defines an inner product on P n .

Example 4
Compute p, q where p(t)= 4+t q(t) = 5-4t2
Refer to P2 with the inner product given by evaluation at -1, 0 and 1 in example 2.
Solution
P(=−1) 3 , P=(0) 4 , P=(1) 5
q (=
−1) 1 , q=
(0) 5 , q=(1) 1

p, q = P(−1)q (−1) + P(0)q (0) + P(1)q (1)

= (3)(1) + (4)(5) + (5)(1)
=3 + 20 + 5

Example 5
Compute the orthogonal projection of q onto the subspace spanned by p, for p and q in
the above example.
Solution
The orthogonal projection of q onto the subspace spanned by p

P(=
−1) 3 , P= =
(0) 4 , P (1) 5
q (=
−1) 1 , q=
(0) 5 , q=
(1) 1

= =
q. p 28 p. p 50

 q. p 28
=q = p (4 + t )
p. p 50
56 14
= + t
25 25

Example 6
Let V be P2 , with the inner product from example 2 where
1
=
t0 0= , t1 and=t2 1
2
Let p ( t=) 12t 2 and q ( t=) 2t − 1
Compute p, q and q, q
Solution

©Virtual University Of Pakistan 243

25-Inner Product Space VU

, q p ( 0) q ( 0) + p
p= ( ) + p (1) q (1)
1

= ( 0 )( −1) + ( 3)( 0 ) + (12 )(1)= 12

2
  1 
q, q = q ( 0 )  +  q    +  q (1) 
2 2

  2 
( −1) + ( 0 ) + (1) =
=
2 2 2
2

Norm of a Vector
Let V be and inner product space with the inner product
denoted by u , v just as in R n , we define the length or norm of a vector V to be the
scalar
= =
2
v u, v or v u, v

1) A unit vector is one whose length is 1.

2) The distance between u & v is u − v vectors u & v are orthogonal if u , v = 0

Example 7
Compute the length of the vectors in example 3.

Solution
2
  1 
 p ( 0 )  +  p  2   +  p (1) 
2 2
p =p, p =
2

  
= ( 0 ) + ( 3) + (12 ) = 153
2 2 2

p = 153

In example 3 we found that

q, q = 2
Hence q = 2

Example 8
Let ℜ2 have the inner product of example 1 and let x=(1,1) and y=(5,-1)
b) Describe all vectors ( z1 , z2 ) that are
2
a) Find x , y and x, y
orthogonal to y.
Solution

a) We have x=(1,1) and y=(5,-1)

And x=
, y 4 x1 y1 + 5 x2 y2

©Virtual University Of Pakistan 244

25-Inner Product Space VU

=x =
x, x 4(1)(1) + 5(1)(1)
= 4 + 5= 9= 3
=y y=
,y 4(5)(5) + 5(−1)(−1)
= 100 + 5= 105

2
x, y = x, y x, y
= [ 4(1)(5) + 5(1)(−1)]
2

= [ 20 − 5]
2

= [15
= ] 225
2

b) All vectors z = ( z1 , z2 ) orthogonal to y=(5,-1)

< y, z >= 0
4(5)( z1 ) + 5(−1)( z2 ) = 0
20 z1 − 5 z2 =
0
4 z1 − z2 =
0
 z1 
[4 − 1]   = 0
 z2 
 4
So all multiples of   are orthogonal to y.
 −1

Example 9
Le V be P4 with the inner product in example 2 involving evaluation of
polynomials at -2,-1,0,1,2 and view P2 as a subspace of V. Produce an orthogonal basis
for P2 by applying the Gram Schmidt process to the polynomials 1, t & t 2 .
Solution

Given polynomials 1 t t2 at -2,-1, 0, 1 and 2

Polynomial: 1 t t2
1  −2  4
1  −1 1 
    
Vector of values: 1 ,  0  ,  0 
    
1 1 1 
1  2   4 
The inner product of two polynomials in V equals the (standard) inner product of their
corresponding vectors in R5. Observe that t is orthogonal to the constant function 1. So

©Virtual University Of Pakistan 245

25-Inner Product Space VU
take p 0 (t) = 1 and p 1 (t) = t. For p 2 , use the vectors in R to compute the projection of t2
5

onto Span {p 0 , p 1 }:
t 2 , p0 = t 2 ,1 = 4 + 1 + 0 + 1 + 4 = 10
p0 , p0 = 5
t 2 , p1 = t 2 , t =−8 + (−1) + 0 + 1 + 8 =0
10
The orthogonal projection of t2 onto Span {1, t} is p0 + 0 p1. Thus
5
p2 (t ) =
t 2 − 2 p0 (t ) =
t2 − 2
An orthogonal basis for the subspace P 2 of V is:

Polynomial: p0 p1 p2

1  −2  2
1  −1  −1
    
Vector of values: 1 , 0 ,  −2 
    
1 1  −1
1  2   2 

Best Approximation in Inner Produce Spaces

A common problem in applied mathematics involves a vector space V whose

elements are functions. The problem is to approximate a function f in V by a function
g from a specified subspace W of V. The “closeness” of the approximation of f
depends on the way f − g is defined. We will consider only the case in which the
distance between f and g is determined by an inner product. In this case the best
approximation to f by functions in W is the orthogonal projection of f onto the
subspace W.

Example 10
Let V be P4 with the inner product in example 5 and let P0 , P1 & P2
be the orthogonal basis for the subspace P2 , find the best approximation to
1
p ( t )= 5 − t 4 by polynomials in P2 .
2

Solution:
The values of p 0 , p 1 , and p 2 at the numbers – 2, –1, 0, 1, and 2 are listed
in R5 vectors in
Polynomial: p0 p1 p2

©Virtual University Of Pakistan 246

25-Inner Product Space VU

1  −2  2
1  −1  −1
    
Vector of values: 1 ,  0  ,  −2 
    
1 1  −1
1  2   2 
The corresponding values for p are: −3,9 2,5,9 2, and –3.
We compute
p, p0 = 8 p, p1 = 0 p, p2 = −31
= p0 , p0 5,= p2 , p2 14
Then the best approximation in V to p by polynomials in P 2 is
p, p0 p, p1 p, p2
pˆ = proj p2 p = p0 + p1 + p2
p0 , p0 p1 , p1 p2 , p2
=
5 p0 + 14 p2 =
8 −31
5 − 14 (t − 2).
8 31 2

This polynomial is the closest to P of all polynomials in P 2 , when the distance between
polynomials is measured only at –2, –1, 0, 1, and 2.

Cauchy – Schwarz Inquality

For all u , v in V
u, v ≤ u v

Triangle Inequality

For all u , v in V
u+v ≤ u + v
Proof
u + v = u + v, u + v
2

=u , u + 2 u , v + v, v
≤ u + 2 u, v + v
2 2

≤ u +2 u v + v
2 2

(u )
2
u+v = + v
2

⇒ u+v = u + v

C [ a, b ]
Inner product for

©Virtual University Of Pakistan 247

25-Inner Product Space VU
Probably the most widely used inner product space for applications
is the vector space C [ a, b ] of all continuous functions on an interval a ≤ t ≤ b , with an
inner product that will describe.

Example 11
For f , g in C [ a, b ] , set
f , g = ∫ f ( t ) g ( t ) dt
a

Show that it defines an inner product on C [ a, b ] .

Solution
Inner product Axioms 1 to 3 follow from elementary properties of
definite integrals

1. f , g = g , f
2. f + h, g = f , g + h, g
3. cf , g = c f , g
For Axiom 4, observe that
b
= ∫ [ f (t )] dt ≥ 0
2
f, f
a

The function [f(t)]2 is continuous and nonnegative on [a, b]. If the definite integral of
[f(t)]2 is zero, then [f(t)]2 must be identically zero on [a, b], by a theorem in advanced
calculus, in which case f is the zero function. Thus f , f = 0 implies that f is the zero
function of [a, b].
b
So f , g = ∫ f (t ) g (t )dt defines an inner product on C[a, b].
a

Example 12
Compute f,g where f (t ) =
1 − 3t 2 and g (t ) = C [ 0,1] .
t − t 3 on v =
Solution
Let V be the space C [ a, b ] with the inner product
f , g = ∫ f ( t ) g ( t ) dt
a

f (t ) =
1 − 3t 2 , g (t ) =
t − t3

∫ (1 − 3t ) ( t − t )dt
1
f,g = 2 3
0

∫ ( 3t − 4t 3 + t )dt
1
= 5
0
1
1 6 4 1 2
= t −t + t
2 2 0
=0
Example 13

©Virtual University Of Pakistan 248

25-Inner Product Space VU

Let V be the space C [ a, b ] with the inner product

f , g = ∫ f ( t ) g ( t ) dt
a

b
Let W be the subspace spanned by the polynomials
P1 ( t ) =
1 , P2 ( t ) =
2t − 1 & P3 ( t ) =
12t 2
Use the Gram – Schmidt process to find an orthogonal basis for W.
Solution

Let q 1 = p 1 , and compute

1 1

p2 , q1 =∫ (2t − 1)(1)dt =(t 2 − t ) =0

0 0

So p 2 is already orthogonal to q 1 , and we can take q 2 = p 2 . For the projection of p 3 onto

W 2 = Span {q 1 , q 2 }, we compute
1 1

p3 , q1 = ∫
0
12t 2 ⋅1dt = 4t 3 = 4
0

1 1

q1 , q1 = ∫ 1 ⋅1dt = t =1
0 0

1 1
p3 , q2 = ∫ 12t 2 (2t − 1)dt= ∫ (24t − 12t 2 )dt= 2
3
0 0
1 1 1
1
q2 , q2 =∫0 (2t − 1) 2 dt =
− =
3
(2 t 1)
6 0 3
p ,q p ,q 4 2
Then projw2 p3 = 3 1 q1 + 3 2 q2 = q1 + q2 =4q1 + 6q2
q1 , q1 q2 , q2 1 13
And q3 = p3 − projw2 p3 = p3 − 4q1 − 6q2
As a function, q 3 (t) = 12t2 – 4 – 6(2t – 1) = 12t2 – 12t + 2. The orthogonal basis for the
subspace W is {q 1 , q 2 , q 3 }

Exercises
Let ℜ2 have the inner product of example 1 and let x=(1,1) and y=(5,-1)
b) Describe all vectors ( z1 , z2 ) that are
2
a) Find x , y and x, y
orthogonal to y.

2) Let ℜ2 have the inner product of Example 1. Show that the Cauchy-Shwarz
inequality holds for x=(3,-2) and y=(-2,1)

Exercise 3-8 refer to P2 with the inner product given by evaluation at -1,0 and 1 in
example 2.
3) Compute p, q where p(t)= 4+t q(t) = 5-4t2
4) Compute p, q where p(t)= 3t - t2 q(t) = 3 + t2
5) Compute P and q for p and q in exercise 3.
6) Compute P and q for p and q in exercise 4.
7) Compute the orthogonal projection of q onto the subspace spanned by p, for p
and q in Exercise 3.

©Virtual University Of Pakistan 249

25-Inner Product Space VU
8) Compute the orthogonal projection of q onto the subspace spanned by p, for p
and q in Exercise 4.

9) Let P 3 have the inner product given by evaluation at -3,-1,1, and 3. Let
= pο (t ) 1=
, p1 (t ) t , and=p2 (t ) t 2
a)Computer the orthogonal projection of P2 on to the subspace spanned by P0 and P1 . b)
Find a polynomial q that is orthogonal to P0 and P1 such tha { p0 , p1 , q} is an orthogonal
basis for span { p0 , p1 , q} . Scale the polynomial q so that its vector of values at (-3,-
1,1,3) is (1,-1,-1,1)
10) Let P 3 have the inner product given by evaluation at -3,-1,1, and 3. Let
= pο (t ) 1=, p1 (t ) t , and= p2 (t ) t 2
Find the best approximation to p (t ) = t 3 by polynomials in Span { p0 , p1 , q} .
11) Let p0 , p1 , p2 be the orthogonal polynomials described in example 5, where the
inner product on P4 is given by evaluation at -2, -1, 0, 1, and 2. Find the orthogonal
projection of t 3 onto Span { p0 , p1 , p2 }
12) Compute f,g where f (t ) =
1 − 3t 2 and g (t ) = C [ 0,1] .
t − t 3 on v =

13) Compute f,g where

f (t ) =
5t − 3 and g (t ) = C [ 0,1] .
t 3 − t 2 on v =
14) Compute f for f in exercise 12.
15) Compute g for g in exercise 13.

16) Let V be the space C[-2,2] with the inner product of Example 7. Find an
orthogonal basis for the subspace spanned by the polynomials 1, t , t 2 .
17)
u1  v1 
Let u =   and v =   be two vectors in R 2 . Show that u= , v 2u1v1 + 3u2 v2
u2  v2 
defines an inner product.

©Virtual University Of Pakistan 250

26-Application of Inner Product Spaces VU

Lecture 26
Application of inner product spaces

Definition

An inner product on a vector space V is a function that associates to each pair of vectors u
and v in V, a real number u , v and satisfies the following axioms, for all u, v, w in V
and all scalars c:
1. u , v = v, u
2. u + v, w = u , w + v, w
3. cu , v = c u , v
4. u , u ≥ 0 and u , u =
0 iff u =
0.

A vector space with an inner product is called an inner product space.

Least Squares Lines

The simplest relation between two variables x and y is the linear

equation = y β 0 + β1 x . Often experimental data produces points ( x1 , y1 ),..., ( xn , yn ) that
when graphed, seem to lie close to a line. Actually we want to determine the parameters
β 0 and β1 that make the line as “close” to the points as possible. There are several ways
to measure how close the line is to the data. The usual choice is to add the squares of the
residuals. The least squares line is the line = y β 0 + β1 x that minimizes the sum of the
squares of the residuals.
If the data points are on the line, the parameters β 0 and β1 would satisfy the
equations
predicted Observed
value value
_________________
β 0 + β1 x1 =
y1
β 0 + β1 x2 =
y2
. .
. .
. .
β 0 + β1 xn = yn

©Virtual University Of Pakistan 251

26-Application of Inner Product Spaces VU

We can write this system as

Xβ = y
1 x1   y0 
1 x2  y 
  1
. .  β0  . 
=
Where X  =  , β =  , y  
. .  β1  . 
. . . 
   
1 xn   yn 

Computing the least-squares solution of X β = y is equivalent to finding the β that

determines the least-squares line.

Example 1

Find the equation = y β 0 + β1 x of the least-squares line that best fits the data points
(2, 1), (5, 2), (7, 3), (8, 3).

Solution

Xβ = y

1 2 1 
1  2
5  β0 
Here X 
= = , β =  , y  
1 7  β1  3 
   
1 8 3 
For the least-squares solution of xβ = y , obtain the normal equations(with the new
notation) :
X T X βˆ = X T y
i.e, compute

1 2
1 1 1 1  1 5   4 22 
X T X =
2 5 7 8 1 7   22 142 
 
1 8
1 
1 1 1 1   2  9 
X T y =
2 5 7 8 3  57 
 
3 

©Virtual University Of Pakistan 252

26-Application of Inner Product Spaces VU

The normal equations are

4 22   β 0   9 
=
 22
 142   β1  57 
Hence,

−1
 β0  4 22   9 
 β  =  22 142  57 
 1 
1 142 − 22   9 
= 
84  −22 4  57 
1  24   2 / 7 
= =
84 30  5 /14 
Thus, the least -squares line has the equation
2 5
y= + x
7 14

Weighted Least-Squares

Let y be a vector of n observations, y1 , y2 ,..., yn and suppose we wish to

approximate y by a vector ŷ that belongs to some specified subspace of  n (as discussed
previously that ŷ is written as Ax so that ŷ was in the column space of A).Now suppose
the approximating vector ŷ is to be constructed from the columns of matrix A. Then we
find an x̂ that makes Axˆ = yˆ as close to y as possible. So that measure of closeness is the
weighted error
Wy − Wyˆ =Wy − WAyˆ
2 2

©Virtual University Of Pakistan 253

 26-Application of Inner Product Spaces VU

Where W is the diagonal matrix with (positive) w1 ,..., wn on its diagonal, that is
 w1 0 ... 0
0 w2 
 
. . .
W = 
. . .
. . .
 
0 . .. wn 
Thus, x̂ is the ordinary least-squares solution of the equation
WAx = Wy

The normal equation for the weighted least-squares solution is

(WA) WAX = (WA ) Wy
T T

Example 2

y β 0 + β1 x that best fits the data (–2, 3), (–1, 5), (0, 5), (1, 4),
Find the least squares line =
(2, 3). Suppose that the errors in measuring the y-values of the last two data points are
greater than for the other points. Weight this data half as much as the rest of the data.

Solution

Write X , β and y
1 −2  3 
1 − 1 5 
  β0   
=X 1 =0  , β = 5 
β1 
, y
    
1 1 4
1 2  3 

For a weighting matrix, choose W with diagonal entries 2 , 2 , 2 , 1 and 1.

Left-multiplication by W scales the rows of X and y:

2 0 0 0 0  1 −2 
 0  1 − 1
0 2 0 0
WX = 0 0 2 0 0  1 0
  
0 0 0 1 0  1 1
0 0 0 0 1  1 2 

©Virtual University Of Pakistan 254

 26-Application of Inner Product Spaces VU

2 − 4 6 
2 − 2 10 
  
=WX = 2 0  , Wy 10 
   
1 1 4 
1 2  3 
For normal equation, compute

14 −9  59 
(WX )T WX =
 −9
T

25  −34 
, and (WX ) Wy
  
And solve
14 − 9   β 0   59 
=
 −9
 25  β1   −34 
−1
 β0  14 − 9   59 
β  =  −9 25   −34 
 1 

 β0  1  25 9   59 
β  =  14   −34 
 1  269  9

 β0  1  25 9   59 
β  =  14   −34 
 1  269  9

 β0  1 1169   4.3
= β  =    
 1  269  55  0.2 
=
Therefore, the solution to two significant digits is β 0 4.3
= and β1 0.20 .
Hence the required line is =y 4.3 + 0.2 x
In contrast, the ordinary least-squares line for this data can be found as:

1 −2 
1 − 1
1 1 1 1 1  5 0
XTX =  1 0 
 −2 − 1 0 1 2   10 
1 
0
1
1 2 

©Virtual University Of Pakistan 255

26-Application of Inner Product Spaces VU

3 
5 
1 1 1 1 1    20 
XT y = 5 
 −2 − 1 0 1 2     −1
4
3 

5 0   β 0   20 
=
0
 10   β1   −1 
−1
 β 0  5 0   20 
 β  = 0 10   −1 
 1 
 β 0  1 10 0   20 
 β  = 0 5   −1 
 1  50 
 β 0  1  200   4.0 
=
β  =    
 1  50  − 5   −0.1

Hence the equation of least-squares line is

=y 1.0 − 0.1x

What Does Trend Analysis Mean?

An aspect of technical analysis that tries to predict the future movement of a stock based
on past data. Trend analysis is based on the idea that what has happened in the past gives
traders an idea of what will happen in the future.

©Virtual University Of Pakistan 256

26-Application of Inner Product Spaces VU

Linear Trend

A first step in analyzing a time series, to determine whether a linear relationship provides
a good approximation to the long-term movement of the series computed by the method
of semi averages or by the method of least squares.

Note

The simplest and most common use of trend analysis occurs when the points t0 , t1,..., tn
can be adjusted so that they are evenly spaced and sum to zero.

Example

Fit a quadratic trend function to the data (-2,3), (-1,5), (0,5), (1,4) and (2,3)

Solution

The t-coordinates are suitably scaled to use the orthogonal polynomials found in Example
5 of the last lecture. We have

Polynomial : p0 p1 p2 data : g
1  −2   2  3 
1  −1   −1  5 
      
Vector of values : 1 ,  0  ,  −2  , 5 
      
1  1   −1  4
1  2   2  3 
< g , p0 > < g , p1 > < g , p2 >
pˆ = p0 + p1 + p2
< p0 , p0 > < p1 , p1 > < p2 , p2 >

20 1 7
= p0 − p1 − p2
5 10 14

and pˆ (t ) =−
4 0.1t − 0.5(t 2 − 2)

Since, the coefficient of p2 is not extremely small, it would be reasonable to conclude

that the trend is at least quadratic.

©Virtual University Of Pakistan 257

26-Application of Inner Product Spaces VU

Above figure shows that approximation by a quadratic trend function

Fourier series

If f is a 2π -periodic function then

a0 ∞
f (t ) = + ∑ (am cos mt + bm sin mt )
2 m =1
is called Fourier series of f where

2π
1
am =
π ∫
0
f (t ) cos mt dt and

2π
1
bm =
π ∫
0
f (t ) sin mt dt

Example

Let C [0, 2π ] has the inner product

2π
< f , g >= ∫
0
f (t ) g (t ) dt

and let m and n be unequal positive integers. Show that cos mt and cos nt
are orthogonal.

Solution

When m ≠ n

2π
< cos mt , cos nt > = ∫ cos mt cos nt dt
0
2π
1
2 ∫0
= [cos (mt + nt ) + cos(mt − nt ) dt

©Virtual University Of Pakistan 258

 26-Application of Inner Product Spaces VU

2π
1  sin (mt + nt ) sin (mt − nt ) 
= +
2  m + n m − n  0
=0

Example

Find the nth-order Fourier approximation to the function

f (t ) = t on the interval [0, 2π ] .

Solution

We compute

2π
1 1 2 
2π
a0 1 1
2 2 π ∫0
= = . t dt =  t  π
2π  2 0 

and for k>0, using integration by parts,

2π 2π
1 11 t 
ak = ∫
π 0
t cos kt dt = 
π k 2
cos kt + sin kt  = 0
k 0

2π 2π
1 11 t  2
bk = ∫
π 0
t sin kt dt =
π  k 2
sin kt −
k
cos kt 
0
=
−
k

Thus, the nth-order Fourier approximation of f (t ) = t is

2 2
π − 2sin t − sin 2t − sin 3t − ⋅⋅⋅ − sin nt
3 n

The norm of the difference between f and a Fourier approximation is called the mean
square error in the approximation.
It is common to write

a0 ∞
f (t ) = + ∑ (am cos mt + bm sin mt )
2 m =1
This expression for f (t) is called the Fourier series for f on [0, 2π ] . The term am cos mt ,
for example, is the projection of f onto the one-dimensional subspace spanned by cos mt .

©Virtual University Of Pakistan 259

26-Application of Inner Product Spaces VU

Example

Let q1 (=
t ) 1, q2 (=
t ) t , and q3 (=
t ) 3t 2 − 4 .Verify that {q1 , q2 , q3 } is an orthogonal set in
C{-2,2] with the inner product

b
< f , g >= ∫ f (t ) g (t ) dt
a

Solution:
2 2
1 2
< q=
1 , q2 > ∫−=
2
1.t dt
2
t=dt
−2
0

∫ 1.(3t
2
< q1 , q3 =
> 2
− 4) dt
= (t 2 − 4t ) = 0
−2
−2

2 2
3
< q2 , q=
3 > ∫−2 t.(3t − 4)=
dt ( t 4 − 2t 2=
2
) 0
4 −2

Exercise

1. Find the equation = y β 0 + β1 x of the least-squares line that best fits the data
points (0, 1), (1, 1), (2, 2), (3, 2).
2. Find the equation = y β 0 + β1 x of the least-squares line that best fits the data
points (-1, 0), (0, 1), (1, 2,),(2, 4).
3. Find the least-squares line = y β 0 + β1 x that best fits the data
(-2, 0), (-1, 0), (0, 2,),(1, 4),(2, 4), assuming that the first and last data points are
less reliable. Weight them half as much as the three interior points.
4: To make a trend analysis of six evenly spaced data points, one can use orthogonal
polynomials with respect to evaluation at the points t=-5, -3, -1, 1, 3and 5
(a). Show that the first three orthogonal polynomials are
3 2 35
p0 =
(t ) 1, p1=(t ) t , and p2 = (t ) t −
8 8
(b) Fit a quadratic trend function to the data
(-5, 1), (-3, 1), (-1, 4), (1, 4), (3, 6), (5, 8)
5: For the space C [0, 2π ] with the inner product defined by
2π
< f , g >= ∫
0
f (t ) g (t ) dt

(a) Show that sin mt and sin nt are orthogonal when m ≠ n

(b) Find the third–order Fourier approximation to f (=t ) 2π − t
3
(c) Find the third order Fourier approximation to cos t , without performing any

©Virtual University Of Pakistan 260

26-Application of Inner Product Spaces VU

integration calculations.

6: Find the first-order and third order Fourier approximations to

f (t ) =
3 − 2sin t + 5sin 2t − 6 cos 2t

©Virtual University Of Pakistan 261

27-Houreholder’s Method and QR Algorithm VU

Lecture 27

Householder’s Method and QR Algorithum

PPT’s slides are available in VULMS/downloads
28-SVD VU

Lecture 28

Singular Value Decomposition

PPT’s slides are available in VULMS/downloads
29-Fixed points of functions of several variables VU

Lecture 29

Fixed Points for Functions of Several Variables

PPT’s slides are available in VULMS/downloads
30- Newton’s Method VU

Lecture 30

Newton’s Method
PPT’s slides are available in VULMS/downloads
31- Quasi-Newton Method VU

Lecture 31

Quasi-Newton Method
PPT’s slides are available in VULMS/downloads