Linear Algebra Lecture Slides

MATH 3341 Advanced Engineering Mathematics

Fairleigh Dickinson University
Prof. Alexander Casti
Matrices and Vectors
Section 7.1 Kreyszig
 Early Example of Coupled Linear Equations
 Perhaps the most fundamental problem in linear algebra concerns the solution of "simultaneous" (coupled)
linear equations. We will soon write these equations in the compact form Ax = b
 The earliest known example comes from the ancient Chinese mathematics book Chiu - chang Suan - shu
(Nine Chapters on Arithmetic; ~200 BC) . Note: A "sheaf" is a bundle of wheat.

x1 price of 1 sheaf
= of good wheat 3x1 + 2 x2 + x3 39
x2 price of 1 sheaf of mediocre
= wheat → 2 x1 + 3x2 + x3 34
x3 price of 1 sheaf of
= bad wheat x1 + 2 x2 + 3x3 26
 3 2 1   x1   39 
Matrix Form: Ax=b →  2 3 1   x  =  34 
  2   
 1 2 3   x   26 
  3   
A is a 3x 3 matrix
xb, are 3x1 column vectors (bold font indicates a matrix or vector)
 Matrix Multiplication (Motivation)
 A primary motivation for our definition of matrix multiplication comes from the desire to express
coupled linear equations as the product of a matrix A times a vector x:
3x1 + 2 x2 + x3 =
39   3 2 1   x1   39 
 2 3 1 x   34 
2 x1 + 3x2 + x3 34
=    2    {x1 , x2 , x3} are the "unknowns"
x1 + 2 x2 + 3x3 =
26   1 2 3   x3   26 
    
A x b
3x3 3 x1 3 x1

 For example, the 2nd entry 34 of the vector b is obtained by taking the
"inner product" (dot product) of the 2nd row of the matrix A with the column vector x
 x1 
(2 3 1)  x2  = (2 )( x1 ) + ( 3)( x2 ) + (1) ( x3 ) = 34
x 
 3

 A solution to this system of equations is a column vector x that simultaneously satisfies

all three of the given equations. If at least 1 solution exists the system is said to be consistent (otherwise inconsistent).
 We will learn the technique of Gaussian Elimination to solve algebraic problems like this.
You will solve this problem in a homework exercise and demonstrate the unique solution
 37   37 
 4  37   4
1 37 17 11
x= 17  =  17 
 4  4  
e.g. (2 3 1)  17 4  =(2 )   + ( 3)   + (1)   =34
 4   4   4 
 11   11   11 
 4  4
 Matrices and Vectors
Def: A matrix is a rectangular array of numbers or functions. An m x n matrix contains m rows and n columns.
A row vector is a special matrix with just 1 row. A column vector is a matrix with just 1 column.
 a11 a12  a1n 
a a22  a2n 
A =aij   21
where aij is the entry in row i and column j
     
 
 am1 am2  amn 
 If A is m x n then i 1,2,...,
= = m (m rows) and j 1,2,..., n (n columns)
( ai1 ai 2  ain ) ∈  n
 The ith row of A is Ai∗ = where  n denotes an n − dimensional Euclidean vector space
 a1 j 
a 
A∗ j   ∈  m
th 2j
 The j column of A is =
  
 
 amj 
Example Example (matrix entries are functions of a variable t )
0 1 2 3  et e −2t 
A(t ) =  −2t 2 x2 matrix (2 rows, 2 columns)
A =  −3 2 1 1  3x 4 matrix (3 rows, 4 columns) e et 

 −1 8 9 −2 
  A1∗ = ( et e −2t ) row 1 (1x2 matrix)
( −3 2 1 1) ∈ 
A2∗ = 4
row 2 (1x 4 matrix)
 e −2t 
2 A∗2 =  t  column 2 (2 x1 matrix)
 1  ∈ 3  e 
A=
∗3   column 3 (3x1 matrix)
9
 
a13 = 2, a34 = −2,...
 More Matrix Definitions
Def: An m x n matrix with m ≠ n is called a rectangular matrix.
Def: A square matrix is an n x n matrix with an equal number of rows and columns.
 a11 a12 a1n 
a   1.6 −3 π 
a22 a2n   
aij   21
A = = Example: A  2 8 0  → 3x3 square matrix
     
   −9 −4 −2.7 
 
 an1 am2 ann 
Def: The main diagonal of a square matrix A is the collection of entries d = ( a11 a22  ann ) along
the diagonal of A beginning at the upper-left entry a11 .
 (previous example) =d (1.6 8 −2.7 )
Def: The n x n square matrix In with ones on the main diagonal and zeros on all off-diagonal entries
is called the Identity Matrix.
1 0  0
0 1  0 
In =  (n rows and n columns)
0 0  
 
0 0  1
Def: A lower triangular matrix L has non-zero entries only on or below the diagonal.
An upper triangular matrix U has non-zero entries only on or above the diagonal.
Examples
 4 0 0 0  3 −6 7 1 
 −1 2 0 0   0 4 −1 −2 
L=  U = 
 3 −3 1 0   0 0 11 9 
   
5 2 9 2
 0 0 0 1
 
LOWER TRIANGULAR UPPER TRIANGULAR
 When are matrices equal?

= iff ( A )ij
A B= ( B )ij

Examples
 −1 2   1 2 
 3 4 ≠  3 4 (upper left entries on main diagonal unequal)
   
 −5 6   −5 6 
   

1 2  3 4 
3 4  ≠ 1 2  (unequal despite containing the same set of 4 numbers)
   
 Basic Algebra of Matrices

Examples (Addition of Matrices)

1 2 0 1 2
A  =
 B  → A + B undefined (unequal sizes)
3 4 3 4 5 
 −1 2 1 0  −1 + 1 2 + 0   0 2 
A=  B=  → A+B =   =  −2 6  = B + A
 −3 4 1 2  − 3 + 1 4 + 2   

Examples (Multiplying Matrices by a scalar)

1 2   1 2   7 ⋅ 1 7 ⋅ 2   7 14 
A =  c =7 → cA =7   = 7 ⋅ 3 7 ⋅ 4  = 21 28 
3 4  3 4     
1  1   −1 ⋅ 1   −1 
A =  2  c = −1 → cA = (−1)  2  =  −1 ⋅ 2  =  −2  Def : − A ≡ (−1) A
 3  3   −1 ⋅ 3   −3 
       
Rules for Matrix Addition and Scalar Multiplication
Matrix Multiplication
Section 7.2 Kreyszig
 Matrix Multiplication (1)
 The fundamental problem Ax = b of linear algebra involving m equations with n unknowns { x1 , x2 , , xn } is
a11 x1 + a12 x2 +  + a1n xn =
b1
a21 x1 + a22 x2 +  + a2n xn =
b2
   = 
am1 x1 + am2 x2 +  + amn xn = bm
where for our purposes the collection of numbers {a11 , a12 , , amn } and {b1 , b2 , , bn } are real numbers in .

 Matrix multiplication is defined such that this coupled system of linear equations (first powers in x j ) can be written
 a11 a12 a13  a1n   x1 
a   x   b1 
 21 a22 a23  a2n   2 b 
 a31 a32 a33  a3n   x3  =  2  ( m x n )( n x 1) → ( m x 1)
      
          b 
a am3  amn   x   m
 m1 am2
   n
b
A x mx1
mxn nx1 vector
matrix vector

 Def: This system of equations is called a linear system because the matrix A satisfies the properties of a linear operator.
Regard the matrix as a mapping from one Euclidean space to another: A :  n   m
Linearity: (1) A(x + y ) = Ax + Ay for any x , y ∈  n (2) A ( cx ) = cAx for any scalar (constant) c
 Loosely speaking, this system is linear because the unknowns x j appear only as first powers.
 Def: (Homogeneous vs. Inhomogeneous)
= The system Ax b is called inhomogeneous if b ≠ 0 and
homogeneous if b = 0.
Matrix Multiplication (2)

Number of columns in A =
Number of rows in B

c21 = a21b11 + a22b21 + a23b31

 Matrix Multiplication (3)
 Each entry of ( C ) jk in the matrix product AB = C can be viewed as a dot product of the jth row Aj∗
with the k th column B∗k
n
( C ) jk ≡ c=
jk ∑ a ji b=ik
i =1
a j1b1k + a j 2b2k +  + a jnbnk

 b1k 
b 
Aj∗B∗k = ( a j1 a j 2  a jn )  2k  → (1 x n )( n x 1) =
(1 x 1) scalar =
c jk
  
 
 bnk 
 You should recognize this as the dot product introduced to you in earlier courses, usually written in the form
 a j1   b1k 
a   
b
c jk = Aj∗ ⋅ B∗k =   ⋅  2k= a j1b1k + a j 2b2k +  + a jnbnk
j2
(dot product)
    
   
 a j 2   bnk 
 The dot product can therefore be viewed as the matrix product of a row vector (on the left)
with a column vector (on the right).
 In more standard notation, if u ,v ∈  n are column vectors, then
 v1 
v  n
T
u ⋅ v= u v= (u1 u2  un )  2 = ∑u v i i where uT is the transpose of u (see later slides)
 i =1
 
 vn 
Matrix Multiplication Examples
 Matrix Multiplication (4)
Column View of Matrix Multiplication
 An alternative view of matrix multiplication is as follows.
( m x n ) and B =
Let A = ( n x p ). Define B∗k ∈  n to be the k th column of B (k =
1,..., p).
 | |  |  |   | |  |  | 
B
AB A=   AB  AB∗ p  ≡ C
 ∗1 B∗2  B∗k  B∗ p   ∗1 AB∗2  AB∗k
 | |  |  |   | |  |  | 
 
 Thus the k th column of B is the matrix product AB∗k ∈  n .
 Notation Comment: Kreyszig denotes the k th column of B as bk (also standard, but ambiguous).

Example: Computing Matrix Products Columnwise

 Matrix Multiplication is not Commutative
 In general AB ≠ BA (although it can sometimes be true that AB = BA). Matrix multiplication is thus non-commutative.
 In the special circumstance that AB = BA we say that the matrix operators A and B commute.

Example: Non-Commuting Matrices

Example: Commuting Matrices Note: The identity matrix I always commutes with any other matrix
IA
= AI
= A
1 0 0  2 4 0

A =0 1 0  B  3 1 0   1 0  a11 a12   a11 a12  1 0   a11 a12 
1 0 2  −1 −4 1  =  a
0 1  a22  =  0 1 
a21 a22  a a22 
     
21
    21

 2  4  0 2 4 0 IA IA A
   
AB =A  3  A  1  A  0    3 1 0 
  −1   −4   1    0 −4 2 
        
 1 0 0 2 4 0
   
BA =B  0  B  1  B  0   3 1 0
 
 1 0 2  0 −4 2 
        
∴ AB = BA
Algebra of Matrix Multiplication
 Matrix Transpose
Def: The Transpose of an m x n matrix A is an n x m matrix AT given by switching the identity
of the rows and the columns of A, i.e. row 1 of AT is column 1 of A, row 2 of AT is column 2 of A, etc.

( A ) = ( A)
T
ij ji
row i , column j entry of AT =row j , column i entry of A

Examples
 −1 3 0 
1 2 3
A =  A  0= 0 1 A (1 2 3 )
4 5 6  
   8 9 2
 
1 4   −1 0 8  1 
AT = 2 5 AT  3= 0 9 AT  2 
     
3 6  0 1 2  3
     

Rules for Matrix Transposition

Note in (d) how transposing a product of matrices leads to a

“reverse order” product of transposes (homework exercise)
 Another Motivation for Definition of Matrix Multiplication
 Suppose we transform variables through a matrix multiplication. Consider the simple case of 2 x 2 matrix operators.
 x  b b12  w1   b11w1 + b12w2 
x =Bw ↔  1  = 11  w  =  b w + b w  B transforms w into x
 x2   b21 b22  2   21 1 22 2 

 y  a a12  x1  a x + a x 
y =Ax ↔  1  = 11 =  11 1 12 2  A transforms x into y
 y2   a21 a22  
 x2   a21 x1 + a22 x2 
 Question: Is there a single matrix C that relates y to w directly?
Answer: YES. y Cw
= = ABw provided we define matrix multiplication as we have!

y1 = a11 x1 + a12 x2 = a11 ( b11w1 + b12w2 ) + a12 ( b21w1 + b22w2 )

y2 = a21 x1 + a22 x2 = a21 ( b11w1 + b12w2 ) + a22 ( b21w1 + b22w2 ) →

 y1   a11b11 + a12b21 a11b12 + a12b22   w1 

y1 = ( a11b11 + a12b21 ) w1 + ( a11b12 + a12b22 ) w2 → y  = a b + a b  
 2   21 11 22 21 a21b12 + a22b22   w2 
y2 = ( a21b11 + a22b21 ) w1 + ( a21b12 + a22b22 ) w2

 The relationship between y and w is thus y = Cw where

a b + a b a b +a b 
C =  11 11 12 21 11 12 12 22  = AB
 a21b11 + a22b21 a21b12 + a22b22 
2 2
c11 = ∑ a1i bi1 =
a11b11 + a12b21 = A1∗ ⋅ B∗1
i 1 =i 1
c12 = ∑ a1i bi2 =⋅
a11b12 + a12b22 = A1∗ B∗2
2 2
c21 = ∑ a2i bi1 =
a21b11 + a22b21 = A2∗ ⋅ B∗1
i 1 =i 1
c22 = ∑ a2i bi2 =⋅
a21b12 + a22b22 = A2∗ B∗2
 Symmetric and Skew-Symmetric Matrices
 Def: The square matrix A is symmetric if AT = A.
 Note that rectangular matrices can never be symmetric since A and AT would have different dimensions and
therefore could not possibly be equal. Symmetry applies to square matrices only.

 −1 2 3  −1 2 3
A  2 5 8  →
= A  2
T
5 8  = A (symmetric)
3 8 6  3 8 6 
 
1 0 0 1 2 3
A  2 1 0  → =AT  0 1 0  ≠ A (non-symmetric)
3 0 0  0 0 0 
 

 Def: The square matrix A is skew-symmetric if AT = − A.

 Note that a skew-symmetric matrix must have only zeros along the main diagonal.
 0 −2 −3   0 2 3
A=  2 0 −8  → AT =  −2 0 8  = − A (skew-symmetric)
   
3 8 0   −3 −8 0 
   
 1 −2 −3   1 2 3
A = 2 0 −8  → AT = −2 0 8  ≠ − A (not skew-symmetric)
3 8 0   −3 −8 0 
   
 Diagonal Matrices
 Def: The square matrix D is diagonal if the only non-zero entries are along the main diagonal.
1 0 0
Note: The term diagonal can also apply to rectangular matrices:   → main diagonal is (1 3)
0 3 0

 −1 0 0  1 0 0 0 0 0
D  =0 5 0  I = 
0 1 0 0 0 0 0
  (diagonal)
 0 0 6 0 0 1 0 0 0
     
1 0 0 0 0 3
A =2 1 0  B 0 1 0
  (non diagonal)
0 0 0 0 0 0
   

 A square diagonal matrix D always commutes with other square matrices A of the same dimension.

 d11 0 0   a11 a12 a13 


D = 0 d22 0   A  a21 a22 a23 

 0 0 d33  a 
  31 a32 a33 
 d11 0 0  a11 a12 a13   d11a11 d11a12 d11a13   a11 a12 a13  d11 0 0 
DA =  0 d22= 0   d a
 a21 a22 a23  = 22 21 d22a22 d22a23  a
 21 a22= a23  
 0 d22 0  AD
 0 0 d33     a  0 d33 
  a31 a32 a33   d33a31 d33a32 d33a33   31 a32 a33  0
∴ DA =
AD

 This calculation can easily be generalized for square matrices of any dimension.
Solving Linear Systems of Equations
Gaussian Elimination
Section 7.3 Kreyszig
 ODE example leading to a linear algebraic system
 An example of how Ax = b arises in practice is in the solution of a boundary value problem (ODE)
d2x
ODE: b(t ), solve for x(t ) over t ∈[ 0,1] , b(t ) some twice-differentiable function
dt 2
BCs: = x(0) 0,= x(1) 0 (boundary conditions)
k
 Discretize time into n + 1 equal intervals of length ∆t: = tk ≡ k∆t = ( k 0,1, , n + 1)
n+1
( x1 x2  xn ) (tk ) , bk b (tk )
T
 Solve
= for x where
= xk x=

d2x x − 2 xk + xk −1
2 ( k)
 Use the 2nd − order accurate central difference discretization for t ≈ k +1
dt ∆t 2
d2x 1
k 1:
= 2 ( 1)
t= ( x0 − 2 x1 + x=
2) b1 (=
x0 0 )
dt ∆t 2
d2x 1
k 2:
= 2 ( 2)
t= ( x1 − 2 x2 + x=
3) b2
dt ∆t 2
  
d2x 1
:
k n=
dt 2 ( n)
t
∆t 2
( xn−1 − 2 xn + xn+1 ) = bn (x n +1 = 0 since xn+1 = x ( tn+1 ) = x(1) = 0 )

 The collection of these coupled linear equations leads to

 2 1 
 − ∆t 2 0 0  0 
∆t 2
 
 1 − 2
2 1
0  0  x1   b1 
 ∆t 2 ∆t ∆t 2  x   b 
 1 2 1  2   2 
 0 − 2  0   x3   b3 
 ∆t 2 ∆t ∆t 2   =   ⇒ Ax = b
 1 2   x4   b4 
 0 0 − 2  0      
 ∆t 2 ∆t     
         xn   bn 
 1 2 
 0 0 0  − 2
 ∆t 2 ∆t 
 Linear Systems of Equations
 We now focus on solving Ax = b for x using the method of Gaussian Elimination.
a11 x1 + a12 x2 +  + a1n xn =
b1
a21 x1 + a22 x2 +  + a2n xn =
b2
   = 
am1 x1 + am2 x2 +  + amn xn =bm

 Gaussian Elimination involves elementary row operations on the matrix A that reduces the system
to the upper triangular form
′ x1 + a12
a11 ′ x2 +  + a1′ n xn =
b1′
′ x2 +  + a2′ n x=
a22 n b2′ = b′
→ Ux
   = 
′ xn = bm′
amn
bm′
 Then solve by back substitution xn = etc.
′
amn

Important: Elementary row operations

do not change the solution space for x.
Geometric Interpretation of Solutions to Linear Algebraic Systems

 a11 a12  x1   b1 
a a  x  =  b 
 21 22  2   2 

Unique Solution Infinite # Solutions No Solution

 Augmented Matrices
 Prior to performing "row reduction" to solve Ax = b

form the augmented matrix A = ( A|b ) , so that whatever algebraic manipulations you do
on the LHS of the equation are also done to the RHS:

Example

Ax = b
 0 −2  x1   5   
0 −2 5 
 1 2  x =   3 → A
= 
    
2  1 2 3 

Then perform row reductions on A

Simple Example: Linear System Already in Triangular Form
 The method of back substitution can be implemented immediately on the following system:
 2 3  x1   1 
2 x1 =
+ 3x2 1 ⇔ =
 0 −2     matrix A is already upper triangular
  x2   4 
− 2 x2 =
4

4
 The bottom equation for x2 is uncoupled to the x1 variable, so solve for x2 = = −2
−2

 Then "back substitute" for x2 = −2 into the equation immediately above the lowest equation:
7
2 x1 + 3 ( −2=
) 1 → 2 x1= 7 → x1=
2

7
 x1   
 The final solution is =x =  2
x
 2  −2
 
 Gaussian Elimination and Row Operations: A Simple Example

The system of equations now has an associated lower triangular matrix and can now easily be solved:
x2 = −2
12
2 x1 + 5 ( −2 ) = 2 → x1 = = 6
2 Important Point: Row operations do not change
the solution space x
x   6 
∴ x=  1 =  
 x2   −2 
Row Operations and Row Equivalent Systems

Matrices effectively replace "equations"

 Elementary Matrices (1)
 Every elementary row operation on a matrix A can be effected by left multiplication of an elementary matrix E .
Here we demonstrate the general form of each elementary matrix type for n × n square matrices.
 We will make use of elementary matrices later when we discuss matrix inverses A−1 and their construction.

Elementary matrices: n × n square matrices Suppose A is an n × n matrix. Elementary matrices E that effect a row
operation can be constructed by making one modification to the n × n identity matrix I . Here we demonstrate the form
of E for 3 × 3 matrices.
0 0 1
 
(a) Row Interchange
= : E  0 1 0  → EA results in the matrix A with rows 1 and 3 interchanged.
1 0 0
 
 E is formed by exchanging rows 1 and 3 of the identity matrix I .
 For the general n × n case, exchanging rows i, j of the identity matrix I produces an elementary matrix E such that
EA is the matrix A with rows i and j interchanged.

0 0 1   1 −2 4   4 3 3 
    
 Examples: EA  0
= 1 08 2 = −2   8 2 −2  ( rows 1,3 exchanged )
1 0 0   4 3 3   1 −2 4 

0 1 0   1 −2 4   8 2 −2 
    
= EA  1 0 0   8 2 −= 2   1 −2 4  ( rows 1, 2 exchanged )
0 0 1   4 3 3   4 3 3 

 Elementary Matrices (2)
1 0 0
 
(b) Multiplying a row by a= scalar: E  0 k 0  → EA results in the matrix A with row 2 multiplied by k .
0 0 1
 
 E is formed by multiplying row 2 of the identity matrix I by k .
 For the general n × n case, multiplying row i of the identity matrix I by k produces an elementary matrix E such that
EA is the matrix A with row i multiplied by k .

 1 0 0   1 −2 4   1 −2 4 
    
 Example: EA =  0 4 0   8 2 =−2   32 8 −8  ( row 2 multiplied=
by k 4)
0 0 1 4 3 3   4 3 3 
    

 1 0 0
 
(c) Adding a multiple of one row to another
= : E  0 1 0  → EA adds k times row 1 of A to row 3 of A.
k 0 1
 
 E is formed by placing the multiplicative factor k in entry ( 3,1) of I → ( I )31 =
k.
 For the general n × n case, placing the factor k in the off-diagonal entry ( i, j ) , i ≠ j , of the identity matrix I
produces an elementary matrix E such that EA is the matrix A with k × ( row j ) added to row i.

 1 0 0   1 −2 4   1 −2 4 
    
 Example:=
EA  0 1 0   8 2 =−2   8 2 −2  ( 2 ∗ ( row 2 ) added to row 3)
    
 0 2 1   4 3 3   20 7 −1 
 Overdetermined and Underdetermined Systems
 Consider Ax = b where A is m x n
 Def: Ax b is an overdetermined system if m > n
This means more equations than variables and usually is inconsistent (no solutions)
Example (=
m 2,= n 1) x1 0
= → No solution!
x1 = 1
 Def: Ax b is an underdetermined system if m < n
This means fewer equations than variables and usually leads to an infinite number of solutions
Example (m = 1, n =
2) x1 + x2 =0 → x1 = − x2
 x1   − x2   −1 
Infinite number of solutions of the form= x =  =  x2  1 
x
 2  2 x  
There are an infinite number of solutions since x2 is an arbitrary scalar

 Def: Ax = b is an consistent system if there exists at least 1 solution.

The previous example of the underdetermined system is an example of a
consistent set of equations.
Gaussian Elimination Example: Infinite # Solutions
Gaussian Elimination Example: Infinite # Solutions
 Pivot variables and free variables
Pivot variables and free variables
 After row reducing a matrix A to an upper triangular form U , we may classify the variables in x as either a
pivot variable or a free variable. Assume the row-reduced upper triangular matrix U is arranged so that the
first non-zero entry in each row occurs in a column to the right of the first non-zero entry in each of the rows above it.

 Pivot variables: The first non-zero entry in each row of U is called a pivot. The components of x in the
matrix multiplication Ax that multiply the pivots are called the pivot variables. In the previous example the pivot
variables are {x1 , x2 }.

 Free variables: The non-pivot components of x are called free variables ({ x3 , x4 } in the previous example). Any
consistent linear system Ax =b → Ux =b′ with 1 or more free variables will have an infinite number of solutions owing
to an infinite number of solutions to the associated homogeneous equation Ax = 0.

 General solution: Write the pivot variables in terms of the free variables to form the general solution vector
x = ( x1 , x2 , , xn ) .
T
 Pivots and free variables: previous example revisited
Previous Example
 After row reducing Ax = b to the (consistent) upper triangular form
 x1 
 3 2 2 −5     8 
   x2   
 0 1 1 −4  x  = 1
 0 0 0 0  3   0 
  x  
 4
we solve for the pivot variables { x1 , x2 } in terms of the free variables { x3 , x4 } :
x2 =− x3 + 4 x 4 + 1
3x1 + 2 x2 =
−2 x3 + 5x4 + 8 → (substitute for x2 using above eqn)
x1 =− x4 + 2

 The general solution x can then be expressed as

 x1   − x4 + 2  2 0  −1 
         
x  − x + 4 x + 1 1 −1 4
x =  2=  3 4
= + x3   + x 4  
 x3   x3  0 1 0
         
 x4   x 4  0
 0
 
 1
particular solution homogeneous solution

 Since x3 and x4 are arbitrary, this linear system has an infinite number of solutions.
Gaussian Elimination Example: Inconsistent System
Gaussian Elimination Example: Unique Solution (square system)
Gaussian Elimination Example: Unique Solution (square system)
 Gaussian Elimination Example: Unique Solution (square system)

 This 3 x 3 system of linear equations thus has a unique solution.

 It is typical for square systems to have a unique solution, but as with rectangular systems you
can have no solution or an infinite number of solutions. You never, for example,
have 2 solutions, 3 solutions, etc. Any finite number of solutions is either 0 or 1.
 In rare circumstances you can have an infinite number of solutions (previous example).
This rare circumstance requires a special conspiracy among the RHS entries in the vector b
Vector Spaces
Determinants and Matrix Inverses
Sections 7.6, 7.8 Kreyszig
 Definition of the Determinant
 The determinant of a square matrix A, denoted as det ( A ) or A , is a number that has many interpretations and can
be used as a quick test for whether the n × n linear system ( Ax = b ) has a unique solution, among many other things.
 The computation of det ( A ) can be done using a method called cofactor expansion, which is a somewhat complicated
topic that we will not cover in depth in this course. Instead, we shall describe formulas to calculate det ( A ) only for 2 × 2
and 3 × 3 matrices, which will be useful in a variety of ways.

2 × 2 matrices
a b a b
Def: The determinant of a 2 × 2 matrix A =   is det ( A ) = ad − bc
=
c d c d
1 2
Example = (1)( 4 ) − ( 2 )( 3) =
−2
3 4
 As we will discuss, because det ( A ) ≠ 0 the matrix A is invertible, and solutions to any linear algebraic system are unique:
1  d −b  1  4 −2 
Ax = A−1b
b → x= where A−1 =   . In this example A −1
=
−  .
det ( A )  −c a  2  −3 1 

 2 −1
Example A=  → det = ( A ) ( 2 )(1) − ( −1)(=
−2 ) 0
 −2 1 
 For this matrix A−1 does not exist, and any linear system Ax = b either has no solutions or an infinite # of solutions.
 Note also that row reducing this matrix (row 2 → row 2 + row 1) yields
 2 −1  2 −1
A=  →   , so Ax b is consistent
= only if the RHS vector satisfies b1 + b2 0.
 −2 1   0 0 
 Cofactor Expansions
 The determinant of a general n × n matrix A can be computed by the method of cofactor expansions.
Theorem The determinant of any n × n matrix A is given by either of the following formulas:
n n
(a) det ( A ) ∑ a C ≡ ∑ ( −1)
i+ j
=ij ij aij M ij (cofactor expansion along row i )
=j 1 =j 1
n n
(b) det ( A ) ∑ a C ≡ ∑ ( −1)
i+ j
=ij ij aij M ij (cofactor expansion along column j )
=i 1 =i 1

( −1)
i+ j
where Cij = M ij (the cofactor of entry aij ), and M ij (the minor of entry aij ) is the determinant of
the matrix formed by deleting row i and column j.

3 2 1
 
Example A =  1 2 3  . Compute the cofactors C11 , C21 , and C32 .
0 1 0
 
3 2 1
2 3
( −1) M 11 = ( −1) 1 2 3 ==
1+1 2
C11 = −3
1 0
0 1 0
3 2 1
2 1
( −1) ( −1) 1
2 +1 3
C21 = M 21 = 2 3 =
− 1
=
1 0
0 1 0
3 2 1
3 1
( −1) M 32 =
( −1) 1
3+ 2 5
C32 = 2 3 =
− −8
=
1 3
0 1 0
 3x3 Determinants
 a11 a12 a13 
 
Example A =  a21 a22 a23  . Compute det ( A ) using cofactor expansions along row 1 and column 3.
a a32 a33 
 31

 Along row 1 the cofactor expansion is

a a a a23 a a22
det ( A ) =
( −1) a11 22 23 + ( −1) a12 21 + ( −1) a13 21
1+1 1+ 2 1+ 3

a32 a33 a31 a33 a31 a32

a22 a23 a a23 a a22
= a11 − a12 21 + a13 21
a32 a33 a31 a33 a31 a32
= a11 ( a22 a33 − a23 a32 ) − a12 ( a21a33 − a23 a31 ) + a13 ( a21a32 − a22 a31 )

 Along column 3 the cofactor expansion is

a21 a22 a11 a12 a11 a12

det ( A ) = ( −1) + ( −1) + ( −1)
1+ 3 2+3 3+ 3
a13 a23 a33
a31 a32 a31 a32 a21 a22
a21 a22 a a12 a a12
= a13 − a23 11 + a33 11
a31 a32 a31 a32 a21 a22
= a13 ( a21a32 − a22 a31 ) − a23 ( a11a32 − a12 a31 ) + a33 ( a11a22 − a12 a21 )

 Remember that you will get the same result for det ( A ) using a cofactor expansion along any row or column.
 Determinant Example
3 2 1
 
Example A =  1 2 3  . Compute det ( A ) using cofactor expansions along row 1 and row 3.
0 1 0
 

 Row 1 cofactor expansion

2 3 1 3 1 2
det ( A ) = ( −1) ( 3) + ( −1) ( 2) + ( −1) (1)
1+1 1+ 2 1+ 3

1 0 0 0 0 1
=( 3)( 0 − 3) − ( 2 )( 0 − 0 ) + (1)(1 − 0 ) =−8
∴ det ( A ) =
−8

 The row 3 cofactor expansion will give the same answer:

2 1 3 1 3 2
det ( A ) = ( −1) ( 0) + ( −1) (1) + ( −1) ( 0)
3+1 3+ 2 3+ 3

2 3 1 3 1 2
=0 − 8 + 0 =−8

 Observe that a cofactor expansion along row 3 is simpler to calculate because all but one of the entries are zero.
 Theorems about the Determinant (1)
Theorem The determinant of an n × n diagonal, lower triangular, or upper triangular matrix A
n
( A)
is the product of its diagonal entries: det= ∏
= a
i =1
ii a11a22  ann

Examples

−3 0 0 0
0 2 0 0
=( −3)( 2 )(1)( −1) =6
0 0 1 0
0 0 0 −1

3 0 0
−9 8 0 =( 3)(8)( −1) =−24
3 4 −1

3 4 −9
0 =
6 −3 ( 3)(=
6 )( 2 ) 36
0 0 2
 Theorems about the Determinant (2)
Theorem Let A′ denote the matrix resulting from an elementary row operation on an n × n matrix A.
Elementary row operations have the following effect on det ( A ) :
(a) det ( A ) =det ( A′ ) if a multiple of one row is added to another (determinant is invariant to this operation).
(b) det ( A ) k det ( A′ ) if a single row or column is multiplied by a scalar k ≠ 0.
(c) det ( A ) = −det ( A′ ) if any two rows or columns are interchanged.
Comment: With this theorem you can use row reduction to compute det ( A ) .

1 2
Example A =   . Calculate det ( A ) by row reducing A to an upper triangular matrix.
3 4
Solution: Subtracting 3 ( row 1) from row 2 does not alter the determinant:

1 2 1 2 
=A   A′ 
→ =  det ( A′ ) = det ( A )
3 4  0 −2 

∴ det ( A ) =(1)( −2 ) =−2

 This answer can be verified directly using the well-known formula for 2 × 2 matrices:
1 2
(1)( 4 ) − ( 2 )( 3) =
= −2
3 4
 Example: Determinant by Row Reduction
3 2 1
 
Example A =  1 2 3  . Compute det ( A ) using elementary row operations.
0 1 0
 
Solution: Begin by interchanging rows 2 and 3
3 2 1 3 2 1
   
A= 1 2 3 → A′ = 0 1 0 det ( A′ ) = −det ( A )
0 1 0 1 2 3
   
1
 Subtract ( row 1) from row 3
3
 
3 2 1  3 2 1
   
A′ =  0 1 0  → A′′ =  0 1 0  det ( A′′ ) = det ( A′ ) =−det ( A )
1 2 3  4 8
  0 
 3 3
4
 Subtract ( row 2 ) from row 3
3
   
3 2 1 3 2 1
   
A′′  0 1 0  = → A′′′  0 1 0  det
= ( A′′′) det ( A′′) = −det ( A )
 4 8  8
0  0 0 
 3 3  3
8
∴ det ( A ) =−det ( A′′′ ) =− ( 3)(1)   = −8
3
 Matrix Inverses
Def: An n × n square matrix A is invertible if there exists another n × n matrix A−1 such that
−1 −1
A= A AA= I,
where I In is the n × n identity matrix.
Theorem If a square matrix is invertible, then A−1 is unique.
Proof: (proof by contradiction) Suppose A has two inverses A1 , A2 , with A1 ≠ A2 . It follows that
A1 A and AA2 I
I= (now multiply second equation on the left by A1 )
→ A1 AA2 =
A1 I =
A1 (now use the fact that A1 A =
I)
→ IA2 = A1 → A2 = A1
 This contradicts our assumption that A1 ≠ A2 . We conclude that A1 = A2 .
a b
Theorem (Inverse of 2 × 2 matrix) If A =   and det ( A ) ≠ 0, then
 c d 
1  d −b  1  d −b 
A−1 =   =  .
ad − bc  −c a  det ( A )  −c a 
Proof: Follows from direct verification that A−1 A = I and the previous uniqueness theorem.

Examples
−1
 2 3 1  1 −3 
  =  
 −2 1  82 2 
−1
 2 4 2 4
  does not exist since =0 ( note that the columns are linearly dependent )
1 2 1 2
 Inverses of Elementary Matrices (1)
Theorem Every square elementary matrix E is invertible, and E −1 is also an elementary matrix.
 Rather than proving this theorem, we will demonstrate it for the case of 3 × 3 elementary matrices.
Suppose A is an n × n matrix, so that every elementary matrix E operating on A is also n × n.

(a) Row Interchange

 Observe that E 2 =I since successive interchanges of the same rows returns the orginal matrix. Thus E −1 =E .
 This same result holds for the general n × n case.

0 0 1   1 −2 4   4 3 3
    
 Example: EA  0
= 1 08 2 = −2   8 2 −2  ( rows 1,3 exchanged )
1 0 0   4 3 3   1 −2 4 

0 1 00 1 0 1 0 0 0 1 0
      
= E2 1 0=01 0 0 0 1 0  =I → E −1 =E =  1 0 0 
0 0 1   0 0 1   0 0 1  0 0 1
  
(b) Multiplying a row by a scalar: Suppose the elementary matrix E multiplies row 2 of a matrix A by a scalar k ≠ 0:
1 0 0
1 0 0  
  1
E 0 k 0
= → E −1  0
= 0
0 0 1  k 
  0 0 1 

 For the general n × n case, the inverse E −1 is formed by taking the reciprocal of each diagonal entry. More specifically,
1
by replacing the diagonal entry k by .
k
 Inverses of Elementary Matrices (2)
(c) Adding a multiple of one row to another: Suppose the elementary matrix E adds 3 × ( row 1) to row 2 of A:
1 0 0  1 0 0
   
3 1
E= 0 → E −1 =
 − 3 1 0 
0 0 1   0 0 1
  
 Observe that if you add k times one row to another, the operation is reversed by subtracting k times that same row.
 For the general n × n case, the inverse E −1 is formed by negating the off-diagonal entry. Specifically, for i ≠ j , if
E is an elementary matrix that effects the adding of k × ( row j ) to row i then
( E )ij =
k → (E )
−1
ij
−k
=

Theorem For any square matrix A the following statements imply one another (if one is true then all are true):
(a) A−1 exists
(b) det ( A ) ≠ 0
(c) The homogeneous equation Ax = 0 has only the trivial solution x = A−1 0 = 0.
(d) The inhomogeneous equation Ax = b has the unique solution x = A−1b.
(e) The reduced row-echelon form R of A is the identity matrix, i.e. elementary row operations
reduce A to R = I.
{ Ei }i =1
r
(f) There exists a sequence of elementary matrices such that Er Er −1  E1 A = I , and thus

A−1 = Er Er −1  E1
(g) The matrix A can be written as a product of elementary matrices: A = E1−1  Er−−11 Er−1
 Matrix Inverses by Gauss-Jordan elimination
 Based on the previous theorem, if A−1 exists then we can multiply A by a sequence of elementary matrices such that
Er Er −1  E1 A = I .
 Since A−1 is unique, this tells us that A−1 = Er Er −1  E1.
 Multiplying the above equation for A−1 by the identity matrix gives
A−1 = Er Er −1  E1 I ,
which tells us that applying the same sequence of row operations on I that reduces A to the identity matrix
results in A−1.

Gauss - Jordan method for constructing A−1

 The Gauss - Jordan algorithm is as follows:

(1) Construct the augmented matrix A = ( A | I ) .
(2) Use elementary row operations on A until A is reduced to the identity matrix I .
Those same row operations will be performed on I in the right-half of the augmented matrix.
(3) The end result of Gauss-Jordan elimination produces A−1:
(A| I) → ( I | A−1 )
( Er Er −1  E1 A | Er Er −1  E1 I ) =
 Matrix Inverse Example (1)
 3 0 −3 
−1  
Example Use the Gauss-Jordan method to find A for A =  −6 1 3  .
 3 0 −2 
 

 3 0 −3 1 0 0
 
 Begin with the augmented matrix A = ( A | I ) =  −6 1 3 0 1 0
 3 0 −2 0 0 1 

 3 0 −3 1 0 0  1 0 0
   
( E1 A | E1 I ) =
 0 1 −3 2 1 0  , E 1 =  2 1 0  row 2 → row 2 + 2 ( row 1)
 3 0 −2 0 0 1  0 0 1
   

 3 0 −3 1 0 0   1 0 0
( E2 E1 A | E2 E1 I ) =  0 1 −3 2 1 0  , E2 = 
 0 1 0

row 3 → row 3 − ( row 1)
 0 0 1 −1 0 1   −1 0 1 
   

 1  1 
 1 0 − 1 0 0   0 0
3 3
    1
( E3 E2 E1 A | E3 E2 E1 I ) =
 0 1 −3 2 1 0  , E 3 =
 0 1 0  row 1 → ( row 1) (cont. next slide)
 0 0 1 −1 0 1   0 0 1 3
   
   
 Matrix Inverse Example (2)
Example (cont.)

 2 
1 0 0 − 3 0 1 1 0 1
   
( E4 E3 E2 E1 A | E4 E3 E2 E1 I )=  0 1 −3 2 1 0  , E4 = 0 1 0 row 1 → row 1 + row 3
 0 0 1 −1 0 1  0 0 1
   
 

 2 
 1 0 0 − 3 0 1 1 0 0
   
( E5 E4 E3 E2 E1 A | E5 E4 E3 E2 E1 I ) =
 0 1 0 −1 1 3  , E5 =
 0 1 3 row 2 → row 2 + 3 ( row 3)
 0 0 1 −1 0 1  0 0 1
   
 

 The inverse A−1 is given by the product of these elementary matrices (verify for yourselves), which you
will see agrees with the right half of the augmented matrix above after row-reducing A to I :

 2 
 − 3 0 1
 
A−1= E5 E4 E3 E2 E1=  −1 1 3 
 −1 0 1 
 
 
Linear Independence, Matrix Rank, Basis
Sections 7.4, 7.9 Kreyszig
 Linear Independence of Vectors
Def: A collection of vectors {v1 , v2 , , vn } in a vector space V is linearly dependent if there exists a set of scalars,
{c1 ,c2 , ,cn } , not all zero, such that
c1v1 + c2v2 +  + cnvn =
0
If the only solution is the trivial solution= ( k 1, ,n) , then the vectors are linearly independent.
ck 0=

Examples (linearly dependent vectors)

Example 1: Any two vectors {v1 , v2 } that are parallel (proportional to one another) are
linearly dependent
= . Suppose v1 kv2 for some scalar k ≠ 0. Then
v1 − kv2 =
0
and thus
c1v1 + c2v2 =
0
has the nontrivial solution c1 = 1, c2 = −k.

Example 2:
 1   −1   −3  
 
The vectors {v1 , v2 , v3} =  2  ,  3  ,  4   are linearly dependent in  3 since
     
 3   1   −1  
v1 − 2v2 + v3 = 0
 Linear Independence Test
Matrix formulation of linear independence test
 Observe that the defining equation for the test for linear dependence
c1v1 + c2v2 +  + cnvn = 0
is equivalent to the linear system
= Ac 0 , where vk ∈  m is the k th column of the m × n matrix A.
 | |  |  c1   0 
 v v  v    =   
 1 2 n    
 | |  |  c   0 
  n   

 In this view, the collection of vectors {v1 , v2 , , vn } with vk ∈  m is linearly dependent if the
m × n linear system Ac 0 has a nontrivial solution c ≠ 0.

Linear independence test for square systems

 For n × n square systems, where v k ∈  n , if you are testing the linear dependence of n vectors
in the vector space V =  n , then the equation
Ac = 0
has only the trivial solution c = 0 (vectors are linearly independent) if any of the following are true:
(a) A−1 exists → c = A−1 0 = 0
(b) det ( A ) ≠ 0
(c) After a series of row reduction steps A → I (n × n identity matrix is the reduced row echelon form)
 Example 1: Linear Independence
Example 1 : linearly independent vectors
The following set of vectors in V =  4 are linearly independent:
{v1 , v2 , v3} ={(1 −1 0 1) , ( 0 2 1 0 ) , ( −1 −1 1 1) . }
T T T

The linear independence test in matrix form, with each vk a column of the matrix A , is
1
0 −1  0
 −1  c1   
2 −1     0 

Ac = 0 →  c2  = ,
0 
1 1   0  
   c3   
1
0 1 0
which after row reductions becomes the homogeneous linear system
1 0 0 0
c
   c1   0 
0
 1 0   1   0  c  =  0 .
c = →  2  
0 0 1   2   0  c  0
  c    3  
0 0 0  3   0 
 Example 2: Linear Independence
Example 2 : linearly independent vectors
The following set of vectors in V =  4 are linearly independent:
{v1 , v2 , v3} ={(1 2 2 1) , ( 2 −1 0 0 ) , (1 1 0 1) , ( 0 0 1 −1) . }
T T T T

The linear independence test in this case is a 4 × 4 linear system:

1 2 1 0  c1   0 
 2 −1 1  c   0 
0 
Ac = 0 →  2
= .
2 0 0 1  c3   0 
    
1 0 1 −1  c4   0 
 det ( A ) = −4
 Since det ( A ) ≠ 0 it follows that A −1 exists and c = A −1 0 = 0 is the only solution.
 We conclude that the given set of vectors is linearly independent.
 Theorems about the Determinant (3)
Theorem det ( A ) = 0 if one or more of the following are true:
(a ) A has a row of zeros or a column of zeros.
(b) The columns or rows of A are linearly dependent.
(c) There exists a nontrivial ( x ≠ 0 ) solution to the homogeneous linear equation Ax = 0.
Comments:
(a) follows directly from the definition of cofactor expansions.
(b) follows from the row reduction method of computing a determinant. If the rows are linearly dependent then
linear combinations of the rows, which does not change det ( A ) , will lead to a row of zeros.
(c) follows from the fact that A−1 exists if det ( A ) ≠ 0, and thus x =
A−1 0 =
0 would be the unique solution.
If det ( A ) = 0, then row operations on A will lead to a row-reduced matrix with one or more rows of zeros,
and thus there will be free variables in the x vector and non-trivial solutions to Ax = 0.

Examples
9 0 −4
3 0 7 =0 since column 2 has all zero entries
−5 0 1

−1 3 2
0 4 8 = −2 ( row 1) → rows are linearly dependent
0 since row 3 =
2 −6 −4
 Spanning Sets
Linear Combinations
Def: A linear combination of vectors {v1 , v2 , , vn } is any weighted sum of the form
c1v1 + c2v2 +  + cnvn
for a set of scalars {c1 ,c2 , ,cn }.
Spanning Sets
Def: The collection of vectors S {v1 , v2 , , vn } is a spanning set for a vector space V if any vector v ∈V
can be expressed as a linear combination
v = c1v1 + c2v2 +  + cnvn
for an appropriate set of scalars {c1 ,c2 , ,cn }. Any such set of vectors is said to span V .

Examples
(a) The standard (natural)
= basis E {e , e ,,=
1 2 e } spans V
n
n
= , where e 1 (1 0  0 ) , etc.
T

(b) The vectors S = {e , e } do not span V =

1 2  3 since any vector v = (v1 v2 v3 ) ∈  3 with v3 ≠ 0
T

cannot be written as a linear combination

 c1 
c ,
c1 e 1 + c2 e 2 =
 2
0
 
which will always have zero for the third component.
 Bases
Basis for a Vector Space
Def: A set of vectors B {b1 , b2 , , bn } is a basis for a vector space V if every v ∈V can be expressed as
v = c1b1 + c2 b2 +  + cn bn
for a unique set of scalars {c1 ,c2 , ,cn }.

 The set B = {b1 , b2 , , bn } will be a basis for the vector space V if it satisfies both of the following:
(1 ) The set B spans V
(2) The set B is linearly independent
A basis is a minimal spanning set
 A basis B = {b1 , b2 , , bn } can be thought of a minimal spanning set for a vector space V in that it
contains the minimal number of vectors required to span V .
 If the set B contained "superfluous" vectors, not necessary for the spanning property, then B would
be a linearly dependent set.
 If the set B contained too few vectors (e.g. m vectors
= for V  n with m < n) then it would not have
the spanning property.
Dimension
Def: The dimension of a vector space V , written as dim (V ) , is the number of vectors in any basis B = {b1 , , bn } , where n could be infinite.
Def: If dim (V ) = n is a finite number then V is a finite dimensional vector space. If dim (V ) = ∞ then V is an infinite dimensional vector space.
 It can be shown that dim (V ) is a unique number (possibly infinity) for any vector space.
 Examples: (i) dim (  m ) = m

| dk f 
(ii) V =  f ( x ) all derivatives k exist for x ∈ ( −∞ , ∞ )  → dim (V ) = ∞
dx
 
∞
f ( ) (0) k
k

∑
Functions f ( x ) ∈V will generally have a Maclaurin Series f ( x ) =
k =0 k!
x

You can think of the infinite set of monomials B = {1, x , x 2 ,} as a basis for this space.
 Bases for Euclidean Vector Spaces
Example
= V m (m − dimensional Euclidean space)
Suppose S {v1 , , vn } is any collection of n vectors with each v k ∈  m . Let v k be the
k th column of an m × n matrix A.
Case 1: n < m
 S cannot be a basis since span{S} ≠  m .
 There will always be vectors b ∈  m such that Ax =
b has no solution, which means that
all possible linear combinations of the columns of A (the set S) cannot produce all possible b ∈  m .
 To see this, note that the row-reduced matrix A → U (upper triangular) will always have at least
1 or more rows of zeros since there are more rows than columns. Therefore, some b can
always be chosen to make Ax = b inconsistent.
Case 2: n > m
 S cannot a basis since S will necessarily be a linearly dependent set.
 The linear independence test Ac = 0 will always involve 1 or more free variables in the
( c1 , , cn )
T
solution c , which means there is always some non-trivial set of weights
= c ≠ 0 such that
c1v1 + c2v2 +  + cnvn = 0.
Case 3: n = m
 S will be a basis if and only if det ( A ) ≠ 0. This means that S =
{v1 , , vn } is a linearly independent set.
 If det ( A ) ≠ 0 then A−1 exists, and=
Ax b always has a unique solution
= x A−1b for all b ∈  m .
This means that span{S} =  m .
 S will then be a set that both spans  m and is linearly independent, and thus a basis.
 Four Fundamental Subspaces of a Matrix
Def: A subspace of a vector space V is a subset S ⊂ V that is also a vector space.
 a11 a12  a1n 
a  a2n 
 Every m × n matrix A =  21 a22
aij  = has 4 fundamental subspaces associated with it.
     
 
 am1 am2  amn 
Column Space R ( A ) Suppose A :  n →  m (i.e. A is m × n ) .

{y ∈ m | Ax =
 R ( A) = y for some x ∈  n }.
 R ( A ) is the span of the columns of A and is called the column space.

Row Space R ( AT ) AT :  m →  n (i.e. A

T
is n × m ) .

 R ( AT ) =
{x ∈ n | AT y =
x for some y ∈  m }.
 R ( AT ) is the span of the rows of A and is called the row space. It is equivalent to the span of the columns of AT .
Nullspace N ( A )

{x ∈ n | Ax =
 N ( A) = 0}. This subspace is called the nullspace of the matrix A.
 N ( A ) is the subspace of all vectors in the domain of A that map to the null vector 0 ∈  m .
Left Nullspace N ( AT )

 N ( AT ) =
{y ∈ m | AT y =
0} =
{y ∈ m |y T A =
0}. This subspace is called the left nullspace of the matrix AT .
 N ( AT ) is the subspace of all vectors in the domain of AT that map to the null vector 0 ∈  n .
 Properties of the Four Fundamental Subspaces
{ }
 The subspaces R ( A ) , R ( AT ) , N ( A ) , N ( AT ) are considered "fundamental" because the domain  n and range  m
of any m × n matrix, A :  n   m , can be decomposed in terms of them in the following sense:
Direct Sums
Def: A vector space V can be written as the direct sum V = V1 ⊕ V2 of subspaces, with V1 ∩ V2 = {0} , if every vector v ∈V
can be expressed uniquely as v =
v1 + v2 for some v1 ∈V1 and v2 ∈V2 (V1 and V2 have only the zero vector 0 in common).
 This concept can be extended to direct sum decompositions of many subspaces {V1 , ,Vs } , where Vi ∩=
Vj {0} ( i ≠ j ) :
V = V1 ⊕ V2 ⊕  ⊕ Vs
if every v ∈V can be written uniquely as a linear combination v = v1 + v2 +  + v s for some v1 ∈V1 , , v s ∈Vs .
Orthogonal Subspaces
Def: Two subspaces V1 ⊂ V and V2 ⊂ V of a vector space V are orthogonal (V1 ⊥ V2 ) if v1 ⊥ v2 ( v1 ⋅ v 2 =
0) for
every v1 ∈V1 and v2 ∈V2 .
Def: Two subspaces V1 ⊂ V and V2 ⊂ V of a vector space V are orthogonal complements
= if V1c V=
2 and V2
c
V1 .
 Note that this implies V = V \ V1 ( the set containing everything in V not in V1 ) .
V1 ⊕ V2 since it follows that V2 =
Theorem: R ( A ) ⊥ N ( AT ) and R ( AT ) ⊥ N ( A ) .

Direct Sums for  m and  n

Theorem: For any m × n matrix A :  n   m , the domain  n and range  m can be written as direct sums
of the four fundamental subspaces:
 n R ( A ) ⊕ N ( AT ) R ( A ) = N ( AT )
c
(i) = where

R ( AT ) ⊕ N ( A ) where R ( AT ) =
c
(ii)  m = N ( A)
 This result is often called the Fundamental Theorem of Linear Algebra.
 Rank of a Matrix
Finding R ( A ) and R ( AT )

 To find bases for the column space R ( A ) and the row space R ( AT ) you perform elementary row operations
A → U until you achieve an upper triangular matrix U , or A → R (R =
reduced row echelon form).

 1 u12  u1n 
 a11 a12  a1n   

a a22  a2 n 
 0 0 1  u2n 
=A  21 = → U 0 0 0 1 0  u3n 
       
      
 am1 am2  amn  0 0 0 0 0  umn 

 Identify the rows and columns with pivots (indicated by 1' s above).
 A basis for R ( AT ) is the set of rows with pivots in either U or the corresponding rows of A.
 A basis for R ( A ) is the set of columns of the original matrix A associated with pivots.

Finding N ( A ) and N ( AT )
 A basis for N ( A ) is the set of vectors involved in the general solution to Ax = 0.
 A basis for N ( AT ) is the set of vectors involved in the general solution to AT y = 0.
Rank
Def: The rank of a matrix A is defined by rank ( A ) = dim ( R ( A ) ) .

( )
Theorem: dim ( R ( A ) ) = dim R ( AT ) .
 The rank of a matrix is the dimension of either the column space R ( A ) or the row space R ( AT ) , which
is determined by the number of pivots in the reduced row echelon form of A (or as determined by a
row reduction to an upper triangular form U).
 Row Operations Change Span of Columns
Example
 A simple matrix illustrates that elementary row operations change the span of the columns of the original
matrix A. Therefore, in general one cannot use a row-reduced matrix A  U and the pivot columns of U as
as basis for R ( A ) .
 Consider the matrix
1 1 1 1
=A =   U   (row 2  row 2 − 2[row 1])
2 2 0 0
dim ( R ( A ) ) dim
 rank ( A ) 1, so there is 1 pivot and = = ( )
R ( AT ) 1
 Row space: We can use the row with the pivot (row 1) of either A or U for a basis, since elementary row operations
do not alter the span of the rows of A (row 1 for A and U happen to be the same in this simple example). Observe that
R ( AT ) = span{(1 1)}
 1    1  
 Column space: Note that the spans of the columns of A and U are different: R ( A ) = span    ≠ span    = R ( U )
 2    0  
 One must use associated pivot columns of the original matrix A to construct a basis for the column space.
 Alternatively, one could find a basis for R ( A ) using the row space basis for AT , since the columns of A become the
rows of AT .
 Nullspaces: You can easily verify that the nullspaces of A are
 1    2  
N ( A ) span
=    and N ( AT ) span   
 −1    −1  
 As an exercise check that R ( A ) ⊥ N ( AT ) and R ( AT ) ⊥ N ( A ) .
To do this, it is sufficient to show that all the basis vectors in these spaces are mutually orthogonal.
Example: Finding the Four Fundamental Subspaces
Example Find bases for the four fundamental subspaces of the following 3 × 4 matrix:
 1 2 4 0
 
A=  −3 1 5 2 
 −2 3 9 2 
 
 Elementary row operations lead to the reduced row echelon form
 6 4
 1 0 − − 
 1 2 4 0 7 7
 

A= −3 1 5 2

→  0 1 17
R=
2 
   7 7 
 −2 3 9 2  0 0 0
 
 0 

 
Column and Row Spaces
Since there are two pivots (indicated by 1' s) we see that rank ( A ) = 2 , so the bases for the column
space and the row space each have 2 vectors:
 1   2  
    
R ( A ) span  −3  ,  1  
= (columns of original A corresponding to columns of R with pivots)
 −2   3  
    
 6 4  17 2  
R ( AT ) span  1 0 −
= −  , 0 1 =   span{(1 2 4 0 ) , ( −3 1 5 2 )}
 7 7  7 7 
 Example (continued)
Nullspace N ( A )
 We need to find a basis for
= the subspace of solutions to Ax 0=
, which can be obtained from Rx 0:
 6 4
 1 0 − 7 − 7   x1   0 
    
 17 2   x2   0 
Rx = 0 → 0 1 =
 7 7   x3   0 
0 0 0 0    
x 0
  4   
 
 The pivot variables are { x1 , x2 } and the free variables are { x3 , x4 }. Since there are 2 free variables, we know
immediately that dim ( N ( A ) ) = 2.
 6   4 
 7   7 
   
6 4 17 2 17 2
x1 = − x3 − x 4 → x =
x 3 + x 4 , x2 = x3  −  + x4  − 
7 7 7 7 7 7
 1   0 
   
 0   1 
   
 6 17  4 2 
 Thus N ( A ) = span  − 1 0  , − 0 1  .
 7 7  7 7 
 Note: I have written the basis for N ( A ) as row vectors because they are naturally paired with the row space
R ( AT ) according to the fundamental theorem of linear algebra.
 Example (continued)
Left Nullspace N ( AT )

 A basis for the left nullspace is defined by the general solution to AT y = 0

The reduced row echelon form of the transpose matrix AT is
 1 −3 −2  1 0 1
   
2 1 3 0 1 1
= A 
T  →
4 5 9  0 0 0
   
0 2 2  0 0 0
The equation AT y = 0 is then obtained from
1 0 1 0
   y   
0 1 1  1   0 
y =
0 0 0   2   0 
  y  
0 0 0  3   0 

( )
 There are two pivot variables {y1 , y2 } and one free variable {y3 } , so dim N ( AT ) = 1.
 −1 
 
− y 3 , y2 =
y1 = − y3 → y=
y 3  −1 
1
 
 −1  
  
 Therefore N(A ) =
T
span  −1  . Again, you should verify that R ( A ) ⊥ N ( AT ) and R ( AT ) ⊥ N ( A ) .
 1  
  
 1   −1   2   −1 
       
For example, to verify R ( A ) ⊥ N ( AT
) observe that  −=
3  ⋅  −1  0 and =1  ⋅  −1  0.
 −2   1   3  1 
       