Review of Vector Spaces and Matrix Algebra

Linear Systems of Equations

A linear equation in n unknowns (with unknowns denoted by {xj }nj=1 ) is

a1 x 1 + a2 x 2 + · · · + an x n = b

and a linear system of m equations in n unknowns is

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
..
. (1)
am1 x1 + am2 x2 + · · · + amn xn = bm

The quantities aij are called the coefficients of the system and are given constants. The system is
said to be homogeneous if all the bj are zero and non-homogeneous otherwise. A solution is a set of
values for the unknowns {xj }nj=1 which when substituted into the system of equations renders the
equations valid, i.e. the equations are satisfied.

In order to solve a linear system there are three allowable operations called elementary operations
which can be preformed on the system to obtain and equivalent system, i.e., a system with exactly
the same solutions.

1. Interchange the position of the equations.

2. Multiply any equation by a nonzero constant.
3. Add a nonzero multiple of one equation to another equation.

The Augmented matrix of the system (1) is

 
a11 a12 ··· a1n b1
 a21 a22 ··· a2n b2 
.
 
 .. .. .. ..
 . . . . 
am1 am2 · · · amn bm
The analog of the elementary operations on equations for the augmented matrix is the Elementary
Row Operations

1. Interchange the position of the rows.

2. Multiply any row by a nonzero constant.
3. Add a nonzero multiple of one row to another row.

1
A system is called Upper Triangular if aij = 0 for all i > j, i.e., if
 
a11 a12 a13 · · · a1n
 0 a22
 a23 · · · a2n 
0
 0 a33 · · · a3n .
 .. .. .. . . .. 
 . . . . . 
0 0 0 · · · ann

One can easily solve an upper triangular system using Back Substitution.

RREF-Reduced Row-Echelon Form

In the method known as Gaussian Elimination the row operations are applied until one obtains
the Row-Echelon Form:

A matrix is in Row Echelon Form if

1. The first nonzero element in every nonzero row is a 1 (this position is called a pivot).
2. In consecutive nonzero rows, the first entry 1 in the lower row appears to the right of
the 1 in the row above it.
3. Rows with all zeros are at the bottom of the matrix.

For the Gauss-Jordan Method one continues using the row operations to obtain the Reduced
Row-Echelon Form (sometimes called the Row Reduced Echelon Form or RREF) :

A matrix is in Row Reduced Echelon Form if

1. The first nonzero element in every nonzero row is a 1 (this position is called a
pivot).
2. In consecutive nonzero rows, the first entry 1 in the lower row appears to the
right of the 1 in the row above it.
3. Rows with all zeros are at the bottom of the matrix.
4. Above any pivot there are all zeros.

(a) A row of only zero entries is called a zero row and the leftmost nonzero entry of a nonzero row
is called a leading 1. The variable corresponding to the column this lead 1 is in is called a lead
variable. A column containing a leading 1 is called a pivot column.

(b) Any variable that is not a lead variable is called a free variable. These variables are called free
because they can chosen to be any arbitrary number. So in the book they call them α, β, etc.

(c) The number of nonzero rows will be denoted by r. This will be an important number in later
sections.

2
A System of equations is said to be consistent if it has at least one solution. If it has no solutions
then it is called inconsistent. If a system is consistent then there are two possibilities:

1. There is a unique solution, i.e. there is only one solution.

2. There are infinitely many solutions.

◦ A homogeneous system of equations is always consistent and a homogeneous m × n system with

m < n always has infinitely many solutions.

◦ A system of equations is called Overdetermined if there are more equations than unknowns. They
are usually consistent.

◦ A system of equations is called Underdetermined if there are fewer equations than unknowns.
They can be consistent but usually they are not. If they are consistent they they have infinitely
many solutions.

Matrix Arithmetic
Definition 1. 1. An m×n Matrix is a rectangular array of entries (numbers, variables, functions,
etc) with m rows and n columns. The entries are called elements.
 
a11 a12 · · · a1n
 a21 a22 · · · a2n 
A =  ..
 
.. 
 . . 
am1 am2 · · · amn

2. The matrix is said to be square if m = n. For a square matrix the main diagonal is the entries
a11 , a22 , · · · , ann and the sum of the main diagonal elements is called the trace of A.
3. A matrix with n rows and 1 column is called a column vector and a matrix with n columns
and 1 row is called a row vector.
4. Two m×n matrices A and B are said to be equal if and only if aij = bij for all i = 1, · · · , m and
all j = 1, · · · , n, i.e., two matrices of the same size are equal if and only if every corresponding
entry is equal.
5. We denote the set of all m × n matrices by M (m, n) and we define an addition for A, B ∈
M (m, n) by
     
a11 a12 · · · a1n b11 b12 ··· b1n a11 + b11 a12 + b12 ··· a1n + b1n
 a21 a22 · · · a2n   b21
  b22 ··· b2n   a21 + b21
  a22 + b22 ··· a2n + b2n 
.. + .. ..  =  .
 
 .. .. ..
 . .   . .   . . 
am1 am2 · · · amn bm1 bm2 · · · bmn am1 + bm1 am2 + bm2 · · · amn + bmn

That is, the i, j-th entry in the sum of two matrices is the sum of the i, j-th entries of each
matrix.

3
 6. We also define a scalar multiplication on M (m, n) by
 
ka11 ka12 ··· ka1n
 ka21 ka22 ··· ka2n 
kA =  .. ..  .
 
 . . 
kam1 kam2 ··· kamn

7. The matrix with all zero entries is called the Zero Matrix and is denoted by O.
8. Properties of Matrix Addition and Scalar Multiplication
Let A, B and C be in M (m, n) and k1 , k2 be any scalars. Then
(a) (commutative property) (A + B) = (B + A).
(b) (distributive law 1) k1 (A + B) = (k1 A + k1 B).
(c) (distributive law 2) (k1 + k2 )A = k1 A + k2 A.
(d) (distributive law 3) k1 (k2 A) = (k1 k2 )A.
(e) (additive identity) A + O = O + A = A.
(f) (additive inverse) A + (−A) = O.
(g) (multiplicative identity) 1A = A.

Definition 2 (Matrix Multiplication). One more very important operation that can be per-
formed with matrices of appropriate sizes is a matrix multiplication. BUT in order for a pair of
matrices A and B be conformable for the multiplication AB it must be that the number of columns
of A must equal the number of rows of B. Notice this means that it is entirely possible that we
can multiply A times B but we cannot multiply B times A.
In order to describe the operation of A times B let us assume that A is m × s and B s × n so
that the compatibility condition is satisfied. Then the matrix C = AB will be in m × n and all
we need to do is tell you the value of Cij for i = 1, · · · , m and j = 1, · · · , n in order to define the
matrix multiplication:
Xs
Ci,j = ai` b`j .
`=1

This multiplication can be interpreted in terms of inner products of vectors as follows. First we
notice that a matrix can be viewed in two useful ways. On the one hand it can be viewed as a
column of row vectors
 
A1
 A2 
A =  ..  where Ai = ai1 ai2 · · · ain ∈ Rn .
   
 . 
Am

On the other hand it can be viewed as a row of column vectors

 
a1j
 a2j 
where Aj =  ..  Rm .
 
A = A1 A2 · · · An
 
 . 
amj

4
With this notation we can see that the ij-th entry of C = AB is nothing more than the dot (or
inner) product of the i-th row of A with the j-th column of B.

So for example Matrix multiplication allows us to express a system of equations

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
am1 x1 + am2 x2 + · · · + amn xn = bm

in the form
     
a11 a12 ··· a1n x1 b1
 a21 a22 ··· a2n   x2   b2 
Ax = b where A =  .. ..  , x =  ..  , b =  ..  .
     
 . .  .  . 
am1 am2 · · · amn xn bm

Properties of Matrix Multiplication

(Assuming all matrices below are conformable for multiplication)
1. Even if A and B are square matrices, in general AB 6= BA.
2. A(BC) = (AB)C.
3. A(B + C) = (AB) + (AC).
4. (B + C)A = (BA) + (CA).
Another useful operation on matrices is the transpose of a matrix A defined by
 
a11 a21 · · · am1
 a12 a22 · · · am2 
AT =  ..
 
.. 
 . . 
a1n a2n · · · amn

Thus we see that AT ∈ M (n, m) is the matrix obtained from A by interchanging the rows and
columns. Therefore if we have a column vector x then xT is a row vector.

Properties of Matrix Transpose

(Assuming all matrices below are conformable for multiplication)
1. (AT )T = A.
2. (A + B)T = AT + B T .
3. (AB)T = B T AT .
4. (kA)T = kAT .

Definition 3 (Other Special Matrices). Other special square matrices include triangular matri-
ces and the special case of diagonal matrices. Another special type of matrix is a triangular matrix.

5
They can be either upper or lower triangular. An upper triangular matrix is a matrix with all zeros
below the main diagonal and a lower triangular matrix is a matrix with all zeros above the main
diagonal.

A matrix is said to be a diagonal matrix is all the entries off the main diagonal are zeros. The
n × n diagonal matrix with all ones on the diagonal is called the identity matrix denoted In . The
identity matrix satisfies properties in common with the number one. Namely, for any n × n matrix
A we have AIn = In A = A.

An n × n matrix A is said to by Symmetric if AT = A. In orther words aij = aji for all i and j.

For square Matrices the identity matrix allows us to consider the notion of the inverse of a matrix.

Definition 4. An n × n matrix A is called nonsingular (or invertible) if there exists a matrix B so

that AB = BA = In . If B exists we call it the multiplicative inverse of A and write B = A−1 .
If a matrix does not have an inverse we say it is singular.

The Elementary Matrices

Let use define the standard row vectors ej ∈ Rn which has zeros in all but the jth position where
it has a 1. Thus it is given by
h i
ej = 0, 0, · · · , 0, 1j , 0, · · · , 1 .

Notice that using these we can write the identity as

 
e1
 e2 
In =  .. 
 
.
en

The Elementary Matrices are matrices that correspond to the elementary row operations. They are
obtained from the identity matrix in a simply way.

I Ei,j is the identity matrix with rows (or columns) i and j interchanged,

II Ei (α) is the identity matrix where the diagonal entry in row i and column i replaced by α,

III Ei,j (α) is the identity matrix where the entry in row j and column i has been replaced by α.
(The subscripts look backwards in the description of Ei,j (α) but they are not).