DM559

Linear and Integer Programming

Lecture 3
Matrix Operations

Marco Chiarandini
Department of Mathematics & Computer Science
University of Southern Denmark
 Matrices
Vectors
Vectors and Matrices
Outline More on Linear Systems

1. Matrices

2. Vectors

3. Vectors and Matrices

4. More on Linear Systems

2
 Matrices
Vectors
Vectors and Matrices
Outline More on Linear Systems

1. Matrices

2. Vectors

3. Vectors and Matrices

4. More on Linear Systems

3
 Matrices
Vectors
Vectors and Matrices
Matrices More on Linear Systems

Definition (Matrix)
A matrix is a rectangular array of numbers or symbols. It can be written as
 
a11 a12 · · · a1n
 a21 a22 · · · a2n 
 
 .. .. . . .. 
 . . . . 
am1 am2 · · · amn

• We denote this array by a single letter A or by (aij ) and

• we say that A has m rows and n columns, or that it is an m × n matrix.

• The size of A is m × n.

• The number aij is called the (i, j) entry or scalar.

4
 Matrices
Vectors
Vectors and Matrices
More on Linear Systems
• A square matrix is an n × n matrix.

• The diagonal of a square matrix is the list of entries a11 , a22 , . . . , ann

• The diagonal matrix is a matrix n × n with aij = 0 if i 6= j (ie, a square matrix with all the
entries which are not on the diagonal equal to 0):
 
a11 0 · · · 0
 0 a22 · · · 0 
 
 .. .. . . .. 
 . . . . 
0 0 · · · amn

Definition (Equality)
Two matrices are equal if they have the same size and if corresponding entries are equal. That is, if
A = (aij ) and B = (bij ) are both m × n matrices, then:

A = B ⇐⇒ aij = bij 1 ≤ i ≤ m, 1 ≤ j ≤ n
5
 Matrices
Vectors
Vectors and Matrices
Matrix Addition More on Linear Systems

Definition (Addition)
If A = (aij ) and B = (bij ) are both m × n matrices, then

A + B = (aij + bij ) 1 ≤ i ≤ m, 1 ≤ j ≤ n

matrix A matrix B
+ + + 
 
a11 a12 · · · a1n  b11 b12 · · · b1n 
   
 a21 a22 · · · a2n   b21 b22 · · · b2n 
   
  + 
 .. ... .. .   .. .. . . .. 
 Eg:

 . . .. 
  . . . . 
 
am1 am2 · · · amn bm1 bm2 · · · bmn    
3 1 2 −1 1 4
A+B = + =?
0 5 −2 2 −3 1
 

c11 c12 · · · c1n 
 c21 c22 · · · c2n 
 

 .. .. .. .

 element-wise operation

 . . . .. 

cm1 cm2 · · · cmn

matrix C = A + B 6
 Matrices
Vectors
Vectors and Matrices
Scalar Matrix Multiplication More on Linear Systems

Definition (Scalar Multiplication)

If A = (aij ) is an m × n matrix and λ ∈ R, then

λA = (λaij ) 1 ≤ i ≤ m, 1 ≤ j ≤ n

Eg:

−2A =?

element-wise operation

7
 Matrices
Vectors
Vectors and Matrices
Matrix Multiplication More on Linear Systems

Two matrices can be multiplied together, depending on the size of the matrices

Definition (Matrix Multiplication)

If A is an m × n matrix and B is an n × p matrix, then the product is the matrix AB = C = (cij )
with

cij = ai1 b1j + ai2 b2j + · · · + ain bnj .

  
b1j
  b2j 
  
  ..  What is the size of C?
ai1 ai2 · · · ain   . 
bnj

8
 B : p rows q columns
 
 b 11 ... b1 j ; ... b 1q 
 
 
 
 
 .. .. .. .. 
 .. 
 . . . . . 
 
 
 
 b k1 ... bk j ; ... b kq 
 

1j
 

b
 

×
 

i1
 .. .. .. .. .. 

a
 

+
. . . . .

..
 

.+
j  
bk  
×  
k  b p1 ... bp j ; ... b pq 
ai

+
..
.+
j
×
bp Not an element-wise
p
ai
   
operation!
 a 11 ... a 1k ... a 1p   c 11 ... c1 j ... c 1q 
   
   
   
   
 .. .. .. .. ..   .. .. .. .. .. 
 . . . . .   . . . . . 
   
   
   
 ai 1 ; ... ai k ; ... ai p ;   ci 1 ... ci j ; ... ci q 
   
   
   
   
 .. .. .. ..   .. .. .. .. 
 ..   .. 
 . . . . .   . . . . . 
   
   
   
 a n1 ... a nk ... a np   c n1 ... c nk ... c nq 

A : n rows p columns C = A × B : n rows q columns

 Matrices
Vectors
Vectors and Matrices
More on Linear Systems

   
1 1 1   3 4
2 3 0
0 1 5 3
AB = 
1
  1 1 =  
2 4 1 14 
−1 3
2 2 −1 9 −1

(2)(3) + (0)(1) + (1)(−1) = 5

The motivation behind this definition is that it allows to deal conveniently with several tasks in
linear algebra. Think about the way we rewrote a system of linear equations using this definition.

10
 Matrices
Vectors
Vectors and Matrices
More on Linear Systems

• AB 6= BA in general, ie, not commutative

try with the example of previous slide...

 
  3 1
2 1 3
A= B = 1 0 AB is 2 × 2 and BA is 3 × 3
1 2 1
1 1
   
1 2 1 1
A= B= ok sizes but AB 6= BA
3 4 0 1

11
 Matrices
Vectors
Vectors and Matrices
Matrix Algebra More on Linear Systems

Matrices are useful because they provide compact notation and we can perform algebra with them

Bear in mind to use only operations that are defined. In the following rules, the sizes are dictated
by the operations being defined.
• commutative A + B = B + A . Proof?

• associative:
• (A + B) + C = A + (B + C )
• λ(AB) = (λA)B = A(λB) Size?
• (AB)C = A(BC )

• distributive:
• A(B + C ) = AB + AC
• (B + C )A = BA + CA Why both first two rules?
• λ(A + B) = λA + λB
12
 Matrices
Vectors
Vectors and Matrices
Zero Matrix More on Linear Systems

Definition (Zero Matrix)

A zero matrix, denoted 0, is an m × n matrix with all entries zero:
 
0 0 ··· 0
0 0 · · · 0
 
 .. .. . . .. 
. . . .
0 0 ··· 0

• additive identity: A − A = 0
• A+0=A
• A−A=0
• 0A = 0, A0 = 0

13
 Matrices
Vectors
Vectors and Matrices
Identity Matrix More on Linear Systems

Definition (Identity Matrix)

The n × n identity matrix, denoted In or I is the diagonal matrix with aii = 1: zero:
 
1 0 ··· 0
0 1 · · · 0
 
I = . . . .. 
 .. .. . . .
0 0 ··· 1

• multiplicative identity (like 1 does for scalars)

A of size m × n.
What size is I ?
• AI = A and IA = A
the identity matrix must be a square
matrix
Exercise: 3A + 2B = 2(B − A + C ) 14
 Matrices
Vectors
Vectors and Matrices
Matrix Inverse More on Linear Systems

• If AB = AC can we conclude that B = C ?

     
0 0 1 −1 8 0
A= , B= , C=
1 1 3 5 −4 4
 
0 0
AB = AC =
4 4
but hold on, this might be just a lucky case

• A + 5B = A + 5C =⇒ B = C
addition and scalar multiplication have inverses (−A and 1/c)

• Is there a multiplicative inverse?

15
 Matrices
Vectors
Vectors and Matrices
Inverse Matrix More on Linear Systems

Definition (Inverse Matrix)

The n × n matrix A is invertible if there is a matrix B such that

AB = BA = I

where I is the n × n identity matrix. The matrix B is called the inverse of A and is denoted by A−1 .
   
1 2 −2 1
A= , B=
3 4 3/2 −1/2
Theorem
If A is an n × n invertible matrix, then the matrix A−1 is unique.

Proof: Assume A has two inverses B, C so AB = BA = I and AC = CA = I . Consider the product

CAB:
CAB = C (AB) = CI = C associativity + AB = I
CAB = (CA)B = IB = B associativity + CA = I
16
 Matrices
Vectors
Vectors and Matrices
More on Linear Systems

• If a matrix has an inverse we say that it is invertible or non-singular

If a matrix has no inverse we say that it is non-invertible or singular
Eg:
    
0 0 a b 1 0
6=
1 1 c d 0 1

• If
 
a b
A= , ad − bc 6= 0
c d
then A has the inverse
 
1 d −b
A−1 = ad − bc 6= 0 check that this is true
ad − bc −c a

• The scalar ad − bc is called determinant of A and denoted |A|.

17
 Matrices
Vectors
Vectors and Matrices
Matrix Inverse More on Linear Systems

Back to the question:

• If AB = AC can we conclude that B = C ?

If A is invertible then the answer is yes:

A−1 AB = A−1 AC =⇒ IB = IC =⇒ B = C

• But AB = CA then we cannot conclude that B = C .

Note: the operation of matrix division is not defined!

18
 Matrices
Vectors
Vectors and Matrices
Properties of the Inverse More on Linear Systems

Let A be invertible =⇒ A−1 exists

• (A−1 )−1 = A

• (λA)−1 = 1 −1
λA
the inverse of the matrix (λA) is a matrix C that satisfies (λA)C = C (λA) = I .
Using matrix algebra:
   
1 −1 1 1 −1 1
(λA) A = λ AA−1 = I and A (λA) = λA−1 A = I
λ λ λ λ

• (AB)−1 = B −1 A−1

19
 Matrices
Vectors
Vectors and Matrices
Powers of a matrix More on Linear Systems

For A an n × n matrix and r ∈ N

Ar = AA . . . A}
| {z
r times

For the associativity of matrix multiplication:

• (Ar )−1 = (A−1 )r

• Ar As = Ar +s

• (Ar )s = Ars

20
 Matrices
Vectors
Vectors and Matrices
Transpose Matrix More on Linear Systems

Definition (Transpose)
The transpose of an m × n matrix A is the n × m matrix B defined by

bij = aji for i = 1, . . . , n and j = 1, . . . , m

It is denoted AT

   
a11 a12 · · · a1n a11 a21 · · · am1
 a21 a22 · · · a2n  a12 a22 · · · am2 
   
A = (aij ) =  . .. .. .  AT = (aji ) =  . .. .. . 
 .. . . ..   .. . . .. 
am1 am2 · · · amn a1n a2n · · · anm
  
1 0 1 3
We reflect the matrix about its main diagonal
3 1 0 1

Note that if D is a diagonal matrix: D T = D

21
 Matrices
Vectors
Vectors and Matrices
Properties of the transpose More on Linear Systems

• (AT )T = A

• (λA)T = λAT

• (A + B)T = AT + B T

• (AB)T = B T AT (consider first which matrix sizes make sense in the multiplication, then
rewrite the terms)

• if A is invertible, (AT )−1 = (A−1 )T

AT (A−1 )T = (A−1 A)T = I T = I using (AB)T = B T AT

(A−1 )T AT = (AA−1 )T = I T = I using (AB)T = B T AT

22
 Matrices
Vectors
Vectors and Matrices
Symmetric Matrix More on Linear Systems

Definition (Symmetric Matrix)

A matrix A is symmetric if it is equal to its transpose, A = AT .
(only square matrices can be symmetric)

23
 Matrices
Vectors
Vectors and Matrices
Outline More on Linear Systems

1. Matrices

2. Vectors

3. Vectors and Matrices

4. More on Linear Systems

24
 Matrices
Vectors
Vectors and Matrices
Vectors More on Linear Systems

• An n × 1 matrix is a column vector, or simply a vector:

 
v1
 v2 
 
v=.
 .. 
vn
The numbers v1 , v2 , . . . are known as the components (or entries) of v .
• A row vector is a 1 × n matrix
• We write vectors in lower boldcase type (writing by hand we can either underline them or add
an arrow over v).
• Addition and scalar multiplication are defined for vectors as for n × 1 matrices:
   
v1 + w 1 λv
 v2 + w 2  λv 
   
v+w = .  λv =  .  element-wise operations
 ..   .. 
vn + w n λv 25
 Matrices
Vectors
Vectors and Matrices
More on Linear Systems

• For a fixed n, the set of vectors together with the operations of addition and multiplication
form the set Rn , usually called Euclidean space

• For vectors v1 , v2 , . . . , vk in Rn and scalars α1 , α2 , . . . , αk in R, the vector

v = α1 v1 + α2 v2 + · · · + αk vk

is known as linear combination of the vectors v1 , v2 , . . . , vk

• A zero vector is denoted by 0;
0 + v = v + 0 = v;
0v = 0

• The matrix product of v and w cannot be calculated

• The matrix product of vT w gives an 1 × 1 matrix
• The matrix product of vwT gives an n × n matrix
26
 Matrices
Vectors
Vectors and Matrices
Inner product of two vectors More on Linear Systems

Definition (Inner product)

Given
   
v1 w1
 v2   w2 
   
v=. w= . 
 ..   .. 
vn wn

the inner product denoted hv, wi, is the real number given by
   
* v1 w1 +
v2  w2 
   
hv, wi =  . ,  .  = v1 w1 + v2 w2 + . . . + vn wn = vT w
 ..   .. 
vn wn

It is also called scalar product or dot product (and written v · w).

27
 Matrices
  Vectors
Vectors and Matrices
w1 More on Linear Systems

   w2 
 
vT w = v1 v2 · · · vn  .  = v1 w1 + v2 w2 + . . . + vn wn
.
 . 
wn

Theorem
The inner product

hx, yi = x1 y1 + x2 y2 + · · · + xn yn , x, y ∈ Rn

satisfies the following properties for all x, y, z ∈ Rn and for all α ∈ R:

• hx, yi = hy, xi
• α hx, yi = hαx, yi = hx, αyi
• hx + y, zi = hx, zi + hy, zi
• hx, xi ≥ 0 and hx, xi = 0 if and only if x = 0

Note: vectors from different Euclidean spaces live in different ’worlds’

28
 Matrices
Vectors
Vectors and Matrices
Outline More on Linear Systems

1. Matrices

2. Vectors

3. Vectors and Matrices

4. More on Linear Systems

29
 Matrices
Vectors
Vectors and Matrices
Vectors and Matrices More on Linear Systems

Let A be an m × n matrix
· · · a1n
 
a11 a12
 a21 a22 · · · a2n 
 . .. . 
 
 .. ..
. . .. 
am1 am2 · · · amn

and denote the columns of A by the column vectors a1 , a2 , . . . , an , so that

 
a1i
 a2i 
ai =  .  , i = 1, . . . , n.
 
 .. 
ami

Then if x = (x1 , x2 , . . . , xn )T is any vector in Rn

Ax = x1 a1 + x2 a2 + . . . + xn an

(ie, vector Ax in Rm is a linear combination of the column vectors of A)

30
 Matrices
Vectors
Vectors and Matrices
Resume More on Linear Systems

• We saw Matrix Algebra

• We can now prove two theorems on linear systems

31
 Matrices
Vectors
Vectors and Matrices
Outline More on Linear Systems

1. Matrices

2. Vectors

3. Vectors and Matrices

4. More on Linear Systems

32
 Matrices
Vectors
Vectors and Matrices
Solution Sets of Linear Systems More on Linear Systems

Theorem
A system of linear equations either has no solution, a unique solution or infinitely many solutions.

Proof.
Let’s assume the system Ax = b has two distinct solutions p and q, that is:

Ap = b Aq=b p − q 6= 0

Let t be any scalar and

v = p + t(q − p), t∈R

Then:

Av = A(p + t(q − p)) = Ap + t(Aq − Ap) = b + t(b − b) = b

that is, v is a solution of Ax = b and since p − q 6= 0 and there are infinitely many choices for t,
then there are infinitely many solutions for Ax = b.
33
 Matrices
Vectors
Vectors and Matrices
More on Linear Systems

Theorem (Principle of Linearity)

Suppose that A is an m × n matrix, that b ∈ Rm and that the system Ax = b is consistent.
Suppose that p is any solution of Ax = b.
Then the set of all solutions of Ax = b consists precisely of the vectors p + z for z ∈ N(A); ie,

{x | Ax = b} = {p + z | z ∈ N(A)}.

Proof: We show that

1. p + z is a solution for any z in the null space of A ({p + z | z ∈ N(A)} ⊆ {x | Ax = b})
2. all solutions, x, of Ax = b are of the form p + z for some z ∈ N(A)
({x | Ax = b} ⊆ {p + z | z ∈ N(A)})
1. A(p + z) = Ap + Az = b + 0 = b so p + z ∈ {x | Ax = b}
2. Let x be a solution. Because p is also we have Ap = b andA(x − p) = Ax − Ap = b − b = 0 so
z = x − p is a solution of Az = 0 and x = p + z
(Check validity of the theorem on the last examples of previous lecture.)
34