Matrix and Numerical Methods
in Systems Engineering
University of Florida
Department of Industrial and Systems Engineering
Fall 2016
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Prologue
I Read the syllabus
I Time commitment
I 4 × 4 = 16 hours/week
I review material after every lecture
I cannot compress time
I Understanding
I do not memorize
I do not filter
I MATLAB
I no calculators
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
How to Study?
I Basic math skills are assumed
I Take notes
I Review notes/slides
I make sure everything is clear: concepts, examples, ...
I book / electronic resources
I office hours
I Problems
I homework
I book
I Goal
I able to solve any problem
I ... not only those on tests
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Linear Algebra
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Systems of Linear Equations
I Linear equation: ax + by = c
I variables: x, y
I constants: a, b, c
I System of linear equations
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
.. ..
. .
am1 x1 + am2 x2 + · · · + amn xn = bm
I n variables
I m equations
I (n + 1)m constants
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 1
2x − y = 0
−x + 2y= 3
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 1
2x − y = 0
−x + 2y= 3
I Matrix form [ ][ ] [ ]
2 −1 x 0
=
−1 2 y 3
or just
Ax = b
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 1
2x − y = 0
−x + 2y= 3
I Matrix form [ ][ ] [ ]
2 −1 x 0
=
−1 2 y 3
or just
Ax = b
I Three interpretations
I row
I column
I matrix
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 1
I Row
I 2x − y = 0 and −x + 2y = 3
I solution: x = 1, y = 2
I number of solutions
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 1
I Row
I 2x − y = 0 and −x + 2y = 3
I solution: x = 1, y = 2
I number of solutions
I Column
I linear combination
[ ] [ ] [ ]
2 −1 0
x +y =
−1 2 3
I solution: x = 1, y = 2
I number of solutions
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 1
I Row
I 2x − y = 0 and −x + 2y = 3
I solution: x = 1, y = 2
I number of solutions
I Column
I linear combination
[ ] [ ] [ ]
2 −1 0
x +y =
−1 2 3
I solution: x = 1, y = 2
I number of solutions
I Matrix
[ ][ ] [ ] [ ] [ ]
2 −1 1 2 −1 0
=1 +2 =
−1 2 2 −1 2 3
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 2
2x−y =0
−x+2y− z = − 1
−3y+4z =4
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 2
2x−y =0
−x+2y− z = − 1
−3y+4z =4
I Row: planes instead of lines
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 2
2x−y =0
−x+2y− z = − 1
−3y+4z =4
I Row: planes instead of lines
I Column
2 −1 0 0
x −1 + y 2 + z −1 = −1
0 −3 4 4
I Solution: x = y = 0, z = 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example 2
2x−y =0
−x+2y− z = − 1
−3y+4z =4
I Row: planes instead of lines
I Column
2 −1 0 0
x −1 + y 2 + z −1 = −1
0 −3 4 4
I Solution: x = y = 0, z = 1
I Ax = b – Does a solution always exist?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Elimination
I Natural
I Two stages
I forward elimination
I backward substitution
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Elimination
I Natural
I Two stages
I forward elimination
I backward substitution
I Example
x+2y+z =2
3x+8y+z =12
4y+z =2
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Elimination
I Natural
I Two stages
I forward elimination
I backward substitution
I Example
x+2y+z =2
3x+8y+z =12
4y+z =2
I Forward step: pivots
1 2 1 1 2 1 1 2 1
A = 3 8 1 → 0 2 −2 → 0 2 −2 = U
0 4 1 0 4 1 0 0 5
I U – upper triangular matrix . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Elimination
I What can go wrong
I pivots 1 & 2 equal to 0: interchange rows
I pivot 3 equals to 0: no way out
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Elimination
I What can go wrong
I pivots 1 & 2 equal to 0: interchange rows
I pivot 3 equals to 0: no way out
I Augmented matrix
1 2 1 2 1 2 1 2 1 2 1 2
3 8 1 12 → 0 2 −2 6 → 0 2 −2 6 = [U c]
0 4 1 2 0 4 1 2 0 0 5 −10
x+2y+ z =2
2y−2z =6
5z = − 10
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Elimination
I What can go wrong
I pivots 1 & 2 equal to 0: interchange rows
I pivot 3 equals to 0: no way out
I Augmented matrix
1 2 1 2 1 2 1 2 1 2 1 2
3 8 1 12 → 0 2 −2 6 → 0 2 −2 6 = [U c]
0 4 1 2 0 4 1 2 0 0 5 −10
x+2y+ z =2
2y−2z =6
5z = − 10
I Backward substitution: z = −2, y = 1, x = 2
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Matrix Multiplication
I matrix-by-column
b1 a11 a12 a13
A b2 = b1 a21 + b2 a22 + b3 a23
b3 a31 a32 a33
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Matrix Multiplication
I matrix-by-column
b1 a11 a12 a13
A b2 = b1 a21 + b2 a22 + b3 a23
b3 a31 a32 a33
I row-by-matrix
[ ] [ ] [ ] [ ]
a1 a2 a3 B = a1 b11 b12 b13 +a2 b21 b22 b23 +a3 b31 b32 b33
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Matrix Multiplication
I matrix-by-column
b1 a11 a12 a13
A b2 = b1 a21 + b2 a22 + b3 a23
b3 a31 a32 a33
I row-by-matrix
[ ] [ ] [ ] [ ]
a1 a2 a3 B = a1 b11 b12 b13 +a2 b21 b22 b23 +a3 b31 b32 b33
I matrix-by-matrix: C = AB
∑
cij = aik bkj
k
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Matrix Multiplication
I matrix-by-column
b1 a11 a12 a13
A b2 = b1 a21 + b2 a22 + b3 a23
b3 a31 a32 a33
I row-by-matrix
[ ] [ ] [ ] [ ]
a1 a2 a3 B = a1 b11 b12 b13 +a2 b21 b22 b23 +a3 b31 b32 b33
I matrix-by-matrix: C = AB
∑
cij = aik bkj
k
I If A is n × m and B is m × p, the C is n × p
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Matrix Multiplication
I Important: AB ̸= BA
I Example [ ][ ]
1 2 5 6
=
3 4 7 8
I Another way to multiply
1 2 [ ] 1 [ ] 2 [ ]
3 7 8
4 = 3 7 8 + 4 9 0
9 0
5 6 5 6
I Block multiplication
[ ][ ]
A1 A2 B1 B2
=
A3 A4 B3 B4
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Back to Elimination
I Step 1
1 2 1 1 2 1 ? ? ?
E21 3 8 1 = 0 2 −2 E21 = ? ? ?
0 4 1 0 4 1 ? ? ?
I Step 2
1 2 1 1 2 1 ? ? ?
E32 0 2 −2 = 0 2 −2 E32 = ? ? ?
0 4 1 0 0 5 ? ? ?
I Forward elimination: E32 (E21 A) = U or
(E32 E21 )A = U
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Back to Elimination
Elementary row operations:
1. Interchanging two rows
2. Multiplying a row with nonzero constant
3. Adding a multiple of a row to another row
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Back to Elimination
Elementary row operations:
1. Interchanging two rows
2. Multiplying a row with nonzero constant
3. Adding a multiple of a row to another row
Definition: An elementary matrix is a matrix obtained by doing an
elementary row operation on the identity matrix of the same size.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Back to Elimination
Elementary row operations:
1. Interchanging two rows
2. Multiplying a row with nonzero constant
3. Adding a multiple of a row to another row
Definition: An elementary matrix is a matrix obtained by doing an
elementary row operation on the identity matrix of the same size.
Theorem: Let B be a matrix obtained by doing an elementary row
operation on matrix A. Then, there exists an elementary matrix E
such that B = EA.
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse of a Square Matrix
I Inverse matrix (if it exists)
A−1 A = I = AA−1
I Invertible, nonsingular
I When the inverse exists?
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse of a Square Matrix
I Inverse matrix (if it exists)
A−1 A = I = AA−1
I Invertible, nonsingular
I When the inverse exists?
I Example: no inverse [ ]
1 3
A=
2 6
I AB ̸= I since columns of AB are on the same line
I exits x ̸= 0 such that Ax = 0:
x = A−1 Ax = A−1 0 = 0
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse of a Square Matrix
I Example [ ][ ] [ ]
1 3 a c 1 0
=I=
2 7 b d 0 1
I A × column j of A−1 = column j of I
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse of a Square Matrix
I Example [ ][ ] [ ]
1 3 a c 1 0
=I=
2 7 b d 0 1
I A × column j of A−1 = column j of I
I Like solving systems of linear equations – two in this case:
[ ][ ] [ ]
1 3 a 1
=
2 7 b 0
[ ][ ] [ ]
1 3 c 0
=
2 7 d 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Gauss-Jordan Method
[ ][ ] [ ]
1 3 a 1
=
2 7 b 0
[ ][ ] [ ]
1 3 c 0
=
2 7 d 1
I Elimination applied to the augmented matrix:
[ ] [ ] [ ]
1 3 1 0 1 3 1 0 1 0 7 −3
[A I] = → → = [I A−1 ]
2 7 0 1 0 1 −2 1 0 1 −2 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Gauss-Jordan Method
[ ][ ] [ ]
1 3 a 1
=
2 7 b 0
[ ][ ] [ ]
1 3 c 0
=
2 7 d 1
I Elimination applied to the augmented matrix:
[ ] [ ] [ ]
1 3 1 0 1 3 1 0 1 0 7 −3
[A I] = → → = [I A−1 ]
2 7 0 1 0 1 −2 1 0 1 −2 1
I Check [ ][ ]
1 3 7 −3
=
2 7 −2 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Gauss-Jordan Method
[ ][ ] [ ]
1 3 a 1
=
2 7 b 0
[ ][ ] [ ]
1 3 c 0
=
2 7 d 1
I Elimination applied to the augmented matrix:
[ ] [ ] [ ]
1 3 1 0 1 3 1 0 1 0 7 −3
[A I] = → → = [I A−1 ]
2 7 0 1 0 1 −2 1 0 1 −2 1
I Check [ ][ ]
1 3 7 −3
=
2 7 −2 1
I Interpretation
E [A I] = [I ?] . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Gauss-Jordan Method
Example:
x+2y+ = 3
−x −2z =−5
−3x−5y+ z =−4
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Gauss-Jordan Method
Example:
x+2y+ = 3
−x −2z =−5
−3x−5y+ z =−4
Ax = b → A−1 Ax = A−1 b → x = A−1 b
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse & Transpose
I Transposition: A⊤
I (A⊤ )ij = (A)ji
I (A⊤ )⊤ = A
I (A + B)⊤ = A⊤ + B ⊤
I (AB)⊤ =
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse & Transpose
I Transposition: A⊤
I (A⊤ )ij = (A)ji
I (A⊤ )⊤ = A
I (A + B)⊤ = A⊤ + B ⊤
I (AB)⊤ =
I Inverse: (AB)−1 =
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse & Transpose
I Transposition: A⊤
I (A⊤ )ij = (A)ji
I (A⊤ )⊤ = A
I (A + B)⊤ = A⊤ + B ⊤
I (AB)⊤ =
I Inverse: (AB)−1 =
I Combine the two:
AA−1 = I
(AA−1 )⊤ = I
(A−1 )⊤ A⊤ = I
(A⊤ )−1 =
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Inverse & Transpose
I Transposition: A⊤
I (A⊤ )ij = (A)ji
I (A⊤ )⊤ = A
I (A + B)⊤ = A⊤ + B ⊤
I (AB)⊤ =
I Inverse: (AB)−1 =
I Combine the two:
AA−1 = I
(AA−1 )⊤ = I
(A−1 )⊤ A⊤ = I
(A⊤ )−1 =
I Symmetric matrix: A⊤ = A
I B ⊤ B is symmetric for any B
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Factorization
I Elimination: A → U
I How A and U relate?
I Example [
] [ ]
2 1 ? ?
A= → =U
8 7 ? ?
E21 A = U
E21 =
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Factorization
I Elimination: A → U
I How A and U relate?
I Example [
] [ ]
2 1 ? ?
A= → =U
8 7 ? ?
E21 A = U
E21 =
A = LU
[ ][ ][ ]
1 0 2 0 1 1/2
= = LDU
4 1 0 3 0 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
3 × 3 case
I No row exchanges – no 0 pivots:
E32 E31 E21 A = U
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
3 × 3 case
I No row exchanges – no 0 pivots:
E32 E31 E21 A = U
I Then
−1 −1 −1
A = E21 E31 E32 U = LU
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
3 × 3 case
I No row exchanges – no 0 pivots:
E32 E31 E21 A = U
I Then
−1 −1 −1
A = E21 E31 E32 U = LU
I Example
1 0 0 1 0 0 1 0 0
E32 E21 = 0 1 0 −2 1 0 = −2 1 0
0 −5 1 0 0 1 10 −5 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
3 × 3 case
I No row exchanges – no 0 pivots:
E32 E31 E21 A = U
I Then
−1 −1 −1
A = E21 E31 E32 U = LU
I Example
1 0 0 1 0 0 1 0 0
E32 E21 = 0 1 0 −2 1 0 = −2 1 0
0 −5 1 0 0 1 10 −5 1
1 0 0 1 0 0 1 0 0
−1
(E32 E21 ) = 2 1 0 0
1 0 = 2 1 0
0 0 1 0 5 1 0 5 1
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
3 × 3 case
I No row exchanges – no 0 pivots:
E32 E31 E21 A = U
I Then
−1 −1 −1
A = E21 E31 E32 U = LU
I Example
1 0 0 1 0 0 1 0 0
E32 E21 = 0 1 0 −2 1 0 = −2 1 0
0 −5 1 0 0 1 10 −5 1
1 0 0 1 0 0 1 0 0
−1
(E32 E21 ) = 2 1 0 0
1 0 = 2 1 0
0 0 1 0 5 1 0 5 1
I Exactly the -multipliers . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example
3 −7 −2 2
−3 5 1 0
A=
6 −4 0 −5
−9 5 −5 12
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Example
3 −7 −2 2
−3 5 1 0
A=
6 −4 0 −5
−9 5 −5 12
I How to use A = LU
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Permutation Matrices
I Row exchanges: needed to avoid 0 pivots
I 3 × 3: 6 matrices
1 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1
0 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0
0 0 1 0 0 1 1 0 0 0 1 0 0 1 0 1 0 0
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Permutation Matrices
I Row exchanges: needed to avoid 0 pivots
I 3 × 3: 6 matrices
1 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1
0 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0
0 0 1 0 0 1 1 0 0 0 1 0 0 1 0 1 0 0
I P −1 = P ⊤ or P ⊤ P = I
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Permutation Matrices
I Row exchanges: needed to avoid 0 pivots
I 3 × 3: 6 matrices
1 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1
0 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0
0 0 1 0 0 1 1 0 0 0 1 0 0 1 0 1 0 0
I P −1 = P ⊤ or P ⊤ P = I
I In general: n! matrices
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
Permutation Matrices
I Row exchanges: needed to avoid 0 pivots
I 3 × 3: 6 matrices
1 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1
0 1 0 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0
0 0 1 0 0 1 1 0 0 0 1 0 0 1 0 1 0 0
I P −1 = P ⊤ or P ⊤ P = I
I In general: n! matrices
I If no row exchanges needed: A = LU
I With row exchanges: P A = LU (any invertible A)
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
