APPLIED NUMERICAL METHODS

SEMM/SKMM3023

Solution to Simultaneous Linear Algebraic Equations

Department of Aeronautics, Automotive, & Ocean Engineering

School of Mechanical Engineering, Faculty of Engineering,
Universiti Teknologi Malaysia

1
 What are simultaneous linear equations?

■ Generally speaking, they are a set of finite linear equations.

Example:
2𝑥 + 3𝑦 = 6
−5𝑥 − 𝑦 = 1
■ Many applications of engineering lead to a system of linear
algebraic equations, involving thousands of simultaneous
linear equations.

2
 Programming Linear Equations

■ In programming, constants in simultaneous linear equations are often

represented by a11, a12, a13 etc as shown below

3
 Programming Linear Equations

■ The set of equations can then be represented as the following:

■ Which is in the from of:

4
 Engineering Applications

■ Linear simultaneous equations are used to solve many engineering

problems such as:

I. Force analysis of structures

II. Current flow in electrical circuits
III. Finite element analysis
IV. Fluid flow in pipes
V. Curve fitting

5
 Matrix Revision

■ Since solving simultaneous equations requires the use of matrices, we

will revise a little bit on this…

■ In mathematics, matrix is generally an array or collection of numbers or

symbols of equations arrange in rows and columns.

■ There are several types of square matrices (matrix with equal number
of rows and columns):

6
Matrix Revision

Symmetrical matrix Diagonal matrix

Upper triangular matrix Lower triangular matrix

7
Matrix Revision

Identity matrix [I] Banded matrix

Transpose Matrix

8
 Matrix Revision

■ Properties of matrices:

9
 Matrix Revision

■ Matrices can only be multiplied if:

𝑀 𝑎×𝑏 × 𝑁 𝑏×𝑐 = 𝑃 𝑎×𝑐

Must be equal for multiplication!

■ Also note that 𝑀 × 𝑁 ≠ 𝑁 × [𝑀]

10
 Matrix Revision

■ Inverse matrix:
[𝐴]−1 𝐴 = 𝐴 𝐴 −1 = 𝐼
−1
1 𝑎22 −𝑎12
[𝐴] =
𝑎11 𝑎22 − 𝑎21 𝑎12 −𝑎21 𝑎11

■ Determinant:

11
 Matrix Revision

■ Matrix norm:
𝐴𝑥Ԧ
𝐴 = max = max 𝐴𝑥Ԧ
Ԧ
𝑥=1 𝑥Ԧ Ԧ
𝑥=1
𝐴 2 = max 𝑖
0<𝑖≤𝑛
𝑛

𝐴 ∞ = max ෍ 𝑎𝑖𝑗
0<𝑖≤𝑛
𝑖=1
𝑛

𝐴 1 = max ෍ 𝑎𝑖𝑗
0<𝑗≤𝑛
𝑖=1
𝑛 𝑚
2
𝐴 𝑓 = ෍ ෍ 𝑎𝑖𝑗
𝑖=1 𝑗=1

12
 Matrix Revision

■ Matrix condition number:

𝐶𝑜𝑛𝑑 𝐴 = 𝐴 ∙ 𝐴−1

If Cond(A) > 1, the matrix is considered ill-conditioned

■ Cramer’s rule:

13
 Matrix Revision

■ MATLAB codes for matrix

[A] = [1 2 3;4 5 6;7 8 9];

A’ (transpose)
det(A)
norm(A)
cond(A)
inv(A)
tril(A)
triu(A)

14
 Available Numerical Methods

■ Elimination methods:
1. Gauss elimination method
2. Gauss-Jordan method

■ Iterative methods:
1. Jacobi’s method
2. Gauss-Seidel method

15
 METHOD 1 (ELIMINATION):
GAUSS ELIMINATION METHOD

16
 Gauss Elimination Method

■ Also know as row reduction method. Requires the matrix to be

transformed into the row echelon form or the upper triangle matrix to
solve
■ Must construct the augmented matrix form first prior to solving:
𝐴 𝑥Ԧ = 𝑏 → [𝐴|𝑏]

■ Method can only be applied for square matrices

■ Once the row echelon form is achieved, x can be determined by using
backward substitution

17
 Gauss Elimination Method
■ Example:
You should have
learnt this before!

18
 Gauss Elimination Method
■ Steps for 3 x 3 matrix:

𝑎21
1. 𝑟𝑜𝑤2 = 𝑟𝑜𝑤2 − 𝑟𝑜𝑤1
𝑎11
𝑎31
2. 𝑟𝑜𝑤3 = 𝑟𝑜𝑤3 − 𝑟𝑜𝑤1
𝑎11
𝑎32
3. 𝑟𝑜𝑤3 = 𝑟𝑜𝑤3 − 𝑟𝑜𝑤2
𝑎22

19
 Gauss Elimination Method

■ Example: Reduce the following matrix to the row echelon form (Gauss
Elimination)

2 −1 1 𝑥1 4
4 3 −1 𝑥2 = 6
3 2 2 𝑥3 15

20
 Gauss Elimination Method

■ First put it in augmented matrix form

Columns (V)
2 −1 1 4
Rows (H) 4 3 −1 6
3 2 2 15

■ Multiply row 1 with a21/a11 and subtract from row 2 to eliminate a21
𝑎21
𝑟𝑜𝑤2 = 𝑟𝑜𝑤2 − 𝑟𝑜𝑤1 = 4 3 −1 6 − 4 −2 2 8
𝑎11

21
 Gauss Elimination Method

Columns (V)
2 −1 1 4
Rows (H) 0 5 −3 −2
3 2 2 15

■ Multiply row 1 with a31/a11 and subtract from row 3 to eliminate a31
𝑎31
𝑟𝑜𝑤3 = 𝑟𝑜𝑤3 − 𝑟𝑜𝑤1 = 3 2 2 15 − 3 −1.5 1.5 6
𝑎11

■ If you have more rows, (4x4 or 5x5 etc..) then just repeat the step above
to eliminate a41, a51 or etc…

22
 Gauss Elimination Method

Columns (V)
2 −1 1 4
Rows (H) 0 5 −3 −2
0 3.5 0.5 9

■ Multiply row 2 with a32/a22 and subtract from row 3 to eliminate a32
𝑎32
𝑟𝑜𝑤3 = 𝑟𝑜𝑤3 − 𝑟𝑜𝑤2 = 0 3.5 0.5 9 − 0 3.5 −2.1 −1.4
𝑎22

■ Again, if you have more rows, (4x4 or 5x5 etc..) then just repeat the step
above to eliminate a42, a52 or etc…

23
 Gauss Elimination Method
2𝑥1 − 1𝑥2 + 1𝑥3 = 4
5𝑥2 − 3𝑥3 = −2
2.6𝑥3 = 10.4

2 −1 1 4 2 −1 1 𝑥1 4
0 5 −3 −2 → 0 5 −3 𝑥2 = −2
0 0 2.6 10.4 0 0 2.6 𝑥3 10.4

■ The matrix has now been reduced into the row echelon form. X can now
be solved using backward substitution.

2.6𝑥3 = 10.4 → 𝑥3 = 4
5𝑥2 − 3 4 = −2 → 𝑥2 = 2
2𝑥1 − 1 2 + 1 4 = 4 → 𝑥1 = 1
■ The same method can be applied for any square matrix!

24
 Gauss Elimination Method

■ Prepare matrix 𝐴 𝑥Ԧ = 𝑏
■ Form the augmented matrix [𝐴|𝑏]
■ Eliminate the 1st column from the 2nd row until the end by multiplying
the 1st row with a constant ratio and subtracting it from the rows
below
■ Eliminate the 2nd column from the 3rd row until the end by multiplying
the 2nd row with a constant ratio and subtracting it from the rows
below
■ Continue process for other columns if necessary, to make a lower
triangular matrix all zeros (row echelon form)
■ Use backward substitution to solve for x

25
 Gauss Elimination Method

■ In MATLAB, you can easily solve this

2 −1 1 𝑥1 4
4 3 −1 𝑥2 = 6
3 2 2 𝑥3 15

𝐴 𝑥Ԧ = 𝑏

■ To find x in MATLAB, simply type x = A\b

26
 Gauss Elimination Method

■ Example:
Spongebob squarepants was trying the figure out the stock market of the
Krusty Krab restaurant. The business of the restaurant can be defined by
the following set of linear equations

2𝑥 + 𝑦 − 2𝑧 = 8
−3𝑥 − 𝑦 + 2𝑧 = −11
−2𝑥 + 𝑦 + 2𝑧 = −3

Generate the row echelon form for the set of equations and hence, solve it
using backward substitution.

27
METHOD 2 (ELIMINATION):
GAUSS-JORDAN METHOD

28
 Gauss-Jordan Method

■ This method is very similar to the Gauss elimination method. However,

the Gauss elimination method only makes the lower triangular half to
zero
𝑎 𝑏 𝑐
0 𝑑 𝑒
0 0 𝑓

Whereas the Gauss-Jordan method reduces the matrix to a unity matrix.

This is also known as the reduced row echelon form
1 0 0
0 1 0
0 0 1

29
 Gauss-Jordan Method

■ Example: Reduce the following matrix to the reduced row echelon form

30
 Gauss-Jordan Method

■ Lets start with the row echelon form we did previously

2 −1 1 4
0 5 −3 −2
0 0 2.6 10.4

■ Normalise the matrix first by dividing each row with the diagonal
row1 = row1/a11 1 −0.5 0.5 2
row2 = row2/a22 0 1 −0.6 −0.4
row3 = row3/a33 0 0 1 4

31
 Gauss-Jordan Method

1 −0.5 0.5 2
0 1 −0.6 −0.4
0 0 1 4

■ Elimanate a12 by multiplying row 2 with a12/a22 and subtracting from

row 1
𝑎12
𝑟𝑜𝑤1 = 𝑟𝑜𝑤1 − 𝑟𝑜𝑤2 = 1 −0.5 0.5 2 − 0 −0.5 0.3 0.2
𝑎22

1 0 0.2 1.8
0 1 −0.6 −0.4
0 0 1 4

32
 Gauss-Jordan Method

1 0 0.2 1.8
0 1 −0.6 −0.4
0 0 1 4

■ Elimanate a13 by multiplying row 3 with a13/a33 and subtracting from

row 1
𝑎13
𝑟𝑜𝑤1 = 𝑟𝑜𝑤1 − 𝑟𝑜𝑤3 = 1 0 0.2 1.8 − 0 0 0.2 0.8
𝑎33

1 0 0 1
0 1 −0.6 −0.4
0 0 1 4

33
 Gauss-Jordan Method

1 0 0 1
0 1 −0.6 −0.4
0 0 1 4

■ Elimanate a23 by multiplying row 3 with a23/a33 and subtracting from

row 2
𝑎23
𝑟𝑜𝑤2 = 𝑟𝑜𝑤2 − 𝑟𝑜𝑤3 = 0 1 −0.6 −0.4 − 0 0 −0.6 −2.4
𝑎33

1 0 01
0 1 02
0 0 14

34
 Gauss-Jordan Method

1 0 01
0 1 02
0 0 14

■ The solution for x is simply the last column in the augmented matrix!
𝑥1 1
𝑥 = 𝑥2 = 2
𝑥3 4

■ Note that the same method can be applied for other square matrix.

35
 Gauss-Jordan Method

■ Prepare the augmented matrix

■ Start with the row echelon form
■ Normalize the matrix by dividing each row with the diagonals
■ Eliminate the non-unity or non-zero (not 1) columns in the 1st row by
multiplying the rows below it with a constant ratio and subtracting it
from the first row.
■ Eliminate the non-unity and non-zero (not 1) columns in the 2nd row
by multiplying the rows below it with a constant ratio and subtracting
it from the 2nd row.
■ Continue process for other rows to make sure the upper triangular
matrix are all zeros

36
METHOD 3(ITERATIVE):
JACOBI’S METHOD

41
 Jacobi’s Method

■ An iterative method to solve simultaneous linear equation matrix where

the diagonals of the matrix are non-zero

■ Consider the following matrix form where a11, a22 and a33 are non-zero.
b1 = a11*x1+a12*x2+a13*x3 -> x1 = (b1-a12*x2-a13*x3)/a11

42
 Jacobi’s Method

■ Let the initial values be the first iteration., where all values of {x} are
assigned an initial guess

■ The {x} values for the next iterations are:

43
 Jacobi’s Method

■ In matrix form it looks like

■ Where the process is repeated by assigning {x}k = {x}k+1 until

|{x}k+1 - {x}k| is less than a certain defined tolerance value
■ It is common to start the guess with x = vector of zeros

44
 Jacobi’s Method
■ Example: Determine if the following simultaneous equations can be
solved using the Jacobi’s method. If yes, solve it until tolerance < 0.01

2 −1 1 4
■ A = 4 3 −1 𝑏 = 6
3 2 2 15
2*x1-x2+x3 = 4 -> x1 = (4+x2-x3)/2

Let the initial guess of x be

𝑥1 0
𝑥 = 𝑥2 = 0
𝑥3 0

45
 Jacobi’s Method
First iteration
4 −1 1
𝑥1𝑘+1 = − 0 − 0 =2
2 2 2
𝑘+1
6 4 −1
𝑥2 = − 0 − 0 =2
3 3 3
𝑘+1
15 3 2
𝑥3 = − 0 − 0 = 7.5
2 2 2
Second iteration

4 −1 1
𝑥1𝑘+1 = − 2 − 7.5 =?
2 2 2
6 4 −1
𝑥2𝑘+1 = − 2 − 7.5 = ?
3 3 3
𝑘+1
15 3 2
𝑥3 = − 2 − 2 =?
2 2 2

46
 Jacobi’s Method
2 −1 1 4
A = 4 3 −1 𝑏 = 6
3 2 2 15
■ If you carry on with the iteration, you will find out that it will not
converge! Why?
■ The reason is that matrix [A] is not diagonally dominant
■ Another reason for this is because matrix A is ill-conditioned and
cond(A) is significantly larger than 1 (cond(A) = 4.04).
■ The limitations of the Jacobi method is that it would most likely fail to
converge if the matrix is not diagonally dominant or significantly ill-
conditioned
■ If the matrix is diagonally dominant but significantly ill-conditioned, it
may still converge!

47
 Jacobi’s Method
■ Lets try a matrix that is less ill-conditioned and diagonally dominant

20 1 −2 17
A= 3 20 −1 𝑏 = −18
2 −3 20 25

■ For this matrix, cond(A) = 1.32. If we use the Jacobi method ({x} = 0)
x1 x2 x3 |x1k+1 - x1k|
0 0 0
Tolerance < 0.85 -0.9 1.25 0.85
0.01 to stop
1.02 -0.965 1.03 0.17
iteration
1.001 -1.002 1.003 0.0188
1 -1 1 0.0009
■ The method converges this time. This is because A is diagonally
dominant and less ill-conditioned

48
 Jacobi’s Method

■ Rearrange the equation into the required form

■ Set the initial value for {x}. The easiest way is to set all initial values
in {x} to {x} = {0, 0 ,0…}
■ Calculate the new value of {x}
■ Repeat iteration process
■ Stop calculation one the desired accuracy has been achieved
■ Remember that for non-diagonally dominant matrix or a significantly
ill-conditioned matrix, the Jacobi method may not converge!

49
METHOD 5 (ITERATIVE):
GAUSS-SEIDEL METHOD

50
 Gauss-Seidel Method

■ This method is very similar to the Jacobi’s method. But instead of

assuming an initial value for the whole array of {x} with n elements,
initial values are assigned to n-1 values. This means that if {x} has 3
elements, then only 2 of the elements are assigned an initial guess.
■ The first element of {x} would be calculated whereas the values of the
other two elements are updated based on the calculated value of the
first element

51
 Gauss-Seidel Method
Gauss-Seidel Jacobi

■ It can be seen in that x1 is computed by assuming x2 and x3.

■ Then the new x1 value and the assumed x3 value are used to calculate a
new x2 value.
■ This process is continued by substituting the new x1 and x2 values to
determine a new x3 value

52
 Gauss-Seidel Method

■ The process is then repeated until |{x}k+1 - {x}k| achieves a certain

tolerance value, just like the Jacobi’s method
■ Example: Determine if the following simultaneous equations can be
solved using the Gauss-Seidel method. If yes, solve it until tolerance <
0.0001
4 + 𝑥2 − 𝑥3
4 − 𝑥3 + 𝑥2 = 2𝑥1 → = 𝑥1
2

■ Let x2 = x3 = 0

53
 Gauss-Seidel Method
First iteration
4 −1 1
𝑥1𝑘+1
= − 0 − 0 =2
2 2 2
𝑘+1
6 4 −1 −2
𝑥2 = − 2 − 0 =
3 3 3 3
𝑘+1
15 3 2 −2 31
𝑥3 = − 2 − =
2 2 2 3 6
Second iteration
4 −1 −2 1 31
𝑥1𝑘+1= − − = −0.9167
2 2 3 2 6
𝑘+1
6 4 −1 31
𝑥2 = − −0.9167 − =?
3 3 3 6
𝑘+1
15 3 2
𝑥3 = − −0.9167 − ? =
2 2 2

54
 Gauss-Seidel Method
2 −1 1 4
A= 4 x1 x2 x3 |x1k+1 - x1k|
3 −1 𝑏 = 6
0 0
3 2 2 15 2 -0.667 5.167
-0.917 4.944 3.931 2.9167
2.507 -0.032 3.772 3.4236
0.098 3.127 4.226 2.4091
1.45 1.475 3.85 1.3525
0.813 2.199 4.081 0.6375
1.059 1.948 3.963 0.2462
0.993 1.997 4.014 0.0664
0.992 2.015 3.997 0.0007
1.009 1.987 4 0.0173
0.994 2.008 4.001 0.0156
1.004 1.996 3.999 0.01
0.998 2.002 4.001 0.0053
1.001 1.999 4 0.0023
1 2 4 0.0008
1 2 4 0.0002
1 2 4 0.0001

55
 Gauss-Seidel Method

■ Rearrange the equation into the required form

■ Set the initial value for {x}, except for the first value. The easiest way
is to set all initial values in {x} to {x} = {-, 0 ,0…}
■ Calculate the new value of {x} by updating the values
■ Repeat iteration process
■ Stop calculation one the desired accuracy has been achieved
■ The Gauss-Seidel method is less sensitive to diagonal dominance or
ill-conditioned matrix, but may still fail in some cases.

56
Thank you

57