Chapter Three

Systems of linear equations

Introduction
Systems of simultaneous equations occur in solving problems in a wide variety of disciplines,
including mathematics and statistics, the physical, biological, and social sciences, as well as
engineering and business. They arise directly in solving real world problems depending on the
structure of the equations. In this chapter we present the solution of n linear simultaneous algebraic
equations in n unknowns. Linear systems of equations are associated with many problems in
engineering and science, as well as with applications of mathematics to the social sciences and
quantitative study of business and economic problems.
A linear system of 𝑛 equations in 𝑛 unknowns 𝑥1 , … , 𝑥𝑛 is a set of the system of algebraic equations
has equations of the form
𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
. . . . (3.1)
. . . .
𝑎𝑛1 𝑥1 + 𝑎𝑛2 𝑥2 + ⋯ + 𝑎𝑛𝑛 𝑥𝑛 = 𝑏𝑛
Where, the coefficients 𝑎𝑗𝑘 and 𝑏𝑗 are constants.
The system (3.1) is homogeneous if all the 𝑏𝑗 are zero, otherwise is non-homogeneous.
In matrix form, the system (3.1) can be written as
𝐴𝑥 = 𝑏, (3.2)
𝑎11 𝑎12 . . . 𝑎1𝑛 𝑥1 𝑏1
𝑎21 𝑎22 . . . 𝑎2𝑛 𝑥2 𝑏2
. . . . .
Where, 𝐴= . . . , 𝑥= . , 𝑏=
.
. . . . .
(𝑎𝑛1 𝑎𝑛2 . . . 𝑎𝑛𝑛 ) (𝑥𝑛 ) (𝑏𝑛 )
There are two different approaches for finding numerical solutions of a system of equations,
namely
1) Direct methods (or exact methods)
2) Indirect methods (or approximated methods)

1|Page
3.1 Direct methods for system of linear equations
Let 𝐴𝑥 = 𝑏 be any given linear system. Then there are different numerical methods to find the
exact solution set of the given system, some of methods to be discussed in this chapter are:
1) Gauss elimination method
2) Gauss-Jordan elimination method
3) Matrix Inversion method
4) LU-decomposition method
The Backward and Forward substitution formulas
In linear system of equations, where the coefficient matrix is triangular are simple to solve. A
system 𝑈𝑥 = 𝐵, where 𝑈an upper triangular matrix is has the form:
𝑢11 𝑥1 + 𝑢12 𝑥2 + ⋯ + 𝑢1𝑛 𝑥𝑛 = 𝑏1
0 + 𝑢22 𝑥2 + ⋯ + 𝑢2𝑛 𝑥𝑛 = 𝑏2
.
.
.
0+⋯ … + 0 + 0 + 𝑢𝑛𝑛 𝑥𝑛 = 𝑏𝑛
If we assume 𝑢𝑖𝑖 ≠ 0, for all 𝑖 = 1,2, … , 𝑛, then the unknowns can be computed in the order of
𝑥𝑛 , 𝑥𝑛−1 , … , 𝑥2 , 𝑥1 as follows:
𝑏𝑛
𝑥𝑛 =
𝑢𝑛𝑛
𝑏𝑛−1 − 𝑢(𝑛−1)𝑛 𝑥𝑛
𝑥𝑛−1 =
𝑢(𝑛−1)(𝑛−1)
.
.
.
𝑏1 − 𝑢1𝑛 𝑥𝑛 − 𝑢1(𝑛−1) 𝑥𝑛−1 − ⋯ − 𝑢12 𝑥2
𝑥1 =
𝑢11
𝒃𝒊 −∑𝒏
𝒌=𝒊+𝟏 𝒖𝒊𝒌 𝒙𝒌
This can be written in more compact form as follows: 𝒙𝒊 = (Backward
𝒖𝒊𝒊

substitution formula). A linear system of equations which has lower triangular coefficient matrix
has the form 𝐿𝑥 = 𝐵 is written as:
𝑙11 𝑥1 = 𝑏1

2|Page
 𝑙21 𝑥1 + 𝑙22 𝑥2 = 𝑏2
.
.
.
𝑙𝑛1 𝑥1 + 𝑙𝑛2 𝑥2 + ⋯ + 𝑙𝑛𝑛 𝑥𝑛 = 𝑏𝑛
Assuming 𝑙𝑖𝑖 ≠ 0, for all 𝑖 = 1,2, … , 𝑛, the unknowns are computed in the order of 𝑥1 , 𝑥2 , … , 𝑥𝑛
𝑏
as follows: 𝑥1 = 𝑙 1
11

𝑏2 − 𝑙21 𝑥1
𝑥2 =
𝑙22
.
.
.
𝑏𝑛 − 𝑙𝑛1 𝑥1 − 𝑙𝑛2 𝑥2 − ⋯ − 𝑙𝑛(𝑛−1) 𝑥𝑛−1
𝑥𝑛 =
𝑙𝑛𝑛
𝒃𝒊 −∑𝒊−𝟏
𝒌=𝟏 𝒍𝒊𝒌 𝒙𝒌
This can be written in more compact form as follows: 𝒙𝒊 = (Forward
𝒍𝒊𝒊

substitution formula)
3.1.1. Gauss elimination method
In this method, the unknowns are eliminated successively and the system is reduced to an upper
triangular system from which the unknowns are found by back substitution.
For the sake of simplicity, we restrict ourselves to the system of 3 equations in 3 unknowns:
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 = 𝑏2 (3.3)
𝑎31 𝑥1 + 𝑎32 𝑥2 + 𝑎33 𝑥3 = 𝑏3
Step-1: elimination of 𝑥1 from the second and the third equations. By assuming that 𝑎11 ≠ 0, we
eliminate 𝑥1 from the second equation by subtracting (𝑎21 /𝑎11) times the first equation from the
second equation. Similarly, we eliminate 𝑥1 from the third equation by subtracting (𝑎31 /𝑎11) times
the first equation from the third equation. We thus, get the new system
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 = 𝑏1
′ ′
𝑎22 𝑥2 + 𝑎23 𝑥3 = 𝑏2′ (3.4)
′ ′
𝑎32 𝑥2 + 𝑎33 𝑥3 = 𝑏3′

3|Page
Note: In a given system of linear equations, an equation which is used to eliminate variables from
other equations is called a pivot equation and its first nonzero coefficient is called a pivot (or
pivot element). Here ‘𝑎11 ’ is called the pivot element (or the first pivot) and the first equation is
called a pivot equation.
′
Step-2: elimination of 𝑥2 from the third equation in the new system (3.4). Assuming that𝑎22 ≠ 0,
′ ′
we eliminate 𝑥2 from the third equation of (3.4) by subtracting (𝑎32 /𝑎22 ) times the second equation
from the third equation. We thus, get the new upper triangular system
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 = 𝑏1
′ ′
𝑎22 𝑥2 + 𝑎23 𝑥3 = 𝑏2′
′′
𝑎33 𝑥3 = 𝑏3′′ (3.5)
Step-3: the unknowns 𝑥3 , 𝑥2 , and 𝑥1 are obtained from the upper triangular system (3.5) by back
substitution.
Remark 1: we make the above steps by applying one of the following elementary operations.
1. Interchanging of two rows
2. Multiplication of one row by a non-zero constant
3. Addition of a multiple of one row to another row.
 Finally, by applying backward substitution formula we find the values of the unknowns.
Example 3.1 Use Gaussian elimination to solve the linear system
10𝑥1 − 7𝑥2 + 3𝑥3 + 5𝑥4 = 6
−6𝑥1 + 8𝑥2 − 𝑥3 − 4𝑥4 = 5
3𝑥1 + 𝑥2 + 4𝑥3 + 11𝑥4 = 2
5𝑥1 − 9𝑥2 − 2𝑥3 + 4𝑥4 = 7
Solution: Set the equations as
10𝑥1 − 7𝑥2 + 3𝑥3 + 5𝑥4 = 6 (i)
−6𝑥1 + 8𝑥2 − 𝑥3 − 4𝑥4 = 5 (ii)
3𝑥1 + 𝑥2 + 4𝑥3 + 11𝑥4 = 2 (iii)
5𝑥1 − 9𝑥2 − 2𝑥3 + 4𝑥4 = 7 (iv)
First, to eliminate 𝑥1 from (ii), (iii), and (iv), we operate [(𝑖𝑖) + 6/10(𝑖)], [(𝑖𝑖𝑖) − 3/10(𝑖)], and
[(𝑖𝑣) − 5/10(𝑖)]. We obtain,
10𝑥1 − 7𝑥2 + 3𝑥3 + 5𝑥4 = 6 (i)
3.8𝑥2 + 3.1𝑥3 − 𝑥4 = 8.6 (v)

4|Page
 3.1𝑥2 + 3.1𝑥3 + 9.5𝑥4 = 0.2 (vi)
−5.5𝑥2 − 3.5𝑥3 + 1.5𝑥4 = 4 (vii)
Second, to eliminate 𝑥2 from (vi) and (vii), we operate [(vi)-3.1/3.8(v)] and
[(𝑣𝑖𝑖) + 5.5/3.8(𝑣)]. We obtain,
10𝑥1 − 7𝑥2 + 3𝑥3 + 5𝑥4 = 6 (i)
3.8𝑥2 + 3.1𝑥3 − 𝑥4 = 8.6 (v)
2.4473684𝑥3 + 10.315789𝑥4 = −6.8157895 (vii)
−2.3421053𝑥3 + 0.0526315𝑥4 = 16.447368 (ix)
Next, to eliminate 𝑥3 from (ix), we operate [(𝑖𝑥) + 2.3421053/2.4473684(𝑣𝑖𝑖𝑖)]. We thus, get
a system of upper triangular form
10𝑥1 − 7𝑥2 + 3𝑥3 + 5𝑥4 = 6
3.8𝑥2 + 3.1𝑥3 − 𝑥4 = 8.6
2.4473684𝑥3 + 10.315789𝑥4 = −6.8157895
9.9249319𝑥4 = 9.9245977
Finally, the unknowns are obtained by back substitution as
𝑥4 = 1, 𝑥3 = −7, 𝑥2 = 4, and 𝑥1 = 5.
Example 3.2 Given the system of equations
2𝑥1 + 3𝑥2 + 5𝑥3 = 23
3𝑥1 + 4𝑥2 + 𝑥3 = 14
6𝑥1 + 7𝑥2 + 2𝑥3 = 26
Solve for 𝑥1 , 𝑥2 and 𝑥3 by using Gaussian elimination method.
2 3 5 23
Solution: The augmented matrix of the system is: [𝐴: 𝑏] = (3 4 1 14)
6 7 2 26
2 3 5 23 −3𝑅 +𝑅 2 3 5 23
2 1 2 1 13 41
Step 1: (3 4 1 14) −3𝑅 (0 −2 − 2 − 2 ) = [𝐴(1) , 𝑏 (1) ]
1 +𝑅3
6 7 2 26 0 −2 −13 −43
2 3 5 23 2 3 5 23
1 13 41 1 13 41
Step 2: (0 − 2 − 2 − 2 ) − 4𝑅2 + 𝑅3 (0 − 2 − 2 − 2 ) = [𝑈, 𝑦]
0 −2 −13 −43 0 0 13 39
2 3 5 23
1 13 41
Here, (0 − 2 − 2 − 2 ) is the augmented matrix of the system of equations given below
0 0 13 39

5|Page
 2𝑥1 + 3𝑥2 + 5𝑥3 = 23 (1)
1 13 41
− 2 𝑥2 − 𝑥3 = − (2)
2 2

13𝑥3 = 39 (3)
Now, to find the values of 𝑥1 , 𝑥2 and 𝑥3 we apply the backward substitution formula. i.e. From
equation (3), we get 𝑥3 = 3 and substitute this into equation (2) and solve for 𝑥2 and we get
𝑥2 = 2. Finally, substitute values of 𝑥2 and 𝑥3 in equation (1) and solve for 𝑥1 , we get 𝑥1 = 1.
Therefore, the solution of the system is 𝑥 = {(𝑥1 , 𝑥2 , 𝑥3 )𝑡 = (1,2,3)𝑡 }.

3.1.2. Gauss elimination with partial pivoting

If the pivot is zero, the entire process in Gaussian elimination fails and if it closes to zero, round
off errors may occur. These problems can be avoided by adopting a procedure called pivoting. In
the first step, the numerically largest coefficient of 𝑥1 is chosen from all the equations from all the
equations and brought as the first pivot by interchanging the first equation with the equation having
the largest coefficient of 𝑥1 . In the second step, the numerically largest coefficient of 𝑥2 is chosen
from the remaining equations (leaving the first equation) and brought as the second pivot by
interchanging the second equation with the equation having the largest coefficient of 𝑥2 . This
process is continued till we arrive at the equation with the single variable. This modified procedure
is called partial pivoting.
Example 3.3 Use Gaussian elimination with partial pivoting method to solve the linear system
8𝑥2 + 2𝑥3 = −7
3𝑥1 + 5𝑥2 + 2𝑥3 = 8
6𝑥1 + 2𝑥2 + 8𝑥3 = 26
Solution: Using Gaussian elimination with partial pivoting, since the first pivot element is zero,
interchanging the third equation with the first equation we have,
6𝑥1 + 2𝑥2 + 8𝑥3 = 26 (1)
3𝑥1 + 5𝑥2 + 2𝑥3 = 8 (2)
8𝑥2 + 2𝑥3 = −7 (3)
Then, to eliminate 𝑥1 from (2), we operate [(2)-3/6(1)]. We thus, get the new system
6𝑥1 + 2𝑥2 + 8𝑥3 = 26 (4)
4𝑥2 − 2𝑥3 = −5 (5)
8𝑥2 + 2𝑥3 = −7 (6)
Next, interchanging (5) with (6) we have the system
6|Page
 6𝑥1 + 2𝑥2 + 8𝑥3 = 26 (7)
8𝑥2 + 2𝑥3 = −7 (8)
4𝑥2 − 2𝑥3 = −5 (9)
Then also, to eliminate 𝑥2 from (8) in the above system, we operate [(9)-4/8(8)]. Thus, we obtain
a system of an upper triangular form
6𝑥1 + 2𝑥2 + 8𝑥3 = 26 (10)
8𝑥2 + 2𝑥3 = −7 (11)
−3𝑥3 = −3/2 (12)
So that the unknowns are obtained by back substitution as follows:
𝑥3 = 1/2
1 1
𝑥2 = 8 [−7 − 2 (2)] = −1
1 1
𝑥1 = 6 [26 − 2(−1) − 8 (2)] = 4.

3.1.3. Gauss-Jordan elimination method

This is a modification of the Gaussian elimination method. In this method, elimination of the
unknowns is performed not in the equations below but in the equations above also, ultimately
reducing the system to a diagonal matrix form, i.e., equation involving only one unknown. From
the resulting system of equations the unknowns can be obtained readily.
Principle of the method: The Gauss Jordan method works by transforming the linear system
𝐴𝑥 = 𝑏 into an equivalent system 𝐴′ 𝑥 = 𝑏 ′ , where 𝐴 is an arbitrary non – singular matrix and 𝐴′
is an identity matrix of order 𝑛, so that 𝑥 = 𝑏 ′ is the solution of the original linear system.
Description of the method: Consider the following matrix equations
𝑎11 𝑎12 . . . 𝑎1𝑛 𝑥1 𝑏1
𝑎21 𝑎22 . . . 𝑎2𝑛 𝑥2 𝑏2
. . . . .
. . . . = .
. . . . .
(𝑎𝑛1 𝑎𝑛2 . . . 𝑎𝑛𝑛 ) (𝑥𝑛 ) (𝑏𝑛 )
Step 1: Assume that 𝑎11 ≠ 0, then replace 𝑎11 by 1 in the augmented matrix [𝐴: 𝑏] and this is
1
possible by pre – multiplying [𝐴: 𝑏] by elementary matrix 𝐸1 (𝑎 ). So after normalization the
11

first row of matrix [𝐴: 𝑏] becomes

𝑎
𝑎1𝑗 (1) = 1𝑗⁄𝑎11, for all 𝑗 = 1,2, … , 𝑛 + 1

7|Page
Now, we make the non – diagonal elements of the first column of 𝐴 to become zero. That is
𝑎𝑖1 = 0, for all 𝑖 = 2,3, … , 𝑛.
Step 2: Assuming 𝑎22 (2) ≠ 0, then replace 𝑎22 (2) by 1 in the augmented matrix [𝐴: 𝑏](1) and this
1
is possible by pre – multiplying [𝐴: 𝑏] by elementary matrix 𝐸2 (𝑎 ). So after normalization the
22

second row of matrix [𝐴: 𝑏](2) becomes

(2) 𝑎2𝑗 (1)

𝑎2𝑗 = ⁄ (1) , for all 𝑗 = 2,3, … , 𝑛 + 1
𝑎22
Now, we make the non – diagonal elements of the second column of 𝐴 to become zero. That is
𝑎𝑖2 = 0, for all 𝑖 = 1,3, … , 𝑛.
1
Step k: Assume 𝑎𝑘𝑘 (𝑘−1) ≠ 0, then multiply [𝐴: 𝑏](𝑘−1) by the elementary matrix 𝐸𝑘 (𝑎 (𝑘−1) ) to
𝑘𝑘

change 𝑎𝑘𝑘 (𝑘−1) to 1. The other coefficients of the 𝑘 𝑡ℎ row are given by:

(𝑘) 𝑎𝑘𝑗 (𝑘−1)

𝑎𝑘𝑗 = ⁄ (𝑘−1) , for all 𝑗 = 𝑘, 𝑘 + 1, … , 𝑛 + 1.
𝑎𝑘𝑘
The non – diagonal elements of the 𝑘 𝑡ℎ column must be zero. That is 𝑎𝑖𝑘 = 0, ∀𝑖 = 1,2, … , 𝑘 −
1, 𝑘 + 1, … , 𝑛.
Remark 2: Similar to the Gaussian elimination method the steps listed above are done by
applying the three elementary operations.
Example 3.4 Using Gauss Jordan method, solve the matrix equation defined by:
1 3 3 𝑥1 0
𝑥
(2 2 0) ( 2 ) = ( 2 )
3 3 6 𝑥3 11
Solution: The augmented matrix of the system of equation is given by:
1 3 3 0
[𝐴: 𝑏] = (2 2 0 2)
3 3 6 11
Step 1: Eliminate the coefficients of the first column below the diagonal element.
1 3 3 0 1 3 3 0
−2𝑅1 +𝑅2
(2 2 0 2 ) −3𝑅 +𝑅 (0 −4 −6 2 ) = [𝐴: 𝑏](1)
1 3
3 3 6 11 0 −6 −3 11
1
Step 2: Normalization is obtained on by pre – multiplying [𝐴: 𝑏](1) by 𝐸2 (− 4). i.e.

1 3 3 0 1 3 3 0
1 3 1
(0 −4 −6 2 ) − 4 𝑅2 (0 1 2
− 2)
0 −6 −3 11 0 −6 −3 11

8|Page
Once again eliminate the non – diagonal elements of the second column. i.e.
3 3
1 3 3 0 1 0 −2 2
3 1 −3𝑅2 +𝑅1
(0 1 2
− 2) (0 1 3 − 1) = [𝐴: 𝑏](2)
6𝑅2 +𝑅3
2 2
0 −6 −3 11 0 0 6 8
1
Step 3: Normalization is obtained on by pre – multiplying [𝐴: 𝑏](2) by 𝐸3 (6). i.e.
3 3
1 0 −2
3 3 1 0 −2 2
2 3 1
1
(0 − 2) 6 𝑅2 0 1 −2
3 1
1 2
2 4
0 0 6 8 0 0 1
( 3 )
Then, eliminate the non – diagonal elements of the third column. i.e.
3 3 7
1 0 −2 1 0 0
2 3 2
3 1 − 𝑅3 +𝑅2 5
0 1 2
−2 3
2
0 1 0 −2 = [𝐴: 𝑏](3)
𝑅 +𝑅1
2 3
4 4
0 0 1 0 0 1
( 3 ) ( 3 )
7
1 0 0 2
5
Here, 0 1 0 − 2 is the augmented matrix of the system of equation given below.
4
0 0 1
( 3 )
7 5 4
𝑥1 = 2, 𝑥2 = − 2, 𝑥3 = 3
7 5 4 𝑡
Therefore, the solution of the system is 𝑥 = {(𝑥1 , 𝑥2 , 𝑥3 )𝑡 = (2 , − 2 , 3) }.

Example 3.5 Use Gauss-Jordan elimination method to solve the system

𝑥1 + 𝑥2 + 𝑥3 = 9
2𝑥1 − 3𝑥2 + 4𝑥3 = 13
3𝑥1 + 4𝑥2 + 5𝑥3 = 40
Solution: Set
𝑥1 + 𝑥2 + 𝑥3 = 9 (i)
2𝑥1 − 3𝑥2 + 4𝑥3 = 13 (ii)
3𝑥1 + 4𝑥2 + 5𝑥3 = 40 (iii)
First, to eliminate 𝑥1 from (ii) and (iii), we operate [(ii)-2(i)] and [(iii)-3(i)]. We get,
𝑥1 + 𝑥2 + 𝑥3 = 9 (i)
−5𝑥2 + 2𝑥3 = −5 (iv)

9|Page
 𝑥2 + 2𝑥3 = 13 (v)
Second, to eliminate 𝑥2 from (i) and (v), we operate [(i)+1/5(iv)] and [(v)+1/5(iv)]. We get,
7
𝑥1 + 5 𝑥3 = 8 (vi)

−5𝑥2 + 2𝑥3 = −5 (iv)

12
𝑥3 = 12 (vii)
5

Next, to eliminate 𝑥3 from (iv) and (vi), we operate [(iv)-5/6(vii)] and [(vi)-7/12(vii)].
Finally, we obtain the diagonal system
𝑥1 = 1
−5𝑥2 = −15
12
𝑥3 = 12
5

Hence, the solutions are obtained directly as 𝑥1 = 1, 𝑥2 = 3, and 𝑥3 = 5.

3.1.4. Matrix inversion using Jordan’s method

In this section we consider some algebra associated with matrices and show how it can be used to
solve problems involving linear systems.
Definition: Two matrices A and B are equal if they have the same number of rows and columns,
say 𝑛×𝑚, and if 𝑎𝑖𝑗 = 𝑏𝑖𝑗 , for each 𝑖 = 1,2, … , 𝑛 and 𝑗 = 1,2, … , 𝑚.
Definition: Let A be an 𝑛×𝑚 matrix and B an 𝑚×𝑝 matrix. The matrix product of A and B,
denoted AB, is an 𝑛×𝑝 matrix C whose entries 𝑐𝑖𝑗 are
𝑐𝑖𝑗 = ∑𝑚
𝑘=1 𝑎𝑖𝑘 𝑏𝑘𝑗 = 𝑎𝑖1 𝑏1𝑗 + 𝑎𝑖2 𝑏2𝑗 + ⋯ + 𝑎𝑖𝑚 𝑏𝑚𝑗 , for each 𝑖 = 1,2, … , 𝑛 and 𝑗 = 1,2, … , 𝑝.

The computation of 𝑐𝑖𝑗 can be viewed as the multiplication of the entries of the 𝑖𝑡ℎ row of A with
corresponding entries in the 𝑗𝑡ℎ column of B, followed by a summation; that is,
𝑏1𝑗
[𝑎𝑖1 … 𝑎𝑖𝑚 ] [ ⋮ ] = 𝑐𝑖𝑗 , where 𝑐𝑖𝑗 = 𝑎𝑖1 𝑏1𝑗 + 𝑎𝑖2 𝑏2𝑗 + ⋯ + 𝑎𝑖𝑚 𝑏𝑚𝑗 .
𝑏𝑚𝑗
This explains why the number of columns of A must equal the number of rows of B for the product
AB to be defined.
Definition: A square matrix has the same number of rows as columns. A diagonal matrix D=[𝑑𝑖𝑗 ]
is a square matrix with 𝑑𝑖𝑗 = 0 whenever 𝑖 ≠ 𝑗. The identity matrix of order 𝑛, 𝐼𝑛 = [𝛿𝑖𝑗 ], is a
1, 𝑖𝑓 𝑖 = 𝑗
diagonal matrix with entries 𝛿𝑖𝑗 = { .
0, 𝑖𝑓 𝑖 ≠ 𝑗

10 | P a g e
Definition: An 𝑛×𝑛 matrix A is nonsingular (or invertible) if an 𝑛×𝑛 matrix 𝐴−1 exists with
𝐴𝐴−1 = 𝐴−1 𝐴 = 𝐼𝑛 . The matrix 𝐴−1 is called the inverse of A. A matrix without an inverse is
called singular (or noninvertible).
Reading assignment: determinant of a matrix
Condition for the existence of inverse
If the determinant of a square matrix A is nonzero, i.e., if det(𝐴) ≠ 0, then the matrix A is
nonsingular.
Consider a system of 𝑛 equations in 𝑛 unknowns
𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
. . .. . .
. . . .
𝑎𝑛1 𝑥1 + 𝑎𝑛2 𝑥2 + ⋯ + 𝑎𝑛𝑛 𝑥𝑛 = 𝑏𝑛
In matrix form, it is written as: 𝐴𝑥 = 𝑏,
𝑎11 𝑎12 … . 𝑎1𝑛 𝑥1 𝑏1
𝑎
Where, 𝐴 = [ 21. 𝑎22. … . 𝑎2𝑛 𝑥 𝑏
. ] , 𝑥 = [ . ] , 𝑏 = [ 2. ]
2
𝑎𝑛1 𝑎𝑛2 … . 𝑎𝑛𝑛 𝑥𝑛 𝑏𝑛
If the coefficient matrix A is invertible, then the solution of the above system is given by the
formula: 𝑥 = 𝐴−1 𝑏
The task to solve a certain system of linear equations using matrix-inversion method is then to find
the inverse of the coefficient matrix. There are several methods of finding the inverse of a matrix.
Some of the methods include:
 method of adjoint,
 Gauss elimination method, and
 Gauss-Jordan method.
In this section we will discuss Gauss-Jordan elimination method in finding the inverse of a
nonsingular coefficient matrix of a system of linear equations. In this method, the matrix is
converted in to an identity matrix. In practice, we apply the elementary row operations for the
augmented matrix formed by combining the coefficient matrix A and the identity matrix 𝐼𝑛 . As
soon as A is reduced to identity, the other matrix represents the inverse 𝐴−1 of the coefficient
matrix.

11 | P a g e
The augmented matrix of the coefficient matrix A and 𝐼𝑛 is of the form
𝑎11 𝑎12 … . 𝑎1𝑛 ⋮ 1 0… 0 1 0…. 0 ⋮ 𝑎11 𝑎12 … 𝑎1𝑛
[𝐴 ⋮ 𝐼] = [ 𝑎21. 𝑎22. … . 𝑎2𝑛
. ⋮ 0 1… 0] ⟹ [0 1…. 0 ⋮ 𝑎21. 𝑎22. … 𝑎2𝑛. ] = [𝐼 ⋮ 𝐴−1 ]
𝑎𝑛1 𝑎𝑛2 … . 𝑎𝑛𝑛 ⋮ 0 0… 1 0 0…. 1 ⋮ 𝑎𝑛1 𝑎𝑛2 … 𝑎𝑛𝑛
Let A be any matrix. Consider the following operation on the rows of A:
(1) Interchange of any two rows of A, symbolically, 𝑅𝑖 ↔ 𝑅𝑗 .
(2) Add scalar multiple one row of A to another row, i.e., 𝑅𝑖 + 𝛼𝑅𝑗 → 𝑅𝑖 , where 𝛼 ≠ 0.
(3) Multiply a row A with a nonzero scalar, i.e., 𝛼𝑅𝑖 → 𝑅𝑖 .
These operations are called elementary row operations of A.
We illustrate this procedure in the following example.
Example 3.6 Use Matrix inversion using Jordan’s method to solve the system
3𝑥1 + 𝑥2 + 2𝑥3 = 3
2𝑥1 − 3𝑥2 − 𝑥3 = −3
𝑥1 + 2𝑥2 + 𝑥3 = 4
3 1 2
Solution: The corresponding coefficient matrix A is given by: 𝐴 = [2 −3 −1] and det(𝐴) = 8.
1 2 1
−1
So that, A is nonsingular and hence 𝐴 exists. Now, the task is to find the inverse of A and we
use the Jordan’s method to do this.
3 1 2 1 0 0
Then the augmented matrix of A and 𝐼3 is defined as: (𝐴|𝐼3 ) = (2 −3 −1|0 1 0)
1 2 1 0 0 1
Using elementary row operations,
3 1 2 1 0 0
2 1
(2 −3 −1|0 1 0) [𝑅2 − 3 𝑅1 → 𝑅2 , 𝑅3 − 3 𝑅1 → 𝑅3 ]
1 2 1 0 0 1
3 1 2 1 0 0
3
~ (0 −11/3 −7/3|−2/3 1 0) [11 𝑅2 → 𝑅2 , 3𝑅3 → 𝑅3 ]
0 5/3 1/3 −1/3 0 1
3 1 2 1 0 0
~ (0 −1 −7/11|−2/11 3/11 0) [𝑅1 + 𝑅2 → 𝑅1 , 𝑅3 + 5𝑅2 → 𝑅3 ]
0 5 1 −1 0 3
3 0 15/11 9/11 3/11 0
7 15
~ (0 −1 −7/11 | −2/11 3/11 0) [𝑅2 − 𝑅3 → 𝑅2 , 𝑅1 + 𝑅3 → 𝑅1 ]
24 24
0 0 −24/11 −21/11 15/11 3
3 0 0 −9/24 27/24 45/24
1 11
~ (0 −1 0 | 9/24 −3/24 −21/24) [ 𝑅1 → 𝑅1 , −𝑅2 → 𝑅2 , − 𝑅3 → 𝑅3 ]
3 24
0 0 −24/11 −21/11 15/11 3

12 | P a g e
 1 0 0 −3/24 9/24 15/24
~ (0 1 0|−9/24 3/24 21/24 ).
0 0 1 21/24 −15/24 −33/24
Note: The symbol ~ is used to say that the given augmented matrix and the successive augmented
matrices are equivalent.
Hence, the inverse of the given coefficient matrix A is
−3/24 9/24 15/24
−1
𝐴 = [−9/24 3/24 21/24 ]
21/24 −15/24 −33/24
Thus, the solution of the system is given by
9 27 60
− − +
−3/24 9/24 15/24 24 24 24
3 27 9 84 1
−1
𝑥 = 𝐴 𝑏 = [−9/24 3/24 21/24 ] [−3] = − − + =[ 2 ]
21/24 −15/24 −33/24 4 24 24 24 −1
63 45 132
[ 24 + 24 − 24 ]
That is, 𝑥1 = 1, 𝑥2 = 2, and 𝑥3 = −1.
Note:
 Matrix inversion using Jordan’s method fails if the coefficient matrix A is a singular matrix.
 Matrix inversion using Jordan’s method is not suitable for a large system.
Example 3.7 Using Gauss Jordan method; find the inverse of the matrix
1 3 3
𝐴 = (2 2 0 )
3 2 6
1 3 3 1 0 0
Solution: The augmented matrix of 𝐴 and 𝐼 is given by: [𝐴: 𝐼3 ] = (2 2 0 0 1 0)
3 2 6 0 0 1
Transform the elements below the diagonal element in the first column of [𝐴: 𝐼3 ] to zero, i.e.
1 3 3 1 0 0 1 3 3 1 0 0
−2𝑅1 +𝑅2
(2 2 0 0 1 0) −3𝑅 +𝑅 (0 −4 −6 −2 1 0) = [𝐴: 𝐼](1)
1 3
3 2 6 0 0 1 0 −7 −3 −3 0 1
1
Now, multiply [𝐴: 𝐼](1) by 𝐸2 (− 4) to change 𝑎22 = −4 to 1. i.e.

1 3 3 1 0 0 1 3 3 1 0 0
1 3 1 1
(0 −4 −6 −2 1 0) − 4 𝑅2 (0 1 2 2
−
4
0)
0 −7 −3 −3 0 1 0 −7 −3 −3 0 1
Then, eliminate the non – diagonal elements of the second column, i.e.

13 | P a g e
 3 1 3
1 0 −2 −2 0
1 3 3 1 0 0 4
3 1 1 −3𝑅2 +𝑅1 3 1 1
(0 1 2 2
− 4
0) 0 1 2 2
−4 0 = [𝐴: 𝐼](2)
7𝑅2 +𝑅3
0 −7 −3 −3 0 1 15 1 7
0 0 −4 1
( 2 2 )
2 15
Now, multiply [𝐴: 𝐼](2) by 𝐸3 (15) to change 𝑎33 = to 1. i.e.
2
3 1 3 3 1 3
1 0 −2 −2 0 1 0 −2 −2 0
4 4
3 1 1 2 3 1 1
0 1 −4 0 𝑅 0 1 −4 0
2 2 15 3 2 2
15 1 7 1 7 2
0 0 −4 1 0 0 1 − 30
( 2 2 ) ( 15 15)

Then, eliminate the non – diagonal elements of the third column, i.e.
3 1 3 2 2 1
1 0 −2 −2 0 1 0 0 −5
4 3 5 5
3 1 1 𝑅 +𝑅1 2 1 1
2 3
0 1 2 2
−4 0 3 0 1 0 5 10
−5 = [𝐴: 𝐼](3)
− 𝑅3 +𝑅2
2
1 7 2 1 7 2
0 0 1 − 30 0 0 1 − 30
( 15 15) ( 15 15 )
Here, the last matrix is the end of the gauss Jordan method and in this 𝐴 is changed to identity
2 2 1
−5 5 5
−1 2 1 1
and this shows 𝐴 is invertible and its inverse is: 𝐴 = −5
5 10
1 7 2
− 30
( 15 15 )
3.1.5. Matrix decomposition method
This method is also known as LU-decomposition method. it is based on the fact a square matrix
𝐴 can be factorized into the form 𝐴 = 𝐿𝑈,where 𝐿 is lower triangular matrix and 𝑈 is unit upper
triangular matrix. Provided that the principal minors of 𝐴 are non – singular. i.e.
𝑎11 𝑎12 𝑎13
𝑎11 𝑎12
𝑎11 ≠ 0, |𝑎 | ≠ 0, | 21 𝑎22 𝑎23 | ≠ 0, etc.
𝑎
21 𝑎22
𝑎31 𝑎32 𝑎33
Now, let us consider the three equations
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 = 𝑏2
𝑎31 𝑥1 + 𝑎32 𝑥2 + 𝑎33 𝑥3 = 𝑏3
𝑎11 𝑎12 𝑎13 𝑥1 𝑏1
That is 𝐴𝑥 = 𝑏, where 𝐴 = (𝑎21 𝑎22 𝑎23 ) , 𝑥 = (𝑥2 ) and 𝑏 = (𝑏2 )
𝑎31 𝑎32 𝑎33 𝑥3 𝑏3

14 | P a g e
 𝑙11 0
0 1 𝑢12 𝑢13
Then, let 𝐴 = 𝐿𝑈, where 𝐿 = (𝑙21 𝑙22 0 ) and 𝑈 = (0 1 𝑢23 )
𝑙31 𝑙32 𝑙33 0 0 1
𝑙11 0 0 1 𝑢12 𝑢13 𝑎11 𝑎12 𝑎13
Therefore, 𝐿𝑈 = (𝑙21 𝑙22 0 ) (0 1 𝑢23 ) = (𝑎21 𝑎22 𝑎23 )
𝑙31 𝑙32 𝑙33 0 0 1 𝑎31 𝑎32 𝑎33
Now, solve for the unknowns, then we get the following
𝑙11 = 𝑎11 , 𝑙21 = 𝑎21 , 𝑙31 = 𝑎31
𝑎
𝑙11 ×𝑢12 = 𝑎12 ⟹ 𝑢12 = 𝑎12
11
𝑎13
𝑙11 ×𝑢13 = 𝑎13 ⟹ 𝑢13 = 𝑎
11
𝑎21 ×𝑎12
𝑙21 ×𝑢12 + 𝑙22 = 𝑎22 ⟹ 𝑙22 = 𝑎22 − 𝑎11
𝑎
𝑙31 ×𝑢12 + 𝑙32 = 𝑎32 ⟹ 𝑙32 = 𝑎32 − 𝑎31 × 𝑎12
11

1 𝑎
𝑙21 ×𝑢13 + 𝑙22 ×𝑢23 = 𝑎23 ⟹ 𝑢23 = 𝑙 (𝑎23 − 𝑎21 × 𝑎13 )
22 11
𝑎
𝑙31 ×𝑢13 + 𝑙32 ×𝑢23 + 𝑙33 = 𝑎33 ⟹ 𝑙33 = 𝑎33 − 𝑎31 × 𝑎13 − 𝑙32 ×𝑢23
11

𝑙11 0 0 𝑥1 𝑏1
Now, 𝐴𝑥 = 𝑏 ⟹ 𝐿𝑈𝑥 = 𝑏, then let 𝑈𝑥 = 𝑣 ⟹ 𝐿𝑣 = 𝑏. ⟹ (𝑙21 𝑙22 0 ) (𝑥2 ) = (𝑏2 )
𝑙31 𝑙32 𝑙33 𝑥3 𝑏3
⟹ 𝑙11 ×𝑣1 = 𝑏1
𝑙21 ×𝑣1 + 𝑙22 ×𝑣2 = 𝑏2
𝑙31 ×𝑣1 + 𝑙32 ×𝑣2 + 𝑙31 ×𝑣3 = 𝑏3
Then, solve for 𝑣1 , 𝑣2 and 𝑣3 by forward substitution formula. Then we have
1 𝑢12 𝑢13 𝑥1 𝑣1
𝑈𝑥 = 𝑣 ⟹ (0 1 𝑢23 ) (𝑥2 ) = (𝑣2 )
0 0 1 𝑥3 𝑣3
⟹ 𝑥1 + 𝑢12 ×𝑥2 + 𝑢13 ×𝑥3 = 𝑣1
𝑥2 + 𝑢23 ×𝑥3 = 𝑣2
𝑥3 = 𝑣3
Then, solve for 𝑥1 , 𝑥2 and 𝑥3 using the backward substitution formula.
Example 3.8 Using matrix decomposition method solve the following system of linear
equations.
𝑥+𝑦+𝑧 =1

15 | P a g e
 4𝑥 + 3𝑦 − 𝑧 = 6
3𝑥 + 5𝑦 + 3𝑧 = 4
Solution: The system is given as:
1 1 1 𝑥 1
(4 3 −1) (𝑦) = (6)
3 5 3 𝑧 4
𝑙11 0 0 1 𝑢12 𝑢13 1 1 1
Let 𝐿𝑈 = 𝐴 ⟹ (𝑙21 𝑙22 0 ) (0 1 𝑢23 ) = (4 3 −1)
𝑙31 𝑙32 𝑙33 0 0 1 3 5 3
Then, solve the unknowns by equating the two matrices and we get
𝑙11 = 1, 𝑙21 = 4, 𝑙31 = 3
𝑢12 = 1, 𝑢13 = 1
𝑙22 = −1, 𝑙32 = 2
𝑢23 = 5 and 𝑙33 = −10
1 0 0 𝑣1 1
Then, we have: 𝐿𝑣 = 𝑏 ⟹ (4 −1 𝑣
0 ) ( 2 ) = (6)
3 2 −10 𝑣3 4
⟹ 𝑣1 = 1
4𝑣1 − 𝑣2 = 6
3𝑣1 + 2𝑣2 − 10𝑣3 = 4
Then, by applying the forward substitution formula we get
1
𝑣1 = 1, 𝑣2 = −2 and 𝑣3 = − 2

1 1 1 𝑥 1
−2
Then, 𝑈𝑥 = 𝑣 ⟹ (0 1 5) (𝑦) = ( 1)
0 0 1 𝑧 −2

⟹𝑥+𝑦+𝑧 =1
𝑦 + 5𝑧 = −2
1
𝑧 = −2

Then, by applying the backward substitution formula we get

1 1
𝑧 = − 2 , 𝑦 = 2 and 𝑥 = 1
1 1 𝑡
Therefore, the solution of the system is 𝑥 = {(𝑥, 𝑦, 𝑧)𝑡 = (1, 2 , − 2) }.

Example 3.9 Use Matrix decomposition method to solve the system

16 | P a g e
 2𝑥1 + 3𝑥2 + 𝑥3 = 9
𝑥1 + 2𝑥2 + 3𝑥3 = 6
3𝑥1 + 𝑥2 + 2𝑥3 = 8
2 3 1
Solution: the corresponding coefficient matrix is: 𝐴 = [1 2 3]
3 1 2
1 0 0 𝑢11 𝑢12 𝑢13 2 3 1
Let [𝑙21 1 0] [ 0 𝑢22 𝑢23 ] = [1 2 3]
𝑙31 𝑙32 1 0 0 𝑢33 3 1 2
i) Clearly, 𝑢11 = 2, 𝑢12 = 3, 𝑢13 = 1, and 𝑙21 𝑢11 = 1, 𝑙31 𝑢11 = 3. So that, 𝑙21 = 1/2, and
𝑙31 = 3/2. For 𝑢22 and 𝑢23 , we have the equations 𝑙21 𝑢12 + 𝑢22 = 2 and 𝑙21 𝑢13 + 𝑢23 = 3.
This implies that, 𝑢22 = 1/2 and 𝑢23 = 5/2. Finally, 𝑙32 and 𝑢33 are obtained from 𝑙31 𝑢12 +
𝑙32 𝑢22 = 1 and 𝑙31 𝑢13 + 𝑙32 𝑢23 + 𝑢33 = 2 and hence, 𝑙32 = −7 and 𝑢33 = 18. It follows that,
1 0 0 2 3 1
𝐿𝑈 = [1/2 1 0] [0 1/2 5/2] (ii)
3/2 −7 1 0 0 18
Then the system of equations can be written as
1 0 0 𝑣1 9
𝐿𝑣 = [1/2 1 0] [𝑣2 ] = [6] (iii)
3/2 −7 1 𝑣3 8
Where,
2 3 1 x1 𝑣1
[0 1/2 5/2] [x2 ] = [𝑣2 ] (iv)
0 0 18 x3 𝑣3
Hence solving the system in (iii) by forward substitution, we get
v1 = 9, v2 = 3/2, and v3 = 5.
Thus, solving the system in (iv) by backward substitution for the unknowns, we obtain
x1 = 35/18, x2 = 29/18, and x3 = 5/18.

3.1.6. Thomas Algorithm for Tridiagonal System

Consider the system of linear simultaneous algebraic equations given by 𝑨𝒙 = 𝑏, Where A is a
tridiagonal matrix, 𝑥 = [𝑥1 , 𝑥2 , … , 𝑥𝑛 ]𝑇 𝑎𝑛𝑑 𝑏 = [𝑏1 , 𝑏2 , … , 𝑏𝑛 ]𝑇 .
Hence, we consider a 4 × 4 tridiagonal system of equations given by
𝑎12 𝑎13 0 0 𝑥1 𝑏1
𝑎 𝑎22 𝑎23 0 𝑥2 𝑏
[ 21 ] [𝑥 ] = [ 2 ] (1)
0 𝑎31 𝑎32 𝑎33 3 𝑏3
0 0 𝑎41 𝑎42 𝑥4 𝑏4

17 | P a g e
Equation (1) can be written as
𝑎12 𝑥1 + 𝑎13 𝑥2 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 = 𝑏2
𝑎31 𝑥2 + 𝑎32 𝑥3 + 𝑎33 𝑥4 = 𝑏3 (2)
𝑎41 𝑥3 + 𝑎42 𝑥4 = 𝑏4
The system of equations given by Eq. (2) is solved using Thomas Algorithm which is described
in three steps as shown below:
𝑎𝑖1 𝑎(𝑖−1)3
Step 1: Set 𝑦1 = 𝑎12 and compute 𝑦𝑖 = 𝑎𝑖2 − 𝑖 = 2,3,4 … 𝑛
𝑦𝑖−1
𝑏 𝑏𝑖 −𝑎𝑖1 𝑧𝑖−1
Step 2: Set 𝑧1 = 𝑦1 = and compute 𝑧𝑖 = 𝑖 = 2,3,4 … 𝑛
1 𝑦𝑖
𝑎𝑖3 𝑥𝑖+1
Step 2: Set 𝑥𝑛 = 𝑧𝑛 and compute 𝑥𝑖 = 𝑧𝑖 − 𝑖 = 𝑛 − 1, 𝑛 − 2 , … ,1.
𝑦𝑖

Example 3.10 Solve the following equations by Thomas Algorithm.

3𝑥1 − 𝑥2 = 5
2𝑥1 − 3𝑥2 + 2𝑥3 = 5
𝑥2 + 2𝑥3 + 5𝑥4 = 10
𝑥3 − 𝑥4 = 1
3 − 1 0 0 𝑥1 5
2 − 3 2 0 𝑥2 5
Solution Here [ ] ⌈𝑥 ⌉=⌈ ⌉
0 1 2 5 3 10
0 0 1 − 1 𝑥4 1
⌈𝑎21 𝑎31 𝑎41 ⌉ = [2 1 1]
⌈𝑎12 𝑎22 𝑎32 𝑎42 ⌉ = [3 − 3 2 − 1]
⌈𝑎13 𝑎23 𝑎33 ⌉ = [−1 2 5]
⌈𝑏1 𝑏2 𝑏3 𝑏4 ⌉ = [5 5 10 1]
𝑎𝑖1 𝑎(𝑖−1)3
Step 1: Set 𝑦1 = 𝑎12 and compute 𝑦𝑖 = 𝑎𝑖2 − 𝑖 = 2,3,4 … 𝑛
𝑦𝑖−1
𝑎21 𝑎13 2(−1) 7
𝑦1 = 3 𝑖=2 𝑦2 = 𝑎22 − = −3 − = −3
𝑦1 3

𝑎31 𝑎23 1𝑥2 20

𝑖=3 𝑦3 = 𝑎32 − =2− =
𝑦2 −7⁄ 7
3
𝑎41 𝑎33 1𝑥5 55
𝑖=4 𝑦4 = 𝑎42 − = −1 − =−
𝑦3 20⁄ 20
7
𝑏 5 𝑏𝑖 −𝑎𝑖1 𝑧𝑖−1
Step 2: Set 𝑧1 = 𝑦1 =3 and compute 𝑧𝑖 = 𝑖 = 2,3,4 … 𝑛
1 𝑦𝑖

18 | P a g e
 𝑏2 − 𝑎21 𝑧1 5 − 2(5⁄3) 5
𝑖=2 𝑧2 = = =−
𝑦2 −7⁄ 7
3
𝑏3 − 𝑎31 𝑧2 10 − 1(−5⁄7) 75
𝑖=3 𝑧3 = = =
𝑦3 20⁄ 20
7
𝑏4 − 𝑎41 𝑧3 1 − 1(75⁄20)
𝑖=4 𝑧4 = = =1
𝑦4 −55⁄
20
𝑎𝑖3 𝑥𝑖+1
Step 2: Set 𝑥4 = 𝑧4 = 1 and compute 𝑥𝑖 = 𝑧𝑖 − 𝑖 = 𝑛 − 1, 𝑛 − 2 , … ,1.
𝑦𝑖

𝑎33 𝑥4 75 5𝑥1
𝑖=3 𝑥3 = 𝑧3 − = − =2
𝑦3 20 20⁄7
𝑎23 𝑥3 5 2𝑥2
𝑖=2 𝑥2 = 𝑧2 − = − =1
𝑦2 7 −7⁄3
𝑎13 𝑥2 5 (−1)(2)
𝑖=1 𝑥1 = 𝑧1 − = − =2
𝑦1 3 3
3.2. Indirect methods for system of linear equations
Introduction: As a numerical technique, Gaussian elimination is rather unusual because it is
direct. That is, a solution is obtained after a single application of Gaussian elimination. Once a
“solution” has been obtained, Gaussian elimination offers no method of refinement. The lack of
refinements can be a problem because, as the previous section shows, Gaussian elimination is
sensitive to rounding error.
Numerical techniques more commonly involve an iterative method. For example, in Calculus we
probably studied Newton’s iterative method for approximating the zeros of a differentiable
function. In this section we will look at two iterative methods for approximating the solution of a
system of n linear equations in n variables.
3.2.1. Gauss Jacobi Method
The first iterative technique is called the Jacobi method, after Carl Gustav Jacob Jacobi
(1804–1851). This method makes two assumptions:
1. The system given by
𝑎11 𝑥1 + 𝑎12 𝑥2 + ⋯ + 𝑎1𝑛 𝑥𝑛 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + ⋯ + 𝑎2𝑛 𝑥𝑛 = 𝑏2
⋮ ⋮ ⋮ ⋮
𝑎𝑛1 𝑥1 + 𝑎𝑛2 𝑥2 + ⋯ + 𝑎𝑛𝑛 𝑥𝑛 = 𝑏𝑛

19 | P a g e
 has a unique solution
2. The coefficient matrix 𝐴 has no zeros on its main diagonal, namely 𝑎11 , 𝑎22 , … , 𝑎𝑛𝑛 are nonzero.
That is, if any of the diagonal entries are zero, then rows or columns must be interchanged to obtain
a coefficient matrix that has nonzero entries on the main diagonal.
Main idea of Jacobi
To begin, solve the 1st equation for 𝑥1 , the 2nd equation for 𝑥2 and so on to obtain the rewritten
equations:
1
𝑥1 = (𝑏 − 𝑎12 𝑥2 − 𝑎13 𝑥3 − ⋯ − 𝑎1𝑛 𝑥𝑛 )
𝑎11 1
1
𝑥2 = (𝑏 − 𝑎21 𝑥1 − 𝑎23 𝑥3 − ⋯ − 𝑎2𝑛 𝑥𝑛 )
𝑎22 2
⋮
1
𝑥𝑛 = 𝑎 (𝑏𝑛 − 𝑎𝑛1 𝑥1 − 𝑎𝑛2 𝑥2 − ⋯ − 𝑎𝑛−1 𝑥𝑛−1)
𝑛𝑛

Then make an initial guess (assumption) of the solution 𝑋 (0) = (𝑥1 (0) , 𝑥2 (0) , … , 𝑥𝑛 (0) ). Substitute
these values into the right hand side the of the rewritten equations to obtain the first approximation,
(𝑥1 (1) , 𝑥2 (1) , … , 𝑥𝑛 (1) ). This accomplishes one iteration.
In the same way, the second approximation (𝑥1 (2) , 𝑥2 (2) , … , 𝑥𝑛 (2) ) is computed by substituting
the first approximation’s 𝑋 − values into the right hand side of the rewritten equations.
By repeated iterations, we form a sequence of approximations
𝑋 (𝑘) = (𝑥1 (𝑘) , 𝑥2 (𝑘) , … , 𝑥𝑛 (𝑘) ), 𝑘 = 1,2,3, ….
The Jacobi Method, for each 𝑘 ≥ 1, generate the components of from 𝑥𝑖 (𝑘) of 𝑋 (𝑘) by

𝑛
(𝑘)
1
𝑥𝑖 = ∑(−𝑎𝑖𝑗 𝑥𝑗 (𝑘−1) ) + 𝑏𝑖 , 𝑓𝑜𝑟 𝑖 = 1,2, … , 𝑛
𝑎𝑖𝑖
𝑗=1,
[ 𝑗≠𝑖 ]
Example 3.11: Apply the Jacobi method to solve
5𝑥1 − 2𝑥2 + 3𝑥3 = −1
−3𝑥1 + 9𝑥2 + 𝑥3 = 2
2𝑥1 − 𝑥2 − 7𝑥3 = 3
Continue the iterations until two successive approximations are identical when rounded to three
significant digits.
Solution: To begin, write the system in the form
20 | P a g e
 1 2 3
𝑥1 = − + 𝑥2 − 𝑥3
5 5 5
2 3 1
𝑥2 = + 𝑥1 − 𝑥3
9 9 9
3 2 1
𝑥3 = − + 𝑥1 − 𝑥2
7 7 7
Because we do not know the actual solution, choose
𝑥1 = 0, 𝑥2 = 0, 𝑥3 = 0 Initial approximation
as a convenient initial approximation. So, the first approximation is
1 2 3
𝑥1 = − + (0) − (0) = −0.200
5 5 5
2 3 1
𝑥2 = + (0) − (0) ≈ 0.222
9 9 9
3 2 1
𝑥3 = − + (0) − (0) ≈ −0.429
7 7 7
That is (𝑥1 (1) , 𝑥2 (1) , 𝑥3 (1) ) = (−0.200, 0.222, −0.429)
Continuing this procedure, we obtain the sequence of approximations shown in Table 3.2.
Table 3.2
𝑛 0 1 2 3 4 5 6 7
𝑥1 0.000 −0.200 0.146 0.192 𝑜. 181 0.185 0.186 0.186

𝑥2 0.000 0.222 0.203 0.328 0.332 0.329 0.331 0.331

𝑥3 0.000 −0.429 −0.517 −0.416 −0.421 −0.424 −0.423 −0.423

Because the last two columns in Table 3.2 are identical, we can conclude that to three significant
digits the solution is
𝑥1 = 0.186, 𝑥2 = 0.331, 𝑎𝑛𝑑 𝑥3 = −0.423.
Note: For the system of linear equations given in Example 3.10, the Jacobi method is said to
converge. That is, repeated iterations succeed in producing an approximation that is correct to
three significant digits. As is generally true for iterative methods, greater accuracy would require
more iteration.

21 | P a g e
3.2.2. The Gauss-Seidel Method
You will now look at a modification of the Jacobi method called the Gauss-Seidel method, named
after Carl Friedrich Gauss (1777–1855) and Philipp L. Seidel (1821–1896). This modification is
no more difficult to use than the Jacobi method, and it often requires fewer iterations to produce
the same degree of accuracy. With the Jacobi method, the values of 𝑥𝑖 obtained in the nth
approximation remain Unchanged until the entire (𝑛 + 1)th approximation has been calculated.
With the Gauss-Seidel method, on the other hand, we use the new values of each as soon as they
are known. That is, once we have determined 𝑥1 from the first equation, its value is then used in
the second equation to obtain the new 𝑥2 . similarly, the new 𝑥1 and 𝑥2 are used in the third
equation to obtain the new 𝑥3 and so on.
Example 3.12 Use the Gauss-Seidel iteration method to approximate the solution to the system
of equations given in Example 3.10.
Solution: The first computation is identical to that given in Example 3.10. That is, using
(𝑥1 , 𝑥2 , 𝑥3 ) = (0,0,0)
as the initial approximation, we obtain the following new value for 𝑥1 .
1 2 3
𝑥1 = − + (0) − (0) = −0.200
5 5 5
Now that we have a new value for 𝑥1 , however, use it to compute a new value for 𝑥2 . That is,
2 3 1
𝑥2 = + (−0.200) − (0) ≈ 0.156
9 9 9
Similarly, use 𝑥1 = −0.200 and 𝑥2 = 0.156 to copmute a new value for 𝑥3 . That is,
3 2 1
𝑥3 = − 7 + 7 (−0.200) − 7 (0.156) ≈ −0.508.

So the first approximation is 𝑥1 = −0.200, 𝑥2 = 0.156, 𝑎𝑛𝑑 𝑥3 = −0.508. Continued iterations

produce the sequence of approximations shown in Table 3.3.
Table 3.3
𝑛 0 1 2 3 4 5
𝑥1 0.000 −0.200 0.167 0.191 0.186 0.186
𝑥2 0.000 0.156 0.334 0.333 0.331 0.331
𝑥3 0.000 −0.508 −0.429 −0.422 −0.423 −0.423

22 | P a g e
Note that after only five iterations of the Gauss-Seidel method, we achieved the same accuracy as
was obtained with seven iterations of the Jacobi method in Example 3.2.
Note: Neither of the iterative methods presented in this section always converges. That is, it is
possible to apply the Jacobi method or the Gauss-Seidel method to a system of linear equations
and obtain a divergent sequence of approximations. In such cases, it is said that the method
diverges.
Example 3.13: (An Example of Divergence) Apply the Jacobi method to the system
𝑥1 − 5𝑥2 = −4
7𝑥1 − 𝑥2 = 6
using the initial approximation (𝑥1 , 𝑥2 ) = (0,0), and show that the method diverges.
Solution: As usual, begin by rewriting the given system in the form
𝑥1 = −4 + 5𝑥2
𝑥2 = −6 + 7𝑥1
Then the initial approximation (0, 0) produces
𝑥1 = −4 + 5(0) = −4
𝑥2 = −6 + 7(0) = −6
As the first approximation. Repeated iterations produce the sequence of approximations shown in
Table 3.4.
Table 3.4
𝑛 0 1 2 3 4 5 6 7
𝑥1 0 −4 −34 −174 −1244 −6124 −42874 −214374
𝑥2 0 −6 −34 −244 −1244 −6574 −42874 −300124

For this particular system of linear equations we can determine that the actual solution is 𝑥1 = 1
and 𝑥2 = 1. So we can see from Table 3.4 that the approximations given by the Jacobi method
become progressively worse instead of better, and you can conclude that the method diverges.
The problem of divergence in Example 3.3 is not resolved by using the Gauss-Seidel method rather
than the Jacobi method. In fact, for this particular system the Gauss-Seidel method diverges more
rapidly, as shown in Table 3.4.
Table 3.4

23 | P a g e
 𝑛 0 1 2 3 4 5
𝑥1 0 −4 −174 −6124 −214374 −7503124
𝑥2 0 −34 −1224 −42874 −1500624 −52521874

With an initial approximation of (𝑥1 , 𝑥2 ) = (0,0) neither the Jacobi method nor the Gauss-Seidel
method converges to the solution of the system of linear equations given in Example 3.3. We will
now look at a special type of coefficient matrix 𝐴, called a strictly diagonally dominant matrix, for
which it is guaranteed that both methods will converge.
Definition: (Strictly Diagonally Dominant Matrix)
An 𝑛×𝑛 matrix A is strictly diagonally dominant if the absolute value of each entry on the main
diagonal is greater than the sum of the absolute values of the other entries in the same row. Thus,
|𝑎𝑖𝑖 | > ∑𝑛𝑗=1|𝑎𝑖𝑗 |
Example 3.14 which of the following systems of linear equations has a strictly diagonally
dominant coefficient matrix?
4𝑥1 + 2𝑥2 − 𝑥3 = −1
3𝑥 − 𝑥2 = −4
𝑎) [ 1 ] 𝑏) [ 𝑥1 + 2𝑥3 = −4 ]
2𝑥1 + 5𝑥2 = 2
3𝑥1 − 5𝑥2 + 𝑥3 = 3
3 −1
𝐴=[
Solution: a) The coefficient matrix of the system is: ]
2 5
And is strictly diagonally dominant because|3| > |−1|𝑎𝑛𝑑 |5| > |2|.
4 2 −1
b) The coefficient matrix of the system is: 𝐴 = [1 0 2]
3 −5 1
is not strictly diagonally dominant because the entries in the second and third rows do not conform
to the definition. For instance, in the second row 𝑎21 = 1, 𝑎22 = 0, 𝑎23 = 2 and it is not true
that|𝑎22 | > |𝑎21 | + |𝑎23 |. Interchanging the second and third rows in the original system of linear
4 2 −1
equations, however, produces the coefficient matrix: 𝐴 = [3 −5 1 ] and this matrix is
1 0 2
strictly diagonally dominant.
The following theorem, which is listed without proof, states that strict diagonal dominance is
sufficient for the convergence of either the Jacobi method or the Gauss-Seidel method.
Theorem 3.2 (Convergence of the Jacobi and Gauss-Seidel methods)

24 | P a g e
If 𝐴 is strictly diagonally dominant, then the system of linear equations given by 𝐴𝒙 = 𝒃 has a
unique solution to which the Jacobi method and the Gauss-Seidel method will converge for any
initial approximation.
In Example 3.13 we looked at a system of linear equations for which the Jacobi and Gauss-Seidel
methods diverged. In the following example you can see that by interchanging the rows of the
system given in Example 3.13, we can obtain a coefficient matrix that is strictly diagonally
dominant. After this interchange, convergence is assured.
Example 3.15: Interchange the rows of the system
𝑥1 − 5𝑥2 = −4
7𝑥1 − 𝑥2 = 6
to obtain one with a strictly diagonally dominant coefficient matrix. Then apply the Gauss-Seidel
method to approximate the solution to four significant digits.
Solution: Begin by interchanging the two rows of the given system we obtain
7𝑥1 − 𝑥2 = 6
𝑥1 − 5𝑥2 = −4
Note that the coefficient matrix of this system is strictly diagonally dominant. Then solve
for 𝑥1 and 𝑥2 as follows.
6 1
𝑥1 = + 𝑥
7 7 2
4 1
𝑥2 = + 𝑥1
5 5
Using the initial approximation (𝑥1 , 𝑥2 ) = (0,0) you can obtain the sequence of approximations
shown in Table 3.5.
Table 3.5
𝑛 0 1 2 3 4 5
𝑥1 0.0000 0.8571 0.9959 0.9999 1.000 1.000
𝑥2 0.0000 0.9714 0.9992 1.000 1.000 1.000
So we can conclude that the solution is 𝑥1 = 1 𝑎𝑛𝑑 𝑥2 = 1.
Note: Theorem 3.2 does not imply that strict diagonal dominance is a necessary condition for
convergence of the Jacobi or Gauss-Seidel methods. For instance, the coefficient matrix of the
system
−4𝑥1 + 5𝑥2 = 1
𝑥1 + 2𝑥2 = 3
25 | P a g e
Here the coefficient of the system is not a strictly diagonally dominant matrix, and yet both
methods converge to the solution 𝑥1 = 1 and 𝑥2 = 1 when you use an initial approximation of
(𝑥1 , 𝑥2 ) = (0,0).

3.3. Solving systems of Non-linear Equations using Newton’s Method

Introduction:
Solving a system of nonlinear equations is a problem that is avoided when possible, customarily
by approximating the nonlinear system by a system of linear equations. When this is
unsatisfactory, the problem must be tackled directly
Definition: A system of nonlinear equations has the form
𝑓1 (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 0
𝑓2 (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 0
⋮ ⋮
𝑓𝑛 (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 0
where each function 𝑓𝑖 can be thought of as mapping a vector 𝒙 = (𝑥1 , 𝑥2 , . . . , 𝑥𝑛 )𝑡 of the 𝑛-
dimensional space ℝ𝑛 into the real line ℝ.
In general, we can usually find solutions to a system of equations when the number of unknowns
matches the number of equations. Thus, we wish to find solutions to systems that have the form:
𝑓1 (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 0
𝑓2 (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 0
⋮ ⋮
𝑓𝑛 (𝑥1 , 𝑥2 , … , 𝑥𝑛 ) = 0. (1)
For convenience we can think of 𝑥1 , 𝑥2 , … , 𝑥𝑛 ) as a vector 𝒙 and (𝑓1 , 𝑓2 , … , 𝑓𝑛 ) as a vector valued
function 𝑓 . With this notation, we can write the system of equations (1) simply as: 𝒇(𝒙) = 𝟎.
That is, we wish to find a vector that makes the vector function equal to the zero vectors.
As in Newton’s method for one variable, we need to start with an initial guess 𝒙0 . In theory, the
more variables one has, the harder it is to find a good initial guess. In practice, this must be
overcome by using physically reasonable assumptions about the possible values of a solution, i.e.
take advantage of engineering knowledge of the problem. Once 𝒙0 is chosen, let
∆𝒙 = 𝒙𝟏 − 𝒙𝟎 .
Linear Approximation for Vector Functions

26 | P a g e
In the single variable case, Newton’s method was derived by considering the linear approximation
of the function 𝑓 at the initial guess 𝒙0. From Calculus, the following is the linear approximation
of 𝑓 at 𝒙0 , for vectors and vector-valued functions:
𝒇(𝒙) ≈ 𝒇(𝒙0 ) + 𝐷𝒇(𝒙0 )(𝒙 − 𝒙0 ).
Here 𝐷𝒇(𝒙0 ) is an 𝑛 × 𝑛 matrix whose entries are the various partial derivative of the components
of. Specifically:
𝜕𝑓1 𝜕𝑓1 𝜕𝑓1
(𝒙0 ) (𝒙0 ) (𝒙0 )
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
𝜕𝑓2 𝜕𝑓2
⋯ 𝜕𝑓2
(𝒙0 ) (𝒙0 ) (𝒙0 )
𝐷𝒇(𝒙0 ) = 𝜕𝑥 𝜕𝑥2 𝜕𝑥𝑛 (2)
⋮ ⋮ ⋱ ⋮
𝜕𝑓𝑛 𝜕𝑓𝑛 𝜕𝑓𝑛
(𝒙0 ) (𝒙0 ) ⋯ (𝒙0 )]
[𝜕𝑥 1 𝜕𝑥 2 𝜕𝑥𝑛

The matrix 𝐷𝑓(𝑥0 ) is called the Jacobian matrix and id denoted by 𝐽(𝒙0 )and has a number of
applications in analysis. That is
𝜕𝑓1 𝜕𝑓1 𝜕𝑓1
(𝒙0 ) (𝒙 ) (𝒙 )
𝜕𝑥1 𝜕𝑥2 0 𝜕𝑥𝑛 0
⋯
𝜕𝑓2 𝜕𝑓2 𝜕𝑓2
𝐽(𝒙0 ) = 𝜕𝑥 (𝒙0 ) 𝜕𝑥2 (𝒙0 ) (𝒙 )
𝜕𝑥𝑛 0
⋮ ⋮ ⋱ ⋮
𝜕𝑓𝑛 𝜕𝑓𝑛 𝜕𝑓𝑛
(𝒙 ) (𝒙 ) ⋯ (𝒙 )
[𝜕𝑥1 0 𝜕𝑥2 0 𝜕𝑥𝑛 0 ]
Newton’s method
We wish to find 𝒙 that makes 𝒇 equal to the zero vectors, so let’s choose 𝒙1 so that
𝒇(𝒙0 ) + 𝐽(𝒙0 )(𝒙1 − 𝒙0 ) = 0
Since 𝐷𝒇(𝒙0 ) is a square matrix, we can solve this equation by
𝒙1 = 𝒙0 − (𝐽(𝒙0 ))−1 𝒇(𝒙0 )
provided that the inverse exists. The formula is the vector equivalent of the Newton’s method
formula we learned before. However, in practice we never use the inverse of a matrix for
computations, so we cannot use this formula directly. Rather, we can do the following. First solve
the equation
𝐽(𝒙0 )∆𝒙 = −𝒇(𝒙0 ) (3)
Since 𝐽(𝒙0 ) is a known matrix and −𝒇 (𝒙0 ) is a known vector, this equation is just a system of
linear equations, which can be solved efficiently and accurately. Once we have the solution vector
∆𝒙, we can obtain our improved estimate 𝒙1 by:

27 | P a g e
 𝒙1 = 𝒙0 + ∆𝒙
For subsequent steps, we have the following process:
 Solve 𝐽(𝒙𝑖 )∆𝒙 = −𝒇(𝒙𝑖 ) for ∆𝒙
 Let 𝒙𝑖+1 = 𝒙1 + ∆𝒙
Example 3.16 Solve the following non-linear system equations using Newton’s method by leting
𝑥0 = (2,1) for about four approximations.
𝑥 2 + 𝑦 2 − 50 = 0
𝑥𝑦 − 25 = 0
Solution: let
𝑓1 (𝑥, 𝑦) = 𝑥 2 + 𝑦 2 − 50 = 0
𝑓2 (𝑥, 𝑦) = 𝑥𝑦 − 25 = 0
𝑥 2 + 𝑦 2 − 50
⟹ 𝑓(𝒙) = [ ]
𝑥𝑦 − 25
2𝑥 2𝑦
From this we have: 𝐽(𝒙) = [ ] and ∆𝒙 = 𝐽(𝒙)−1 (−𝑓(𝒙))
𝑦 𝑥
−45
Where 𝑓(𝒙0 ) = [ ] because 𝒙0 = (2,1), which implies
−23
45 4 2 1 2 −2
−𝑓(𝒙0 ) = [ ] and 𝐽(𝒙0 ) = [ ] ⟹ 𝐽(𝒙0 )−1 = 6 [ ]
23 1 2 1 4
22
3
Hence: ∆𝑥0 = 𝐽(𝒙0 )−1 (−𝑓(𝒙0 )) = [53]
6

28
3 9.33333
⟹ 𝒙1 = 𝒙0 + ∆𝒙0 = [59 ]=[ ]
8.83333
6

9.33333
Again consider 𝒙1 = [ ] as initial value and hence calculate the next Approximation.
8.83333
9.33333 115.1389 −115.1389
Here 𝒇(𝒙1 ) = 𝒇 [ ]=[ ] ⟹ −𝒇(𝒙1 ) = [ ]
8.83333 57.44444 −57.44444
18.66667 17.66667 0.513761 −0.97248
and 𝐽(𝒙1 ) = [ ] ⟹ 𝐽(𝒙1 )−1 = [ ]
8.833333 9.333333 −0.48624 1.027523
−3.29052
∆𝒙1 = 𝐽(𝒙1 )−1 (−𝑓(𝑥1 )) = [ ]
−3.04052
6.042813
Thus: 𝒙2 = 𝒙1 + ∆𝒙1 = [ ]
5.792813

28 | P a g e
 6.042813
Again consider 𝒙2 = [ ] as initial value and hence calculate the next approximation. Then
5.792813
6.042813 20.072289 −20.0723
𝒇(𝒙2 ) = 𝒇 [ ]=[ ] ⟹ −𝒇(𝒙2 ) = [ ]
5.792813 10.00489 −10.0049
12.08563 11.58563 1.021123 −1.95775
And 𝐽(𝒙2 ) = [ ] ⟹ 𝐽(𝒙2 )−1 = [ ] and
5.792813 6.042813 −0.97888 2.042245
−0.90914
∆𝒙𝟐 = 𝐽(𝒙2 )−1 (−𝑓(𝒙2 )) = [ ]
−0.78414
Hence
5.133673
𝒙3 = 𝒙2 + ∆𝒙2 = [ ]
5.008673
5.133673
Again consider 𝒙3 = [ ] as initial value and hence calculate the next approximation. Then
5.008673
5.133673 1.441411 −1.441411
𝒇(𝒙3 ) = 𝒇 [ ]=[ ] ⟹ −𝒇(𝒙3 ) = [ ]
5.008673 0.712893 −0.712893
10.26735 10.01735 2.024649 −3.9507
And 𝐽(𝒙3 ) = [ ] ⟹ 𝐽(𝒙3 )−1 = [ ] and
5.008673 5.133673 −1.97535 4.049298
−0.10192
∆𝒙𝟑 = 𝐽(𝒙3 )−1 (−𝑓(𝒙3 )) = [ ]
−0.03942
5.031749
Hence: 𝒙4 = 𝒙3 + ∆𝒙3 = [ ]
4.969249
Note: We precede this process until we get the idea ‖∆𝑥𝑛 ‖ ≤ 𝜀 or ‖𝑥𝑛 − 𝑥𝑛−1 ‖ ≤ 𝜀. Where 𝜀 is
called pre specified error (or tolerance) and is usually given.
>> F=@(x)[x(1).^3+3*x(2).^2-21;x(1).^2+2*x(2)+2];J=@(x)[3*x(1).^2 6*x(2);2*x(1) 2]; x=[1;-
1];eps=10^(-6);
>> [x, iteration_counter] = Newton_system(F, J, x,eps)

29 | P a g e