COMPUTATIONAL METHOD ECEG - 3042 2022

Chapter Three
Systems of linear algebraic equations and the solutions
In more complicated modeling situations it may be that there are many linear relationships,
each involving a large number of variables. For example, in the stress analysis of a structure
such as a bridge or an aeroplane, many hundreds (or indeed thousands) of linear relationships
may be involved. Typically, we are interested in determining values for the dependent variables
using these relationships.

In this chapter various methods are compared for solving linear systems of equations. Both
direct and indirect methods are considered. Direct and Indirect methods are widely used for
solving large sparse and dense problems. In direct methods, two methods will be considered:
Gaussian Elimination and

LU-Factorization methods. Jacobi and Gauss-Seidel Methods will be discussed as an iterative

(indirect) method. The results show that the iterative method is more efficient than the direct
methods. The criteria considered are time to converge, number of iterations, memory
requirements and accuracy.

Keywords: Linear system equation, direct method, indirect method.

When the number of equations (and consequently the number of variables in those equations)
becomes large it can be tedious to write the system out in full. In any case, we need some
notation which will allow us to specify an arbitrary linear system in compact form so that we
may conveniently examine ways of solving such a system. Hence we write linear systems using
matrix and vector notation as Ax = b; Where A is a matrix, known as the coefficient matrix (or
matrix of coefficients), b is the vector of right-hand sides and x is the vector of unknowns to
be determined.

A linear equation in n variables: a1x1 + a2x2 + … + anxn = b

Solution
Solution of the linear system (3) is a sequence of n numbers s1,s2,s3,...,sn , which satisfies
system (3) when we substitute x1 =s1,x2 =s2,x3 =s3,..., xn =sn .

Example.1. Solve the system of equations

x - 3y =-3 →1
2x + y = 8 →2

AMIT FECE Page 1

 COMPUTATIONAL METHOD ECEG - 3042 2022

Solution: -2R1 + R2→

-2x + 6y = 6

2x + y =8

+7y= 14⟹y = 2
From equation 1 x = -3 +3=x = -3 + 6 = 3
Solution is x = 3 and y = 2

Example.2. Solve the system of equations

x - 3y = -7 →1
2x - 6y = 7 →2
Solution: 2R1 - R2=

2x - 6y = -7
- 2x - 6y = -14
0 + 0 = -21

This makes no sense as 0 = -21, hence there is no solution.

NOTE:

Inconsistent, the system of equations is inconsistent, if the system has no solution.

Consistent, the system of equations is consistent if the system has at least one solution

Example: Inconsistent and consistent system of equations

For the system of linear equations which is represented by straight lines

a1x - b1y =c1 →l1

a2x - b2y =c2 →l2

There are three possibilities:

No solution one solution infinite many solutions

[Inconsistent] [Consistent] [Consistent]

AMIT FECE Page 2

 COMPUTATIONAL METHOD ECEG - 3042 2022

Note: 1. A system will have unique solution (only one solution) when number of unknowns is
equal to number of equations
Note:2. A system is over determined; if there are more equations then unknowns and it will be
mostly inconsistent.
Note:3. A system is under determined if there are less equations then unknowns and it may
turn inconsistent.

Augmented Matrix
Linear Algebraic Equations;

a11x1+ a12x2+ a13x3+ … + a1nxn= b1

a21x1+ a22x2+ a23x3+ … + a2nxn= b2

an1x1+ an2x2+ an3x3+ … + anxn= bn

Where all aij's and bi'sare constants.

Collections of linear equations are called linear systems of equations. They involve same set
of variables. Various methods have been introduced to solve systems of linear equations. There
is no single method that is best for all situations. These methods should be determined
according to speed and accuracy. Speed is an important factor in solving large systems of
equations because the volume of computations involved is huge. Another issue in the accuracy
problem for the solutions rounding off errors involved in executing these computations.

The methods for linear systems of equations can be divided into

1. Direct Methods
2. Iterative Methods

3.1 Direct Methods

Direct methods are not appropriate for solving large number of equations in a system,
particularly when the coefficient matrix is sparse, i.e. when most of the elements in a matrix
are zero.

For small (n ≤ 3), linear algebra provides several tools to solve such systems of linear equations:
 Direct substitution
 Graphical method
 Cramer’s rule
 Method of elimination

AMIT FECE Page 3

 COMPUTATIONAL METHOD ECEG - 3042 2022

A. Determinants and Cramer’s Rule

X1 + X3 =4 4X +4Y =2
X1 + X2 + X3 =0
B. Gauss -Elimination Method

A linear system may be reduced to upper triangular form by means of successively subtracting
multiples of the first equation from the second, third, fourth and so on, then repeating the process
using multiples of the newly-formed second equation, and again with the newly-formed third
equation, and so on until the penultimate equation has been used in this way. Having reduced
the system to upper triangular form backward substitution is used to find the solution. This
systematic approach is known as Gaussian elimination. It has two parts:

A. Forward Stage
B. Backward Stage

Forward Stage: First part is linked with the manipulation of equations in order to eliminate
some unknowns from the equations and constitute an upper triangular system or echelon
form. It reduces Ax=b to an upper triangular system Tx=b’

AMIT FECE Page 4

COMPUTATIONAL METHOD ECEG - 3042 2022

a 11
a 12
a 13
b 1

a 21
a 22
a 23
b 2

a 31
a 32
a 33
b 3
Forward
⇓ Elimination
a 11
a 12
a 13
b 1
' ' '
0 a 22
a 23
b 2
'' ''
0 0 a 33
b 3

𝑎𝑖1 ∗𝑎1𝑗 𝑎𝑖1

Where a’ij =𝑎𝑖𝑗 − and 𝑏1′ = 𝑏𝑖 − 𝑏 Thus, we eliminated the first unknown X1
𝑎11 𝑎11 1
from the second and third equations. Repeat the procedure selecting the second row as pivot,
Backward Stage: This stage uses the back substitution process on the reduced upper
triangular system and results with the actual solution of the equation.

𝑏3′′ 𝑏2′ −𝑎23

′ 𝑥
3
𝑥3 = ′′ 𝑥2 = ′ Back
𝑎33 𝑎22
Substitution
𝑏1 −𝑎13 𝑥3 −𝑎12 𝑥2
𝑥2 =
𝑎11
Example: solve the following linear algebraic equation

2X1 + 3X2 +4X3 − 2X4 = 1

X1 − 2X2 +4X3 − 3X4 = 2
4X1 + 3X2 − X3 + X4 = 2
3X1 − 4X2 +2X3 − 2X4 = 5
As a first step towards an upper triangular system we eliminate X1 from (R2), (R3), and (R4).
To do this we subtract 1/2 times (R1) from (R2), 2 times (R1) from (R3), and 3/2 times (R1)
from (R4). In each case the fraction involved in the multiplication (the multiplier) is the ratio
of two coefficients of the variable being eliminated from (R2) up to (R4).

2X1 + 3X2 +4X3 − 2X4 = 1

-3X2 − 9X3 + 5X4 =0

AMIT FECE Page 5

COMPUTATIONAL METHOD ECEG - 3042 2022

To proceed we ignore (R1) and repeat the process on (R2) to (R4). We eliminate X2 from
(R3) and (R4) by subtracting suitable multiples of (R2). We take (−3)/ (− ) times (R2) from
(R3) and (−17)/(−7) times (R2) from (R4)to give (after the removal of common
denominators) the system
2X1 + 3X2 +4X3 − 2X4 = 1

Finally we subtract 62/75 times (R3) from (R4) to give the system
2X1 + 3X2 +4X3 − 2X4 = 1

We now have an upper triangular system which may be solved by backward substitution.
From (R4) we have X4 = 3.Substituting this value in (R3) gives X3 = 2. Substituting the
known values for X4 and X3 in (R2) gives X2 =−1. Finally using (R1) we find X1 = 1.

Drawback of Elimination Methods

Division by zero: It is possible that during both elimination and back-substitution phases a
division by zero can occur. It occur in A zero on diagonal term

For example:
0X1+2X2 + 3X3 = 8 0 2 3
4X1 + 6X2 + 7X3 = -3 A= 4 6 7
2X1 + X2 + 6X3 = 5 2 1 6
 Round-off errors: Because computers carry only a limited number of significant figures,
round off errors will occur and they will propagate from one iteration to the next. This
problem is especially important when large numbers of equations (100 or more) are to be
solved. Since round off errors can induce small changes in the coefficients, these changes
can lead to large solution errors in ill-conditioned systems.
 System may be ill-conditioning, i.e. det[A] 0
 No solution or an infinite # of solutions, i.e. det[A] = 0

Techniques for Improving Solutions by Elimination Methods

AMIT FECE Page 6
COMPUTATIONAL METHOD ECEG - 3042 2022

 Use of more significant figures

 Pivoting - If a pivot element is zero, normalization step leads to division by zero. The
same problem may arise, when the pivot element is close to zero. Problem can be
avoided:
 Scaling - used to reduce the round-off errors and improve accuracy

C. LU Decomposition Methods

LU-Factorization is based on the fact that it is a matrix decomposition and non-singular square
matrix ’A’ can be replaced as the product of lower triangular and upper triangular matrices.
That is why this method is known as LU-Factorization method. This method is also known as
LU-Decomposition method. LU-Factorization is actually variant of Gaussian Elimination
method. Consider the following linear systems of equations.

a11x1+ a12x2+ a13x3+ … + a1nxn= b1

a21x1+ a22x2+ a23x3+ … + a2nxn= b2

an1x1+ an2x2+ an3x3+ … + anxn= bn

Which can be written as follows AX = b → 1
Then A takes the form, A = LU
Where L is lower triangular matrix and U is upper triangular matrix. So equation 1 becomes

LUX= b

Suppose we apply Gaussian elimination to an n x n matrix A without interchanging any rows.

For example

a11 a12 a13 a11 a12 a13 a11 a12 a13

a21 a22 a23 0 a22(1) a23(1) 0 a22(1) a23(1)
a31 a32 a33 0 a32(1) a33(1) 0 0 a33(2)

Lij

L L L
Example Consider the following matrix

AMIT FECE Page 7

 COMPUTATIONAL METHOD ECEG - 3042 2022

1 2 3 1 2 3 1 2 3
A= 2 -3 2 -7 -4 -7 -4
3 1 -1 -5 -10 -50/7

L21=2 L31=3 L32=5/7

And thus 1 0 0 1 2 3

L= 2 1 0 and U= 0 -7 -4
3 5/7 1 0 0 -50/7
If we can compute the factorization A = LU, then

Ax = b , LUx = b , Ly = b, where Ux = y.

Therefore, the following steps can be used to solve Ax = b:

• Compute the factorization A = LU.

• Solve Ly = b using forward substitution.

• Solve Ux = y using backward substitution.

Example 1 Find the LU decomposition of the matrix

25 5 1
[𝐴]=[ 64 8 1]
144 12 1
1 0 0 u11 u12 u13
Solution: [𝐴] = [𝐿][𝑈] = [ℓ21 1 0 ] [ 0 u22 u23 ]
ℓ31 ℓ32 1 0 0 u33
The [𝑈] matrix is the same as found at the end of the forward elimination of Naïve Gauss
elimination method, that is
25 5 1
[𝑈] = 0 -4.8 -1.56
0 0 0.7

AMIT FECE Page 8

COMPUTATIONAL METHOD ECEG - 3042 2022

To find 21and 31, find the multiplier that was used to make the a21 and a31 elements zero in
the first step of forward elimination of the Naïve Gauss elimination method. It was

64
ℓ21 = = 2.56
25
144
ℓ31 = = 5.76
25
To find32 , what multiplier was used to make a32 element zero? Remember a32 element was
made zero in the second step of forward elimination. The [𝐴]matrix at the beginning of the
second step of forward elimination was

25 5 1

[0 − 4.8 − 1.56 ]
0 − 16.8 − 4.76

−16.8
So 32 = −4.8 =3.5

1 0 0

Hence [L]= [2.56 1 0]

5.76 3.5 1

Example 1.3 Consider the following system of equations

The triangular matrices L and U determined using the relations (1.30) are equal to

The determinant of the matrix of coefficients satisfies the equation detA = detL· det U =
84 · 1 = 84. The solution obtained by using the LU decomposition method is x1 = 1.0, x2 =
0.7, x3 = 0.4, and x4 = 0.1.

AMIT FECE Page 9

COMPUTATIONAL METHOD ECEG - 3042 2022

D. Gauss – Jordan Elimination Method

𝑎11 𝑎12 𝑎13 𝑏1 1 0 0 𝐵1

[𝑎21 𝑎22 𝑎23 𝑏2 ] ⟶ [0 1 0 𝐵2 ]
𝑎31 𝑎32 𝑎33 𝑏3 0 0 1 B3
Example. Solve the system of linear equations by Gauss - Jorden elimination method
x1 + x2 + 2x3 = 8
- x1- 2x2 + 3x3 = 1
3x1 - 7 x2 + 4x3 = 10
Solution: Augmented matrix is
1 1 2 8
[ −1 − 2 3 1 ]
3 −7 4 10
1 1 2 8
[ 0 −1 5 9 ] R1+R2,
0 − 10 − 2 − 14 -3R1+R3

1 1 2 8
[0 1 − 5 − 9 ]10R2+R3
0 0 − 52 − 104

1 1 2 8
[0 −1 5 9] -R2,
0 0 1 2 -R3/52

1 1 0 4 -2R3+R1,
[0 1 0 1] 5R3+R2
0 0 1 2
1 0 0 3 -R2+R1
[0 1 0 1]
0 0 1 2
Equivalent system of equations form is: x1 = 3, x2 = 1, x3 = 2 is the solution of the system.

AMIT FECE Page 10

COMPUTATIONAL METHOD ECEG - 3042 2022

E. The Method of Inverse Matrix

The method of inverse matrix also finds an application for the task of repeatedly solving the
system of linear equations

A·X=B (1)

For which the matrix of coefficients A remains unchanged. In other words, the equation system
is being solved for different values of the free terms forming the vector B.

As we know from the extensive literature on the subject, application of the above method is
legitimate, if the number of solution processes applied to the equation system (1) is greater than
2n, where n is the rank of the matrix A, equal to the number of equations in the system. The
inverse A−1 of a nonsingular square matrix A (having the determinant D different from zero) is
also the nonsingular square matrix of the same rank. Product of these matrices, i.e.

A−1 · A = A · A−1 = E (2)

is equal to the unitary matrix E, having also the same rank. The equation system

(1) Will remain unchanged after multiplication of both sides by an inverse matrix, i.e.,

A−1 · A · X = A−1 · B (3)

Substituting relation (2) in the expression (3), we obtain

E · X = A−1 · B (4)
The product of the unitary matrix E of the rank n by a column vector X with n elements is
identical to the vector X. Due to this property, Eq. (4) can be written as

X = A−1 · B (5)

Expressing the essence of the method under consideration. It follows from the above equation
that the solution vector X can be found by simple multiplication of the inverse matrix A−1 by
the vector of free terms B. Determination of the inverse matrix A−1 therefore constitutes an
essential and most difficult problem, which must be solved in the first stage. Different
algorithms available in the literature on linear algebra can be used for this purpose. In case of
the matrix of a small rank (n ≤ 3), the relations given in Appendix B may prove to be useful.
One of the most popular algorithms used for calculating the inverse matrix is presented below.
Assume that a square nonsingular matrix A is given. Denote the elements of this matrix by aij,
where 1 ≤ i ≤ n,1 ≤ j ≤ n. Elements (terms) of the inverse matrix

AMIT FECE Page 11

COMPUTATIONAL METHOD ECEG - 3042 2022

A−1 are denoted by xij, where 1 ≤ i ≤ n,1 ≤ j ≤ n. Product of this two matrices, i.e.,

A · A−1 = E (6)

Can be presented in the following equivalent form:

ij (7)

Where δij is the Kronecker symbol taking the value 1 for i = j and the value 0 for i . It
follows from Eq. (1.37) that, if we want to determine elements of the column j of the matrix
A−1, the following system of equations should be solved:

a11x1j + a12x2j + ... + a1nxnj = 0

a21x1j + a22x2j + ... + a2nxnj = 0
.................................................... (8)
aj1x1j + aj2x2j + ... + ajnxnj = 1
....................................................
an1x1j + an2x2j + ... + annxnj = 0
In order to find all elements of the matrix A−1, the equation system (8) should be solved n
times, namely for j = 1,2,3, ...,n. The matrix of coefficients A of this system remains
unchanged, and therefore, it can be effectively solved by using the LU decomposition method
described in the previous section. The product (2) can be used to evaluate precision obtained
for the calculated inverse matrix A−1. In case this precision is not satisfactory, the main
equation system can be solved once more, this time by using the relation (5).

Example 1.4 Solve the following system of equations using the method of inverse matrix

The inverse A−1 of the coefficients matrix A of the system given below is equal to (see
Appendix B)
A

According to the relation (1.35), we have

Finally, we find our solution: x1 = 7, x2 = 5, and x3 = 5.

AMIT FECE Page 12

COMPUTATIONAL METHOD ECEG - 3042 2022

3.2. Iterative Methods

Iterative methods for solving general, large sparse linear systems have been gaining popularity
in many areas of scientific computing. A number of efficient iterative solvers were discovered
and the increased need for solving very large linear systems triggered a noticeable and rapid
shift toward iterative techniques in many applications. Also, iterative methods are gaining
ground because they are easier to implement efficiently on high-performance computers than
direct methods.

Those methods are called direct that terminate after finitely many operations with an exact
solution (up to rounding errors).Because of round-off errors, direct methods become less
efficient than iterative methods when they applied to large systems. In addition, the amount of
storage space required for iterative solutions on a computer is far less than the one required for
direct methods when the coefficient matrix of the system is sparse. Thus, especially for sparse
matrices, iterative methods are more attractive than direct methods.

Advantages Iterative Schemes:

1. May be more rapid if coefficient matrix is "sparse"

2. May be more economical w/ respect to memory
3. May also be applied to solve nonlinear systems

Disadvantages Iterative Schemes::

1. May not converge or may converge slowly
2. Not appropriate for all systems

A. Jacobi method
Jacobi calculates all new values of Xi’s to calculate a set of new xi values If any of the aii = 0
and the matrix A is nonsingular, then the equations can be reordered so that all aii = 0.
Convergence (if possible) is accelerated by taking the aii as large as possible. Because necessary
condition for convergence is that the set be diagonal dominant. Starting with

AX=b
a11x1 + a12x2 + a13x3+ ...+ a1nxn = b1
a21x1+ a22x2+a23x3+ ... + a2nxn = b2
:
an1x1+ an2x2 + an3x3 + ... + annxn = bn

AMIT FECE Page 13

 COMPUTATIONAL METHOD ECEG - 3042 2022

Solve each equation for one variable:

x1[k+1] = [b1 – (a12x2[k]+a13x3[k] + ... + a1n xn[k])]/a11

x2[k+1] = [b2 – (a21x1[k]+a23x3[k] + ... + a2n xn[k])]/a22
a32x2[k]+ ... + a3nxn[k])]/a33 :

xn[k+1] = [bn – (an1x1[k]+an2x2[k+1] + ... + an,n-1xn-1[k]]/aii

𝑏 𝑎𝑖𝑗 𝑥𝑗
Final form 𝑥𝑖 = 𝑎 𝑖 − ∑𝑛𝑗=1 for i = 1, 2, …, n provided aii≠ 0 0.
𝑖𝑖 𝑎𝑖𝑖
𝑎𝑖𝑗 𝑥𝑗
We generate x(k-1) from x(k) for k≥0 by; xi(k-1== ∑𝑛𝑗=1(− + 𝑏𝑖 ) 𝑖 = 1,2, … , 𝑛
𝑎𝑖𝑖
𝑗≠𝑖
Theorem: If the matrix A is strictly diagonally dominant, that is, if |𝑎𝑖𝑖 |> ∑𝑛𝑗=1 𝑎𝑖𝑗 i = 1, 2,n.
𝑗≠𝑖
Example: Solve by point Jacobi method:
5x +y =10
2x + 3y = 4

⇒X [j+1] =[10-1*y]/5 Y[j+1] = [4-2x]/3

Start with solution guess X [0] = -1, Y [0]= -1 and start iterating on the solution

This is a converging process keep on going until the desired level of accuracy is achieved X [k]
= 2.00000, Y [k] = 0.00000

B. Gauss–Seidel Method

The Gauss-Seidel Method allows the user to control round-off error. Elimination methods such
as Gaussian elimination and LU decomposition are prone to prone to round-off error. Also: If
the physics of the problem are understood, a close initial guess can be made, decreasing the
number of iterations needed. This method is very similar to the Jacobi method except that

AMIT FECE Page 14

COMPUTATIONAL METHOD ECEG - 3042 2022

Gauss Seidel uses the most recently computed values for in its computations. Using all
updated values of increases the convergence rate (twice as fast as Jacobi)
Starting with AX=b

a11x1 + a12x2 + a13x3+ ...+ a1nxn = b1

a21x1+ a22x2+a23x3+ ... + a2nxn = b2
:
an1x1+ an2x2 + an3x3 + ... + annxn = bn
Solve each equation for one variable:
Solve each equation for one variable:
x1[k+1]= [b1 –(a12x2[k] + a13x3[k] +...a1n xn[k])]/a11
x2[k+1]= [b2 –(a21x1[k+1] + a23x3[k] + +...a2n xn[k])]/a22
x3[k+1] = [b3– (a31x1[k+1] + a32x2[k+1] + ...a3nxn[k])]/a33 :
xn[k+1]= [bn –(an1x1[k+1]+ an2x2[k+1] +...+an,n-1xn-1[k])]/ann
1
Generally; 𝑥𝑖𝑘+𝑖 = − 𝑎 [∑𝑖−1 𝑘+1
𝑗=1 𝑎𝑖𝑗 𝑥𝑗 − ∑𝑥𝑗=𝑖+1 𝑎𝑖𝑗 𝑥𝑗𝑘 − 𝑏𝑖 ]
𝑖𝑖

 Start with an initial estimate of {x}0,

 substitute into the right-hand side of all of the above equations

 generate a new approximation {x}1
 Repeat until maximum number of iterations is reached or: ‖x𝑗+𝑖 − x𝑗 ‖ ≤ 𝛿 + 𝜀‖x𝑗+𝑖 ‖
 Calculate the Absolute Relative Approximate Error
x inew - x iold x100
∈𝑎 =
i x inew

Iterative convergence

 Is the (k+1)th solution better than the kth solution? Iterative process can be
convergent/divergent
 A necessary condition for convergence is that the set be diagonal.
o This requires that one of the coefficients in each of the equations be greater
than all others and that this “strong coefficient” be contained in a different
position in each equation.
o We can re-arrange all strong elements onto diagonal positions by switching
column

AMIT FECE Page 15

 COMPUTATIONAL METHOD ECEG - 3042 2022

 A sufficient condition to ensure convergence is that the matrix is diagonally dominant

| aii| >∑ 𝑗=1 |𝑎𝑖𝑗| i=1, N
𝑖≠𝑗

 A poor first guess will prolong the iterative process but will not make it diverge if the
matrix is such that convergence is assured.
o Therefore better guesses will speed up the iterative process
 Criteria for ascertaining convergence:
x j - x j-1
∈a , i = i i x100<𝜀 s
xj
i

o Relative convergence criteria ⇒Where 𝜀=a user specified tolerance or accuracy

Example: Solve by point Gauss-Seidel Method rounded to four decimal places.

10x1 - x2 + 2x3 = 6
– x1 + 11x2 – x3 + 3x4 = 25
2x1 - x2 + 10x3 – x4 = –11
3x2 - x3 + 8x4 = 15
The equations can be written as follows:

AMIT FECE Page 16

COMPUTATIONAL METHOD ECEG - 3042 2022

= 0.8789

Using x(1) we get; x(2) =(1.0300, 2.037, – 1.014, 0.9844) T

And we can check that; x(5) =(1.0001, 2.0000, – 1.0000, 1.0000) T

AMIT FECE Page 17