University of Tripoli

Engineering Faculty
Computer Engineering Department

Lecture notes of

Numerical analysis
EC372

Dr. Amna Elhawil

Spring 2013

Contents
1

Introduction ............................................................................................................. 3
1.1

1.1.1

Identifying Significant Digits ..................................................................... 3

1.1.2

Numerical Errors ........................................................................................ 3

1.1.3

Uncertainty: well- or ill-conditioned? ......................................................... 4

1.1.4

Why Study Numerical Methods.................................................................. 5

1.2
2

Accuracy and precision ..................................................................................... 3

Taylor Series ..................................................................................................... 5

Roots of equations ................................................................................................... 9

2.1

Bisection Method .............................................................................................. 9

2.2

False-Position Method ..................................................................................... 10

2.3

Fixed-point Iteration ........................................................................................ 13

2.4

Newton-Raphson Method ................................................................................ 15

2.4.1

Drawbacks of the Newton-Raphson Method............................................. 18

2.4.2

Additional features for Newton-Raphson.................................................. 19

2.5

Secant Method ................................................................................................ 20

2.5.1
2.6
3

Difference between the secant and false-position method ......................... 21

Rate of convergence ........................................................................................ 23

System of Linear equations .................................................................................... 25

3.1

Guess Elimination ........................................................................................... 25

3.1.1

How is a set of equations solved numerically? .......................................... 25

3.1.2

Forward Elimination of Unknowns:.......................................................... 25

3.1.3

Back Substitution: .................................................................................... 27

3.1.4

Pitfalls of the Nave Gauss elimination method ........................................ 32

3.1.5

Techniques for improving the Nave Gauss elimination method ............... 37

3.1.6

Gaussian elimination with partial pivoting ................................................ 37

3.2

LU-Decomposition .......................................................................................... 41

3.3

Inverse of matrix ............................................................................................. 46

3.4

Gauss-Seidel Method ...................................................................................... 50

1 Introduction
1.1 Accuracy and precision
The errors associated with both calculations and measurements can be characterized with
regard to their accuracy and precision. Accuracy refers to how closely a computed or
measured value agrees with the true value. Precision refers to how closely individual
computed or measured values agree with each other.
Significant Figures
Number of significant figures indicates precision. Significant digits of a number are
those that can be used with confidence, e.g., the number of certain digits plus one
estimated digit.
53,800
5.38 x 104
5.3800 x 104

How many significant figures?

3
5

Zeros are sometimes used to locate the decimal point not significant figures.
0.00001753
4
0.001753
4
1.1.1 Identifying Significant Digits
All non-zero digits are considered significant:
For example, 91 has two significant figures, while 123.45 has five significant
figures.

Zeros appearing anywhere between two non-zero digits are significant:

For example: 101.1002 has seven significant figures.

Leading zeros are not significant:

For example: 0.00052 has two significant figures.

Trailing zeros in a number containing a decimal point are significant.

For example: 12.2300 has six significant figures: 1, 2, 2, 3, 0 and 0. The number
0.000122300 still has only six significant figures (the zeros before the 1 are not
significant). In addition, 120.00 has five significant figures.

The significance of trailing zeros in a number not containing a decimal point

can be ambiguous. For example, it may not always be clear if a number like
1300 is accurate to the nearest unit. Various conventions exist to address this
issue.

1.1.2 Numerical Errors

Roundoff
Roundoff occurs when digits in a decimal point (0.3333...) are lost (0.3333) due to a limit
on the memory available for storing one numerical value.
Truncation
Truncation error occurs when discrete values are used to approximate a mathematical
expression.
For both types, the relationship between the exact, or true, result and the approximation
can be formulated as
True value = approximation + error
From this equation we find that the numerical error is equal to
Et = true value approximation
where Et is used to designate the exact value of the error. The subscript t is included to
designate that this is the true error. The relative error is expressed as

True percent relative error: t

True value Approximation

100%
True value

where t designates the true percent relative error.

Approximate Error
For numerical methods, the true value will be known only when we deal with functions
that can be solved analytically.
Approximate Error = Current Approximation(i) Previous Approximation(i-1)

Approximate Relative Error :

Approximate error
100%
Approximation

1.1.3 Uncertainty: well- or ill-conditioned?

Errors in input data can cause uncertain results input data can be experimental or rounded
leads to a certain variation in the results well-conditioned: numerical results are
insensitive to small variations in the input ill-conditioned: small variations lead to
drastically different numerical calculations (poorly conditioned)

1.1.4 Why Study Numerical Methods

1. No numerical method is completely trouble free in all situations!
How should I choose/use an algorithm to get trouble free and accurate
answers?
2. No numerical method is error free!
What level of error/accuracy do I have the way Im solving the problem?

Identify error! (e.g. movement of a building)

3. No numerical method is optimal for all types/forms of an equation!
Efficiency varies by orders of magnitude!!!

One algorithm for a specific problem seconds to solve on a

computer

Another algorithm for the same problem decades to solve on

the same computer

In order to solve a physical problem numerically, you must understand the behavior of
the numerical methods used as well as the physics of the problem.

Our Job
As numerical analysts, we need to 1 solve a problem so that the calculation is not
susceptible to large round-off error 2 solve a problem so that the approximation has a
tolerable truncation error How? incorporate roundoff-truncation knowledge into I the
mathematical model I the method the algorithm the software design awareness ! correct
interpretation of results.

1.2 Taylor Series

How does a CPU compute the following functions for a specific x value?
cos(x)

sin(x)

ex

log(x)

etc

Taylor series provides a means to predict the value of a function at one point in terms of
the function value and its derivatives at another point.

f ( xi 1 ) f ( xi )

f ' ( xi )
f " ( xi )
f ( n ) ( xi )
( xi 1 xi )
( xi 1 xi ) 2
( xi 1 xi ) n Rn
1!
2!
n!

Define the step size as h=(xi+1- xi), the series becomes:

f ( xi 1 ) f ( xi )

f ' ( xi )
f " ( xi ) 2
f ( n ) ( xi ) n
h
h
h Rn
1!
2!
n!

Any smooth function can be approximated as a polynomial.

 zero order approximation:

f(x) f(xi)
first order approximation
f ( x) f ( xi ) f ' ( xi )( x xi )

Second order approximation:

f ( x) f ( xi )

f ' ( xi )
f " ( xi )
( x xi )
( x xi ) 2
1!
2!

nth order approximation

f ( x) f ( xi )

f ' ( xi )
f " ( xi )
f ( n ) ( xi )
( x xi )
( x xi ) 2
( x xi ) n Rn
1!
2!
n!

Example (1.1)
Approximate the function f(x) = sin (x) from x0 = 0.2 then find f(0.6).
Solution
f(x) = sin(x) - f(0.2) = 0.1987
f(x) = cos(x) f(0.2)= 0.9801
f(x) = -sin(x) f(0.2) =- 0.1987
f(x)= - cos(x) f(0.2)=- 0.9801
using zero order approximation:
T0 = f(x) f(xi)=f(0.2)= 0.1987
using first order approximation
f ( x) f ( xi ) f ' ( xi )( x xi )

T1 =f(x) = 0.1987+ 0.9801 (x 0.2)

x = 0.6 T1 = 0.5907
using second order approximation

f ' ( xi )
f " ( xi )
f ( x) f ( xi )
( x xi )
( x xi ) 2
1!
2!
T2 =f(x) =0.1987 0.9801( 0.2)

( 0.2)

( 0.2)

At x =0.6 T2 = 0.5748
using third order approximation
T3 =f(x) =0.1987 0.9801( 0.2)

( 0.2)

At x =0.6 T3 = 0.5643
The exact value sin(0.6) = 0.5646. Table 1.1 summarizes the results.
Table 1.1 Taylor series
x0
0
0.1
0.2
0.4
0.5
1

h =x-x0
0.6
0.5
0.3
0.2
0.1
-0.4

T0
0
0.0998
0.1987
0.3894
0.4794
0.8415

T1
0.6000
0.5973
0.5907
0.5736
0.5672
0.6254

T2
0.6000
0.5849
0.5748
0.5658
0.5648
0.5580

T3
0.5640
0.5641
0.5643
0.5646
0.5646
0.5638

T4
0.5640
0.5644
0.5646
0.5646
0.5646
0.5647

Et=|T4 sin(0.6)|
6.4247e-004
2.5542e-004
8.2187e-005
2.4192e-006
7.2449e-008
5.0704e-005

Note that
Be aware that many functions still cannot be expressed as power series using this
formula. For example, the function f(x) = 1/x has no Taylor series, since f(0) is
undefined. In general, any function for which is undefined for some will f n(0) to
be analytic.

Example (1.2)
Approximate the function f(x) = 1.2 - 0.25x - 0.5x2 - 0.15x3 - 0.1x4 from xi = 0 with h
= 1 and predict f(x) at xi+1 = 1.
Solution
f(x) = 1.2 - 0.25x - 0.5x2 - 0.15x3 - 0.1x4
f(x) = -0.25 - x - 0.45 x2 - 0.4 x3
f(x) = -1 - 0.9x - 1.2x2
at x = 0

f (0) = 1.2, f(0) =-0.25 and f(0)= -1

using zero order approximation:
f(x) f(xi)=f(0)=1.2
using first order approximation
f ( x) f ( xi ) f ' ( xi )( x xi ) =1.2+f(1.2) *h= 1.2 + f(0) x h=1.2 0.25 *1=0.95

using second order approximation

f ( x) f ( xi )

f ' ( xi )
f " ( xi )
( x xi )
( x xi ) 2 =1.2 +
1!
2!

Fig. 1.1

(1) +

(1) = 0.45

2 Roots of equations
2.1 Bisection Method
The steps of Bisection method to find the root of the equation f ( x) 0 are
1. Choose x and xu as two guesses for the root such that f ( x ) f ( xu ) 0 , or in
other words, f (x ) changes sign between x and xu .
2. Estimate the root, xm , of the equation f ( x) 0 as the mid-point between x and

xu as
xm =

x xu
2

3. Now check the following conditions:

a) If f ( x ) f ( xm ) 0 , then the root lies between x and xm ; then x x and

xu xm .
b) If f ( x ) f ( xm ) 0 , then the root lies between xm and xu ; then x xm and

xu xu .
c) If f ( x ) f ( xm ) 0 ; then the root is xm . Stop the algorithm if this is true.
4. Find the new estimate of the root
x xu
xm =
2
Find the absolute relative approximate error as

a =

x mnew - x mold
100
x mnew

where
xmnew = estimated root from present iteration
xmold = estimated root from previous iteration

5. Compare the absolute relative approximate error a

with the pre-specified

relative error tolerance s . If a s , then go to Step 3, else stop the algorithm.

Note one should also check whether the number of iterations is more than the

maximum number of iterations allowed.

algorithm and notify the user about it.

If so, one needs to terminate the

Example (2.1)
Use Bisection method to find the root of f (x) = 3x + sin(x) - exp(x) = 0.
Solution
The figure of this function is shown in Fig. 2.1. It has two roots, one between [0 0.5]
and the second is between [1.5 , 2].
1.5
1
0.5
0
-0.5

0.5

1.5

Fig. 2.1

Let us choose two guesses x =0 and xu = 0.5

Iteration
no.
1
2
3
4
5
6
7
8
9
51

Table 2.1 Example (3) using Bisection method

x
xu
xm
f( x )f( xu )

0
0.25
0.2500
0.3125
0.3594
0.3594
0.3594
0.3594

0.5
0.5
0.3750
0.3750
0.3750
0.3672
0.3633
0.3613

0.25
0.375
0.3125
0.3438
0.3672
0.3633
0.3613
0.3604

-0.3307
-0.0948
-0.0104
-0.0044
-9.5043e-005
-4.4235e-005
-1.8722e-005
-5.9386e-006

33.3333
20
9.0909
4.3478
2.1277
1.0753
0.5405

0.3604

0.3604

0.3604

-9.8608e-032

6.1607e-14

So one of the roots of 3x + sin(x) - exp(x) = 0 is approximately 0.3605.

2.2 False-Position Method

The poor convergence of the bisection method as well as its poor adaptability to higher
dimensions (i.e., systems of two or more non-linear equations) motivate the use of better
techniques. One such method is the Method of False Position. Here, we start with an
initial interval [x1,x2], and we assume that the function changes sign only once in this

10

interval. Now we find an x3 in this interval, which is given by the intersection of

the x axis and the straight line passing through (x1,f(x1)) and (x2,f(x2)). It is easy to verify
that x3 is given by

Now, we choose the new interval from the two choices [x1,x3] or [x3,x2] depending on in
which interval the function changes sign.
The steps of the False-position method to find the root of an equation f x 0 are
1. Choose xU, xL as two guesses for the root such

f xL f xU 0
2. Estimate the root,
xm

xU f xL xL f xU
f xL f xU

3. Now check the following, if f xL f xm 0 , then the root lies between xL and
xm; then xL = xL and xL = xm
4. If f xL f xm 0 , if f xL f xm 0 , then the root lies between xm and xU; then
xL = xm and xU = xU
5. If f xL f xm 0 , then the root is xm and stop. If this condition is not true, find
the absolute relative approximate error as
x new x old
a m new m 100
xm
6. say s 10 3 0.001. , if a s go to steps 2 to 4, else stop.

Note that:
-

The False-Position and Bisection algorithms are quite similar. The only difference
is the formula used to calculate the new estimate of the root xm.
It converges faster to the root because it is an algorithm which uses appropriate
weighting of the initial end points xL and xU using the information about the
function, or the data of the problem. In other words, finding xm is
a static procedure in the case of the bisection method since for a given xL and xU,
it gives identicalx3, no matter what the function we wish to solve. On the other

11

hand, the false position method uses the information about the function to arrive
at xm.
Example (2.2)
Use False-position method to find the roots of the function f(x) = x3 + 2x2 3x -1 , take
[xL, xU] = [1, 2].
Solution
Function f is continuous on [1, 2] and f(1) = -1, f(2) = 9 so there is a root in [1, 2].
=

=1

) ( )
( ) ( )

(2 1) (1)
= 1.1
9+1

Table 2.2 Example (2.2) using False-position method

x
xu
xm
f( x )f( xu )
a

Iteration
no.
1
2
3
4
5
6
11
12

1
1.10
1.1517
1.1768
1.1886
1.1941
1.1987
1.1987

2
2
2
2
2
2
2
2

1.1
1.1517
1.1768
1.1886
1.1941
1.1966
1.1987
1.1987

-9
-4.9410
-2.4696
-1.1767
-0.5479
-0.2524
-0.0010
1.8040e-004

4.4926
2.1326
0.9916
0.4565
0.2092
8.6600e-004
-2.1859e-004

From Fig. 2.2 we can see that the function has two solutions. One is between [-0.5, 0] and
the second is between [1, 1.5]. The False-position method gives the second solution
which is 1.1987 after 12 iteration.
8
7
6
5
4
3
2
1
0
-1
-2
-1

-0.5

0.5

Fig. 2.2

12

1.5

2.5

2.3 Fixed-point Iteration

In this method the function is rearranged so that x is on the left-hand side of the equation:
f ( x) 0 g ( x) x
xk g ( xk 1 ) xo is given , k 1, 2, ...

Example (2.3)
Find the roots of the equation: f ( x) x 2 2 x 8 0 . Approximate error a not greater
than 10-4.
Solution
This is second order equation. It has two roots -2 and 4. Let us use the Fixed-point
iteration method to find one of these roots.
1. Rewrite the equation as : x g ( x)

x2 8
2

or
x 2x 8

Let us choose the first option.

2. Select initial guess x0 = 2, and calculate x1 using the formula:
xi 1

xi2 8
2

x 02 8 4 8
x1

2
2
2

3. Compute the approximate error:

a
a

xmnew xmold
100
xmnew

2 (2) 100 2
2

4. Calculate the new value of x2

13

x12 8 22 8
x2

-2
2
2
5. Next we calculate the new approximate error
- 2 (2) 100 0
a
-2

6. Since the error is 0, so the root of the equation is -2.

However, if we choose another initial guess, for example x0 = 1. The procedure is
summarized in Table 2.3.
Table 2.3 Fixed-point iteration, example (2.3)
Iteration
xi
xi 1
a
no.
1
2
3
4
5
6
7
8
9

1
-3.5000
2.1250
-1.7422
-2.4824
-0.9189
-3.5778
2.4005

-3.5000
2.1250
-1.7422
-2.4824
-0.9189
-3.5778
2.4005
-1.1189

1.2857
2.6471
2.2197
0.2982
1.7016
0.7432
2.4905
3.1455

It is clear that, the iterations diverges (do not converge) from the root . Note that if we use
the same initial guess (x0 = 1) but with the formula

x 2x 8
The method converges to the root after 10 iterations, as shown in Table 2.4.
Table 2.4 Fixed-point iteration, example (2.3)
Iteration
xi
xi 1
a
no.
1
2
3
4
5
6
7
8
9
10

1
3.1623
3.7848
3.9458
3.9864
3.9966
3.9992
3.9998
3.9999
4.0000

3.1623
3.7848
3.9458
3.9864
3.9966
3.9992
3.9998
3.9999
4.0000
4.0000

14

0.6838
0.1645
0.0408
0.0102
0.0025
6.3629e-004
1.5907e-004
3.9767e-005
3.9767e-005
9.9417e-006

Fixed-point methods may sometimes converge, depending on the starting point

(initial guess) and how the function behaves.

2.4 Newton-Raphson Method

This method is most widely used formula for locating roots. It can be derived using
Taylor series or the geometric interpretation of the slope in the figure

Fig. 2.3

f (xi )

f ( xi ) 0
( xi xi 1 )

rearrange to obtain :

xi 1 xi

f ( xi )
f ( xi )

The steps of the Newton-Raphson method to find the root of an equation f x 0 are
1. Evaluate f x symbolically
2. Use an initial guess of the root, xi , to estimate the new value of the root, xi 1 ,
as
f xi
xi 1 = xi
f xi
3. Find the absolute relative approximate error a as

a =

x i 1 xi
100
xi 1

15

4. Compare the absolute relative approximate error with the pre-specified

relative error tolerance, s . If a > s , then go to Step 2, else stop the
algorithm. Also, check if the number of iterations has exceeded the maximum
number of iterations allowed. If so, one needs to terminate the algorithm and
notify the user.
Example (2.4)
Use Newton-Raphson method to find the root of the function

x 3 0.165 x 2 3.993 10 4 0
Solution

f x x 3 0.165x 2 3.993 104

f x 3x 2 0.33x
Let us assume the initial guess of the root of f x 0 is x0 0.05 m.
Iteration 1 : The estimate of the root is

x1 x0

f x0
f x0

3
2

0.05 0.1650.05 3.993 10 4

0.05
2
30.05 0.330.05

0.05

1.118 10 4
9 10 3

0.05 0.01242

0.06242

The absolute relative approximate error a at the end of Iteration 1 is

x1 x0
100
x1

0.06242 0.05
100
0.06242
19.90%

16

The number of significant digits at least correct is 0, as you need an absolute relative
approximate error of 5% or less for at least one significant digit to be correct in your
result.
Iteration 2: The estimate of the root is
x2 x1

f x1
f x1

0.06242

0.062423 0.1650.062422 3.993 10 4

2
30.06242 0.330.06242

0.06242

3.97781 10 7
8.90973 10 3

0.06242 4.4646 10 5

0.06238

The absolute relative approximate error a at the end of Iteration 2 is

x2 x1
100
x2

0.06238 0.06242
100
0.06238

0.0716%

The maximum value of m for which a 0.5 102 m is 2.844. Hence, the number of
significant digits at least correct in the answer is 2.
Iteration 3: The estimate of the root is
x3 x 2

f x 2
f x2

0.06238

0.062383 0.1650.062382 3.993 10 4

2
30.06238 0.330.06238

0.06238

4.44 10 11
8.91171 10 3

0.06238 4.9822109

0.06238

The absolute relative approximate error a at the end of Iteration 3 is

17

0.06238 0.06238
100
0.06238

The number of significant digits at least correct is 4, as only 4 significant digits are
carried through in all the calculations.
2.4.1 Drawbacks of the Newton-Raphson Method
1. Divergence at inflection points: : If the selection of a guess or an iterated value
turns out to be close to the inflection point of f(x), that is, near where f(x)=0, the
roots start to diverge away from the true root.
2. Division by zero or near zero: The formula of Newton-Raphson method is

xi 1 xi

f(xi )
f (xi )

Consequently if an iteration value, xi is such that f xi 0 , then one can face division
by zero or a near-zero number. This will give a large magnitude for the next value, xi+1.
An example is finding the root of the equation

f x x 3 0.03x 2 2.4 10 6
in which case

f x 3x 2 0.06 x
For x0 0 or x0 0.02 , division by zero occurs. For an initial guess close to 0.02, of

x0 0.01999 , even after 9 iterations (Table 2.5), the Newton-Raphson method is not
converging.
Table 2.5
Iteration
Number
0
1
2
3
4
5
6
7
8
9

xi

0.019990
-2.6480
-1.7620
-1.1714
-0.77765
-0.51518
-0.34025
-0.22369
-0.14608
-0.094490

100.75
50.282
50.422
50.632
50.946
51.413
52.107
53.127
54.602

18

f(xi)
-1.6000 x 10-6
-18.778
-5.5638
-1.6485
-0.48842
-0.14470
-0.042862
-0.012692
-0.0037553
-0.0011091

3. Root jumping: In some case where the function f(x) is oscillating and has a
number of roots, one may choose an initial guess close to a root. However, the
guesses may jump and converge to some other root. For example for solving the
equation
sin x 0

if you choose x0 2.4 7.539822 as an initial guess, it converges to the root

of x=0 as shown in Table 2.6. However, one may have chosen this an initial
guess to converge to x 2 6.2831853 .

Iteration
Number
0
1
2
3
4
5

f(x)

Table 2.6
f(xi)

xi
7.539822
4.461
0.5499
-0.06303
6.376x10-4
-1.95861x10-13

0.951
-0.969
0.5226
-0.06303
8.375x10-4
-1.95861x10-13

68.973
711.44
971.91
7.54x104
4.27x1010

1.5

0.5

0
-2

-0.06307

0.5499

4.461

7.539822

10

-0.5

-1
-1.5

Fig. 2.4 Root jumping from intended location of root for f x sin x 0

2.4.2

Additional features for Newton-Raphson

Plot your function to correctly choose the initial guess.
Substitute your solution into the original function.
The program must always include an upper limit on the number of iterations.
The program should alert the user of the possibility of f ' x 0 .

19

2.5 Secant Method

The secant method is obtained by replacing the derivative in Newtons method, f0(xn), by
the following finite difference approximation:

xi xi 1
f ( xi ) f ( xi 1 )

xi 1 xi f ( xi )

Fig. 2.5

The secant method can also be derived from geometry:

From Fig. 2.6 ABE and DCE are similar triangles.
f (x)

f (xi )

f (xi1)

C
E
xi+1

Fig. 2.6

Hence

20

D
xi1

A
xi

AB DC

AE DE
f ( xi )
f ( xi 1 )

xi xi 1 xi 1 xi 1

On rearranging, the secant method is given as

xi 1 xi

f ( xi )( x i xi 1 )
f ( xi ) f ( x i 1 )

This method requires two initial estimates xo, x1. However, it is not a bracketing
method. The Secant Method has the same properties as Newtons method. Convergence
is not guaranteed for all xo, f(x).
The algorithm of the secant method can be written as follows:
Step 1: Choose two starting points x0 and x1.
Step 2: Let

Step 3: if |x2-x1|<a (Acceptable approximated error) then let root = x2, else x0 =
x1 ; x1 = x2 ; go to step 2.
Step 4: End.

2.5.1 Difference between the secant and false-position method

In the false position method, the latest estimate of the root replaces whichever of the
original values yielded a function value with the same sign as f(xr). The root is always
bracketed by the bonds and the method will always converge.
For the secant method, xi+1 replaces xi and xi replaces xi-1. As a result, the two values can
sometimes lie in the same side of the root and lead to divergence. But when, the secant
method converges, it usually does it at a quicker rate.
Example (2.5)
Find the root of the function f ( x) 5 2e x 0 use Secant method. Plot the function to
determine the initial guess.

21

Solution
1. First we plot the function. From Fig. 6 we see that the function has a root between
[-2, 0]. Let us choose x0 = -2, x1 = 0.
5

-5

-10
-2

-1

Fig. 2.7

2. Determine f (x0) and f (x1): f ( x 0 ) 5 2e x0 -9.7781 ,

f ( x1 ) 5 2e x1 3

3. Calculate x2 using the formula

=0

(2 0) (3)
= 0.4696
(9.7781 + 3)

4. Now the approximate error equals to | x2 x1| = 0.4696.

5. Now x0 = 0 and x2 = - 0.4696. Repeat steps 2 to 4 until the approximate error is
close to zero. This occurs after 7 iterations as shown in Table 2.7. However the
solution is x2 = -0.9163.

Iteration No.

x0

1
2
3
4
5
6
7

-2
0
-0.4696
-1.1753
-0.8572
-0.9089
-0.9165

Table 2.7 Secant method Example (2.7)

x1
f(x0)
f(x1)
0
-0.4696
-1.1753
-0.8572
-0.9089
-0.9165
-0.9163

-9.7781
3
1.8014
-1.4782
0.2869
0.0368
-0.0011

22

3
1.8014
-1.4782
0.2869
0.0368
-0.0011
4.0809e-006

x2

a =| x2 x1|

-0.4696
-1.1753
-0.8572
-0.9089
-0.9165
-0.9163
-0.9163

0.4696
0.7057
0.3181
0.0517
0.0076
2.2151e-004
8.1609e-007

2.6 Rate of convergence

Fig. 2.8 compares the convergence of all the methods.

Fig. 2.8

Example (2.6)
Use Fixed-point and Secant methods to find the root of the function f ( x) x 6 x 1 0
Plot the function to determine the initial guess.
Solution
Fig. 8 shows the function. It has two roots close to -1 and 1.
15
10
5
0
-5
-2

-1

23

Fig. 2.9
Table 2.8 Fixed-point method, initial guess are x0 = 1

Iteration
No.
1
2
3
4

Iteration No.
1
2
3
4
5
6

xi

f(xi)

xi 1

1
1.12246
1.13363
1.13462

-1
-0.12246
-0.01117
-0.00099

1.12246
1.13363
1.13462
1.13471

0.10910
0.00985
0.00087
7.75029e-05

Table 2.9 Secant method, initial guesses are x0 = 1, x1 = 1.5.

x0
x1
f(x0)
f(x1)
x2

a =| x2 x1|

1
1.500000
1.050552
1.083627
1.147187
1.133111

0.44944
0.03307
0.06356
0.01407
0.00156
4.813e-05

1.50000
1.05055
1.08362
1.14718
1.13311
1.13467

-1
8.89062
-0.706217
-0.464506
0.132135
-0.0165321

8.890625
-0.70621
-0.46450
0.132135
-0.01653
-0.000493

1.050552
1.083627
1.147187
1.133110
1.134676
1.134724

From Tables 2.8 and 2.9 we can see that both methods converge to the root 1.1347.
Fixed-point method converges faster than Secant method.

24

3 System of Linear equations

3.1 Guess Elimination
3.1.1 How is a set of equations solved numerically?
One of the most popular techniques for solving simultaneous linear equations is the
Gaussian elimination method. The approach is designed to solve a general set of n
equations and n unknowns

a11 x1 a12 x2 a13 x3 ... a1n xn b1

a21x1 a22 x2 a23 x3 ... a2 n xn b2
.

an1 x1 an 2 x2 an3 x3 ... ann xn bn

Gaussian elimination is a method to solve simultaneous linear equations of the form
[A][X]=[C]
It consists of two steps
1. Forward Elimination of Unknowns: In this step, the unknown is eliminated in
each equation starting with the first equation. This way, the equations are
reduced to one equation and one unknown in each equation.
2. Back Substitution: In this step, starting from the last equation, each of the
unknowns is found.
3.1.2 Forward Elimination of Unknowns:
In the first step of forward elimination, the first unknown, x1 is eliminated from all rows
below the first row. The first equation is selected as the pivot equation to eliminate x1 .
So, to eliminate x1 in the second equation, one divides the first equation by a11 (hence
called the pivot element) and then multiplies it by a21 . This is the same as multiplying
the first equation by a21 / a11 to give

a21 x1

a21
a
a
a12 x2 ... 21 a1n xn 21 b1
a11
a11
a11

25

Now, this equation can be subtracted from the second equation to give

a
a
a
a 22 21 a12 x 2 ... a 2n 21 a1n x n b2 21 b1
a11
a11
a11

or

a22 x2 ... a2 n xn b2
where

a22 a22

a21
a12
a11

a2n a2n

a21
a1n
a11

This procedure of eliminating x1 , is now repeated for the third equation to the n th
equation to reduce the set of equations as

a11 x1 a12 x2 a13 x3 ... a1n xn b1

x3 ... a2 n xn b2
a22 x2 a23
x2 a33
x3 ... a3 n xn b3
a32
.

an 2 x2 an3 x3 ... ann xn bn

This is the end of the first step of forward elimination. Now for the second step of
forward elimination, we start with the second equation as the pivot equation and a22 as
the pivot element. So, to eliminate x2 in the third equation, one divides the second

. This is the same as

equation by a22 (the pivot element) and then multiply it by a32
/ a22 and subtracting it from the third equation.
multiplying the second equation by a32
This makes the coefficient of x2 zero in the third equation. The same procedure is now
repeated for the fourth equation till the n th equation to give

26

a11 x1 a12 x2 a13 x3 ... a1n xn b1

x3 ... a2 n xn b2
a22 x2 a23
x3 ... a3n xn b3
a33
.

an3 x3 ... ann xn bn

The next steps of forward elimination are conducted by using the third equation as a pivot
equation and so on. That is, there will be a total of n 1 steps of forward elimination. At
the end of n 1 steps of forward elimination, we get a set of equations that look like

a11 x1 a12 x2 a13 x3 ... a1n xn b1

x3 ... a2 n xn b2
a22 x2 a23
x3 ... a3n xn b3
a33
.

.
n 1
a nn
x n bnn 1

3.1.3 Back Substitution:

Now the equations are solved starting from the last equation as it has only one unknown.
xn

bn( n 1)
( n 1)
a nn

Then the second last equation, that is the (n 1) th equation, has two unknowns: xn and

x n 1 , but xn is already known. This reduces the (n 1) th equation also to one unknown.
Back substitution hence can be represented for all equations by the formula

27

bii 1 aiji 1 x j
j i 1

xi

a iii 1

for i n 1, n 2, ,1

and
xn

bn( n 1)
( n 1)
a nn

Example (3.1)
The upward velocity of a rocket is given at three different times in Table 3.1.
Table 3.1 Velocity vs. time data.

Time, t (s) Velocity, v (m/s)

5

106.8

177.2

12

279.2

The velocity data is approximated by a polynomial as

vt a1t 2 a2t a3 ,

5 t 12

The coefficients a1, a2 , and a3 for the above expression are given by
25 5 1 a1 106.8
64 8 1 a 177.2

144 12 1 a 3 279.2

Find the values of a1, a2 , and a3 using the Nave Gauss elimination method. Find the
velocity at t 6, 7.5, 9, 11 seconds.
Solution

Forward Elimination of Unknowns

28

Since there are three equations, there will be two steps of forward elimination of
unknowns.
First step
Divide Row 1 by 25 and then multiply it by 64, that is, multiply Row 1 by 64/25 2.56 .

25

64

5 1
12.8 2.56

106.8 2.56 gives Row 1 as

273.408

Subtract the result from Row 2

64
64

8
1
177.2

12.8 2.56 273.408

0 4.8 1.56 96.208

to get the resulting equations as

5
1 a1 106.8
25
0 4.8 1.56 a 96.208

1 a3 279.2
144 12

Divide Row 1 by 25 and then multiply it by 144, that is, multiply Row 1 by
144/25 5.76 .

25

5 1

106.8 5.76 gives Row 1 as

144

28.8 5.76

615.168

Subtract the result from Row 3

144
144
0

12
1
279.2
28.8 5.76 615.168
16.8 4.76 335.968

to get the resulting equations as

5
1 a1 106.8
25
0 4.8 1.56 a 96.208

0 16.8 4.76 a 3 335.968

Second step

29

We now divide Row 2 by 4.8 and then multiply by 16.8, that is, multiply Row 2 by
16.8/ 4.8 3.5 .

4.8 1.56

96.208 3.5

16.8 5.46

336.728

gives Row 2 as

Subtract the result from Row 3

0
0

16 .8 4.76

16 .8 5.46
0
0
0.7

335.968
336.728
0.76

to get the resulting equations as

5
1 a1 106.8
25
0 4.8 1.56 a 96.208

0
0
0.7 a3 0.76

Back substitution

From the third equation

0.7a3 0.76
a3

0.76
0.7

1.08571

Substituting the value of a3 in the second equation,

4.8a2 1.56a3 96.208

a2

96.208 1.56a3
4.8
96.208 1.56 1.08571
4.8

19.6905

Substituting the value of a2 and a3 in the first equation,

30

25a1 5a2 a3 106.8

a1

106.8 5a 2 a3
25
106.8 5 19.6905 1.08571
25

0.290472

Hence the solution vector is

a1 0.290472
a 19.6905
2

a3 1.08571

The polynomial that passes through the three data points is then
vt a1t 2 a 2 t a3

0.290472t 2 19.6905t 1.08571, 5 t 12

Since we want to find the velocity at t 6, 7.5, 9 and 11 seconds, we could simply
substitute each value of t in vt 0.290472t 2 19.6905t 1.08571 and find the
corresponding velocity. For example, at t 6
2

v6 0.2904726 19.69056 1.08571

129.686 m/s
However we could also find all the needed values of velocity at t = 6, 7.5, 9, 11 seconds
using matrix multiplication.

vt 0.290472 19.6905

t 2

1.08571 t
1

So if we want to find v6, v7.5, v9, v11, it is given by

31

6 2
v6 v7.5 v9 v11 0.290472 19.6905 1.08571 6
1

0.290472

19.6905

129.686 165.104

7. 5 2
7.5
1

9 2 112

9 11
1
1

36 56.25 81 121
1.08571 6
7.5
9 11
1
1
1
1

201.828

252.828

v(6) 129.686 m/s

v(7.5) 165.1 04 m/s
v(9) 201.828 m/s
v(11) 252.828 m/s
3.1.4 Pitfalls of the Nave Gauss elimination method
There are two pitfalls of the Nave Gauss elimination method.
1. Division by zero: It is possible for division by zero to occur during the beginning
of the n 1 steps of forward elimination.
For example

5x2 6x3 11
4 x1 5 x2 7 x3 16
9 x1 2 x2 3x3 15
will result in division by zero in the first step of forward elimination as the coefficient of
x1 in the first equation is zero as is evident when we write the equations in matrix form.
0 5 6 x1 11
4 5 7 x 16

2
9 2 3 x3 15

But what about the equations below: Is division by zero a problem?

32

5x1 6 x2 7 x3 18
10 x1 12x2 3x3 25
20x1 17 x2 19x3 56
Written in matrix form,
5 6 7 x1 18
10 12 3 x 25

2
20 17 19 x3 56

there is no issue of division by zero in the first step of forward elimination. The pivot
element is the coefficient of x1 in the first equation, 5, and that is a non-zero number.
However, at the end of the first step of forward elimination, we get the following
equations in matrix form
7 x1 18
5 6
0 0 11 x 11

0 7 9 x3 16

Now at the beginning of the 2nd step of forward elimination, the coefficient of x2 in
Equation 2 would be used as the pivot element. That element is zero and hence would
create the division by zero problem.
So it is important to consider that the possibility of division by zero can occur at the
beginning of any step of forward elimination.
2. Round-off error: The Nave Gauss elimination method is prone to round-off
errors. This is true when there are large numbers of equations as errors propagate.
Also, if there is subtraction of numbers from each other, it may create large errors.
See the example below.
Example (3.2)
Remember Example 3.1 where we used Nave Gauss elimination to solve

20x1 15x2 10x3 45

3x1 2.249x2 7 x3 1.751
5x1 x2 3x3 9

33

using six significant digits with chopping in your calculations? Repeat the problem, but
now use five significant digits with chopping in your calculations.
Solution
Writing in the matrix form
15
10 x1 45
20
3 2.249 7 x = 1.751

5
1
3 x3 9

Forward Elimination of Unknowns

First step
Divide Row 1 by 20 and then multiply it by 3, that is, multiply Row 1 by
3 / 20 0.15 .

20

15 10

45 0.15 gives Row 1 as

2.25 1.5

6.75

Subtract the result from Row 2

3
3
0

2.249 7 1.751
2.25 1.5 6.75
0.001 8.5
8.501

to get the resulting equations as

10 x1 45
20 15
0 0.001 8.5 x = 8.501

5
1
3 x3 9

Divide Row 1 by 20 and then multiply it by 5, that is, multiply Row 1 by 5 / 20 0.25 .

20

15 10
3.75 2.5

45 0.25 gives Row 1 as

11.25

Subtract the result from Row 3

34

5
5

1
3
9

3.75 2.5 11.25

0 2.75 0.5 2.25

to get the resulting equations as

15
10 x1 45
20
0 0.001 8.5 x = 8.501

0 2.75 0.5 x3 2.25

Second step
Now for the second step of forward elimination, we will use Row 2 as the pivot equation
and eliminate Row 3: Column 2.
Divide Row 2 by 0.001 and then multiply it by 2.75, that is, multiply Row 2 by
2.75 / 0.001 2750 .

0.001 8.5

8.501 2750 gives Row 2 as

2.75 23375

23377.75

Rewriting within 5 significant digits with chopping

2.75 23375

23377

Subtract the result from Row 3

0
0
0

2.75
2.75
0

0.5
23375
23375

2.25
23377
23374

Rewriting within 6 significant digits with chopping

0 23375

23374

to get the resulting equations as

10
20 15
0 0.001 8.5

0
0
23375

x1
x =
2
x3

45
8.501

23374

This is the end of the forward elimination steps.

35

Back substitution
We can now solve the above equations by back substitution. From the third equation,

23375x3 23374
x3

23374
23375

0.99995

Substituting the value of x3 in the second equation

0.001x2 8.5 x3 8.501

8.501 8.5 x3
0.001
8.501 8.5 0.99995

0.001

x2

8.501 8.499575
0.001
8.501 8.4995

0.001

0.0015
0.001

1. 5

Substituting the value of x3 and x2 in the first equation,

20x1 15x2 10x3 45

x1

45 15 x 2 10 x3
20
45 15 1.5 10 0.99995
20

45 22.5 9.9995
20
22.5 9.9995 12.5005 12.500

20
20
20

0.625

36

Hence the solution is

x1
X x2
x3

0.625
1.5
0.99995

Compare this with the exact solution of

x1 1
X x2 1
x3 1

3.1.5 Techniques for improving the Nave Gauss elimination method

As seen in Example 3.2, round off errors were large when five significant digits were
used as opposed to six significant digits. One method of decreasing the round-off error
would be to use more significant digits, that is, use double or quad precision for
representing the numbers. However, this would not avoid possible division by zero
errors in the Nave Gauss elimination method. To avoid division by zero as well as
reduce (not eliminate) round-off error, Gaussian elimination with partial pivoting is the
method of choice.

3.1.6 Gaussian elimination with partial pivoting

At the beginning of each step of forward elimination Therefore, before each row is
normalized, it is advantageous to determine the largest available coefficient in the column
below the pivot element. The rows can then be switched so that the largest element is the
pivot element. This is called partial pivoting. If columns as well as rows are searched for
the largest element and then switched, the procedure is called complete pivoting.
Complete pivoting is rarely used because switching columns changes the order of the xs
and, consequently, adds significant and usually unjustified complexity to the computer
program. The following example illustrates the advantages of partial pivoting. Aside
from avoiding division by zero, pivoting also minimizes round-off error. As such, it also
serves as a partial remedy for ill-conditioning.
Example (3.3)
In the previous two examples, we used Nave Gauss elimination to solve

20x1 15x2 10x3 45

3x1 2.249x2 7 x3 1.751

37

5x1 x2 3x3 9
using five and six significant digits with chopping in the calculations. Using five
significant digits with chopping, the solution found was
x1
X x2
x3

0.625
1.5
0.99995

This is different from the exact solution of

x1
X x2
x3

1
1
1

Find the solution using Gaussian elimination with partial pivoting using five significant
digits with chopping in your calculations.
Solution
15
10 x1
20
45
3 2.249 7 x = 1.751

5
9
1
3 x3

Forward Elimination of Unknowns

Now for the first step of forward elimination, the absolute value of the first column
elements below Row 1 is

20 , 3 , 5
or
20, 3, 5
So the largest absolute value is in the Row 1. So as per Gaussian elimination with partial
pivoting, the switch is between Row 1 and Row 1 to give
15
10 x1
20
45
3 2.249 7 x = 1.751

5
9
1
3 x3

38

Divide Row 1 by 20 and then multiply it by 3, that is, multiply Row 1 by

3 / 20 0.15 .

20

15 10

45 0.15 gives Row 1 as

2.25 1.5

6.75

Subtract the result from Row 2

3
3

2.249 7 1.751
2.25 1.5 6.75
0.001 8.5
8.501

to get the resulting equations as

10 x1 45
20 15
0 0.001 8.5 x = 8.501

5
1
3 x3 9

Divide Row 1 by 20 and then multiply it by 5, that is, multiply Row 1 by 5 / 20 0.25 .

20

15 10
3.75 2.5

45 0.25 gives Row 1 as

11.25

Subtract the result from Row 3

5
5

1
3
9

3.75 2.5 11.25

0 2.75 0.5 2.25

to get the resulting equations as

15
10 x1
20
0 0.001 8.5 x

2
0 2.75 0.5 x3

45
8.501

2.25

This is the end of the first step of forward elimination.

Now for the second step of forward elimination, the absolute value of the second column
elements below Row 1 is

0.001 , 2.75

39

or
0.001, 2.75
So the largest absolute value is in Row 3. So Row 2 is switched with Row 3 to give
15
10 x1
20
7
0 2.75 0.5 x = 2.25

0 0.001 8.5 x3
8.501

Divide Row 2 by 2.75 and then multiply it by 0.001, that is, multiply Row 2 by
0.001 / 2.75 0.00036363 .

2.75 0.5

2.25 0.00036363 gives Row 2 as

0.00099998 0.00018182

0.00081816

Subtract the result from Row 3

0
0
0

0.001
0.00099998
0

8.5
0.00018182
8.50018182

8.501
0.00081816
8.50018184

Rewriting within 5 significant digits with chopping

0 8.5001

8.5001

to get the resulting equations as

15
10
20
0 2.75
0.5

0
0
8.5001

x1
45
x = 2.25
2

x3
8.5001

Back substitution

8.5001x3 8.5001
x3

8.5001
8.5001

=1
Substituting the value of x3 in Row 2

40

 2.75x2 0.5x3 2.25

x2

2.25 0.5 x 2
2.75
2.25 0.5
2.75

2.25 0.5 1
2.75

2.75
2.75

Substituting the value of x3 and x2 in Row 1

20x1 15x2 10x3 45

x1

45 15 x 2 10 x3
20
45 15 1 10 1
20

45 15 10
20
30 10

20

20
20

So the solution is
x1
X x2
x3

1
= 1
1

This, in fact, is the exact solution. By coincidence only, in this case, the round-off error
is fully removed.

3.2 LU-Decomposition
As described in previous section, Gauss elimination is designed to solve systems of linear
algebraic equations,
[A]{X} = {B}

41

Although it certainly represents a sound way to solve such systems, it becomes inefficient
when solving equations with the same coefficients [A], but with different right-hand-side
constants (the bs).
LU-Decomposition method: For most non-singular matrix [A] that one could conduct
Nave Gauss Elimination forward elimination steps, one can always write it as
[A] = [L][U]
where
[L] = lower triangular matrix
[U] = upper triangular matrix
If solving a set of linear equations [A][X] = [C]
If [A] = [L][U] then [L][U][X] = [C]
Multiply by [L]-1
Which gives [L]-1[L][U][X] = [L]-1[C]
Remember [L]-1[L] = [I] which leads to [I][U][X] = [L]-1[C]
Now, if [I][U] = [U] then [U][X] = [L]-1[C]
Now, let [L]-1[C]=[Z]
Which ends with

[L][Z] = [C] (1)

And

[U][X] = [Z] (2)

Steps of LU-decomposition method:

Given [A][X] = [C]
1. Decompose [A] into [L] and [U]:

a11
a
[ A] 21

a n1

a12
a 22

an2

a1n 1
a2 n 21



a nn n1

0
1

n2

0 u11
0 0

1 0

u12 u1n
u 22 u2n

0 unn

The elements of the U matrix are exactly the same as the coefficient matrix one
obtains at the end of the forward elimination steps in Nave Gauss elimination.

42

The lower triangular matrix L has 1 in its diagonal entries. The non-zero
elements on the non-diagonal elements in L are multipliers that made the
corresponding entries zero in the upper triangular matrix U during forward
elimination.
2. Confirm L U A .
3. Solve [L][Z] = [C] for [Z]
4. Solve [U][X] = [Z] for [X]
that gives the solution vector X .
Example (3.4)
Use the LU decomposition method to solve the following simultaneous linear equations.
25 5 1 a1 106.8
64 8 1 a 177.2

144 12 1 a 3 279.2

Solution
1. Decompose [A] into [L] and [U]:

A L U
1
21

31

0
1
32

0 u11
0 0

1 0

u12
u22
0

u13
u23

u33

The U matrix is the same as found at the end of the forward elimination of Nave Gauss
elimination method, that is
5
1
25

U 0 4.8 1.56
0
0
0.7

To find 21 and 31 , find the multiplier that was used to make the a 21 and a31 elements
zero in the first step of forward elimination of the Nave Gauss elimination method. It
was

43

 21

64
25

2.56

31

144
25

5.76

To find 32 , what multiplier was used to make a32 element zero? Remember a 32
element was made zero in the second step of forward elimination. The A matrix at the
beginning of the second step of forward elimination was
5
1
25
0 4.8 1.56

0 16.8 4.76

So
32

16.8
4. 8

3.5

Hence
0 0
1

L 2.56 1 0
5.76 3.5 1

2. Confirm L U A .
0 0 25
5
1
1

LU 2.56 1 0 0 4.8 1.56

5.76 3.5 1 0
0
0.7

3. Solve L Z C
0 0 z1 106.8
1
2.56 1 0 z 177.2

5.76 3.5 1 z 3 279.2

to give
z1 106 .8
2.56 z1 z 2 177 .2

44

25 5 1
64 8 1
144 12 1

5.76z1 3.5z2 z3 279.2

Forward substitution starting from the first equation gives
z1 106.8
z 2 177 .2 2.56 z1

177.2 2.56 106.8

96 .208

z3 279.2 5.76z1 3.5z2

279 .2 5.76 106 .8 3 .5 96 .208

0.76

Hence
z1
Z z 2
z 3

106.8
96.208
0.76

This matrix is same as the right hand side obtained at the end of the forward elimination
steps of Nave Gauss elimination method. Is this a coincidence?
4. Now solve U X Z
5
1 a1 106.8
25
0 4.8 1.56 a 96.208

0
0
0.7 a3 0.76
25a1 5a 2 a 3 106 .8

4.8a2 1.56a3 96.208

0.7a3 0.76
From the third equation

0.7a3 0.76
a3

0.76
0.7

1.0857

45

Substituting the value of a3 in the second equation,

4.8a2 1.56a3 96.208

96.208 1.56a3
4.8

a2

96.208 1.56 1.0857

4.8

19.691

Substituting the value of a 2 and a3 in the first equation,

25a1 5a2 a3 106.8

a1

106.8 5a 2 a3
25
106.8 5 19.691 1.0857
25

0.29048

Hence the solution vector is

a1 0.29048
a 19.691
2

a3 1.0857

3.3 Inverse of matrix

How do I find the inverse of a square matrix using LU decomposition?
A matrix B is the inverse of A if

AB I B A.
How can we use LU decomposition to find the inverse of the matrix? Assume the first
column of B (the inverse of A ) is
[b11 b12 ... ... bn1 ]T

Then from the above definition of an inverse and the definition of matrix multiplication

46

 b11 1
b 0
A 21


bn1 0
Similarly the second column of B is given by

b12 0
b 1
A 22


bn 2 0
Similarly, all columns of B can be found by solving n different sets of equations with
the column of the right hand side being the n columns of the identity matrix.
Example (3.5)
Use LU decomposition to find the inverse of the matrix
25 5 1
A 64 8 1
144 12 1

Solution
Knowing that

A L U
0 0 25
5
1
1

2.56 1 0 0 4.8 1.56

5.76 3.5 1 0
0
0.7

We can solve for the first column of [ B ] A1 by solving for

25 5 1 b11 1
64 8 1 b 0

21
144 12 1 b31 0

First solve

47

L Z C ,
that is
0 0 z1 1
1
2.56 1 0 z 0

2
5.76 3.5 1 z 3 0

to give
z1 1
2.56 z1 z 2 0

5.76z1 3.5 z 2 z 3 0
Forward substitution starting from the first equation gives
z1 1
z 2 0 2 .56 z1
0 2.56 1

2.56

z 3 0 5.76z1 3.5 z 2
0 5.76 1 3.5 2.56

3.2

Hence
z1
Z z 2
z 3

1
2.56
3.2

Now solve

U X Z
that is
5
1 b11 1
25
0 4.8 1.56 b 2.56

21

0
0
0.7 b31 3.2

48

25b11 5b21 b31 1

4.8b21 1.56b31 2.56
0.7b31 3.2
Backward substitution starting from the third equation gives
b31

3.2
0.7

b21

2.56 1.56b31
4.8

b11

4 .571

2.56 1.56(4.571)
4.8

0 .9524

1 5b21 b31
25
1 5(0.9524) 4.571
25

0.04762

Hence the first column of the inverse of A is

b11 0.04762
b 0.9524
21

b31 4.571

Similarly by solving
25 5 1 b12 0
b12 0.08333
64 8 1 b 1 gives b 1.417

22
22

144 12 1 b32 0
b32 5.000

and solving
b13 0.03571
25 5 1 b13 0
64 8 1 b 0 gives b 0.4643
23

23
b33 1.429
144 12 1 b33 1

Hence

49

0.04762 0.08333 0.03571

0.9524
1.417
0.4643
4.571
5.000
1.429

Can you confirm the following for the above example?

AA1 I A1 A

3.4 Gauss-Seidel Method

Given a general set of n equations and n unknowns, we have

a11 x1 a12 x2 a13 x3 ... a1n xn c1

a21 x1 a22 x2 a 23 x3 ... a2 n xn c2
.

an1 x1 an 2 x2 a n3 x3 ... ann xn cn

If the diagonal elements are non-zero, each equation is rewritten for the corresponding
unknown, that is, the first equation is rewritten with x1 on the left hand side, the second
equation is rewritten with x2 on the left hand side and so on as follows

x2

c2 a21 x1 a23 x3 a2 n xn
a22

xn 1
xn

cn 1 an1,1 x1 an1, 2 x2 an 1,n 2 xn 2 an 1,n xn

an 1,n 1

cn an1 x1 an 2 x2 an ,n 1 xn1
ann

These equations can be rewritten in a summation form as

50

c1 a1 j x j
j 1
j 1

x1

a11
n

c 2 a2 j x j
j 1
j 2

x2

a 22

.
.
.
n

c n 1

n 1, j

xj

j 1
j n 1

x n 1

a n 1,n 1
n

c n a nj x j
j 1
j n

xn

a nn

Hence for any row i ,

n

ci aij x j
j 1
j i

xi

aii

, i 1, 2, , n.

Now to find xi s, one assumes an initial guess for the xi s and then uses the rewritten
equations to calculate the new estimates. Remember, one always uses the most recent
estimates to calculate the next estimates, xi . At the end of each iteration, one calculates
the absolute relative approximate error for each xi as
a

x inew x iold
x inew

100

where xinew is the recently obtained value of xi , and xiold is the previous value of xi .

51

When the absolute relative approximate error for each xi is less than the pre-specified
tolerance, the iterations are stopped.
Example (3.6)
Find the solution to the following system of equations using the Gauss-Seidel method.

12 x1 3x2 5x3 1
x1 5x2 3x3 28
3x1 7 x2 13x3 76
Use
x1 1
x 0
2
x 3 1

as the initial guess and conduct two iterations.

Solution
The coefficient matrix
12 3 5
A 1 5 3
3 7 13

is diagonally dominant as
a11 12 12 a12 a13 3 5 8
a 22 5 5 a 21 a 23 1 3 4
a33 13 13 a 31 a32 3 7 10

and the inequality is strictly greater than for at least one row. Hence, the solution should
converge using the Gauss-Seidel method.
Rewriting the equations, we get
x1

1 3x 2 5 x3
12

52

x2

28 x1 3 x3
5

x3

76 3x1 7 x 2
13

Assuming an initial guess of

x1 1
x 0
2
x 3 1

Iteration #1
x1

1 30 51
12

x2

28 0.50000 31
5

x3

76 30.50000 74.9000
13

0.50000

4.9000

3.0923

The absolute relative approximate error at the end of the first iteration is
a 1

0.50000 1
100
0.50000

100.00%

a 2

4.9000 0
100
4.9000

100.00%

3.0923 1
100
3.0923

67.662%

The maximum absolute relative approximate error is 100.00%

Iteration #2
x1

1 34.9000 53.0923
12

x2

28 0.14679 33.0923
5

0.14679

3.7153

53

x3

76 30.14679 73.7153
13

3.8118

At the end of second iteration, the absolute relative approximate error is

a 1

0.14679 0.50000
100
0.14679

a 2

3.7153 4.9000
100
3.7153

31.889%

3.8118 3.0923
100
3.8118

18.874%

240.61%

The maximum absolute relative approximate error is 240.61%. This is greater than the
value of 100.00% we obtained in the first iteration. Is the solution diverging? No, as you
conduct more iterations, the solution converges as follows.
Iteration

x1

a 1 %

x2

a 2 %

x3

a 3 %

1
2
3
4
5
6

0.50000
0.14679
0.74275
0.94675
0.99177
0.99919

100.00
240.61
80.236
21.546
4.5391
0.74307

4.9000
3.7153
3.1644
3.0281
3.0034
3.0001

100.00
31.889
17.408
4.4996
0.82499
0.10856

3.0923
3.8118
3.9708
3.9971
4.0001
4.0001

67.662
18.874
4.0064
0.65772
0.074383
0.00101

This is close to the exact solution vector of

x1 1
x 3
2
x 3 4

Example (3.7)
Given the system of equations

3x1 7 x2 13x3 76
x1 5x2 3x3 28
12x1 3x2 - 5x3 1
find the solution using the Gauss-Seidel method. Use

54

 x1 1
x 0
2
x 3 1

as the initial guess.

Solution
Rewriting the equations, we get
x1

76 7 x 2 13 x3
3

x2

28 x1 3x 3
5

x3

1 12 x1 3x 2
5

Assuming an initial guess of

x1 1
x 0
2
x 3 1

the next six iterative values are given in the table below.
Iteration
1
2
3
4
5
6

x1
21.000
196.15
1995.0
20149
2.0364 105
2.0579 106

a 1 %
95.238
110.71
109.83
109.90
109.89
109.89

x2
0.80000
14.421
116.02
1204.6
12140
1.2272 105

a 2 %
100.00
94.453
112.43
109.63
109.92
109.89

x3
50.680
462.30
4718.1
47636
4.8144 105
4.8653 106

You can see that this solution is not converging and the coefficient matrix is not
diagonally dominant. The coefficient matrix
3 7 13
A 1 5 3
12 3 5

is not diagonally dominant as

55

a 3 %
98.027
110.96
109.80
109.90
109.89
109.89

a11 3 3 a12 a13 7 13 20

Hence, the Gauss-Seidel method may or may not converge. However, it is the same set of
equations as the previous example and that converged. The only difference is that we
exchanged first and the third equation with each other and that made the coefficient
matrix not diagonally dominant.
Therefore, it is possible that a system of equations can be made diagonally dominant if
one exchanges the equations with each other. However, it is not possible for all cases.
For example, the following set of equations

x1 x2 x3 3
2 x1 3x2 4 x3 9
x1 7 x2 x3 9
cannot be rewritten to make the coefficient matrix diagonally dominant.

56