An Introduction to Numerical Analysis

with MATLAB

Lecture Notes

Mohammad Sabawi

Department of Mathematics
College of Education for Women
Tikrit University

Email: [email protected]

March 2019

1
List of Tables

2.1 Bisection Method Solution of Example 9 . . . . . . . . . . . . 20

2.2 False Position Method Solution of Example 10 . . . . . . . . . 21
2.3 Newton’s Method Solution of Example 11 . . . . . . . . . . . 23
2.4 Newton’s Method Solution of Example 12 . . . . . . . . . . . 24
2.5 Secant Method Solution of Example 14 . . . . . . . . . . . . . 30
2.6 Fixed Point Method Solution of Example 16 . . . . . . . . . . 33
2.7 Fixed Point Method Solution of Example 16 . . . . . . . . . . 34
2.8 Newton’s and Accelerated Newton’s Methods Solutions of Ex-
ample 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.1 Jacobi Iterative Solution of Example 28 . . . . . . . . . . . . . 61

3.2 Gauss-Siedel Iterative Solution of Example 29 . . . . . . . . . 64

2
Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1 Introduction 6
1.1 Numerical Analysis: An Introduction . . . . . . . . . . . . . . . . . 6
1.2 Numbers Representation in Computer . . . . . . . . . . . . . . . . 7
1.2.1 Floating-Point Numbers . . . . . . . . . . . . . . . . . . . . 8
1.3 Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.1 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.2 Sources of Error in Numerical Computations . . . . . . . . 9
1.3.3 Absolute and Relative Errors . . . . . . . . . . . . . . . . . 10
1.3.4 Roundoff and Truncation Errors . . . . . . . . . . . . . . . 11
1.4 Stable and Unstable Computations: Conditioning . . . . . . . . . . 11
1.5 Convergence and Order of Approximation . . . . . . . . . . . . . . 13
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2 Numerical Solutions of Nonlinear Equations 16

2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 Bisection (Interval Halving) Method . . . . . . . . . . . . . . . . . 17
2.3 False-Position Method . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Numerical Solutions of Systems of Nonlinear Equations . . . . . . 25
2.6 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.7 Fixed Points and Functional Iteration . . . . . . . . . . . . . . . . 30
2.8 Acceleration of Iterative Methods . . . . . . . . . . . . . . . . . . . 34
2.8.1 Modified Newton’s Methods . . . . . . . . . . . . . . . . . . 34
2.9 Computing Roots of Polynomials . . . . . . . . . . . . . . . . . . . 36
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3 Solving Systems of Linear Equations 39

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 Norms of Matrix and Vectors . . . . . . . . . . . . . . . . . . . . . 39
3.3 Direct Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3.1 Backward Substitution Method . . . . . . . . . . . . . . . . 43
3.3.2 Forward Substitution Method . . . . . . . . . . . . . . . . . 45

3
 3.3.3 Gaussian Elimination Method . . . . . . . . . . . . . . . . . 46
3.3.4 Gauss-Jordan Elimination Method . . . . . . . . . . . . . . 51
3.4 LU and Cholesky Factorisations . . . . . . . . . . . . . . . . . . . 54
3.5 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.5.1 Jacobi Method . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.5.2 Gauss-Siedel Method . . . . . . . . . . . . . . . . . . . . . . 61
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

4 Interpolation and Extrapolation 66

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.2 Lagrange Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3 Newton Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.3.1 Finite Divided Differences . . . . . . . . . . . . . . . . . . . 71
4.3.2 Newton’s Interpolation Divided Difference Formula . . . . . 73
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4
 Preface
The aim of these class notes is to cover the necessary materials in a stan-
dard numerical analysis course and it s not intended to add to the plethora
of Numerical Analysis texts. We tried our best to write these notes in con-
cise, clear and accessible way, to make them more attractive to the readers.
These lecture notes cover the basic and fundamental concepts and principles
in numerical analysis and it is not a comprehensive introduction to numer-
ical analysis. We emphasise in these notes on the mathematical principles
via explaining them by the aid of numerical software MATLAB. The pre-
requisite material for this course are a course in Calculus, Linear Algebra
and Differential Equations. A basic knowledge in MATLAB is helpful but
it is not necessary. There is a glut of numerical software nowadays, among
these we chose to use MATLAB because of its wide capabilities in scientific
computing.These notes consist of ten chapters and each chapter ends with a
set of exercises address the topics covered in each chapter.

5 Mohammad Sabawi/Numerical Analysis

Chapter 1

Introduction

1.1 Numerical Analysis: An Introduction

Numerical analysis is a branch of mathematics studies the methods and al-
gorithms which used for solving a variety of problems in different areas of
todays life such as mathematics, physics, engineering, medicine and social
and life sciences. The main objective of numerical analysis is investiga-
tion finding new mathematical approaches for approximating the underlying
problems, and also development of the current algorithms and numerical
schemes to make them more efficient and reliable. The advent of computers
revolutionise numerical analysis and nowadays with parallel and super com-
puters the numerical computations became more easier compared with the
past where solving simple problems take a long time, much effort and require
hard work. In principle, numerical analysis mainly focuses on the ideas of
stability, convergence, accuracy, consistency and error analysis. In the lit-
erature numerical analysis also known as scientific computing, scientific
computation, numerics, computational mathematics and numerical
mathematics. Numerical analysis can be divided into the following fields:

1. Numerical Solutions of Linear Algebraic Equations.

2. Numerical Solutions of Nonlinear Algebraic Equations.

3. Interpolation and Extrapolation.

4. Approximation Theory and Curve Fitting.

5. Numerical Differentiation.

6. Numerical Integration.

6
 CHAPTER 1. INTRODUCTION

7. Numerical Optimisation.

8. Numerical Solutions of Eigenvalue Problems.

9. Numerical Solutions of Ordinary Differential Equations.

10. Numerical Solutions of Partial Differential Equations.

11. Numerical Solutions of Integral Equations.

12. Numerical Modelling.

13. Numerical Simulation.

Numerical analysis is dated back to the Babylonians works in approxi-

mating the square root of 2. During this long journey of evolution many
many scientists contributed to its development and progress among these we
just name a few such as Lagrange, Gauss, Newton, Euler, Legendre
and Simpson.

1.2 Numbers Representation in Computer

Human beings do arithmetic in their daily life using the decimal (base 10)
number system. Nowadays, most computers use binary (base 2) number sys-
tem. We enter the information to computers using the decimal system but
computers transform them to the binary system by using the machine lan-
guage.

Definition 1 (Scientific Notation). Let k be a real number, then k can be

written in the following form

k = m × 10n ,

where m is any real number and the exponent n is an integer. This notation is
called the scientific notation or scientific form and sometimes referred
to as standard form.

Example 1. Write the following numbers in scientific notation:

1. 0.00000834.

2. 25.45879.

3. 3400000.

7 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

4. 33.
5. 2, 300, 000, 000.
Solution:
1. 0.00000834 = 8.34 × 10−6 .
2. 25.45879 = 2.545879 × 101 .
3. 3400000 = 3.4 × 106 .
4. 33 = 3.3 × 101 .
5. 2.3 × 109 .

1.2.1 Floating-Point Numbers

In the decimal system any real number a 6= 0 can be written in the decimal
normalised floating-point form in the following way

a = ±0.d1 d2 d3 · · · dk dk+1 dk+2 · · · × 10n , 1 ≤ d1 ≤ 9, 0 ≤ di ≤ 9,

for each i = 2, · · · , and n is an integer called the exponent (n can be
positive, negative or zero). In computers we use a finite number of digits in
representing the numbers and we obtain the following form

b = ±0.d1 d2 d3 · · · dk × 10n , 1 ≤ d1 ≤ 9, 0 ≤ di ≤ 9,
for each i = 2, · · · , k. These numbers are called k-digit decimal ma-
chine numbers.

Also, the normalised floating-point decimal representation of the number

a 6= 0 can be written in other way as
1
a = ±r × 10n , (
≤ r < 1),
10
the number r is called the normalised mantissa.

The floating-point representation in binary number system can be defined

by the same way as in the decimal number system. If a 6= 0, it can be
represented as
1
a = ±p × 2m , ( ≤ p < 1),
2
where p = (0.b1 b2 b3 · · · )2 , b1 = 1.

8 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

1.3 Errors
Occurrence of error is unavoidable in the field of scientific computing. In-
stead, numerical analysts try to investigate the possible and best ways to
minimise the error. The study of the error and how to estimate and min-
imise it are the fundamental issues in error analysis.

1.3.1 Error Analysis

In numerical analysis we approximate the exact solution of the problem by
using numerical method and consequently an error is committed. The numer-
ical error is the difference between the exact solution and the approximate
solution.

Definition 2 (Numerical Error). Let x be the exact solution of the underlying

problem and x∗ its approximate solution, then the error (denoted by e) in
solving this problem is

e = x − x∗ .

1.3.2 Sources of Error in Numerical Computations

• Blunders (Gross Errors) These errors also called humans errors,
and are caused by humans mistakes and oversight and can be minimised
by taking care during scientific investigations. These errors will add to
the total error of the underlying problem and can significantly affect
the accuracy of solution.

• Modelling Errors These errors arise during the modelling process

when scientists ignore effecting factors in the model to simplify the
problem. Also, these errors known as formulation errors.

• Data Uncertainty These errors are due to the uncertainty of the

physical problem data and also known as data errors.

• Discretisation Errors Computers represent a function of continuous

variable by a number of discrete values. Also, scientists approximate
and replace complex continuous problems by discrete ones and this
results in discretisation errors.

9 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

1.3.3 Absolute and Relative Errors

Definition 3 (Absolute Error). The absolute error ê of the error e is defined
as the absolute value of the error e

ê = |x − x∗ |.

Definition 4 (Relative Error). The relative error ẽ of the error e is defined

as the ratio between the absolute error ê and the absolute value of the exact
solution x

ê |x − x∗ |
ẽ = = , x 6= 0.
|x| |x|
Example 2. Let x = 3.141592653589793 is the value of the constant ratio π
correct to 15 decimal places and x∗ = 3.14159265 be an approximation of x.
Compute the following quantities:

a. The error.

b. The absolute error.

c. The relative error.

Solution:

a. The error

e = x − x∗ = 3.141592653589793 − 3.14159265 = 3.589792907376932e − 09

= 3.589792907376932 × 10−9 = 0.000000003589792907376932.

b. The absolute error

ê = |x−x∗ | = |3.141592653589793−3.14159265| = 3.589792907376932e−09.

c. The relative error

ê |x − x∗ | 3.141592653589793 − 3.14159265
ẽ = = =
|x| |x| 3.141592653589793
3.589792907376932e − 09
= = 1.142666571770530e − 09.
3.141592653589793

10 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

1.3.4 Roundoff and Truncation Errors

Computers represent numbers in finite number of digits and hence some
quantities cannot be represented exactly. The error caused by replacing a
number a by its closest machine number is called the roundoff error and
the process is called correct rounding.

Truncation errors also sometimes called chopping errors are occurred

when chopping an infinite number and replaced it by a finite number or by
truncated a series after finite number of terms.
Example 3. Approximate the following decimal numbers to three digits by
using rounding and chopping (truncation) rules:

1. x1 = 1.34579. 4. x4 = 3.34379.
2. x2 = 1.34679.
3. x3 = 1.34479. 5. x5 = 2.34579.

Solution:

(i) Rounding: (ii) Chopping:

(a) x1 = 1.35. (a) x1 = 1.34.

(b) x2 = 1.35. (b) x2 = 1.34.
(c) x3 = 1.34. (c) x3 = 1.34.
(d) x4 = 3.34. (d) x4 = 3.34.
(e) x5 = 2.35. (e) x5 = 2.34.

1.4 Stable and Unstable Computations: Con-

ditioning
Stability is one of the most important characteristics in any efficient and
robust numerical scheme.
Definition 5 (Numerical Stability). The numerical algorithm is called sta-
ble if the final result is relatively not affected by the perturbations during
computation process. Otherwise it is called unstable.
The stability notion is analogous and closely related to the notion of
conditioning.

11 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

Definition 6 (Conditioning). Conditioning is a measure of how sensitive

the output to small changes in the input data. In literature conditioning is
also called sensitivity.

• The problem is called well-conditioned or insensitive if small changes

in the input data lead to small changes in the output data.

• The problem is called ill-conditioned or sensitive if small changes

in the input data lead to big changes in the output data.

Definition 7 (Condition Number of a Function). If f is a differentiable

function at x in its domain then the condition number of f at x is

|xf 0 (x)|
Cond(f (x)) = , f (x) 6= 0.
|f (x)|
Note: Condition number of a function f at x in its domain sometimes
denoted by Cf (x).

Definition 8 (Condition Number of a Matrix). If A is a non-singular n × m

matrix, the condition number of A is defined by

cond(A) = kAkkA−1 k,
where
kAxk
kAk = max ,
x6=0 kxk
and x is a m × 1 column vector.

Definition 9 (Well-Posed Problem). The problem is well-posed if satisfies

the following three conditions:

a. The solution exists.

b. The solution is unique.

c. The solution depends continuously on problem data.

Otherwise, the problem is called ill-posed.

Remark 1. Note that:

1. The problem is ill-posed or sensitive if cond  1.

2. The problem is well-posed or insensitive if cond < 1.

12 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

√
Example 4. Find the condition number of the function f (x) = x.

Solution:
√ 1
f (x) = x =⇒ f 0 (x) = √ , x 6= 0,
2 x
implies that
|xf 0 (x)| | 2√x x | 1
cond(f (x)) = = √ = .
|f (x)| | x| 2
This indicates that the small changes in the input date lead to changes in the
output data of half size the changes in the input data.

Example 5. Let  
1 −1 1
A =  1 0.5 3  ,
0.1 1 0.3
the inverse of A can be computed by using MATLAB command inv(A) to
obtain  
4.7500 −2.1667 5.8333
A−1 =  0.5000 −0.3333 1.6667  .
−3.2500 1.8333 −4.1667
Also, the condition number of A and its inverse can be computed using MAT-
LAB commands cond(A) and cond(inv(A)) to have cond(A) = 37.8704
and cond(A−1 ) = 37.8704. We notice that the matrix A and its inverse have
the same condition number.

Definition 10 (Accuracy). It is a measure of closeness of the approximate

solution to the exact solution.

Definition 11 (Precision). It is a measure of closeness of the two or more

measurements to each other.

Remark 2. Note that the accuracy and precision are different and they are
not related. The problem maybe very accurate but imprecise and vice versa.

1.5 Convergence and Order of Approxima-

tion
Convergence of the numerical solution to the analytical solution is one of the
important characteristic in any good and reliable numerical scheme.

13 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

Definition 12 (Convergence of a Sequence). Let {an } be an infinite sequence

of real numbers. This sequence is said to be convergent to a real number a
(has a limit at a) if, for any  > 0 there exists a positive integer N () such
that

|an − a| < , whenever n > N (),

Otherwise it is called a divergent sequence, a is called the limit of the
sequence an . Other commonly used notations for convergence are:
lim an = a or an → a as n, or lim (an − a) = 0,
n→∞ n→∞

this means that the sequence {an } converges to a otherwise it diverges.

Definition 13 (Order of Convergence). Let the sequence {an } converges to
a and set en = an − a for any n > 0. If two positive constants M 6= 0 and
q > 0 exist, such that
|an+1 − a| |en+1 |
lim q
= lim = M,
n→∞ |an − a| n→∞ |en |q

then the sequence {an } is to be convergent to a with the order of conver-

gence q.

If q = 1, the convergence is called linear.

If q = 2, the convergence is called quadratic.
If q = 3, the convergence is called cubic.

Note that the convergence gets more rapid as q gets larger and larger.
Example 6. Consider the sequence { n1 }, where n is a positive integer. Ob-
serve that n1 → 0 as n → ∞, it follows that
1
lim = 0.
n→∞ n

Definition 14 (Order of Approximation O(hn )). The function f (h) is said

to be big Oh of the function g(h), if two real constants c, and C exist such
that
|f (h)| ≤ C|g(h)| whenever h < c,
and denoted by f (h) = O(g(h)). The order of approximation is used to
determine the rate at which a function converges.
Example 7. Consider the functions f (x) = x + 1 and g(x) = x2 , where
x ≥ 1. Observe that x ≤ x2 and 1 ≤ x2 for x ≥ 1, hence f (x) = x + 1 ≤
2x2 = 2g(x) for x ≥ 1. Consequently, f (x) = O(g(x)).

14 Mohammad Sabawi/Numerical Analysis

 CHAPTER 1. INTRODUCTION

Exercises

Exercise 1. Write the following numbers in scientific form:

1. 23.123. 4. 39776444.

2. 30, 000, 000. 5. −345.386443.

3. 0.000001573. 6. −23000000.

Exercise 2. Evaluate error, absolute error and relative error of the following
values and their approximations:

1. x = 1, 000, 000, x∗ = 999, 999.

2. y = 0.00012887765, y ∗ = 0.00012897766.

3. z = 9776.96544, z ∗ = 9775.66544.

Exercise 3. Approximate the following numbers to four digits using rounding

and chopping:

1. 1.98876. 3. 8.98879.

2. 33.87654. 4. 2.88778.

Exercise 4. Compute the condition number of the following functions:

1. f (x) = cos(x).

2. f (x) = cos−1 (x).

15 Mohammad Sabawi/Numerical Analysis

Chapter 2

Numerical Solutions of
Nonlinear Equations

2.1 Introduction
Nonlinear algebraic equations are wide spread in science and engineering
and therefore their solutions are important scientific applications. There are
a glut of numerical methods for solving these equations, and in these lecture
notes, we study the most commonly used ones such as bisection, secant and
Newton methods. Locating positions of roots of nonlinear equation is a topic
of great importance in numerical mathematical analysis. The problem under
consideration maybe has a root or has no root at all.

Definition 15 (Zero of a Function). Let f be a real or complex valued func-

tion of a real or complex variable x. A real or complex number r satisfies
f (r) = 0 is called zero of f or also called a root of equation f (r) = 0.

For example, the function f (x) = x2 − 5x + 6 = (x − 2)(x − 3) has two real

zeros r1 = 2 and r2 = 3, whereas the corresponding equation x2 − 5x + 6 =
(x − 2)(x − 3) = 0 has two real roots r1 = 2 and r2 = 3.

Definition 16 (Order of a Zero). Let f and its derivatives f 0 , f 00 , · · · , f (M )

are continuous and defined on an interval about the zero x = r. The function
f or the equation f (x) = 0 is said to be has a zero or a root of order M ≥ 1
at x = r if and only if

f (r) = 0, f 0 (r) = 0, f 00 (r) = 0, · · · , f (M −1) (r) = 0, f (M ) 6= 0.

16
 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

If M = 1 then r is called a simple zero or a simple root, and if M > 1 it

is called a multiple zero or a multiple root. A zero (root) of order M = 2
is called a double zero (root), and so on. Also, the zero (root) of order M
is called a zero (root) of multiplicity M .

Lemma 1. If the function f has a zero r of multiplicity M , then there exists

a continuous function h such that f can be factorised as

f (x) = (x − r)M h(x), lim h(x) 6= 0.

x→r

Theorem 5 (Simple Zero Theorem). Assume that f ∈ C 1 [a, b]. Then, f has
a simple zero at r ∈ (a, b) if and only if f (r) = 0 and f 0 (r) 6= 0.

Example 8. Show that the function f (x) = e2x − x2 − 2x − 1 has a zero of

multiplicity 2 (double zero) at x = 0.

Solution:

f (x) = e2x − x2 − 2x − 1, f 0 (x) = 2e2x − 2x − 2, and f 00 (x) = 4e2x − 2.

Hence,

f (0) = e0 −0−0−1 = 0, f 0 (0) = 2e0 −0−2 = 0, and f 00 (0) = 4e0 −2 = 2 6= 0,

so, this implies that f ha s a double zero at x = 0.

2.2 Bisection (Interval Halving) Method

It is a bracketing method used to find a zero of a continuous function f on the
interval [a, b] where a and b are real numbers, i.e. find x such that f (x) = 0.
This method based on the Intermediate Value Theorem and also known as
Bolzano or binary search method. This method requires f (a) and f (b)
have opposite signs since f is continuous its graph is unbroken and according
to the Intermediate Value Theorem it has a root x = r somewhere in the
interval. The first step in the solution process is is to compute the midpoint
c = a+b2
of the interval [a, b] and to proceed we consider the three cases:

1. If f (a)f (c) < 0 then r lies in [a, c].

2. If f (c)f (b) < 0 then r lies in [c, b].

3. If f (c) = 0 then the root is c = r.

17 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

a1 +b1
To begin, set a1 = a and b1 = b and let c1 = 2
be the midpoint of the
interval [a1 , b1 ] = [a, b].

• If f (c1 ) = 0 then the root is r = c1 .

• If f (c1 ) 6= 0 then either f (a1 )f (c1 ) < 0 or f (c1 )f (b1 ) < 0.

(i). If f (a1 )f (c1 ) < 0 then r lies in [a1 , c1 ], and set a2 = a1 and b2 = c1 ,
i.e. [a2 , b2 ] = [a1 , c1 ].
(ii). If f (c1 )f (b1 ) < 0 then r lies in [c1 , b1 ], and set a2 = c1 and b2 = b1 ,
i.e. [a2 , b2 ] = [c1 , b1 ].
a2 +b2
(iii). Compute c2 = 2
the midpoint of the interval [a2 , b2 ].
(iv). Then, we proceed in this way until we reach the nth interval [an , bn ]
and then compute its midpoint cn = an +b2
n
.

• Finally, construct the interval [an+1 , bn+1 ] which brackets the root and
its midpoint cn+1 = an+ +b
2
n+1
will be an approximation to the root r.

Remark 3. (a). The interval [an+1 , bn+1 ] is wide as half as the interval [an , bn ]
i.e. the width of each interval is as half as the width of the previ-
ous interval. Let {`n } be a sequence of widths of intervals [an , bn ], i.e.
`n = bn −a
2
n
, n = 1, 2, · · · . Hence limn→∞ `n = 0, where  is the preas-
signed value of the error (tolerance) i.e.

|rn+1 − rn | ≤ , n = 0, 1, · · · ,

(b). The sequence of left endpoints an , n = 1, 2, · · · , is increasing and the

sequence bn , n = 1, 2, · · · of right endpoints is decreasing i.e.

a0 ≤ a1 ≤ a2 ≤ · · · ≤ an ≤ · · · ≤ r ≤ · · · ≤ b n ≤ · · · ≤ b 2 ≤ b 1 ≤ b 0 .

Theorem 6 (Bisection Method Theorem). Let f ∈ C[a, b] such that f (a)f (b) <
0 and that there exists a number r ∈ [a, b] such that f (r) = 0, and {cn } a
sequence of the midpoints of intervals [an , bn ] constructed by the bisection
method, then the error in approximating the root r in the nth step is:
b−a
|en | = |r − cn | ≤ , n = 0, 1, · · · .
2n+1
Hence, the sequence {cn } is convergent and its limit is the root r, i.e.

lim cn = r.
n→∞

18 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

Proof. For Proof see the References or have a look in any standard numerical
analysis text.

Remark 4. Remark

• The number N of repeated bisections required to compute the nth ap-

proximation (midpoint) cn of the root r is:
 ln(b − a) − ln() 
N = int .
ln(2)

• The width of the nth interval is:

|b0 − a0 |
|bn − an | = .
2n

Example 9. (a) Use bisection method to show that f (x) = x sin(x) − 1 = 0

has a real root in [0.5, 1.5]. Compute eleven approximations (i.e. use
n = 10) to the root.

(b) Evaluate the number of computations N required to ensure that the error
is less than the preassigned value (error bound)  = 0.001.

Solution:
(a) We start with initial interval [a0, b0] = [0.5, 1.5] and compute f (0.5) =
−0.76028723 and f (1.5) = 0.49624248. We notice that f (a0 ) and f (b0 ) have
opposite signs and hence, there is a root in the interval [0.5, 1.5]. Compute
the midpoint c0 = a0 +b
2
0
= 0.5+1.5
2
= 1 and f (1) = −0.15852902. The function
changes sign on [c0 , b0 ] = [1, 1.5], so, we set [a1 , b1 ] = [c0 , b0 ] = [1, 1.5], and
compute the midpoint c1 = a1 +b 2
1
= 1+1.5
2
= 1.25 and f (1.25) = 0.18623077.
Hence, the root lies in the interval [a1 , c1 ] = [1, 1.25]. Set [a2 , b2 ] = [a1 , c1 ] =
[1, 1.25] and continue until we compute c10 = 1.11376953125. The details are
explained in Table 2.1.

19 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

n Left Endpoint an Midpoint cn Right Endpoint bn Function Value f (cn )

0 0.5 1 1.5 −0.15852902
1 1 1.25 1.5 0.18623077
2 1 1.125 1.25 0.01505104
3 1 1.0625 1.125 −0.07182663
4 1.0625 1.09375 1.125 −0.02836172
5 1.09375 1.109375 1.125 −0.00664277
6 1.109375 1.1171875 1.125 0.00420803
7 1.109375 1.11328125 1.1171875 −0.00121649
8 1.11328125 1.115234375 1.1171875 0.00149600
9 1.11328125 1.1142578125 1.115234375 0.00013981
10 1.11328125 1.11376953125 1.1142578125 −0.00053832

Table 2.1: Bisection Method Solution of Example 9

(b)

 ln(1.5 − 0.5) − ln(0.001)   ln(1) − ln(0.001) 

N = int = int =
ln(2) ln(2)
 0 − (−6.90775528)   6.90775528 
int = int = int(9.96578429) = 10.
0.69314718 0.69314718

2.3 False-Position Method

It also known as regula falsi method, it is similar to the bisection method
in requiring that f (a) and f (b) have opposite signs. This method uses the
abscissa of the point (c, 0) at which the secant line called it SL joining the
points (a, f (a)) and (b, f (b)) cross the x-axis instead of using the midpoint of
the interval as approximation of the zero of the function f as in the bisection
method. To evaluate c, we need to compute the slope of line SL between the
two points (a, f (a)) and (b, f (b)):

f (b) − f (a)
m= .
b−a
Now, compute the slope of line SL between the two points (c, f (c)) = (c, 0)
and (b, f (b)):
f (b) − f (c) f (b) − 0 f (b)
m= = = .
b−c b−c b−c

20 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

By equating the two slopes, we obtain

f (b) − f (a) f (b) f (b)(b − a)
= =⇒ c=b− .
b−a b−c f (b) − f (a)
Now, we have the same possibilities as in the bisection method:
• If f (c0 ) = 0 then the root is r = c0 .
• If f (c0 ) 6= 0 then either f (a0 )f (c0 ) < 0 or f (c0 )f (b0 ) < 0.
(i). If f (a0 )f (c0 ) < 0 then r lies in [a0 , c0 ], and set a1 = a0 and b1 = c0 ,
i.e. [a1 , b1 ] = [a0 , c0 ].
(ii). If f (c0 )f (b0 ) < 0 then r lies in [c0 , b0 ], and set a1 = c0 and b1 = b0 ,
i.e. [a1 , b1 ] = [c0 , b0 ].
f (b1 )(b1 −a1 )
(iii). Compute c1 = b1 − f (b1 )−f (a1 )
.
(iv). Then, we proceed in this way until we reach the nth interval [an , bn ]
n −an )
and then compute cn = bn − ff(b(bnn)(b
)−f (an )
.

Example 10. Show that f (x) = 2x3 − x2 + x − 1 = 0 has at least on root in

[0, 1].
Solution:
Since f (0) = −1 and f (1) = 1, then Intermediate Value Theorem implies that
this continuous function has a root in [0, 1]. Set [a0 , b0 ] = [0, 1] and compute
0 −a0 )
c0 = b0 − ff(b(b00)(b
)−f (a0 )
= f (1) − ff (1)−f
(1)(1−0)
(0)
1(1−0)
= 1 − 1−(−1) = 0.5, also compute
f (c0 ) = f (0.5) = −0.5. Hence, the root lies in [c0 , b0 ] we squeeze from the
left and set a1 = c0 = 0.5 and b1 = b0 = 1, to have [a1 , b1 ] = [0.5, 1]. Now,
1 −a1 )
compute the new approximation to the root c1 = b1 − ff(b(b11)(b )−f (a1 )
= f (1) −
f (1)(1−0.5) 1(1−0.5)
= 1 − 1−(−0.5)
f (1)−f (0.5)
= 2/3 ≈ 0.66666667, f (c1 ) = −0.18518519. The
function has opposite signs on the interval [c1 , b1 ], set a2 = c1 = 0.66666667
and b2 = b1 = 1. so we have [a2 , b2 ] = [0.66666667, 1]. Continue this we and
stop at c7 = 0.73895443.
n an cn bn f (cn )
0 0 0.5 1 −0.5
1 0.5 0.66666667 1 −0.18518519
2 0.66666667 0.71875000 1 −0.05523681
3 0.71875000 0.73347215 1 −0.01532051
4 0.73347215 0.73749388 1 −0.00416160
5 0.73749388 0.73858180 1 −0.00112399
6 0.73858180 0.73887530 1 −0.00030311
7 0.73887530 0.73895443 1 −0.00008171

21 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

Table 2.2: False Position Method Solution of Example 10

2.4 Newton’s Method

Newton’s method is also known as Newton-Raphson method is one of
the most powerful and efficient numerical methods for root-finding problems.
It is well-known and popular method and there are several variants and
extensions of this method. There are more than one approach for deriving
this method such as the graphical technique and Taylor series technique, and
0 00
here we will use the latter. Let f, f , f are continuous functions on the
interval [a.b] (i.e. f ∈ C 2 [a, b]. Let r0 ∈ [a, b] be an approximation to the
0
zero r of the function f such that f (r0 ) 6= 0 and r0 is ”sufficiently close to
r i.e. |r − r0 | is relatively small”. Let’s start with first Taylor polynomial of
f (x) expanded about the initial approximation r0 and compute it at x = r:
0 (r − r0 )2 00
f (r) = f (r0 ) + (r − r0 )f (r0 ) + f (ξ(r)),
2
where ξ(r) lies between r0 and r. Using the fact that f (r) = 0, this leads to
0 (r − r0 )2 00
0 = f (r0 ) + (r − r0 )f (r0 ) + f (ξ(r)).
2
Since |r − r0 | is small then (r − r0 )2 is much smaller, so we can neglect the
third term in Taylor’s expansion which contains this term (quadratic power
term) to have
0
0 = f (r0 ) + (r − r0 )f (r0 ).
Solving for r yields
f (r0 )
r = r0 − .
f 0 (r0 )
To, proceed, set r = r1 in the Newton’s formula to compute r1 by using the
known value r0
f (r0 )
r1 = r0 − 0 ,
f (r0 )
and then we compute r2 using the known value r1
f (r1 )
r2 = r1 − ,
f 0 (r1 )
and by following the same fashion, we compute r3 , r4 and so on. The general
or nth form of Newton’s method is:
f (rn−1 )
rn = rn−1 − , n = 1, 2, · · · .
f 0 (rn−1

22 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

This is called Newton’s formula or Newton-Raphson formula.

Example 11. Use Newton’s method to find the positive root accurate to
within 10−5 for f (x) = 3x − ex = 0. Start with the initial guess r0 = 1.5.

Solution: Start by finding the derivative of f (x):

0 0
f (x) = 3x−ex , f (x) = 3−ex , r0 = 1.5, f (r0 ) = 0.01831093, f (r0 ) = −1.48168907,

so, the Newton-Raphson iteration formula for this problem is:

f (rn−1 ) 3rn−1 − ern−1

rn = rn−1 − 0 = rn−1 − , n = 1, 2, · · · , .
f (rn−1) 3 − ern−1
Computing r1 by using the known value r0 ,

f (r0 ) 3r0 − er0 3(1.5) − e1.5

r1 = r0 − 0 = r0 − = 1.5 − = 1.51235815.
f (r0 ) 3 − er0 3 − e1.5
0
Now, compute f (r1 ) and f (r1 ),
0
f (r1 ) = 3r1 − er1 = −0.00034364, f (r1 ) = 3 − er1 = −1.53741808.

Next, we compute r2 ,

f (r1 ) 3r1 − er1 0

r2 = r1 − 0 = r1 − r
= 1.51213463, f (r2 ) = −1.1e−07, f (r2 ) = −1.53640399.
f (r1 ) 3−e 1

A summary of the computations is given in Table 2.3.

n rn f (rn ) f 0 (rn )
0 1.50000000 0.01831093 −1.48168907
1 1.51235815 −0.00034364 −1.53741808
2 1.51213463 −0.00000011 −1.53640399
3 1.51213455 −0.00000000 −1.53640365
4 1.51213455 0 −1.53640365

Table 2.3: Newton’s Method Solution of Example 11

Newton’s Method for Finding the nth Roots

We start with square
√ roots. Let B > 0 a real number and r0 be an initial
approximation
√ to B. Our goal is to find a square root of a number B. Let
x = B, so x2 = B, which implies that x2 −B = 0, define f (x) = x2 −B = 0.

23 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

√
Note that this equation has two roots x = ± B. Now, find the derivative
of f , f 0 (x) = 2x and use the Newton’s fixed point formula
B
f (x) x2 − B x2 + B x+ x
x = g(x) = x − 0 =x− = = .
f (x) 2x 2x 2

Now, using Newton’s iteration formula

B
rn + rn
rn+1 = , n = 0, 1, · · · .
2
√
The sequence of iterations {rn }∞
n=0 converges to B. Note that in computing
the square root of B, we do not need to evaluate f and f 0 and this makes
the calculations easier and faster since we just need the values of the iterates
rn , n = 0, 1, · · · .
√
Example 12. Use Newton’s square-root algorithm to find 3, use r0 = 1.

Solution: Starting with r0 = 1 when n = 0, we have

3
r0 + r0 1+3
r1 = = = 2.
2 2
For n = 1,
3 3
r1 + r1 2+ 2
r2 = = = 1.75.
2 2
3 3
r2 + 1.75 + 1.75
r2
n = 2, r3 = = = 1.732142857142857.
2 2
r3 + r33
n = 3, r4 = = 1.732050810014727.
2
A summary of results is given in Table 2.4

n rn
0 1
1 2
2 1.75
3 1.732142857142857
4 1.732050810014727
5 1.732050807568877
6 1.732050807568877

Table 2.4: Newton’s Method Solution of Example 12

24 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

2.5 Numerical Solutions of Systems of Non-

linear Equations
Some phenomena in nature are modelled by systems of N nonlinear equa-
tions in N unknowns. These systems can be handled firstly by linearising
them and then solving them in repeated way. Newton’s method for a single
nonlinear equation follows the same approach and can be easily extended for
solving a system of nonlinear equations.

The general form of a system of N nonlinear equations in N unknowns

xi is:

f1 (x1 , x2 , · · · , xN ) = 0
f2 (x1 , x2 , · · · , xN ) = 0
..
.
fN (x1 , x2 , · · · , xN ) = 0.

Using vector notation, this system can be written in this concise form

F (X) = 0,
where

F = [f1 , f2 , · · · , fN ]T
X = [x1 , x2 , · · · , xN ]T .

Newton’s formula for a single nonlinear equation can be extended to a

system of nonlinear equations in the following form
0
X (k+1) = X (k) − [F (X (k) )]−1 F (X (k) ),
0
where F (X (k) ) is the Jacobian matrix which will be defined below. It
(k) (k) (k)
contains the partial derivatives of F evaluated at X (k) = [x1 , x2 , · · · , xN ]T .
The above-mentioned formula is similar to the Newton’s formula for a single
nonlinear equation except that the derivative appeared in the numerator as
an inverse of the Jacobian matrix. In practice, the inverse will not be com-
puted since this is impractical because its computational cost and instead we
will solve a related linear system.

25 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

The method will explained by solving a system of three nonlinear equa-

tions

f1 (x1 , x2 , x3 ) = 0
f2 (x1 , x2 , x3 ) = 0
f3 (x1 , x2 , x3 ) = 0.

The Taylor series expansion in three variables x1 , x2 , x3 :

∂fi ∂fi ∂fi

fi (x1 + h1 , x2 + h2 , x3 + h3 ) = fi (x1 , x2 , x3 ) + h1 + h2 + h3 +··· ,
∂x1 ∂x2 ∂x3
where the partial derivatives are evaluated at the point (x1 , x2 , x3 ). We
consider just the linear terms in step sizes hi for i = 1, 2, 3. Assume that we
have in vector notation
0
0 ≈ F (X (0) + H (0) ) ≈ F (X (0) ) + F (X (0) )H (0) ,

0
where F (X (0) ) is the Jacobian matrix at the initial guess X (0) = (x01 , x02 , x03 ),

 
∂f1 ∂f1 ∂f1
∂x1 ∂x2 ∂x3
0  ∂f2 ∂f2 ∂f2 
F (X (0) ) =  ∂x1 ∂x2 ∂x3  ,
∂f3 ∂f3 ∂f3
∂x1 ∂x2 ∂x3

where the partial derivatives are evaluated as follows:

∂fi ∂fi (X (0) )

= , i, j = 1, 2, 3.
∂xj ∂xj
0
If the Jacobian matrix F (X (0) ) is non singular i.e its inverse is existed, then
solving for H, we find
0
H (0) = −[F (X (0) )]−1 F (X (0) ).

The next iteration after correction X (1) = X (0) + H (0) is closer to the root
than X (0) . Hence, Newton’s formula for the first iteration is

0
X (1) = X (0) − [F (X (0) )]−1 F (X (0) ).

26 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

Consequently, the general form of Newton’s method for solving the non-
linear system is:
0
X (k+1) = X (k) − [F (X (k) )]−1 F (X (k) ), k = 0, 1, ... .

To avoid computing the inverse of the Jacobian matrix at each iteration,

we instead resort to solving the Jacobian linear systems
0
[F (X (k) )]H (k) = −F (X (k) ), k = 0, 1, ... ,
Hence, the next Newton’s iteration is computed using the formula

X (k+1) = X (k) + H (k) , k = 0, 1, ... .

Example 13. Solve the following nonlinear system using Newton’s method.
Start with the initial guess X (0) = (x01 = 1, x02 = 0, x03 = 0). The exact
solution to this system is X = (x1 = 0, x2 = 1, x3 = 1).

x1 + x2 + x3 = 2
x21 + x22 + x23 = 2
ex1 + x1 x2 − x1 x3 = 1.

Solution: We compute the Jacobian matrix

 
∂f1 ∂f1 ∂f1  
1 1 1
0  ∂x
∂f
1 ∂x2
∂f2
∂x3
∂f2 
F (X) =  ∂x21 ∂x2 ∂x3  = 2x1 2x2 2x3  .
∂f3
∂x1
∂f3
∂x2
∂f3
∂x3
ex1 + x2 − x3 x1 −x1

The Jacobian matrix at the initial guess X (0)

 
∂f1 ∂f1 ∂f1    
∂x1 ∂x2 ∂x3 1 1 1 1 1 1
0
F (X (0) ) =
 ∂f2
 ∂x1
∂f2
∂x2
∂f2 
∂x3  = 2x01 2x02 2x03  =  2 0 0  .
(0) (0) (0) (0) (0)
∂f3
∂x1
∂f3
∂x2
∂f3
∂x3
ex1 + x2 − x3 x1 −x1 e1 1 −1

Now, solving the Jacobian linear system for H (0)

0
[F (X (0) )]H (0) = −F (X (0) ),

27 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

we get

   0  
1 1 1 h1 −1
 2 0 0  h02  = −  −1  ,
e1 1 −1 h03 e1 − 1

implies that

 
0.5000
H (0) = −1.2887 .
1.7887

Hence,
     
1 0.5000 1.5000
X (1) = X (0) + H (0) = 0 + −1.2887 = −1.2887 .
0 1.7887 1.7887
Now, compute the Jacobian matrix for X (1) .

   
1 1 1 1.0000 1.0000 1.0000
0
F (X (1) ) =  2x11 2x12 2x13  = 3.0000 −2.5774 3.5774  .
(1) (1) (1) (1) (1)
e x1 + x2 − x3 x1 −x1 1.4043 1.5000 −1.5000

   
f1 (x1 , x2 , x3 ) 0
F (X (1) ) = f2 (x1 , x2 , x3 ) =  5.1102  .
f3 (x1 , x2 , x3 ) −1.1344

Solving for H (1) , we have

   1  
1.0000 1.0000 1.0000 h1 0
3.0000 −2.5774 3.5774  h12  = −  5.1102  ,
1.4043 1.5000 −1.5000 h13 −1.1344

implies

 
−0.5172
H (1) =  0.8788  .
−0.3616

28 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

The next approximation is

     
1.5000 −0.5172 0.9828
X (2) = X (1) + H (1) = −1.2887 +  0.8788  = −0.4099 .
1.7887 −0.3616 1.4271
Continuing as before, until we reach the required results.

2.6 Secant Method

Newton’s method is a very powerful and efficient technique for solving root-
finding problems but one of the drawbacks of the method is the need of
derivative evaluations of f at the approximations rn , n ≥ 0, and this is not a
trivial task. To avoid this we introduce secant method which is a variation
of Newton’s method. Secant method is similar to the false position method
but it differs in the way of choosing the succeeding terms.We start with two
initial points (r0 , f (r0 )) and (r1 , f (r1 )) near the point (r, 0), where r is the
root of equation f (x) = 0. Define the point (r2 , 0) to be the point of intersec-
tion of the secant line joining the points (r0 , f (r0 )) and (r1 , f (r1 )) with the
x-axis. Geometrically, the abscissa of the point of intersection r2 is closer to
the root r than to either r0 and r1 .

The slope of the secant line relating these three points (r0 , f (r0 )), (r1 , f (r1 ))
and (r2 , f (r2 )) is:

f (r1 ) − f (r0 ) f (r2 ) − f (r1 ) 0 − f (r1 ) −f (r1 )

m= and m = = = .
r1 − r0 r2 − r1 r2 − r1 r2 − r1
Equating the two values of the slope, we have
f (r1 ) − f (r0 ) −f (r1 )
= .
r1 − r0 r2 − r1
Solving slope’s equation for r2 , we obtain
f (r1 )(r1 − r0 )
r2 = r1 − .
f (r1 ) − f (r0 )
So, the general form of the secant method is:
f (rn+1 )(rn+1 − rn )
rn+2 = rn+1 − , n = 0, 1, · · · , .
f (rn+1 ) − f (rn )

29 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

Example 14. Find the root of equation x − cos(x) = 0 using the secant
method and the two initial guesses r0 = 0.5 and r1 = 0.6.

Solution: To compute the first approximation r2 , we need to compute f (r0 )

and f (r1 )

f (r0 ) = f (0.5) = 0.5 − cos(0.5) = −0.377582560000000,

f (r1 ) = f (0.6) = 0.6 − cos(0.6) = −0.225335610000000.

So,

f (r1 )(r1 − r0 ) f (0.6)(0.6 − 0.5)

r2 = r1 − = 0.6 − = 0.748006655882730,
f (r1 ) − f (r0 ) f (0.6) − f (0.5)

f (r2 ) = r2 − cos(r2 ) = 0.014960500949714.

Now, we compute the next approximation r3 ,

f (r2 )(r2 − r1 )
r3 = r2 − = 0.738791967963291, f (r3 ) = −0.000490613128583.
f (r2 ) − f (r1 )

Continuing until satisfying the required accuracy. A summary of the calcu-

lations is given in Table 2.5.

n rn f (rn )
0 0.500000000000000 −0.377582560000000
1 0.600000000000000 −0.225335610000000
2 0.748006655882730 0.014960500949714
3 0.738791967963291 −0.000490613128583
4 0.739084558312839 −0.000000962163319
5 0.739085133252381 0.000000000062293
6 0.739085133215161 0
7 0.739085133215161 0

Table 2.5: Secant Method Solution of Example 14

2.7 Fixed Points and Functional Iteration

Iteration is a fundamental concept in computer sciences and numerical anal-
ysis and is used for solving a wide variety of problems. There is a strong
connection between root-finding problems and fixed point problems, and in
this section, we use fixed point problems to solve root-finding problems.

30 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

Definition 17 (Fixed Point). The number p is called a fixed point of the

function g if p = g(p).

We start by transforming the root-finding problem f (x) = 0 to a fixed

point problem x = g(x) by algebraic manipulations. There are more than
one way of rearranging f (x) = 0 into an equivalent form x = g(x). Note that
if r is a zero of the function f ( i.e. r is a root of the equation f (r) = 0) then
r = g(r) i.e. r is a fixed point of the function g. Conversely, if g has a fixed
point at r then the function f (x) = x − g(x) has a zero at r. Geometrically,
the fixed points of a function y = g(x) are the points of intersection of its
curve with the straight line y = x.

Example 15. Find the fixed points of the function g(x) = 2 − x2 and verify
that they are the solutions to the equation f (x) = x − g(x) = 0.

Solution : The fixed points of g are the points satisfying the fixed point
equation x = g(x), so intersect the graph of y = g(x) with the graph of the
straight line y = x
x = g(x) = 2 − x2 ,
which implies that

−x2 − x + 2 = −(x2 + x − 2) = −(x − 1)(x + 2) = 0.

So, either (x − 1) = 0 implies x = 1 or (x + 2) = 0 implies x = −2. Hence,

the fixed points are x = 1 and x = −2. We notice that these fixed points are
the same the zeros of f (x) = x − g(x) = −(x2 + x − 2) = −(x − 1)(x + 2) = 0.

Definition 18 (Fixed Point Iteration). The iteration rn+1 = g(rn ), n =

0, 1, · · · , obtained by using fixed point formula r = g(r) is called a fixed
point iteration or functional iteration.

Theorem 7. Assume that g ∈ C[a, b] such that

1. If g(x) ∈ [a, b] for all x ∈ [a, b], then g has a at least one fixed point r
in [a, b].

2. If also, g 0 (x) existed and defined on (a, b) and there exists a positive
constant K < 1 such that |g 0 (x)| ≤ K < 1, for all x ∈ (a, b), then g
has a unique fixed point r in [a, b].

Theorem 8 (Fixed Point Theorem). Assume that g ∈ C[a, b] such that

1. g(x) ∈ [a, b] for all x ∈ [a, b], then g has a fixed point r in [a, b].

31 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

2. g 0 (x) existed and defined over (a, b) and there exists a positive constant
K < 1 such that |g 0 (x)| ≤ K < 1, for all x ∈ (a, b), then g has a unique
fixed point r in [a, b].

3. If g satisfies the conditions (1) and (2) , then for any number r0 in [a, b]
the sequence {rn }∞ n=0 of iterations generated by fixed point iteration
rn+1 = g(rn ), n = 0, 1, · · · , converges to the unique fixed point r in
[a, b].

Corollary 1. If g satisfies the hypotheses of Fixed Point Theorem, then the

error bounds for approximating r using rn are given by

|r − rn | ≤ K n max |r − r0 |,

and
Kn
|r − rn | ≤ max |r1 − r0 |, for all n ≥ 1.
1−K
Example 16. Use the fixed point method to find the zero of the function
f (x) = x3 − 3x2 + 2 in [0, 2], start with r0 = 1.5.

Solution: There are many possibilities to write f (x) = 0 as a fixed point

form x = g(x) using mathematical manipulations.
−2
(1) x = g1 (x) = x − x3 + 3x2 − 2. (5) x = g5 (x) =
.
x x−3
 3 1/2
(2) x = g2 (x) = x 3+2 .
 1/3
 3 1/2 (6) x = g6 (x) = 3x2 − 2 .
(3) x = g3 (x) = − x 3+2 .
 1/2  1/2
2 2
(4) x = g4 (x) = 3−x
. (7) x = g7 (x) = 3x − x
.

For example, to obtain g1 (x) just add x to both sides of the equation
−f (x) = 0 and this is the simplest way to write the problem as a fixed point
form

−f (x) = 0, −x3 + 3x2 − 2 = 0, so x = x − x3 + 3x2 − 2 = g1 (x).

Also, g2 (x) and g3 (x) can be obtained as follows:

3 2 2 3 x3 + 2
2
x − 3x + 2 = 0, so 3x = x + 2, and x = ,
3

32 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

implies that
 x3 + 2 1/2  x3 + 2 1/2  x3 + 2 1/2
x=± , so g2 (x) = , and g3 (x) = − .
3 3 3
Note that it is important to check that the fixed point of each derived function
g is a solution to the problem f (x) = 0. For example, because the solution is
positive and lies between 0 and 2, so we choose the positive function g2 (x),
since the negative function g3 (x) is not a choice here. The results are outlined
in Tables 2.6 and 2.7 below.

n g1 (x) g2 (x) g3 (x) g4 (x) g5 (x)

0 1.5 1.5 1.5 1.5 1.5
1 2.875 1.33853153 −1.33853153 1.15470054 0.88888889
2 1.90820313 1.21081272 0 − 0.36i 1.04107393 1.06578947
3 3.88369668 1.12177435 1.01042940 0.97018561
4 −11.44518863 1.06639827 1.00261759 1.01559099
5 1.8788e + 03 1.03484519 1.00065504 0.99238449
6 −6.6210e + 09 1.01787695 1.00016380 1.00385153
7 2.9025e + 29 1.00905819 1.00004095 0.99808533
8 1.00455985 1.00001024 1.00096009
9 1.00228772 1.00000256 0.99952065
10 1.00114582 1.00000064 1.00023985
11 1.00057340 1.00000016 0.99988012
12 1.00000004 1.00005995
13 1.00000001 0.99997003
14 1.00000000 1.00001499
15 0.99999251
16 1.00000375
17 0.99999813
18 1.00000094
19 0.99999953
20 1.00000024
21 0.99999988

Table 2.6: Fixed Point Method Solution of Example 16

33 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

n g6 (x) g7 (x)
0 1.5 1.5
1 1.68098770 1.77951304
2 1.86406700 2.05295790
3 2.03474597 2.27698696
4 2.18422416 2.43979654
5 2.30913228 2.54944095
6 2.40992853 2.61989258
7 2.48919424 2.66388582
8 2.55035309 2.69088731
9 2.59688496 2.70728880
10 2.63192563 2.71718971
11 2.65811433 2.72314423
12 2.67757909 2.72671736
13 2.69198765 2.72885862
14 2.70262171 2.73014079
15 2.71045296 2.73090817
16 2.71621092 2.73136731
17 2.72043954 2.73164198
18 2.72354236 2.73180628
19 2.72581768 2.73190456
20 2.72748542 2.73196334
21 2.72870741 2.73199849

Table 2.7: Fixed Point Method Solution of Example 16

2.8 Acceleration of Iterative Methods

The linear convergence a sequence {rn } to the limit r such as the sequences
of the fixed point iterations can be accelerated by using some techniques such
as Aitken’s ∆2 method (Aitken’s acceleration) and Steffensen’s method. For
more details see references [4, 15] and the references therein.

2.8.1 Modified Newton’s Methods

Newton’s method is a fixed point method since it can be written as

f (x)
x = g(x) = x − ,
f 0 (x)

34 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

and in iterative way

f (rn−1 )
rn = g(rn−1 ) = rn−1 − , n = 1, 2, · · · ,
f 0 (rn−1 )
and this is called Newton-Raphson iteration formula or simply New-
ton’s iteration. The convergence of Newton’s method can be modified to
accelerate its rate of convergence at the root x = r of order M > 1
f (rn−1 )f 0 (rn−1 )
rn = rn−1 − , n = 1, 2, · · · .
(f 0 (rn−1 ))2 − f (rn−1 )f 00 (rn−1 )
This formula is called a modified Newton’s method.

Also, Newton’s method can be accelerated in an another way.

Theorem 9 (Acceleration of Newton’s Iteration). Assume that Newton’s
method produces a linearly convergent sequence to the root x = r of order
M > 1. Then Newton’s iteration formula
M f (rn−1 )
rn = rn−1 − , n = 1, 2, · · · ,
f 0 (rn−1 )

produces a quadratically convergent sequence {rn }∞

n=0 to the root x = r.

Example 17. Show that r = 1 is a double zero (double root) of f (x) =

−x3 + 3x − 2 = 0. Start with r0 = 1.25 as an initial guess of r and compare
the performance of Newton’s method and accelerated Newton’s method for
solving f (x) = 0.
Solution : Since r = 1 is a double root then M = 2, so the accelerated
Newton’s method becomes
3
2f (rn−1 ) 2(−rn−1 + 3rn−1 − 2)
rn = rn−1 − 0
= r n−1 − 2
, n = 1, 2, · · · ,
f (rn−1 ) −3rn−1 +3
or
3
−2rn−1 + 6rn−1 − 4
rn = rn−1 − 2
, n = 1, 2, · · · .
−3rn−1 +3
Start by computing r1

−2r03 + 6r0 − 4 −2(1.25)3 + 6(1.25) − 4

r1 = r0 − = 1.25 − = 1.00925926.
−3r02 + 3 −3(1.25)2 + 3
Table 2.8 compares the performance of both methods.

35 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

n Newton’s Method Accelerated Newton’s Method

0 1.25 1.25
1 1.12962963 1.00925926
2 1.06612990 1.00001422
3 1.03341772 1.00001422
4 1.01680039 1.00000000
5 1.00842352
6 1.00421765
7 1.00211030
8 1.00105552
9 1.00052785
10 1.00026395
11 1.00013198
12 1.00006599
13 1.00003300
14 1.00001650
15 1.00000825
16 1.00000413
17 1.00000207
18 1.00000104
19 1.00000052
20 1.00000026
21 1.00000013
22 1.00000007
23 1.00000004
24 1.00000002
24 1.00000001
25 1.00000001

Table 2.8: Newton’s and Accelerated Newton’s Methods Solutions of Exam-

ple 17

2.9 Computing Roots of Polynomials

Computing roots of polynomials has important applications in different areas
of mathematics and other sciences.

Definition 19 (nth Degree Polynomial). A polynomial of degree n has the

general form

P (x) = an xn + an−1 xn−1 + · · · + a1 x + a0 ,

36 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

where the coefficients ai , i = 0, 1, · · · , n, are real numbers (constants) and

an 6= 0. The nth degree polynomial P (x) is sometimes referred to as Pn (x),
and also named algebraic polynomial.

Note that the zero function P (x) = 0 is a polynomial but has no de-
gree.There are several techniques for finding zeros of polynomials in the liter-
ature such as Müller method, Laguerre’s method, Bairstow method,
Brent’s method and Jenkins-Traub method, and these methods are
beyond the scope of this lecture notes and interested readers can see the
references.

37 Mohammad Sabawi/Numerical Analysis

 CHAPTER 2. NUMERICAL SOLUTIONS OF NONLINEAR
EQUATIONS

Exercises

Exercise 10. Solve Example 10 using the bisection method and compare the
solution with false position method’s solution of the same problem.

Exercise 11. Repeat solving Example 9 using the false position method and
compare the results with the solution of the bisection method for the same
problem.

Exercise 12. Find the solution to the equation ex − x − 1 = 0 accurate to six

decimal places (i.e.  = 0.0000001) using Newton’s and modified Newton’s
methods. Start with r0 = 0.6. Compare the results of both methods.

Exercise 13. Use the secant method to find the solution accurate to within
10−5 to the following problem x sin(x) − 1 = 0, 0 ≤ x ≤ 2.

Exercise 14. Use the fixed point method to locate the root of f (x) = x−e−x =
0, start with an initial guess of x = 0.1.

Exercise 15. Let f (x) = x2 − 5 and r0 = 1.5. Use bisection, false position,
secant, fixed point, Newton’s and modified
√ Newton’s methods to find r7 the
approximation to the positive root r = 5.

Exercise 16. Use modified (accelerated) Newton’s method to solve the equa-
tion x2 − 3x − 1 = 0 in the interval [−1, 1].

38 Mohammad Sabawi/Numerical Analysis

Chapter 3

Solving Systems of Linear

Equations

3.1 Introduction
Many phenomena and relationships in nature and real life applications are
linear, meaning that results and their causes are proportional to each other.
Solving linear algebraic equations is a topic of great importance in numerical
analysis and other scientific disciplines such as engineering and physics. So-
lutions to Many problems reduced to solve a system of linear equations. For
example, in finite element analysis a solution of a partial differential equation
is reduced to solve a system of linear equations.

3.2 Norms of Matrix and Vectors

In error and convergence analyses we need a measure to determine the dis-
tance (difference) between the exact solution and approximate solution or to
determine the differences between consecutive approximations.

Definition 20 (Vector Norm). A vector norm is a real-valued function

k.k : Rn → R satisfies the following conditions:

(i) kxk ≥ 0 for all x ∈ Rn .

(ii) kxk = 0 if and only if x = 0 for all x ∈ Rn .

(iii) kαxk = |α|kxk for all α ∈ R and x ∈ Rn .

(iv) kx + yk ≤ kxk + kyk for all x, y ∈ Rn (Triangle Inequality).

39
 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Definition 21 (l1 Vector Norm). Let x = (x1 , x2 , · · · , xn )0 . Then the l1

norm for the vector x is defined by
n
X
kxk1 = |xi |.
i=1

Definition 22 (Euclidean Vector Norm). Let x = (x1 , x2 , · · · , xn )0 . Then

the Euclidean norm (l2 norm) for the vector x is defined by
n
X 1/2
kxk2 = x2i .
i=1

Definition 23 (Maximum Vector Norm). Let x = (x1 , x2 , · · · , xn )0 . Then

the maximum norm (l∞ norm) for the vector x is defined by

kxk∞ = max |xi |.

1≤i≤n

Remark 5. Note that when n = 1 both norms reduce to the absolute value
function of real numbers.
Example 18. Determine the l1 norm, l2 norm and l∞ norm of the vector
x = (1, 0, −1, 2, 3)0 .
Solution: The required norms of vector x = (1, 0, −1, 2, 3)0 in R5 are:

5
X
kxk1 = |xi | = |x1 | + |x2 | + |x3 | + |x4 | + |x5 | = |1| + |0| + | − 1| + |2| + |3| = 7,
i=1

5
X 1/2  1/2
kxk2 = x2i = x21 + x22 + x23 + x24 + x25
i=1
 1/2  1/2
= (1)2 + (0)2 + (−1)2 + (2)2 + (3)2 = 15 ,

and

kxk∞ = max |xi | = max{|x1 |, |x2 |, |x3 |, |x4 |, |x5 |}

1≤i≤5
= max{|1|, |0|, | − 1|, |2|, |3|} = 3.

40 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Definition 24 (Matrix Norm). A matrix norm is a real-valued function

k.k : Rn×m → R satisfies the following conditions:
(i) kAk ≥ 0 for all A ∈ Rn×m .

(ii) kAk = 0 if and only if A = 0 for all A ∈ Rn×m .

(iii) kαAk = |α|kAk for all α ∈ R and A ∈ Rn×m .

(iv) kA + Bk ≤ kAk + kBk for all A, B ∈ Rn×m (Triangle Inequality).

If matrix norm is related to a vector norm, then we have two additional
properties:
(v) kABk ≤ kAkkBk for all A, B ∈ Rn×m .

(vi) kAxk ≤ kAkkxk for all A ∈ Rn×m and x ∈ Rn .

We give here some equivalent definitions of the matrix norm particularly

when matrix norm is related to the vector norm.
Definition 25 (Subordinate Matrix Norm). Let A is a n × n matrix and
x ∈ Rn , then the subordinate matrix norm is defined by

kAk = sup{kAxk : x ∈ Rn and kxk = 1}.

or, alternatively
kAk = max kAxk.
kxk=1

Definition 26 (Natural Matrix Norm). Let A is a n × n matrix and for any

z
z 6= 0, and x = kzk is the unit vector. Then the natural / reduced matrix
norm is defined by
 z  kAzk
max kAxk = max A = max ,
kxk=1 z6 = 0 kzk z6 = 0 kzk
or, alternatively
kAzk
kAk = max .
z6=0 kzk
Definition 27 (l1 Matrix Norm). Let A is a n×n matrix and x = (x1 , x2 , · · · , xn )0 .
Then the l1 matrix norm is defined by
n
X
kAk1 = max kAxk1 = max |aij |.
kxk1 =1 1≤i≤n
i=1

41 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Definition 28 (Spectral Matrix Norm). Let A is a n × n matrix and x =

(x1 , x2 , · · · , xn )0 . Then the spectral / l2 -matrix norm is defined by
p
kAk2 = max kAxk2 = max |σmax |,
kxk2 =1 1≤i≤n

where σi are the eigenvalues of AT A, which are called the singular values of
A and the largest eigenvalue in absolute value (|σmax |) is called the spectral
radius of A.
Definition 29 (l∞ Matrix Norm). Let A is a n×n matrix and x = (x1 , x2 , · · · , xn )0 .
Then the l∞ (maximum)matrix norm is defined by
n
X
kAk∞ = max kAxk∞ = max |aij |.
kxk∞ =1 1≤i≤n
j=1

Remark 6. Note that kIk = 1.

Example 19. Determine kAk∞ for the matrix
 
1 −1 2
A= 0 5 3 .
−1 6 −4

Solution: For i = 1, we have

3
X
|a1j | = |a11 | + |a12 | + |a13 | = |1| + | − 1| + |2| = 4,
j=1

and for i = 2, we obtain

3
X
|a2j | = |a21 | + |a22 | + |a23 | = |0| + |5| + |3| = 8,
j=1

for i = 3, we get
3
X
|a3j | = |a31 | + |a32 | + |a33 | = | − 1| + |6| + | − 4| = 11.
j=1

Consequently,
3
X
kAk∞ = max |aij | = max{4, 8, 11} = 11.
1≤i≤3
j=1

42 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

3.3 Direct Methods

Direct methods are techniques used for solving and obtaining the exact
solutions (in theory) of linear algebraic equations in a finite number of steps.
The main widely used direct methods are Gaussian elimination method
and Gauss-Jordan method.

Consider the following linear system of dimension n × (n + 1)

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
an1 x1 + an2 x2 + · · · + ann xn = bn .

This system can be written in concise form by using matrix notation as

AX = B as follows:

    
a11 a12 · · · a1n x1 b1
 a21 a12 · · · a1n   x2   b2 
   
..   ..  =  ..  ,
 
 .. .. ..
 . . . .  .   . 
an1 a12 · · · a1n xn bn

where An×n is square matrix and is called a coefficient matrix, Bn×1 is

a column vector known as the right hand side vector and Xn×1 is a column
vector known as unknowns vector. Also, this system can be written as
 
a11 a12 · · · a1n b1
 a21 a12 · · · a1n b2 
[A|B] =  .. ..  ,
 
.. . . ..
 . . . . . 
an1 a12 · · · a1n bn
where [A|B] is called the augmented matrix.

3.3.1 Backward Substitution Method

Backward substitution also called back substitution is an algorithm or
technique used for solving upper-triangular systems which are systems
such that their coefficient matrices are upper-triangular matrices. Assume
that we have the following upper-triangular system

43 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

a11 x1 + a12 x2 + a13 x3 + · · · + a1n−1 xn−1 + a1n xn = b1

a22 x2 + a23 x3 + · · · + a2n−1 xn−1 + a2n xn = b2
a33 x3 + · · · + a3n−1 xn−1 + a3n xn = b3
..
.
an−1n−1 xn−1 + an−1n xn = bn−1
ann xn = bn .

To find a solution to this system we follow the following steps provided

that xrr 6= 0, r = 1, 2, · · · , n:

(1) Solve the last (nth) equation for xn :

bn
xn = .
ann

(2) Substitute xn in the next-to-last ((n − 1)th) equation and solve it for
xn−1 :
bn−1 − an−1n xn
xn−1 = .
an−1n−1

(3) Now, xn and xn−1 are known and can be used to find xn−2 :

bn−2 − an−1n−1 xn−1 − an−1n xn

xn−2 = .
an−2n−2

(4) Continuing in this way until we arrive at the general step:

br − nj=r+1 arj xj
P
xr = , r = n − 1, n − 2, · · · 1.
arr

Example 20. Solve the following linear system using back substitution method

3x1 + 2x2 − x3 + x4 = 10
x2 − x3 + 2x4 =9
3x3 − x4 =1
3x4 =6

44 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Solution: Solve the last equation for x4 to obtain

6
x4 = = 2.
3
Substitute x4 = 2 in the third equation, we have
1 + x4 1+2 3
x3 = = = = 1.
3 3 3
Now, use values x3 = 1 and x4 = 2 in the second equation to find x2

x2 = 9 + x3 − 2x4 = 9 + 1 − 4 = 6.

Finally, solve the first equation for x1 yields

10 − 2x2 + x3 − x4 10 − 12 + 1 − 2 −3
x1 = = = = −1.
3 3 3

3.3.2 Forward Substitution Method

Forward substitution is an algorithm or technique used for solving lower-
triangular systems which are systems such that their coefficient matrices
are lower-triangular matrices.

a11 x1 = b1
a21 x1 + a22 x2 = b2
a31 x1 + a32 x2 + a33 x3 = b3
..
.
an−11 x1 + an−12 x2 + an−13 x3 + · · · + an−1n−1 xn−1 = bn−1
an1 x1 + an2 x2 + an3 x3 + · · · + ann−1 xn−1 + ann xn = bn .

To find a solution to this system we follow the following steps provided

that xrr 6= 0, r = 1, 2, · · · , n:

(1) Solve the first (1st) equation for x1 :

b1
x1 = .
a11

(2) Substitute x1 in the second equation (2nd) equation and solve it for x2 :
b2 − a21 x1
x2 = .
a22

45 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

(3) Now, x1 and x2 are known and can be used to find x3 :

b3 − a31 x1 − a32 x2
x3 = .
a33
(4) Continuing in this way until we arrive at the general step:
br − r−1
P
j=1 arj xj
xr = , r = 2, 3, · · · n.
arr
Example 21. Use the forward substitution method for solving the following
linear system
4x1 =8
2x1 + x2 = −1
x1 − x2 + 5x3 = 0.5
0.1x1 + 2x2 − x3 + 2x4 = 12
,
Solution: Solving the first equation for x1 yields
8
x1 = = 2.
4
Using the value of x1 to find x2
−1 − 2x1 −1 − 2(2)
x2 = = = −2.5.
2 2
Use x1 and x2 to find x3
0.5 − x1 + x2 0.5 − 2 − 2.5 −4
x3 = = = = −0.8.
5 5 5
Finally, solve for x4 to have
12 − 0.1x1 − 2x2 + x3 12 − 0.1(2) − 2(2.5) − 0.8 6
x4 = = = = 3.
2 2 2

3.3.3 Gaussian Elimination Method

Gaussian elimination method is also known as Gauss elimination
method or simply elimination method. It is a direct method used for
solving a system of linear algebraic equations. In this method we transform
the linear system to an equivalent upper or lower triangular system and then
solve it by backward or forward substitution. The process of transforming
the linear system to an equivalent upper or lower triangular system is called
trianguarisation.

46 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Definition 30 (Equivalent Systems). Two linear algebraic systems of dimen-

sion n × n is said to be they are equivalent if they have the same solution
sets.
Definition 31 (Elementary Transformations). The following operations per-
formed on a linear system transform it to an equivalent system:
• Interchanges: Changing the order of any two equations in the system.
• Replacement: Any equation of the system can be replaced by itself
and a nonzero multiple of any other equation in the system.
• Scaling: Multiplying any equation in the system by a nonzero real
constant.
Definition 32 (Elementary Row Operations). The following operations per-
formed on a linear system transform it to an equivalent system:
• Interchanges: Changing the order of any two rows in the matrix.
• Replacement: Any row in the matrix can be replaced by its sum and
a nonzero multiple of any other row in the matrix.
• Scaling: Multiplying any row in the matrix by a nonzero real constant.
Definition 33 (Pivoting). The process of interchanging the rows of the coef-
ficient matrix A containing the element (entry) akk and using it to eliminate
elements ark , r = k + 1, k + 2, · · · n is called pivoting process. The element
akk is called pivotal element and the kth row is called pivotal row.
Example 22. Write the following linear system in the augmented form and
then solve it by using Gauss elimination method.

x1 + 2x2 − x3 + 4x4 = 12
2x1 + x2 + x3 + x4 = 10
−3x1 − x2 + 4x3 + x4 = 2
x1 + x2 − x3 + 3x4 = 6

.
Solution: The augmented matrix is
 
1 2 −1 4 12
 2 1 1 1 10 
 .
 −3 −1 4 1 2 
1 1 −1 3 6

47 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

The first row is the pivotal row, so the pivotal element is a11 = 1 and
is used to eliminate the first column below the diagonal. We will denote
by mr1 to the multiples of the row 1 subtracted from row r for r = 2, 3, 4.
Multiplying the first row by m21 = −2 and add it to the second row to have
 
1 2 −1 4 12
 0 −3 3 −7 −14 
 .
 −3 −1 4 1 2 
1 1 −1 3 6
Now, multiply the first row by m31 = 3 and add it to the third row to
obtain
 
1 2 −1 4 12
 0 −3 3 −7 −14 
 .
 0 5 1 13 38 
1 1 −1 3 6
Multiplying the first row by m41 = −1 ) and adding it to the fourth row
yields
 
1 2 −1 4 12
 0
 −3 3 −7 −14 
.
 0 5 1 13 38 
0 −1 0 −1 −6
Now, the pivotal row is the second row and the pivotal element is a22 =
−3. Multiply the second row by m32 = 53 to have
 
1 2 −1 4 12
 0 −3 3 −7 −14 
 .
 0 0 6 4/3 44/3 
0 −1 0 −1 −6
−1
Multiply the the second row by m42 = 3
and add it to the fourth row
to obtain
 
1 2 −1 4 12
 0
 −3 3 −7 −14  .
 0 0 6 4/3 44/3 
0 0 −1 4/3 −4/3
Now, the pivotal row is the third row and the third element is a33 = 6.
Finally, multiply the third row by m43 = 61 to the fourth row to have

48 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

 
1 2 −1 4 12
 0 −3 3 −7 −14 
 .
 0 0 6 4/3 44/3 
0 0 0 14/9 10/9
Now, note that the coefficient matrix is transformed into an upper trian-
gular matrix and can be solved by backward substitution method. Firstly,
we from the last row we compute

10/9 5
x4 = = .
14/9 7
Use the third row to solve for x3
44/3 − 4/3(5/7) 288/21 16
x3 = = = .
6 6 7
Now, solve the second equation for x2
−14 − 3x3 + 7x4 −14 − 3(16/7) + 7(5/7) 111 37
x2 = = = = .
−3 −3 21 7
Finally, solve the first equation for x1
6
x1 = 12 − 2x2 + x3 − 4x4 = 12 − 2(37/7) + 16/7 − 4(5/7) = .
7
Example 23. Solve the following linear system using Gauss elimination
method by using forward substitution technique

x1 + 2x2 + x3 + 4x4 = 13
2x1 + 0x2 + 4x3 + 3x4 = 28
4x1 + 2x2 + 2x3 + x4 = 20
−3x1 + x2 + 3x3 + 2x4 = 6

.
Solution: We start our solution strategy by transforming this square system
to equivalent lower-triangular system and then solve it by using forward
substitution method. Write the system in augmented matrix form
 
1 2 1 4 13
 2 0 4 3 28 
 4 2 2 1 20  .
 

−3 1 3 2 6

49 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Note that now the pivotal row is the fourth row and the pivotal element
is a44 = 2. Multiply the fourth row by the multiple m14 = −2 and it to the
first row to have
 
7 0 −5 0 1
 2 0 4 3 28 
 4 2 2 1 20  .
 

−3 1 3 2 6
Multiply the fourth row by m24 = −3
2
and add it to the second row to
obtain
 
7 0 −5 0 1
 13/2 −3/2 −1/2 0 19 
 .
 4 2 2 1 20 
−3 1 3 2 6
Now multiply the fourth equation by m34 = −1
2
and add it to the third
row to have
 
7 0 −5 0 1
 13/2 −3/2 −1/2 0 19 
 .
 11/2 3/2 1/2 0 17 
−3 1 3 2 6
The pivotal row now is the third row and the pivotal element is a33 = 1/2.
Add the third row to the second row (i.e. multiply it by m23 = 1) to get
 
7 0 −5 0 1
 12 0 0 0 36 
 11/2 3/2 1/2 0 17  .
 

−3 1 3 2 6
Now
 
12 0 0 0 36
 11/2 3/2 1/2 0 17 
 .
 7 0 −5 0 1 
−3 1 3 2 6
The pivotal row (third row) is used to eliminate elements in the second
1
row and the pivotal element is a33 = −5. Multiply the third row by m23 = 10
to have

50 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

 
12 0 0 0 36
 31/5 3/2 0 0 171/10 
 .
 7 0 −5 0 1 
−3 1 3 2 6
Now, use forward substitution to solve the lower-triangular matrix. solve
the first equation for x1
36
x1 = = 3.
12
Use the equation to find x2

171/10 − (31/5)3
x2 = = −1.
3/2
Now, solve the third equation for x3

1 − 7(3)
x3 = = 4.
−5
Finally, solve the fourth equation for x4

6 − (−3)(3) − 1(−1) − 3(4)

x4 = = 2.
2

3.3.4 Gauss-Jordan Elimination Method

In this method instead of transforming the coefficient matrix into upper or
lower triangular system, we transform the coefficient matrix into diagonal (in
particular identity) matrix using elementary row operations.

Example 24. Solve the following linear system using Gauss-Jordan elimi-
nation method

3x1 + 4x2 + 3x3 = 10

x1 + 5x2 − x3 = 7
6x1 + 3x2 + 7x3 = 15

Solution: Express the system in augmented matrix form

51 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

 
3 4 3 10
 1 5 −1 7  .
6 3 7 15
The pivot row is the first row and the pivot element is a11 = 3. Multiply
it by m11 = 1/3 to get
 
1 4/3 1 10/3
 1 5 −1 7 .
6 3 7 15
Subtract the second equation from the first (i.e. multiply it by m21 = −1)
and multiply the fist equation by m31 = −6 and add it to the third equation
to have
 
1 4/3 1 10/3
 0 −11/3 2 −11/3  .
0 −5 1 −5
Now, the pivot row is the second row and the pivot element a22 = −11/3.
Multiply it by m22 = −3/11 to have
 
1 4/3 1 10/3
 0 1 −6/11 1 .
0 −5 1 −5
Multiply the first and third rows by m12 = −4/3 and m32 = 5 to obtain
 
1 0 19/11 2
 0 1 −6/11 1  .
0 0 −19/11 0
The pivot element now is third row and the pivot element is a33 = −19/11.
Multiply it by m33 = −11/19 to get
 
1 0 19/11 2
 0 1 −6/11 1  .
0 0 1 0
Finally, multiply the third row by m13 = −19/11 and m23 = 6/11 and
add it to the first and second rows to have
 
1 0 0 2
 0 1 0 1 .
0 0 1 0

52 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Hence, we have x1 = 2, x2 = 1 and x3 = 0.

Example 25. Solve the following linear system using Gauss-Jordan elimi-
nation method

−2x1 + x2 + 5x3 = 15
4x1 − 8x2 + x3 = −21
4x1 − x2 + x3 = 7

Solution: Write the system in augmented matrix form

 
−2 1 5 15
 4 −8 1 −21  .
4 −1 1 7
Multiply the first row by m21 = m31 = −2 and it to the second and third
rows respectively, to obtain
 
−2 1 5 15
 0 −6 11 9  .
0 1 11 37
Now, multiply the second row by m12 = m32 = 16 and it to the first and
third rows respectively, to have
 
−2 0 41/6 33/2
 0 −6 11 9 .
0 0 77/6 77/2
Finally, multiply the third row by m13 = −41
77
and m32 = −6
7
and it to the
first and third rows respectively, to obtain
 
−2 0 0 −4
 0 −6 0 −24  ,
0 0 77/6 77/2
implies that

−4 −24 77/2
x1 = = 2, x2 = = 4 and x3 = = 3.
−2 −6 77/6

53 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

3.4 LU and Cholesky Factorisations

In this section we will discuss the triangular factorisations of matrices.

Definition 34 (Positive Definite Matrix). Let An×n be symmetric real matrix

and x ∈ Rn a nonzero vector. Then, A is said to be positive definite
matrix if A = AT and xT Ax > 0 for any x.

Remark 7. Note that the matrix A is nonsingular by definition.

Definition 35 (Triangular Factorisation). Assume that A is a nonsingular

matrix. It said to be A has a triangular factorisation or triangular
decomposition if it can be factorised as a product of unit lower-triangular
matrix L and an upper triangular matrix U :

A = LU.

or in matrix form
    
a11 a12 a13 1 0 0 u11 u12 u13
 a21 a22 a23  =  l21 1 0   0 u22 u23  .
a31 a32 a33 l31 l32 1 0 0 u33

Note that since A is nonsingular matrix this implies that urr 6= 0 for all
r and this is called Doolittle factorisation.

Also, A can be expressed as a product of lower-triangular matrix L and

unit upper triangular matrix U :
    
a11 a12 a13 l11 0 0 1 u12 u13
 a21 a22 a23  =  l21 l22 0   0 1 u23  ,
a31 a32 a33 l31 l32 l33 0 0 1

and this is called Crout factorisation.

To solve the linear system AX = B using LU factorisation, we do the

following two steps:

1. Using forward substitution to solve the the lower-triangular linear sys-

tem LY = B for Y .

2. Using backward substitution to solve the upper-triangular linear system

U X = Y for X.

54 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Example 26. Solve the following linear system using LU (Doolittle) decom-
position

2x1 − 3x2 + x3 = 2
x1 + x2 − x3 = −1
−x1 + x2 − x3 = 0

Solution: Express the system in matrix form

 
2 −3 1 2
 1 1 −1 −1  .
−1 1 −1 0
Factor A as follows:
    
2 −3 1 1 0 0 u11 u12 u13
 1 1 −1  =  l21 1 0   0 u22 u23  .
−1 1 −1 l31 l32 1 0 0 u33

Find the values of the entries of matrices L and U . From the first column
we have

2 = 1u11 =⇒ u11 = 2,
and

1 = l21 u11 = l21 2 =⇒ l21 = 0.5,

finally

−1 = l31 u11 = l31 2 =⇒ l31 = −0.5.

In the second column, we have

−3 = 1u12 =⇒ u12 = −3,

and

1 = l21 u12 + 1u22 = −1.5 + u22 =⇒ u22 = 2.5,

so

1 = l31 u12 + l32 u22 = (−0.5)(−3) + l32 (2.5) =⇒ l32 = −0.2.

55 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Finally, in the third column we have

1 = 1u13 =⇒ u13 = 1,
and

−1 = l21 u13 + 1u23 = 0.5 + u23 =⇒ u23 = −1.5,

finally,

−1 = l31 u13 + l32 u23 + 1u33 = −0.5(1) + (−0.2)(−1.5) + u33 =⇒ u33 = −0.8.

Now, we have the LU factorisation

    
2 −3 1 1 0 0 2 −3 1
A= 1 1 −1  =  0.5 1 0   0 2.5 −1.5  = LU.
−1 1 −1 −0.5 −0.2 1 0 0 −0.8

Now, we have the following lower-triangular linear system LY = B for Y

    
1 0 0 y1 2
 0.5 1 0   y2  =  −1  .
−0.5 −0.2 1 y3 0

Write the system in augmented matrix form

 
1 0 0 2
 0.5 1 0 −1  .
−0.5 −0.2 1 0

Solve this system by forward substitution to have

y1 = 2, y2 = −1 − 0.5(y1 ) = −1 − 0.5(2) = −2,

and

56 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

y3 = 0 + 0.5(y1 ) + 0.2(y2 ) = 0.5(2) + 0.2(−2) = 0.6.

Now, we have the following upper-triangular linear system U X = Y

    
2 −3 1 x1 2
 0 2.5 −1.5   x2  =  −2  .
0 0 −0.8 x3 0.6

Express the system in augmented matrix form

 
2 −3 1 2
 0 2.5 −1.5 −2  .
0 0 −0.8 0.6

Finally, use the values of Y to solve the upper-triangular linear system

U X = Y by back substitution to have

0.6 −3 −2 + 1.5(x3 ) −2 + 1.5(−3/4)

x3 = = , x2 = = = −5/4,
−0.8 4 2.5 2.5
and

2 + 3(x2 ) − 1(x3 ) 2 + 3(−5/4) − (−3/4)

x1 = = = −1/2.
2 2
Definition 36 (Cholesky Factorisation). Let A be a real, symmetric and
positive definite matrix. Then, it can be factored or decomposed in a
unique way A = LLT , in which L is a lower-triangular matrix with a positive
diagonal, and is termed Cholesky factorisation.
    
a11 a12 a13 l11 0 0 l11 l21 l31
 a21 a22 a23  =  l21 l22 0   0 l22 l32  .
a31 a32 a33 l31 l32 l33 0 0 l33
Example 27. (a) Determine the Cholesky decomposition of the matrix
 
2 1 0
A= 1 2 1 
0 1 2

57 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

(b) Then, use the decomposition from part (a) to solve the linear system

2x1 + x2 + 0x3 = −1
x1 + 2x2 + x3 = −4
0x1 + x2 + 2x3 = 2

Solution: Factor A as a product LLT as follows:

    
2 1 0 l11 0 0 l11 l21 l31
 1 2 1  =  l21 l22 0   0 l22 l32  .
0 1 2 l31 l32 l33 0 0 l33

From the first column we obtain

2
√
2 = l11 =⇒ l11 = 2,

√ 1
1 = l21 l11 + l22 (0) =⇒ 1 = l21 2 =⇒ l21 = √ ,
2
√
0 = l31 l11 + l32 (0) + l33 (0) =⇒ 0 = l31 2 =⇒ l31 = 0.

Now, from the second column we have

r
2 2 1 2 3
2 = l21 + l22 =⇒ 2 = + l22 =⇒ l22 = ,
2 2

r √
3 2
1 = l31 l21 + l32 l22 + l33 (0) =⇒ 1 = 0 + l32 =⇒ l32 = √ .
2 3
Finally, from the third column we get

r
2 2 2 2 2 4 2
2 = l31 + l32 + l33 =⇒ 2 = 0 + + l33 =⇒ l33 = =√ .
3 3 3

58 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

3.5 Iterative Methods

Direct methods are more efficient in solving linear systems of small dimen-
sions in less computational cost than iterative methods. For large linear
systems in particular for sparse linear systems iterative methods are more
efficient for solving linear systems in terms of computational cost and effort
compared to direct methods. In this section we will study the most common
and basic iterative methods for solving linear algebraic systems which are
Jacobi method and Gauss-Siedel method.

3.5.1 Jacobi Method

The general form of Jacobi iterative method for solving the ith equation
in the linear system AX = B for unknown xi , i = 1, , · · · , n is:

n 
X aij xjk−1  bi
xki = − + , j 6= i, aii 6= 0, for i = 1, · · · , n, k = 1, · · · , n.
j=1
aii aii

It is also known as Jacobi iterative process or Jacobi iterative tech-

nique
Example 28. Solve the following linear system using Jacobi iterative method

2x1 + x2 + x3 = 0
x1 + 3x2 + x3 = 0.5
x1 + x2 + 2.5x3 = 0

Solution: These equations can be written in the form

−x2 − x3
x1 = ,
2
0.5 − x1 − x3
x2 = ,
3
−x1 − x2
x3 = .
2.5

Writing these equations in iterative form

−xk2 − xk3
xk+1
1 = ,
2
59 Mohammad Sabawi/Numerical Analysis
 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

0.5 − xk1 − xk3

xk+1
2 = ,
3
−xk1 − xk2
xk+1
3 = .
2.5

Let us start with initial guess P0 = (x01 , x02 , x03 ) = (0, 0.1, −0.1). Substi-
tuting these values in the right-hand side of each equation in above to find
the new iterations

−x02 − x03 −0.1 − (−0.1) −0.1 + 0.1

x11 = = = = 0,
2 2 2
1 0.5 − x01 − x03 0.5 − 0 − (−0.1)
x2 = = = 0.2,
3 3
−x01 − x02 −0 − 0.1
x13 = = = −0.04.
2.5 2.5

Now, the new point P1 = (x11 , x12 , x13 ) = (0, 0.2, −0.04) is used in the
Jacobi iterative form to find the next approximation P2

−x12 − x13 −0.2 + 0.04 −0.16

x21 = = = = −0.08,
2 2 2
2 0.5 − x11 − x13 0.5 + 0.04 0.54
x2 = = = = 0.18,
3 3 3
−x11 − x12 −0 − 0.2 −0.2
x23 = = = = −0.08.
2.5 2.5 2.5

The new point P2 = (x21 , x22 , x23 ) = (−0.08, 0.18, −0.08) is closer to the
solution than P0 and P1 and is used to find P3

−x22 − x23 −0.18 + 0.08 −0.1

x31 = = = = −0.05,
2 2 2
0.5 − x21 − x23 0.5 + 0.08 + 0.08 0.66
x32 = = = = 0.22,
3 3 3
−x21 − x22 0.08 − 0.18 −0.1
x33 = = = = −0.04.
2.5 2.5 2.5

60 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

This Jacobi iteration process generates a sequence of points {Pn } =

{(xn1 , xn2 , xn3 )} that converges to the solution (x1 , x2 , x3 ) = (−3/38, 4/19, −1/19) =
(−0.078947368421053, 0.210526315789474, −0.052631578947368). The out-
line of the results is given in the Table 3.1.

n xn1 xn2 xn3

0 0.000000000000000 0.100000000000000 −0.100000000000000
1 0.000000000000000 0.200000000000000 −0.040000000000000
2 −0.080000000000000 0.180000000000000 −0.080000000000000
3 −0.050000000000000 0.220000000000000 −0.040000000000000
4 −0.090000000000000 0.196666666666667 −0.068000000000000
5 −0.064333333333333 0.219333333333333 −0.042666666666667
6 −0.088333333333333 0.202333333333333 −0.062000000000000
7 −0.070166666666667 0.216777777777778 −0.045600000000000
8 −0.085588888888889 0.205255555555556 −0.058644444444444
9 −0.073305555555556 0.214744444444444 −0.047866666666667
10 −0.083438888888889 0.207057407407407 −0.056575555555556
11 −0.075240925925926 0.213338148148148 −0.049447407407407
12 −0.081945370370370 0.208229444444444 −0.055238888888889
13 −0.076495277777778 0.212394753086420 −0.050513629629630
14 −0.080940561728395 0.209002969135802 −0.054359790123457
15 −0.077321589506173 0.211766783950617 −0.051224962962963
16 −0.080270910493827 0.209515517489712 −0.053778077777778
17 −0.077868719855967 0.211349662757202 −0.051697842798354
18 −0.079825909979424 0.209855520884774 −0.053392377160494
19 −0.078231571862140 0.211072762379973 −0.052011844362140
20 −0.079530459008916 0.210081138741427 −0.053136476207133

Table 3.1: Jacobi Iterative Solution of Example 28

3.5.2 Gauss-Siedel Method

An improvement of Jacobi method can be made by using the recent values
xki , i, k = 1, · · · , n, in the calculations once their values are obtained. This
improvement is called Gauss-Sedel iterative method and its general form
for solving the ith equation in the linear system AX = B for unknown
xi , i = 1, , · · · , n is:

61 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

i−1  n 
X aij xkj  X aij xk−1
j
 b
i
xki = − + − + , j 6= i, aii 6= 0,
j=1
aii j=i+1
aii aii

for i = 1, · · · , n, and k = 1, · · · , n.

It is also known as Gauss-Sedel iterative process or Gauss-Sedel

iterative technique
Example 29. Solve the following linear system using Gauss-Siedel iterative
method

2x1 − 4x2 + x3 = −1
x1 + x2 + 6x3 = 1
3x1 + 3x2 + 5x3 = 4

.
Solution: Rearrange the system in above such that the coefficient matrix is
strictly diagonally dominant

3x1 + 3x2 + 5x3 = 4

2x1 − 4x2 + x3 = −1
x1 + x2 + 6x3 = 1

.
These equations can be written in the form
4 − 3x2 − 5x3
x1 = ,
3
−1 − 2x1 − x3 1 + 2x1 + x3
x2 = = ,
−4 4
1 − x1 − x 2
x3 = .
6

This suggests the following Gauss-Siedel iterative process

4 − 3xn2 − 5xn3
xn+1
1 = ,
3
62 Mohammad Sabawi/Numerical Analysis
 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

1 + 2xn+1
1 + xn3
xn+1
2 = ,
4
1 − xn+1
1 − xn+1
2
xn+1
3 = .
6

We start with initial guess P0 = (x01 , x02 , x03 ) = (1, 0.1, −1). Substitute
x02 = 0.1 and x03 = −1 in the first equation and have

4 − 3x02 − 5x03 4 − 3(0.1) − 5(−1) 8.7

x11 = = = = 2.9.
3 3 3
Then, substitute the new value x11 = 2.9 and x03 = −1 into the second
equation to obtain

1 + 2x11 + x03 1 + 2(2.9) + (−1)

x12 = = = 1.45.
4 4
Finally, substitute the new values x11 = 2.9 and x12 = 1.45 in the third
equation and get

1 − x11 − x12 1 − 2.9 − 1.45 −3.35

x13 = = = = −0.558333333333333.
6 6 6

Now, we have the now point P1 = (x11 , x12 , x13 ) = (2.9, 1.45, −0.558333333333333)
is used to find the next approximation P2 .

Substitute x12 = 1.45 and x13 = −0.558333333333333 in the first equation

and get

4 − 3x12 − 5x13 4 − 3(1.45) − 5(−0.558333333333333)

x21 = =
3 3
2.441666666666666
= = 0.813888888888889.
3
Then, substitute the new value x12 = 0.813888888888889 and x13 = −0.558333333333333
into the second equation to obtain

1 + 2x21 + x13 1 + 2(0.813888888888889) + (−0.558333333333333)

x22 = =
4 4
2.069444444444445
= = 0.517361111111111.
4
63 Mohammad Sabawi/Numerical Analysis
 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Finally, substitute the new values x12 = 0.813888888888889 and x22 =

0.517361111111111 in the third equation and get

1 − x21 − x22 1 − 0.813888888888889 − 0.517361111111111

x23 = =
6 6
−0.331250000000000
= = −0.055208333333333.
6

This iteration process generates a sequence of points {Pn } = {(xn1 , xn2 , xn3 )}
that converges to the solution (x1 , x2 , x3 ) = (32/39, 25/39, −1/13) =
(0.820512820512820, 0.641025641025641, −0.076923076923077). The results
are given in the Table 3.2.

n xn1 xn2 xn3

0 1.000000000000000 0.100000000000000 −1.000000000000000
1 2.900000000000000 1.450000000000000 −0.558333333333333
2 0.813888888888889 0.517361111111111 −0.055208333333333
3 0.907986111111111 0.690190972222222 −0.099696180555556
4 0.809302662037037 0.629727285879630 −0.073171657986111
5 0.825558810763889 0.644486490885417 −0.078340883608218
6 0.819414981794946 0.640122269995419 −0.076589541965061
7 0.820860299946349 0.641282764481909 −0.077023844071376
8 0.820423642303718 0.640955860134015 −0.076896583739622
9 0.820538446098689 0.641045077114439 −0.076930587202188
10 0.820505901555874 0.641020303977390 −0.076921034255544
11 0.820514753115183 0.641027117993706 −0.076923645184815
12 0.820512290647653 0.641025234027623 −0.076922920779213
13 0.820512967271065 0.641025753440729 −0.076923120118632
14 0.820512780090325 0.641025610015504 −0.076923065017638
15 0.820512831680559 0.641025649585870 −0.076923080211072
16 0.820512817432583 0.641025638663523 −0.076923076016018
17 0.820512821363173 0.641025641677582 −0.076923077173459
18 0.820512820278183 0.641025640845727 −0.076923076853985
19 0.820512820577581 0.641025641075294 −0.076923076942146
20 0.820512820494949 0.641025641011938 −0.076923076917814

Table 3.2: Gauss-Siedel Iterative Solution of Example 29

64 Mohammad Sabawi/Numerical Analysis

 CHAPTER 3. SOLVING SYSTEMS OF LINEAR EQUATIONS

Exercises

Exercise 17. Solve Example 22 Using Gauss elimination with forward sub-
stitution method. Compare the solution with solution of the same example.
Exercise 18. Solve Example 23 Using Gauss elimination with backward sub-
stitution method. Compare the solution with solution of the same example.
Exercise 19. Repeat Example 28 with Gauss-Siedel iteration. Compute five
iterations and compare them with Jacobi iterations in the same example.
Exercise 20. Redo Example 29 with Jacobi iteration. Compute five itera-
tions and compare them with Gauss-Siedel iterations in the same example.
Exercise 21. Use Gauss elimination with backward substitution method and
three-digit rounding arithmetic to solve the following linear system
x1 + 3x2 + 2x3 = 5
x1 + 2x2 − 3x3 = −2
x1 + 5x2 + 3x3 = 10
.
Exercise 22. (a) Determine the LU factorisation for matrix A in the linear
system AX = B, where
   
−1 1 −2 2
A=  2 −1 1  and B = 1  .

−4 1 −2 4

(b) Then use the factorisation to solve the system

−x1 + x2 − 2x3 = 2
2x1 − x2 + x3 = 1
−4x1 + x2 − 2x3 = 4
.
Exercise 23. Solve the following linear system using Gauss-Jordan elimina-
tion method
−4x1 − x2 − 2x3 = −9
−x1 − x2 + 3x3 = 9
−2x1 − 4x2 + x3 = 5
.

65 Mohammad Sabawi/Numerical Analysis

Chapter 4

Interpolation and
Extrapolation

4.1 Introduction
In applied sciences and engineering, scientists and engineers often collect a
number of data points of different scientific phenomena via experimentation
and sampling. In many cases they need to estimate (interpolate) a function
at a point its functional value is not in the range of the collected data. Inter-
polation is a branch of numerical analysis studies the methods and techniques
of estimating an unknown value of a function at an intermediate value of its
independent variable. Also, interpolation is used to replace a complicated
function by a simpler one.

4.2 Lagrange Interpolation

Suppose that we would like to interpolate an arbitrary function f at a set of
limited points x0 , x1 , · · · , xn . These n+1 points are known as interpolation
nodes in interpolation theory. Firstly, we need to introduce a system of n+1
special polynomials of degree n known as interpolating polynomials or
cardinal polynomials. These polynomials are denoted by `0 , `1 , · · · , `n and
defined using Kronecker delta notation as follows

(
1 if i = j,
`i (xj ) = δij =
0 6 j.
if i =
.

66
 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

Then, we can interpolate the function f by a polynomial Pn of degree n

defined by
n
X
Pn (x) = `i (x)f (xi ),
i=0
this polynomial is called Lagrange polynomial or Lagrange form of
the interpolation polynomial, and it is a linear combination of the cardi-
nal polynomials `i , i = 0, 1, · · · , n. Moreover, it coincides with the function
f at the the nodes xj , j = 0, 1, · · · , n, namely
n
X
Pn (xj ) = `i (xj )f (xj ) = `j (xj )f (xj ) = f (xj ).
i=0
The interpolating polynomials can be expressed as a product of n linear
factors
n
Y (x − xj )
`i (x) = , i = 0, 1, · · · , .
j6=i
(xi − xj )
i.e.

(x − x0 ) (x − x1 ) (x − xi−1 ) (x − xi+1 ) (x − xn )
`i (x) = ··· ··· .
(xi − x0 ) (xi − x1 ) (xi − xi−1 ) (xi − xi+1 ) (xi − xn )
Example 30. Determine the linear Lagrange polynomial that passes through
the points (1, 5) and (4, 2) and use it to interpolate the linear function at
x = 3.
Solution: Writing out the cardinal polynomials

(x − x1 ) (x − 4) −1
`0 (x) = = = (x − 4),
(x0 − x1 ) (1 − 4) 3
and

(x − x0 ) (x − 1) 1
`1 (x) = = = (x − 1).
(x1 − x0 ) (4 − 1) 3

Hence, the Lagrange polynomial is

1
X
P1 (x) = `i (x)f (xi ) = `0 (x)f (x0 ) + `1 (x)f (x1 ) =
i=0
−1 1
(x − 4)(5) + (x − 1)(2) = −x + 6.
3 3
67 Mohammad Sabawi/Numerical Analysis
 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

So,

P1 (3) = −(3) + 6 = 3.
Note that

P1 (1) = −(1) + 6 = 5 = f (1), and P1 (4) = −(4) + 6 = 2 = f (4).

Example 31. Find the Lagrange polynomial that interpolates the following
data

x 1 2 2.5 3 4 5
f (x) 0 5 6.5 7 3 1
Solution: The cardinal polynomials are:

(x − x1 )(x − x2 )(x − x3 )(x − x4 )(x − x5 )

`0 (x) =
(x0 − x1 )(x0 − x2 )(x0 − x3 )(x0 − x4 )(x0 − x5 )
(x − 2)(x − 2.5)(x − 3)(x − 4)(x − 5)
=
(1 − 2)(1 − 2.5)(1 − 3)(1 − 4)(1 − 5)
1 5 11 4 53 3 221 2 505 25
= − x + x − x + x − x+ ,
36 24 18 24 36 3

(x − x0 )(x − x2 )(x − x3 )(x − x4 )(x − x5 )

`1 (x) =
(x1 − x0 )(x1 − x2 )(x1 − x3 )(x1 − x4 )(x1 − x5 )
(x − 1)(x − 2.5)(x − 3)(x − 4)(x − 5)
=
(2 − 1)(2 − 2.5)(2 − 3)(2 − 4)(2 − 5)
1 5 31 4 61 3 509 2 655
= x − x + x − x + x − 50,
3 6 2 6 6

(x − x0 )(x − x1 )(x − x3 )(x − x4 )(x − x5 )

`2 (x) =
(x2 − x0 )(x2 − x1 )(x2 − x3 )(x2 − x4 )(x2 − x5 )
(x − 1)(x − 2)(x − 3)(x − 4)(x − 5)
=
(2.5 − 1)(2.5 − 2)(2.5 − 3)(2.5 − 4)(2.5 − 5)
5000 5 75000 4 425000 3 1125000 2 1370000 600000
= − x + x − x + x − x+ ,
7031 7031 7031 7031 7031 7031
68 Mohammad Sabawi/Numerical Analysis
 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

(x − x0 )(x − x1 )(x − x2 )(x − x4 )(x − x5 )

`3 (x) =
(x3 − x0 )(x3 − x1 )(x3 − x2 )(x3 − x4 )(x3 − x5 )
(x − 1)(x − 2)(x − 2.5)(x − 4)(x − 5)
=
(3 − 1)(3 − 2)(3 − 2.5)(3 − 4)(3 − 5)
1 5 29 4 79 3 401 2 235
= x − x + x − x + x − 50,
2 4 2 4 2

(x − x0 )(x − x1 )(x − x2 )(x − x3 )(x − x5 )

`4 (x) =
(x4 − x0 )(x4 − x1 )(x4 − x2 )(x4 − x3 )(x4 − x5 )
(x − 1)(x − 2)(x − 2.5)(x − 3)(x − 5)
=
(4 − 1)(4 − 2)(4 − 2.5)(4 − 3)(4 − 5)
1 3 137 3 109 2 365 25
= − x5 + x 4 − x + x − x+ ,
9 2 18 6 18 3

(x − x0 )(x − x1 )(x − x2 )(x − x3 )(x − x4 )

`5 (x) =
(x5 − x0 )(x5 − x1 )(x5 − x2 )(x5 − x3 )(x5 − x4 )
(x − 1)(x − 2)(x − 2.5)(x − 3)(x − 4)
=
(5 − 1)(5 − 2)(5 − 2.5)(5 − 3)(5 − 4)
1 5 5 55 149
= x − x4 + x3 − x2 + x − 1.
60 24 24 60

Hence, the Lagrange polynomial is

69 Mohammad Sabawi/Numerical Analysis

 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

6
X
P5 (x) = `i (x)f (xi ) = `i (x)f (xi ) + `i (x)f (xi ) +
i=0
`i (x)f (xi ) + `i (x)f (xi ) + `i (x)f (xi ) + `i (x)f (xi ) =

1 5 11 4 53 3 221 2 505 25
− x + x − x + x − x+ (0) +
36 24 18 24 36 3
1 5 31 4 61 3 509 2 655
x − x + x − x + x − 50 (5) +
3 6 2 6 6
5000 5 75000 4 425000 3 1125000 2 1370000 600000
− x + x − x + x − x+ (6.5) +
7031 7031 7031 7031 7031 7031
1 5 29 4 79 3 401 2 235
x − x + x − x + x − 50 (7) +
2 4 2 4 2
1 3 137 3 109 2 365 25
− x5 + x4 − x + x − x+ (3) +
9 2 18 6 18 3
1 5 5 55 149
x − x4 + x3 − x 2 + x − 1 (1).
60 24 24 60
After some mathematical manipulation, we have

8 8 2722272566677 3 200167100491 2
P5 (x) = − x5 + x4 − x + x
316395 21093 1266637395197952 35184372088832
10969157106929 187476506320011
− x− .
1583296743997440 8796093022208

Note that the Lagrange interpolant is used to interpolate a function at a set

of non-equally spaced points.

4.3 Newton Interpolation

Newton interpolation is used to interpolate a function at a set of given equally
spaced points x0 , x1 , · · · , xn . Before we start we need to define the finite
divided differences.

70 Mohammad Sabawi/Numerical Analysis

 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

4.3.1 Finite Divided Differences

The first finite divided difference of the function f is in general given by

f (xj ) − f (xi )
f [xi , xj ] = .
xj − xi
The second finite divided difference is the difference between the two divided
difference, is represented by

f [xj , xk ] − f [xi , xj ]
f [xi , xj , xk ] = .
xk − xi
Likewise, the nth finite divided difference is expressed by

f [x1 , · · · , xn−1 , xn ] − f [x0 , x1 , · · · , xn−1 ]

f [x0 , x1 , · · · , xn−1 , xn ] = .
xn − x0
Note that the zero-order difference is defined as

f [xi ] = f (xi ) = fi .
Also, observe that

f (xj ) − f (xi ) f (xi ) − f (xj )

f [xi , xj ] = = = f [xj , xi ].
xj − xi xi − xj

The divided differences is summarised in the divided difference table given

below:

xi fi f [xi , xi+1 ] f [xi , xi+1 , xi+2 ] f [xi , xi+1 , xi+2 , xi+3 ]

x0 f0 f [x0 , x1 ] f [x0 , x1 , x2 ] f [x0 , x1 , x2 , x3 ]

x1 f1 f [x1 , x2 ] f [x1 , x2 , x3 ] f [x1 , x2 , x3 , x4 ]
x2 f2 f [x2 , x3 ] f [x2 , x3 , x4 ]
x3 f3 f [x3 , x4 ]
x4 f4

Example 32. Compute the divided differences of the following data

71 Mohammad Sabawi/Numerical Analysis

 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

x 0.5000 1.000 1.500 2.000 2.5000

f (x) 1.1250 3.000 7.3750 15.0000 26.6250
Solution: Using the standard notation the first finite divided differences are:

f (x1 ) − f (x0 ) 3 − 1.1250 1.8750

f [x0 , x1 ] = = = = 3.7500.
x1 − x0 1 − 0.5 0.5
f (x2 ) − f (x1 ) 7.3750 − 3 4.3750
f [x1 , x2 ] = = = = 8.7500.
x2 − x1 1.5 − 1 0.5

f (x3 ) − f (x2 ) 15 − 7.3750 7.6250

f [x2 , x3 ] = = = = 15.2500.
x3 − x 2 2 − 1.5 0.5

f (x4 ) − f (x3 ) 26.6250 − 15 11.6250

f [x3 , x4 ] = = = = 23.2500.
x4 − x 3 2.5 − 2 0.5
Now, using the computed first divided differences, we compute the second
divided differences

f [x1 , x2 ] − f [x0 , x1 ] 8.7500 − 3.7500 5

f [x0 , x1 , x2 ] = = = = 5.000.
x2 − x0 1.5 − 0.5 1

f [x1 , x2 ] − f [x0 , x1 ] 8.7500 − 3.7500 5

f [x0 , x1 , x2 ] = = = = 5.000.
x2 − x0 1.5 − 0.5 1

f [x2 , x3 ] − f [x1 , x2 ] 15.2500 − 8.7500 6.5000

f [x1 , x2 , x3 ] = = = = 6.5000.
x3 − x1 2−1 1

f [x3 , x4 ] − f [x2 , x3 ] 23.2500 − 15.2500 8.0000

f [x2 , x3 , x4 ] = = = = 8.0000.
x4 − x2 2.5 − 1.5 1
Finally, we compute the third divided differences using the computed second
divided differences

f [x1 , x2 , x3 ] − f [x0 , x1 , x2 ] 6.5000 − 5.0000 1.5000

f [x0 , x1 , x2 , x3 ] = = = = 1.0000.
x3 − x0 2 − 0.5 1.5000

f [x2 , x3 , x4 ] − f [x1 , x2 , x3 ] 8.0000 − 6.5000 1.5000

f [x1 , x2 , x3 , x4 ] = = = = 1.0000.
x4 − x1 2.5 − 1 1.5000
The results are outlined in the following table

72 Mohammad Sabawi/Numerical Analysis

 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

xi fi f [xi , xi+1 ] f [xi , xi+1 , xi+2 ] f [xi , xi+1 , xi+2 , xi+3 ]

0.5000 1.1250 3.7500 5.000 1.0000

1.000 3.000 8.7500 6.5000 1.0000
1.5000 7.3750 15.2500 8.0000
2.000 15.0000 23.2500
2.5000 26.6250

4.3.2 Newton’s Interpolation Divided Difference For-

mula
The general form of Newton’s interpolation polynomial of order n for n + 1
data points is

Pn (x) = d0 + d1 (x − x0 ) + d2 (x − x0 )(x − x1 ) + d3 (x − x0 )(x − x1 )(x − x2 ) +

· · · + dn (x − x0 )(x − x1 )(x − x2 ) · · · (x − xn−1 ),

where

d0 = f [x0 ],

d1 = f [x0 , x1 ],

d2 = f [x0 , x1 , x2 ],

d3 = f [x0 , x1 , x2 , x3 ],

..
.

dn = f [x0 , x1 , · · · , xn ].

Example 33. Use the data from Example 32 to construct Newton’s inter-
polation divided difference formula, and use it to evaluate f (0), f (3), and
f (3.25).

73 Mohammad Sabawi/Numerical Analysis

 CHAPTER 4. INTERPOLATION AND EXTRAPOLATION

Solution: The Newton’s polynomial of third order for the data in the table
above is

P3 (x) = d0 + d1 (x − x0 ) + d2 (x − x0 )(x − x1 ) + d3 (x − x0 )(x − x1 )(x − x2 ) =

1.1250 + 3.75(x − 0.5) + 5(x − 0.5)(x − 1) + (x − 0.5)(x − 1)(x − 1.5).

After some mathematical manipulation, we have

P3 (x) = x3 + 2x2 − x + 1.

Hence,

f (0) = P3 (0) = 1, f (3) = P3 (3) = 43, f (3.25) = P3 (3.25) = 53.2031.

74 Mohammad Sabawi/Numerical Analysis

Bibliography

[1] A. Householder, Numerical Treatment of a Single Nonlinear

Equation, McGraw-Hill, New York, 1970.
[2] A. Ralston and P. Rabinowitz, A First Course in Numerical
Analysis, 2nd Edition, McGraw-Hill, New York, 1978.
[3] C. Gerald and P. Wheatley, Applied Numerical Analysis, 2nd
Edition, Addison-Wesley, Reading, Mass, MA, 1987.
[4] C. Gerald and P. Wheatley, Applied Numerical Analysis,
7th International Edition, Pearson/Addison-Wesley, Boston,
2004.
[5] D. Kincaid and W. Cheney, Numerical Mathematics and
Computing, 4th Edition, Brooks/Cole Publishing Company,
1999.
[6] D. Kincaid and W. Cheney, Numerical Mathematics and
Computing, 5th Edition, Brooks/Cole Publishing Company,
2004.
[7] D. Kincaid and W. Cheney, Numerical Mathematics and
Computing, 7th International Edition, Brooks/Cole Publish-
ing Company, 2013.
[8] F. Hildebrand, Introduction to Numerical Analysis, 2nd Edi-
tion, McGraw-Hill, New York, 1974.

75
 BIBLIOGRAPHY

[9] G. Phillips and P. Taylor, Theory and Applications of Nu-

merical Analysis, Academic Press, New York, 1973.
[10] K. Atkinson, Elementary Numerical Analysis, John Wiley
& Sons, 1985.
[11] L. Hageman and D. Young, Applied Iterative Methods, Aca-
demic Press, New York, 1981.
[12] M. Heath, Scientific Computing: An Introductory Survey,
2nd Edition, McGraw-Hill, New York, 2002.
[13] P. Linear, Theoretical Numerical Analysis, John Wiley &
Sons, New York, 1979.
[14] R. Burden and J. Faires, Numerical Analysis, 3rd Edition,
Mass Prindle, Weber & Schmidt, 1985.
[15] R. Burden and J. Faires, Numerical Analysis, 9th Edition,
Brooks/Cole, 2011.
[16] S. Conte and C. deBoor, Elementary Numerical Analysis,
3rd Edition, McGraw-Hill, New York, 1980.
[17] W. Press, S. Teukolsky, W. Vetterling and B. Flannery,
Numerical Recipes: The Art of Scientific Computing, 3rd
Edition, Cambridge University Press, 2007.

76 Mohammad Sabawi/Numerical Analysis