Computational Methods

June 2, 2018

1 Floating point representation of numbers

Consider the real number,

427.325 = 4 × 102 + 2 × 101 + 7 × 100 + 3 × 10−1 + 2 × 10−2 + 5 × 10−3

The round off error if the ’n+1’ th digit is removed, is < 1 × 10−n . It is < 0.5 × 10−n if the
’n’ th digit is rounded off.
In general, a nonzero real number x can be represented in the form x = ±r × 10n . where
1
10
≤ r < 1 and n is an integer.
similarly, in binary system, a nonzero real number x can be represented in the form x =
±q × 2m . where 12 ≤ q < 1 and m is an integer.
q is the mantissa and m is the exponent.
A real number expressed as in Equation () is said to be in normalized floating- floating point
form.
Example: Consider a 32 bit word.
2 sign bits (each one for mantissa and exponent) 7 bits for exponent. 23 bits for mantissa.
Since first digit is always 1, effectively 24 bits.
Maximum range is between 2−127 (10−38 ) to 2127 (1038 ) since |m| = (27 − 1) = 127
In this machine numbers have a limited precision of roughly seven decimal places, since the
least significant bit in the mantissa represents units of 2 24 (or approximately 10 7 ).
Machine overflow and under flow
x > 224 overflow
x < 2−24 underflow
Machine epsilon

1
For a number system and a rounding procedure, machine epsilon is the maximum relative
error of the chosen rounding procedure.
x−x∗
Fractional error x
≤ 2−24 .
x = x∗ (1 + δ)
epsilon = 1.0;
while (1.0 + 0.5 * epsilon) 1.0: epsilon = 0.5 * epsilon
Chopping: Store x as c, where |c| < |x| and no machine number lies between c and x.
Rounding : Store x as r, where r is the machine number closest to x. IEEE standard
arithmetic uses rounding .

2 Numerical methods
mathematically equivalent formulas are not numerically equivalent
Numerical method: A procedure used to solve a mathematical problem through the use
of numbers , an arithmetical approach. Approximate solutions to complex problems in terms
of simple arithmetic operations. A special care or procedure should be adopted if problem
is illconditioned or unstable. General strategyis to replace difficult problem by easier one
having same or closely related solution.
infinite by finite
differential by algebraic
nonlinear by linear
complicated by simple

Numerical algorithm: A precise step by step description of the implementation of a

numerical method.
Numerical Analysis : Checking the range of validity of the numerical method chosen.
Error analysis of the approximation made. Choice of numerical methods suitable for a given
problem.
Typically, scientific computing require massive amounts of calculations (usually floating-
point) and are often executed on supercomputers or distributed computing platforms.
Errors in computation The errors introduced in the computation at various stages are
important. Typically sources (apart from blunders) are ,
Before Computation
Modeling error
Data or mesurement error/uncertainties
Algorithmic error
Error from privious computation

2
input data error: systematic and random errors

During Computation

1) rounding error : finite number of digits of storage

2) truncation error: Limiting process is truncated. infinite series is represented by finite

number of terms.

3 Big-O Notation
A function f(x) is order g(x) denoted as f (x) = O(g(x)) if,

| f (x) |< C | g(x) |; ∃C > 0 and ∀x > x0

In numerical analysis,

|f (x)| < C | g(x) |; ∃C > 0 and ∀x < x0

4 Approximation of derivatives

0 f 00 (x) 2
f (x + ∆x) = f (x) + f (x)∆x + ∆x + O(∆x3 )
2!

xj + ∆x = xj+1 ; f (xj ) = fj and∆x = h

Forward difference approximation,

fj00 2 2
fj+1 = fj + fj0 h + h + O(h3 ) (1)
2!

divide by h and re-arrange,

fj+1 − fj
= fj0 + O(h)
h
truncation error is O(h). What this means?

3
backward difference approximation,

fj00 2 2
fj−1 = fj − fj0 h + h − O(h3 )
2!

divide by h and re-arrange

fj − fj−1
fj0 = + O(h)
h
To get the expression for central difference, subtract equation (3) from (1), to get,

fj+1 − fj−1 = 2fj0 h + O(h3 ) (2)

(3)

fj+1 − fj−1
fj0 = + O(h2 )
2h
If we refine the mesh by factor 2, we expect the truncation error to reduce by factor 4.
Here is fourth order formula, from function evaluation at additional points.

fj−2 − 8fj−1 + 8fj+1 − fj+2

fj0 = + O(h4 )
12h
This has problem near boundaries.
We can get an expression for second derivative by adding equation (1) and (3), to get,

fj+1 − 2fj + fj−1

fj00 = + O(h2 )
h2

5 First order ODE-Basic Numerical Methods

The objective is to approximate the solution to the problem,
dy
= f (y, t)
dt

t ≥ t0 ; y(t0 ) = y0

Here , f is sufficiently well behaved function that maps [t0 , ∞) × R → R.

4
`[2.0,2.0]

we require f to obey the Lipschitz condition,

kf (x, t) − f (y, t)k ≤ λ k x − y k

Analytic functions

6 Properties of numerical methods

The following criteria are crucial to the performance of a nume rical algorithm:
1. Consistency The discretization of a PDE should become exact as themesh size tends
to zero (truncation error should vanish)
2. Stability Numerical errors which are generated during the solution of discretized equa-
tions should not be magnified
3. Convergence The numerical solution should approach the exact solution of the PDE
and converge to it as the mesh size tends to zero
4. Conservation Underlying conservation laws should be res pected at the discrete level
(artificial sources/sinks are to be avoided)
5. Boundedness Quantities like densities, temperatures, co ncentrations etc. should re-
main nonnegative and free of spurious wiggles
These properties must be verified for the numerical scheme.

6.1 Consistency, stability and convergence

Consistency:
differential equation ⇔ discretized equation
Definition: Truncation error should vanish as step size tend to zero.
Convergence:
Solution of the differential equation ⇔ numerical solution of the discretized equation
Definition: A numerical scheme is said to be convergent if it produces the exact solution of
the underlying PDE in the limit h → 0 and t → 0. Lax equivalence theorem: stability +
consistency = convergence

5
6.2 Euler’s methods
Z t
y(t) = y(t0 ) + f (y, τ )dτ ≈ y0 + f (y0 , t0 )(t − t0 )
t0

with tn+1 − tn = h, and y(tn + h) = yn+1 ,

yn+1 ≈ yn + hf (yn , tn )

is the celebrated Euler’s method. It is the corner stone of numerical analysis of differential
quations.
implicit and explicit Euler’s methods

dy h2 d2 y
y(t + h) = y(t) + h + + O(h3 )
dt 2! dt2

yn+1 = yn + hf (yn , tn ) + O(h2 )

Local error is O(h2 ) and global error is O(h), it is called a first order method.

dy h2 d2 y
y(t − h) = y(t) − h + − O(h3 )
dt 2! dt2

dy
y(t) = y(t − h) + h + O(h2 )
dt

yn = yn−1 + hf (yn , tn ) + O(h2 )

Implicit Euler’s method.

6.3 Stability of Euler’s methods

ẏ(t) = f (y, t)

If f (ye ) = 0, then ye is an equilibrium point.

Stability of the solution:
Bounded input → bounded output.
Perturbations from equilibrium remain bounded.

6
Assymtotic stability.

∀ > 0; ∃δ() > 0

such that,
k y(t) − ye k< if k y(0) − ye k< δ()

assymptotic stability
Limt → ∞ k y(t) − ye k= 0

Linearize with respect to the equilibrium point ye ,

d
(ye + y) = f (ye + y, t)
dt

d ∂f
y= |(y ) y + O(y 2 )
dt ∂y e

with f 0 (ye ) = λ,

dy
= λy
dt
where ∈ C is a system parameter which represent the eigenvalues of linear systems of
differential equations. The equation is stable if Real() ≤ 0. In this case the solution is
exponentially decaying. limt → ∞y(t) = 0).
Under what conditions the numerical solution yn also decays, limn → yn = 0 ?
Let us examine the explicit Euler’s method with f (y, t) replaced by λy.

yn+1 = yn + hλyn

yn+1 = yn (1 + λh)

yn+1 = yn−1 (1 + λh)2 = y0 (1 + λh)n+1

This implies that the solution is stable, when | 1 + λh |< 1.

2
−2 < λh < 0 implies that for real λ < 0, h < λ

In the complex plane ((1 + λr h)2 + (λi h)2 < 1), implies that the region of convergence is a
unit disc in the λh plane centered at -1.

7
Therefore, we can conclude that while the anlytical solution is stable for all values of λ in
the left half of complex plane, the numerical solution is stable in a much smaller region only.
This is known as conditional stability.
Let us examine the implicit Euler’s method with f (y, t) replaced by λy.

yn+1 = yn + hλyn+1

yn
yn+1 =
(1 − λh)

yn−1 y0
yn+1 = =
(1 − λh)2 (1 − λh)n+1

this implies that, the region of convergence is defined by,

| 1 − λh |> 1

for λ < 0 this implies that the method is convergent for all λ > 0.
Or in the complex plane (λh) , it convergent outside the unit disc (1 − λr h)2 + (λi h)2 ≤ 1
centered at 1.
Implementation of implicit methods

yi+1 = yi + hf (ti+1 , yi+1 )

The following are some of the ways to solve foryi+1 :

1. Fixed point iteration
2. Regula Falsi (method of false position)
3. Bisection method
4. Newton’s method
Fixed point iteration:

F (x) = 0; (4)
rewriteF (x) = x − f (x)x = f (x) (5)
xn+1 = f (xn ) (6)
(7)

convergence | f 0 (x) |< 1

8
Fixed point iteration:
k is the iteration counter, i the integration step counter
k+1 k
yi+1 = yi + hf (ti+1 , yi+1 )

Newton iteration:
yi+1 = yi + hf (ti+1 , yi+1 )
this implies,
yi+1 − yi − hf (ti+1 , yi+1 ) = 0 i.e., F (yi+1 ) = 0

F 0 (yi+1 )∆yi+1 = −F (yi+1 )

k+1 k
yi+1 = yi+1 + ∆yi+1

6.4 Stiff ODE

If the differential equation is

ˆ nonstiff: explicit Euler
ˆ nonstiff: implicit Euler with fixed point iteration
ˆ stiff: implicit Euler with Newton iteration

6.5 Multi step methods

Leap frog method:

yn+1 = yn−1 + 2hf (yn , tn ) + O(h3 ) (8)

6.6 Runge-Kutta Methods

6.7 Pade Approximation

7 Numerical Solution of PDE

7.1 Semi-discretisation

Diffusion equation for the function u(x, t),

9
 ∂u ∂ 2u
=α 2
∂x ∂x

∆x2 ∆x3
u(x + ∆x) = u(x) + u0 (x)∆x + u00 (x) + u000 (x) + O(∆x4 ) (9)
2 6
∆x2 ∆x3
u(x − ∆x) = u(x) − u0 (x)∆x + u00 (x) − u000 (x) + O(∆x4 ) (10)
2 6
Adding the above two eq.

u(x + ∆x) − 2u(x) + u(x − ∆x)

u00 (x) = + O(∆x3 )
∆x2
Applying the above 2nd order finite difference scheme to the space variable,

du u(x + ∆x) − 2u(x) + u(x − ∆x)

'
dt ∆x2
Let the interval to be discretised is [0-L]. If the interval is divided into N meshes, there
will be a total of N + 1 points with 2 boundary points and N − 1 interior points.
Let the first point be 0 and the last point N 0,1,2,3,4....N; There are N intervals, N+1
points Interior points are 1 to N-1.
This can be written in a matrix ((N − 1) × (N − 1)) form

du
= Au(t)
dt
α
where A = ∆x2
[1, −2, 1], is the tridiagonal matrix.
The solution in terms of the eigen values λi and eigen vectors vi of matrix A is,

X
u(t) = ci vi eλi t
i

Full discretisation, explicit method.

du u(t + ∆t) − u(t)

=
dt ∆t

u(t + ∆t) = u(t) + ∆tAu(t)

Implicit method,
with h = ∆t,

10
 α∆t
u(t + ∆t) = u(t) + Au(t)
∆x2

αh
u(t + h) = u(t) + Au(t + h)
∆x2

(I − Ah)u(t + h) = u(t) + boundary terms

The general form of the above set of equations is,

Ax = b

8 Linear Equations

8.1 Sparse Matrix Representation

PDE discretisation or linearisation leads to ,

Ax = b

where, A is (n,n) matrix, x and b are column vectors.

when A is non-singular, it is invertible. There exists an unique inverse.
Under determined systems
Over determined systems
Least squares method

AT (Ax̄ − b) = 0
x̄ is the best fit estimate (x̄ lies in the column space of A , which is subspace of Rrows .
proof :
BL A BR = BL I = I BR

Solving x = A−1 is expensive. It takes O(n3 ) steps.

exactly, 2n3 + O(n2 )
LU decomposition
Let,
A = LU

11
 LU x = b

first solve L y = b by forward substituton and then solve U x = y by back substitution.

LU factorisation requires 2/3n3 + O(n2 ) operations. computing x from factors requires
2n2 + O(n). LU is 3 times faster.

9 Direct methods
Thomas tridiagonal matrix algorithm (TDMA)
Gaussian elimination algorithm
Rule of thumb: direct methods for 1D and 2D and iterative methods for 3D.

10 Iterative Methods
Performance better with round off errors and computationally efficient.

10.1 Stationary Iteration

xk+1 = f (xk )

For convergence, f needs to be a contraction map.

df
<1
dx

Ax = b

xk+1 = Lxk + b

x̄ = Lx̄ + b

equation 8.1 - 8.1,

rk+1 = Lrk

rk+1 = Lk r1

12
 Lim k
r = 0  Lk = 0
k→∞
In case of a matrix operator, the spectral radius ρ(A) < 1.
where, ρ(L) = max |λ|

10.2 Point Jacobi Method

A = (D + L + U )
Ax = b
Dx = b − (L + U )x
xk+1 = D−1 [b − (L + U )xk ]

For the elements,

1 X
xk+1
i = [bi − aij xkj ]
aii j6=i

10.3 Gauss Seidel Method

make use of new x.

Dxk+1 = b − Lxk+1 − U xk

The matrix elements are,

i−1 n
1 X X
xk+1
i = [bi − k+1
aij xj − aij xkj ]
aii j=1 j=i+1

10.4 Succesive Over-relaxation Method

xk+1 = f (xk )

xk+1 = (1 − w)xk + wf (xk )

i−1 n
w X X
xk+1
i = (1 − w)xki + [bi − aij xk+1
j − aij xkj ]
aii j=1 j=i+1

13
In matrix form,
Ax = b

A=D+L+U

(D + L + U )x = wb + (1 − w)b

[D + w(L + U ) + (1 − w)(L + U )]x = wb + (1 − w)(D + L + U )x

[D + w(L + U )]x = wb + (1 − w)Dx

(D + w(L + U )x = wb + (1 − w)Dx

(D + wL)x = wb + [(1 − w)D − wU ]x

(D + wL)xk+1 = wb + [(1 − w)D − wU ]xk

Dxk+1 = (1 − w)Dxk − w(b + Lxk+1 + U xk )

xk+1 = (1 − w)xk − D−1 w(b + Lxk+1 + U xk )

11 Advanced Methods

11.1 Krylov Space Methods

Conjugate Gradient method

Generalised Minimal Residual Method (GMRES)

12 geometric numerical integration

14