Numerical Integration of ODEs

December 5, 2016

Numerical Integration of ODEs

Single Step Methods : Taylor series

The numerical solution of the ODE

y 0 = f (x, y ) with y (x0 ) = y0 (1)

at a given point x can be found by finding the coefficients of the Taylor

series expansion about the initial point x0 . The actual solution will be:

h2 00
y (x) = y (x0 + h) = y (x0 ) + hy 0 (x0 ) + y (x0 ) + (2)
2
where h = x x0 .
Thus we need to provide the coefficients:

y 00 = f 0 (x, y ) , y 000 = f 00 (x, y ) , . . . (3)

Numerical Integration of ODEs

Taylor series : Example
We will find the y (1) for the ODE
y0 = x + y with y (0) = 1 (4)

y 0 (x0 ) = y 0 (0) = 0 + 1 = 1
y 00 = 1 + y 0 = 1 + x + y = 1 + y 0
y 00 (x0 ) = 1 + y 0 (0) = 2
y 000 = 1 + y0
000
y (x0 ) = 1 + y 0 (0) = 2
y (4) = 1 + y0
y (4) (x0 ) = 1 + y 0 (0) = 2
Thus
h3 h4
y (0 + h) = 1 + h + h2 + + + (5)
3 12
and the error will be
y (5) () 5
E= h for 0<<h (6)
5!
Numerical Integration of ODEs
Taylor series : Example

x y Analytic Error Error *

0 1 1
0.1 1.110342 1.110342 1.7 107
0.2 1.24280 1.24281 5.5 106 3.7 107
0.3 1.39968 1.39972 4.3 105 6.2 107
0.4 1.383467 1.383649 1.8 104 9.1 107
:
1.0 3.416667 3.436564 2 102 4.2 106

Numerical Integration of ODEs

Single Step Methods : Euler-Heun
EULER We keep only the 1st
term of the Taylor series:

y (x0 + h) = y (x0 ) + hy 0 (x0 ) (7)

with and obvious error:

y 00 () 2
E= h for 0<<h
2

EULER - HEUN : This is a predictor - corrector method

1st step : yn+1 = yn + hyn0 + O(h2 ) (8)

h 0 0
+ O(h3 )


2nd step : yn+1 = yn + yn + yn+1 (9)
2
Can you explain the smaller error?
Numerical Integration of ODEs
Propagation of Errors
If we consider the first order ODE y 0 = f (x, y ), with y (x0 ) = y0 then if
Yn is the calculated value at xn and yn the true value at xn the error at
xn will be
n = yn Yn (10)
Lets try to study the propagation of the error for the Euler method:

n+1 = yn+1 Yn+1 = yn + h f (xn , yn ) Yn h f (xn , Yn )

(f (xn , yn ) f (xn , Yn ))
= n + h n = n [1 + h fy (xn , yn )]
yn Yn
1 f
= (1 + h k) n + h2 y 00 (n ) where k = . (11)
2 y

I.e. the propagation of the error is linear. If |1 + hk| 1 the error

increases while |1 + hk| 1 decreases.
This leads to the necessary condition for absolute convergence:
f
2 < hk < 0 or < 0. (12)
y

Numerical Integration of ODEs

Convergence
Lets assume an ODE of the form
dy
= Ay (13)
dx
which has an obvious solution of the form y = e Ax .
Eulers method gives the approximate solution via the recurrence relation:

yn+1 = yn + hAyn = (1 + hA)yn for n = 0, 1, 2, ... (14)

Thus if we use this relation n-times we will get:

yn+1 = (1 + hA)n+1 y0 for n = 0, 1, 2, ... (15)

But for small h we know that 1 + hA e hA thus

yn+1 = (1 + hA)n+1 y0 e (n+1)hA y0 = e (xn+1 x0 )A y0 = e Axn+1 (16)

where h = (xn+1 x0 )/(n + 1).

This means that the numerical solution for small h converges to the
analytic solution: y = e Ax .
Numerical Integration of ODEs
Runge - Kutta Methods
Lets assume the ODE
dy
= f (x, y ) (17)
dx
A possible recurrence relation for the estimation of yn can be of the form
yn+1 = yn + ak1 + bk2 (18)
where
k1 = hf (xn , yn ) (19)
k2 = hf (xn + Ah, yn + Bk1 ) (20)
Lets try to find the equivalent Taylor expansion we get
h2 0
yn+1 = yn + hf (xn , yn ) + f (xn , yn ) + . . . (21)
2
But since
df dy
= fx + fy = fx + fy f (22)
dx dx
we get
h2
yn+1 = yn + hfn + (fx + ffy )x=xn (23)
2
Numerical Integration of ODEs
By comparing with (18), which can be written as:

yn+1 = yn + ahf (xn , yn ) + bhf [xn + Ah, yn + Bhf (xn , yn )] (24)

1
we can get the Taylor series expansion

yn+1 = yn + h (a + b) fn + h2 (A b fx + B b f fy )n (25)

and we come to the following relations for the arbitrary constants a, b, A

and B:
1 1
a + b = 1, Ab = and B b = (26)
2 2
By setting a = 32 , 12 , 45 . . . we can estimate the other 3 unknowns.
here we choose a = 12 then b = 12 and A = B = 1, i.e.

1
yn+1 = yn + (k1 + k2 ) (27)
2
where
k1 = hf (xn , yn ) , k2 = hf (xn + h, yn + k1 ) (28)
which is Euler-Heuns method.
1
Here we use: f (xn + Ah, yn + Bhf (xn , yn )) (f + fx Ah + fy Bhf )x=xn
Numerical Integration of ODEs
4th order Runge - Kutta Method
If we repeat the same procedure for a Taylor series up to h4 , we will
create a system of 11 equations with 13 unknowns.
Then with the appropriate choice of two of them we come to a
recurrence relation of the form :

1
yn+1 = yn + (k1 + 2k2 + 2k3 + k4 ) (29)
6
where

k1 = hf (xn , yn ) (30)
 
1 1
k2 = hf xn + h, yn + k1 (31)
2 2
 
1 1
k3 = hf xn + h, yn + k2 (32)
2 2
k4 = hf (xn + h, yn + k3 ) (33)

This the 4th order Runge - Kutta with local error E O(h5 ) and global
error after n steps E O(h4 ).
Numerical Integration of ODEs
4th order Runge - Kutta - Merson Method
In this method the order is not given by the number of ks, but by the
global error:

k1 = hf (xn , yn ) (34)
 
1 1
k2 = hf xn + h, yn + k1 (35)
3 3
 
1 1 1
k3 = hf xn + h, yn + k1 + k2 (36)
2 6 6
 
1 1 3
k4 = hf xn + h, yn + k1 + k3 (37)
2 8 8
 
1 3
k5 = hf xn + h, yn + k1 k3 + 2k4 (38)
2 3

1
yn+1 = yn + (k1 + 4k4 + k5 ) + O(h5 ) (39)
6
ERROR:
1
E= (2k1 9k3 + 8k4 k5 ) (40)
30
Numerical Integration of ODEs
Runge - Kutta - Fehlberg Method

k1 = hf (xn , yn ) (41)
 
1 1
k2 = hf xn + h, yn + k1 (42)
4 4
 
3 3 9
k3 = hf xn + h, yn + k1 + k2 (43)
8 32 32
 
12 1932 7200 7296
k4 = hf xn + h, yn + k1 k2 + k3 (44)
13 2197 2197 2197
 
439 3680 845
k5 = hf xn + h, yn + k1 8k2 + k3 k4 (45)
216 513 4104
 
1 8 3544 1859 11
k6 = hf xn + h, yn k1 + 2k2 k3 + k4 k5(46)
2 27 2565 4104 40
Then a first estimation for yn+1 is:
 
25 1408 2197 1
y n+1 = yn + k1 + k3 + k4 k5 (47)
216 2565 4104 5
here the local error is h5 .
Numerical Integration of ODEs
The next step will include k6 and gives:
 
16 6656 28561 9 2
yn+1 = yn + k1 + k3 + k4 k5 + k6 (48)
135 12825 56430 50 55

with local error h6 and global h5 .

A formula for the estimation of error of the Runge - Kutta - Fehlberg
method is
1 128 2197 1 2
E= k1 k3 k4 + k5 + k6 (49)
360 4275 7524 50 55
Since the k1 , k2 , ..., k6 are known in every step we can always test the
accuracy of the method and if is worst than the demanded we subdivide
the step h.

Numerical Integration of ODEs

Multistep Methods : Adams

The principle behind a multistep method is to utilize the past values of

y and/or y 0 to construct a polynomial that approximates the derivative
function, and extrapolate this into the next interval.
The number of past points that are used sets the degree of the
polynomial and is therefore responsible for the truncation error.
The order of the method is equal to the power of h in the global error
term of the formula, which is also equal to one more than the degree of
the polynomial.
We will write the ODE in the form:
dy
= f (x, y ) then dy = f (x, y ) dx (50)
dx
and then we integrate between xn and xn+1 :
Z xn+1 Z xn+1
dy = yn+1 yn = f (x, y ) dx (51)
xn xn

Numerical Integration of ODEs

Multistep Methods : Adams
We can use the trapezoidal rule for the integration
Z xn+1
h h3
yn+1 yn = f (x, y ) dx = [f (xn , y (xn )) + f (xn + h, y (xn + h))] + f 00 ()
xn 2 12
(52)
thus the new scheme will be:
h 0 0

h3 000
yn+1 = yn + yn + yn+1 y () (53)
2 12
alternatively:

1 h3
yn+1 = yn + hyn0 + yn00 h2 + y 000 ()
2 6
0
yn0 h3

0 1 y n+1 h 000
= yn + hyn + y () h2 + y 000 ()
2 h 2 6
3
1 0 0

h 000
= yn + yn + yn+1 y () (54)
2 12
Did you ever met this relation again?
Numerical Integration of ODEs
NOTE: Can you prove the following expression?

h 0

h4 (4)
yn+1 = yn + 5yn+1 + 8yn0 + yn1
0
y (55)
12 24
or
h h4
yn+1 = yn + (5fn+1 + 8fn + fn1 ) y (4) (56)
12 24

HINT
The previous relations suggest that we can use (appropriately) the
methods that we developed for the numerical evaluation of integrals in
order to derive more accurate relations.
The typical way was to integrate the term on the right of (51) by using an
approximate form of f (x, y ) e.g. using an interpolating polynomial in x.
NOTE: In deriving this we have to use several past points.

Numerical Integration of ODEs

Multistep Methods : 3rd order Adams
Here we use quadratic approximation
Z xn+1
dy = yn+1 yn
xn
Z xn+1  
(s + 1) s 2
= fn + sfn1 + fn2 + Error dx
xn 2
Z s=1  
(s + 1)s 2
= h fn + sfn1 + fn2 ds
s=0 2
Z s=1
s(s + 1)(s + 2) 3 000
+h h f ()ds
s=0 6
then
 
1 5
fn1 + 2 fn2 + O h4


yn+1 yn = h fn +
2 12
 
1 5
(fn 2fn1 + fn2 ) + O h4


= h fn + (fn fn1 ) +
2 12
h 3
yn+1 = yn + [23fn 16fn1 + 5fn2 ] + h4 f 000 () (57)
12 8
Numerical Integration of ODEs
Multistep Methods : 4th order Adams

If we use cubic approximation we get:

h
[55fn 59fn1 + 37fn2 9fn3 ] + O h5


yn+1 = yn + (58)
24
ERROR TERM : We get the error term of the 4th order Adams formula
by integrating the error of the cubic interpolating polynomial
251 5 (5)
E h y () , xn3 xn+1 (59)
720

Numerical Integration of ODEs

Multistep Methods : Milne
This is an alternative method to Adams. We follow the same procedure
but we use different limits of integration. Thus the ODE
dx
= f (x, y ) (60)
dy
can be written as:
Z xn+1 Z xn+1
dy = yn+1 yn3 = f (x, y ) dx (61)
xn3 xn3

and if we replace f (x, y ) with a 2nd order polynomial

4h
(2fn fn1 + 2fn2 ) + O h5


yn+1 = yn3 + (62)
3
ERROR TERM
28 5 (5)
E h y () , xn3 xn+1 (63)
90

Numerical Integration of ODEs

Predictor - Corrector Methods

We estimate the value of yk+1 = y (xk+1 ) by using two successive

approximations.
Prediction : we use the m previous points

yk+1 = P (ykm , ykm+1 , ..., yk ) (64)

Correction : we include the predicted value of yk+1

yk+1 = C (ykm , ykm+1 , ..., yk , yk+1 ) (65)

This 2nd step can be repeated a few times until the two corrected
values converge i.e. until
 
 (j) (j+1) 
yk+1 yk+1  < E

where j is the number of iterations and E is the maximum allowed error.

Numerical Integration of ODEs

Predictor - Corrector Methods : Milne
A corrector formula for Milnes method can be derived by just shifting
the integration limits
Z xn+1 Z xn+1
dy = yn+1 yn1 = f (x, y ) dx (66)
xn1 xn1

and by using Simpsons rule for the integral we get

h
yn+1 = yn1 + (fn+1 + 4fn + fn1 ) (67)
3
with error:
h5 (5)
E y () where xn1 < < xn+1 (68)
90
Milnes Predictor-Corrector formulas
4h 28
yk+1 = yk3 + (2fk2 fk1 + 2fk )+ h5 y (5) (1 ) (xk3 < 1 < xk+1 )
3 90
(69)
h 1
yk+1 = yk1 + (fk1 + 4fk + fk+1 ) h5 y (5) (2 ) (xk1 < 2 < xk+1 )
3 90
(70)
Numerical Integration of ODEs
Predictor - Corrector Methods : Milne
Adams-Multon Method
PREDICTION (xk3 < 1 < xk+1 )

h 251 5 (5)
yk+1 = yk + (55fk 59fk1 + 37fk2 9fk3 ) + h y (1 ) (71)
24 720
CORRECTION (xk2 < 2 < xk+1 )

h 19 5 (5)
yk+1 = yk + (9fk+1 + 19fk 5fk1 + fk2 ) h y (2 ) (72)
24 720
Hamming Method
PREDICTION (xk3 < 1 < xk+1 )

4h 28
yk+1 = yk3 + (2fk2 fk1 + 2fk ) + h5 y (5) (1 ) (73)
3 90
CORRECTION (xk2 < 2 < xk+1 )

1 3h
yk+1 = (9yk yk2 ) + (fk1 + 2fk + fk+1 ) (74)
8 8

Numerical Integration of ODEs

Systems of ODE
For the numerical solution of systems of ODEs we follow the methods
developed earlier.
For example, the 4th order Runge-Kutta will look like:
1
yn+1 = yn + (k1 + 2k2 + 2k3 + k4 ) (75)
6
1
zn+1 = zn + (l1 + 2l2 + 2l3 + l4 ) (76)
6
where
k1 = h f (xn , yn , zn ) (77)
1 1 1
 
k2 = hf xn + h, yn + k1 , zn + l1 (78)
2 2 2
h 1 1
 
k3 = h f xn + , yn + k2 , zn + l2 (79)
2 2 2
k4 = h f (xn + h, yn + k3 , zn + l3 ) (80)

l1 = h g (xn , yn , zn ) (81)
1 1 1
l2 = h g (xn + h, yn + k1 , z n + l1 ) (82)
2 2 2
1 1 1
 
l3 = h g xn + h, yn + k2 , zn + l2 (83)
2 2 2
l4 = h g (xn + h, yn + k3 , zn + l3 ) (84)

Numerical Integration of ODEs

Instabilities in the numerical schemes
We will demonstrate that Milnes method can become unstable even for
simple ODEs.
Consider the ODE
dy
= Ay with general solution for x = xn the yn = y0 e A(xn x0 )
dx
If we solve this ODE with Milnes method we will use the corrector
formula
h 0
yn+1 + 4yn0 + yn1
0


yn+1 = yn1 +
3
where by substituting yn0 = Ayn we get
h
yn+1 = yn1 + (Ayn+1 + 4Ayn + Ayn1 )
3
and by rearranging, we get
   
hA 4hA hA
1 yn+1 yn 1 + yn1 = 0
3 3 3
This is a 2nd order difference equation.
Numerical Integration of ODEs
The difference equation admits a solution of the form

yn = c1 Z1n + c2 Z2n

where Z1 and Z2 are the roots of the quadratic equation:

   
hA 4hA hA
1 Z2 Z 1+ =0
3 3 3

If we set r = hA/3 the roots of the above quadratic equation will be

2r + 3r 2 + 1
Z1 = 1 + 3r e 3r + O(r 2 ) = e hA
1r

2r 3r 2 + 1
Z2 = 1 + r e r + O(r 2 ) = e hA/3
1r
Hence the Milne solution is represented by

yn = c1 e nhA + c2 e nhA/3 = c1 e A(xn x0 ) + c2 e A(xn x0 )/3 , xn x0 = nh

The 1st term is as expected

The 2nd term is parasitic : For A > 0 dies out as xn increases, but
for A < 0 it will grow exponentially with xn .
Numerical Integration of ODEs
EXAMPLES

Solve numerically the ODE

y 0 = 10y , with y (0) = 1
from x = 0 tox = 2 by using Milnes and Adams-Multons methods.
What are your conclusions about the error?
Use Milnes method to solve numerically the ODE
y 0 = 2(x + 1), with y (1) = 3
Is there any indication of instability?

Numerical Integration of ODEs

Convergence Criteria
We will look for a criterion to show how small h should be in the
Adams-Multon method so that re-corrections are not necessary.
Let :
yp = value of yn+1 from predictor formula
yc = value of yn+1 from corrector formula
ycc , yccc , ... = value of yn+1 if successive re-corrections are made,
y = values to which successive re-corrections converge
D = yc yp
The change of yc by re-correcting would be
 
h
9yc0 + 19yn0 5yn1
0 0


ycc yc = yn + + yn2
24
 
h 9h 0
9yp0 + 19yn0 5yn1
0 0
yc yp0



yn + + yn2 =
24 24
By manipulating the difference (yc0 yp0 ) we get:
f (xn+1 , yc ) f (xn+1 , yp )
yc0 yp0 = f (xn+1 , yc ) f (xn+1 , yp ) = (yc yp )
yc yp
= fy (1 )D where yc 1 yp . (85)
Numerical Integration of ODEs
Hence
9h 0
9h
ycc yc = y yp0 = fy (1 )D (86)
24 c 24
If recorrected again, the result is
 
9h 0 0 9h 9h 9hD
yccc ycc = (y yc ) = fy (2 )(ycc yc ) = fy (2 ) fy (1 )
24 cc 24 24 24
 2
9h 2
= [fy ()] D (87)
24
On further recorrections we will have a similar relation. Finally, we get
y by adding all the corrections of yp together:
y = yp + (yc yp ) + (ycc yc ) + (yccc ycc ) + . . .
 2  3
9hfy () 9hfy () 9hfy ()
= yp + D + D+ D+ D + ...
24 24 24
D 9hfy ()
= yp + D 1 + r + r 2 + r 3 + . . . = yp +
 
, r=
1r 24
The above geometrical series will converge only if the ratio is smaller
than unity:
h|fy ()| h|fy (xn , yn )|
|r | = = <1
24/9 24/9
Numerical Integration of ODEs
Convergence Criteria - Adams-Multon Method

Thus the 1st convergence criterion will be:

24/9
h<
|fy (xn , yn )|

If we wish to have yc and y the same to within one in the Nth decimal
places, then
 
D rD
y yc = yp + (yp + D) = < 10N
1r 1r

If r  1 the fraction will be approximately r /(1 r ) r we can write a

2nd convergence criterion which ensures that the 1st corrected value is
adequate (i.e. it will not changed in the Nth decimal place by further
corrections)  
N
1 24/9
D 10 <   =
r h|fy (xn , yn )|

Numerical Integration of ODEs

Convergence Criteria - Milne Method

The previous criteria were derived for the Adams - Moulton method.
Similar criteria can be found for the Milne method :
1st convergence criterion :
3
h<
|fy (xn , yn )|

2nd convergence criterion :

 
1 3
D 10N <   =
r h|fy (xn , yn )|

NOTE: These criteria are for a single 1st order ODE only. A similar
analysis for a system is much more complicated.

Numerical Integration of ODEs

Errors & Error Propagation

The previous error analysis examined the error of a single step only,
the so called local truncation error.
The accumulation of these errors, termed as the global truncation
error is important.
There are other sources of error in addition to the truncation error.
Original data errors : If the initial conditions are not known exactly
or exressed inexactly as terminated decimal number, the solution will be
affected to a greater or lesser degree depending on the sensitivity of the
equation.
Round-off errors : Both floating-point and fixed-point calculations
in computers are subject to round off errors.
Truncation errors of the method: These are the types of error
due to the use of truncated series for approximation in our work, when
infinite series is needed for exactness.

Numerical Integration of ODEs

Error Propagation
When we solve ODEs numerically we must worry about the propagation
of the previous errors through the subsequent steps.
If we consider the 1st order ODE y 0 = f (x, y ), with y (x0 ) = y0 then if Yn
is the calculated value at xn and yn the true value, then the error in Yn
will be:
n = yn Yn (88)
By the Euler algorithm :
1
n+1 = yn+1 Yn+1 = yn + hf (xn , yn ) Yn hf (xn , Yn ) + h2 y 00 (n )
2
f (xn , yn ) f (xn , Yn ) 1 2 00
= n + h n + h y (n )
yn Yn 2
1 2 00
= n (1 + hfy (xn , n )) + h y (n ) where n between yn , Yn
2
1 2 00 f
= (1 + hk) n + h y (n ) where k = . (89)
2 y
I.e. the error propagation is linear. If |1 + hk| 1 the error increases
while for |1 + hk| 1 decreases. This observation leads to the condition:
2 < hk < 0 or f /y < 0 (90)
Numerical Integration of ODEs
 1
n+1 = (1 + hk) n + h2 y 00 (n ) (91)
2
Since Y0 = y0 we get:

0 = 0
1 1
1 (1 + hk)0 + h2 y 00 (0 ) = h2 y 00 (0 )
 2  2
1 2 00 1 2 00 1
2 (1 + hk) h y (0 ) + h y (1 ) = h2 [(1 + hk)y 00 (0 ) + y 00 (1 )]
2 2 2
1 2
h (1 + hk)2 y 00 (0 ) + (1 + hk)y 00 (1 ) + y 00 (2 )

3
2
1 2
h (1 + hk)n1 y 00 (0 ) + (1 + hk)n2 y 00 (1 ) + ... + y 00 (n1 )

n
2
If k 0 the truncation error at every step is propagated to the next
step after beingamplified by the factor (1 + hk). Finally, for h 0 the
error at any point is the sum of all previous errors.
If the fy is negative and of magnitude such that kh 2 the errors are
propagated with diminishing effect.

Numerical Integration of ODEs

We now show that the accumulated error after n steps is O(h); that is
the global error of the simple Euler method after n steps is O(h).
If |y 00 (x)| < M with M > 0 then the previous equation will be written as:

1
|n+1 | (1 + hk)|n | + h2 M
2
which is a difference equations of the form:
1
Zn+1 = (1 + hk)Zn + h2 M with Z0 = 0
2
With obvious solution
hM hM
Zn = (1 + hk)n
2k 2k
since 1 + hk < e hk for k > 0 we get
hM hk
n hM hM nhk
hM  (xn x0 )k 
Zn < e = e 1 = e 1 = O(h)
2k 2k 2k 2k
It follows that the global error n is O(h)
Numerical Integration of ODEs