UNIVERSITY of NIS GAF

FACULTY OF CIVIL ENGINEERING AND ARCHITECTUR E

NUMERICAL METHODS
In Computational Engineering
G.V. MILOVANOVIC, sci. advisor
&
D. R. DORDEVIC, lecturer

financed by ·
Austrian
~ Development Cooperation

Nis, 2001
 UNIVERSITY of NIS
FACULTY OF CIVIL ENGINEERING AND ARCHITECTURE

NUMERICAL METHODS
in Computational Engineering
G.V. MILOVANOVIC, sci. advisor

(http:/ /gauss.elfak.ni.ac.yu/)

D. R. DORDEVIC, lecturer

(http:/ jwww.gaf.ni.ac.yu/cdp/lecturer.htm)

Nis, 2007.
Prof. Gradimir V. ?IIilm·;moYit. lTni\'l'rsitY ~is
Prof. Don1e R. Don1PYi(, Unin'rsitY ~is ·
Numerical Methods in Computational Engineering
ISBN 978 86-80295-81-7

The publishing of this script is part of the project CDP+ 53/2006 financed by Austrian
Cooperation through WUS Austria
This copy is not for sale
Ova skripta je objavljena u okviru projekta WUS Austria CDP+ 53/2006 finansiranog
od strane Austrian Cooperation
Besplatan primerak

Printed hy Grafilm Petkoyi(. Nis, 2007 (100 copies)

 Contents

Preface . IX

1. Mathematics and Computer Science 1

1.1. Calculus l
1.2. Number represeut.at.ion :)
1.3. Error, accuracy, aud stability
1.4. Programming 10
1.5. Numerical :-;oftware 10
1.6. Case stu<l.r: errors, round-off, and stability 11
Bibliography . 12
As.~igmnent- I (http://www.gaf.ni.ac.yu/CDP/Assignment-I.pdf)
2. Linear Systems of Algebraic Equations: Direct Methods 15
2 .1. Elements of matrix cakulu:-; . l;)
2.1.1. LR. factmization of quadratic matrix lG
2.1.2. l'viat.rix cige!lvPct.nrs awl eigenvalues 17
2.2. Direct met.hocl:-; in linear algebra 17
2.2.1. Iut.ro<luctiou 17
2.2.2. Gauss <~limi!latioll with pivoting 18
2.2.3. lVIatrix iHver:-;iou usiug Gauss method 21
2.2.4. Factorization md.hods 21
2.2.5. Program realization 24
Bibliography . ~B

3. Linear Systems of Algebraic Equations: Iterative Methods :3;J

3.1. Intmcluc:tiou :35
3.2. Simple iteration metho<l 3G
3.3. Gauss-Seidel method . 3G
3.4. Program n~alization 37
3.5. Packages fm systems of linear algebraic equation:-; 41
Bibliography . 42
Assignment-II-III (http://www. ~af. ni. ac. yu/CDP I Assignment- II- III. pdf)
4. Eigenvalu~ Problems 45
4.1. Introduction 4ij
4.2. Localization of eigenvnlues 50
4.3. l'viethocls for dominant eigenvalnes 51
4.4. Niethocls for subdominant <~igmwalues G3
4.5, Eigenvalm~ problem for synnnetric tridiagonal matrices 57
4,6. LR ancl QR algorithms . :)9
4.7. Soft1va.re eigenpacka.ges GO
4.8. Generalized a.nd nonlinear eigenvalue problems G1

v
 BibliogTaphy . 62
Assignment-IV (http:llwww.gaf.ni.ac.yuiCDPIAssignment-IV.pdf)
5. Nonlinear Equations and Systems of Equations 65
5 .1. N onlineai' equations 65
5.1.0. Introduction 65
5.1.1. Newton's method 68
5.1.2. Bisection method 72
5 .1.3. Program reali,.-;ation 73
5.2. System of nonlinear equations 83
5.2.1. Newton-Kantorowitch (R.aphson) method 83
5.2.2. Gradient method 87
5.2.3. Globally convergent methods 91
Bibliography . 94
Assignment-V (http;/ lwww. gaf. ni. ac. yuiCDP I Assignment-V. pdf)
6. Approximation and Interpolation 97
6.1. Introduction 97
6.2. Chebyshev systems 98
6.3. Lagrange's interpolation 99
6.4. Newton's interpolation with divided differences 100
6.5. Newton's interpolation formulas 102
6.6. Spline functions and interpolation by splines 104
6.7. Prony's interpolation 106
6.8. Packages for interpolation of functions 107
Bibliography . 108
Assignment-VI (http: I lwww. gaf .ni. ac. yuiCDP I Assignment-VI. pdf)
7. Best Approximation of Functions 109
7.1. Introduction 109
7.2. Best L 2 approxiination 112
7.3. Best I"· approximation 114
7.4. Packages for approximation of fnnctions 119
Bibliography . 119
Assignment-VII . (http: I lwww. gaf. ni. ac. yuiCDPI Assignment-VII. pdf)
8. Numerical Differentiation and Integration 121
8.1. Numerical differentiation 121
8.1.1. Introduction 121
8.1.2. Formulas for numerical differentiation 121
8.2. Numerical integration- Quadrature formulas 12.3
8.2.1. Introduction 123
8.2.2. Newton-Cotes formulas 124
8.2.3. Generalized qHadra.tHne formulas 126
8.2.4. R.ombl~rg integration 128
8.2.5. Program n~alization 128
8.2.6. On numerical evaluation of a class of double integrals 132
8.2.7. Packages for nmnerical integration 134
Bil>liography . 13.5
Assignnwnt-VIII (http: I lwww. gaf. ni. ac. yuiCDP I Assignment-VIII. pdf)
9. Ordinary Differential Equations - ODE 137
9.1. Introduction 1:3 7

vi
 0.2. Euler's method 1.38
0.3. Ge1wr;d liiwm multi-st.<•p uwthod 1:30
0.4. Clwin~ of initial valuPs 141
0.:). Predict.or-corrP.ct.or llH'.tlwcls 1-.11
0.6. Program n·alizat.ion of Imdti-st.<~p methods 142
0.7. RungP-Kut.t.a lll<-'t.lwds 14.)
0.8. Progrmu realization of Rnng<'-Kut.ta methods 1;j()
0.0. Solution of s.vst.<~lll of <-'quat.ious and equations of higher order l;j:j
0.10. Bmmdary prohbus 158
0.11. Packag<!S for ODEs Hil
Bibliography . 161
Assignment-IX (http://www.gaf.ni.ac.yu/CDP/Assignment-IX.pdf)
10. Partial Differential Equations - PDE 1Ci3
10.1. Introduction 163
10.2. Grid method . 164
10.3. Laplace eqnation 165
10.4. Vlave eqnat.ion 167
10.5. Packages for PDE.~ 160
Bibliography 170
Assignment-X (http://www.gaf.ni.ac.yu/CDP/Assignment-X.pdf)
11. Integral Equations . 173
11.1. Introclnction 173
11.2. Mc!t.lwcl of sucu~ssive approximations 17:)
11.3. Application of q1wdra.t.me formnlas 175
11.4. Program n~alization 176
Bibliography 178
Assignnwnt-XI (http://www.gaf.ni.ac.yu/CDP/Assignment-XI.pdf)

Appendices
A.l. Equations of Technical Physics (http://www.gaf.ni.ac.yu/CDP/ETPH.pdf)
A.2. Special Functions (http://www.gaf.ni.ac.yu/CDP/SPEC.pdf)
A.3. Numerical Methods in FEM (http://www.gaf.ni.ac.yu/CDP/NMFEM.pdf)
A.4. Numerical Methods in Informatics (http://www.gaf.ni.ac.yu/CDP/NMINF.pdf)

vii
 Preface

A course in Numerical Methods in Computational Engineering, oriented to engineering

education, originates at first from the course in numerical analysis for graduate students of
Faculty of Civil Engineering and Architecture of Nis (GAF), and then from course Numer-
ical Methods held in English language at Faculty of Civil Engineering in Belgrade in the
frame of project DYNET (Dynamical Network) in common of Faculty of Civil Engineering
of University of Bochum, Faculty of Civil Engineering and Architecture of University of
Nis, Faculty of Civil Engineering of University Belgrade, and IZIIS (Institute for Earth-
quake Engineering and Seismology) of University Skopje. The subject Numerical Analysis
was held in the first semester of postgraduate studies at GAF by Prof G. V. Milovanovic
for years. In continuation, following Bologna process, the new structured subject entitled
Numerical Analysis is ~o be introduced to PhD students at GAF. In addition, having in
n1ind that course in numerical analysis become accepted as an important ingredient in the
undergraduate education in engineering and technology, it was with its main topics involved
a
in undergraduate subject Informatics II at GAF Nis (As collateral case, in Appendix A.4.
-in electronic fonn -are given numerical methods in Informatics, what could be interesting
for students of this orientation).
The backbone of this script are famous books of G. V. Milovanovic, Numerical Anal-
ysis, Part I, II, and III, Naucna knjiga, Beograd, 1988 (Serbian). In addition, the book
Programming Numerical Methods in Fortran, by G. V. Milovanovic and Dj. R. Djordjevic,
University of Nis, 1981 (Serbian), with its engineering-oriented text and codes, was rather
used.
As previously noted, this textbook is supporting undergraduate studies, master and
doctoral study at GAF; and international master study in the frame of DYNET project.
Presentation on GAF site would enable distance lear~ing technique and on-line consulta-
tions with lecturer. By up-to-day engineering oriented applications the supporting of life
long education of civil engineers will be enabled.
This script will be available on the site of GAF (http: I lwww. gaf. ni. ac. yu) under In-
ternational Projects and can be reached by chapters using address
http: I lwww. gaf. ni. ac. yul cdplsubj ect_syllabus .htm. Each chapter concludes with a ba-
sic bibliography and suggested further reading. Tutorial exercises in form of selected as-
signments are also presented on the site of GAF. Some hints for solutions are given in the
same files.
Devoted primarily to students of Civil Engineering (undergraduate and graduate -
master & PhD), this textbook is dedicated also to industry and research purposes.

Authors

lX
Faculty of Civil Enginc(~ring Faculty of Civil Engineering and Architectme
Belgrade Nis
Master Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON I

1. Mathematics and Computer Science

1.1 Calculus
The principal topics in calculus are the real and complex number systems, the
concept of limits ancl convergence, and the properties of functions.
Convergence of a sequence of numbers :ri is defined as follows:
The sequence :r.;, converges to the limit :c* if7 given any tolerance E > 0,
there is an index N = N(E) so that for all ·i 2:: N we have Ixi- :r* I ::=;E. The
notation for this is
lim x.;, = x*.
i--too

Convergence is also a principal topics of numerical computation, but with a different

emphasis. In calculus one studies limits and convergence with analytic tools; one tries
to obtain the limit or to show that convergence takes place. In computations, one has
the same problem but little or 1io theoretical knowledge about the sequence. One is
frequently reduced to using empirical intuitive tests for convergence; often the principal
task is to actually estimate the value of the tolerance for a given x.
The study of functions in calculus revolves about continuity, derivatives, and inte-
grals. A function f (:r) is continuous if •
lim .f(x.;,) = .f(x*)
:r:;~:I;*

holds for all :r* and all ways for the xi to converge to x*. We list six theorems from
calculus which are useful for estimating values that appear in numerical computation.
Theorem 1 (Mean value theorem for continuous functions). Let .f(x:) be con-
tinuous on the interval [a, b]. Consider points XHI and XLOW in [a, b] and a value y so
that .f(X LOW) ::=; y ::=; .f(X HI). Then there is a point pin [a, b] so that
.f(p) = y.
Theorem 2 (Mean value theorem for sums). Let .f(x) be continuous on the inter-
val [a, b], let x 1 , :r 2 , ... , :rn be points in [a, b] and let w1 ,w 2 , ... , 'Wn be positive numbers.
Then there is a point p in [a, b] so that
n n

L w.;,(:r:).f(xi) = .f(p) L 'Wi·

i=l

Theorem 3 (Mean value theorem for integrals). Let f (:c) be continuous on the
interval [a, b] and let w(x) be a nonnegative function [w(x) 2:: 0] on [a, b]. Then there is
. a point pin [a, b] so that
·b (b
!
. o. w(:r)f(:r,)dx = f(p) .la. w(x)dx,

1
2 Numerical Methotls in Computational Engineering

Theorems 2 and 3 show the analogy that exists between sums and integrals. This
fact dt~rives
from the definition of th<-~ integral as

where the points :r:'i with :1:.i. < :r:i+l are a partition of [a, bj. This analogy shows up
for many nmnerical methods when~ one variation applies to sums and another applies
to int<-~grals. Tht~orem 2 is prow~cl from Tlwon~m 1, and then Theorem 3 is proved
by <1. similar method. The assumption that 'w (:r:) ~ 0 (w.;. > 0) may be i·eplaced by
w(:r) :::; 0 (w.i. < 0) in tlws<~ tlworems; it is essential that w(:r:) be on one sign shown hy
the exalllplc w(:1:) = .f(:1:) = :1: and [(}.. b] = [-1, 1].
Theorem 4 (Continuous functions assume max/min values). Let .f(:1:) !Je coll-
~imzous on tlw i~1tenral [~~,.b] with Ia I, lbl :::; oo. Tl1e~11 there are points XHI n.ncl XLOVV
lll [a, b] so that for all :1: m [a, b]

.f(XHI):::; .f(:r:):::; f(XLOW).

The derivative of .f (:~:) is defined lw

rf;f = .f'(:r) = lim .flr +h)- .f(:r:).

rh h-+O h

As an illustration of the dift(~rew:c l>etvvef~ll tlwmy and pra.ct;ice, the CJIUI.lltity [.f (:1: +
h) - .f(:1:)]/h uw be l'(~placed hy f[(:r +h) - f(:~:- h)]/(2h) with 110 cha.np,e in the
t;heory lm t wi tlr dn!.lll<l Uc iluprove11W11t in the nlte of ccmvergeuce; that is, mw:h nw1·e
accurate (-~.'-ltimat;c.'-1 of .f'(:1:) are o!Jt;ailled f(H· a given W!.lue of h. The /;;-th deriva-
t;iw~ is Uw derivative of Uw (k -1)th derivative: they arc dcuoted h_y d1"f/rl:r 1" m
f" (:r), f"' (:r: ), .f( 4 ) (:r:). f(S) (:1:) ....
Theorem 5 (Mean value theorem for derivatives). Let; .f(:r:) he continuous ;:md
difi(~rcntial>le i11 [a, 1!], with lal, lhl < oo. Then there is a point fJ in [u.,b] so that

f (/J) - f ((J.) = f' ((!)

l;- (1,

f(:r) = .f(c) + f'(f!)(:r:- c)

TlH~
special case of Tlwmcm G with .f(a) = .f(l1) = 0 is known as Dolle's thc~on~m. It
stat<~s tl1at if .f(o.) = f(l;) = 0, tlwn there is a poillt. (J between n, a11<lh su that .f'(fJ) = 0.
This is d<~riv(~d fnnu Thc~orclll [; l >y nmltiplyiug tlmmgh by lJ - rL rcumuiug o.. l; as :1:. r: .
all<l t.lwu applying tht-~ hrst. l'onn t.o the smallt~r iutcTval [:r, c] m [r:, :r], dcpencliug 011 tlw
n~latiou hdwec11 :1: awl r:.
A very impmtaut tool in llllllterical aualysis is tlw <~xtcnsiou of the secoll<l part of
Thcon~m G to usc· highc~r dt•rivativ<~s.

Theorem 6 (Tailor series ' with remainder). Let I (:1:) haFc n + 1 c:cmti111wlls
rleri,rat:ivi~S in [u.. li].

Givcu points :1: awl r: iu [a.lJ] we hav<~

·(:1:) = /'(r:) + /"(r·)(:l:- r:) + /"'(r:) (:1:- rV + (" (:1:- rV + ... + f(n)(c) (:r- r:)n
·/ · · · 2! · 3! n!
 L<~sson I - Mathematics and Computer Science

wlwre Rn+l has <~ith<~r mw of tlw following forms (pis a point betweeu :t: awl c):

- .t•(n+l) ((! ) -
R·n+l (.1.' ·) - (:I: - c)n+l
----
(n + 1)!

Rn+l (:r:) = A
n. .
~·:r (:r:- t)"'.f(n+l) (t)dt
c

If a function f d<~peuds on sPveral variables, one can differentiate it with respect to one
:variable, say :r:, while ke<~pinp; all t.hc rest fixed. This is a partial derivative off and it
1s <lenoted by of/ih: or fr:· Higlwr mder and mixed derivatives are defined by successive
difff~rent.iation. Taylor's S(~l'i!~S for functions of several variables is a direct extension of
the formula in Theorem G, althonp;h the number of terms in it grows rapidly. For two
variables it is ·

f(:r:,y) = f(c:,rl) + lr:(:r- c)+ fy(y- d)+ ~[f:1::~:(:r:- c) 2 + 2f:r:y(:r- c)(y- d)

+fyy(:tJ- d) 2 ] + ... '

where all the partial d<~rivatives arc evaluated at the point (c, d).
Theorem 7 (Chain rule for derivatives). Let f(:I:, y, ... , z) have contimzous first
partial derivatives with respect to a.ll its variables. Let :r: = x(t), y = y(t), ... , z = z(t)
l>e continuous cliHerentia.IJle fimctions oft. Then

,r; (t) = f (:r (t) , y (t) , . . . , z (t) )

is contiwzously diffi,~rentia/;le and

,r;'(t) = f: 1,:r:'(t) + f:yy'(t) +.· .: + fzz'(t).

Finally, we state
Theorem 8 (Fundamental theorem of algebra). Let p(:1:) be a polynomial of
degree n 2: 1, tha.t i,.,,

where the a£ are real or C:Olll]Jlex wzmhers and O.n =/= 0. Then, there is a complex lWJnbeT
p so that p(p) = 0.

1.2. Number representation

Numbers are represented in number systems. Any number of bases can be employed
as the base of a number system, fm example, the base 10 (decimal), 8 (octal), 12
(duodecimal), 16 (hexadecimal), or the base 2, (binary) system. The base 10, i.<~.
decimal system is the most common system used in human communication. In spite
of not being optimal (optimal would be theorf~tical system with base e, base of natural
logarithm, or teclmical system with base 3, trinary system), digital computers use, due
to electronic technology, system with base 2, or binary system. In a digital computer,
a bi1iary number consists of a number of binary bits. The number of binary bits in
a binary number determines tJH-~ precision with which the binary number represents a
cleciir1al number. The inost common size of binary number is a 32-bit number (we say,
the inachine word is 32 bits long, what defines the "32-bits word computer"), what
can represent approximately 7 digits of a decimal number. Some computer have 64
bits binary m1mbers, i.e. (:)4 bits machine word length, which can represent 13 to 14
decimal digits. For many engineering and scientific calculations, 32 bit arithmetic is
 4 N unwrical Methods in Colilputational Engineering

g·ood enough. But, for mc-my other applications, G4 bit arithmetic is required. Hig·her
precision (i.e. 64 bit, or even 128 bit) can he reached by software means, using Double
precision or Quad precision, resp<~ctively. Of course, such software enhancement must
be payed by even 10 times execution times of single precision calculation.
As already told, computers store numbers not with infinite precision but rather in
some approximation that can be packed into a fixed number of bits (binary digits) or
bytes (groups of 8 bits). Almost all computers allow the programmer a tho ice amm;tg·
several different such representations or data types. Data types can differ in~ the number
of bits utilized (the word-length), but also in the more fundamental respect of whether
the stored number is represented in fixed- point (also called integer) or fioating·- point
(also called real) format. A number in integer representation is exact. . Arithmetic
between numbers in int<-~ger n~prcsentation is also exact, with the provisos that
(a) the answer is not ontsi<h~ tlw range of (usually signed) integers that can be n~pn~­
sented, all<l
(b) division i::; interpn~t<~<l a::; producing an integer result, throwing a.way any integer
remainder.

,.-A-.,,

=0
lf2 1000 000 0 1000000 000 00000 000000 00 (a)
3=0 10 00 00 10 1 100000000 00000 000000 00 (b)
I-~ = 0 0 11 1111 1 10000 00000 0000 000000000 (c)
() OI101001 l 10 10 I I 0 I 0 I I 1 11 I 100 I 0 I 0
---c=··. (d)
= 0 I 0 00 00 10 o o o o o o o o o o o o o o o o o o o o o o ol (e)
3+ I0- 7 =0 1 0 0 0 0 0 I 0 .1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (f)

Figure 1. 2 .1.
In Fig. 1.2.1. an~ giv<~u Hoatiug point rcpn~::;entations of nmnhcrs in a typical 32-bit
(4-hyte) format., with the following <~xmnples:
(<t) Tlw mnuber 1/2 (not<~ the bias in the <~xponcnt);
(b) the mnnbcr :3;
(c) tlw mnulwr 1/4;
(d) the nmuher 1 o- 7 , r<~pn~s<~ll tt~d to machine accuracy;
(P) tlw sanw lllllllh<~r 10- 7. lmt shifted so as to h;.w<~ the same expom~nt as the number
3: with this shifliug. ·' :-;ignificaucc is lost ancl 10- 7 becomes zero; shifting to a
connnon exponeut 1111. 'l('<:ur before two nmnh<~rs can be added;
(f) smn of tlw numbers ;~ -i w- 7 , which c~qnals :3 to nmchiw~ accuracy. Even though
10- 7 can he n~pres<~Ut(~<l au:uratdy by it::;elf. it cannot accurately he add<~d to a
much larger Hmnb~~r.
In fioating-point repn~sentatiou, a rnunl ><~r is represented internally by a sig1~ ~ Ji t
s (int(~rprd.<~d as plns or mim1s), au exact iut<~gcr <-~xponent e, awl an exact pos1t.1v<~
iuteg<~r manl.is::;a J\1. T<lk<-~ll togct.ltcr tlw::;c n~prcsent tlw uuml><~r

(1.2.1) s x NIx n"-B

where B is t.lw base of t.lw rqm~s<~ut.ation (nsually B = 2, lmt sometimes B = Hi),

and E is t.lw bias of t-.lw PX]Hnwut.. a. fix<~d iut(~gcr coust.aut. f(H· any given machiw~ awl
rqJn~:·H!llt.at.iou. ,
Several HoatiHg-poiuL hit pat.t.<Tus can n~pn~s<~ul the sauw umulwr. If B = 2.
for cxau1ple, a 1wmtissa with lec-u ling (high-order) z<To hits can b<~ left-shiftecL i.e.,
multipliPd by a power of 2. if tlw <~xponent is decreased by a. comp<~usating muount.
 Lesson I - I'viatlwnmtics awl Co1npntc~r Scic~ncc

Bit pat.t.e~rus that m'(~ ··as ldt-shift.e~d as t.lwy can he~" are t.fTnH~d uonwtlizc~d. Most
comput.e~1·s always prodnce~ um·ul;tlized resnlt.s. since these do not waste any hit.s of t.he~
mantissa ancl thus allow a 1-';r<~ater accuracy of the representation. Since tJ;e-~ hip;l1-on leT
bit. of a properly nonuali:wd mautissa (wlwu B = 2) is always one, souw compnte~rs
do not ston~ this hit at all. giving one extra hit of significaw:e. Arithme~tic au1ong
muuhcrs iu floating-point n~presc~ntatiou is not exact., even if the operawls happ<~n t.o
he~ <~xa<:tly rcpn~se~ntcd (i.e~.. have~ <~xact valw~s in the form of e~quation ( 1. 2.1). For
<~xamph\ two fioatiug nmuh<~rs an~ added by first right-shifting (divicliug by two) tlw
m;u1t.issa of the small<~r (in 111agui tw le) ollc, sinmltaneously iw:reasillg i t.s c~xpmwuL
nntil the two operands hav<~ tlw smuc c~xpmwnt. Low-order (least significant) hits of
the smaller opcc~nmd an~ lost by this shifting. If the two operamls differ too greatly in
1wtgnitude, tlwu t.lw smaller opc~mwl is efte-~<:t.ivdy replaced by zc~ro, siw:c~ it is rigllt-
shiftecl to oblivion. Tlw sm;\.llc~st (iu nutgnitndc) floating-point nmnber which. wlwll
ad< lc-~d to the fioating- poi11 t. nmnl HT 1. 0, prod nu~s a floating-poillt result eliffc~rcnt from
1.0 is tenned the lmtc:hinc ac:cmac:y ·m. A typical compntc~r with B = 2 ancl a 32-hit
word-length has '///. aronwl ;) x 10- 8 . Gc~n<~rally speaking, th<~ machine accuracy ·1n is
the fractional au:nrac:y to w hic:h fj oating-point munbers are n~rn·csentc~cl, conespou< ling
to a dmng<~ of one iu the l<~ast significant bit of the mantissa.

1.3. Error, accuracy, and stability

Exu~pt for int<~gcrs all<l sonw fractions, all binary n~preseutatiolls of decimal umu-
hers arc approximabous, owing to tlw finite word length of binary mnnbcrs.Tlms, some
loss of pn~cision iu the binary n~plTS<~ntation of clec:in1alunmber is nnavoiclahle. R.esnlt of
aritlm1d.ic op<·Tatioll among binary Hmnbers is typically a longer 1>inary m1ml>cr which
cmmot be repn~s<~nted witl1 tlw nmnbcr of aw:tilahlc hits of the digital computc~r. Tlms,
the results are romHl<~<l off in t.lw bst available hinmy bit. This rmmcling-off is called
ronnel-off <~rror. WelL pretty nmch ally aritlnnetic operation among floating muul>ers
slwuld be thought of ;\.s in trod ncing au additional fractional en or of at least f 111 , called
ronncloif en or. H is import aut to nllclcrstancl that fm. is not the smallest. fioatiug- poillt
1mmher that <:all he rcprcs<~ntc<l ou a lllc\.chiuc. That nmnhcr depends 011 how lLtauy
hit:-: tluTc an~ in the <~xpon<~llt. while E-m cl<~pe~w1s on how many bits there arc in the
nmntissa. Roundoff <~nms accnumlate with incn~~tsiup; mnounts of calcnlation. If, in
tlw conrse of obtailling a c;\knlatcd value, ow~ performs n such arithmetic operations,
he might be satisfied with a total roundoff <~nor on the order of jnc 111 , if the ronwloff
Pnors <·.mrw in randomly np or (lown. (Tlw square root comes from a random-walk.)
Howcvc-~r, this estimate~ call he v<~ry haclly off tlw mark for two reasons:
(i) It very fn~qnently happ<~us that the regnlarities of calculation, or the p<;cnliarities of
c:ompntcr, cause tlw ronw loff cnors to accmnulate preferentially in one direction.
In this case the total will lH~ of onlcr nc./11.
(ii) Some especially nnfavorablc oc:cnnenccs can vastly increase tlw ronllcloft" en or of
single opcra!".iolls. Ge~ncrally tlwse can be traced to the snbtractiou of two very
lH~a:dy equal umuhc~rs. giving a rcsnlt whose only significant 1Jits me those (few)
low-onlcr oucs ill which the operands differed. You might think that snd1 a '' coinci-
clelltal'. sn1>tractiou is 1mlikdy to occnr, wh<:tt is not always true. Some mathemat-
ical c~xprcssions magnif\r its probability of occnrrcnce tn~nwndonsly. For cxmuplc,
in the familiar formnla for the solntion of a quadratic equation,

(1.3.1)'
2a
wlwn o.c < < [J 2 the addition lH~c:omcs critical and ronnel-off could ruin the~ calcula-
tion (seE-~ .s<~c:tion 1. (j.). .
. Roundoff enor is a charactc~ristic ofcomputcr hardware. There is another, different,
kind bf error that is a characteristic of the program ot algorithm used, in<lepenclent of
the liartlware on which the program is executed. Many numerical algorithms compute
,. discrete" approximatiolls to som<~ desired "continuous" quantity. For example~, an
integral is evaluated 1mnwrically by computing a function at. a discrete set of points,
rather than at ,. every"· point. Or, a fnnction may be evaluated by smmning; a finite
 Numerical Methods in Computational Engineering

number of leading terms in its infinite series, rather than all infinity terms. In cases like
this, there is an ad_iustable parameter, e.g., tlw number of points or of terms, such that
the "true" answer is obtained only when that parameter goes to infinity. Any practical
calculation is done with a finite, hut sufficiently large, choice of that parameter. .
The discrepancy between the true answer and the answer obtained in a practical
calculation is called tlw tnmcation error. Truncation error would persist even on a
hypothetical, "perfect" computer that had an infinitely accurate represe1itation and no
roundoff error. As a general rule there is not much that a programmer c~1n do about
roundoff (~nm·, othe1· than to choose algorithms that do not magnify it umwcessarily.
Truncation (~nor, on tlw otlwr hand, is entirely under the programmers control. In fact,
it is only a slight exaggeration to say that clever minimization of truncation error is
practically the entire content of the field of numerical analysis.
Most of the time, truncation error and roundoff error do not strongly inter~tct with
one another. A calculation can be imagined as having, first, the truncation error that it
would have if nm ou an infinite-precision computer, and in addition, the roundofF enor
associated with the umnb(T of op<~rations IH~rfonned.
Some comrmtations arc very s<~nsitive to ronnel-oft' and others are not. In some
problems sensitivity to rouwl-off can he eliminated by changing the formula or method.
This is always possible; th(~rc arc many prohl(-mlS which arc inherently sensitive to
round-off and any other 1mu~rtaiuties. Thus we must distinguish between sensitivity of
methods and sensitivity inherent inproblems.
The word stability app<~ars (lming numerical computations and refers to continu-
ous dep(~uclence of a solution on the data of the problem or rnethod. If one says that
a lll(~thod is n'umerically 'I.Lnstable, one nwans that the round-off effects are grossly
magnified by the nwthocl. Stability also has precist~ technical meaning (not always the
sanw) in <liff(-~rent an~as as well as in this g(meral one. ·
Solving diff(~rcntial <~qnations n::;nally lc-~ads to clifh~rence equations, like

Here, the sequence :1: 1 . :1::2, . .. is defined, alHl for given initial conditions :r1 and :r:2 of
clifferential equation, we~ gd the initial conditions for difference equation. For example,
:r 1 = :30 . :r: 2 = 25. Cmupnting in snccc~ssion for 4, 8, 16, 32,64 decimal cligits gives
th<~ n~snlts that can h<~ compared with the exact one, :r:.;. = 3G/(5/6)'i. (Compute: in
Mathematica, using N[:1:[I + 2], !.:]. where k = 4, !::5, 1G, 32,64 nmnbcr of decimal digits).

1 4 8 1G True value
1 :30.00 30.00 30.00 30.00
2 25.00 25.00 25.00 25.00
3 20.!::5:3 20.8333 20.8:33:3 20.!::5:B3
4 17.~)() 17.:3Gll 17.3611 17.:3Gll
5 14.4() 14.,!li7G 14.4G76 1~1.4li7G
(j l2.07 12. 05G:J 12.0563 12. OSG:)
7 10.00 10.0470 10.04G9 10.04G9
8 o.G1!::5 8.3724 8.3724 8.3724
9 ().541 G.9773 G.9770 G. 9770
10 7.121 5.!::51:33 5.8142 5.8142
11 .925 4.!::5,!7!::5 4.8452 4.84G2
12 1G. 790 4.0290 4.037G 4.0:HG
1:) -:31.!)20 3.3888 3.3G47 3.3G"17
H 108.700 2.7:n8 2.8039 2.80:39
1li !)5'l. GOO 1.2!)78 1.9472 1.9472
1!::5 8G7G.OOO -4.4918 1.3522 1.:)522
20 77170.000 -51.6565 .9390 .9390
22 G.~) X lOS '--472.7080 .G521 .GS21
2S -l.t\ X 10 7 12701.1000 .:377G .3774
20 S.O X 10 8 -:5<15079.0000 .2U4 .2lo"J
:)() .J.fi X 10 9 -~·L 1 X lOG .1071 .1S17
 Lesson I - l\llathcm<ttic:s and Compntf~r Scit~nc:e 7

3 .Jr: -1.1 X 10 12 7.5 X 10 8 10.8822 .0609

40 -1.1 X 10 14 -1.8 X lOll -2629.5300 .024S
so l.S X 10 19 -1.0 X 10 16 -l.Sxl0 8 .00:39
31 27
75 1.:3 X 1() 9.2 X 10 1.3 X 10 20 .00
This cliff<~l·<~ncc <~quation is nustable and om~ c:an see that the compnta.tion quickly
"blows up". Om~ nice thing al>ont nustahl<~ compntat.ion is that they usually produc·<~
huge, nonsense mm1bers that mw is not tempt<~d to accept as correct. However, imagine
that one wanted only 30 t.<TlllS of tlw :r.i and was using the computer with 1() decim<tl
digits. How would ow~ know that the last term is in error by GO percent '?
Tlw wmd condition is used to descrilH~ the sensitivity of probl<~ms to tmcertainty.
Imagine the solutimt of a problem being obtained hy f~valuation a fnnction f (:1:). Then,
if :r: is change<l a litth~ to :r + rh, the value f(:r:) also changes. The relative condition
number of this ch;-mgc is
If (:r + r5 :r:) - f (:r) I/ I (b: I
If (:r:) I :r: '
or
f(:r: + (b:)- f(:r:) :r
------~----~ X ---
rh f(:r)'

and, for i5:r v<~ry small, con<lition nnmber c is

:r:f' (:r:)
C;v - - - - ' -
, f(:I:) .

This number et>timates how mnch an nncertainty in the data :r of a problem is mag-
nifi<-~d in its t>olnt.ion f (:r:). If this mnnlwr is large, then the problem is said to be
'ill-conditioned m pomly conditioned.
The given fmmula is for t.lw simph~st case ofa function of a single variable; it is not
<~asy to obta.in such formulas for more complex problems that depend on many variables
of different types. We can se<~ three different ways that a problem can have a large
condition number:
1. f'(:r) ·may br. laT_(]f: while :r: and .f(:r) are not;
If we evaluate 1 + Jl:r:- 11 for :r very close to 1, then .T and f(:r) an~ nearly 1, but
.f'(:c) is large and the computed value is highly sensitive to change in :r.
2. f(:r) ·may be snuLll wh·ile :r: and f'(:r:) arc nut;
The Taylor's series for sin :r m~ar 7r or e-'c . with .7: larg<~ exhibit this form of ill
concli tioning.
3. ;r n1.ay be la:r.qe while f'(:r) and .f(;r) aTe not;
The evaluettion of sin :r: for :r 'near 10000007r is poorly conditioned.
One can alsci say that computation is ill-coiiclitioned and this is the same as saying
it is mm1erically m~stahle. Th<~ condition number gives more information than just.
saying somet.hirig is nmn<~rically m1stable. It is rarely possible to obtain accmate values
fo~· condition numbers but cnw rarely neecls much accuracy; an order of magnitucl<-~ is
often enough to know.
Note that is almost impossible for a method to be numerically stable for an ill-
conditioned problem.
Example 1.3.1. · Au ill-conditi~ned line intersection problem consists in
computing the point of iilt<-~rscc:tion P of two nearly parallel lines. It it> cle~r t~mt. a
minor change in one line changes the point of intersection to (P + 8P) wh1ch rs far
hom P. A mathematical model of this problem is obtained by introducing a coordinate
system and writing <~quations

y = a 1 :r + b1
Y = a2:r: + b2
8 N unwrical Methods in Computational Engineering

what leads to solving a syst<~m of equations

(J,l:J: - y = -bl
a 2 :r: - y = -b 2

with the a1 and a2 lWctrly equal since the lines are nearly parallel. This rmme'rical
problem is 1mstablf~ or ill-cowlitiolHxl, as it reflects the ill-conditioning of the original
problem. ·
A mathematical model is obtained by introducing a coordinate system. Any two
vectors will do for a lmHis, and if we chm;e to use the unusual basis

bl = (0.5703958095, 0.8213701274)
b2 = (0.5703955766, 0.8213701274)

then every w~ctor x um be <~xpressccl as

so that the equations of the two lines in thiH coordinate system are

y = -0.0000000513 + 0.9.999998843.'r:
y = -0.0000045753 + 1.00000159G:r:
with tlw point of intersectioi1 P with coordinates
( -0.8903429339, 0.8903427796). Note that mathematical model is very ill-conditioned:
a change of 0.0000017117 in tlw data makes the two lines parallel, with no solution.
The poor choice of a basis in the given example made the problem poorly concli-
timwd. In more complex prohl<~ms it is not so easy to see that a poor choice has been
macle. In fact, a poor choiu~ is sometimes the most natural thing to do. For example, in
problems involving the polynomials, mw naturally takes vectors based on 1, :r:, :r 2 , ... , :;:n
as a baHis, bnt thes<~ an~ terribly ill-conditioned even for n moderate in size.
Example 1.3.2. Syst<~m of equations (input information)

2:r: + Gy = 8
2:r: + G.0001y = 8.00()1
have a solutions (ontpnt infonnation) :r: = 1, y = 1. If th<~ coefficients of second equation
slightly change, i.<~. if one takes the <~qnation

2:r: + G.99999y = 8.00002,

the solutions are :r: = 10, y = -2. This is typicalrouud-off <~nor.

Errors in nH~thods occm· usually because in mmH~rical mathematics the problem to
1><~ solved is replac<~d l>y auotlwr mw, dos<~<l to original, which is easi<~r to solve.
Example 1.3.3. Iutcp;ml .J::.' f(:r:)rl:r: can he approximately calculated, for <~xawple,
by replacing the hmctiou f by souH~ polynomial P on segment [a, b], which is iu som<-~
S(~nse dose to given fnuctiou. However, for approximative calculation it is posHihl(~ to
US<~ the SUlll

L f(:I:.i)!J.:r:.,:.
·i=l

Iu both cases t.lw llwtJwd error occm·s.

Iu some seus<~. t.hc rmmd-off ITror an~ also md.lwd cnorH. Smn of all errors nml-ws
tlw total <~nor.
Souwtiuws. how<~vcr. au ot.lwrwise attractive uwthod can lH~ uustahl<~. This uwans
that auy. rmuHl~)ff <~nm that. h<~couH~s "mixed iuto" the cakubtio;1
. at au <~<trly. stage is
 Le:-;son 1 - Matlwmaticf:> awl Computer Science

:·mu:c~ssivdy magnifif~d nut.il it co11ws to swamp the true answer. An unstable~ method
would he nsdul on a hypotlwtical, perfect computer; but in this imperfr~ct world it is
lH~ccssary for us to n~quin~ that algorithms be stable or if unf:>table that we use them
with great caution. Here is a simple~, if somewhat artificial, example of an unstahh~
algorithm (see [4], p.20).
Example 1.3.4. Suppose that it is d<~sired to calculate all integer pow<~n; of the
so-call eel "Golden Mean,'' tlw munht~r given by

(1.3.2) <I> =J52-1. ~ 0.61803398

Powers of <I>n satisfy simple n~cnrrence relation

(1.:3.3)

Well, knowing the first two values <I> 0 = 1 and <I> 1 = 0.61803398, we can apply (1.3.3)
by subtraction, rather than a slow<~r multiplication by <I>, at each stage. Unfortunat<~ly,
the recurrence (1.3.3) also has another solution, namely the value - ~ ( VG + 1). Since
the recunenc:e is linear, and since this undesired solution has magnitude greater than
unity, any small admixture of it iHtroduced by roundoff errors will grow exponentially.
On a typical machine with 32-hit word-length, (1.3.3) starts to give completely wrong
answers by about n = 16, at which point <I>n is down to only 10- 4 . Thus, the recurrence
(1.3.3) is unstabl<\ and cannot be used for the purpose stated.

On the end of this s<~ction, it remains the questjon: How to estimate errors and
uncertainty '?
One almost uewer knows the error in a computed result unless one already knows
the true solution, and so ouc must settle for estimates of the error. There are three
basic approaches to error estimates. The first is forward error analysis, when one
u:-;es the theory of tlw mmHTical mdlwd plus information about the uncertainty in the
problem and attempts to predict the error in the computed result. The information one
might use includes
- the size of ronnel-off,
- the measurement enors in problem data,
- the truncation enors in obtaining the nunwrical model from the mathematical
model,
- the differences betweeu the nmthematic:al model and the original physical model.
The secoud approach is backward error analysis, where one takes a computed
solution and sees how close it comes to solving the original problem . The backward error
is often ~alled the the residual in equations. This approach requires that the problems
involve satisfying some couditions (such as an equation) which can be ~ested with a
trial solution. This prevents it hom being applicable to all numerical computations,
e.g. nmnerically estimating the value of 1r or the value of an integral.
. The third approach is experimental error analysis, where one experiments
with changing the computations, the method, or the data to see the effect they have on
the results. If one truly wants certainty about the accuracy of a computed value, then
one should give the problem to two (or even more) different groups and ask to solve
it. The groups arc not allowed to talk together, preventing a wrong idea from being
passing around.
The relationship between these. three approaches could be illustrated graphically,
10 Numerical Methods in Computational Engineering

as given in the following figure.

_)+

Baci<w2~
.
P~-
. ~~~-t_--·------
.---------..

- CornPut .
. eo. .
·
--------~.x
.
+
.·
~~
. .
.
+

Forw3rd
Error

Error . ./ +
\

a ·-- --------- y
Exact

P = true problem and data x = exact result for true problem

Q = perturbed probkm and data y = computed result for true problem
= exact ie:mlt for perturbed problem
+= computed results u>ing other .
. methods, programs, etc.

Figure 1.3.1

1.4. Programming

There are several a.reas of knowledge ahont programming that are 1weded for sci-
entif-ic computation. These induclf~ kuowledge about:

- The programming languag<~ (FORTRAN, Pascal, C, .Java, Mathematic:a (MatCAD,

Matlab).
- The computer system in which the language runs
- Program debugging and verifying the correctness of results
- Computation organization awl expressing tlwm clearly.

Debugging programs is au art as well a.s a science, and it must be learned through
practice. There are sev(~ral dfectivc tactics to use, like:
- Intermediate output
- Consultations about program with experienced user
- Use compiler and d<~lmgging tools.

Sonw abilities of compilers:

- Cross- n~fereuu~ tal >ks
- Tracing
- Snbscript dwckiug
- Laugnag<~ stand<mls dwckiug.

Smm~ hints:
- Use lots of cmm1wnts
- Usc meaniugfnlwmws for variables
- Make t.lw typ<~s of variables obvious
- Use simph-~ logical coutrol stnH:tnres
- Usc program p<u:kag<~s and systems (Mathematic:a, Matlab) wherever possible
- Ust strnctnn~d programming
- Use (if possible) OOP t.<~dmics for technical problems.

1. 5. Numerical software

Tlwn~ an~ s<~V<~ral .ionrnals tha.t publish iwlividnal computer programs:

- ACM Tmusactiow-; 011 l\!Iat.hcntatical Soft.wan~ (IMSL, Int<~mational l\!Iatlwmatical
Scientific Lil mtry)
 Lesson I - Matltc~matic:s aucl Computer Science 11

- Applied Statistics
- BIT
- The~ Compnte~r .Tonrnal
- N m1wrische Matlwmatik
The ACM Algorithms se~rics contains more than thousand items and is available~ as
the Collected Algorithms of the Association for Computing Machinery.
Three general libraries of programs for numerical computations are widely available:
IMSL Inkmational Mathematical Scientific Library
NAG Nnnwrical Algorithms Group, Oxford University
SSP Scie~ntific Snbroutiw~ Package, IBM Corporation
Then~ are a substantial nmnher of important, specialized software packages. Most
of the packages listed hdow an~ available from IMSL, Inc.

MP Multiple Precision Arithmetic Package

BLAS Basic: Liuc~ar Algd)ra Subroutines
DEPACK Differential Equation Package
DSS Diff'cn~utial System Simulator
EISPACK Matrix Eigensystems Routines
FISHPACK Routines for the Helmholtz Problem in Two or Three Dimensions
FUNPACK Special Function Subroutines
ITPACK Iterative Methods
LINPACK Linear Algebra Package
PPPACK Pie~c:ewise Polynomial and Spline Routines
ROSEPACK Rolmst Statistics Package
ELLPACK Elliptic Partial Differential Equations
SPSS Statistical Package for the Social Sciences.

User interface to the IMSL library:

PROTRAN .John R Rice, Purdue University

1.6. Case study: Errors, round-off, and stability

Example 1.6.1. Solve qna,dratic formnla

a:r 2 + b:r + c = 0

with 5, 10, 1G, ... 100 decimal digits ~1sing FORTRAN and Mathematica code. Take a=
1, c = 2, b = G.2123(10)10G.2123. Use the following two codes:
DIS=SQRT(B*B-4. *A *C) DIS=SQRT(B*B-4. *A *C)
Xl=(-B+DIS)/(2*A) IF(B.LT-.0) THEN
X2=(-B-DIS)/(2*A) , X1=(-B+DIS)/(2*A)
ELSE
Xl=( -B-DIS) I (2* A)
END IF
X2=C/X1
Compare the obtained results.
There are two impmt;-mt lessons to be learned from example 1.6.1.:
1. Round~off error can completely ruin a short, simple computation.
2. A simple change in the method might eliminate adverse round-off
effects.
Example 1.6.2. Calculation of 1r.
Using five following algorithms, calculate 1r in order to illustrate the various effects
of round-ofF on somewhat different computations.
12 Numerical Methods in Computational Engineering

Algorithm 1.6.2.1. Infinite altm·nate series

7r = 4(1 - 1/3 + 1/5- 1/7 + 1/9---:. ")

Algorithm 1.6.2.2. Taylor's series of arcsin(1/2) = 1r /6

2
(0.5) 3(0.5) 4 3_ X 5(0.5) 6
r:
7r = G(() .o + .
--. +1X + 1 X
+ ... )
2x3 2x4x5 2x4x6x7

Algorithm 1.6.2.3. ArchimecleR' method. Place 4, 8, 16, ... , 2n triangles inside a

circle. The area od each triangle is 1/2 sin( H). The values of sin( H) are computed by
the half angle formula ·
sin(H) = }[1 - cos(2H)]/2
and
cos(11) = ) I - sin 2 H.
The calculation is initialized by sin(7r/4) = cos(7r/4) = 1/J'i. As the number oftriangles
grows, they fill up the circle ancl their total area approaches 1r. (Archimed carried a
similar procedure by hand with 9G triangles and obtained

- . 1137
' ' c
3.1409 ... - 3 < 7r < 3 1335 -- 3.1428
- )
... )
8069 9347

Algorithm 1.6.2.4. Instead of inscribing triangles in a circle, we inscribe trape-

zoids in a quarter circ:k. As a mnuber of trapezoids increases, the sum of their areas
approaches 1r /4.
Algorithm 1.6.2.5. Monte Carlo integration.
Monte Carlo integration for .J;~ l~:r: d:r is proceeded by choosi11g a pair (:t, y) at
random with :r:, y in [0, 2], and comparing y with 2/(1 + :r). If y :::; 2/(1 + :r) the--m the
point (:r:, u) is nll<lc-~r thc~ curve y = 2/(1 + :r:) and variable SUM is increased by 1. After
M p<tirs, the int.egntl is cstimatc-~<1 by the fraction SUM/M of points that are under tlw
cnrvc~. Use this algorithm to estimate~ the area of the quarter circle.

Bibliography (Cited references and further reading)

[1] Milovanovi{, G.V., Numerical Analysis I Naucna kn.iiga, Beograd, 1988 (Sc~r­
bian).
[2] Milovanovi<:, G.V. and D_jonl_jevif:, D_j.R.., Programiranje numer·ickih metoda
na FORTRAN jez·ih;u. Instit.nt za clokmm~ntaci.iu zastite na radn "Edvanl
Kanld,j", Ni~, 1981 (Serbian).
[3] Hoffman, .J.D., Numerical Methods for Engineers and Scientists. Taylor
& Francis, Boca R.aton-Lowlon-Ncw York-Singapore, 2001.
[4] Pn~ss, W.H .. Flaum~ry. B.P ... Tc~nkolsky, S.A., ancl Vetterling, W.T., N'U,mer·ical
Recepies - The Art of Scientific Computing. Cambriclgc~ University Press,
1989.
[5] Rice, .J.R.., N'Umer-ical Methods, Software, and Analysis. McGraw-HilL New
York. 1!J8~j.
[G] Abramowitz. M .. and St.t~p;mL I.A .. Handbook of Mathematical Functions.
Natimml Bnn~an of St.<mdmds, Applic~d Matlwmatics Series, Washington, 1%4
(n~printc~d 1!J()i:) hy DovcT Publications, N cw York).
[7] Hildebrand. F.B., IntToduct£on to Numerical Analysis. Mc.Graw-Hill, New
York.. 1!J7 4.
 Lesson I - Mathematics and Computer Science 13

[8] Stoer, J., ancl. 'Bulirsch, R., Introduction to Numerical Analysis. Springer-
Verlag, New York, 1980.
[9] Kahaner, D., Moler, C., and Nash, S., Numerical Methods and Software.,
Prentice Hall, Englewood Cliffs, 1989.
[10] Johnson, L.W., and Riess, R.D., Numerical Analysis. 2nd ed., Addison- Wesley,
Reading, 1982.
[11] Wilkinson, .J.H., Rounding Errors in Algebraic Processes. Prentice-Hall,
Englewood Cliffs, 1964.
[12] Milovanovic, G.V. aml Kovacevic, M.A., Zbirka resenih zadataka iz nu-
mer·icke analize. Nauc:na kn.iiga, Beograd, 1985. (Serbian).
[13] Forsythe, G.E., Malcolm, M.A., and Moler, C.B., Computer Methods for
Mathematical Computations. Englewood Cliffs, Prentice-Hall, NJ, 1977.
[14] IMSL Math/Library Users Manual, IMSL Inc., 2500 City West Boulevard,
Houston TX 77042.
[15] NAG Fortran Library, Numerical Algorithms Group, 256 Banbury Road, Ox-
ford OX27DE, U.K.
Faculty of Civil Engineering Faculty of Civil Engineering and Architecture
Belgrade ', · Nis
Master Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON II

2. Linear Systems of Algebraic

Equations: Direct Methods

2.1. ELEMENTS OF MATRIX CALCULUS

2 .1.1. LR factorization of quadratic matrix

During solution of systems of linear equation there is often case to present a

quadratic matrix in a form of product of two triangular matrices. This section is devoted
to this problem.
Theorem 2.1.1.1. If all determinants of form

an alk:
6A:= (k=1, ... ,n-1)

are different fi-om zero, the 1natrix A = [aij ln. x n can be written in form

(2.1.(1) A=LR,

where L lower, and R upper triangular matrix.

Triangular matrices L and R of order n are of following forms:

(2.1.1.2) L = [lij]nxn (l,ij = 0 ¢= i < _j),

(2.1.1.3) R = [r-.-i.ilnxn ('r",;,_j = 0 ¢= i > j).

Decomposition (2.1.1.1), known as LR factorization (decomposition), is not unique,

having in mind the equality

LR = (cL)( ~R) (Vc# 0).

c

Nevertheless, if diagor1alelements of matrix R (or L) take fixed values, not one being
equal to zero, the decomposition ifl unique. In regards to (2.1.1.2) and (2.1.1.3), and
having in mind
ma.or:('i,j)
a,i.i = L lik:Tkj ('i, j = 1, ... , n)
A:=l

15
 16 Numerical Methods in Computational Engineering

~he elements of matrice.s L and R can be easy determined by recursive procedure, giving
rn, advance the values for elements T·i·i(=l= 0) or li·i(=/= 0) (i ____: 1, ... , n). For example, if
given numbers ·r.;,.;,(::j: 0) (i = 1, ... , n), it holds

_ an
ln- - .
Tn

(i=2, ... ,n);

('i=2, ... ,n).

(j = i + 1, ... , n) ;

. In similar way can he defined recursive procedure for determination of matrix ele-
ments of matrices Land R, if the numbers l,ii(=l= 0) ('i = 1, ... , n) are given in advance.
In practical applications one usually takes '~"·i·i = 1 ('i = 1, ... , n) or l·i·i = 1('i = 1, ... , n).
Very frequent case in application is of multi-diagonal matrices, i.e. matrices
with elements different from zero on the main diagonal and around the main diag-
onal. For example, if a.;,.i =I= 0 for li - .il : : ; 1 and O.ij = 0 for li - .il > 1 , the
matrix is tri-diagonal. The elements of such a matrix are wmally written as vectors
(a2, ... , a..n.), (bl·,: .. , bn), (cL ... , Cn-1), i.e.

h cl 0 0 0
(],2 b2 c2 0 0
(2.1.1.4) A= 0 (},3 b3 0 0

0 0 0 an bn

If a.;,.i 1- 0 (i'i, - .i I : : ; 2) and o..;,.i = 0 (I ·i - .i I > 2), we have a case of five-diagonal matrix.
Let m; now suppose that tri-diagonal matrix (2.1.1.4) fulfills the conditions of Theorem
2.1.1.1. For decomposition of snch a matrix it is enough to suppose that

fJl () 0 0 0
CY.2 (-J 2 0 0 0
L= 0 (Y.3 /h 0 0 ({31(32 ... f3n =/= 0)

0 0 () CYn fJ.n
awl
1 11 0 0 0
0 1 12 0 0
R= 0 0 1 0 0

0 () 0 0 1
 · Lesson II - Linear Systems of Algebraic Equations: Direct Methods 17

By comparing corresponding elements of matrix A and matrix

!31 /31"(1 0 0 0
Ct2 Ct2'Y1 + !32 f32'Y2 0 0
LR= 0 C't3 C't3'Y2 + !33 0 0

0 0 0 an an'Yn-1 + f3n

we get the following recursive formulas for determination of elements Cti, f3i, 'Y·i:

c1
'Y1 = /31)
Ci
'Y·i = - (i = 2, ... , n - 1) ,
13i

2.1.2. Matrix eigenvectors and eigenvalues

Definition 2.1.2.1. Let A complex quadratic matrix of order n. Every vector x E en
different from zero-vector is named eigenvector of matrix A if there exists scalar .A E C
such that

(2.1.2.1) A.i = >.x.

Scalar .A is then na.med the conesponding eigenvalue.

Considering that (2.1.2.1) can be written in form

(A- .AI)x = 6,
one can conclude that equation (2.1.2.1) has non-trivial solutions (in:£) if and only if
det(A- .AI) = 0.

2.2. DIRECT METHODS IN LINEAR ALGEBRA

2.2.1. Introduction
Numerical problems in linear algebta can be classified in several groups:
1. .Solution of system of linear algebraic equations

A.i= b,
where
. . A regular matrix, calculation of determinant of matrix A, and matrix A
mverswn;
2. Solution of arbitrary system of linear equations using least-square method;
3. Determination of eigenvalues and eigenvectors of given quadratic matrix;
4. Solution of problems in linear programming.
For solutioi1 of these problems, a number of methods is developed. They can be
sepai·ated in. two classes, as follows.
The first class contains so-called direct methods, known sometimes as exact meth-
ods. The basic characteristic of those methods is that after final number of transfor-
mations (steps) one gets the result. Presuming all operations being performed exact,
the gained result would be absolutely exact. Of course, because the performed compu-
tations are performed with rounding intermediate results, the final result is of limited
exactness.
 18 Numerical Methods in Computational Engineering

The second class is made of iterative methods, obtaining the result after infinite
number of steps. As initial values for iterative methods are usually used the results
obtained by some direct method. .
Note that at solution of systems with big number of equation, used for solution of
partial differential equations, the .iterative methods are usually used.

2.2.2. Gauss elimination with pivoting

Consider the system <)f linear algebraic equations

an:rl + a12X2 + · · · + a1nXn = b1

a21:r1 + (1,221:2 + · · · + a2nXn = b2
(2.2.2.1)

or, in matrix form

(2.2.2.2)

where

all
;r;ll
A=
' (],21 a2n
aln l' ~
b=
rbtl
b2
. ' x-
X2

[
(],.:1.1 CLnn b:n. r:r:n
Suppose that system of equation (2.2.2.2) has an unique solution. It is very known that
solutions of system (2.2.2.1), i.e. (2.2.2.2), can be expressed using Crammer's rules

<let A.i
:r.i = ('!: = 1, 2, · · · n.),
· clet A

where matrix Ai is obtained from matrix A by replacing ·i-th column by vector b.

Nevertheless, these formulas are inappropriate for practical calculations because for
calculation of n + 1 determinants one needs a big number of CE1.lculations. Namely, if
we would like to calculate the value of determinant of ·n-th degree by developing of
determinant through rows or columns, it would be necessary to proceed Sn = n! - 1
additions and Mn rv n!(r.- 1) multiplications (n > 4), what gives the total number of
·calculations P~~, = 1\!In. + Sn ~ n!e. Supposing that one operation demands 100p,s (what
is the case with fast computei·s), the total time for calculation of value of determinant
of order thirty (n = 30) wonlrl take approximately 2.3 · 10 20 y<~ars. Generally speaking,
such one procedun~ is practically mmsable for determinants of order n > G. One of the
most suitabk direct nu~tlw<ls for solution of system of linear equations is Gauss method
of elimination. This nwtho<l is based on re<luction of system (2.2.2.2), using equivalent
transformations, to the triangular system

(2.2.2.3) Rx=c,
when~

R=
'I'll
Tin
1'2n I
r Tnn '
 L<~ssou II - Liuem Systems of Algebraic Equations: Direct Methods 1!)

System (2.2.2.3) is- solv<~<l sw:ccssivdy starting fmm the last equation. Namely,

"/1.
1
·1· - -.. -
···i-
I ii
(r·"i.- L r.;,,,.r.h) ('i=n-1, ... ,1).
l.:=·i.+l

Note that coeffici<~uts .,.,, # 0, lH~canse of assumption that systt~m (2.2.2.2), i.e. (2.2.2.3)
has an unique solution.
We will show uow how system (2.2.2.1) can he reduced to equivalent system with
triangular matrix.
Supposing an # 0, ld us compute first the factors
(J,i.l
'/nil =- ('i = 2, ... 'n),
au
and then, by multiplication first equation in system (2.2.2.1) by m. and subtracting hom
·i-th equation, one gds the system of n - 1 equations

o.W :r:2 + ... + a~~? :rn = b~ )

2

(2.2.2.4)

where
o..~.~) = b.~ ) =b.;,-
2
aij- ·m.;,1o.1.i, ·m.ilbl (i,j = 2, ... ,n).
Assuming o. 22 -::f. 0, and applying the same procedure to (2.2.2.4), with
a..;.2
Tn.;,2 =- ('i = 3, ... , n),
0.22

one gets the system of n - 2 equations

(3)' (3)'. - (3)
0.33 .C3 + · · · + 0.3nXn- b3

where
(3) - (2) ' . (2) l (3) - ,. (2) - ' .. b(2) (. . 3 )
a..;:,; - a.ij - 111..,.2o.2j ' J.;, - J.;. - rn.1.2 2 ?., J = '· · · 'n ·

Continuing this procedure, after n. - 1 steps, one gets the equation

From the' obtained systems, taking the first equations, one gets the system of equations
. (1),. (1),. (1) ... (1) - b(l)
a.ll .r.1 + (1,12 .1.2 + al3 X3 + + a.ln Xn - 1
(2) '1.. + rl (2) 'I: +
(1•22 + '"'2n
n (2) X - b(2)
· ·2 · '"'23 · ·3 · · · · ·n - 2
(3) (3) - l (3)
0.33 X3 + · · · + 0.3n Xn - J3
20 Numerical Methods in Computational Engineering

1
where we put ag) = a.i..i, b.) l = b.i..
The presented triangular reduction, or as often called Gauss elimination, is actually
cleterrnirmtion of coefficients . ·

(A:)
o,·i.k
(1.:)'
aH:
(A:+l) - (1.:) ' . (1.:)
ai.i - 0'-i.,j - m'l.kJLAj '

b)/,:+l) = b.)A:) - 'IILi.A:bi~:) ('i, j = f.:+ 1, ... , n)

for k = 1, 2, ... , n - 1. Note that the elemer1ts of matrix R and vector care giv(~n a.s
.
r··-a('i.)
'/,.} - '·i.J ' c·-b('i)
' /. - ri, ··(·z--1
) - i ... '
·r'·J·-·z·
IJ'. - ' ... i
'1')
(., .

In onler the presented reduction to exists, it is necessary to obtain the condition

ai:~? # 0. Elements o.i'2
are known as main denwnts (pivotal elements or pivot). As-
smni.ng matrix A of systcn.l (2.2.~.2) l?eing regular, the conditions
obtamed by pernmtatwn of (Xpwtwns m syst(:m.
# 0 are to be ai'2
Moreover, from the point of vi(~W of accuracy of results, it is necessary to use so
known strategy of choice of pivotal elements. Modification of Gauss elimination method
in this sens(~ is calle(l Gauss mdJtod with clwic(~ of pivotal element. In ac:c:orclance to
this method, for pivotal ckmeut in k-th elimination step one takes the element a;.~), for
which holds ·
. (A:) . . (A:)
Ia·-!·
1
•·
I= l.:<·i.<n
max lo, 1. I, 1
''

with permutation of k-th awl T-th row.

If one obtains in addition to p<~nnutation of equations the pcnuutation ohmknowns,
it is the best w<ty to tak<~ for pivotal dement in the k-th elimination step tlH~ dement
a.;';?,fm which it holds
Ia.,(!.:)I
.., -- ·
lllc)X
/i:~·I.,.J~'/1.
IU.·i.j(1.:) I
.

with pPntmtatiou of k-th and r-t.l1 row (equations) awl k-th and s-th colunm (un-
lntowJts). Such method is call(~<l th(~ nH~tlwd with total choice of pivotal (~lc~mt-~nt..
Orw can show (sc~c [1], pp. 2:3:)-2:~4) that total number of calculations by applying;
Gauss methocl i~

For ·11. hip; f-~noug;h, om~ g;cts N(n) ~ 2n 3 j:3. It was long time opinion that Gauss nu~tltod
is optimal n~g;anling llllllllHT of computatiow,;. Nowad;:tys, V. St.rassPll.. by involvillJ!,
it(~rativ(~ alg;mitlm1 for umH.iplyiiLg awl invc~rsc of matrices, gave a new md.lwd for
solntiou of system of liucar cqnatious, by which th<~ munhcr of computations is of orrl<T
n 10 !2;" 7 . Strasscn uwt.lwd is t.lms better than Gauss method lug 2 7 < :).
Triaugula.r rcdntJ.iou ohtaius simpl(~ computation of system dd<~nuinant. Namdy.
it. holds
•t A_ (1) (2) (n)
( l<. • - o.ll n.22 · · ·a.,.., ·

\VlH'll usPd Ganss uwtlwd with dJOicc of pivot;d dement. ow~ slwuld take care ahont.
llllllllwr of JH~ntmt.at.ious of rows (awl cohmms i>y using md.hod of total choice of pivotal
ckuw11t.). what iu!inenc<~s the sigu of dctenuiu;lut.. This way of clctermiuaut cakulat.imt
i~ high dhci(~llt. For cxawpl<·. for udcnlatio11 of dct.tTmin;mt of order n = :)(). ouc uccds
O.los. p1-csmuing that. ouc arit.huwl.ic O]H~r;li.iou t.c1kcs lOtt..'i.
 · L<~sson II - LilH~m Syst<~ms of Alg·ehraic Equations: Direct Methods 21

2.2.3. Matrix inversion using Gauss method

Let A = [a.;.i ]., x n 1H~ n~gular matrix ancl let

X=
[ :1:21
'11
:r:12
:1:22 <ln
:r:2n 1- [:I\
~

:D2

:/:.;,_[ :r:n2 :r: '11.'11.

be its inverse matrix. V<~ctors :1! 1 , :1! 2 , . . . , :l!n are first, second, ... , n-th colmnn of matrix
X. Let us now define v<~ct.m·s (71 , P2 , ... £;,, as

R<-~ganling to eqnality

the problem of determination of inV<-~rse matrix can reduce to solving of n systems of

lirwar equations

(2.2.3.1) A:l!.; =f.;, ('i = 1, ... , n).

For solving of syst<~m (2.2.3.1) it is convenient to nse Ganss method, taking in account
that matrix A app<-~ars as a matrix of all syst<~ms, so that its triangular reduction shell
be done once only. By this proccdun~ all the transformations necessary for triangular
reduction of matrix A should he applied to the unit matrix I = [e'1 e2 ... €.n.] to(i. In
this way matrix A tnmsfonns to triangular matrix R, and ma.trix I to matrix C =
[21 22 ... C.,J Finally, triangular systems of form

R:l!.i = C.; (i = 1, ... , n)

should be solved.

2.2.4. Factorization methods

Factorization methods for solving of system of linear equations are based on fac-
torization of matrix of syst<~m to product of two matrices in such form that enables
reduction of system to two systems of equations which can be simple successive solved.
In this section we will show up at the methods based on LR matrix factorization (see
Section 2 .1.1.).
Given the system of <:~quations

(2.2.4.1) Ax=b,
•
with quadratic matrix A, which all main diagonal minors are zero different. Then,
based on Theorem 2.1.1.1, it exists factorization matrix A= LR, where L lower awl
R upper triangular matrix. The factorization is unique defined, if, for example, one
adopts unit diagonal of matrix L. In this case, system (2.204.1), i.e. system LR:r = b
can be presented in equivalent form

(2.2.4.2) Lz7 = b, R:.Z = i].

Based on previous, for solving of system of equations (2.2.4.1), the following method
can bP formulated:
1. Put l-i·i = 1 ('i ___.: 1, . .. ,n);
22 Numerical Methods in Computational Engineering

2. Determine other elements of matrix L = [l.;,jJ.nxn and matrix R [rij]nxn (see

Section 2.1.1);
3. Solve first system of equations in (2.2.4.2);
4. Solve second system of equations in (2.2.4.2).
Steps 3. and 4. are simple to be performed. Namely, let
b= [b1 b2···bn]T, ;1/= [:Y1 :IJ2···Ynf, X= [x1 X2 ... :rnf· ·
Then
i-1
:1J1 = b1, Yi =b.,:- Llik:IJk (i = 2, ... ,n)
k:=1
and
n
Yn 1 ~
:En=-, ;r;.;, = -(y.,:- 6 Tik::r;k) (i = n- 1, ... , 1).
Tnn T··
'/,'/, k:=i+1

The method presented is known in bibliography as method of Cholesky. In the case

when matrix A is normal, i.e. symmetric and positive definite, the Cholesky method can
be simplified. Namely, in this case one can take that L = RT. . Thus, the factorization
of matrix A in form A = RTR should be performed. Based on formulas from Section
2.1.1 for elements of matrix R it holds:
ru = JUll
(],1j
T1: = -· (.j = 2, ... , ·n),
.7 'tn

·i-1

T·i:i. = . o,;,.;, - L ·rL.

1.:=1

('i = 2, ... , n).

1 i-1

. = -'{'· ( O,ij.
T.;,7· - ~ 'f"l.:·iTA:·i)
6 ·' (j = ·i + 1, ... , n) .
't:t. A:=l

In this case the systems (2.2.4.2) become

RT;1} = b, Ri = f].
Remark 2.2.4.1. The determinant of normal nwtrix can be calculated by method of
square root as
<let A~ (rn T22 ... Tnn) 2 .
Factorization mdhods are SlH~cially conw~nient for solving of systems of linear equa-
tions where matrix of systems does not change, hut only free vector b. Such systems
are very frequent in cng;im~<~ring.
Now it will be shown that Gauss method of elimination can be interpret<~d as LR
factorization of matrix A. Take matrix A such that during the elimination process
pernmtation of rows awl columns should not be pf~rfonnecl. D<~note the starting system
as A (1 ) .7! = {} 1 ). Gauss diminatiou proceclme gives n - 1 equivalent systems A (2 ) :l:' =
b( 2 ) ... A(n);(;' = {;(n). where matrix A(l.:) is of form
(1) (1) (1) (1)
(1,11 (},12 o,H: a,1n
(2) (2) (2)
(1,22 (],2/i: (},2n

(!.) (k:)
(J,·nl.: (J.n11
 Lesson II - Linear Systems of Alg·ebraic Equations: Direct Methods 23

Let us analyze mo'Tlification of elements a.;,j ( = o..)~)) during the process of triangular
reduction. Because, f{n· k = 1, 2, ... , n- 1,
(h:+1) _ (h:) (A:)
a..i.j - a.,.i - m.;,A,ah::i (i, .i = k + 1, ... , n),
and
(1.:+1) - (1.:+1) - - (k+1) ()
a.;.l - a.;,2 - ... -a.;,~;, = (i=k+1, ... ,n),
by summation we get

('i ::; j)

and

(i > j).

By defining m,;,.; = 1 ('i = 1, ... , ·n), the last two equalities can be given in form

(2.2.4.3) (i,j=1, ... ,n),

where p = min('i, .i). Equality (2.2.4.3) is pointing out that Gauss elimination procedure

l
gives LR factorization of matrix A, where

'f'12

[ '11/.21
1 1
['11 'f'22 'In
't2n
L=

'lrl,n1 'fr/,n2 J R=

Tnn

and Tk.i = ai,~?. During program realization of Gauss method in order to obtain LR
factorization of matrix A , it is not necessary to use new memory space for matrix
L, but it is convenient to load factors m.iA: in the place of matrix A coefficients which
are annulled in procest> of triangular reduction. In this way, after completed triangular
reduction, in the memory space of matrix A will be memorized matrices L and R,
according to following scheme:
A* LR
Cormider that cliagoual ele1i1ents of matrix L, all equal to unit, should not be
memorized.
Cholesky method, based on LR factorization, is used when matrix A fulfils condi-
tions of Theorem 2.1.1.1. Neverthdess, usability of this method canbe broaden to other
systems with regular matrix, taking in account permutation of equations in system. For
factorization is used Gauss elimination method with pivoting. There will be LR = A',
where matrix A' i8 obtained hom matrix A by finite number of row interchange. This
means that in elimination process set of indices of pivot elements I = (Pl, ... , Pn-1),
where PA: it> number of row from which the main element is taken in k-th elimination
step, should be memorized. By solving of system A.i = b, after accomplishing a pro-
cess of factorization, .according to set of indices I, coordinates of vector b should be
permuted. In this way the transformed vector b' is obtained, so that solving of given
system reduces to successive solving of triangular systems

(2.2.4.4) .L y__, = ),..., l R.1. . . -_ y.....

24 Numerical Methods in Computational Engineering·

2.2.5. Program realization

This section is devoted to software realization of methods previously exposed in this
chapter. For successful following of material in this subchapter it is necessary knowledge
exposed in all previous suLchapters of this chapter. In presented subprograms the
matrices are treated as vectors.

Program 2.2.5.1. Subprogram for matrix transpose MTRN is of form:

SUBROUTINE MTRN (A, B, N, M)

c
C TRANSPONTOVANJE MATRICE A
c
DIMENSION A(1)' B(1)
IC=O
DO 5 I=1, N
IJ=I-N
DO 5 J=1, M
IJ=IJ+N
IC=IC+1
5 B(I C) =A (IJ)
RETURN
END

Parameters in the list of subprogram parameters have tlw following meaning:

A - input matrix of type N x M, treated as vector of length NM (taken in form
column by column);
B - output matrix of type M x N (B = AT). Matrix is treated in the same way as
matrix A.

Program 2.2.5.2. Subprogram for multiplication of matrices A (of dimension

N x M) and B (of dimension !VI x L) is of form

SUBROUTINE MMAT (A, B, C, N, M, L)

c
C MATRICA A TIPA N*M
C MATRICA B TIPA M*L
C MATRICA C TIPA N*L
C MNOZENJE MATRICA C=A*B
c
DIMENSION A(1), B (1), C (1)
IC=O
I2=-M
DO 5 J=1,L
I2=I2+M
DO 5 I=1, N
IC=IC+1
IA=I-N
IB=I2
C(IC)=O.
DO 5 K=1, M
IA=IA+N
IB=IB+1
5 C(IC)=C(IC) + A(IA)*B(IB)
RETURN
END
 - L<~sson II - Linear Systems of Algebraic Equations: Direct Methods 25

Output matrix'-G (C =Ax B) i:-:; of dimension N x L.

Program 2.2.5.3. Ld u:-:; write a program for computing matrix BT A, by u:-:;iug

previously giv<~n subprograms, for given matrices A and B. Let matrix A be of type
N x ]\1[, and matrix B of type N x K (with maximal munber of matrix elements for
both matrices 100).
This progntm has the following fcmn:

DIMENSION A(100), B(100), C(100)

OPEN(8,FILE='MTMM.IN')
OPEN(S,FILE='MTMM.OUT')
READ (8,10) N,M,K
10 FORMAT (3I2)
NM=N*M
NK=K*M
KM=K*M
READ (8,20) (A(I), I=1, NM), (B(I), I=1, NK)
20 FORMAT(16F5.0)
CALL MTRN( B, C, N, K)
CALL MMAT (C, A, B, K, N, M)
WRITE (5,30) ((B(J), J=I, KM, K), I=1, K)
30 FORMAT (5X, 'MATRIX C=B(TR)* A'// (2X,4F6.1))
CLOSE(8)
CLOSE(S)
STOP
END
Test of propp-uu, being proe<~<~de<l with matrices

r~~
J 0

[~
-:~

A= ()
4
1
1
-2
2]
5
0 '
and B=
2
4
-1
~62 j '
-2 J 1 J -1 5 1

gave the following n~sult:

MATRIX C=B(TR)* A
1.0 2.0 -5.0 -1.0
-3.0 21.0 11.0 29.0
-8.0 -19.0 -9.0 -27.0

Program 2.2.5.4. Method of Cholesky for solving of system of linear <~quations

(see f:lnbc:hapter 2.2.4) can be n~al1zcd in the following way:

C===~==========~=========================~===========
C CHOLESKY METHOD
C==============:::r=====================================
DIMENSION A(10,10), B(10)
OPEN(8,FILE='CHOLESKY.IN')
OPEN(S,.FILE='CHOLESKY.OUT')
33 READ(8,100)N
100 FORMAT(I2)
IF(N)11,22,11
11 READ(8,101)(B(I),I=1,N)
101 FORMAT(8F10.4)
26 Numerical Methods in Computational Engineering

C READ IN THE UPPER MATRIX TRIANGLE OF A

READ(8,101)((A(I,J),J=1,N),I=1,N)
WRITE(5,102)N
102 FORMAT(/ 5X,'MATRIX DIMENSION =',I3//
1 5X,'MATRICA A',
2 <(N-1)*12+3>X,'VEKTOR B'/)
WRITE(5,103)((A(I,J),J=1,N),B(I),I=1,N)
103 FORMAT(1X,<N>F12.7,F13.7)
C FACTORIZATION OF MATRIX A TO THE FORM A~L*R
DO 10 I=2,N
10 A(1,I)=A(1,I)/A(1,1)
DO 25 I=2,N
I1=I-1
S=A(I,I)
DO 20 K=1,I1
20 S=S-A(I,K)*A(K,I)
A(I,I)=S
IF(I.EQ.N) GO TO 40
IJ=I+1
DO 25 J=IJ,N
S=A(I,J)
T=A(J, I)
DO 30 K=1,I1
S=S-A(I,K)*A(K,J)
30 T=T-A(J,K)*A(K,I)
A(I, J) =S/ A(I, I)
25 A(J,I)=T
40 WRITE(5,107)
107 FORMAT(//5X,'MATRIX L'/)
DO 111 I=1,N
111 WRITE(5,103)(A(I,J),J=1,I)
WRITE(5,108)
108 FORMAT(//5X,'MATRIX R'/)
N1=N-1
DO 222 I=1,N1
II=I+1
M=N-I
222 WRITE(5,99) .(A(I,J) ,J=II,N)
WRITE(5,99)
99 FORMAT(<12*I-8>X, '1.0000000' ,<M>F12.7)
C OBTAINING THE VECTOR OF SOLUTIONS
B(1)=B(1)/A(1, 1)
DO 55 I=2,N
I1=I-1
DO 45 K=1,I1
45 B(I)=B(I)-A(I,K)*B(K)
55 B(I)=B(I)/A(I,I)
DO 50 J=1,N1
I=N-J
I1=I+1
DO 50 K=I1,N
50 B(I)=B(I)-A(I,K)*B(K)
WRITE(5,109)
109 FORMAT(//13X,'VEKTOR OF SOLUTIONS'/)
WRITE(5,104)(B(I) ,I=1,N)
104 FORMAT(12X,F12.7)
GO TO 33
22 CLOSE(5)
CLOSE(8)
 · Lesson II - Linear Systems of Algebraic Equations: Direct Methods 27

STOP
END

For factorization of matrix A(= LR) we take in upper triangular matrix R unit
diagonal, i.e. 7'-i·i = 1 ('i, = 1, ... , n). Program is organized in this way so that matrix
A transforms to matrix A 1 , which lower triangle (including main diagonal) is equal to
matrix L, and strict upper triangle to matrix R. Note that diagonal elements in matrix
R are not memorized, hut only formally printed, using statement FORMAT. Note also
that in Section 2.2.4. the unit diagonal has been adopted into matrix L.
By applying this program to the applicable system of equations, the following
results are obtained:

MATRIX DIMENSION = 4
MATRICA A VEKTOR B
1.0000000 4.0000000 1.0000000 3.0000000 9.0000000
.0000000 -1.0000000 2.0000000 -1.0000000 .0000000
3.0000000 14.0000000 4.0000000 1.0000000 22.0000000
1.0000000 2.0000000 2.0000000 9.0000000 14.0000000
MATRIX L
1.0000000
.0000000 -1.0000000
3.0000000 2.0000000 5.0000000
1.0000000 -2.0000000 -3.0000000 2.0000000
MATRIX R
1.0000000 4.0000000 1.0000000 3.0000000
1.0000000 -2.0000000 1.0000000
1.0000000 -2.0000000
1.0000000
VEKTOR OF SOLUTIONS
1.0000000
1.0000000
1.0000000
1.0000000

Program 2.2.5.5. In similar way can be realized square root method for solution
of system of linear equations with symmetric, positive definite matrix. In this case it is
enough to read in only main diagonal elements of matrix A, and, for example, elements
from upper triangle. ·
The program and output listing for given system of equations are g·iven in the
following text. Note that from the point of view of memory usage it is convenient to
treat matrix A as a vector. Nevertheless, due to easier understanding, we did not follow
this convenience on this place.
Program is· organized in this way so that, in addition to solution of system of
equation, the determinant of system is also obtained. In output listing the lower triangle
of symmetric matrix is omitted.

$DEBUG
C=================================================
C SOLUTION OF SYSTEM OF LINEAR EQUATIONS
C BY SQARE ROOT METHOD
C=================================================
biMENSION A(10,10),B(10)
OPEN(B,FILE='SQR.IN')
OPEN(5,FILE='SQR.OUT')
3 READ(8,100)N
100 FORMAT(I2)
IF (N) 1, 2, 1
28 Nmnt~rical Methods in Computational Engineering

C READ IN VECTOR B
1 READ ( 8 , 10 1) ( B(I ) , I= l, N)
101 FORMAT(8F10.4)
C READ IN UPPER TRIANGULAR PART OF MATRIX A
READ ( 8 , 10 1) ( (A (I , J) , J =I , N) , I= 1 , N)
WRITE(5,102)
102 FORMAT(////5X, 'MATRIX OF SYSTEM'/)
WRITE(5,99)((A(I,J),J=I,N),I=1,N)
99 FORMAT(<12*I-11>X,<N-I+1>F12.7)
WRITE(5,105)
105 FORMAT(//5X, 'VECTOR OF FREE MEMBERS'/)
WRITE(5,133)(B(I),I=1,N)
133 FORMAT(1X,10F12.7)
C OBTAINING OF ELEMENTS OF UPPER TRIANGULAR MATRIX
A(1,1)=SQRT(A(1,1))
DO 11 J=2,N
11 A(1,J)=A(1,J)/A(1,1)
DO 12 I=2,N
S=O.
IM1=I-1
DO 13 K=1,IM1
13 S=S+A(K,I)*A(K,I)
A(I,I)=SQRT(A(I,I)-S)
IF(I-N) 29,12,29
29 IP1=I+1
DO 14 J=IP1,N
S=O.
DO 15 K=1,IM1
15 S=S+A(K,I)*A(K,J)
14 A(I,J)=(A(I,J)-S)/A(I,I)
12 CONTINUE
C CALCULATION OF DETERMINANT
DET=1.
DO 60 I=1,N
60 DET=DET*A(I,I)
DET=DET*DET
C SOLUTION OF SYSTEM L*Y=B
B(1)=B(1)/A(1, 1)
DO 7 I=2,N
IM1=I-1
S=O.
DO 8 K=1, IM1
8 S=S+A(K,I)*B(K)
P=1. I A(I, I)
7 B(I)=P*(B(I)-S)
c
C SOLUTION OF SYSTEM R*X=Y
C MEMORIZING OF RESULTS INTO VECTOR B
c
B(N)=B(N)/A(N,N)
NM1=N-1
DO 30 II=1,NM1
JJ=N-II
S=O.
JJP1=JJ+1
DO 50 K=JJP1,N
50 S=S+A(JJ,K)*B(K)
30 B(JJ)=(B(JJ)-S)/A(JJ,JJ)
c
 · Lcssou II - Liw~ar Syst.c~ms of Alg<-~hmic Equaticms: Direct MPthods 29

C PRINTING ~oF RESULTS

c
WRITE (5,201)
201 FORMAT(//5X, 'MATRIX R'/)
Pause 1
C DO 222 I=1,N
222 WRITE(5,199)((A(I,J),J=I,N),I=1,N)
199 FORMAT(<12*I-11>X,<N-I+1>F12.7)
WRITE(5,208) DET
208 FORMAT(//5X,'SYSTEM DETERMINANT D=' ,F11.7/)
WRITE(5,109)
109 FORMAT(//5X,'SYSTEM SOLUTION '/)
WRITE(5,133)(B(I),I=1,N)
GO TO 3
2 CLOSE(5)
CLOSE(8)
STOP
END

MATRIX OF SYSTEM
3.0000000
.0000000
1.0000000
2.0000000
1.0000000
1.0000000
VECTOR OF FREE MEMBERS
4.0000000 3.0000000 3.0000000
MATRIX R
1. 7320510
. 000000'0
.5773503
1.4142140
.7071068
.4082483
SYSTEM DETERMINANT D= 1.0000000
SYSTEM SOLUTION
.9999999 .9999998 1.0000000

Program 2.2.5.6. Mctlw<l of factorization for solution of systems of linear equa-

tions based on Ganss elimination with choice of pivotal element (see Sections 2.2.2 etml
2.2.4) can lH~ prop;r;umnalllc rcali?;c~<lnsing the following subprograms:

SUBROUTINE LRFAK(A,N,IP,DET,KB)
DIMENSION A(1),IP(1)
KB=O
N1=N-1
INV=O
DO 45 K=1,N1
IGE= (K -1) .*N+K
c
C FINDING THE PIVOTAL ELEMENT IN K-TH
C ELIMINATION STEP
c
GE=A (IGE)
I1=IGE+1
I2=K*N
30 Numerical Methods in Computational Engineering

IMAX=IGE
DO 20 1=11,12
IF(ABS(A(I))-ABS(GE)) 20,20,10
10 GE=A(I)
IMAX=I
20 CONTINUE
IF(GE)25,15,25
15 KB=1
c
C MATRIX OF SYSTEM IS SINGULAR
c
RETURN
25 IP(K)=IMAX-N*(K-1)
IF(IP(K)-K) 30,40,30
30 I=K
IK=IP(K)
c
C ROW PERMUTATION
c
DO 35 J=1,N
S=A(I)
A(I)=A(IK)
A(IK)=S
I=I+N
35 IK=IK+N
INV=INV+1
c
C K-TH ELIMINATION STEP
c
40 DO 45 I=I1,I2
A(I)=A(I)/GE
IA=I
IC=IGE
K1=K+1
DO 45 J=K1,N
IA=IA+N
IC=IC+N
45 A(IA)=A(IA)-A(I)*A(IC)
c
C CALCULATION OF DETERMINANT
c
DET=1.
DO 50 I=1,N
IND=I+(I-1)*N
50 DET=DET*A(IND)
IF(INV-INV/2*2) 55,55,60
60 DET=-DET
55 RETURN
END
c
c
SUBROUTINE RSTS(A,N,IP,B)
DIMENSION A(1),IP(1),B(1)
c
C SUCCESSIVE SOLUTION OF TRIANGULAR SYSTEMS
c
N1=N-1
C VECTOR B PERMUTATION
DO 10 I=1,N1
 Lesson II - Linear Systems of Algebraic Equations: Direct Methods 31

I 1=IP'{ I)
IF(I1-I) 5,10,5
5 S=B(I)
B(I)=B(I1)
B(I1)=S
10 CONTINUE
C SOLUTION OF LOWER TRIANGULAR SYSTEM
DO 15 K=2,N
IA=-N+K
K1=K-1
DO 15 I=1,K1
IA=IA+N
15 B(K)=B(K)-A(IA)*B(I)
C SOLUTION OF UPPER TRIANGULAR SYSTEM
NN=N*N
B(N)=B(N)/A(NN)
DO 25 KK=1,N1
K=N-KK
IA=NN-KK
I=N+1
DO 20 J=1,KK
I=I-1
B(K)=B(K)-A(IA)*B(I)
20 IA=IA-N
25 B(K)=B(K)/A(IA)
RETURN
END

Parameters in subprogram list of LRF AK are of following meaning:

A - Input matrix of order N stored columnwise (column by column). After N-1
elimination steps matrix A transforms to matrix which contains triangular matrices L
and R (see sectiou 2.2.4);
N - order of matrix A; ·
IP - vector of length N-1, which is formed during elimination procedure and con-
tains indices of pivot dements (see section 2.2.4);
DET - output vaxiahle contailling determinant of matrix of system A, as product of
elements on diagonal of matrix R, with accuracy up to sign. This value are corrected
by sign on the en<l of procedme, having in mind number of row permutations during
elimination process;
KB - control 1mmber with value KB=O if factorization is correctly performed, and
KB= 1 if matrix of system is singular. In the last case, LR factorization does not exist.
Subroutine RSTS solves suc:cessively systems of equations (2.2.4.4). Parameters in
list of subroutine parameters are of following meaning:
A - matrix obtained in subroutine LRFAK;
N - order of matrix A;
IP - vector obtained in subroutine LRF AK;
B - vector offree members in system to be solved. This vector transforms to vector
of system.solutions.
Main program is written in such way that, at first, given matrix A is factorized by
means of subroutine LRF AK, and then is possible to solve system of equations A;£ = b
for arbitrary numb<~r of Vf~ctors b, by calling subroutine RSTS.
· Main program aU<l output listing are of form:

DIMENSION A(100),B(10),IP(9)
OPEN(8,FILE='FACTOR.IN')
OPEN(5,FILE='FACTOR.OUT')
READ(8,5)N
32 Numerical Methods in Computational Engineering

5 FORMAT(I2)
NN=N*N
READ(8,10)(A(I),I=1,NN)
10 FORMAT(16F5.0)
WRITE(5,34)
34 FORMAT(1H1,5X,'MATRICA A'/)
DO 12 I=1,N
12 WRITE(5,15)(A(J),J=I,NN,N)
15 FORMAT(10F10.5)
CALL LRFAK(A,N,IP,DET,KB)
IF(KB) 20,25,20
20 WRITE(5,30)
30 FORMAT(1HO,'MATRICA JE SINGULARNA'//)
GO TO 70
25 WRITE(5,35)
35 FORMAT(1H0,5X, 'FAKTORIZOVANA MATRICA'/)
DO 55 I=1,N
55 WRITE(5,15)(A(J),J=I,NN,N)
WRITE(5,75)DET
75 FORMAT(/5X,'DETERMINANTA MATRICE A='F10.6/)
50 READ(8,10,END=70) (B(I),I=1,N)
WRITE(5,40)(B(I),I=1,N)
40 FORMAT(/5X,'VEKTOR B'//(10F10.5))
CALL RSTS(A,N,IP,B)
WRITE(5,45) (B(I),I=1,N)
45 FORMAT(/5X,'RESENJE'//(10F10.5))
GO TO 50
70 CLOSE(5)
CLOSE(8)
STOP
END

1 MATRICA A
3.00000 1.00000 6.00000
2.00000 1.00000 3.00000
1.00000 1.00000 1.00000
0 FAKTORIZOVANA MATRICA
3.00000 1.00000 6.00000
.33333 .66667 -1.00000
.66667 .50000 -.50000
DETERMINANTA MATRICE A= 1.000000
VEKTOR B
2.00000 7.00000 4.00000
RESENJE
18.99999 -7.00000 -8.00000
VEKTOR B
1.00000 1.00000 1.00000
RESENJE
.00000 1.00000 .00000
Program 2.2.5. 7. Using snbroutine LRFAK and RSTS, having in mind section
2.2.3, it is Pasy to writ<~ pro~ram for matrix inversion. The corresponding program and
output result (for matrix from pn~vious example) have the following forms:

C================================================
C INVERZIJA MATRICE
C=============================================~==
 · Lesson II - Linear Systems of Algebraic: Equations: Direct Methods 33

DIMENSION A(100), B(10), IP(9),AINV(100)

open(8,file='invert.in')
open(S,file='invert.out')
READ(8,5) N
5 FORMAT (I2)
NN=N*N
READ(8,10)(A(I),I=1,NN)
10 FORMAT(16F5.0)
WRITE(5,34)
34 FORMAT(1H1, SX, 'MATRICA A'/)
DO 12 I=1,N
12 WRITE(5,15) (A(J) ,J=I,NN,N)
15 FORMAT(10F10.5)
CALL LRFAK(A,N,IP,DET,KB)
IF(KB) 20,25,20
20 WRITE(5,30)
30 FORMAT(1HO,'MATRICA A JE SINGULARNA'//)
GO TO 70
25 DO 45 I=1,N
DO 40 J=1,N
40 B(J)=O.
B(I)=1.
CALL RSTS(A,N,IP,B)
IN=(I-1)*N
DO 45 J=1,N
IND=IN+J
45 AINV(IND)=B(J)
WRITE(5,50)
50 FORMAT(1H0,5X, 'INVERZNA MATRICA'/)
DO 55 I=1,N
55 WRITE(5,15)(AINV(J),J=I,NN,N)
70 CLOSE(5)
CLOSE(8)
STOP
END

1 MATRICA A
3.00000 1.00000 6.00000
2.00000 1.00000 3.00000
1.00000 1.00000 1.00000
0 INVERZNA MATRICA
-2. 00000 5. 00000 .-3; 00000
1.00000 -3.00000 3.00000
1.00000 -2.00000 1.00000

Bibliography
[1] Milovanovic:, G.V., Numerical Analysis I, Naucna knjiga, Beograd, 1988 (Ser-
bian).
[2] Milovanovic, G.V. and Djordjevic, Dj.R., Programiranje numerickih metoda
na FORTRAN jeziku. Institut za dokumentaciju zastite na radu "Edvard
Kardelj", Nis, 1981 (Serbian).

(The full list of references and further reading is given on the end of Chapter 3.)
Faculty of Civil Engineering Faculty of Civil Engineering and Architecture
Belgrade · · · Nis
Master Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON III

3. Linear Systems of Algebraic

Equations: Iterative Methods

3.1. Introduction
Consider system of linear equations

an:r1 + a12:z:2 + · · · + a1nXn = b1

a21X1 + a22X2 + · · · + a2nXn = b2

(3.1.1)

which can be written in matrix form

(3.1.2) Ax=b,
where

A=
[auan
(J,,;J,l
(},12
(},22 a1n
a2n l [x1 l
'
x=
X2

:r::n
'
b=
~

l:tl
an2 ann

In this chapter we always suppose that system (3.1.1), i.e. (3.1.2) has an unique solution.
Iterative methods for solving systems (3.1.2) have as goal determination of solution
x with accuracy given in advance. N arnely, -starting with arbitrary vector if( D) ( =
(O) (O))T) b y 1teratlve
. . x~(k)· ( = [ x 1(A:). ... Xn(A:))T) sueh
. met lw:d s one d efi nes t l1e senes
[:r 1 ... Xn
that
lim fi(k) = x.
k-++oo

3.2. Simple iteration method

One of the most simplest methods for solving system of linear equations is method
of simple iteration. For application of this method it is necessary to transform previously
system (3.1.2) to the following equivalent form

(3.2.1)

35
36 Numerical Methods in Computational Engineering

Then, the method of simple iteration is given as

(3.2.2) :f!(k) = B:f!U:- 1 ) + j3 (k = 1, 2, ... ).

Starting from arbitrary vector :f!(O) and using (3.2.2) one generates series {.i(h:)}, which
under some conditions convergc~s to solution of given system. . ·
If

bl2
B=
[1!11
b21 b22
.
bl,.
b2n 1 and - = [ !32
fii. 1'
(3
bnl bn'2 bnn ' f3n

iterative method (3.2.2) can be written in scalar forn1

where k = 1, 2, ....
One can prove (see [1]) that iterative process (3.2.2) converges if all eigenvalues
of matrix B are by modulus less than one. Taking in account that determination
of eigenvalues of matrix is rather complicated, in practical applications of method of
simple iteration only sufficient convergence conditions are examined. Namely, for matrix
B several norms can be <ldined, as for example,

IIBih = (LbTj) 112 ,

·ij
n
(3.2.3) IIBII2 = mpx L llJ.i.il,
j=1
n

IIBih =max L lbiJI·

.} ·i=1

It is not difficult to prov<~ that iterativ<~ process (3.2.2) converges if liB II < 1, for arbitrary
iui tial vectm :[C 0 ) . . .

3.3. Gauss-Seidel method

Gauss-Seicl<~lmethocl is constructed by modification of simple iterative method. As
we~ have seen, at simple~ iteration metho<l, the value of ·i.-th component :dk:) of vector
:1:-(h·) .
· IS o1J t <:unec
. l f'nnu va11ws :~: (1.:- 1) , ... , :rn(h:- 1) , 1.<).
1

(/.:) = I: r)· ·')'.

')'.
"·t.
n
(/;:-1)
'/.,) ".J
+ tJ..
1. ('i=1, ... ,n; k=l,2, ... ).
j=1
 Lesson III - Linear Systems of Algebraic Equations: Iterative Methods 37

This method can he modified in this way so that for computation of :dkl are used
.c1 ll prcvwus ~,
. , . ·ly compn t.e< 1 va.
. llWS. .r., .(h:) , ... , .:r:.,_
, (A:) (A:- 1 ) ( k: - 1 ) d ~ .
1 1 , :r:., , ... , Xn an the rest w1ll be part
of vector, obtained in previous iteration, i.e

(3.3.1) (·i = 1, ... , n; k = 1, 2, ... ) .

.i=1 .i=1

Noted modification of simple iterative method is known as Gauss-Seidel method.

The iterative process (3.3.1) can be written in matrix form too. Namely, let

where

Bl=
[ b21
0
.
0
0
0
0
B2=
h2
b22 bl,.
IJ2n 1
b.,~.l bn2 bn,n-l
:1 ' [I 0 bnn

Then (3. 3.1) becomes

(3.3.2)

Theorem 3.3.1. For arbitrary vector :£( 0 ), iterative process (:3.3.2) converges then and
only then if all mots of equation

bn- A

det[B 2 - (I- B 1 )A] =

b21A
:
bln
b2n 1=0
[
bnlA bn.n - A

are by modulus less tll;;w one.

3.4. Program realization

Program 3.4.1. Let's write a program for solving a system of linear equations of
form :Z = B:r+.B, by simple iteration method. Because this method converges when norm
of matrix B is less than one, for examination of this condition we will write a subroutine
NORMA, in which, depending on k, are computed norms (k = 1, 2, 3).in accordance with
formula (3.2.3). Parameters in list of parameters are of following meaning:
A - matrix stored as vector, which norm is to be calculated;
N - order of matrix;
K- number which defines norm (K=1, 2, 3);
ANOR - corresponding norm of matrix A.

SUBROUTINE NORMA(A,N,K,ANOR)
DIMENSION A(1)
NU=N*N
ANOR=O
GO TO (10, 20,40),K
10 DO 15 I=1,NU
38 NumeriC:al Methods in Computational Engineering·

15 ANOR=ANOR+A(I)**2
ANOR=SQRT(ANOR)
RETURN
20 DO 25 I=1,N
L=_:N
S=O.
DO 30 J=1,N
L=L+N
IA=L+I
30 S=S+ABS(A(IA))
IF(ANOR-S) 35,25,25
35 ANOR=S
25 CONTINUE
RETURN
40 L=-N
DO 50 J=1,N
S=O.
L=L+N
DO 45 I=1,N
LI=L+I
45 S=S+ABS(A(LI))
IF(ANOR-S) 55,50,50
55 ANOR=S
50 CONTINUE
RETURN
END

Main prognun is organized iu this way that before iteration process begins, the
converg'C~nce is examined. Namdy, if at least one norm satisfies liB Ilk < 1 (k = 1, 2, 3),
iterative process proce<~<ls. and if not, the message that convergence comlitions are not
fulfilled is printed and program terminates.
For multiplying matrix B by vector :r(k+l) we use subroutine MMAT, which is
given in 2.2.5.2. As initial v<~ctm we take f(O).
- As criteria for t<~nuiuation of iterative process we adopted

I...1.,,,(1.: l - .1....,,(1.:- 1 l I -< c ('i=1, ... ,n) .

On output we print the bst iteration which fulfills above given criteria.

DIMENSION B(100), BETA(10), X(10), X1(10)

OPEN(8,FILE='ITER.IN')
OPEN(5,FILE='ITER.OUT')
READ(8,5) N, EPS
5 FORMAT(I2,E5.0)
NN=N*N
READ(8,10)(B(I),I=1,NN),(BETA(I),I=1,N)
10 FORMAT(16F5.1)
WRITE(5,13)
13 FORMAT(1H1,5X,'MATRICA B', 24X,'VEKTOR BETA')
DO 15 I=1,N
15 WRITE(5,20) (B(J),J=I,NN,N),BETA(I)
20 FORMAT(/2X,4F8.1,5X,F8.1)
DO 30 K=1,3
CALL NORMA(B,N,K,ANOR)
IF(ANOR-1.) 25,30,30
30 CONTINUE
WRITE(5,35)
 ·Lessou III- Li1war Systems of Algebraic Equatious: Iterative l\!Id.lwds :39

35 FORMAT{5X, 'USLOVI ZA KONVERGENCIJU'

1' NISU ZADOVOLJENI')
GO TO 75
25 ITER=O
DO 40 I=1,N
40 X{I) =BET A(I)
62 ITER=ITER+1
CALL MMAT(B,X,X1,N,N,1)
DO 45 I=1,N
45 X1(I)=X1(I)+BETA(I)
DO 55 I=1,N
IF(ABS(X1(I)-X(I))-EPS)55,55,60
55 CONTINUE
WRITE(5,42)ITER
42 FORMAT(/3X,I3,' .ITERACIJA'/)
WRITE(5,50)(I,X1(I),I=1,N)
50 FORMAT(3X,4(1X, 'X(' ,I2, ')=' ,F9.5))
GO TO 75
60 DO 65 I=1,N
65 X(I)=X1(I)
GO TO 62
75 CLOSE(8)
CLOSE(5)
STOP
END

Taking H.ccnracy c = lo-s, for one concrete system of equation of fourth degree
(see output listiug) we g(~('. the solution in fomtcenth iteration.

MATRICA B VEKTOR BETA

-.1 .4 .1 .1 .7
.4 -.1 .. 1 .1 .7
.1 .1 -. 2 .2 1.2
.1 .1 .2 -.2 -1.6
14.ITERACIJA
X( 1)= 1.00000 X( 2)= 1~00000 X( 3)= 1.00000
X( 4)= -1.00000
Program 3.4.2. Write a code for obtaining a matrix S = cA wlwre A is given
square matrix of mdcr n, by using fonnnb

(3.4.2.1)

Let ·'h be k-th partial Slllll of series (3.4.2.1), and uh: its general mem1H'~r. Then tlH~
equalities ·

(3.4.2.2)

hold, whereby U 0 = S 0 =I (unity matrix of order n). By using equality (3.4.2.2) one
can write a program for smnmation ofseries (3.4.2.1), where we usually take as criterict
for termination of sm1muttion the case when norm of matrix is lesser than in advance
given small positiw~ number E:. In om case we will take norm 11.112 (see formula (3.2.3))
and c = 10- 5 .
40 Nnmel'ical Methods in Computational Engineering

By using subroutine MMAT fm rnatrices multiplication and subrmitine NORMA for

calculation of matrix norm, we h<we written the following program for obtaining the
matrix ~A

C===========================~====================~=
C ODREDJIVANJE MATRICE EXP(A)
C==================================================
DIMENSION A(100), S(100), U(100), P(100)
OPEN(S,FILE='EXPA.IN')
OPEN(5,FILE='EXPA.OUT')
READ(8,10) N,EPS
10 FORMAT(I2,E5.0)
NN=N*N
READ(8,15)(A(I),I=1,NN)
15 FORMAT (16F5. 0) _
C FORMIRANJE JEDINICNE MATRICE
DO 20 I=1,NN
s (I) =0.
20 U(I)=O.
N1=N+1
DO 25 I=1,NN,N1
S(I)=1.
25 U(I)=l.
C SUMIRANJE MATRICNOG REDA
K=O
30 K=K+1
CALL MMAT(U,A,P,N,N,N)
B=1./K
DO 35 I=1,NN
U(I)=B*P(I)
35 S(I)=S(I)+U(I)
C ISPITIVANJE USLOVA ZA PREKID SUMIRANJA
CALL NORMA(U,N,2,ANOR)
IF(ANOR.GT.EPS)GO TO 30
WRITE(5,40) ((A(I) ,I=J,NN,N) ,J=1,N)
40 FORMAT(2X,<5*N-9>X,'M AT RIC A A'
1 //(<N>F10.5))
WRITE(5,45)((S(I),I=J,NN,N),J=1,N)
45 FORMAT(//<5*n-9>X, 'MAT RIC A EXP(A)'
1 //(<N>F10.5))
CLOSE(8)
CLOSE(5)
END
This program has been t<~ste< l on the exampl<~

A= [442 ~ =~]'
4 -3

for which can be obtained analytically

(3.4.2.3) A=
3e- 2 3e- 3
2c- 2 2c- 1
-3e ++ 3]
-2e 2 .
r 4r:- 4 4e- 4 -4e +5
Outpnt listiup; is of form
 LeHson III - LilH~ar SyHtemH of Algebraic Equations: Iterative Methods 41

M AcT RI C A A
4.00000
3.00000
-3.00000
2.00000
3.00000
-2.00000
4.00000
4.00000
-3.00000
MA T R I C A EXP(A)
16.73060
14.01232
-14.01232
9.34155
12.05983
-9.34155
18.68310
18.68310
-15.96482

By using (3.4.2.3) it is not hard to prove that all figures in obtained results are
exact.
It is suggested to readers to write a code for previous problem using program
Mathematica.

3.5. Packages for systems. of linear algebraic equations

For many computer-<~arly years ago (late sixties and early seventies of previous
century) the most popular program (at least at Nis University), linear equations solver,
was SIMQ fi:·om SSP (Scientific Subroutine Package) by IBM corporation.
Nowadays, in many cases you will have no alternative but to use sophisticated black-
box program packages. Several good ones are available. LINPACK was developed at
Argonne National Laboratories ancl deserves particular mention because it is published,
documented, and available for fi:·ee use. A successor to LINPACK, LAPACK, is becoming
available. Packages available commercially include those in the IMSL and NAG libraries.
One should keep in mind that the sophisticated packages are designed with very large
linear systems in mind. They therefore go to great effort to minimize not only the
number of operations, but also the required storage. Routines for the various tasks are
usually provided in several versions, corresponding to several possible simplifications in
the form ofthe input coefficient matrix: symmetric, triangular, banded, positive definite,
etc. If one has a large matrix in Qne of these forms, he should certainly take advantage
of the increased efficiency provided by these different routines, and not just use the
form provided for general matrices. There is also a great watershed dividing routines
that are direct (i.e., execute in a predictable number of operations) from routines that
are iterative (i.e., attempt to converge to the desired answer in however many steps
are necessary). Iterative methods become preferable when the battle ?tgainst loss of
significance is in danger of being lost, either due to large n or becau8e the problem is
close to singular. Very interesting techniques are those the borderline between direct
and iterative methods,~ namely the iterative improvement of a solution that has been
obtained by direct methods.
Many commercial software packages contain Holvers for systems of linear algebraic
equations. Some of the more prominent packages are Matlab and Mathcad. The
spreadsheet Excel can also be used to solve systems of equations. More sophisticated
packages, such as Mathernatica, Macsyrna, and Maple also contain linear equations
solvers.
The book Numerical Recipes [4] contains several subroutines for solving systems
of linear algebraic equation8. Some algorithms, from which some are codded, are given
in book Numerical Methods for Engineers and Scientists [3] (see Chapter 1).
42 Numerical Methods in Computational Engineering

Some general guidelines for selecting a method for solving systems oflinear algebraic
equations are given as follows.
• Direct elimination methods are preferred for small systems (n ~ 50 to 100) and
systems with few zeros (nonsparse systems). Gauss elimination is the method of
choice. ·
• For tridiagonal systems, the Thomas algorithm is the method of choice ([3], Char)ter
1).
• LU factorization methods are the methods of choice when more than one vectors b
must be considered. ,
• For large systems th<tt are not diagonally dominant, the round-off errors can be
large.
• Iterative methods are preferred for large, sparse matrices that are diagonally domi-
nant. The SOR (Successive-Over-Relaxation) method is recommended. Numerical
experimentation to find the optimum over-relaxation factor w is usually worthwhile
if the system of equations is to be solved for many vectors b.

Bibliography (Cited references and further reading)

[1] Milovanovic, G.V., Numerical Analysis I, Naucna knjiga, Beograd, 1988 (Ser-
bian).
[2] Milovanovic, G.V. and Djorcljevi{:, Dj.R., Programiranje numerickih metoda
na FORTRAN jeziku. Institut za clokumentaciju zastite na radu "Eclvard
Kardelj", Nis, 1981 (Serbian). ·
[3] Hoffman, .J.D., Numerical Methods for Engineers and Scientists. Taylor
& Francis, Boca Raton-London-New York-Singapore, 2001.
[4] Press, W.H., Flannery, B.P., Teukolsky, S.A., and Vetterling, W.T., Numerical
Recepies - The Art of Scientific Computing. Cambridge University Press,
1989.
[5] Albrecht, .J., Fehlerabschiitzungen bei Relaxationsverfahren zur nu-
merischen Auflosung linearer Gleichungsysteme. Numer. Math. 3(1961),
188-201.
[6] Apostolatos, N. unci Kulisch, U., Uber die Konvergenz des Relaxionsvr-
fahrens bei nicht-negativen und diagonaldominanten M atrizen. Com-
puting 2(19G7), 17-24.
[7] Golub, G.H. & Varga, R.S., Chebyshev semi-iterative methods, successive
overrelaxation iterative methods and second order Richardson itera-
tive methods. Nunu~r. Math. 3(1961), 147-168.
[8] Soutwell, R.V., Relaxat'ion Methods in Theoretical Physics. 2 vols. Oxford
Univ. Press. Fair Lawn., New .Jersey, 1956.
[9] Varga, R.S., Matrix Iterative Analysis. Prentice Hall, Englewood Cliffs, New
Jersey, 1962.
[10] Young, D.M., Iterative Solution of Large Systems. Academic Press, New
York, 1971.
[11] Ralston,A., A First Course in Numerical Analysis. McGraw-Hill, New York,
1965.
[12] Hildebrand, F.B., Introduction to Numerical Analysis. Me.Graw-Hill, New
York, 1974.
[13] Acton, F.S., Numerical Methods That Work (corrected edition). Mathemat-
ical Association-of America, Washington, D.C., 1990.
[14] Abramowitz . M., aucl Steguu, I.A., Handbook of Mathematical Functions.
National Bnn~au of St.awlanls, Applied Mathematics Series, Washington, 1904
(n~printed ElG8 hy Dowr Publications, New York).
[15] Rice, .T.R., Numerical Methods, Software, and Analysis. McGraw-Hill, New
York, 1983.
 ·l_A~SSOll 111 - LllW<Il. 0'{SWlllS 01 .tuger>r<tH: .DqUaciOns: !Wl'af.lVe lVleClHHlS -!.j

[Hi] Forsythe, G.E.,, Malcolm, M.A .. awl Moler, C.B., Computer Methods faT
M athemat·ical C omp·utat·ions. Engl<~woocl Cliffs, Prentic<~- Hall, N .J, 1977.
[17] Forsythe, G.E., Solving linear algebra·ic equations can be interesting. Bnll.
Amer. Math. Soc. 59(1%3), 299-329.
[18] Forsythe & I\!Iolcr, C.B., Computer Solution of Linear Algebraic Systems.
Prentice-Hall. Englewood Cliffs, N.J, 1%7.
[19] Kahaner, D., Moler, C .. awl Nash, S., 191)9, Numerical Methods and Soft-
ware. Eugl<~woocl Cliffs. Prentice Hall, NJ, 1989.
[20] Hamming, R.W., Num.erical Methods for Engineers and Scientists. Dov<~r,
N<~w York, 1%2 (rcpriut(~<l 191:lG).
[21] Fcrziger, .J.I-L Numerical Methods for Engineering Applications. Stanford
Univ<~rsity, John Willey & Sons, Inc., New York, 1998
[22] P<~arson, C.E., Numerical Methods ·in Engineering and Science. University
of \Vashington. Van Nost.mwl R<~inholcl Company, New York 198G.
[23] Steplwnsou, G. awl Raclmon~. P.M., Advanced Mathematical Methods for
Engineer-·ing and S ciencc Students. Imperial College Low lou, University Col-
lege, Loudon CmHhridgc Univ. Press, 1999
[24] l\!Iilovanovi{:, G.V. and Kova<:<~vi(:, M.A., Zb·irka re,9enih zadataka iz nu-
merick:e anal·ize. Nau<:na lm.iiga, Beogra<l, 191:l5. (Serbian).
[25] IMSL Math/Libr-ary Users Manual , IMSL Inc., 2500 City West Boulevard,
Houston TX 77042
[2G] NAG Fortran L-ibrary, Nmm~rica.l Algorithms Group, 25G Banbnry Road, Ox-
ford OX27DE. U.K.
[27] Dongarra, .T..T., et al., LINPACK Users Guide. S.I.A.M., Philadelphia, 1979.
[28] Golub, G.H., awl Vau Loau,C.F., Matr-ix Computations, 2nd ed. (Baltimme:
Johns Hopkins Univ<~rsit.y Pn~ss), 1989.
[29] Gill, P.E., Murray, W .. and Wright, M.H., Numerical Linear Algebra and
Opt·imizat·ion, val. 1. Addison-Wesley, Redwood City, CA, 1991.
[30] St.ocr, J., and BulirsdL R., Introduction to Numerical Analysis. Springer-
V<·~rlag, N<~w York, 1980.
[31] Colen~an, T.F.,and Van Loan,C., Handbook for Matrix Compv.tations.
S.I.A.M., Philacldphia, 1988.
[32] Wilkinson, J.I-I., awl Reinsch, C. , L-inear Algebra, val. II of Handbook for
A·utomatic Computat·ion. Springer-Verlag, New York, 1971.
[33] \iV<~stlake, J.R A Handbook of Numerical Matr-ix Inversion and Solution
of Linear Eq1wtions. vViky, New York, 1%8.
[34] Johnson, L.W., awl Riess, R.D., Numerical Analysis, 2nd ed. A<ldison- ·wesley,
R<~ading, MA, 1982.
[35] Ralston, A., and Ral>iuowitJ~;, P., A First Course in Numerical Analysis,
2nd ed. McGraw-Hill. N<~w York. 1978.
[36] Isaacson, E., awl Kdl<,T, 1-I.B., A~alysis-o.f Numerical Methods. Wiley, New
York. 1966.
[37] Hom.,R.A., alHl Johnson, C.R, Matrix Analysis. Cambridge University Press,
C~tmhriclge, 1~)85.
Faculty of Civil Engine<~ring Faculty of Civil Engineering and Architect.un~
Belgrade ~ Nis
Master Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON IV

4. Eigenvalue Problems

4.1. Introduction
An nxn matrix A is said to have an eigenvectorx and corresponding eigenvalue
). if

(4.1.1) A· x = ,\x
Obviously any multiple of an eigenvector x will also be an eigenvector, hut we won't
consider such multiples as being distinct eigenvectors. (The zero vector is not considered
to be an eigenvector at all). Evid<mtly (4.1.1) can hold only if

(4.1.2) clet lA- ,\II = o,

which, if expanded out, is a.n nth degree polynomial in ). whose roots are the eigenval-
ues. This proves that there are always n (not 1iecessarily distinct) eigenvalues. Equal
eigenvalues coming from multiple roots are called degenerate. Root searching in the
characteristic equation (4.1.2) is usually a very poor computational method for finding
eigenvalues (see [2], pp. 449-453).
The above two equations also prove that every one of the n eigenvalues has a (not
necessarily distinct) corresponding eigenvector: If ). is set to an eigenvalue, then the
matrix A - ,\I is singular, and we. know that every singular matrix has at least one
nonzero vector in its null-space (consider singular value decomposition).
If you add TX to both sides of (4.1.1), you will easily see that the eigenvalues
of any matrix can be changed or shifted by an additive constant T by adding to the
matrix that constc-1nt times the identity matrix. The eigenvectors are unchanged by this
shift. Shifting, as we will see, is an important part of many algorithms for computing
eigenvalues. We see also that there is no speci(lJ significance to a zero eigenvalue. Any
eigenvalue can be shifted to zero, or any zero eigenvalue can be shifted away from zero.

Definitions
A matrix is called symmetric if it is equal to its transpose,

(4.1.3)

It is called Hermitian or self-adjoint if it equals to the complex-conjugate of its

transpose (its Hermitian conjugate, denoted by "t")

(4.1.4)

45
46 NunH~1·ical Methods in Computational Engineering

It is termed orthogonal if its transpose equals its inverse

(4.1.5)

and unitary if its Hermitian conjugate equals its inverse. Finally, a matrix is called
normal if it commutes with its Hermitian conjugate,

(4.1.6)

For real matrices, Hennitiau means the same as symmetric, unitary means 'the same as
orthogonal, and both of these distinct classes are normal.
The reason that "Hermitian" is an important concept has to do with eigenvalues.
The eigenvalues of a Hermitian matrix are all real.: In particular, the eigenvalues of a
n~al symmetric matrix are all real. Contrariwise, the eigenvaln<~s of a real nonsymmetric
matrix may inchul<~ real values, bnt 1ll<W abo inclmk lJttirs of conjugate v<1lues; and the
eigenvalues of a complt~x matrix that is not Hermitian will in general be complex.
The reason that "normal" is an important concept has to do with the eigenvectors.
The eigenvectors of n. normal matrix with non-degenerate (i.e., distinct) eigenvalues
are compld.e and orthogonal, spanning th<~ ·n-dimensional vector space. For a nonnal
matrix with degenerate eigenvalues, we have the additional b:eedom of replacing the
eigenvectors corn~spondinp; to a d(~generate eigenvalue by linear combinations of them-
selves. Using this fi:e<~clom, we can always perform Gramm-Schmidt orthogonalization
and find a set of eigenvectors that arc cmnpktc and orthogonal, just as in the non-
dcgelH-~rate case. The matrix whose columns are an orthonormal set of <~igenvectors is
evidently unitary. A special case is that the matrix of eigenvectors of a real symmetric
matrix is orthogonal, sinu~ tlw t~igcnvectors of that matrix are all real.
When a matrix is not uon1Jal, as typiiiecl hy any random, nonsymmetric, real
matrix, then in general we canuot find any orthonormal set of eig·envectors, nor even
any pairs of eigenv<~ctors that arc orthogoual (except perhaps by rare chance). vVhile
the n 110n-orthononual eigenv<~ctors will "mmally" span the n-<limensioual vector spact\
they do not always do so; tlmt is, the eigcuvectors are not always complete. Such a
nmtrix is said to l><~ <lef(-~ctivc.

Left and Right Eigenvectors

Whil(-~ tlw cig<~nvcctms of a non-normal matrix arc not particularly mthogonal
mnong tlwmsdves, they do have an orthogonality rdation with a <liff<Tent set of vectors,
which W(-~ umi'lt now define. Up to uow om cig(~nvcctors have-~ been colunm vectors that
are nmltipli<~<l to the right of a nmtrix A, as in (4.1.1). Tlwse, mon~ <~xplicitly. an~
t<~rnwd T"ight e'iqcnvectoTs. We could also, lww<-~ver, try to find row vectors, which
nmltiply A to tlw ldt awl sat.isfy

(4.1.7) :i! . A = >-.:1!.

These an~ call<~<l left e·igenvectoTS. By takiug the -tnwspm;c of (Ll.l. 7), mw cm1
s<~c that every ldt cigcuvcdor is the transpose of a right cig<~nvcctm of the tnmspos<~ of
A. Now by compariug to (:1.1.2). all<l using the fact that the det<~rmiuant of a matrix
<~qm11s the <letcnniuaut of its transpose, we also s<~c that the kft awl right cip;euvalncs
of A an~ i<kntical.
If th<~ matrix A i,-; symllldric t.lH~u tlw ldt awl right cigenw~ctors arc just trauspos(~s
of each other. that is, h<tv<~ tlw sauw umlwrical vahws as compoucuts. Likewise, if the
matrix is sdf-ad.ioiut.. t.lH~ ldt awl right <~ig(~nvcctors an~ Hermitian conjugates of each
ot.lwr. For tlw g(~lwralnou-uonual cas<~, lww<~vcr. we h;w<~ the following cakulatiou: Let
Xn he the matrix formed hy cohtlllllS from tlw rigltt eigenvectors, alHl XL he the matrix
formed hy rows from the ldt <~ig<~llV<~d.ors. Then (11.1) and ( 4.1. 7) can h<~ rcwritt.<~n as

A· Xn = Xn · di<tg(>q ... >-..,): XL· A= diag(>-.1 ... >-..,)·XL.

 Lesson IV - Eigenvalue Problems 47

Multiplying the first, of these equations on the left by XL, the second on the right by
XR, and subtracting the two, gives

(4.1.9)

This says that the matrix of clot products of the left and right eigenvectors comnmtes
with the diagonal matrix of eigenvalues. But the only matrices that commute with a
diagonal matrix of distinct elements are themselves diagonal. Thus, if the eigenvalues
are non-degenerate, each left eigenvector is orthogonal to all right eigenvectors except
its corresponding one, and vice versa. By choice of normalization, the dot products of
corresponding left and right eigr.nvectors can always be made unity for any matrix with
non-degenerate eigenvalues.
If some eigenvalues are degenerate, then either the left or the right eigenvectors
corresponding to a degenerate <~ig<~nvalue must be linearly combined among themselves
to achieve orthogonality with the right or left ones, respectively. This can always be
clone by a procedure akin to Gram-Schmidt orthogonalization. The normalization can
then be ad.iustecl to give unity for the nonzero clot products between corresponding left
and right eigenvectors. If the clot product of corresponding left and right eigenvectors
is zero at this stage, then you have a case where the eigenvectors are incomplete. Note
that incomplete eigenvectors can occur only where there are degenerate eigenvalues,
but do not always occur in such cases (in fact, never occur for the class of "normal"
matrices).
In both the degenerate and non-degenerate cases, the final normalization to unity
of all nonzero dot products produces the result: The matrix whose rows are left eig<~n­
vectors is the inverse matrix of tlw matrix whose columns are right eigenvectors, if the
inverse exists.
Diagonalization of a Matrix
Multiplying the first equation in (4.1.8) by XL, and using the fact that XL and
XR are matrix iuver:ses, we get

(4.1.10)

This is a particular case of a s·im·ilarity transform of the matrix A,

(4.1.11) A-+ z- 1 ·A· Z

for some transformation matrix Z. Similarity transformations play a crucial role in the
computation of eigenvalues, because they leave the eigenvalues of a matrix unchanged.
This is easily seen fi:·om

det IZ- 1 ·A· Z- >-II = det IZ- 1 ·(A- >.I)· Zl

(4.1.12) = clet IZI clet lA- >-II clet IZ- 1 1

= clet lA- >-II

Equation (4.1.10) shows that any matrix with complete eigenvectors. (which: includes
all normal matrices aud "most" random non-normal ones) can be clmgonahzed by a
similarity transformation, that the columns of the transformation matrix that effects
the cliagonalization are the right <~igenvectors, and that the rows of its inverse are the
left eigenvectors. ·
For real, symmetric matrices, the eigenvectors are real and orthonormal, so the
transformation matrix is orthogonal. The similarity transformation is then also an
orthogonal transformation of the form

(4.1.13) A-+ zT ·A· Z

 48 Numerical Methods in Computational Engiheering

While real nonsymmetric matrices can be diagonalized in their usual case of complete
eigenvectors, the transformation matrix is not necessarily reaL It turns out, however,
that a real similarity transformation can "almost" do the job. It can reduce the matrix
down to a form with little two-by-two blocks along the diagonal, all other elements zero.
Each two-by-two block corresponds to a complex-conjugate pair. of complex eigenvalues.
The "grand strategy" of virtually all modern eigensystem routines is to nudge the
matrix A towards diagonal form by a sequence o~ similarity transformations,

A----+ PI -l . A. pl ----+ P2 1 . P1 1 . A. pl . p2
(4.1.14) 1
----+ P3 · P2 1 · P1 1 ·A· P 1 · P2 · P3 ----+ etc.

If we get all the way to diagonal form, then the eigenvectors are the columns of the
accumulated transformation

(4.1.15)

Sometimes we do not want to go all the way to diagonal form. For example, if we
are interested only in eigenvalues, not eigenvectors, it is enough to transform the matrix
A. to be triangular, with all elements below (or above) the diagonal zero. In this case
the diagonal elements are already the eigenvalues, as you can see by mentally evaluating
(4.1.2) using expansion by minors. .
There are two rather different sets of techniques for implementing the strategy
, (4.1.14). It turns out that they work rather well in combination, so most modern
eigensys.tem routines use both. The first set of techniques constructs individual Pi's
as explicit "atomic" transformations designed to perform specific tasks, for example
zeroing a particular off-diagonal element (.Jacobi transformation), or a whole particular
row or column (Householder transformation, elimination method). In general, a finite
sequence of these simple transformations cannot completely diagonalize a matrix. There
are then two choices: either use the finite sequence of transformations to go most of the
way (e.g., to some special form like tridiagonal or Hessenberg) and follow up with
the second set of techniques about to be mentioned; or else iterate the finite sequence of
simple transformations over and over until the deviation of the matrix from diagonal is
negligibly small. This latter approach is conceptually simplest. However, for n greater
than rv 10, it is computationally inefficient by a roughly constant factor rv 5.
The second set of techniques, called factorization methods, is more subtle.
Suppose that the matrix A can be factored into a left factor F L and a right factor FR.
Then
(4.1.16) A= FL · FR or equivalently F£ 1 ·A= FR
If we now multiply back together the factors in the reverse order, and use the second
equation in (4.1.16) we get

(4.1.17)
which we recogniz(~ as having effected a similarity transformation on A with the trans-
formation matrix being FL. The QR method which exploits this idea will be explained
later.
Factorization methods also do not converge exactly in a finite number of transfor-
mations. But the better ones do converge rapidly and reliably, and, when following an
appropriate initial reduction by simple similarity transformations, they are the methods
of choice. The presented considerations are very important for those dealing with dy-
namics of construction and seismic engineering, especial in the phase of modelling and
dynamic response computation. ·

Definitions and theorems regarding eigenvalue problem

For further considerations w!~ need some theorems and definitions, as follows (see
[1], pp. 211-213).
 Lc~ssou IV - Eigenvahw Prohlc~ms 40

Definition 4.1.1. Lot A= [a·i.i] IH! cOHlplex sq1wre matrix ofonler n. Evmy vector .FE
en, which is diffi~ro11t hom xen>-vector, is called eigenvector of matrix A if exists scalar
). E C sw:l1 that lwlds (4.1.1). Scalar). in (4.1.1) is called corresponding eigenvalm!.
Hewing in mind that (4.1.1) can l>o ]mJsented in tlw form
(A- >.I):r = 0,
we concll!de Uw.t e(jllil.tion ( 4.1.1) lws 11011- trivi;d solutions (i11 :/;') then a11d only tl1en if
holds (4.1.2).
Definition 4.1.2. If A is S(JWIH-! matrix, then polynomial). ---t P(>.) = cld.(A- >.I) is
called clwracteristic polynomial, awl conespondinp,· (-!rJIZation P(>.) = 0 its clu-t.racf;eristic
eqllation.
Ld. A= [a·i,i]nxn· Tlw dmracteriHtic polynomial can be expresHed iu the form
(J,ln
(},2n
P(>.) =

or
P(>.) = ( -1)n(xn- ]>lxn.-l + ]J2xn- 2 - · · · + (-1)n-l]Jn-1A + (-1)nPn),
wh<~re ]JJ.: is snm of all principal minorH of order k of cletermimmt of matrix A, i.e.

Note that
'fl.

JJ1 = L ai·i. = tr A and P·n. = clet(A).

-i=l
Often, in place of charactc~ristic polynomial P is nsed so known normed characteristic
polynomial H, clefi1wd by
H(>.) = (-1)1/.P(>.) = Xn.- Plxn.-l + P2An- 2 - · · · + (-1)"''P.n.·
Eigenvalues of matrix A (i.e. zeros of polynomial P) )..i (·i = 1, ... , ·n.) will be
denoted as >..i(A).
Definition 4.1.3. Tlw set of all eigenvalues of square matrix A is ca.llecl spectrmv of
tha.t matrix and denoted with S]J (A).
Definition 4.1.4. SJH·!dnd radius fJ(A) of square matrix A is number
f! (A) = max
.,.
I>..i (A) I·
Theorem 4.1.1. Every matrix is;. in matrix sense, null of its cha.racteristic polynomial.
This theorem is known as Cayley-Hamilton theorem.
Theorem 4.1.2. Let >. 1 , ... , An Le eigenvalues of matrix A = [a·i.i] of mder n and
:r: ---t Q(:r) scalar polynomial of dep,Tee m .. Then

Q(>.I), · · · 'CJ(>.n)
are eigenvalues of matrix Q(A).
Theorem 4.1.3. Let A1, ... , An he eigeliW!.lnes of regular matrix A of order n. Then
,-1 \-1
/\1 ' · · · ' /\n

are eigenvalues of matrix A - l .

Theorem 4.1.4. Eigenvalues of triangular matrix are equal to diagonal elements.
The following themem gives recursive procedure for obtaining characteristic poly~
nomial of tridiagonal inatrix.
50 Numerical Methods in Computational Engineering

Theorem 4.1.5. Let

and
0 0

Normed characteristic polynomial A -----1 H(A) (= Hn(A.)) of matrix A(= An) is to be

obtained by recursive procedure

Hk(A) =(A- bA:)Hk:-l(A)- ak:-1Ck:-1Hk-2(A) (k = 2, ... , n),

where Ho(A) = 1 and H 1 (A) =A- b1 ._

Definition 4.1.5. For mati"ix B one says to be siinilar to matrix A if there exists at
least one Tegular matrix C such that

B = c- 1 AC.
Theorem 4.1.6. Similar matrices have identical characteristic polynomials, and there-
with identical eigenvalues.

4.2. Localization of Eigenvalues

A lot of problems reduce to eigenvalue problem. Here we will give some results
regarding localization of eigenvalues in complex space (see [1], pp. 290-292).
Theorem 4.2.1. (Gersllgorin). Let A= [aij]nxn square matrix of order nand Ci (i =
n
1, ... n) discs in complex space with centers in aii and radiuses ri = 2.::: laij I, i.e.
j=l
j =/=i

C,i = {z liz- O,;,il :'S: Ti} (i = 1, ... , n).

If we denote with C union of these discs, then all eigenvalues of matrix A are in C.
Remark 4.2.1. Regarding fact that matrix AT has same eigenvalues as matrix A, on
the basis of previous theorem one can conclude that all eigenvalues of matrix A are
located in the union of D discs

n
where s.i = 2.::: Iaij 1-
i=l
'i.=f=j

Based on previous one concludes that all eigenvalues of matrix A lie in the cut of
sets C and D.
Theorem 4.2.2. Ifm. discs from Theorem 4.2.1. form connected area which is isolated
from other discs, then exact m eigenvalues of matrix A are located in this area.
The proof of this theorem could be found in extraordinary monograph of Wilkinson
[7].
 Lesson IV - Eigenvalue Problems 51

Example 4.2.1.
Take matrix

A= ~
r-0.2
0.1
2
-0.1]
0.4
0 3
Based on theorem 4.2.1 eigenvahws are located in discs

CI = {z liz- 1[::; 0.2}, c2 = {z liz- 2[::; 0.4}, c3 = {z liz- 3[::; 0.2}.

Note that, based on remark 4.2.1., it follows that discs DI, D 2 , D 3 have radiuses
0.2, 0.1, 0.5, respectively. By the way, the exact values of eigenvalues, given with seven
figures, are )q = 0.9861505, A2 = 2.0078436, A3 = 3.0060058, and normed characteristic
polynomial is
H(A) = A3 - 6A 2 + 10.98A- 5.952.
Theorem on localization of eigenvalues has theoretical and practical importance (for
example, for determining initial values at iterative methods, for analysis at perturbation
problems, etc).
For determining eigenvalues there are a lot of methods, whereby some of them
enable finding of all eigenvalues, and others only some of them, for example, dominating
ones, i.e. with maximum modulus. Some of methods perform only determination of
coefficients of characteristic polynomial, so that some of methods for solution of algebraic
equations have to be used (see Chapter 5). Such approach is not recommended, being in
most cases numerically unstable, i.e. ill-conditioned. Namely, because the coefficients
of characteristic polynomials are: in general, subjects to round-off error, due to ill-
conditioning of characteristic polynomials, the big errors in eigenvalues occur.

4.3. Methods for dominant eigenvalues

Very often, in many applications (i.e. in dynamic of constructions), one needs only
maximal (by module) eigenvalue and corresponding eigenvector.
Let AI, ... , An be eigenvalues and :r\, ... , Xn corresponding eigenvectors of matrix
A= [ai.J]·n,xn· If

we say that AI, ... , A.,. arc dominant eigenvalues of matrix A. In this section we will
consider a method for determination of-dominant eigenvalue and corresponding eigen-
vector, as well as some modifications of this method. We suppose that eigenvectors are
linearly independent, forming a basis in nn. Therefore, the arbitrary non-zero vector
'1!0 can be expressed as

(4.3.1)

where a.i some scalars. Define now the iterative process

fh: = A'iJk-I (k = 1, 2, ... ).

Then n
_,
Vk = _,
Avk-I = A2 'l!!i:-
_, 2 = ... = Ak _,
. Vo = "'""""'
L-t ai Ak x.i,
_,
i=I

or, regarding to (4.3.1) and assertion of Theorem 4.1.2,

n
(4.3.2) Th = L CYiA~:Xi·
i=I
52 N lmwricnl Methods in Computational Engineering .

• The speeial interesting caRe hm·e is when one dominant eigenvalue A1 (r = 1) exists.
Assuming a1 =I 0, on the basis of (4.3.2) we have ·.

where EA: ---+ 0, when k---+ +oo.

Introduce now notation (if).;, for 'i-th coordinate of Rome vector iJ. Then 'i-th coor-
dinate of vector fh: is

Because of
'ih+L = a1AA:+ 1(:r1 + EA:+l),
based on previous, for every 't (1 ::; 'i ::; n) we have

Based on this fact, the method for determination of dominant eigenvalue A1, known
as power method, can be formulated. Vector fh, is thereby an approximation of non-
normed eigenvector which corresponds to dominant eigenvalue*. By practical realization
of this method the nonning of eigenvector is perforined, i.e. of vector f!A: after every
iteration step. Norm-setting is performed by dividing vector 'l'h: by its coordinate with
maximal module. So, power method can be expressed by

where "YA: is coordinate of v<~ctor ZA: with maximal module, i.e., "YA: = (iA:)·;, and l(zA}i.l =
IliA: II· Note that "YA: ---+ A1 and Th, ---+ ll'~rll , when k ~ +oo.
,[,1 00

Speed of convergence of this method d<~pends on ratio IAI/ A2 1. Namely, it holds

(4.3.3)

Note that by deriving of this method we. suppose that a1 i- 0, meaning that method
converges if A1 is <lominant <~igcnvalue and if initial vector 'Vo has a component with same
direction as eigenvector :1! 1 . On h<~havior of this method without those assumptions
one can find in the monograph of Wilkinson [7, p. 570] ancl Parlett ancl Poole [11].
Practically, due to round-oft' errors in iterative process, the condition a. 1 i- 0 will be
satisfied after few steps, although starting assumption for vector ·uo not being fulfilled.
Example 4.3.1.
Let
-2Gl 209
-49]
A= -G30 422 -98 '
[ -t\00 G31 -144

with eigenvalues A1 = 10, A2 = 4, A3 = 3.

By taking for initial vector fi0 = [0 0 - l]T, by power method we get the results
given in Table 4.3.1.

*
If :i! eigenvector, then n: ( c i- 0) IS also (~igenvector corresponding to the same
eigenvalue.
 Lesson IV- Eigenvalue Problems 53

Table 4.3.1
k 'Yk ('ihh ('uh: )2 Uhh
1 144.0000 0.340278 0.680556 1.
2 13.2083 0.334911 0.669821 1.
3 10.7287 0.333774 0.667549 1.
4 10.2038 0.333463 0.666926 1.
5 10.0599 0.333372 0.666744 1.
6 10.0179 0.333345 0.666690 1.
7 10.0054 0.333337 0.666674 1.
8 10.0016 0.333334 0.666669 1.
9 10.0005 0.333334 0.666667 1.
10 10.0001 0.333333 0.666667 1.
11 10.0000 0.333333 0.666667 1.

Because of linear conv<)rgence of the power method, for convergence acceleration

the Aitken c5 2 method can be used. A simple method for convergence acceleration is
given in [1], pp. 303-305.

4.4. Methods for subdominant eigenvalues

Suppose that eigenvalues of matrix A are ordered in a way

In this section the methods for determination of subdominant eigenvalues, i.e .

...\2, ...\3, ... , Am (nL < n) will be considered. The three methods will be explained.
1. Method of orthogonalization. Suppose, at first, that matrix A is symmetric,
and that eigenvector :f:'1 which corresponds to dominant eigenvalue ...\1 (I.A1l > J.A.iJ, -£ =
2, ... , n) has been detenniw~d by, for example, power method. Starting with arbitrary
vector z, let us form vector ·u0 which is orthogonal to vector x1. So we have (see
Gram-Schmidt's method of orthogonalization)

(4.4.1)

Because of (i10 , :r\) = 0, from theoretical point of view, the series 17A: = A·uh:-l (k =
1, 2, ... ) in the power method could be used for determination of ...\ 2 and corresponding
eigenvector :£2 . Nevertheless, regardless of fact that ·iJ0 does not have the component
in direction of eigenvector :r1 , pmyer method would, because of round-off errors, after
some number of iterations, converge toward eigenvector x1 . This fact was mentioned in
the previous section.
It 1s possible to eliminate this influence of round,..off errors using so known periodical
"purification" of vector ·iJ0 from component in direction of i\. That means that, after,
say, T steps, we compute ·u0 using ·11, in place of z in (4.4.1), i.e. by means

In this way, if the period of "purification" is small enough so that it cannot happen sig-
nificant accumulation of round-off:' error, by power method can be determined eigenvalue
:r
...\2 and eigenvector 2 .
By continuation of this procedure we can f11rther determine ...\ 3 and x3 .
54 Numerical Methods in Computational Engineering

Generally, if we detc~rmine -X 1 , ... , AI/ and corresponding vectors :l1 , ... , ;l1/ (u < rn)
it is possible to determine \/+1 and :£1/+l using power method, by forming vector ·i70
orthogonal to :£1 , ... , .1!1/. So, starting fi:·om arbitrary vect<)l' z, we have

I/ (--+ --+ )
(4.4.2) --+ --+ """" z' :r: i --+
'U() = z- ~ (·--:. ,-t.) :~:-;,
·i=1 ,f,,,, .f.'/,

meaning that vector 'lf.o has components only in direction of residual eigenvectors, i.e.

Power method applied to ·i70 gives :ru+I and A1/+ 1 in absence of round-off errors. Being·
not the case, it is necessary frequent "purification" of vector 'VA: from components in
direction :l1 , ... , .in. In other words, after r· steps, one should determine again 0 using v
(4.4.2), by using 'if.r in place of z.
Also, in the case when matrix A is not symmetric, but has complete system of
eigenvectors, the given orthogcmalizing procedure can be applied.
2. Inverse iteration method. This method is applied to general matrix A and is
based on solution of system of equations

(4.4.3)

where p is constant, and ·iT0 arbitrary vector. System (4.4.3) is usually to be solved by
Gauss method of dimiuation or Cholesky method by factorization of matrix B = A- pl.
Note that tl.1~~ method of i~1verse iteratio.1~ is equi.valen~ to the po~er uHpwd applied.
to B. Therefore, by applymg mdhocl of mverse 1teratwn tlw domman!./f'Jgenvalne of
matrix B is obtained, i.e. /Lu = 1/(Au- p) for which it holds

minl\i- PI = I-X11- PI·

.'l

Eigenvalue A11 is closest eigenvahw of matrix A to the numher p. Eigenvector ohtainrd

thereby is the same oue for matrices B an<l A.
By convt~nieut choice of parameter ]J all eigenvalues of matrix A can be, in principle,
obtained.
Similm to power methocl, lH~n~ is also suitable to perform nonning of vector ·ih, so
that w<~ hav<~

(4.4.4)

wlwre "'(/;: is <:Oordillatc of V<~ctOl' Z~, with grcakst module.

Example 4.4.1.
Using m<~thod of iuv<~rsc iteration for matrix

1
10
1

we will determine cig<~nvaln<~ dns<~st to m1mlwr ]J = 9, as well as corresponding eigen-

vector.
Using factorization by Gauss nwthod with pivoting for matrix B = A- 91, we get

LR = PB.
 L<~sson IV - Eigenvalw~ Problems GG

21~Gl
1 () 5 1
= ()]
L
[
-4/G 1
~ ' R=
[
~ 9/G
- '1/r:.) 2/3 0 -1

and matrix P ddim~d by in(h~x s<~ries I = (1, 3).

IH~nnut.ation
The method of invt~rs<~ iteration (4.4.4) can now be expressed in the form

by which application the results giv<~n in Tahl<~ 4A.1 are obtained.

Table 4.4.1
k Uh,h Uh:h Uh:h (J,,
1 0. 1. -1. 6.
2 -0.2 1. -O.G 9.3
3 -0.17241 1. -0.48276 9.34483
4 -0.17200 1. -0.48000 9.34800
5 -0.17185 1. -0.47980 9.34835
6 -0.17184 1. -0.47977 9.34838

For initial vector we took -IJ0 = [1 0 OjT. Iu the last column of table is given the
quantity jJ 1, = p + 1//~.:, which giv<~s appmximatiou for corresponding eig<~nvalue A. One
can see that this eigenvalue has approximate value 9.34838.
3. Deflation uwtho<ls. The methods of this kind are composed from construction
of sequence of matrices A~~. ( = A), A~~. _ 1 , ... , A 1 , which order is equal to index and
thereby
Sp (An) ~ Sp (An-1) ~ · · · ~ Sp (A1),
where Sp (A A:) denotes spcctnun of matrix A~,:.
We will describe now a special awl impmtant case of deflation method, when matrix
A is Hermitian.
Let :c = [:r 1 :1: 2 ... :rnJT be eigenvector of matrix A corresponding to eigenvalue A
and nonned
(:1!, :C) = :c * :1! = II:ZII~ = 1
with first coordinate :~: 1 being nonnegative.
Have a look over matrix

(4.4.G) P = I- 2'iihv*,

where the vector 'lv = [w 1 w2 ... 'll!nJT is defined by first vector e1 = [1 0 ... OjT from
natural basis of spac<~ nn in the following way:

(4.4.6) 'IV* 'IV= 11:1711~ = l, 'W1 2 0,

(4.4. 7)
The matrix P is of form

P=
[
1- 2'W1'llJ1
-2'W27lJl
.
-2Wl'1V2
1- 2w211J2
-2'W1'1Vn
-2w2'i'iJn
I
-2'Wn,'llJl 1- 2Wn'lVn
5G Nnnwrical MetlwclR in Computational Engineering

Note that P* = P. what lJWauR that matrix P iR Hermitian, too. Moreover,

regarding to ( 4.4.G), by clin~ct multiplication we s<~e that

p*p = p2 =I ,

and concludf~ that matrix P is unitary.

Based on (4.4. 7) we iiud coordinates of vector w. So, from 1 - 2w·1Ti 1
-2'WA:'IV1 = :r:,,, (k = 2, ... , n) it follows

and w,,. = - -2w1- :r~,:

(k = 2, ... , n).

Note that ·1T! 1 = '1111 > 0.

Now, basecl ou (4.4.7) and A:l! = A:c we find that APe1 = P<:'-'1 , wherefrom we
conclnd<-~ that P* APf1 = Ac"1, i.<~. rc"i'1 is eig-<-~nvector of matrix B = P*AP = PAP.
Note, a1Ro, that iirst colnnm in matrix B is just veotor Ae1 , i.<~.

A h2 !Jl.? h1n
0 !J22 1!2:3 b2n
~T ]
0,~-1
() b32 1!33 bn-1
B= h3n
[ An-1
() bn2 1Jn3 bnn

where with An- 1 we d<~note<lmatrix of order n- 1 which matches with enclosed block.
~ T~

On-1 is zero-vector of onler n- L and, finally, h.~_ 1 = [b 12 b13 · · · b1.,] .

Regarding the fact that mahix B iR similar (we say also unitary Rimilar) to llJ<-1-1-.T'lX
A, w<~ conclude that

Sp(An-1) = Sp(An)\(A) (An= A).

In order to get 1wttrix A.,_ 2 w<~ are proceeding in a similar way. In place of matrix P
we use matrix
~T

P1=
[ On-1
~
1
o,Z/],
where matrix Q is of order '/1, - 1 awl of form (4. i.G), satisfying the conclitions ( L1.4. G)
1
ancl (4.4. 7) reganlinp.; cip.;euvcct.or il ;mel <~igcnvalnc JJ of matrix An-1· Becanse of P1 =
Pi = P 1 we condwlc that matrix P 1 is unitary, too.
Now matrix C = P 1 BP 1 = P 1 PAPP 1 has a form

A (:12 r:13 C1n A (;12 (:13 C1n

() ,,, (:23 r:2n 0 fl C23 C2.,_
() ()
c - (:33 C:Jn () ()

An-2
() () C.,_ () ()
Cn3 'I/,

wh<~re matrix A~~._ 2 is of onkr n - 2. By colltillninp.; this procedure Wf-~ get npp<T
triangnlar matrix which is n11itary similar to iuitial matrix A. Having in mind that
matrix A is Hl~rmit.iau. wr~ cmH:lwk that it is nuitary similar to diag-onal matrix.
Tlw lll'<~s<-~uted pro<:<~dnn~ d<~umwls. hd'orc of every st<~p, clctc~nuinatioll of ow~ cig<~ll­
V<thw ;mel concspondinp.; <~ig<~nv<~ctor, what can he <lone hy some of previonsly pn~s<~ut.ed
mctlwds. TlmR, hdorc thr~ Jirst stq>. mw has to dct.<~nniw~ eig<~uvalnc A <-UJd <~ig<~uvcd.m
 Lesson IV - Eigenvalm~ Probl<~ms G7

:r of matrix A, befd-re the second step eigenvalue Jl. and eigenvector :1] of matrix A~~._ 1 ,
and so on.
~t is. clear that eigenvalnes of matrix A are diagonal elements of obtained triangnlar
matnx, 1.e. A1 = A, A2 = JL, etc. It remains the question what is with eigenvectors of
matrix A? It is dear that. :1\ = :7. We will show how, based on obtained results. the
second eigenvector of matrix A can b<~ found. ·
If the coordinates of eigenvector :IJ are y 2 , ... , Yn., in order to find, at first, eigenvector
:IJ' of matrix B, put :t/ 1 = [:1;1 :Y2 ... YnJT and try to determine :y 1.
Because of

B :tJ.... i = [ .. A
On.-1

l.e.

it follows
:t/1 + b,'7,_1:1J= :Y1·
If A =1- JJ, by virtue of previous equality we get
1 _,T _, 1
:Y1 = -,--bn-1:Y = -,--(b12:Y2
A-fl. /\-fl.
+ · · · + b1nYn)·
Now simply find the eigeuvector :r2 of matrix A, corresponding· to eigenvalue A2 =1- p..
Indeed, because of PAP:IJ' = JL:/j', i.e. A(P:t7') = /I(P:IJ') we conclude that 2 = P:t7'. x
In a similar way t.l1<~ other eigenvectors can be determined.

4.5. Eigenvalue problem for symmetric tridiagonal matrices

Let A be real symmetric tridiagonal matrix of order n which non-zero elements
will be denoted as
O+i = b.; ('i = 1, ... , n.),
O..i.;i-1 = (J,,i-l;i = C.i, ('i = 2, ... , n).
With ph:( A) denote main minor of order k of matrix A- AI, i.e.
b1 - A c:2
C2 b2 - A C3 0

0 Ck:-1 bk:-1 -A Ck:

Ck bk:- A
and define p 0 (A) = 1. Note that p!(A) = b1- A.
By developing of determinant Ph: (A) up to elements of last row we get

PA:(A) = (bk- A)Pk:-l(A)- c~:Pk:-2(A) .

•
The value of characteristic polynomial of matrix A can be simple evaluated, based on
previous, using three-term recurrence relation
ph:(>..) = (b,,:- A)JJk-1 (A)- c~:PA:-2(A) (k = 2, ... , n),
(4.5.1)
Po(A) = 1, JJI(A) = b1- A.
A simple method for determination of eigenvalues of symmetric tridiagonal matri-
ces is based on usage of recurrent relation (4.5.1), method of interval bisection, and
statement of the following theorem, which is simple to prove:
58 Numerical Methods in Computational Engineering

Theorem 4.5.1. (Givens) Let all elements~ Ck =1- 0 of symmetric tTidiagonal matrix A
of order n. Then it lwld8:
(1) Zeros of every polynomial Pk ( k = 2, ... , n) are real, diffe:i·ent, and separated by
zeros of polynomial Ph:-1; ~
(2) If Pn (>.) =1- 0, number of eigenvalues of matrix A less than .\ is equal to numb~r of
sign change s ( >.) in the series .

(4.5.2) ]Jo(>.), P1(>.), · · · ,Pn(>.).

If some Pk:(>.) = 0, then on this place in series (4.5.2) can be taken arbitrary sign,
regarding to Ph:-1 (>.)PA:+1 (>.) < 0.
Remark that in theorem there exists condition Ck =1- 0 for every k = 2, .. :, n. If,
for example, for some k = m, c.m. = 0, then problem simplifies, because it splits in two
problems of lower order (m and n- m.). Namely, matrix A becomes
----

A' 0 ]
A== [ 0 A" '

where A' and A" an~ tridiagonal syminetric matrices of order rn and n- m., respectively,
and in this case is

clet(A- >.I) = det(A'- >.I) det(A"- >.I).

Using multiple values for >.it is possible by systematic application of Theorem 4.5.1
to determine disjunct intervals in which lie eigenvalues of matrix A. Thus, if we find
that -

based on Theorem 4.5.1 we have that in interval (>. 1, >. 2) .lies one only eigenvalue of
matrix A. Then for its determination the simple method of halving of interval (bisection
method) can be used, by contraction of this starting interval up to desired exactness.
For determination of intervals in which lie eigenvalues it can be used also theorem
of Gershgorin, so that those intervals are

[h- ic2l, b1 + jc2j],

[b.i.- lc·i.i-lr:·i+11, b.;+ jcil + ic·i+11], ('i = 2, · .. , n- 1),
[bn- knl, bn + lc.nj].
Unfortunately, these intervals are not disjunct, aiHl in general case contain not only one
eigenvalue of matrix A.
Example 4.5.1.
For given matrix

we have

Po(>.)= 1, 1'1(>.) = 1- >., P2(>.) = (3- >.)pi(>.)- Po(>.),

P3(>.) = (G- >.)1J2(>.)- 4pl(>.), P4(>.) = (7- >.)p3(>.)- 9p2(>.).

Let >. = 0. Then we have JJo(O) = 1, JJI(O) = 1, P2(0) = 2, p3(0) = 6, P4(0) = 24.
Thus, in the series (4.G.2) an~ + + + + +, what means that there ip no sign change, i.e.
 Lesson IV - Eigenvalue Problems 59

:s(O). =. 0. ~c~orclirti? !-o Theorem 4.5.1, matrix A does not have negative eigenvalues,
1.e. 1t 1s pos1tlve-clefimte.
Taking in sequence for >. values 1, 2, 4, 5, 7, 9, 10 we get the results given in Table
4.5.1.

Table 4.5.1
). Po ( >.) P1 (>.) ]J2(>.) P3 ().) P4 ().) .s(>.)
1 1 0 -1 -4 -15 1
2 1 -1 -2 -2 8 2
4 1 -3 2 14 24 2
5 1 -4 7 Hi -31 3
7 1 -(:j 23 -22 -207 3
9 1 -8 47 -1G6 -111 3
10 1 -9 62 -274 264 4

Based on values of .s(>.) we conclude that in each interval (0, 1), (1, 2), (4, 5), (9, 10)
is located one eigenvalue of matrix A. These eigenvalues with six figures are .

A1 ~ 0.322548, A1 rv 1.745761, A1 ~ 4.536620, A1 ~ 9.395071.

Note that these are zeroes of Laguerre polynomial L 4 .

4.6. LR and QR algorithms

This section is devoted to so known factorization methods. First such method for so-
lution of problem of eigenvalues for arbitrary matrix A was described by H. Rutishauser
([14]) in the year 1958, which called it LR transformation. Method consists in construc-
tion of series of matrices {AdA: EN, starting from A 1 =A, in the following way: Matrix
A factorizes to lower triangular matrix Lh: with unit diagonal and upper triangular
matrix Rk, i.e.

(4.6.1)

and then the following member is determined by multiplication of obtained factors in

opposite sequence, i.e.
AA:+1 = Rk:Lk:.

Note that matrices Ak:+ 1 and AA: are similar, because they are connected with trans-
formation of similarity

(4.6.2)

Factorization of (4.6.1) can be performed by Gauss method of elimination.

If we put
L (k:) = L 1 ... L k: ancl
based on (4.6.2) we have
- AL(k)
L (k:)A k:+1- '

wherefrom it follows
•
L(k)R(k) = L(k:- 1 ) Ak:R(k:- 1) = AL(k:- 1 )R(k:- 1 ).

By iterating the last equality, we get

60 Numerical Methods in Computational Engineering

what means that L(k)R(k) is factorization of matrix A k:. Using this facts, Rutishauser
(see, also [7]) showed that under certain conditions series of matrices { Ak} converges
towards some upper triangular matrix, which elements on th,e main diagonal give eigen-
values of matrix A. Usually, LR method is applied to matrices previously reduced to
upper Hessenberg form (aij = 0 for 'i ::::: ,j + 2). If, by means ofsome method, general
matrix reduced to lower Hessenberg form we apply LR method to transposed matrix,
which has the same eigenvalues. All matrices in series { Ak:} have Hessenberg form.
Acceleration of convergence of series {A~;:} can be done by convenient shifting Pk:, so
that, in place of A~;: we factorize Bk: = Ak: -p~;;l = L~;:Rk:, whereby Ak:+ 1 = p~;:I+RK:L~;:.
Unfortunately, LR algorithm has several disadvantages (see monograph of Wilkin-
son [7]). For example, factorization does not exist for every matrix. One better factor-
ization method was developed by J.G.F. Francis ([15]) and V.N. Kublanovskaya ([16]),
where matrix L is replaced with unitary matrix Q. So one gets QR algorithm defined
by .

(4.6.3)

starting from A 1 =A. Note that A~;:+ 1 = Qt:Ak:Qk:.

If we put .

(4.6.4) Q (k:) -- Q 1... Q k: and - R k:... R 1,

R (k:)-

similar as LR method, we find

(4.6.5)
Theorem 4.6.1. If matrix A reg,uhu·, then exists decomposition A = QR, where Q
is unita1y, and R upper triangular matrix. Moreover, if diagonal elements of matrix R
are positive, decomposition is unique.
QR factorization ( 4.6.3) can be performed by using unitary matrices of form I -
2wv)*. So, in order to transform Ak: to Rk:, i.e. reduction of columns to Ak:, we have

(4.6.6)
The matrix Qk: is then

(4.6.7)

QR algorithm is efficient if initial matrix has (upper) Hessenberg form. Then, pre-
viously mentioned unitary matrices reduce to two-dimensional rotations. All matrices
Ak: are of Hessenberg form. Thus, eigenvalue problem for general matrix is most con-
venient to be solved in two steps. At first, reduce matrix to Hessenbcrg form, and th(~n
apply the QR algmithm.
In special case, when initial matrix is tridiagonal, matrices Ak in QR algorithm
are also tridiagonal. In that case, using conveniPntly chosen shift Pk, QR algorithm
becomes very efficient for solving (~ip;<:~nvalue problem of tridiagonal matrices.
Similar to QR algorithm, it is developed QL algorithm ([18]), where L is lower
triangular matrix, and Q unitary matrix. Also, it has been developed so known implicit
QL algorithm ([19]).

4.7. Software eigenpackages

Some general guidelines for solving eigenproblems are su1mnarizerl hdow [22].
• When only the largest awl (m) t.he smallest eigenvalue of a matrix is required, the
pow(~r method can be employed.
• Although it is rath(~r inefficient, the power method can he used to solve fm inter-
mediate eigenvalw~s.
 Lesson IV - Eigenvalue Problems 61

• The direct method is not a good method for solving linear eigenproblems. However,
it can be used for solving nonlinear eigenproblems.
• For serious eigenproblems, the QR method is recommended.
• Eigenvectors corresponding to a known eigenvalue can be determined by one ap-
plication of the shifted inverse power method.
Almost all software routines in use nowadays trace their ancestry back to routines
published in Wilkinson and Reinsch's hoock Handbook for Automatic Compu-
tation, Vol. II, Linear Algebra [13]. A public-domain implementation of the
Handbook routines in FORTRAN is the EISPACK set of programs [3]. The routines pre-
sented in majority of most frequently used software packages are translations of either
the Handbook or EISPACK routines, so understanding these will take a lot of the way
towards understanding those canonical packages.
IMSL [4] and NAG [G] each provide proprietary implementations in FORTRAN of what
are essentially the Hand book routines.
Many commercial software packages contain eigenproblem solvers. Some of the
more prominent packages are Matlab and Mathcad. More sophisticated packages, such
as Mathernatica, Macsyrna, and Maple also contain eigenproblem solvers. The book
Numerical Recepies [2] contains subroutines and advice for solving eigenproblems.
A good "eigenpackage" will provide separate routines, or separate paths through
sequences of routines, for the following desired calculations
• all eigenvalues and no eigenv<~ctors
• all eigenvalues and some corresponding eigenvectors
• all eigenvalues and all corresponding eigenvectors.
The purpose of these distinctions is to save compute time and storage; it is wasteful
to calculate eigenvectors that you don't need. Often one is interested only in the eigen-
vectors corresponding to the largest few eigenvalues, or largest few in the magnitude,
or few that are negative. The method usually used to calculate "some" eigenvectors is
typically more efficient than calculating all eigenvectors if you desire fewer than about
a quarter of the <~igenvectors.
A good eigenpackage also provides separate paths for each of the above calculations
for each of the following special forms of the matrix:
• real, symmetric, tridiagonal
• real, symmetric, banded (only a small number of sub- and super-diagonals are
nonzero)
• real, symmetric
• real, nonsymmetric
• complex, Hermitian
• complex, non-Hermitian.
Again, the purpose of these distinctions is to save time and storage by using the
least general routine that will serve in any particular application.
Good routines for tht~ following paths are available:
• all eigenvalues ancl eigenvectors of a real, E;ymmetric, tridiagonal matrix
• all eigenvalues and eigenvectors of a real, symmetric, matrix
• all eigenvalues and eigenvectors of a complex, Hermitian matrix
• all eigenvalues and no eigenvectors of a real, nonsymmetric matrix.

4.8. Generalized and nonlinear eigenvalue problems

Many eigenpackages also deal with the so-called generalized eigenproblem [6],

(4.8.1) A· x= >.B · x

where CA and B are both matrices. Most such problems, where B is nonsingular, can
be handled by the equivalent

(4.8.2)
62 Numerical MetlwclH in Computational Engineering

Often A and B are symmetri<~ nncl B is positive definite . The matrix B- 1 . A in

(4.8.2) is not Rymnu~tric, hut we can recover a symmetric eigenvalue problem by uRing
th<~ Cholesky decompoHition B = L ·LT. Multiplying equation (4.8.1) by L - 1 we get

(4.8.3) C · (LT · :£) = A(LT · :Z)

where

(4.8.4)

The matrix C is symmetric aucl its eigenvalues are the same as those of the original
problem (4.8.1); its <~ig<~nfnnd.ious an~ LT · :Z. The efficient way to form C is first to
solve the equation '

(4.8.5)

for the lower triangl<~ of the matrix Y. Then solve

(4.8.G) L·C=Y

for the lower triangle of the symnwtric matrix C.

Another gem~rali:;mtion of the stan<l;:ml <~igenvalue problem is to problems nonlinear
in the eigenvahw A, for example,

(4.8.7) (AA 2 + BA +C)· :r = 0

This can be tmned into a lint:~m problem by introclucing an additional nnknown
eigenv<~ctor:1/ and solving tJw 2n x 2n eigensystmH,

[ ()
-A - l · C
I
-A- 1 -B ]
.-·]
).
/.

[ if
= .r.' .
A [~]
:1/.

This technique g<~n<~raliz;<~s to higlwr-onl<~r polynomials iu A. A polynomial of degree

m. pro<lue<-~s a lim~ar m:n x urn eig<~nsyst<~lll, as given in [7].

Bibliography (Cited references and further reading)

[1] J\!Iilovanovi(:. G.V .. Numer·ical Analysis I, Nau(:na kn.iiga, BeognHl, 1988 (Ser-
bian).
[2] Press, W.H., Flmmcry, B.P., Tenkolsky, S.A., and Vetterling, W.T., Numer·ical
Recepies - The Art of Scient~fic Computing. Cambridge Uuiw~rsity Press.
Hl89.
[3] Smith, B.T., <~t al., Matr·i:r: E-igensystem Routines - EISPACK Ou:idc,
2nd ed., val 6 of Lect·uTe Notes in Compv,ter Science. Spring<~L N<~w
York, 197G.
[4] IMSL M ath/L-ibTary Users M an·ual , IMSL Inc., 2500 City \i\Tt~st BonlcvanL
Honston TX 77042
[5] NAG Fortran L-ibrary, Nmncrical Alp;mitlnns Gronp, 25G Banlmry Road, Ox-
fonl OX27DK U.K., Clmpt<~r F02.
[G] Golnh, G.H., awl Van Loan, c.·F., Matri:E Computation, Johns Hopkins Uni-
V<~rsity Press, Baltimm<~ 191)9.
[7] Wilkinsou, The Algebraic E·igenvalue Problem, Oxford Univc~rsity Pn~ss, N<~w
Ymk, 19GG.
[IS] Acton, F.S .. Nume·r-ical Methods that Work, corrected edition, Matlwmatical
Association of Auwrica. Clw.ptcr B . Washiup;ton, 1970.
 Lesson IV - Eigenvalue Problems 63

[9] Horn, R.A., <urd Johnson, C.R.., Matrix Analysis, Cambridge University Press,
Cambridge, 1985.
[10] Milovanovi<':, G.V. and Djonljevic, Dj.R., Programiranje numerickih metoda
na FORTRAN jeziku. Institut za dokumentaciju zastite na radu "Edvard
Kardelj", Nis, 1981 (Serbian).
[11] Parlett, B.N. ancl Poole, W.G., A geometric theory for the QR, LU, and
power iterations. SIAM .J. Numer. Anal. 10(1973), 389-412.
[12] Bart, W., Martin, R.S., Wilkom;on, .J.H. Calculation of the eigenvalues of
a symmetric tridiagonal matrix by the bisection method. Numer Math.
9(1967), 386-393.
[13] Wilkinson, .J.H. & Reisch, C., Handbook for Automatic Computation. Vol.
II Linear Algebra. Springer Verlag, Berlin-Heidelberg-New York, 1971.
[14] Rutishauser, H., Solution of eigenvalue problem with the LR-transfor-
mation. Appl. Math. Ser. Nat. Bur. Stand. 49(1958), 47-81.
[15] Francis, J.G.F., The QR transformation - a unitary analogue to the LR
transformat·ion. Compnt. J. 4(1961/62), 2G5-271, 332-345.
[16] Kublanovskaya, V.N., 0 nekotoryh algori.fmah dlja resenija polno{ prob-
lemy sobstvennyh znaceni{ Z. VyCisl. Mat. i Mat. Fiz. 1(1961), 555-570.
[17] Golub, G.H. & Welsch, .J.H., Calculation of Gauss quadrature rules. Math.
Comp. 23(1969), 221-230.
[18] Bowdler, H., Martin, R.S., Reinsch, C. Wilkinson, .J.H., The QR and QL algo-
rithms for symmetric matrices. Nmner. Math. 11(1968), 293-306.
[19] Dubrulle, A., Martin, R.S., Wilkinson, .J.H., The implicit QR algorithm. Nu-
mer. Math. 12(1968), 377-383.
[20] Stoer, J., and Bulirsch, R., Introduction to Numerical Analysis, Springer,
New York, 1980.
[21] Mitrinovic, D.S. and Djokovi(:, D.Z., Polinomi i matrice. ICS, Beograd, 1975
(Serbian). .
[22] Hoffman, .J.D., Numerical Methods for Engineers and Scientists. Taylor
& Francis, Boca Raton-London-New York-Singapore, 2001.
Faculty of Civil Engineering Faculty of Civil Engineering ancl Architecture
Belgrade '- Nis
Master Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON V

5. Nonlinear Equations and Systems

5.1. Nonlinear Equations

5.1.0. Introduction
We consider that most basic of tasks, solving equations numerically. While most
equations are born with both a right-hand side and a left-hand side, one traditionally
moves all terms to the left, leaving

(5.1.0.1) .f(x) = 0

whose solution or solutions are desired. When there is only one independent variable,
the problem is one-dimensional, namely to find the root or roots of a function. Figure
5.1.0.1 illustrates the problem graphically.

f(x)

Figure 5.1.0.1
With more than one independent variable, more than one equation can be satisfied
simultaneously. You likely once learned the implicit function theorem which (in this
context) gives us the hope of satisfying n equations in n unknowns simultaneously.
Note that we have only hope, not certainty. A nonlinear set of equations may have no
(real) solutions at all. Contrariwise, it may have more than one solution. The implicit
function theorem tells us that generically the solutions will be distinct, pointlike, and
separated from eetch other. But, because of nongeneric, i.e., degenerate, case, one can
get a continuous family of solutions. In vector notation, we want to find one or more
n-dimensional solution vectors .if such that

(5.1.0.2)

.f
where is the n-dimensional vector-valued function whose components are the individ-
ual equations to be satisfied simultaneously. Simultaneous solution of equations in n
dimensions is much more difficult than finding roots in the one-dimensional case. The
principal difference between one and many dimensions is that, in one dimension, it is

65
66 Numerical Methods in Computational Engineering

possible to bracket or "trap" a root between bracketing values, and thei1 find it out di-
rectly. In multidimensionR, you can never be sure that the root is there at all until you
have found it. Except in linear problems, root finding invariably proceeds by iteration,
and this is equally true in one or in many dimensions. Starting from some approximate
trial solution, a useful algorithm will improve the solution until some predetermined
convergence criterion is satisfied. For smoothly varying functions, good algorithms will
always converge, provided that the initial guess is good enough. Indeed one can even
determine in advance the rate of convergence of most algorithms. It cannot bP overem-
phasized, however, how crucially success depends on having a good first g·uess for the
solution, especially for multidimensional problems. This crucial beginning usually de-
pends on analysis rather than numerics. Carefully crafted initial estimates reward you
not only with reduced computational effort, but also with understanding and increased
self-esteem. Hammings motto, "the purpose of computing is insight, not numbers,"
is particularly apt in the area of finding roots. One should repeat this motto aloud
whenever program converges, with ten-digit accuracy, to the wrong root of a problem,
or whenever it faib to converge because there is actually no root, or because there is
a root but initial estimate was not sufficiently close• to it. For one-dimensional root
finding, it is possible to give some straightforward answers:. You should try to get some
idea of what your functiou looks like before trying to find its roots. If you need to
mass-produce roots for many different functions, then you should at least know what
some typical members of the ensemble look like. Next, you should always bracket a root,
that is, know that the function changes sign in an. identified interval, before trying to
converge to the roots value. Finally, one should never let iteration method get outside
of the best bracketing bounds obtained at any stage. We can· see that some pedagogi-
cally important algorithms, such as secant method or Newton-Raphson, can violat<-~ this
last constraint, and are thus not recommended unless certain fixups are implemented.
Multiple roots, or very close roots, are a real problem, especially if the multiplicity is
au even number. In that case, tlwr<~ may b<~ no readily apparent sign change in the
function, so the notion of bracketi1ig a root and maintaining tlu~ bracket becomes diffi-
cult. We nc~vertheless insist on bracketing a root, even if it takes the minimum-searching
techniques to determine whether a tantalizing dip in the function really does cross zero
or not. As usual, we want to discourage the reader from using routines as black boxes
without understanding them.

f{x)f . f{~ ~
1
(a)
+--+----:~ (b)

f(x)
f(x)l \ . ;

(c)
~ (d)

f~x) 11
·. -rbx)

a 1 =~~:~=a, X
a 1 =~ x
(e) (f)

1 1 (\ ~X)r !"""'\ 1
~ ~
(g) (h)

Figmc G.1.0.2
Nonlin<~ar equations can hchav<~ in various ways in tlw vicillity of a root. Algc~hraic
<m<l transc<~lHl<~utal <~qnations may haw~ distinct. (i.e. simple) real )·oots, repeated (i.e.
 Lesson V- Nonlinear Equations and Systems ()7

multiple) real roots, or complex roots. Polynomials may have real or complex roots.
If the polynomial coefficients are all real, complex roots occnr in conjugate pairs. If
the polynomial coefficients are complex, single complex roots can occur. Figure 5.1.0.2
illustrates several distinct types of behavior of nonlinear equations in the vicinity of a
root. (a) illustrates the case of a single real root, called simple root. (b) illustrates a
case where no real roots exist. Complex roots may exist in such a case. Two and three
simple roots are showed on (c) and (d), respectively. Two and three multiple roots are
illustrated on (e) and (f), respectively. A case with one simple root and two multiple
roots is given in (g), and in (h) is illustrated the general case with any number of simple
and multiple roots.
There are two distinct phases in finding the roots of nonlinear equation (see [2],
pp. 130-135):
(1) Bounding the solution, and
(2) Refining the solution.
In general, nonlinear equations can behave in many different ways in the vicinity
of a root. ·
(1) Bounding the solution
Bounding the solution involves finding a rough estimate of the solution that can
be used as the initial approximation, or the starting point, in a systematic procedure
that refines the solution to a specified tolerance in an efficient manner. If possible, it
is desirable to bracket the root between two points at which the value of the nonlinear
function has opposite signs. The bounding procedures can be:
1. Drafting the function,
2. Incremental search,
3. Previous experience or similar problem,
4. Solution of a simplified approximate model.
Drafting the function involves plotting the nonlinear function over the range of
interest. Spreadsheets generally have graphing capabilities, as does Mathematica,
Matlab and Mathcad. The resolution of the plots is generally not precise enough for
accurate result. However, they are accurate enough to bound the solution. The plot
of the nonlinear function displays the behavior of nonlinear equation and gives view of
scope of problem.
An incren1ental search is conducted by starting at one end of the region of interest
and evaluating the nonlinear function with small increments across the region. When
the value of the fimction changes the sign, it is assumed that a root lies in that interval.
Two end points of the interval containing the root can be used as initial guesses for a
refining method (second phase of solution). If multiple roots are suspected, one has to
check for sigh changes in·the derivative of the function between the ends of the interval.

(2) Refining the solution

Refining the solution involves determining the solution to a specified tolerance by
an efficient procedure. The basic methods for refining the solution are:
2.1 Trial and error,
2.2 Closed domain methods (bracketing method),
2..3 Open domain methods.
Trial and error methods simply presume (guess) the root, x = a, evaluate f(a.),
and compare to zero. If f(cv.) is close enough to zero, quit, if not guess another a. and
continue until .f (a) is close enough to zero.
Closed domain (bracketing methods) are methods that start with two values of x
which bracket the root, :r: = a, and systematically reduce the interval, keeping root
inside of br:ackets (inside of interval). Two most popular methods of that kind are:
· 2.2.1 Interval ha1virlg (bisection),
2.2.2 False position (Regula Falsi). ... ·
Bi'acketing methods are robust and reliable, since root is always inside of closed
interval, but can be slow to convergence.
Open domain methods do not restrict the root to remain trapped in a closed in-
terval. Therefore, there are not as robust as bracketing methods and can diverge. But,
uu l ~ l.l.l.LJ.,:-:-;.t.l\.. CH .Lv--.1\.~V.llUI....J.D 11.1 VUll.l_lJ UlJctldU.llClo.l D.l.lt)l.l,.ltt:a ll~t)

they use information about the nonlir1ear function itself to come closer with estimation
of the root. Thus, they are much more efficient than bracketing methods.

Some general hints for root finding

Nonlinear equations can behave in various ways in the vicinity of a root. Algebraic
and transcendental equations may have simple real roots, multiple real roots, m complex
roots. Polynomials may have real or complex roots. If the polynomial eoefficieilts are all
real, complex root occur in conjugate pairs. If the polynomial coefficients are complex,
single complex roots cc:m occur.
There are numerous methods for finding the roots of a nonlinear equation. Some
general philosophy of root finding is given below. ·
1. Bounding method should bracket a root, if possible.
2. Good initial approximations are extremely important.
3. Closed domain methods are more robust than open domain methods bec;:tuse they
keep the root in a closed interval.
4. Open domain methods, when--converge, in the general case converge faster than
closed domain methods.
5. For smoothly varying functions, most algorithms will alwa.ys converge if the initial
approximation is close enough. The rate of convergence of most algorithms can bP.
determined in advance. ·
6. Many problems in engineering and science are well behaved and straightforward.
In such cases, a straightforward open domain method, such as Newton's method,
or the secant method, can be applied without worrying about special cases and
strange behavior. If problems arise during the solution, then the peculiarities of
the nonlinear equation and the choice of solution method can be reevaluated.
7. When a problem is to be solved only once or a few times, theu the efficiency of
method is not of major concern. However, when a problem is to be solved many
times, efficiency is of major concern.
8. Polynomials can be solved by any of the methods for solving nonlinear equations.
However, the special techniques applicable to polynomials should b<~ considered.
9. If a nonlinear <~qnation has complex roots, that has to be anticipated when choosing
a method.
10. Time for problem anaJysis versus computer time has to be considered dming
method selection.
11. Generalizations about root-finding methods are generally not possible.
The root-finding algorit.hms should contain the following features:
1. An npp<-~r limit on the muuber of iterations.
2. If the method nsP.s tlw d<~rivative .f'(:J:), it should be monitored to ensure that it
does not approach zen>.
3. A convergeuce test for the change in the magnitude of the solution, j:r:.i+l - .1:.;,[, or
the magnitude of the nonlinear function, Ixi+ 1l, has to be included.
4. When convergence is indicated, tlw final root estimate should be inserted into the
nonlinear fnm:tiou f (:1:) to guarantee that f (:1:) = 0 within the desired toleraw:e.

5.1.1. Newton's method

Newton's or oftm1 c:alkd Nt:wton-Raphson method is basic method for dd.P.nniua-
tion of isolated zeros of nonlim~<n <~quations.
Let isolated nniqne siwpl<~ root :r: = u, of <~quation (5 .1.0 .1) exist on scgmP.nt [rv, fJ]
and let f E C[n, fJ]. Then . nsing Taylor dewlopnwnt, we get

(G.1.1.1)

when~ ~· = :~: 0 + fJ(o,- :r: 0 ) (0 < 0 < 1). Having in mind that f(a) = 0. by nt:glect.ing
rlast member on t.lw right-hand side of (5.1.1.1), we get

""--' f(:J:o)
o, = :ro - . ( '!' )
' II '·()
 L!~sson V - Nonlinear Equations and Systems (i9

If we denote left-h<incl sicl!~ of last approximative equation with x 1 , we get

f(:Do)
(5.1.1.2) :r;l = xo- f1(:rof

Here :1: 1 represents the abscissa of intersection of tangent on the curve :y = f(:r) in the
point (xo, f(:r 0 )) with :~:-axis (see Figure 5.1.1.1).

y=f(x) y=f(x)

X
0

Figure 5.1.1.1 Figure 5 .1.1. 2

The equality (5.1.1.2) suggests the construction of iterative formula

(5.1.1.3)

known as Newton's method or tangent method.

We can examine the convergence of iterative process (5.1.1.3) by introducing the
additional assumption for function f, namely, assume that f E C2 [ cv., ,8]. Because the
iterative function ¢ is at Newton's method given as
f(.'D)
c/J (:r: ) = :r:- f1(x)'

by differentiation we get

1
• f'(:r: 2 ) - f(.T)f"(:r:) f(:r)f"(x)
(5.1.1.4) cp (:£:) = 1- f1(:J:)2- .f1(x)2
Note that cp(a) = a and (p 1 (a) = 0. Being, based on accepted assumptions for .f,
functi9n cf/ continuous on [cv., (J], and cf/ (a) = 0, there exist,13 a neighborhood of point
x = a, denoted as U (a) where it holds

1 l.f(x)f"(x) I
(5.1.1.5) lc/J (:DJI = .f'(:I:)2 S q < 1.
Theorem 5.1.1.1. If :z: 0 E U(a), series {xk} generated using (5.1.1.3) converges to
point x = a, whereby
. .'Dk:+l- a .f"(a)
(5.1.1.6) lnn (.'Dk: - a, ) 2 = ,2, .1 ( a, ) •
k:-t+= 1
(see [1], pp. 340-341).
70 Numerical Methods in Computational Engineering

Example 5.1.1.1.
Find the solution of equation

.f(:c) = x- cos :1: = 0

on segment [0, 1r /2] using Newton's method

. _ .. 1:h: - cos :rk Xk sin Xk: +cos :Dk:

.1•k:+l -.LA:- . = (/,; = 0, 1, ... ).
1 + sm :rk: 1 + sin Xk:

Note that .f' (2:) = 1 +sin :c > 0(\i:c E [0, 1r /2]). Starting with ::r 0 = 1, we get the results
given in Table 5.1.1. -
•
Table 5.1.1
'k 1:,,
0 1.000000
1 0.750364
2 0.739133
3 0.739085
4 0.739085

The lcist two iterations give solution of equation in consideration with six exact
figures.
Example 5.1.1.2.
By applying thP Newton's method on solution of equation .f(:r) = x'~~- o. = 0 (o. >
> 1) we obtain the iterative formula for determination of n-th root of positive
0, n
number a

(k = 0, 1, ... ).

A special case of this formula, for n = 2 gives as a result square root.

At application of Newton method it is often problem how to chose initial valne of
:r 0 in order series {:c,Jk:EN to be monotonons. One answer to this question was given
by Fourier. Namely, if .f" <loet-J not change a sign on [a, ,8] and if :r(J is chosen in such
~ waytl.mt .l(;o).f"(:r 0 ) > 0, the series {.1:,Jk:EN will be monotonous. This statement
follows from (o.l.l.4).
Based on Theorem 5.1.1.1 we conclude that Newton's method applied to det!~nni­
nation of simple root :r = o. has square convergence if f"(a.) i- 0. In this case factor of
convergence (asymptotic constant of error) is

The case .f"(a) is specially to be analyzed. Namely, if we suppose that .f E C 3 [n,j3] one
can prove that
. :r:/,:+1- a f"'(a)
. lnn . .
1•. --t+oo (.r.,., - a) 3 :3.f' (o.) .
Example 5.1.1.3.
Consider the !~quatiou

.f(:r:) = :r:
3
- 3:I: 2 + 4:J:- 2 = 0.
 L<~ssm1 V- Nonliw~ar Eqnations and Systems 71

B<~cans<-~ of .f(O) = -2 awl f(1.5) = O.G25 w<~ conclncle that on seguwnt [0. 1.5] this
equation has a mot. Ou the otlwr hand, .f' (:r:) = ::b: 2 - G:r: + 4 = 3(:r:- 1) 2 + 1 > 0, what
means that. tlw root is simple, enabling application of Newton's method. Starting with
:1: 0 = 1.5, we get the n~sults in Table 5.1.2.

Table 5.1.2
k :r,,,
() 1.5000000
1 1.1428571
2 1.0054!)44
:3 1. 0000003

The exact vahw fm root is a= 1, br.canse .f(:1:) = (:r:- 1):3 + (:r:- 1).
In mder to n~<lncc umnh<~r of caknlations, it is often used th<~ following modification
of N <-~wton method
f (:r:/,:)
:1:/,:+1 = :r1.,- /''(·. ) (k = 0, 1, ... ).
. ./.()

Geometrically, :r: 1,,+ 1 n~prescnts abscissa of intersection of :r:-axes a.nd straight line passing
through point (:r 1,, f(:r:,J) and being paralld to ta.ngent of cnrvi-~ y = f(:r:) in the point
(:r:o, .f(:r:o)) (see Figme 5.1.1.2).
ltcrativ<~ function of such mo< lified Newton's method is

f' (a) .
Because of qJ 1 (a) = a awl (/>~(o.) = 1- ·.,( ) , w<-~ conclude that method has order of
.f :r:o
convergence one, i.e. it holds

whereby the condition

f'(:r:) ~---
1-
.f''(·.r:o ) :'S (j < 1,

is analogous to co11<lition (5.1.1.5)

By approximation of first derivative f' (:r:k) in Newton's method with divided clif-

ferencc
f (r: A:) - .f (:r: 1.:-1) one gd.s secant method
:r: /,: - :r: /,:- 1

:r: /.:+ 1 = :r: A:

:r~.: - :rl.:-1 f'(·..r. r· ) (k = 1, 2, ... ),
(5.1.1.7) -
.f (:r ,J - f (:r: k -1) · ·

which belongs to open domains methods (two steps method). For starting of iterative
process (5.1.1. 7) two initial values :r 0 and :r: 1 are needed. Geometrical interpretation of
secant method is giv<~n in Figure 5.1.3.1.
 72 Numerical Methods in Computational Engineering

y y

X X

Figure 5.1.3.1 Figure 5.1.3.2

The secant method can be modified in such a .way that

(5.1.1.8) (k = 1, 2, ... ).

This method is often called regula falsi or false position method. Differently from
secant method, where is enough to take .1: 1 # :r: 0 , at this method one needs to take
.1:1 and :J:o on different sides of ro()t :r: = a. Geometric interpretation of false position
method i8 given in Figure 5.1.3.2.

5.1.2. Bisection method

Let on segment cv., (:J exist isolated simple root :r = a of equation
(5.1.2.1) f(x:) = 0,

where f E C[a, fJ]. Metho<l of interval bisection for Holution of equation (5.1.2.1) couHists
in com;truction of series' of iutervals { (:c A:, !JJ,:)} I.: EN such that
1
:UA:+1- :rl.:+l = 2(:1J~.:- :r:~.:), (k=1,2, ... )

having thereby lim :r1.: = lim 'lJA: = a. The noted process of construction of intervals
1.:--+ +oo 1.:--++oo
is interrupted when, for example, interval length becomes lesser than in advance given
small positive nmnhcr E. This method can be cleHcribecl with fom steps:
I. k := 0, :~: 1 = o~, y 1 = {3:
II. k := k + 1, ZJ.: := !(:r:J.: + yk);
III. If
.f(zJ.:)f(:r,,:) < 0 take :r~.:+ 1 := :r:A:, :t/k:+1 := ZJ.:,
> 0 take :r/.:+1 := z~.:, :UA:+1 := :U~.:,
= 0 talu! a := z1.:; encl of calculation
IV. If
go to II,
1
<E ZJ.:+ l := 2(:r: !.~+ 1 + :tJI,:+ 1)
end of calculation.
 Lesson V - Nonlinear Equations and Systems

Not<-~ that error <~stimat.iou for approximation of ZJ.:+l is

5.1.3. Program realization

In this sec:tiou w<~ present smne simple prop;rmns for solution of nouliw~ar equa-
It is n~comme1Hk<l to rc~c-u l<T to write a short code for the same pmgrams iu
t,i(mS.
Mathematica, aud, if haviup; pleasmc, in Pascal, i.e. Delphi.
Example 5.1.3.1.
Writ<~ a program for solving eqnatiou

f(:l:) = 1- :r-n + (1- :r:)-n = 0 (a= 0.5 (0.1) 2.8),

using Newton's nwtho<l, with accuracy c = 10- 5 . For initial approximation take :£: 0 =
0.5. On output print value of paranwter a, root :c and corresponding value of f(:r:).
Crit<~rion for interrupting iterative~ process is accuracy E. Namely, we consider th-~ root
to 1Je found with accuracy c if f (:r:) changes the sign in interval (:rn - c, :Dn + c). The
program and output list arc of form:
C========================================================
C SOLVING EQUATION
C 1 - X**(-A) + (1-X)**(-A) = 0
C BY NEWTON'S METHOD
C========================================================
FUNK(X,A)= 1 - X**(-A) + (1-X)**(-A)
PRIZ(X,A)= A*X**(-A-1) + A~(1-X)**(-A-1)
OPEN(6,File='NEWT1.out') .
WRITE(6,10)
10 FORMAT(10X,'A-', 10X, 'X', 12X, 'F(X)'/)
EPS=1.E-5
DO 11 I=5,28
A=I*0.1
X0=0.5
6 X=XO-FUNK(XO,A)/PRIZ(XO,A)
IF(FUNK(X+EPS,A)*FUNK(X-EPS,A).LT.O.) GO TO 7
XO=X
GO TO 6
7 Y=FUNK(X,A)
WRITE(6,20)A,X,Y
20 FORMAT(9X, F3.1, 5X, F9.6, 5X, F9.6)
11 CONTINUE
STOP
END
and the output list of results is
A X F(X)
.5 .219949 -.000014
.6 .267609 .000000
.7 .305916 -.000026
.8 .336722 -.000003
.9 .361641 .000000
1.0 .381966 .000000
1.1 .398689 .000000
1.2 .412563 .000000
1.3 . 424159- -.000084
74 Numerical Methods in Computational Engineering

1.4 .433933 ~.000044

1.5 .442217 -.000024
1.6 .449281 -.000013
1.7 .455337 -.000007
1.8 .460554 -.000005
1.9 .465068 -.000002
2.0 .468990 -.000001
2.1 .472410 -.000001
2.2 .475402 .000000
2.3 .478029 -.000001
2.4 .480340 .000000
2.5 .482380 -.000001
2.6 .484184 .000000
2.7 .485784 -.000001
2.8 .487205 .000000
Example 5.1.3.2.
If the equation f(:r:) = 0 has a .root :1: =a in interval a, /3, where .f(a) .f(/3) < 0,
write a program for finding a root :r: = a with accuracy c, using bisection. Use clonble
precision arithmetic. For program testing use the following example:

.f (:r) = e;r: - 2(:r- 1) 2 , ( a,{J) = ( -O.G, 1.0), E = 10- 12 .

The program code is

C======================,==================================
C SOLVING NONLINEAR EQUATION
C BY BISECTION
C========================================================
DOUBLE PRECISION X,Y,Z,F,FZ,EPS
F(X)=DEXP(X)-2.*(X-1.)**2
OPEN(8,File='Bisect.in')
OPEN(6,File='Bisect.out')
C PRINTING OF TABLE HEADING
WRITE(6,9)
9 FORMAT(2X,'K', 2X,'(',8X,'X(K)',8X,',',8X,'Y(K)',
18X,))) ,5X, 'F(Z(K))) I )
C READ IN THE INITIAL INTERVAL (ALPHA, BETA)
READ(8,10)ALFA, BETA
10 FORMAT(2F5.0)
EPS=1.D-12
K=-1
X=ALFA
Y=BETA
5 K=K+1
Z=0.5*(X+Y)
FZ=F(Z)
IF(K/5*5-K.LT.O)GO TO 25
WRITE(6,20)K,X,Y,FZ
20 FORMAT(1X,I2,2X,'(', D20.13,',' ,D20.13,')' ,2X,D12.5)
25 IF(FZ*F(X))1,2,3
1 Y=Z
GO TO 4
2 IF(K/5*5-K.EQ.O) GO TO 6
GO TO 7
3 X=Z
4 IF(DABS(Y-X) .GE.EPS) GO TO 5
Z=0.5*(X+Y)
 Lesson V - Nonlinear Equations and Systems 75

K=K+1
FZ=F(Z)
7 WRITE(6,20) K,X,Y,FZ
6 WRITE(6,30) Z,EPS
30 FORMAT(/5X, 'A= ', D20.13,' (WITH EXACTNESS EPS =
1 D7. 1, ') ')
STOP
END
and the output list of results is
K ( X(K) Y(K) ) F(Z(K))
0 ( -.5000000000000D+OO,' .1000000000000D+01) .15903D+OO
5 ( .2031250000000D+OO, .2500000000000D+OO) .57870D-01
10 ( .2119140625000D+OO, .2133789062500D+OO) -.29038D-02
15 ( .2132873535156D+OO, .2133331298828D+OO) .70475D-05
20 ( .2133073806763D+OO, .2133088111877D+OO) -.23607D-05
25 ( .2133086323738D+OO, .2133086770773D+OO) .89352D-07
30 ( .2133086337708D+OO, .2133086351678D+OO) .53733D-09
35 ( .2133086343383D+OO, .2133086343820D+OO) .58801D-10
40 ( .2133086343465D+OO, .2133086343479D+OO) .19761D-11
41 ( .2133086343465D+OO, .2133086343472D+OO) .48077D-12
A = .2133086343468D+OO (WITH EXACTNESS EPS = .1D-11)

Example 5.1.3.3.
Write a program for solving nonlinear equation .f (x) = 0 by regula-falsi method

(i = 2,3, ... ).

Iterative process interrupt when the condition f(:J:,;,- c)f(x,i, +c) ~ 0 is fulfilled. For
program testing use the following example:

f(:c) = e'r:- 2(:r: -1) 2 , :co= -0.5,.TI = 1.0, E = 10- 5 .

The program code and output list are of form:
C========================================================
C SOLVING NONLINEAR EQUATION
C EXP(X) - 2*(X-1)**2 = 0
C BY REGULA-FALSI METHOD
C========================================================
F(X)=EXP(X)-2.*(X-1.)**2 .
OPEN (6, File= 'Reg-Fal. out')·
WRITE(6,10)
10 FORMAT(9X,'I', 1bX,'Xi',14X, 'F(Xi)' / )
XO= -0.5
X1=1
I=2
3 X=(XO*F(X1)-X1*F(XO))/(F(X1)-F(XO))
Y=F(X)
WRITE(6,20)I,X,Y
20 FORMAT(8X,I2,5X,E14.7,2X, E14.7)
IF(F(X-1.E-5)*F(X+1~E-5)) 1,1,2
2 X1=X
I=I+1
GO TO 3
1 STOP
END
76 Numerical Methods in Computational Engineering

and the output list of l'<$nlts is

I Xi F(Xi)
2 .3833067E+OO .7065066E+OO
3 .2476403E+OO .1489089E+OO
4 .2200995E+OO .2971124E-01
5 .2146460E+OO .5861370E-02
6 .2135718E+OO .1153714E-02
7 .2133604E+OO .2269801E-03
8 . 2133188E+OO .4465835E~04
9 .2133106E+OO .8795895E-05
Example 5.1.3.4.
For polynomial of form P(:r;) = a 1 z 3 + a 2 z 2 + a3 z + a4 (a 1 #- 0) write a program
for determination of zeros using followi11g algorithm:
1° One root find using Newton's method (see previous examples) with accuracy
10- 7 (i:rn+l- :r:.,i < 10- 7);
2° With, in this way ohtaim~d zero z 1 evaluate coefficients of polynomial Q(z) =
P(z)/(z- zl); ·
3° Solv<~ the square equation Q(z) = 0 by standard formula.
For c.alculation of polynomial value use subroutine of type FUNCTION. Algorithm
steps 1° and 2° rcaliz<~ in subprogram of type SUBROUTINE. The program should solve
arbitrary number of equations. On output print coefficients of polynomials P and Q
and roots of equat-ion P(z) = 0.
For program testing use the following polynomials. P(z) = 3z 3 - 7 z 2 + 8z - 2 and
P(z) = z 3 - 5z 2 - z + 5.
For calculation of polynomial value, subprogram of type FUNCTION, named PL using
Horm~r's scheme, is writt<~ll. Arguments in parameter list hav<~ the following meaning:
Z - value of argument; .
A - polynomial codficic~nt.s;
N - degree of polynomial.
This subprogram obtains polynomials P(z) a.ud P' (z).
FUNCTION PL(Z,A,N)
DIMENSION A(1)
PL=A(1)
DO 10 I=1,N
10 PL=PL*Z+A(I+1)
RETURN
END
c
c
SUBROUTINE KJ(A,B,C,X1,Y1,X2,Y2)
D=B*B-4*A*C
IF(D) 25,10,20
10 X1=-B/2./A
X2=X1
15 Y1=0.
Y2=0.
RETURN
20 X1=(-B+SQRT(D))/2./A
X2=(-B-SQRT(D))/2./A
GO TO 15
25 X1=-B/2./A
X2=X1
Y1=SQRT(-D)/2./A
Y2=-Y1
RETURN
 Lessou V - Nonlinear Equations and Systems 77

END
c
c
SUBROUTINE NEWT(A,B,N,Z1)
DIMENSION A(1), B(1)
C EVALUATION OF COEFFICIENTS P'(Z)
DO 5 I=1,3
5 B(I)=A(I)*(4-I)
C EVALUATION OF REAL ROOT Z(1)
ZO=O.
10 Zl=ZO-PL(ZO,A,N)/PL(ZO,B,N-1)
IF(ABS(Z1-Z0)-1.E-7) 20,15,15
15 ZO=Z1 ·
GO TO 10
C EVALUATION OF COEFFICIENTS Q(Z)
20 B(1)= A(1)
DO 25 I=2,3
25 B(I)=A(I)+B(I-1)*Z1
RETURN
END
For solving of square equation Q(z) = az 2 + bz + c = 0 we formed subprogram KJ.
Arguments in parameter list of subprogram an~ of following meaning: ·
A, B, C - coefficients of equation;
X1, Y1 - real and imaginary part of the first root of equation;
X2 , Y2 - real and imaginary part of the second root of equation.
For algorithm steps 1° and 2° the subroutine NEWT is written, with following argu-
ments:
A - coefficients of polynomial P;
B - coefficients of polynomial P' and Q;
N - degree of polynomial P (N = 3);
Z1 - real root of equation P(z) = 0 obtained by Newton's method.
Main program and output list of results are of following form:
C========================================================
C SOLVING NONLINEAR EQUATION
C OF DEGREE THREE
C========================================================
DIMENSION A(4), B(3), ZR(3), ZI(3)
OPEN(6,File='POL.OUT')
OPEN(8,File='POL.IN')
5 READ(8,10,END=99)(A(I),I=1,4)
10 FORMAT(4F10.0)
IF(A(1)) 15,99,15
15 CALL NEWT(A,B,3,Z1)
ZR(1) =Z1
ZI(1)=0.
WRITE(6,20) (I,A(I), I=1,4)
20 .FORMAT(/ 22X, 'COEFFICIENTS OF POLYNOMIAL P(Z)'// 5X,
*4 (l A(' , I 1, ') =' , F8. 5, 3X) / /)
WRITE (6, 25) (I, B(I), I=1, 3)
25 FORMAT(/23X,'COEFFICIENTS OF POLYNOMIAL Q(Z)'//5X,
*3('B(' ,I1, ')=.' ,F8.5,3X)// )
WRITE(6,30)
30 FORMAT(/23X, ' ZEROS OF POLYNOMIAL P(Z)'//27X,
*'REAL',8X,'IMAG'/ .)
CALL KJ(B(1),B(2),B(3),ZR(2),ZI(2),ZR(3),ZI(3))
WRITE(6,35) (I,ZR(I) ,ZI(I) ,I=1,3)
35 FORMAT(/18X,'Z(',I1, ')=', 2F12.7)
GO TO 5
78 Numerical Methods in Computational Engineering

99 STOP
END
COEFFICIENTS OF POLYNOMIAL P(Z)
A(1)= 3.00000 A(2)=-7.00000 A(3)= 8.00000 A(4)=-2.00000
COEFFICIENTS OF POLYNOMIAL Q(Z)
B(1)= 3.00000 B(2)=-6.00000 B(3)= 6.00000
ZEROS OF POLYNOMIAL P(Z)
REAL IMAG
Z(1)= .3333333 .0000000
Z(2)= 1.0000000 1.0000000
Z(3)= 1.0000000 -1.0000000
COEFFICIENTS OF POLYNOMIAL P(Z)
A(1)= 1.00000 A(2)=-5.00000 A(3)=-1.00000 A(4)= 5.00000
COEFFICIENTS OF POLYNOMIAL Q(Z)
B(1)= 1.00000 B(2)= .00000 B(3)=-1.00000
ZEROS OF POLYNOMIAL P(Z)
REAL IMAG
Z(1)= 5.0000000 .0000000
Z(2)= 1.0000000 .0000000
Z(3)= -1.0000000 .0000000

Example 5.1.3.5.
Write <1. program fm <~valuation of coefficients of polynomial of form

P( ·~"') = C'n+J.Z n + c~'n'~.,;n- 1 + · · · + C'·2Z + C~1 (Cn+1 = 1)

if all zeros ZA: = :r~.: + 'i /JA: (/;: = L ... , n) are known.
Let

Ph: (z-) <~f

- II(-z- z.,-) -·-
h:
C'(l.:) A:
'!.:+ 1 z + C'(h:)_,k--,1
'li: ,~ + ... + c,(h:)_
2 z + ,1 .
C'(J.:)

·i.=l

Then for polynomial P(z) it l1ol<ls P(z) = Pn(z), i.e. C;. = c;n) ('i = 1, ... , n + 1).
B<~cans<·~ of
P,,:(z) = (z- z,JP,,._ 1 (z)
the following recmTence rdations hold

C 1(l.:) -
-z,,: '1
-
- C'(l.:-1)
'

C i(!..l -- c'U···-1)
'·i.-l -
-
Z/,: 'i
cu.:-1) ('i=2, ... ,1.:),
cu.: l = cu·- 1l.
/,:+1 '··

stmting with cil) = -Zl, ci ) = L

1
Based on pn~vious, the sulmmtiw~ VIETE is written, with following arguments:
Z - vect.or of zeros of l<~ugth N;
N - dq~TC<~ of polynomial:
C - polynomial co<~I-hci<~uts:
KB- ftag (KB = 0 for corn~ctly 12;iwn polyumHial, KB = 1 fm polynomial with <h~­
gree kss thau one).
Program ront.incs arc n~aliz<~<l iu cmnplex mit.lnnetic. Main prognuu. snlmmt.im~,
and outr)nt list with results an~ giv<~ll as follows.
C========================================================
C EVALUATION OF POLYNOMIAL COEFFICIENTS
 L<~ssml V - N oulill<~m Eqnatiims aud Systellls 7D
C FROM GIVEN~ POLYNOMIAL ROOTS
C========================================================
COMPLEX 2(10), C(11)
OPEN(6,File='VIETE.OUT')
OPEN(S,File='VIETE.IN')
C READ-IN POLYNOMIAL ZEROS
2 READ(8,10,END=99) N, (Z(I),I=1,N)
10 FORMAT(I21(10F8.2))
C SUBPROGRAM VIETE
CALL VIETE(Z,N,C,KB)
IF(KB.EQ.O) GO TO 1
WRITE(6,20)
20 FORMAT(I5X, 'DATA ERROR:'
*' POLYNOMIAL DEGREE LESS THAN 1'11)
GO TO 2
C PRINTING ZEROS AND COEFFICIENTS OF POLYNOMIAL
1 WRITE(6,25) (i,Z(I), I=1,N)
25 FORMAT(I16X,'POLYNOMIAL ZEROS'II23X,'REAL' ,5X,
*' IMAG' I I (10X, 'Z (' , I2, ') =' , 4X, F10. 7, 1X, F10. 7) )
N=N+1
WRITE(6,30) (I,C(I), I=1,N)
30 FORMAT(I16X,'POLYNOMIAL COEFFICIENTS'II23X,'REAL' ,5X,
*'IMAG'II(10X,'C(' ,I2,')=' ,4X,F10.7,1X,F10.7) )
GO TO 2
99 STOP
END
c
SUBROUTINE VIETE(Z,N,C,KB)
COMPLEX Z(1),C(1),A,B
IF(N.GE.1) GO TO 5
KB=1
RETURN
5 KB=O
C(1) =-Z (1)
c (2) =1.
IF(N.GE.2) GO TO 20
RETURN
20 DO 15 K=2,N
A=C (1)
C(1) =-Z (K) *C (1)
DO 10 I=2,K
B=C (I)
C(I) =A-Z (K) *B
10 A=B
15 C(K+1)=A
RETURN
, END
POLYNOMIAL ZEROS
REAL IMAG
Z( 1)= 1.0000000 1.0000000
Z( 2)= 1.0000000 1.0000000
Z( 3)= 1.0000000 -1.0000000
POLYNOMIAL COEFFICIENTS
REM_. IMAG
c ( 1) = -2.0000000 -2.0000000
c ( 2) = 4.0000000 2.0000000
c ( 3) = -3.0000000 -1.0000000
C( 4)= 1.0000000 .0000000
80 Numerical Methods in Computational Engineering

Example 5.1.3.6.
· Write a program for evaluation of complex root of transcendental equation f (z) = 0
using Newton's method · .

Zn +1 = Zn - l.f1(zn)
( )
( 1
.f (Z.
11.) =/::- 0
)
1
. Zn

where Zn = :r:n + ·i Yn (n. = 0: 1, ... ). Iterative process is to be interrupted when the

conditions

where c is in advm.1ce given accuracy, are contemporary fulfilled.

Program is organized in the following way:
1° Functions Re(f(z)), Im(.f(z)), Re(f 1 (z), Im(.f 1 (z))) are given in subroutine of type
FUNCTION; . .
2° In subprogram of type SUBROUTINE is caknlat;~d one iterative step of Newton's
method;
3° Main program reads in initial values :r 0 , y0 , c, calls subroutine for iteration, checks
the termination condition awl prints result. .
The example taken is f(z) = e'1: - 0.2z + 1,z0 = 1+ Jr'i,E = 10- 6 . Two separate
programs are written, in real and in complex aritlnnetics.
By separation of real and imaginary part in Newton's formula, one gets

:cn+l = :rn- ~ (Re(f(z. .))Re(.f (zn)) + Im(f(zn))Im(.f

11
1 1
(zn)))

Yn+l = /Jn- ~ (Im(.f(z.n.))Re(.f (zn))- Re(.f(zn))Im(f

1 1
(zn))),

where 6 = lf'(zn)l 2 = 1Rc{l'(zn))l 2 + 1Im(.f 1 (zn))l 2 .

Being, in our case\ f(z) = cz- 0.2z + 1 and f 1 (z) = ez- 0.2, we have Rc~(.f(z)) =
ez cosy- 0.2:r: + 1, Im(f(z)) = e'1' siuy- 0.2y, R({l'(z)) = c'1' cosy- 0.2, Im(f 1 (z) =
e'r: siny), what is given hy fnnction subroutine EF.
Program routines arc n~alized in real aucl complE~x arithmetic. Main program,
subroutine, and output list with results are given as follows.
C========================================================
C EVALUATION OF COMPLEX ROOT OF TRANSCENDENT
C EQUATION F(Z)=O BY NEWTON'S METHOD
C USING REAL ARITHMETIC
C========================================================
OPEN(6,File='NEWT-TRR.OUT')
OPEN(S,File='NEWT-TRR.IN')
READ(8,5) XO, YO, EPS
5 FORMAT{2F10.0,E5.0)
WRITE(6,10)
10 FORMAT(//10X, 'NEWTON''S METHOD FOR SOLVING TRANSCEN'
*'DENT EQUATION'//18X,'F(Z)=EXP(Z) - 0.2*Z + 1 = 0'
*//5X,'ITER.No. ',4x,'REAL(Z)',5X,'IMAG(Z)' ,4X,
*'REAL(F(Z))',2X,'IMAG(F(Z))'/)
ITER=O
KBR=1
15 A=EF(XO,Y0,1)
B=EF(XO,Y0,2)
WRITE(6,20) ITER,XO,YO,A,B
20 FORMAT(5X,I4,2X,2F13.7,2F12.6)
GO TO (22,50),KBR
22 ITER=ITER + 1
 Lesson V - Nonlinear Equations and Systems 81

XS=XO
YS=YO
CALL TRANS(XO,YO,A,B,R)
IF(R) 25,25,35
25 WRITE (6,30)
30 FORMAT(//5X,'FIRST DERIVATIVE OF FUNCTION= 0')
GO TO 50
35 IF(ABS(XO-XS)-EPS) 40,40,15
40 IF(ABS(YS-YO)-EPS) 45,45,15
45 KBR=2
GO TO 15
50 WRITE(6,55)EPS
55 FORMAT(/5X,'SPECIFIED ACCURACY OF CALCULATION '
*'EPSYLON = ',E7.1)
STOP
END
c
c
FUNCTION EF(X,Y,I)
GO T0(10,20,30,40),I
10 EF=EXP(X)*COS(Y)-0.2*X+1
RETURN
20 EF=EXP(X)*SIN(Y)-0.2*Y
RETURN
30 EF=EXP(X)*COS(Y)-0.2
RETURN
40 EF=EXP(X)*SIN(Y)
RETURN
END
c
c
SUBROUTINE TRANS(XO,YO,A,B,R)
C=EF(XO,Y0,3)
D=EF(XO,Y0,4)
R=C*C+D*D
IF(R) 5,10,5
5 XO=XO-(A*C-B*D)/R
YO=YO-(B*C-A*D)/R
10 RETURN
END
NEWTON'S METHOD FOR SOLVING TRANSCENDENT EQUATION
F(Z)=EXP(~) - 0.2*Z + 1 = 0
ITER.No. REAL(Z) IMAG(Z) REAL(F(Z)) IMAG(F(Z))
0 1.0000000 3.1415920 -1.918282 -.628317
1 .3426673 2.9262880 -.444708 -.284296
.2 .0372190 2.7002840 .054076 -.096737
3 .0497327 2.6425620 .067235 -.025535
4 .0911207 2.6459620 .018186 -.008234
5 .1015006 2.6458960 .006090 -.002721
6 .1049549 2.6459040 .002026 -.000909
7 .1060995 2.645904.0 .000678 -.000304
8 .1064820 2.6459040 .000227 -.000102
9 .1066103 2.6459040 .000076 -.000034
10 .1066533 2.6459040 .000026 -.000011
11 .1066677 2.6459040 .000009 -.000004
12 .1066726 2.6459040 .000003 -.000001
13 .1066742 2.6459040 .000001 .000000
14 . 106674 7 2. 6459040 . 000000 . .000000
82 Num<~rical Methods in Computational Engineering

SPECIFIED ACCURACY OF CALCULATION EPSYLON = .1E-05

The given snbprogrmns and main program can be rather simple realized in complex
aritlnn<~tic, what is, with n~sults, giw~n in continuation.

C========================================================
C EVALUATION OF COMPLEX ROOT OF TRANSCENDENT
C EQUATION F(Z)=O BY NEWTON 1 S METHOD
C USING COMPLEX ARITHMETIC
C========================================================
COMPLEX Z,ZO,F,Y,A
OPEN(6,File='NEWT-TRC.OUT')
OPEN(8,File='NEWT-TRC.IN')
READ(8,10) ZO
10 FORMAT(2E14.7)
EPS=1. E-6
WRITE(6,20)
20 FORMAT(//10X, 'NEWTON''S METHOD FOR SOLVING TRANSCEN'
*'DENT EQUATION'//18X,'F(Z)=EXP(Z) ~ 0.2*Z + 1 = 0'
*//5X,'ITER.No. ',4x,'REAL(Z)',5X,'IMAG(Z)',4X,
*'REAL(F(Z)) ',2X,'IMAG(F(Z))'/)
ITER=O
Y=F(Z0,1)
13 WRITE(6,30)ITER,ZO,Y
30 FORMAT(5X,I4,2X,2F13.7,2F12.6)
Y=F(Z0,2)
B=CABS(Y)
IF(B.EQ.O.) GO TO 99
CALL NEW(Z,ZO)
ITER=ITER+1
Y=F(Z,1)
A=Z-ZO
IF(ABS(REAL(A)) .GT.EPS) GO TO 95
IF(ABS(AIMAG(A)).LE.EPS) GO TO 98
95 ZO=Z
GO TO 13
99 WRITE(6,40)
40 FORMAT(//5X,'FIRST DERIVATIVE OF FUNCTION= 0')
GO TO 97
98 WRITE(6,30) ITER,Z,Y
97 WRITE(6,55) EPS
55 FORMAT(/5X, 'SPECIFIED ACCURACY OF CALCULATION '
*'EPSYLON = ',E7.1)
STOP
END
c
c
COMPLEX FUNCTION F(Z,I)
COMPLEX Z
GO T0(1,2)I
1 F=CEXP(Z) - 0.2*Z + 1
RETURN
2 F=CEXP(Z) - 0.2
RETURN
END
c
c
SUBROUTINE NEW(Z,ZO)
COMPLEX Z,ZO,F
Z=zn - F(Z0,1)/F(Z0,2)
 Lesson V - Nonlinear Equations and Systems 8:5

RETURN
END
NEWTON'S METHOD FOR SOLVING TRANSCENDENT EQUATION
F(Z)=EXP(Z) - 0.2*Z + 1 = 0
ITER. No. REAL(Z) IMAG(Z) REAL(F(Z)) IMAG(F(Z))
0 1.0000000 3.1415920 -1.918282 -.628317
1 .3426675 2.9262880 -.444709 -.284296
2 .1036775 2.7002840 -.023705 -.066273
3 .1054019 2.6458710 .001517 -.000634
4 .1066756 2.6459040 -.000001 .000000
5 .1066750 2.6459040 .000000 .000000
SPECIFIED ACCURACY OF CALCULATION EPSYLON = .1E-05

5.2. Systems of nonlinear equations

5.2.1. Newton-Kantorowitch (Raphson) method

Consider syst<~m of nonlinear equations

(5.2.1.1) f,;(:r1, ... ,:rn) =0 ('i= 1, ... ,n).

By taking .i! = [:1: 1 ... :r:n]T, (-} = [0 ... O]T, where (-} is null-vector, we can write

(5.2.1.2)

Basic iterative method for solving equations (5.2.1.2) is method of Newton-Kan-

torowich, which generalizes classical Newton's method. Fundamental results regarding
existence and uniqueness of solutions of eq. (5:2.1.2) and convergence of the method
are given by L.V. Kantorowich (see [22]).
Let a = [a1 (],nV be exact solution of this system. Using Taylor development
0 0 0

for functions which appear in (5.2.1.1), we get

EJ f,; ( _ .1.1
' k:) )
to • (
. '/, a1, 0 0. ,an
) _ f" •(,
- . '/, .1.1
0 ( h:)
'0 0 0
0

,.1.n
( k) )
+ u:E1
~) 0,1
0 (

+ 0 0 0

of; (an _ :Ln

+.·uXn
;l
, .(kl) + ,.(k)
. r ·i (i=1, ... ,n),

where partial derivatives on the right-hand side of given equations are calculated in
point :£(k). T.~k:) represents corresponding remainder term in Taylor's formula.
Because of .f,;, ( a 1 , ... , an) = 0 ('i = 1, ... , n), previous system of equations can be
represented in matrix form

and
84 Nmnerical Metlwds in Computational Engineering

where .p(h:) = [,.ili:J ... ,..~~h:)f. If .Tac:ohiau matrix for .f is regular, then we have

By neglecting the very last member on tlH~ right-hand size, in place of of vector r7 we
get its new approximation, denoted with :l! (h:+l)_ In this way, one gets

(G.2.1.3)

where :J!(I.:) = [:r:i":) ... :r:g·')]r. This method is oft!~n called Newton-Raphson method.
Mdhod (G.2.1.3) can lH~ modific~~d in the sense~ that inverse matrix of VV(:i') is not
evahw.tc~cl at every step, hut only at first. Thus,

(5.2.1.4)

In [1, pp. 384-38G] the Ncwton-Kautorowich method is illustrated with systc~m of

nonlim~ar ~~quationin two unknowns. It is suggested to n~ader to write a program code
iu Mathematica and Fortran.
Example 5.2.1.1. Solve Uw sysl;em of nonlinear equation

f.(·1·
.1·•1., ·1· 2 ) -
.. <l·J·
- ;; 2 ·r·
.. 1·•2 + 4·r·' '22 - 3G' :-
- 0
h(:t:l. :r:2) = l6:r~- :r:t + :r:2 + 1 = 0,
which has a. solHtion in first r1w1.dnwt (:r:1; :r2 > 0).
Using graphic pn~sent.at.ion of implicit functions .fi and h in first quadrant. on<~
can sc~e that solntion i7. is locate< l in tlw neighborhood of point ( 2, 1), so that w<~ tak<~
for initial vector :J!(O) = [2 l]T, i.e~. :t:~o) = 2 and :r~o) = 1.
By partial clcrivatiou of .fi ;t.ncl h one gds the .Jacohian

9:r:i + 8:1:2] .
32:r:2 +1 '

awl its invPrsc-~

vv-1 (·/!)
.
= _1_ [ 32:r2 .+ 1
A(,-:)
Ll .J, ±.'/'.3
Ll
'1
- (9:r:~ -~ _8:1:2)
18.t 1·[, 2
l'
when~
6(:/!) = l8:r:1:r:2(32:r:2 + 1) + ,±:d(9:I:i + 8:1:2).
By pntt.ing .t.?'') =
f.;(:r:~l.:). :r:~l.:)) aud 61., =
6(:rY.')) ('i = 1, 2; k = 0, 1 ... ) in t.he S<:il.lar
form of Newtou-Kaut.orowich f(mnnla (5.2.1.3), we~ get the iteration formula

' '(1.: +1 ) -
./,1 -
' . ( /,;) -
·1'1
-
1- { (.) ')' .(1.:)
·J~.I")
+ ]-) t• 1(1.:) - ( 9'
.L1
'(/.:) ~ + °· 1 (1.:) ) t• (/;:) }
Q' .
·?
6,,, - . - . ')- ·
'
,. (/.: + 1)
·1·2
-
-
' . ( /,;) -
·1·2
_.!:_____ {
6
I ' . ( /.:) :1
-±./.1
t• (/,;) + 1o./.1
. 1
<..>' , (/,:) ' , (/.:) t• (/.;) }
.1.2 . 2 _ ·
/,;

The appropriate~ Fortran c:odc~ for solving givcu uoulincar equation is

Double precision x1,x2,x10,x11,x20,x21,f1,f2,Delta,EPS
F1(x1,x2)=9*x1**2*x2 + 4*x2**2-36
F2(x1,x2)=16*x2**2 - x1**4 + x2 + 1
 L<~ssou V- Nouliw~ar Eqnatious and Systems

Oelta(x1,x2);18*x1*x2*(32*x2+1)+4*x1**3*(9*x1**2+8*x2)
Open(1, File='Newt-Kant.out')
x10=2.d0
x20=1.d0
EPS=1.d-6
Iter=O
write (1, 5)
5 format(1h ,// 3x, 'i' ,7x, 'x1(i) ',9x, 'x2(i) ',
* 9x, 'f1(i) ', 9x, 'f2(i) '/)
write(1,10)Iter, x10,x20,F1(x10,x20),F2(x10,x20)
1 x11=x10-((32*x20+1)*f1(x10,x20)-(9*x10**2+8*x20)*
* f2(x10,x20)) /Oelta(x10,x20)
x21=x20-(4*x10**3*f1(x10,x20)+18*x10*x20*f2(x10,x20))
* /Oelta(x10,x20)
Iter=Iter+1
write(1,10)Iter, x11,x21,F1(x11,x21),F2(x11,x21)
10 Format(1x,i3, 4014.8,2x)
If(Oabs(x10-x11).lt.EPS.and.Oabs(x20-x21) .lt.EPS)stop
If(Iter.gt.100)Stop
x10=x11
x20=x21
go to 1
End
and the output list of results is
i x1(i) x2(i) f1(i) f2(i)
0 .200000000+01 .100000000+01 .400000000+01 .200000000+01
1 .198305080+01 .922958400+00 .731363450-01 .881108350-01
2 .198370710+01 .920743220+00-.286940530-04 .683484410-04
3 .198370870+01 .920742640+00-.103241860-10-.569948530-10
4 .198370870+01 .920742640+00 .000000000+00-.155431220-14
Example 5.2.1.2. vVrite p1·op,'l·am f(n· approximEI.tive solving a system of equations
F(:~:, y) = 0,
G(:r, y) = 0,
where F and G are contimzous differentiable iiznctions, using Newton-Raphson Jnetlwd

('n. = 0, 1, 2, ... )

starting with some approximate val11es of :ro and Yo, where

J(:r:n, Yn) = # 0.

F,I, (:Dn, !In) F~(:r:n, Yn)

1
6:rn =
J(:r:n, Yn)
G (:r n , lin.) a;lrn, Yn)

F;~(:rn, Yn) F(:rn, y.,J

1
6yn =
J(:I:n, :tin) G~,(:r.~~., Yn) G(:rn, Yn)
86 Numerical Methods in Computational Engineering

Partial derivathres are to be ol>tainerl mzmericaJly. The itern.tions inteiTupt when tlw
conditicms
j:r:n+l- :J:,nl s
E, and IYn+l- Ynl E, s
an! fizlfilled. t is accuracy given in advance. For test exn,mple take

F(:r,:y) = 2:r 3 - ;y 2 -1 =0
3
G(:r:, y) = :ry - y- 4 = 0,

with initial values :c 0 = 1.2, y 0 = 1.7 and E = 10- 10 .

For obtaining partial derivatives of function .f(:r, y) we use following expi"essions

Df ~ .f(:r: + h, y)- .f(:r:- h, u)

U:r 2h
D.t ~ f(:t:, y +h) - f(:1~·, y- h)
Dy 2h

where h is small <~nongh iucrcnwnt (here is tak<m h 10- 5 ). Fuuctions F and G

are given in subprogram of typ<~ FUNCTION. Program is realized iu clonhle precisiou
arithmetic. Program code is of following form:
C========================================================
C SOLVING OF SYSTEM OF NONLINEAR EQUATIONS
C BY NEWTON-RAPHSON METHOD
C=====================~==================================
IMPLICIT REAL*8 (A~H,O-Z)
DIMENSION R(2)
OPEN(6,File='NEWT-RAPH.OUT')
OPEN(8,File='NEWT-RAPH.IN')
READ(8,15) XP,YP,EPS,H
15 FORMAT(4D10.0)
WRITE(6,16)
16 FORMAT(/10X,'NEWTON-RAPHSON''S METHOD FOR SYSTEMS OF'
*' EQUATIONS'///30X,'2.*X**3-Y**2-1.=0. '//
*30X, 'X*Y**3-Y-4.=0. '///1X,'ITER. ',6X,'X' ,15X,'Y',12X,
*'F(X,Y)' ,10X, 'G(X,Y) '/)
ITER=O
20 ITER=ITER+1
CALL NEWT(XP,YP,XK,YK,FD,H,A,B)
IF(FD) 5,6,5
6 WRITE(6,17)
17 FORMAT(/1X,'JACOBI MATRIX SINGULAR')
GO TO 90
5 DO 8 I=1,2
8 R(I)=EEFF(I,XK,YK)
WRITE(6,18) ITER,XK,YK,(R(J),J=1,2)
18 FORMAT(1X,I3,4F15.10)
IF(DABS(A)-EPS) 30,30,40
30 IF(DABS(B)-EPS) 90,90,40
40 XP=XK
YP=YK
GO TO 20
90 STOP
END
c
c
SUBROUTINE NEWT(XP,YP,XK,YK,FD,H,A,B)
 Lc~ssou V - Nouliucm· Equations aucl Syst.Plus ....
01
-
IMPLICIT REAL*8 (A-H,O-Z)
DIMENSION R(2),DX(2),DY(2)
X=XP
Y=YP
DO 4 I=1,2
R(I) =EEFF (I, X, Y)
DX(I)=0.5DO/H*(EEFF(I,X+H,Y)-EEFF(I,X-H,Y))
4 DY(I)=0.5DO/H*(EEFF(I,X,Y+H)-EEFF(I,X,Y-H))
FD=DX(1)*DY(2)-DY(1)*DX(2)
IF(FD) 5,6,5
5 A=(R(1)*DY(2)-R(2)*DY(1))/FD
B=(R(2)*DX(1)-R(1)*DX(2))/FD
XK=X-A
YK=Y-B
6 RETURN
END
c
c
FUNCTION EEFF(J,X,Y)
IMPLICIT REAL*8 (A-H,O-Z)
GO TO (50,60),J
50 EEFF=2.DO*X**3-Y*Y-1.DO
RETURN
60 EEFF=X*Y**3-Y-4.DO
RETURN
END
and the output list of n~snlts is
NEWTON-RAPHSON'S METHOD FOR SYSTEMS OF EQUATIONS
2.*X**3~Y**2-1.=0.
X*Y**3-Y-4.=0.
ITER. X y F(X,Y) G(X,Y)
1 1.2348762633 1.6609796808 .0073200054 -.0022831784
2 1.2342746753 1.6615262759 .0000023823 -.0000008838
3 1.2342744841 1. 6615264668 .0000000000 .0000000000
4 1.2342744841 1.6615264668 .0000000000 .0000000000

5.2.2. Gradient method

Because Newt.on-Kautorowich mctlw(l demands .obtaining inverse operator F'(v~,
what can be vc~ry coml>lic:atccl, mi.cl c~vcn impossible, it have been developc~cl a whole
class of quasi-Newton methods, which nse sonwapproximat.ionR of noted operator (see
[21], [22], [23]: [24]). One~ of this mctlwds is graclic~nt method.
Consider system of uonliw~ar cqnations (G. 2 .1.1), with matrix form

(G.2.2.1) .run = o.
The gradient. mcthocl for solving a given system of equations is based ou minimi!';ation
of functional
'fl,

U(:J!) = L f,;(:r1, ... , :1~ 11 ) 2 = cf(:;), }(:;)).

·i.=l

It is easy to show that. t.lw cqniva.leuce U(:E) = 0 -¢:::=::} _{(:;) = 0 holds.

Suppose that equation (5.2.2.1) has unique solution :Z = a, for which functional
U riches minimum value. Let :z(o) be initial approximation of this solution. Let ns
88 Numerical Methods in Computational Engineering

construct series {:G(A:)} tmch that U(:£( 0 )) > U(:i!( 1 )) > U(:f!( 2 )) > · · ·. In a same way as
at linear equationR, we take

(5.2.2.2) .-:(k+I)
,[, = ,(, , "U(·-:(k))
.-:(k) _ /\/;; v ,[, (k = 0, 1, ... ),

where \7U(:1) = grad(£) = [

au
!), . · · ·
DU] T
~ . Parameter Ak: is to be determined from
u.1.1 u.l.n
condition that scalar function S, defined with S(t) = U(:if(k) - t\7U(:if(k))) has a min-
imum value in point t = Ah:. Having in mind that equation S' (t) = 0 is non-linear,
proceed its lineari7,a.tion aronud t = 0. In thiR case we have

so that linearized equation is

wherefrom we obtain
n
""""H·
6 '/., f'·(·-:(k))
'/. .(,
\ ·i.=1
(5.2.2.3) /\1.
"·
== t = - -n - - -
~H[
·i.=1

where we put H;, = (\7f;.(:f!(h:)), \7U(:f!(h:))) (i = 1,, .. , n). Because of

au _ _!!___{~i··(·-..)
• - ~ '
2} '/, .[,
_
-
?~f·-(.-..)af,(x)
~
.:...1 :J, • ' '/,

D:r:.1· U:r: .1· .

•=1
.
•=1
D.r.1·

we have

(5.2.2.4)

where W(:?,) is .Jacobian matrix.

According to previonR, (5.2.2.3) reduceR to

1 (f-;'(A:) w.wrt-:'(k))
\ - . ' h. /;;.
/\f· - -. - -
· 2 (Wl.wr
"· h: .
t· (J,:) ' w,.wr
•. h: .
f' (A:l)

where [(h:) = /(:f(l.:)) and W 1,, = W(:f(l.:l). Finally, gradient method can be represented
in the form

(5.2.2.5)

As we see, iu plac<~ of matrix w- 1 (:7(1.:)) which appears in Newtou-Kantorowich

method, we h<tv<~ HOW matrix 2A/,: W[.
 LPI"I"Oll V- Nonlim~ar Equations and Systems

Example 5.2.2.1.. System ofwmlinear eqw1.tion-; given in example 5.2.1.1 willl>f~ solved
using gmdient metlwrl, starti11g with tlw same i11itial vector :J:(O) = [2 1jT. giviw~ t;lw
following list of results · '

l x1(i) x2(i) 2lam_k

0 .20000000000+01 .10000000000+01 .3057873950-03
1 .19755370080+01 .92599945040+00 .5387476890-03
2 .19832101790+01 .92018713060+00 .3395536230-03
3 .19836435590+01 .92078400320+00 .5355965390-03
4 .19837052300+01 .92073878450+00 .3393286040-03
5 .19837082700+01 .92074293170+00 .5355733540-03
6 .19837087090+01 .92074260960+00 .3393325160-03
7 .19837087310+01 .92074263910+00 .5359906240-03
8 .19837082340+01 .92074263680+00 .3377933010-03
9 .19837087340+01 .92074263700+00
Nok that the conwrgcw:c hen~ is nmch slower than at Newton-K;:mtorowidl
method, clue to fact that p;radicnt nwtJwd is of first order.
Gn1(lient lll(~thod is successfully US(~d in many optimization pro blcms of nonlinear
programmiug, with large 11mnber of nwthods, especially of gradient typ<\ which are
basis fm muuber of prognumning padmges for solving nonlinear programming problems
(see nwnograph [G] for sylllholic implemeutation of uoulinear optimization).
The methods of mlconl"traincd optimizatiou, hasecl ou derivatives of goal fnnctiou
can be roughly sqmratc(l iu two da.sscs. To the first group belong methocls which use
only first derivative of goal function aud tlwy an~ called gradient methods of .first
order. The most known gmclicnt. metho(l of first order is Cauchy method of st(-)(-)pest
descent. This nwthod has linear couv(~rgence alHl characteristic of good progress to
optimum point from "distauf' iuitial approximations, but slow convcrgeuce in vicinity
of optimum point. To the second class belong methods which nse first and second
derivative of goal fund. ion (m SO Ill(~ approximatiou of them) which are called grad'ient
methods of second order. The most known method of Sl)C(mcl orckr is Newtou·s
method. It is characterized hy square conv(-)rgcnC<), wh<tt means that, when couw~rgf~S,
is faster than Cauchy's m(~1,h~Hl. But, it is less reliable than Cauchy's method. Best
characteristics of 1lOth nwtlHH ls belong to so kw JWll Qnat>i-N (~wton (Variable liH~tric)
methods. Thes(~ nwtJwds have at l(~ast liw~ar onlcr of convergence, with quadratic
asymptotic PlTOl'.
Senne basic t,()l'lllS, ucu)ssary for mHh-)rstanding of gradient methods of optimization
are given below.
Vector-gradient VQ in n-dimensional space has n components, equal to partial
derivatives in every govcruing paramct(~r in point :Zh,, i.e.

(5.2.2.6)

where Q(:Z) Q(:1: 1 , :~: 2 .... , :I:n) Dnc to simplicity, we will dcuotl~ q(h:) = Q(:Z(h:)), so
that gradient in point :J:(k) will be .
DQ(I.:) aQU•:)}
VQ('r(/.:)) -- horHl =
(5.2.2.7) ' . ' ,,
(1(·?(/;:))
' . • {- - ' ... ) -
!l,.
u.I.l
-
!l,.
u.l,n

Gradient vector VQ(T) is i11 CV(~l'Y point :Z(i.:) = (:r:~k:)), ... , :r:.~/.·')) normal to the
plane with constant value Q(f) awl go(~S tlmmgh given point. This vcctm has in ev(TY
point f(h:) orieutation of fast<~st grow of Q (:Z) fi:·om this point. Algorithm of gradil:nt
optimization methods consists in procedure that, starting from given or computed pomt
:Z(k), one goes to the next point :zU:+l) with step D.:Z(Ii:) in din~ction of gradient, clming
calculation of maximum
(5.2.2.8)
90 Numerical Methods in Computational Engineering

or in opposite direction of gradient, when calculating minimum

(5.2.2.9)

When given parameter of step j{(k) = (h~k)), i = 1, ... , n, move in direction of gradient
is realized by formulas ·

' (h:+l) - ' (k:) (k:) fJQ(k:)

(5.2.2.10) r;.
• ?.
- r;.
• '1.
+h..,. -
fJ.x- . (·i-
- 1' ... ' n) '
. "/,
during finding a maximum, and

(k:)
(5.2.2.11) • (h:+l) - • (k:) (k:) fJQ
.E.i - .Ei -hi - - - (1: = 1, ... , n),
8 Xi
during finding minimum of function Q(x). In formulas (5.2.2.10) and (5.2.2.11) the
move is in direction of gradient only if all h.;k), (i = 1, ... , n) are same. Nevertheless, in
some methods are steps chosen arbitrary, or by some criteria.
At gradient methods the forrnulas with coordinates of normalized gradient vector
can be uses.

(5.2.2.12) (1:=1, ... ,n),

In formula (5.2.2.12) normed gradient-vector shows up to the direction of fastest

change of goal function, but does not define the velocity of moving through extremum.
This is given by steps, hih:), ('i = 1, ... , n). Normalization of gradient improves method
stability.
. Theoretically, the procedure of gradient search terminates in point in which all
coordinates of gradient are equal to zero, i.e. in which is Euclid's norm of gradient
equal to zero:

(5.2.2.13)

Thus, the following criterion for termination of gradient search may be used:

(5.2.2.14)

where c is given small uumber. For the same criterion can be used also Chebyshev
gradient norm

I fJQ(h:) I
(5.2.2.15) L--
n

·i.=l
fJ:J:
~c.
i
.

The exposed formulas (~uable writing a code in procedural and symbolic languages
for gradient methocls, what is snggc~sted to readers.
 L<~ssml V - Nonliw~ar Equations and Systems ~1

5.2.3. Globally convergent methods

V!e hav~ seen th~1t Newtons _met.hoc l and N ewtou-like nwthods (quasi- N ewtonnwth-
?~s) for solv1_ng nonlm~~<~r equatiOns has an unfortunate tendem:y not to converge if thf~
nnt.Ial guess IS not sufficwntly dose to the root. A global method is one that cm1vc-~rges
to a solution from almost any starting point. Therefore, it is om goal to clt~vdop an
algorithm that combines tlw rapicl local c:onv<~rgeuce of Newtous method with a glob-
ally convergent strategy that, will guarantee sonw progress towards the solution at each
iteration. The algorithm is dosdy rdated to the quasi-Newton method of minimizatiou
(see [S], p. 376).
From (5.2.1.3), N<-~wton-R.aphson method, w<~ have so known Newton step in itera-
tion formula

(5.2.3.1) (k = 0, 1, 0 0 .)

where W is .Jacobian matrix. Tlw qnest.iou is how one should decide to accept tltr~
Newton step rh? If w<~ <lcnotr~ F = .{(:J!U')), a reasonable strat<~gy for step acceptance
is that IFI 2 = F · F <l<~cn~as<~s, what is the same requirement oJH~ would impose if trying
to minimize

1
t = -F·F.
o
(S.2.3.2)
0 2

Every solution of (5.2.1.1) miuimizes (5.2.3.2), but there may be some local minima
of (5.2.3.2) that an-~ not solution of (S.2.1.1). Thus, simply applying some minimum
finding algorithms can be wrong.
To develop a l><~t.ter strategy, note that Newton step (5.2.~~.1) is a descent direction
for .f:

3 3.)
r. ') ..
( o.~ \lf · r):J! =(F. W) · (- w- 1 . F)= -F. F < 0.
Thus, the strategy is qnite simple. One shonld first try the full Newton step. l H'<·ause
once we are close enough to the solntion, w<-~ will get quadratic convergence. lluwl~V<~r,
we should check at each iteration that tlw proposed step rwlnces f. If not, we go
back (backtrack) along tlw Newton direction until we get acceptable step. Because the
Newton direction is descent direction for f, we fill find for sure an acceptable step by
backtr a eking.
It is to mention that this strategy essentially minimizes f by taking Newton steps
determined in such a way that bring \l.f to zero. In spite of fact that this method can
occasionally lead to local minimnm -of .f, this is rather rare in practice. In such a case,
one should try a new starting point.

Line Searches and Backtracking

V\Then we are not close enough tt) the minimum off, taking the full Newton step jJ =
r5:? need not decrease Uw fnnction; wc~ may move too far for the quadratic approximation
to be valid. All we an~ guarante<~cl is that initially .f decreases as we move in the Newton
direction. So the goal is to mow~ to a new point :r:new along tlw direction of the N<~wton •
step j), but not necessarily all the way (see [5], pp. 377-378):

(5.2.3.4) (0<>.::;1)

The aim is to fiml /\.so that f(:l! 01 cJ+>.iJ) has decreased sufficiently. Until the early 1970s,
standard practice \\-;Is to choos<-~ >. so that :l!.,.,.,w exactly minintizes .f in t~h<-~ direction fl.
However, we now know that it is extremely wasteful of function evaluations to do so. A
better strategy is as follows: Sine<~ jJ is always the Newton direction in our algorithms, •
we first try ), = 1, the full Newton step. This will lead to quadratic convergence
·when :I! is sufficiently close to the solution~ . However, if .f(.i!nc·w) does not meet onr
92 Numerical Methods in Computational Engineering

acceptance criteria, we backtrack along the Newton direction, trying a smaller value
of A, until we find a suitable point. Since the Newton direction is a descent direction,
we are guaranteed to decrease of for sufficiently small A. What should the criterion for
accepting a step be? It is not sufficient to i·equire merely that f(xne-w) < f(iotd)· This
criterion can fail to converge to a minimum off in one of two ways. First, it is possible
to construct a sequence of steps satisfying this criterion with f decreasing too slowly
relative to the step lengths. Second, one can have a sequence where the step lengths
are too small relative to the initial rate of decrease of .f. A simple way to fix the first
problem is to require the average rate of decrease off to be at least some fraction a of
the initial rate of decrease \7 f · j)

(5.2.3.5)

Here the pararneter a satisfies 0 < a < 1. We. can get away with quite small values
of a; a = 10- 4 is a good choice. The second problem can be fixed by requiring the
rate of decrease of f at :Ene·w to be -greater than some fraction f3 of the rate of decrease
of f at Xold· In practice, we will not need to iinpose this second constraint because
our backtracking algorithm will have a built-in cutoff to avoid taking steps that are too
small.
Here is the strategy for a practical backtracking routine. Define

(5.2.3.6) q(A) = f(:lald + APJ

so that

(5.2.3.7) g'(A) = 'Vf j) 0

If we need to backtrack, then we model g with the most current information we have
and choose A to minimize the model. We start with g(O) and g'(O) available. The first
step is always the Newton step, A = 1. If this step is not acceptable, we have available
g(1) as well. We can therefore model g(A) as a quadratic:

(5.2.3.8) g(A) ~ [g(1)- g(O)- g'(0)],\ 2 + g(O).

By first derivative of this function we find the minimum condition

A- - g'(O)
(5.2.3.9)
- 2[g(1)- g(O)- g'(O)].

Since the Newton step failed, we can show that >.,::; ~ for small a. We need to guard
against too small a value of A, however. We set Am.in = 0.1.
On second and subseqU<~nt backtracks, we model g as a cubic in A, using the previous
value g(A 1 ) and the second most recent value g(A2)·

(5.2.3.10) y(A) = aA 3 + bA 2 + g'(O)A + g(O)

Requiring this expression to give the correct values of g at A1 and A2 gives two
equations that can be solved for the coefficients a and b.

(5.2.3.11)
-1/ A~]
AI/A~
[g(AI)- g'(O)AI- g(O)
. g(A2)- g'(O)A2- g(O) .
l
The minimum of the cubic (5.2.3.10) is at

A= -b + Jh 2 - 3ag'(O).
(5.2.3.12)
3a
 Lesson V - Nonlinear Equations and Systems !J3

One should enfmce /that A lie be~tween Am 11.:1: = 0.5,\ 1 and Am·i.n = 0.1,\ 1 . The corn~­
sponcling code iu FORTRAN is given in [5], pp. 378-381. It it suggested to reader to
write the c:orresponeling c:oele in Mathematica.

Multidimensional Secant Methods: Broyden's Method

Newton's nwthod as use~cl prcvionsly is ratlwr efficient, but it still has several di:-;-
advantages. One of the most important is that it needs .Jacobian matrix. In many
proble~ms the .Jacobian matrix is not available, i.e. there do not exist analytic ckriva-
tives. If the function evalnation is complicatexl, the finite-difference determination of
.Jacobian can bt~ prohibitive~. Tlwn~ an~ quasi-Newton methods which provide-~ clwap
approximation to the~ .Taco hi an for the purpose of zero finding. The methods are oftt~n
called secant methods, be~canse tlwy reeluce in one dimension to the secant method.
One of the best of those nH~thods is Broyclen's met.ho(l (see [21]).
If one denote-~s approximate .Jacobian by B, then the ·i-th quasi-Newton ste-~p r):P.; 1s
the solution of -

(5.2.3.13) B.;,· r5:r.;, = -F.;,,

where c5x.;, = :?.;,+ 1 - :r.;,. Qnasi-Newton or secant condition is that BH 1 satisfy

(5.2.3.14)

where r)F.;, = F.;,+ 1 - F.;,. Thi::; is generalization of the one-dimensional secant approxi-
mation to the derivative\ bF/rh. Howe~ver, e-~qnation (5.2.3.14) doe::; not determine-~ B.;,+ 1
uniquely in mon-~ than mw dime~n::;iou. Many different. auxiliary conditions to determine~
B.;,+ 1 have be~en examim~d, hut the be::;t one results from the Broyclen's formula. This
formula is based on idea of getting B.;+ 1 by making a least change to B.;, in accordn.JH:e
to the secant equation (5.2.3.14). Broyden gave the formula

(5.2.3.15)

One can check that BH 1 satis:fie::; (5.2.3.14).

Early implementation::; of Brnyclcn's methoel used the Sherman-Morrison formula.
to invert equation analytically,
1
B_ 1 _ B(-1) (!5:f.;,- B.i 1 · r5F.i) 0 r5.i.,; · B.i
(5.2.3.16) +
H1 - ·i.
(]\,--:_
.r.., . u-'B-:-1
'/, . u-'F.-
f,

Thus, insteael of solving eqnaticnt (5.2.3.1) hy, for example, LU decomposition, orw
determined

(5.2.3.17) · ()_:r:.,:
___, = - B-
·i.
1 l"
• < ·i

by matrix multiplication iu O(n 2 ) operations. Th1~ disadvantage of this method i::; t:lmt
it cannot b<~ easily emhexlde~el in a globally convergent strategy, for which the graclwnt
of equation (5.2.3.2) requires B, not B- 1

(5.2.3.11::1)

Accordingly, one should implement. the upcla.t.e formula iu the form (5.2.3.15). However,
we canstill preserve the O(n 2 ) solution of (5.2.3.1) by using QR decomposition of B.i+l
in O(n. 2 ) operations. Allm~ech~d is initial approximation B 0 to start process. It is ofteil
accepted to take identity matrix, awl tlwn allow O(n.) updates to produce a n~asonable
 94 Numerical M<c~thocls in Computational Engineering

approximation to the .T acobian. In [5], p. 382-:383, the first n function evaluations

/are spent on a finite-difference approximation in order to initialize B. Since B is not
/exact Jacobian, it is not guarantec(l that &i! is descent direction for f = ~F · F (sf"~e eq.
(5.2.3.3)). That, has a consequence that the line search algorithm can fail to retmn the
suitable step if B is far from the t.me .T acobian. In this case we simply n~initializi~ B.
Like the secant method in one dimension, Broyden's method converges snperlinearly
once you get close enough to the root. Embedded in a global strategy, it is almost as
robust as Newton's method, and oftm1 needs far fewer function evaluations to determine
a zero. Note that the final value of B is not always close to the true .Jacobian at the
root, in spite of fc1.ct that method converg·es. .
The programme code ([5], pp. 383-385) of Broyclen's method differs from Newto-
nian methods in using QR decomposition instead of L U, and detennination of Jacobia.n
by finite-difference approximation instead of direct evaluation.

More Advanced Implementations

Ow~ of the principal ways that the methods d(~scrihed above can fail is if matrix W
(Newton-Kantorowic:h) or B (Broydcn's method) becomes singular or nearly singular,
so that 6:1: cannot he determined. This situation will not occur very often in practice.
Methods develope(l so far to d(~al with this probl(~lll involve tlw monitoring of condition
number of W aiHl perturbing W if singularity m near singularity is detected. This
feature is most easily implemented if QR decomposition instead of L U decomposition
in N cwton (or qnasi-N cwtou) uwtlwd is applied. However, in spite of fact that this
method can solv<-~ problems wh(~n W is exactly singular and Newton's aucl Newton-like
methods fail, it is occasionally less rolmst on other problems where L U decomposition
succ<~<~ds. Implementation details, like roundoff, underflow, etc. are to be considered
and taken in account.
In [5], considering dfcctiverwcos of strategies for minimization a.nd zero finding, the
global strategies have been based on l·ine seaTches. Other global algorithms, like hook
step ancl dogleg step methods, arc based on the model-trust region approach,
which is related to the L(~V<~lll)(~rg-Marqnardt ;:1lgorithm for nonlinear least-squares. In
spite being mon~ complicated than lim~ seardws, these methods have a repntation for
robustness even when starting far from rlesired ;;ero or minimum.
Numerous libraries awl software pac:kag<-~s arc available f(n· solving nonlinear equa-
tions. Ma.ny workstations awl mainframe compnters have snch libraries a.ttach<~d to
operating systems. 1\!Iany cmmucrcial software 1mckages contain nonlinear equ<ttion
solvers. Very popnlar among cnp;iw~(~rs an~ Matlab and Matcad. Mon~ ,-;ophisticatt~d
package:-> like Mathematica, IMSL, Macsyma, <tlHl Maple contain programs for nouliu-
ea.r eqnation solving. The hook Num,erical recipes [5] contains umlH·~rons programs
fOl' solving nonlinear (~quatiou.
For symbolic impknwul.ati(llJ of nonlinear optimization, see [G], containing uot only
very us<~fnl co(lP lmt also concspouding thcon~tical backgronlHl.

Bibliography (Cited references and further reading)

[1] Milovanovit:, G.V .. Numerical Analysis I. Nau{:na kn.iiga, B<~ograd, 1988 (S(~r­
hiau).
[2] Hoffman, J.D., N·umer'ical Methods for Engineers and Scientists. Taylm
& Francis, Boca Raton- Low lou- N cw Ymk-Sillg<tpore, 2001.
[3] lVIilovanovi<':. G.V. aud D_jonl.icvir':, Dj.IL Programiran.Je numer'i{:kih metoda
no. FORTRAN _jez,iku. Institnt Z<t dolomwutaci.in zast.it(~ na radn "'Edvanl
Karrldf, Nis . 1!!81 (Serbian).
[Ll] StmT, .J., awl I3ulirsclJ. R .. Introd·uction to Numerical Analys1:s, Spriup;cr.
N(~W Ymk. 1~)80.
[G] Press. W.H .. FlamHTy. I3.P .. Tcnkolsky, S.A., and Vet.terli11p;. vV.T .. Nv:rne:rical
Recepies - The Art of Sc·ienti.fic Computing. C<ullbrid?,"c Univnsity Press.
1~)8~).
 Lesson V - N ouliuear Equations aucl Systems

[o] Milovauovi<:, G.V., Stauimirovi<':, P.S., Symbolic Implementation of Noniin-

ear Optim'izat'ion. Uuiv<~rsity of Nis, Faculty of Ele.ctrouic Engine<~ring, Nis, 2002
(Serbian).
[7] D.iordj<~vic:, L.N., An iterative solution of algebraic equations with a pa-
rameter to accelerate convergence. Univ. Beograd. Publ. El<~ktrotdm. Fak.
Ser. Mat.Fi:;,. No. 412- No. 4G0(1973), 179-182.
[8] Tihonov, O.N., 0 bystrorn vychysln'iz najbolshih korne{ mnogoelena. Zap.
L<-~ningrad. gom. in-b. 48, 3(1 9G8). 36-41.
[9] Ostrowski, A., Sol,ution of Equations and System of Equations. Aca<l<~mic
Press, N<~w Ynrk, 19GG.
[10] Wegsteiu, .J.H., Accelerating convergence of iterative processes. Comm.
ACM 1 (1958), 9-13.
[11] Ralston, A., A Pirst Co'urse 'tn Numerical Analysis. McGraw-HilL N<~w
York, 1965.
[12] MilovanoviC:, G.V. & Pdkovi{:, M.S., On some modifications of a third order
method for solv'ing equat'ions. Univ. Beograd. PubL Elektrotd1. Fak. Ser. Ma.t.
Fiz. No. 678- No. 71G (1980), pp. 63-G7.
[13] MilovanoviC:, Kovac:cvi(:, M.A., Two 'iterative processes of third order with-
out derivatives. IV Zmmstvcni slmp PPPR.. Stubic:ke Toplice, 1982, Proc<~ediugs.
pp. G3-67 (Serhiau).
[14] Var.iuhin, V.A. awl Kasjan.ink, S.A., Ob deracionnyh metodah utocnen~ja
kornez uravnen:i{ Z. Vychysl. Mat. i Mat. Fiz. 9(1969), 684-G87.
[15] Lika, D.K., Ob 'iteracionnyh metodah V'issego pmjadka. Dokl. 2-I Nan<:.-
tclm. respuhL konf. Moldavii. Kishiuev, 1965. pp.13-16.
[16] Djonljevif:, L.N. & Milovauovi<':, G.V., A combined iterative formula for solv-
ing equations. Infonnatika 78, Bled 1978, :::(207).
[17] Pctkovif:, M.S., Some 'iterative interval methods for solving eq'uations.
Ph.D. thesis, Uuiw~rsity Nis, 1980.
[18] Pdkovi{:, M.S. & P<~tkovi<':, D. Lj., On a method for two-sided approach'ing
for solving equat'ions. Frcilmrg<~r Iut<~rvall-Bericht<-~ 10(1980), pp. 1-10.
[19] IMSL Math/Libm,ry Users Manual, IMSL Inc., 2500 City West Boulevard,
Honstou TX 77042
[20] NAG Fortran Library. Ntmwrical Algorithms Group, 256 Banbury RoacL Ox-
ford OX27DE, U.K., Clmpt.<T F02.
[21] Broyden . C. G., Qv,asi- Newton methods and their application to funct'ion
m'in'im'izat'ion, Math. Cmnp. 2!)(1%7), 368-381.
[22] Kautorowich, L.V., Funkcional'nyi anal'iz i pr'ikladnaja matemat,ika., Us-
pehi Mat. Nank 3(1948), 8!J-J85.
[23] Ortega, .T.M. & Rlwinboldt., vV.C., Iterative solution of nonlinear eq'Uat'ions
in several var'iable8, Aca<lmnic Press, New York, 1970.
[24] Rall, L., Computat,ional .solution of nonlinear operator. equations, New
York. 196!). ·
[25] K1tl;~ickii, O ..Jn. & Simeh~vi<::. L.I., 0 nahoiden'i{ nacal'nogo pribiien'ija .. Z.
Vy<:isl. Mat. i Mat. Fiz. 14(1!l74), pp. 1016-1018.
[26] Dennis, .J.E. awl Sdmahd, R.B., Numer-ical Methods for Uncon8tra'ined
Optimizat'ion and Nonlinear Eq'uat,ions, Englewood Cliffs, N.J: Prentice Hall,
1983.
Faculty of Civil Enp;incerinp; Faculty of Civil Engine<~ring and Architec:tnn~
Belgrade ·c · Nis
MaHter Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON VI

6. Approximation and Interpolation

6.1. Introduction
This lesson is devot<~d to one of the most important areas of theory of approxima-
tion - interpolation of func:tions. In addition to theoretical importance in constrnction
of nunH~ricalmetlw<ls for solving a lot of problems like numerical differentiation, nmtH~r­
ical integration and simila.r, it has pmc:tical application to many engineering problems,
including FEM problems.
Th<-~ory of approximation deals with replacing of function .f ddiw~d on some S<~t X
by another function iD. Let function (]) depend on n + 1 parameter o.o, a1; ... , !J.n, i.<~.

Problem of approximation of function f by function (]) reduces to determination of pa-

rameters o,.i, ·i = 1, ... , n, according to some criterion. Depending on chosen criterion, we
differ several HOlts of approximations. Generally, depending on form of approximation
function, they can be divided to linear and nonlinear approximation functions. The
general form of linear approximation function is

(6.1.1)

whereby system of functions {(])A:} fulfills some given conditions. Linearity of function
<1? means linearity regarding pararneters a.i ('i = 0, 1, ... , n). When <I>A: = :r:k: (k =
0, 1, ... , n), i.e.

we have approximation by algebraic polynomials. In the cast~ when { <I>,j = { 1, cos :1:,
sin :r:, cos 2:r:, siu2:~:, ... } we h;-we approximation by trigonometric polynomials, i.<~.
trigonometric appl'oximation. For the case

( ·'1"• - '1" )m, ( :r: 2: :r: k:) '

• 1n. ··A:
<I>J.:(:r:) = (:r:- :~:,.:)+ = {
0 (:r: < :c:,.,),

where rn is fixed natural1mmber, we have spline approximation.

We will mention two of nonlin<~ar approximations:
1. Exponential approximation

'f (
<±> I >(:1:;
:1: ) = < c0 , fJn, ... , c.,., fJ,,. ) = co eb0 :r: + . . . + c.,...eb,:r; ,
where n +1= 2(r + 1), i.e. n = 2T+ 1.
97
98 Numerical Methods in Computational Engineering

2. Rational approximation

where n = T + s + 1.
Let function f be given on segment [a, b] by set of pairs (x~;;, fk:) (k = 0, 1, ... , n),
where f~;; .f(:r:k:)· If for approximation of function f by function <I? the criterion for
choice of parameters a 0 , a 1, ... , an is given by system of equations

(6.1.2) <1?(.-rk:; ao, a1, ... , an) = .h .(k = 0, 1, ... , n),

we have problem of function interpolation. Function <!? is called in this case interpolation
function and points .T~;; (k = 0, 1, ... , n) interpolation nodes.
Problem of interpolation could be !ilore complicated then noted. More general case
appears when, in addition to function values in intei~polation nodes, the derivatives of
function are also included.

6.2. Chebyshev systems

Let function f be given by its values .h: = f(.-r~;;) in points x:~;; (xk: E [a, b])(k =
0, 1, ... , n). If we have linear interpolation problem, i.e. interpolation by function
(6.1.1), system of equation (6.1. 2) reduces to system of linear equations in parameters
a.;, ( i = 0, 1, ... , n) ,

l[ l
1.e.

<Po (:ro) <1?1 (:r:o)

<Po (:r:1) <P1(x1) <Pn(:r:o)
<Pn (:r1) ao]
a1 . h
[ .fo
(6.2.1)
[
<J? 0 (::r: n) il>,(,cn) . a.n - f,. .

In order above given interpolation problem to have unique solution it is necessary that
matrix of system (6. 2.1) be regular.
To the system of functions (<!?A:) should be intruded such conditions under which
there not exists linear combination

which has n + 1 dif-ferent zeros on [a, b]. System of functions with such characteristic
are called Chebyshev (Tchebyshev) systems, or T-systems. There exists extraordinary
monograph regarding T -systems [6].
Theorem 6.2.1. If the fimctio11s <I? A: : [a, b] ---+ R (k = 0, 1, ... , n) are n + 1 tinws
differentiable and if fbr every k = 0, 1, ... , n the vVronsky determinant VVA: is cliff(~rent
from zero, i.e.
<Po (:r:) <!? 1(:r:) <!? k ( :J:)
<P;ll:) <!?~ (:r) <!?~; (:r:)
vv,,, = # 0,
<!? ~/'') (:r:) <!? i
k:) ( :r) ;r)(k)(··)
'± k: ,[,

system of fwu:tions {<I> A:} is Clwhyshev (T) system.

 L<-~ssml VI - Approximation and Interpolation 99

6.3. Lagrange~interpolation

Let function f be given by its values .h = f(:I:k,) in points Xk: (:.rk: E [a, h]) (k
0, 1, ... , n). Without decn~asing the generality, assume

(6.3.1)

If we take points :1:h, for int.<~rpolation knots and put <D~.,(:r) = :r: 1·' (k = 0, 1, ... , n) we
have a probl<~m of int<~rpolation of function f by algebraic polynomial. Denote this
polynomial with P~~., i.e.

Then we have the int<~rpolating polynomial

n
(6.3.2) P,.(:r:) = L .f (:r:~;; )Lk(:r:)'
k:=O

where

(:r:- :r:h:)w'(:J:h:)'
w(:r:) = (:r:- :ro)(:r- :r:1) ... (:r- :r:n),
w' (:rh:) = (:r: A: - :ro) ... (:D '·' - :r: h:-1) (:r: k: - :r: k+l) ... (:r: h: - :1:.,.).
The formula (6.3.2) is called Lagrange inkrpolation formula, and polynomial P~~. La-
grange interpolation polynomial. ~Then programming, it is suggested to use the follow-
ing form of Lagrang<-~ fornmla.

;r: - .7:.;
P,,(:r:) = n _( .f(:r:~;,) IJ,.
L_ n _J,,,'~- )
. .L 1.:
k:=O .,.=n
-;,p,,

Having in mind that determinant of system (6.2.1) is Vanclennonde determinant, i.e.

1 :r:o :r:()'
1 :r:l :r'1
=II (:r.;. -
:r.i)'
·i.>j
1 :r:n :~:;~

and the assumption (6.3.1), it follows that the Lagrange polynomial ( 6.3.2) is unique.
Example 6.3.1. For hmc:tion wr.lues given in tabular form find tlle Lagrange interpo-
lation polynomiaL
:r: k: .f (:r:k)
-1 -1.
0 2.
2 10.
3 35.
100 Numerical Methods in Computational Engineering

P (:J:) = (_ 1) . (:r:- 0)(:~:- 2)(:r:- 3) (:r + 1)(:r- 2)(x- 3)

2
( -1 -- 0) (-1 - 2) ( ~1 - 3) +
3
(0 + 1) (0 - 2) (0 - 3)
+ (:r: + 1)(:r:- O)(:D- 3) . (:1: + 1)(x- O)(:r ~2)
10 35
(2 +1)(2- 0)(2- 3) + (3 + 1)(3- 0)(3- 2)'.
Le.
5 4
P3 (0''r·) = -'1':3
3''
- :._,1.2
3''
+ ')~.
Example 6.3.2. Determine <tppmximately zem of function given in example 6.3.1.
Lagrange interpolation polynomial for function y ---+ .f- 1 (u) is

p (·t) = (-1) (y- 2)(:1J- 10)(:u- 35) C) (y + 1)(y- 10)(y- 35)

3
.'/ (-1- 2)(-1-10)(-1- 35) + (2 + 1)(2 -10)(2- 35)
+ ') (y + 1)(y- 2)(y- 35) '3 (:tj + 1)(y- 2)(y- 10)
~ (10 + 1)(10- 2)(10- 35) +' (35 + 1)(35- 2)(35- 10)'
w hercfrom, for y = 0 we get :~.c~ro of a function f
;r; rv P3 (()) = - (). ()5()1)

6.4. Newton interpolation with divided differences

For function f given by its values .h: _ f(:r:,J in points :c,,, (k = 0, 1, ... , n), define
first divided cliff(~r<~nces. Tlw ratio

is callc~d clividc~cl llif-fercuu~ of first. order (of hmc:tion fin points :r:o and :1:1) and denotc~cl
as [:r:o, :r:1; .f].
Dividecl clitfcrenc:e of order T are clefim~cl recursively by

.. ,. ,. . ·]- [:r:l, ... ,:r:.,.;.f]-[:ro, ... ,:r.,._l;.f]

(G.c!.l) [.l.o,.r.l, ... ,.r..,. . .f- ,. ,. '
.r •.,. - .J,o

when~ [:r:; .f] = f(:r:).

H.dation ( (j .4 .1) euablc~s c:onstrnc:tion of table of divided differences

k :t:/,; fl.: 6.f'. /,: 6_2_h: 6. 3J,,,

() :r:o .fo
[:t:o. :r1: .f]
1 :r:l fl [T '/'·1·.'/'·2'. .I]
• ·())
0

[:r: 1,:1:2: .f] [:r:o. :r:l, :r2, :r:3: .f]

2 :r:2 h [T"'1•.'/'·2;.T·3,.. f]
[:r:2, :r:3: .f]
:~ :r: :3 f:J

Om~ call show that dividc~cl cliffcn~ncc of mclc~r .,. has characteristic: of li11earity. i.e.
 L<~sson VI - Approximatiou and Interpolatiou lOl

where c1 and c:2 me~arhitrary constants. Bt~canse of, based on (G.4.1),

one can prove by matlwmatical iuclnction

where w(:r:) = (:r,- :i: 0 )(:E:- :r: 1 ) ... (:r- :r:.,.).

Let f E C'"'[a,b] awl t.lw condition (6.3.1) holds. Then, for every r:::; n, the fornmla

=II· . I
1 tl t.,._l

[:ro, :r:1, ... , :r:.,.; .f] f('r)(:r:o + t(:r:- :r:.,_I)t.;,)rlt1 dt2 ... (it;.,.
() 0 0 '1,=1

holds. This can be proved by mathematic<ll induction.

Applying tlwon~m on integral mean value, fi:·om last expression follows

II· . I
1 tl t-.. -1

[:co, :r:1, ... , :r:.,.; f] = f('r)(O rlt1 dt2 ... (lt.,.

() 0 ()

= !_f('r)(() (a< (<b).

'{' l

Taking :r.;, ----t :.r:o ('i = 1, ... , ·r) in last equality, we get

·r· '
·f]----'" 1 '"
(6.4.2) '!' 'I'
[ ·•0,··1.,···,•"/'). ---, -,. f(r)(·r·)
··0.
'f'.

Let ns express now value of function f (:r:.,.) (7' :::; n) by means of divided differences
[:r:o, ... , :r:.;,; f] (·i = 0, 1, ... , T).
ForT= 1, based on definition (6.4.1), we h<we

In similar way, for .,. = 2,

· f(:r2) = f(:c1) + (:1:2- :r:1)[:r:1, :~:2; .f]

= (f(:ro) + (:r:1- :t:!i)[:ro, :r1; f]) + (:r:2- :r:1)([:ro, :1:1; f]
+ (:~:2- :r:o)[:r:o, :r:1, :r:2; f]),
I.e.

In general case, it holds

f(.T.,.) = f(:r:o) + (:r:j.- :I:o)[:ro, :r:1; f] + (:r:-r- :ro)(:x:j.- x1)[:r:o, :x:1, x2; f]
+ ... + (:r:.,. - :r 0) (x :.r:1) ... (x.,. - :x:.,._I)[:Eo, x1, ... , :r:-r; .f].
1 . ,......
102 N llllleric:al Metlw<ls in Computational Engineering

Using divided differences for sc-~t of cla.ta (:r:h:, f(:r:A:)) (A: = 0, ... , n) the interpolation
polynomial of the following form can be con:-:tructed.

Pn(:r:) = .f(:ro) + (:r:- :ro)[:r:o, :r:1; f] + (:r:- :r:o)(:r:- :r:l)[:r:o, :r:1, :r:2; .f]
+ ... + (:r- :r:o)(:r:- :r:I) ... (:r- :r:.n-l)[:r:n, :r1, · · ·, :I;n: f].
ThiR polynomial iR calle<l Newton·:-: iut<~rpolation polynomial.
Having in mind 1miqn<~1w:-:s of algebraic interpolation polynomial, we conch1<lr-~ that
Newton's interpolation polynomial is equivalent to Lagrange polynomial.. Note that
construction of N ewtou's int<~rpolation polynomial clemandR previous forming of ta.bl<~
of divided difh~n~uces, what was not the case with Lagrange polynomial. On We other
lmud, involving a new interpolation node in order to .reduce interpolation error, is mme
convenient with Newton':-: polynomial, because do not demand repeating of whole cal-
culation. Namely, at Newton's interpolation we have

Pn+1 (:r:) = P, (:~:) + (:r:- :ro) (:r:- :rl) ... (:r:- :r.,)[:r:o, :r:1; ... , :rn+1: .f].

If w<~ pn t :r'i -t :rn iu N<~v,.rtou 's intm-polation polynomial P,, based on (6.4. 2) it
n~duc:es to Taylor polyuumial.
Example 6.4.1. Based 011 tahle of values of fimction :1: -t ch:r: f(mn ta.IJle of divided
differences and write Netytdn 's int:mpolatio11 J>oly11omial.

k 0 1 2 3
:r:,,, 0.0 0.2 0.5 1.0
f(:r:k) 1. 0000000 1.0200668 1.1276260 l.G430806

k [:r~;:, :rl.:+l: f] . ·/,:;.T·1,:+1:.T·/;:+2:.

[T . f] [:r,,,, :r:~;,+1, :r/,:+2, :1:,,,+3: f]
0
() .1 00333~
1 O.G163938
0. ~3G8G:J07 0.07407!)5
2 O.G9047:33
0. 830!)09:3
3

N ewtou 's iut<~rpolatiou polyuomial is tlwu

P.1 (:r) = 1. + 0.1()(););):31-);r + 0.51G:39:31-l:r:(:J:- 0.2) + 0.0740795:J:(:r- 0.2)(:r- O.G).

For <~xamplc. for :t: = o.:J, r:h 0.:3 ~ P:3 (0.3) = 1.04[)1474.

6.5. Newton's interpolation formulas

By uwaus of finit<~ diffcn~w:c calcnlns t.lw s<~wral intcrpolatiou t<wumlas iu cqnidis-
tcmt. uodr~s cau b<~ <~v;dnat<~d. Tlw olclcst mw is Nr~wtou's int.<~rpolat.iou fmmnla.
Ld hm<:t.ion f h<~ given 011 [u., h] by pairs of vahws :1:1.:: f,,, when~ f,,, = f(:c!.:) awl
:t: 1,, = :r 0 + l.:h (1.: = (L 1. .... 'It). Fm giv<~u set of data the tabl<-~ of finite diffcrcw:<!s ca11
h<~ fomwd. In th<~ l'ollowiug table is nsccl operator 6 cldiued hy 6f(:1:) = f(:t: +h) -
 L<~ssou VI - Approximation an(l Interpolation

.f(:1:) (h =canst. > 0).

:to fo
~~·
.0
:J:1 .f1 f
~2 .0
~f
.1 ~3.fo
:t:2 h ~2 .f1 ~4f
.0
~f
.2 ~3f
.1
:~:.'3 h ~2 .f2
~f
.3
:~:4 .f4

:t: - :r:o
Let :1: = :1:o +ph (0 ~ p ~ n), i.<-~. p = . Because of
1/,

EP = (1 + ~)1) = t (f) ~h:,

A:=O
where E is shift OJH~rator (Ef(:t:) = f(:r +h)), we have

EPf'
. 0 ~ (p)~h:f·
=D k ~ (p)~A:f·
.0 = D k . 0 +R·n (f'···)
. ':L '
h:=O h:=O
l.e.

(6.5.1) f(:ro+ph) = t (~)~kfo+R.,.{f;::c:).

/.:=0
The remainder term Rn, having in mind tlw uniqueness of interpolation polynomiaL is
equal to remaind<~r of Lagrange interpolation formula
lt,'n.+1
R.,.(.f';:c) = ( )'p(p- 1) ... (p- n)f(n+1)(e),
n+ 1.
where ~ is point in interval (:r 0 , :r:.,).
The polynomial

(6.5.2) Pn.(:1:) = t (~) ~AJo

h:=O
(ph= :r:- :ro)

obtained in a given way, is call<~cl first Newton':-:; interpolation polynomial. This polyno-
mial can be defined recnrsively as.

H:(:r) = P/,:-1(:r:) + (p) - -

k ~AJo (k = 1, ... , n),

starting_ with P 0 (:r:) = .f0 . The <leveloped form of polynomial is

. p (· ·) - ,. ' A ,.
n .1. - . o + z>u. n +
p(p - 1) A 2 f'
') 1 u , o + ... + p(JJ - 1) •..,n..(p
- 'fl, + 1) uAn,.. o,
~. 1
Le
~f . ~ f 2

Pn.(:D). .fo + - ·0 (:r- :ro) + ~,·;) (:r:- :r:o)(:c- :L·1) + ...

h 2. L
tl'/1. f'0
+ -,-·.-
n.hn.
(:r- :ro)(:r- :r:1) ... (:r- :r:n-1)·
First Newton's interpolation polynomial is used for interpolation on begin of llll-
terval, i.e. in neighborhood of interval starting point x 0 . Interpolation of function for
:r: < :r: 0 is called extrapolation.
104 Numerical Methods in Computational Engineering

Remark. The reader is encouraged to write a code in Mathematica fm progran1

realization of Erst Newton's interpolation formula, using all four previously given fc)rm.s.
. . For ~igger tables of ~nite differences, usually by applying finite element method,
It Is· very Important knowmg propagation of accidental error offunction value in: some
interpolation nodes. Let us have the value .h: + E in place of .h:, where E is error. Then
we have the following table obtained using operator .6. .
:r-t .f,;, .6-.f,;, .6,2 • r
1.
r
.6,3 '1, r
.6. 4'1,
:E/;:-4 .h:-4
.6-.h:-4
:r:k:-3 .h:-3 .6. 2.fr.:-4
.6-f''
'/,,-3 .6. 3.h:-4
.Tk:-2 .lr.:-2 .6. 2 fk-3 .6. 4f'A:-4 +E
.6-.h:-2 .6, 3.f'k:-3 +E
2 .6. 4h:-3 - 4s
:rli:-1 .lr.:-1 .6. .f'K:-2 +E
.6-fk-1 +E .6. 3 .h-2 - 3s
.fA:+ E 2 4
:r.k .6. h-1 - 2E .6. h:-2 + 6c
.6-,h- E .6. 3.h-1 + 3s
:rr.:+1 .6.2.h +c 4
h:+1 .6. h-1 - 4E

.6. .f1.:+1. .6.3.h -E

:r:r.:+2 h+2 .6. 2.fr.: + 1 .6.4h:+s

3
.6.h+2 .6. .h+1
:I:k:+3 2
h:+3 .6. .f'A:+2
6.h:+3
:r~;:+4 h+4
The erroneous differences in table are underlined. We see the progressive propagation
. l'ff'
of error, sot l1at error In< I erencc u'm.
A f
·li:-w.+i (·i = 0, 1, ... , ·m ) is (n,.,·L)
" (-1)'i,c.
~

6.6. Spline functions and interpolation by splines

Physical device named spline consists of a long strip fixed in position at a number
of points that relaxes to form a smooth curve passing through those points. Before
computers were used for creating engineering designs, drafting tools were employed by
designers drawing by hand. To draw curves, especially for shipbuilding, draftsmen often
used long, thin, fiexiblt~ strips of wood, plastic, or metal called a spline (or a lath, not to
be confused with lathe). The spline:.,; were held in place with h~ad weights (called ducks
because of their <luck like shape). The elasticity of the spline material comhinecl with
the constraint of the control points, or knots, would cause the strip to take the shape
which minimizes the cm~rgy r~c~quired for bending it between tlw fixed points, and tlms
adopt the smoothest possible shape. One can recreate a clraftsm<:J,n's spline device with
weights and a l<~ngth of thin stiff. plastic or rubber tubing. The weights are attach<~d
to the tube (by gluing or pinning). The tubing is then placed over drawing paper.
Crosses are mark<~d on the papPr to designate the knots or control points. The tube is
then adjusted so that it passes over the control points. Supposing uniform elasticity of
spline, one can say that its potential energy, when bent, is proportional to th(~ integral
along it (curvilinear int<~gral along curve) of quadrate of convolution K. Thus, if spline
lies along plane curve y = S (:I:), o. ::; :I: ::; IJ, its potential energy is proportional to the
int<~gral

(0.6.1)
.I 2
. K(:r:) ds =.
;·
b

(1
S"(:r)2
+ S'(:r) 2)G/ 2 d:r
L n
 Lesson VI - Approximation and Interpolation lOS

and stabilize<l slmp<{ it tak<~s is such that minimiz<~s (6.6.1) under given limitations.
In the similar way is ddined mathematical spline, by discarding S' (x) 2 in nominator
of (6.6.1), what is dose to previous case, wlwn S'(:r:) << 1. Thus, now is to minimiz<~
the int<-~gral

!J

(6.6.2) ./ S"(:r:) 2 rh:.

Mathematical splin<~ can be uwre generally defined by using higher derivative~ tlwu two
in (6. 6. 2) .
First results regarding splin<~ fuuctions appeared in papers of Quade and Collat.z
( [11], 1938) an< l Coura.nt ( [12]. In 1946 mathematicians started studying the spline
shape, and derived the pi<~cewisc polynomial formula. known as the spline cmve or
function (Schoenberg [13]). This has led to the widespread us<-~ of such functious in
computer-aided design, csr><-~cially iu the surface desigus of v<~hicles. Scho<-~nherg gave
the spline function its name after its resemblance to the mechanical spline used by
draftsmen. The origins of the spline in wood-working may show in the conjectured
<~tymology which connects the wonl spline to th<~ word splint<~r. Later craftsmen have
made splines out of rubber, ste<~l, and other dastomeric materials. Spline devices help
bend the wood for pianos, violins, violas, etc. The Wright brothers used one to shape
the wings of their aircraft.
The extensive development of spline functions and usage of their approximation
properties begun in sixtic~s last C(~ntury. Th<~ splines are greatly applied to 1mmc-~rical
mathematics, in particular to int<~rpolation, numerical differentiation, numerical int<-~­
gration, differential equations, etc:. The extremal and approximative attributes of so
known natural cubic spliw~ arc given in [1], pp. 81-86.
Let on segment [a, b] given IH~twork of nodes

(6.6.3)

Denote with P.,~~, set of algebraic polynomials. of order not greater than m.
Definition 6.6.1. Function

is called polynomial spline of degn~e m, and deif~ct k (1:::; k:::; m.) with nodes (6.6.3), if
satisfies tile conditions
1° Sm. E Pm on every snbsegment [:r:.;,-r, :r:.;,] ('i = 1, ... , ·n),
2° Sm. E c·m-li:[a, b].
Points .'D.;, are called nodes of spline. --
We will further consiller polynomial splines of defect 1 and for Sm(:r:) = Sm,l(:~:) say
to be a spline of degree ·m. Very impmtant kind of splines, interpolation cubic .splill<~,
with rn = 3 are most frequently used and applied in engineering design. Therefor<~ we
join to the network nodes 6.n rcalmnnbers .fo, .fr, ... , f.n·
Definition 6.6.2. Function S 3 (:r) = S 3 (:1:; f) is called interpolation cubic spline f(n·
fimction f on tile netvvork 6.n (n 2: 2) if the following conditions are fulfilled:
1° S 3 (:r, f) E P 3 if :r: E [:L;.-1, :r.i] ('i = 1, ... , n),
2° S3 (:r:; f) E C 2 [a, bJ,
3° · S 3 (;c.;,; f)= f.;,= .f(:r:.;,) ('t = 0, ... , n,).
We see that condition 3° does not appear in Definition 6.6.1. The spline defined
in this way is called simple cubic spli1ie. It interpolates function .f in network nodes
(condition 3°), it is continuous on [a, b] together with its derivatives SH1:) and S~(1:)
(condition 2°) and defined on every .subsegment between neighbor nodes with polynomial
106 Numerical Methods in Computational Engineering

of degTee not greater than 3. So, the third derivative of cubic spline· is discontinuous,
being, part by part of constant value.
Cubic spline has two free parameters, determined usually by some additional bound-
ary conditions. The typical ones are: ·

(6.6.4) S~(a) = S~(b), S~(a) = S~(b);

(6.6.5)

(6.6.6)

(6.6.7) S~(:r:1- 0) = S~(:r:1 + 0), S~(.Tn-1- 0) = S~(xn-1 + 0),

where an, bn, An, B, are given real numbers.
Conditions (6.6.4) define so known periodic spline. These conditions are used when,
for example, interpolating function f is periodic with period b - a.
If function .f is differentiable and we know values of derivatives in boundary points
a and b, then the additional conditions (6.6.5), an= f'(a) and bn = J'(b), or (6.6.6),
An = .f" (a) and En = .f" (b), are to be used, what is often case at mechanical models.
The corresponding spline is called natural cubic spline.
The conditions (6.6. 7) are most complicated, but they obtain continuity of third
derivatives of spline in points :r: = .1:1 and x = Xn_ 1 .
The most interesting spline approximation is cubic spline interpolation. The algo-
rithm for construction of cubic spline is given in ([1], pp. 73-81). To interested reader
is suggested to write a code f(n· construction of t>pline (taking· care of Hermite inter-
polation) and, if possible, include graphical interface. For some programming details,
see Fortran subroutines Spline and Splint in ([5], pp. 109-110). For obtaining a
higher order of smoothness in two-dimensional interpolation (applicable in many areas
of engineering, and specially in computer graphics), one can use bicubic spline and code
given in ([5], pp. 120-121.)

6. 7. Prony's interpolation
Dating from 1795, Prony's interpolation ([14]) is often known as Prony's exponen-
tial approximation, and until nowadays not applied as it by its sophisticated nature
deserves. It is suggested to students and readers to apply the following formulas in de-
veloping algorithms for pro!2;rannning of Prony's method. Some benchmarking research
for comparison of applic(),tion of this method, cubic spline, and, for exmnple, least square
method to some physical problems is also strongly encouraged.
If we have interpolation function of form

' f·(· ·)
,[,
~
-
(' 1 cPt;r; + ('-2
> >
(,n2tr:
' + ... + ("
"11
ea,nx

where l'·h: = P.nh. If function f is given on set of equidistant points { (xk, .h) h=D,1, ... ,2n-1,
and :r:k- :r:k_ 1 = h = r:on.'d (k = 1, 2,.,., 2n -1), by replacing x = .1: 0 +kh data S(~t can
be replaced by {(k, .fdh=<LL .. ,2n-1, where :c = 0, 1, ... , 2n-1. By setting interpolation
problem

(6.7.1) <D(k) = f,,, (k = 0, 1, ... , 2n- 1),

 Lesson VI - Approximation and Interpolation 107

we get the system of~equations

+ c2 + ... + Cn = fo
c1

r;1/'·1 + C2/J·2 + ·. · + c.,p.n = h

(6.7.2) Cl/l·i + C2J.l.~ + ... + CnJ.l.~. = f2

.
(.1//,1
N-l . N-1
+ C2fl2 . N-1
+ · · · + CnJ.l•n =.
f N-1·

If p.'s are known (or pn~assigned.) awl N = n, system (6.7.2) is to be solved exactly <ts
system of linear eqnations, and if N > n approximately by h~ast squares method (se<~
next chapter).
If 11.'s are to he det<-~nnined, we need at least 2n. equations, but we have system of
nonlinear equation, which, as we know, in gen<~ral case could he unsolvable. Therefore,
we can assume that t"'s arc the roo·ts of algebraic polynomial of form

Jl. n + (\'.1f/· Tl.-1 + · · · + C\'.n-l/1, + C\'.n = O·.

(6.7.3) I.e

By multiplying all equations in (0.7.21) by CY.n, o~n- 1 , ... , a 1 , 1, we get the system

foC\'.n + .f'Ic\'.n-1 + hcv.n-2 + · · · + .f.n.-lC\'.1 = -fn

finn+ han-1 + hc\'.n-2 + · · · + fn0'.1 = -fn+1
(6.7.4)

If determinant
fo fn-1
h fn

fN-n-1 fN-n fN-2

the solution of system (6.7.4) is unique. If N = 2n we get system of linear equations,
and if N > 2n we solve this system by least squares method.
After the o:'s are determined, the n Jl.'s ar_e found as the roots of (6.7.3). The
equation (6.7.2) then become liw~ar in the n c's. with known coefficients. Thus the
nonlinearity of the system is conce1itrated in the single algebraic equation (6.7.3).

6.8. Packages for interpolation of functions

Many libraries and software packages are available for interpolation and extrapo-
lation of functions. Many workstations and mainhame computers have such libraries
attachc~d to their operating systems.
Many commen:ial software packages contain algorithms and programs for inter-
polation. Some of mon-~ prominent packages are Matlab and Mathcad. Some more
sophisticated packages, such as IMSL, MATHEMATICA, MACSYMA, and MAPLE, also con-
tain routines for polynomial interpolation and approximation of functions. The book
Numerical Recipes [5] contains a rOlitines for interpolation (See chap. 3, Interpo-
lation and Extrapolation), and the book Numer·ical Methods for Engineers and
. Scientists ([2]) program code for difFerence formulas and numerical difFerentiation.
For demands on interpolation in tw<) or more dimensions, higher order for accuracy, ar
higher order of smoothness (bicuhic interpolation, bicubic spline) see code in ([S], p
118-122).

Bibliography (Cited references and further reading)

[1] Milovanovic, G.V., Numerical Analysis II, Naucna knjiga, Beograd, 1988 (Se
bian). .
[2] Hoffman, J.D., Numerical Methods for Engineers and Scientists. Tayl1
& Francis, Boca Raton-London-New York-Singapore, 2001.
[3] Milovanovic, G.V. and Djordjevic, Dj.R.., Programiranje numerickih metoc.
na FORTRAN jeziku. Institut za dokumentaciju zasti.te na radu "Edva1
Kardelj", Nis, 1981 (Serbian). .
[4] Stoer, J., and Bulirsch, R.., Introduction to Numerical Analysis,. Springe
New York, 1980.
[5] Press, W.H., Flannery, B.P.-, Teukolsky, S.A., and Vetterling, W.T., Numeric,
Recepies - The Art of Scientific Computing. Cambridge University Pres
1989.
[6] Karlin, S. and Studden, W ..J., Tchebyshev Systems With Applications 1
Analysis and Statistics. Interscience Publishers, New York, 1966.
[7] Milovanovic, G.V. and Kovac:evic, M.A., Zbirka resenih zadataka iz m
mericke analize. Nauc.na knjiga, Beograd, 1985. (Serbian).
[8] Goertzel, G., An algorithm for the evaluation of finite trigonometr
series. Amer. Math. Monthly 65(1958).
[9] Cooley, J.W., Lewis, P.A.W, Welch, P.D., Historical notes on the fast Fouri1
transform. Proc. IEEE 55(1967), 1675-1677.
[10] Cooley, J.W., Tukey, .J.W., An algorithm for the machine calculation 1
complex Fo·urier series. Math. Comp. 19(1965), 297-301.
[11] Quade,W. and Collatz, L., Zur lnterpolationstheorie der reellen periodi.
chen Funktionen. S. -B. Preus. Akacl. Wiss. Phys.-Math. Kl. 30(1938), 383-42
[12] Courant, R.., Variational methods for the solution of problem of equilil
rium and vibrations. Bull. Amer. Math. Soc. 49(1943), 1-23.
[13] Schoenberg,I..T., Contributions to the problem of approximation of equ
distant data by analytic functions. Quart. Appl. Math. 4(1946), 45-99; 11
141.
[14] Prony, R.. De, Essai experimentale et analytique ... J. Ec. Polytech. Par
1(2) (1795), 24-76.
[15] Ralston,A., A First Course ·in Numerical Analysis. McGraw-Hill, New Yor
1965.
[16] IMSL Math/Library Users Manual, IMSL Inc., 2500 City West Boulevar
Houston TX 77042
[17) NAG Fortran Library, Numerical Algorithms Group, 256 Banbury Road, 0
ford OX27DE, U.K., Chapter F02.
Facnlty of Civil Eng·iq<~<~riug Facnlty of Civil Engi1wering ancl Architc~ctnn~
Bdp;radc Nis
Mast<~r Study Doctmal Stu< ly
COMPUTATIONAL ENG INEER.ING

LECTURES

LESSON VII

7. Best Approximation of Functions

7.1. Introduction
This chapter is clc~vot.c~cl to approximations of functious most applied in <liffc~reut
areas of scic~ucc~s ancl eugineeriu p;.
Let functicm f : [(J,./J] -t R p;ivc~u by set of valnc~ pairs (:r 1. f 1) (.j = 0, 1, ... , n1.)
where
fi = f (:ri). Consider the pro1>lc~m of approximation of function f by linc~a.r approxima-
tion function
I n
<D(:r:) = <D(:1:; rJ.o, ... , an)= L (l.i(h(:r:).
·i.=O

w hc-~re m. > n (for '/11, = n we have int<~rpolation). Procec-~cling like at intcrpolatimL we

gd. so known oven ldinc~cl system of e~quatious
'1/.

(7.1.1) L aicfJ.i (:r:J) = fi (.j = 0, 1, ... , rn),

·i.=O

which in ge~ncral case cloes not have solution, i.e. all equations of system (7.1.1) can not
l>e conteml>Ol'ary :-;atisfied. If we clchm~ c)n l>y
'TI,

(7.1.2) r5n(:1:) = f(:~:)- L o,dJ.;(:r),

·i=O

it is possible to :-;can:h fm ":-;olntion" of :-;y:-;tcm (7.1.1) so that

(7.1.:3) 1}5.~. l·l·r = minllr5,llr·,

(}..;

where ''1/1,

llr5nll·,. = (L l(5n(:1:j)l~') 1
/·r· (T 2: 1).
j=O

The equality (7.1.:3) gives the~ critc~ria. for detcnnination of paranH~ters ao, a1 ..... U.n in
approximation function <D .. The~ quantity llr5.~. 11'1'1 which exists always, is called the value
of he:-;t approximation in f1'. Optimal values of parameters a.;. = ~.;. ('i = 0, 1, ... , n) in
sense of (7.1.3) give tlw 1H~st rr approximation function
'//.

(~*(:r:) = L~·i(;).;(:I:).
·i=O

109
llO Nurnerical Methods in Computational Engineering

Most frequently is taken

m.
t r = 1, ll6nlh = I: l6n(:rj)l (best t 1 approxim~tion),
j=O
rn
2. r = 2, ll6nll2 = (2:: 6n('Ej) 2) 112 (best l 2 approximation),
j=O
3. r = +oo, ll6nlloo = max lc5n(:r:j)l (Tchebyshev min-max approxirm1tion).
O:'S:J:'S:m.
In a similar way can be considered problem of best approximation of function f in
space Lr(a, b). Here we have

(1::; r < +oo)

and

By introducing weight function p : (a, b) ---+ R+ the more general case of mean-square
approximations can be considered, where the corresponding norms for discrete and
continuous case are given as (see [1], pp. 90-91)

(7.1.4)

and

(7.1.5)
a.

respectively.
Example 7.1.1. Function :r: ---+ f(:r:) :x: 1 13 is to approximate with function :r ---+
cp(:r:) = ao + a 1 x in space

Here we have c5 1 (:r) = :r: 1 13 - a0 - a 1 :r; (0::; :1:::; 1). (see {1}, pp. 91-93).
1° We get the best L 1 (0, 1) approximation by minimization of norm
1

llr51lh = ./ i:r 113 - ao- a1xld:r.

0

. -861 = -1, and -861 = -:r, t 11e optnna

Havmg . 1 va1ues o f' parame
. t.ers ao an(1 a1 are
.
r5ao rSa1
to be determined from the system of equations
1

./ sgn r51 (:r:) &r: = 0,

()
(7.1.6)

l
1

:r: sgn r5 1 (:r:) rl:r: = 0.

()
 L<~HHOll VII - B<~Ht: Approxilllatiou of FtinctiouH 111

Having in lllill< 1 that r) 1 chaugc~s Hign on S<~guwnt [0, 1] in points :r 1 awl :r:2 ( S<~<~ hg.
7.1.1). system of equations (7.1.G) n~dnces to system

1 1
'}'
"2-. 1 -
'(' - 2' '1'2
" 2 - .. 1-2'
'['2 -

wlwrefrolll it follows :r1 = ~ awl :r2 = ~-

Thns, dd.<mnining tlw hc~st. U(O, 1) approximation recluc<~s to intc-~rpolation, L<~.
determining of iutcrpolatiou polyuomial <I>* which satisfies t.lw conditions

<J)*(l/c!) = f(l/4) = iff <T>*(J/4) = f(3/4) = ff,

I.e.
2 :1{;) 1 :l(;)
<I>*(:r:) = - . (v3 -l):r + -.,-(3- v3)
\14 2\/4
~ O.GG720:~: + 0.49066.

X . X

Figmc 7.1.1 Fignre 7. 1. 2

I(n.o,al) ~llr5lll~ = ./(:~: 1 1 3 -ao-a1:r) 2d:r.

()

From the couditions

1
()J - -2
' ;· (.1.
'.,1/:3 - ao -- a ' '.z.)d.i.- 0,
-. -- 1' ''-
dao .
()

1
()J
-. - -- -2' ;· .r.(.1.
,, ,.1/3-'- ao - a1.1.)d.r.
,.. ,. -- 0,
dn.1 .
() '

it follows
1 3 1 1 3
ao + 20.1 = 4' -ao + ·~o.1 = -.
2 3 7
112 Numerical Methods in Computational Engineering

1.e. ao = ao = ¥, a1 = a1 = 9
14 , so that the best mean-square approximation is
given with
<D* (x) = 73 + 149 x o.,: 0.42857x + 0.64286x.
3° For determining min-max approximation we will use the following si1i1ple geomet-
rical procedure. Through the end-points of curve y = j(1:) = x 113 (0 :::; J; :::; 1) we
will draw the secant, and then tangent on the curve which is parallel to this secant
(see Fig. 7.1.2). The corresponding equations for those straight lines are

2V3
y= Ysec = :r and Y = Yta.n = X + - 9-'

so that the best min-max approximation is

1 . }3
<D*(:r:) = 2(Y.sec + Yta:n,) =X+ g ~X+ 0.19245,

whereby the the value of best approximation is llt5i II= = '{!.

7.2. Best L 2 approximation
Here, we will consider the problem of best L 2 approximation of function f : [a, b] ----+
R using linear approximation function

n
<D (:r:) = L a.i. <D (:r:)'
·i.
·i.=O

where { <Doi.} is system oflinear independent functions from the space L 2 (a, b), with sc:e1lar
product introduced by

I!

(f,g) = .lp(:r:).f(:r:)g(:r:)d:r: (f,g E L 2 (a,b)),

where p: (a, b) ----+ R+ is giwn weight function.

From the previous section we can conclude that for the best mean-square approx-
imation for .f it is necc~ssary to minimize the norm (7.1.5) by parameters a.i (-i. =
0,1, ... ,n)..
If we put I(ao, a 1 , ... , o..,.) = llr)n.ll 2 = (c5.,., c5n), then hom

I!
of
-.-. = 2
;· p(:r:)(.f(:r:)- l.:o
n
..i.<D·i.(:r:))(-<D.i(x))&r: =
0

0 (j = 0, 1, ... , n)
oa 1 .
().
·i.=O

it follows system of equations f(n· cktc~nnination of approximation parameters

(7.2.1) L(<Di, <D.i)(J,;_ = (.f, <D.7) (j = 0, 1, ... , n).

·i.=O
 Lesson VII - B<~st Approximation of Functions 113

This system can be i'epresented in matrix form as

(<I>o, <I>o) (<I>1,<I>o)

(cDn,<I>o)l
(<I>o,<I>l) (<I>1, <I>1) (<I>r,,<I>l) 0,1
aol (f, <I>l)
[(f,<I>o)l
[
(<I>o, <I>r,.) (<I>.,., <I>r,.)
. [ a,. = (f, ~,.) .

Matrix of this system is known as Gram's matrix. It can be shown that this matrix is
regular if system of functions {<I>.;.} is linearly independent, having unique solution of
given approximation problem.
System of equations (7.2.1) can be simple solved if the system of functions {<I>}
is orthogonal. Namely, then all oif-<liagonal elements of matrix of system are equal to
zero, i.e. matrix is diap.;onal one, having as solutions

(7.2.2) ('i=0,1, ... ,n).

It can be shown that by taking in the giv<m way chosen parameters ai. ('i
0, 1, ... , n) the function I reaches its minimal value. Namely, because

where (lh:j is Cronecker delta, we have

n
(pI= 2 L II<I>k:ll da~, > 0.
2

i.=O

Thus, the best L 2 approximation offunction .fin subspace Xn = L( <P 0 , <P1; ... , <I>n),
where { <P.;} is orthogonal system of functions,· is given as

(7.2.3)

A very important class of mean-square approximations is approximation by alge-

braic polynomials. In this case, the orthogonal basis of subspace Xn is constructed by
Gramm-Schmidt orthogonalizatimi procedure, starting, for example, from natural basis
{ 1, x, x 2 , ... , :rn}; or general methods for orthogonalization.
Example 7.2.1. For function :1: -+ .f(:r:) = i:£1 on segment [-1, 1] determine in the
set of polynomials not greater degree than three, best L 2 appr;dximation, with weight
fimctio;1 x-+ p(:r:) = (1- :r: 2)312.
Let us compute integral

(k: E No)

needed for further considerations (see [4], pp. 92-93). By partial integration over the
integral
+1
Nk:-1 - Nk: -
-. ' 2(k:-1)
X (1 - X,2 ) 5/2
. d.r, (kEN),
./ -1
 114 Numerical Methods in Computational Engineering

5 . 2k- 1 3
we get Nk:-1 -Nr.: = 2k _ 1 Nk:, 1.e. Nk: = k + Nk:- 1, (kEN) so that, with No= 7r
2 4 8
(2k- 1)!!
we have Nk: = 37r ( k + )!! (kEN). Starting from natmal basis {1, x, x 2 , .. . }, using
2 4
Gramm-Schmidt orthogonalisation, we get subsequently

<Do(:z:) = 1
(x, <Do)
<D1(:r:) = :r- ( ) <l?o(x) == x,
<Po, <Do
2 1 2 1
(T) ('l')-
±2., - , 'I'
, - N 1 N-
0 - . 1'
- , - -6'

- .'r' 3 - N 2 N- ' - :x' - ~x

<J? 3 (T)
.' -
1
1 x-
3 8 ,,

and corresponding norms

Because of
2 1
(f, <Do) = 5' (f, <J?I) = 0, (f, <1?2) =
21
, (f, <1?3) = 0,

using (7.2.2) we get

16 128
ao = - . a1 = 0, a2 = -, a3 = 0,
157r' :~57r

having, finally approximation in the form

* 16 128 2 1 . . ' 2
<J? (:z:) = - + -(:r - -) ~ 0.14551309 + 1.1641047x .
157r 357r 6
This function is, in addition, best approximation in the set of polynomials of degree not
greater than two.
Some further very valuable c:ousicleratiom; can be found in [1], pp. 96-99.

7.3. Best l 2 approximation

In previous sections W(~ considered problem of best approximation of function
in space L 2 (a, b). Now we will consider a particular case, mentioned in introduc-
tory section. Namely, let. function f : [a, b] -7 R be given on set of pairs of values
{ (:~:j, fj )}j=o, 1 , ... ,111,, when~ fi = f(:r:.i ). We will consider the problem of best approxima-
tion of given function by linear approximation function
'
n
(7.3.1) <J?(:r) = L a.;,<J?.;(:r) (n < m)
i=O

in sense of minimization of norm (7.1.4), where ]J: [a, b] -7 R+ is given weight function
and r5n cl1~finecl by (7 .1. 2). By involving the matrix notation

<Do (:r:o) <D1 (:ro)

laoa1
I
<Do(:1:1) <I> 1 ( :J: t) <Dn(:ro)l
<Dn(:r:1) 7 h
[fol ~
X= . f = '(/, = .

<Do /r.111 ) <Dn (:r:.,.,) , , )n ·, a:,

 Lesson VII - Best Approximation of Ftmctions llG

P = diag(p(:I:o), p(:r1), ... , p(:rm)), 'li = .f- Xc1,

square of norm, defined by (7.1.4), can be represented as
rn

(7.3.2) F = llc5nll = ll(5nll~ = LP(:r,j)Dn(;Ej) 2 = ·iJTP·u.

2

.i=O

~~r ~etermi~ation of best discrete mean-square approximation (7.3.1) it is necessary

to mmnmze F, giVen by (7.3.2). Thus, based on

DF _ 2 ~·(···)-
!-) . -
(· ·)86n(:.c.i)
~ p J,.J ()n .r1 ,
_
- 0 (i=0,1, ... ,n)
ua.,. .i=O aa.;.

we get normal system of equations

'/11.

(7.3.3) LP(:rj)(5n(:r,.i)<D.i(x.i) = 0 (i = 0, 1, ... , n)

.i=O

for determination of parameters a.i ('i = 0, 1, ... , n). The last system of equations can
be given in matrix form
XTP·u= 0'
I.e.

(7.3.4)

Note that normal system of equations (7.3.3), i.e (7:3.4) is obtained from overdefined
system of equations (7.1.1), given in matrix form as

Xa=.T,
by simple multiplication by matrix X Tp from the left side.
Diagonal matrix P, which is called weight matrix, is of meaning so that larger
weights P.i p(:r:.i) are assigned to the values of function f.i with greater accuracy. This
is of importance when approximating experimental data, which are obtained during
measures by different accuracy. For example, for 'measurements realized with different
dispersions, which relations are known, the weights P.i are chosen as inverse of disper-
sions, i.e, such that
1 -- 1 1
Po : PI : ... : Pm = 2 : 2 : ... : -2-.
. 0' 0 (} 1 O'm

When the measurements are realized with same accuracy, but with diff'erent numbers
of meas1u·ements, i.e. for every value of argument x · are proceeded m..i measurements,
and for fj taken arithmetic means of obtained re~mlts in series of measurements, then
for weights are taken numbers of measurements in series, i.e. Pj = mj (j = 0, 1, ... , m.).
Nevertheless, usually are the weights equal, i.e. P is unit matrix of order m + 1. In this
case, (7.3.4) reduces to

(7.3'.5)

Vector of coefficients a is determined fror:n (7.3.4) or (7.3.5). From (7.3.5) it follows

116 Numerical Methods in Computational Engineering

In case when the syt>t<~rn of basic functions is chosen so that <P.i(:r) :r'i ('i.

l
0, 1, ... , n) w<~ hav~~

:ro :r2
'() 0
:rl '1'2 :r:'~,
. '1 x"
X=

The method considen~d

r: :r.,n . '1'2
·'m,

it> often called least-squares method. Interesting

,.n
.l.,,n
casf~ is when
n = 1, i.e. when the approximation function is of form <P(:r:) = a. 0 + a 1 .r . Then the
system (7.3.4) becomes
·~11
[ s21
·:12]
.s22
[ ao]
n1
= [ bo
.b1
l,
where
m. m. m.
sn = 2..::.=IJ.;, s12 = s21 = 2..:.:P.i:r:.i,. s22 = 2..:.:P.i:r:.7,
.i=O .i=O .f=D
'Til. '/1),

bo = 2..::.= P.i .fi, b1 = 2..::.= P.i :r:.i fi ·

.i=O .i=O
The asked approxiu1ation parameters are

where D = sns22 - si 2 .
Example 7.3.1. Find panmwters o. 0 n.nd o. 1 in approxinwtion function <D(:r:) = ao+o. 1 :r:
using lea.st-squares method. f(n fuuction given in tabular form, as a set of values pairs

{(1.1, 2.G), (1.9,3.2), (4.2,4.G), (6.1,6.0)}.

For weight matrix P we can take unit matrix. The previously given formulas can
be directly applied, but we can start from overdefined system of equations

[j ao]
1.11 2.51
3.2
1.9 . [ h •
4. 2 (],1
[ 4 ,;)
G.1 6.0

6~1 ] from the left side, we get the

1 1 1
By multiplying with matrix xr = [ 1.1 1.9 4.2
normal system of equations

13.3]· [ a 0
G9.67 a1
l 16.2]
[ 64.33 '

wherefrom it follows

[ao]
(),1 = 61.79
1 [G9.!>7
-1:3.3
-13.3]· [ 16.2]
4. 64.33
1. 797 4G91]
[ 0.6774559 '

Thus, we have <P(:1:) = 1.7974591 + 0.6774559:1:.

 L<~~~llll VII - l3<~~t. Approxiumt.iou of Fnw:t.iou~ 117

Iu mauy an~a~ M' ~ci<~uc<~ and t<~dmolop;y, d<~aliug with P.xperimental data. w<-~ hav<~
oft<~npw hkm of paranwt.cr d<~t.<~nuiuation iu ~o ]mown P.mpiricnl formula~ which <~xpn~~~
hmct.ional rdation h<~tw<~cu two m 11101'<~ variabl<~s. Fm <~xmuplf~, ld hmctional ndat.iou
he giwm as
:tJ == f (:r:; (J.o, u.l, · · · , an),

wlwn~ o..; ('i. = 0. L .... '11.) arc paraJtwt-.<~rs which an~ to lH' dd.<~nuim~d nsiup; the followiug
t.almlat<~d data o ht.aiw~d hy nH~a~llr<~lll<~nt.

'/, () 1 m.
:r.;. :J:o :r:l :r.m
.1/i :t/o :t/1 Urn

Th<-~ measnn~d data coutaiu accid<~utal cnors of measm·eluP.uts. i.<-~. ''1wise" iu cx-
p<~rinwut. Dd<~nuiuntion of parauwters (J..i ('i. = 0, L ... , '11.) is, from the point of tlwmy
of approximation, possi1Jl<~ <mly if m. ~ n. Iu cas<~ of m. = n, we lmvP. interpolation, which
is, iu gcw~ral cas<~ nouliw~ar, what. dcp<~uds on fnuct.iou shape. In on kr to dimiwtt.<~
"noise" in data, awl obt.ain gn~atcr accuracy awlrdiahilit.y, the umnber of nH~asnreuwnt.s
shonl<l h<~ larg<-~ Pnoup;h. Tlwu, t.h<~ most. ns<~cl uwtlwd fm clet<~rmination of parauwtm·s
is least-square nwtho<L i.e. minimi,;ation of variahlf~ F clPfinecl by

'IJJ,

(7.3.0) F = L):t}j- f(:r;: ao, (l.J, ... '(J.n))2'

J=O

or nsmg
'Ill,

where arc inclwkd weights Jlj· If fnnctionalrdation between s<~veral variables is givr~u
as
Z = f(:~:, ;tJ; (J.(), a1, · · ·, (!.n)

for clct.ennination of approximation parameters we have to minimize

'1}/,

Iff is linc~ar approximation funct.i.on (in parameters a 0 , a 1 , .... an), i.e. of form (7.:3.1),
the prohl<-~m .is to 1H~ solvecl in pn~vionsly <~xplaine<l way. Nevc~rt.hdess, iff is nonlinear
approximation function, then the concspml<'iing; nmmal system of equation

uF
-=0 ('i = 0, 1, ... , n)
(7.3.7)
ua-i

is nonlinear. Fm solving of this system can be nse<l some method for solution of syst<~m
of nonlinear eqnat.ions, like N cwton-Kantmovich method~ whereby this proceclme is
rather complicat.c~d. In order to solve problem in c~asier way, there are some simplihecl
methods of transfonuation of snclt problems to linear approximation nwt.hod. N amdy,
hy introducing some sulJstitnt.ions, like

(7.3.8) X= .r;(:r:), Y = h(y)

nonlim~ar problem reduces to linear one.

118 Numerical Methods in Computational Engineering

For example, let :1;---:- .f(:r:; a 0, o, 1) = a0 ea 1 :r'. Then, by logaritmization and substitu-
tion ·
X=:c, Y=logy, b0 =loga 0 , b1 =o, 1,
the problem reduces to lirwar one, because now Y = b0 + b1 X. Thus, by minimization
of
·m.
(7.3.9) G = G(bo, h)= I)Y;- bo- b1Xjf,
j=O

where X.i = :r:_i and ~i = log Y:i (.j = 0, 1, ... , n), we determine the parameters b0 and
b1, and then
ao = ebn and a1 = b1.
Nevertheless, this procedure does not give the same result like the minimization of
function
'In

F = F(ao, an)= L(Y.i ~ a 0ea 1 '1':i) 2 .

.i=ll

Moreover, the obtained n~sults can significantly deviate, because the problem we are
solving is different from stated om~, having in mincl transfonnation we have done (Y =
logy). But, for many practical engineering problems the parameters obtained in this
way are satisfactory.
We will specify some typical functional dependencies with possible transformation
of variables.
1° y=o, 0 :c 1]. 1 , X=log:r, Y=log:y, bo=logao, b1=a1;
2° y = a0 a'[, X= :r:, Y =logy, bo= logan, h = loga1;
(},1 1
3° ''lJ = a 0 + -.
:J: '
X= -.
:r: '
Y = 'IJ, 1J 0 = ao, b1 = 0,1;
'
0 (1,1
4 :IJ = ao + -,
:I:
.X= :1:, Y = :ry, bo = a1, h = ao;
1
5() :lJ= - - - -
+
ao a1:1:'
:r:

Example 7.3.2. Rc.'mlt of 11H!<I.SHrcmeuts of values 1: and y are gwen 111 f(>llowinp,
talmlar f(mn.
'/, 0 1 2 3 4 s
:r.;. 4.48 4.98 5.GO G.ll 6.62 7.42
Y·i. 4.15 1.95 1.31 1.03 0.74 0.63

1
Ify reduce to linea.r prohlem and approximate llsmg least-.s·quare
(/,() + (J,l:r:'
method.
By involving X = :r:. Y = 1 /:1; and nsing 1<-~ast-sqnare method we get approximation
1
fnnction <D(X) ~ 0.4u8X- 1.843. wlwrefrom it f<lllows Y ~ OAGS:r~ _ 1. ·
843
 L<~ssou VII - Best Approxilllation of Ft1nctinus ll~J

From the previof1s mw can condnde that. dep<~ucling on f the collv<~nient replac<~­
ments (7.3.8) shoul<l be dwseu so that they Puabl<~ reducing of

Y = f(:~:; ao, a1, ... , a,.)

to linear form oL for <~xmupl<~. polyuomial type

(7.3.10)

!t i~ clear that functions .11 and h umst have their inverse functions, so that (7.3.10) is:
m fact, equivalent to
h- 1 (Y) = f(_q- 1 (X); ao, a1, ... , a.,),
whereby parameters b.;, depeucl on parameters a.;, in rather simple way.

7 .4. Packages for approximation of functions

Procedures fm clevdoping polynomials for discrete data are very important in <~ngi­
neering practice. The clin~ct fit polynomials, the Lagrange polynomial, and the divicl<-xl
difference polynomial work well for nonequally spa.c<~d data. For eqnally spaced <lata,
polynomials base< l on diff<~rcnces are recommended.
Procednres for <l<~vcloping least squares approximatious for discrete data an~ also
valuable in engineering practice. L<~ast squares approximations are nseful for larg<' s<~ts of
data ancl sets of rongh data. L<>.ast sqnare polynomia.l approximation is straightJonvnn 1.
for both one inclepcmlent variabk and more than one variable. The least sqnan:s twnnnl
equations corresponding to polynomial approximating functions are linear, w hic:h l<~ads
to very efficient solving proccdun~s. For nonlinear approximating functions, the l<~ast
squares normal equatious are nonlinear, which leads to complicated solution procednres.
As previously mentioned, conv<-~nient mapping of nonlinear approximating function to
linear one (i.e. linearization) can solve this prohlc1n usually good enough. Least sqnares
polynomial approximation is a straightforward, simple, and accurate procedure for ob-
taining approximating functions f(w large sets of data or sets of rough experimental
data.
Numerous libraries and software packages are available for approximation of func:-
tions, especially for polynomial approximation.
Many comnH:~rcial software pa.ckages contain routines for fitting approximating poly-
nomials. Sonw of the more prominent packages are Matlab and Mathcad. More so-
phisticated packages, such as IMSL, Mathematica, and Macsyma contain also rcmtines
for fittiug approximating polynomials. The book Numerical Recipes ([G]) contains
numerous subroutines for fitting approximating polynomials (see Chapter 15, Mocldling
of Data), and the book Numerical Methods for Engineers and Scientists ([2])
prograrn code for fitting <tpproximating polynomials (t>ee Chapter 4, Polynomial Ap-
proximation and Interpolation).

Bibliography (Cited references and further reading)

[1] Mih;vanovic, G.V., Numer'ical Analys'is II, Nauc:na knjiga, Beograd, 1988 (Ser-
bian). · •
[2] HofFman, .J.D., N'umerical Methods for Engineers and Scient'ists. Taylor
& Francis, Boca Raton-Lonclou-New York-Singapore, 2001.
[3J MilovanoviC:, G.V. and Djorcljevit:, Dj.R., Programiranje numerickih metoda
na FORTRAN jeziku. Institnt za dokumentaciju zastite na raclu "Eclvard
Kardelj", Nis, 1981 (Serbian). .
[4] Milovanovic:, G.V., N·umerical Analysis I Nauc:na kn.iiga, Beogi·ad, 1988 (Ser-
bian). ·
[5] Stoer, .J., and Bulirsch, R, Introduction to Numerical Analysis, Springer,
New York, 1980.
120 Numerical Methods in Computational Engineering

[6] Press, W.H., Flannery, B.P., Teukolsky, S.A., and Vetterling, W.T., Numerical
Recepies - The Art of Scientific Computing. Cambridge University Press,
1989.
[7] Milovanovic, G.V. and Wrigge, S., Least squares approximation with con-
straints. Math. Comp. 46(1986), 551-565.
[8] Milovanovic, G.V. and Kovacevic, M.A., Zbirka resenih zadataka iz nu-
mericke analize. Naucna knjiga, Beograd, 1985. (Serbian).
[9] Dordevic, D.R., Ilic, Z.A., Realization of Stieltjes procedure and Gauss-
Christoffel quadrat'ure formulas in system Mathematica. Zb. rad. Gradj.
fak. Nis 17(1996), pp. 29-40. .
[10] Todd, .J., Introduction to the Constructive Theory of Function§. New
York, 1963.
[11] Schoenberg,I..J., Contributions to the problem of approximation of equi-
distant data by analytic functions. Quart. Appl. Math. 4(1946), 45-99; 112-
141. -
[12] Ralston,A., A Pirst Course 'in Numerical Analysis. McGraw'-Hill, New York,
1965.
[13] Brass, H., Approximation durch Teilsummen von Orthogonalpolynom-
reihen. In: Numerical Methods of Approximation Theory (L. Collatz, G. Meinaruf:i,
H. Werner, eels.), 69-83, ISNM 52, Birkhauser, Basel, 1980.
[14] Brass, H., Error estimates for least sq'uares approximation by polyno-
mials . .J. Approx. Theory 41(1984), 345-349.
[15] Collatz, L. and Krahs, W., Approximationstheorie. Tschebyscheffsche Ap-
proximation with Anwendungen. B.G. Teubnc~r, Stutgart, 1973.
[16] Davis, P ..J., Interpolation and Approximation. Blaif:idell Publ. Comp., New
York, 1963.
[17] Feinerman, R.P. & Newman, D ..J., Polynomial Approximation. Williams &
Wilkins, Baltimore, 1974.
[18] Fox, L. & Parker, I.B., Chebyshev Polynomials in Numerical Analysis.
Oxford Univ. Press, 1972.
[19] Hildebrand, F.B., Introduction to Numerical Analysis.Mc.Graw-Hill, New
York, 1974.
[20] Rivlin, T ..J., An Introduction to the Approximation ad Functions. Dover
Publications, Inc., New York, 1981.
[21] IMSL Math/Library Users Manual, IMSL Inc., 2500 City West Boulevard,
Houston TX 77042 ·
[22] NAG Fortran Library, Numerical Algorithms Group, 256 Banbury Road, Ox-
ford OX27DE, U.K., Chapter F02.
Faculty of Civil Eugirwminp; Faculty of' Civil Engirw<~ring and An:hitect.m<~
Belgrade ~ NiH
Mast<~r Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON VIII

8. Numerical Differentiation and Integration

8.1. Numerical Differentiation - ..

In this section the rnnnerical differentiation of real functiom; d('fi ned u11 ~ u, 1
: ,.; II

be considered.

8.1.1. Introduction
The need for mmH~rical differentiation appears in following cases:
a. When values of function an-~ known only on discrete set of points on [a, b], i.e.
function f is given in tabular form;
b. When analytic:al expression for hmction f is rather complicated.
Numerical differentiation is chiefly based on approximation of function f by func-
tion <D on [a, b], and then <liffen~ntiating <D desirable times. Thus, based on f(:r) rv
<D (x) (a :S: :r :S: b)), we have

For function <D are mostly taken algebraic interpolation polynomials, because being
simple differentiating. Let <D be interpolating polynomial of 'ri-th degn~e, i.e.

<D(:r:) = Pn(:r:).

If known error Rn (f; :r:) of approximation polynomial

(8.1.1.1) f (:r) = Pn. (:r) + Rn (f ; :r) (a :S: x :S: b)

it is possible to estimate error in formula for differentiation, i.e. fi:·om (8.1.1.1) it follows

It is me~mingful to take for order of derivative only k < n.

It is obvious that numerical differentiation has smaller accuracy than interpolation.
So, for example, for interpolation is error in nodes equal to zero, what is uot in case of
differentiation.

8.1.2. Formulas for numerical_differentiation

If known values of function f. on set of equidistant points { :r:o, x1, ... , :Em} C [a, b],
with step h, let --
+
.h = f(:th:) = f(:r:o k ·h) (k = 0, 1, ... , m,).

121
122 Numerical Methods in Computational Engineering

Construct over set {x.;,,x.;,+l, ... ,:ri+n} (0 ::; i ::; m- n) first Newton interpolation
polynomial (see Chapter 6) ·

Po .(x) = f·.
n , '·
+ p ~ j'.·'· + P (p2!- 1 ) ~ 2 1·.
~
+ P (p ~ 13!) (p - 2)
u
A 3
1.~ + ...
. ·
(8.1.2.1)
P (p- 1) ... (p- n + 1) A nf··
+ n.' u 'tl

. p = (1:-
wh ere :1:.;,
1 dPn(x) , by d1fferentlat10n
)/ 7L. B ecause Pn1 ( x ) = - . . . (8.1.2.1) we g·et
h dp . .

2
(8.1.2.2) 1 (• ·) _
Pn:J, ---;;,~./.;.+
1( 2p - 1 2 .
~Ji+
3p - 6p + 2 ~Ji+
3
... ).
2 6

By further differentiation of (8.1.2.2) we get, in turn P;;, P;:~, and so on. For example,

(8.1.2.3) Pn. (:r:)

II
= -
1(2• 3·
~ .fi + (p -1)~ .fi + ... ).
72
),

For x = :1:.;,, i.e. p = 0, formulas (8.1.2.2) and (8.1.2.3) reduce to

1
1 ( A f' 1A2 ·
PnI ( :r..• ;, ) -_ -1
.
u, ·i.
I,
- -2 u f.;. + -31 ~ 3 .fi· - ... + ( -1) n -
,n ~
n .)
./.;, '
Pn (x.;,) -_1(
II 2· .3• 11 4· )
h,2 ~ f.;. -A f.;.+ 12 ~ fi- . ' ..

Some useful formulas for the first derivative are

f'(x.;,) =*(f.;,- .f.i-1) + ~.t" (Til), (:ri-1 < ' 11 < x.;,),
2
1 ( . . . h
h, 3.f.;.- 4.f.;,_1 + .f?.-2) + 3.f
·I ( ) ,Ill
j Xi = (T/2), (Xi-2 < 7/2 <Xi),
2
~6 1• (llf.;.- 18.fi.-1 + 9.fi-2- 2.fi-3) + 1~ .f 1 V (713)
3

f'(.Ti) = (xi_ 3 < 'f/ 3 <:c.;,),.

Previous formulas fm: first derivative in node :r;i are obviously asymmetric and are usually
applied on the interval [o., IJ] boundaries. Typical application of these formulas is at
approximation of differentiable boundary conditions in contour problems of differential
equations.
For nodes inside of segment [a, h] is better to use symmetric differentiation formulas.

where

The most usf~cl and simplest f()rmula for approximation of second derivative is

D 2 .f'-
'1. - 2_(f'·
1), 2 ' 1.+ 1-2f'·+f'·
. '/, ' '1,- 1)+T(j) )
 Lesson VIII - Numerical Differentaiation and Integration 123

where, under condition f E C 4 [a, b], the remainder term is

8.2. Numerical integration- Quadrature formulas

8.2.1. Introduction
Numerical integration of functions is dealing with approximative calculation of
definite integrals on the basis of the sets of values of function to be integrated, by
following some fornnila.
Formulas for calculation of single integrals are called quadrature formulas. In
similar way, formulas for dm.1ble integrals (and multi-dimem;ional integrals, too) are
called cubature formulas.
In our considerations, we will deal mainly with quadrature formulas.
The need for numerical integration appears in many cases. Namely, Newton-
Leibnitz formula
·b
(8.2.1.1) . a. f (x) dx = F (b) - F (a) ,
/

where F is primitive function of function .f, cannot always be applied. Note some of
these cases:
1. Function F cannot be represented by finite number of elementary functions (for
•)

example, when f(:r;) =e-x" ).

2. Application of formula (8.2.1.1) leads often to very complicated expression, even at
calculation of integral of rather simple functions, e.g.

r· 1.+dxx 3
=log \/Ia+ II- -6~ log(a 2 - a+ 1) + ~3arctg2a~a·
./ 0 v<>

3. Integration of functions with values known on discrete set of points (obtained, for
example, by experiments), is not possible by applying formula (8.2.1.1).
Large immber of quadrature formulas are of form

(8.2.1.2)

1
where .h = f(:r:k) (a :S :r: 0 < ... < Xn :S b). If .To =a and Xn = b, formula (8.1.1.2) is
of closed kind, and in other cases is of open kind.
For integration of differentiable functions are used also formulas which have, in
addition to function values, values of·its derivatives. The formulas for calculation of
integrals of .form
·b
. a. p(x)f(:r;)d1:,
/

where x -+ p(x) is given weight function, are also of concern.

One simple way for construction of quadrature formulas is founded on application
of interpolation. Formulas obtained in this way are called interpolating quadrature
formulas. .
Let the values of function fin given points xo, 1:1, ... , xn( E [a, b]) be fo, h, · · ·, fn
respectively, i.e.
(k=0,1, ... ,n).
l:L4 l~mrwncallVletlwcls m Uomput.atwnal J:!.Jngmeenng

On the ba:-;is of tlwH<~ <lata., w<~ can construct Lagrange interpolation polynomial

where w(:r) = (:r- :r:o) (:r:- :1:1) · · · (:r- :r:n).

Then
., ;·b p(:D)Pn(:r)d:r + Rn+L(.f),
!
. (),
J{r:).f(:r)d:r =
. (),

1.e.

(8.2.1.3)

where we put
., p(;r)w(:~:)
Ak=
· !
. n (:r - :r:J.:)w' (:r:J.:)
dx (k=0,1, ... ,n.).

In formula (8.2.1.3), Rn+ 1 (f) is called rmnainder term, residue (rest, residuum) of
quadrature formula and reprcsentt-> error dmw by replacing of integral by finite snn1.
Index n, + 1 in nmmiwl<~r term <lcnotes that int<~gral is approximate calculated based on
values of function to be integrated in n + 1 points.
Denote with rrn set of all polynomials of degree not greater than n.
Because .f(:r:) = :r": (k = 0, 1. ... , n), f(:r) = Pn.(:r), we have Rn+l(:rh:) = 0 (k =
0, 1, ... , n), wherdrom we conclude that formula (8.2.1.3) is exact for every f E rrn,
regardless of choiu~ of int<~rpolatiou nodes :1: 1.: (k = 0, 1, ... , n) and in this case we say
that (8.2.1.3) has algebraic <lcgn~e of accuracy n.

8.2.2. Newton-Cotes formulas

In this section we will d<~velop quaclratme formulas od dosed type in which the
interpolation nocles :r:~,: = :r 0 + kh (k = 0, 1, ... , n) are taken <~quidistantly with a st<~p
h=lJ-o._
n
If w<~~ introduce substitution :r - :r 0 = ]Jh, we have

(8.2.2.1) w(:r) = (:r:- :1:o)(:1:- :r:I) ... (:r:- :r:.,.) = hn+ 1 p(p- 1) ... (p- n)

and

w'(:rk) = (:1:1,:- :ro)(:I:J,:- :1:1) ... (:1:1.:- :I:J,:-l)(:r:k- :r;l.:+l) ... (:r,,:- :r:n)
(8.2.2.2)
= 11. 11
( -1)"-l.:!.:!(n- k)!

By introducing notation for gcnerali7:ed d<~gree :1:(s) = :r(:1: -1) ... (:r- s + 1), based
on (8.2.2.1), (8.2.2.2) and n~:-mlts from previous section, we get

(k = 0, 1, ... , nL

l.P.
A,,: = (IJ- o.)H1,: (k = 0, 1, ... , n),
 Lesson VIII - Numerical Differentaiation and Integration 125

where we put

(8.2.2.3)
_
Hk: = Hk(n) -
_ ( -1)n-h:
1
(n) •
~r·n p(n+l)
• dp (k = 0, 1, ... , n).
n.n k . 0 p- k

Coefficients Hk are known in literature as Newton-Cotes coefficients, and corresponding

formulas ·

X~n~b ~ b- a
(8.2.2.4) f(:r:)(11: = (b- a) L Hx:.f(a + k - ) (kEN)
. n
:r;o=a. k:=O

as Newton~Cotes formulas.
Further on we will give survey of Newton-Cotes formulas for n ~ 8. Here we use
denotations h = b- a, h: = .f(.'EA:) (k = 0, 1, ... , n).
n
1. n = 1 (Trapezoid rule )

X!

./ f(:I:)dx =~(.to+ h)- ~~!"(6);

:r.o

2. n = 2 (Simpson's rule )

I..
~£·)
-
. .f(:r:)d.'E =
h .
3 (./o + 4h
.
+h)-
h
5

90
! IV( )·
6 ,
xa

3. n = 3 (Simpson's rule t)
xa t r.:::

8 uo +3ft+ 3]2 +h)-

. . 3h . . . 3h" IV( )
. .f(:I:)dx =
80
f 6 ;
/
:co

4. n = 4 (Boole's rule)

.l
xo
f(x)dx = ~~ (7fo +32ft+ 12/z + 32h + 7f•)- !~~!vi (e.);
5. n = 5

.j'' f(:E)dx.
.'{;5

=
5h . . .
(19.f0 +75ft+ 50]2 + 50h + 75!4 + 19!5)-
275 1 ·(6)
12096
f (~5);
} 7

288
xo

6. n = 6
:r;ci

:r;o
/ f(x )dx =
1 ~0 (41f 0 + 216.fl + 27 h + 272h + 27 !4

9h9 j'8(t )·
+ 216!5 + 41!6 ) ~ 1400 <,6 '
126 Numerical Methods in Computational Engineering

7. n = 7

I.'(
.'L7

)
.. .f x dx =
7h
17280
(751.fo + 3577 h + 1323h + 2989.!3 + 2989f4
xo

8. n = 8

.I ()
xs

. f :r d.r =
4h
14175
(989fo + 5888h - 928h + 10496h- 4540f4

~36678;;~ f(lO) ((8 );

1

+ 1049Gfs - 928-;f'G + 5888h + 989f8 ) -

where 6: E (:ro, :r~;:) (h: = 1, ... , 8).

In general case, the residue Rn+ 1 (f) is of form

R ·n+1 (t. ') = C n 71. m.,·(m-1)(t

. <..;n
) (.To < ~n < :rn),
where ·m = 2[~] + 3~ Giv<~n cqna.lity has a meaning if function f E cm-l [a., b].

8.2.3. Generalized quadrature formulas

In order to compute value of integrals more accurate it is necessary to divide seg-
ment [o., b] to the set of subsegments, and then to apply to each of them some quadrature
formula.. In this way w<~ get generalized or composite formulas. In this section we will
consider generalized fonrmlas obtained on the basis of trapezoid or Simpson's formula.
Divide the segment [a, IJ] on set of subsegmeuts [:~:.i_ 1 , :r.i] so that :r.; = a+ ·ih and
h = (b- a.)/n (see Fig. 8.2.3.1)).

y
y = f(x)

/~
I

Figure 8. 2.3.1
By applying the trap<~zoid formula on every subsegment, we get

Le.
. IJ
. '3 L'/ . .f
/, ·II

I
.(}.
f(:r)rl:J: = Tn-
12 ·i=1
((i),
 L<~ssou VIII - N muerical Diffr.n~ntaiation ai1d Int<~gration 127
where
Tn. = Tn(.f: h)= h(~fo + h + · · · + fn.-1 + ~fn.)
and
.'J'·
"1.-1 < '-,·!.
(,. < '"/,
'['. (·,·-
. - 1' 2' ''1)
..• ' ,, .

Theorem 8.2.3.1. Iff E C 2 [a, b] the equality

{,

. .( ) (b - (},) 2 •// )
./
.f :~: d:r:- Tn. =- 1'). 2 .f ((
~11.
(a<(<b)
.(}.

holds.
Quadrature formula

,,
./ f (:1:) rl:r ~ Tn. (f; !1.) (h=b-a)
'II.
(),

is called generalize<l trapezoid formula.

b- (J. . .
Suppose now that h = -,-, 1.e. :r.;, =a+ ·th ('i = 0, 1, ... , 2n) (See Fig. 8.2.3.2),
2n
ancl then apply Simpson's rule to suhsegments

In this way we get gcw~ralizer l Simpson's fonnula

,,
./ f(:r)d:c ~ S.,Jf; h) _ b-a)
('1.- '
2n
(),

where

S., = S.n.(.f; h)= ~{fo + 4(.f1 + · ·: + hn-1) + 2(.1'2 + · · · + hn-2) + hfl.}.

y

y = f(x)
I
I
I

X
x =a X X =b
0 2n-:-2 2n

Figure 8.2.3.2
128 Numerical Methods in Computational Engineering

Theorem 8.2.3.2. Iff E C 4 [a, b] the equality

/ .f(:z:)rh:- Sn = - ~8~0(;;42 f(IV)(~) (a,<~< b)

(),

holds.

8.2.4. ·Romberg integration

For calculation of definite integrals in practice is most frequently used generalized
trapezoid formula in a. special form, known as Romberg integration.·
Denote with Tk~o) trapezoid approximation Tn.(/; h) (n = 2k), i.e. h = (b ;: a,)).
Romberg integration consists of construction of two-dimensional set Tk~m) (rn =
0, 1, ... , k; k = 0, 1, ... ) using

(8.2.4.1)

Using (8.2.4.1) one can construct so known T table

T(O) -t T.(l) -t T.(2) -t 1'(3)
0 0 0 0

/' /' /'

T(O) -t T(1) -t T(2) -t
1 1 1

/' /'
T(O) -t T(1) -t
2 2

/'
T(O) -t
3
by taking k = 0, 1, ... and m. = 1, 2,.. .. In first column of this table arc in turn
approximate values of integral I obtained by rm~etns of trapezoid formula with h~;: =
(b- a)/2k: (~: =.0, 1, ... ). Second column is obtained based on the first, using formula
(8.2.4.1), t.lurd from second, aml so on.
Iterative process, <lefincd by (8.2.4.1) is the standard Romberg method for numer-
ical integration. One can proV<~ that series {T~;~·m.) hENa and {rtn.) }mE No (by columns
and rows in T-table) converge to I. At practical application of Romberg integration,
iterative process (8.2.4.1) is usually interrupted when ITJm)- r(;m- )1 ~ f, where cis
1

in advance allowe<l error, an<l then as result is taken I'?!: rJm).

8.2.5. Program realization

In this section we give program realization of Simpson's and Romberg integration.

Program 8.2.5.1.
For integration using g<~m~raliz<~<l Simpson's formula the subroutine INTEG is writ-
ten. Parameters in parameter list are of meaning explained in C- comments of suhpro-
gram source coch~. Function to 1w integ;ratecl is given in subroutine FUN, and clepencls
on one panuneter Z. By intc~gcr parameter J is provided simultaneous specifying more
functions to integrate.
 Lesson VIII - Numerical Differentaiation arid Int1~gration 129

Subroutine INTEG is organizecl in this way that initial number of subsegnwnts

can be improved (by reduction of step h to h/2) up to MAX=10000 In case when
relative difference in integral:-:; values, obtained by steps hand h/2, is less than 10- 5 , the
calculation interrupts awl value of integral calculated with the smallest step is taken
as definitive vahw of integral. If this criterion cannot be fulfilled with less than MAX
subsegments, the message KBR= 1 is printed (and in opposite case KBR=O) 0
As a test examples for this subroutine, the following integrals are taken:

1
cz:r:

1 +z
0

2 2 (h: (z = 1.0(001)1.5),
0 :r:
0
1/2

. / 1r sin( 1r zoy,) d:r: (z = 1.0(002)1.4)

()

2
log(:1: + z) sin:cd,·
1
0

1
°

z 2 + e' :c
1
'
,J, (z = 000(001)005)0

Subroutines, main program, and output listing are of form:

C========================================================
C COMPUTATION OF DEFINITE INTEGRAL OF FUNCTION F(X,Z,J)
C BY SIMPSON'S RULE
C========================================================
SUBROUTINE INTEG(A, B, S, F, J, KBR, Z)
C A - 'LOWER LIMIT OF INTEGRAL
C B - UPPER LIMIT OF INTEGRAL
C S - VALUE OF INTEGRAL WITH ACCURACY EPS=1.E-5
C KBR -· CONTROL NUMBER
C KBR=O INTEGRAL CORRECTLY COMPUTED
C KBR=1 INTEGRAL NOT COMPUTED WITH SPECIFIED ACCURACY
C Z - PARAMETAR OF INTEGRATED FUNCTION
C INITIAL NUMBER OF SEGMENTS IS 2*MP MAXIMAL IS MAX=1000
MP=15
MAX=1000
KBR=O.
N=2.*MP
SO=Oo
SAB=F(A,Z,J)+F(B,Z,J)
H=(B-A)/FLOAT(N)
X=A
S1=0o
N2=N-2
DO 5 I=2, N2, 2
X=X+2.*H
5 S1=S1+F(X,Z,J)
10 S2=0.
X=A-H
N1=N-1
. DO 15 I=1, N1, 2
X=X+2o*H
15 S2=S2+F(X,Z,J)
S=H/3.*(SAB+2.*S1+4,*S2)
REL= (S-SO) /S
IF (ABS(REL)-1.E-5) 35,35,20
20 IF (N-MAX) 25,25,30
130 Numerical Metho<lt> in Computational Engineering

25 N=2*N
H=0.5*H
C NUMBER OF INTERVALS IS DOUBLED AND
C NEW VALUE FOR S1 IS COMPUTED
S1=S1+S2
SO=S
GO TO 10
30 KBR=1
35 RETURN
END
FUNCTION FUN(X,Z,J)
GO TO (10,20,30) ,J
10 FUN=EXP(Z*X)/(X*X+Z*Z)
RETURN
20 PI=3.1415926535
FUN=PI*SIN(PI*X*Z)
RETURN
30 FUN=ALOG(X+Z)/(Z*Z+EXP(X))*SIN (X)/X
RETURN
END
EXTERNAL FUN
OPEN(8,File='Simpson.IN')
OPEN(6,File='Simpson.out')
WRITE(6,5)
5 FORMAT (1H1,2X, 'IZRACUNAVANJE VREDNOSTI INTEGRALA',
1 ' PRIMENOM SIMPSONOVE FORMULE ' //14X,
2 'TACNOST IZRACUNAVANJA EPS=1.E-5'
3 ///11X, 'J' ,4X, 'DONJA' ,5X, 'GORNJA' ,3X, 'PARAMETAR',
4 3X,' VREDNOST'/ 16X, 'GRANICA', 3X,'GRANICA',
5 5X,'Z' ,7X,'INTEGRALA'//)
DO 40 J=1,3
READ (8,15) DG, GG, ZP, DZ, ZK
15 FORMAT(5F5.1)
Z=ZP-DZ
18 Z=Z+DZ
IF (Z.GT.ZK+0.000001) GO TO 40
CALL INTEG (DG,GG,S,FUN,J,KBR,Z)
IF(KBR) 20,25,20
20 WRITE (6,30)
30 FORMAT (/11X, 'INTEGRAL NIJE KOREKTNO IZRACUNAT'/)
GO TO 18
25 WRITE (6,35) J,DG,GG,Z,S
35 FORMAT (11X,I1,F8.1,2F10.1,F15.6/)
GO TO 18
40 CONTINUE
STOP
END

0. ,1. ,1. ,0.1,1.5

0. ,0.5,1. ,0.2,1.4
1. ,2. ,0. ,0.1,0.5

IZRACUNAVANJE VREDNOSTI INTEGRALA PRIMENOM SIMPSONOVE FORMULE

TACNOST IZRACUNAVANJA EPS=i.E-5
J DONJA GORNJA PARAMETAR VREDNOST
GRANICA GRANICA Z INTEGRALA
1 .0 1.0 1.0 1,270724
 Les:->011 VIII - N nmerkal DiffereutaiaLion aiul Intc~gration 1~1

1 .0 1.0 1.1 1.153890

1 .0 1.0 1.2 1.059770
1 .0 1.0 1.3 .983069
1 .0 1.0 1.4 .920013
1 .0 1.0 1.5 .867848
2 .0 .5 1.0 1.000000
2 .0 .5 1.2 1.090848
2 .0 .5 1.4 1.134133
3 1.0 2.0 .0 .048047
3 1.0 2.0 .1 .059595
3 1.0 2.0 .2 .069940
3 1.0 2.0 .3 .079052
3 1.0 2.0 .4 .086920
3 1.0 2.0 .5 .093558
Program 8.2.5.2.
Now we give program realization of Romberg integration in double arithmetic com-
puter precision DOUBLE PRECISION. List in subroutine is of following meaning:
DG - lower limit of integral;
GG - upper limit of integral;
FUN - name of function suhroutim~ which defines function to he integrated;
EPS - demanded accuracy of computation;
VINT - value of integral for given accuracy EPS, if KB=O;
KB - control number (KB=O - integral correctly computed; KB=1 - accuracy of com-
puting not reached after 15 proposed steps, i.e. with numbers of subsegments 215 ). For
testing of this subroutine is taken tabulating of function ·
:r:
2
F(:r:) = ./ e-t dt (:r: = 0.1(0.1)1.0),
0

with accuracy 10- 5 . Routines codes and output listings are of form:
C=================================================================
C ROMBERGOVA INTEGRACIJA
C======================~==========================================
DOUBLE PRECISION GG, FUN, VINT
EXTERNAL FUN
open(6,file='rornberg.out')
EPS=1.E-8
WRITE (6,11)
11 FORMAT(1H0,5X, 'X', 7X, 'INTEGRAL(O. ,X)'/)
DO 10 I=1, 10
GG=0.1*I
CALL ROMBI(O.DO,GG,FUN,EPS,VINT,KB)
IF (KB) 5,15,5
5 WRITE (6,20) GG
20 FORMAT (5X,F3.1,4X,'TACNOST NE ZADOVOLJAVA'//)
GO TO 10
15 WRITE(6,25)GG,VINT
25 FORMAT(5X,F3.1,4X,F14.9)
10 CONTJ;NUE
STOP
END
SUBROUTINE ROMBI (DG,GG,FUN,EPS,VINT,KB)
DOUBLE PRECISION FUN,VINT,T(15),DG,GG,H,A,POM,B,X
. KB=O
H=GG-DG
 132 Numerical Methods in Computational Engineering

A=(FUN(DG)+FUN(GG))/2.
POM=H*A
DO 50 K=1, 15
X=DG+H/2.
. 10 A=A+FUN (X)
X=X+H
IF (X.LT.GG) GO TO 10
T(K)=H/2.*A
L "'••
B=1.
IF (K.EQ.1) GO TO 20
.K1=K-1
. 'DO 15 M=1, K1
I=K-M
B=4. *B
15 . T(I)=(B*T(I+1)-T(I))/(B-1.)
·. 20 B=4.*B
VINT=(B*T(1)-POM)/(B-1.)
IF(DABS(VINT-POM).LE.EPS) RETURN
POM=VINT
50 H=H/2.
KB=1
RETURN
END
FUNCTION FUN(X)
DOUBLE PRECISION FUN,X
FUN=DEXP(-X*X)
RETURN
END

0 X INTEGRAL ( 0. , X)
.1 .099667666
.2 .197365034
.3 .291237887
.4 .379652845
.5 .461281012
.6 .535153533
.7 .600685674
.8 .657669863
.9 .706241521
1.0 .746824138

8.2.6. On numerical computation of one class of double integrals

In this section we will point ont to one way for approximate calculation of double
integrals of form

(8.2.6.1) ./lf(:r, y) dxdy,

G

where area of int<~gration i:-; nnit circle, i.e. G = {(x, y) I x 2 + :t/ ::; 1}. Namely, for
numerical computation of the integral (8.2.6.1) in literature is known formula

(8.2.6.2) ./lf(:r:, y) d:r:dy ~ -i(2.f(O) + t .f(Mi)),

G .,.=1 .
 LeHHou VIII - N nuwric:al Diffen~utaintion and Int<~grat.iou

when~ 0 iH origin, iA 0 = (0, 0), awl poiutH Nii have polar coordinates

('i.=1,2, ... ,u).

According to fonnnla (~.2.u.2) we will n~ali~<~ program for computation of double

integralH, with unit circle~ aH an~a of int<~gration. Program organization will br. such that
by function t>ubroutine EF can lH~ defined several different functions f to he integrated.
Paramet<~rH iu list of panmwt<~rH an~ of following meaning:
X - value of argnuwnt :1:;
Y - value of argnnwnt y;
K - integer that defines diff<~rcnt fnnc:tiow; to be intP.grated.
Formula (8.2.u.2) iH n~ali~~,<~d hy Hnbroutine DVINT, which parameters in list are of
following meaning:
EF ~ lHtuH-~ of function :·:mhron tim~;
K - integer with sauw nwauing lilw in snbmutine EF;
VRINT- computed valn<~ of illt<~gral, obtaiw~d by using formula (8.2.6.2).
SUBROUTINE DVINT(EF, K,VRINT)
PI=3.1415926535
RO=SQRT(2./3)
PI3=PI/3
FI=-PI3
VRINT=2.*EF(O. ,0. ,K)
DO 10 I=1,6
FI=FI+PI3
X=RO*COS (FI)
Y=RO*SIN (FI)
10 VRINT=VRINT+EF(X,Y,K)
VRINT=PI/8.*VRINT
RETURN
END
Main program is of form:
C====================================================
C IZRACUNAVANJE DVOSTRUKOG INTEGRALA
C====================================================
EXTERNAL EF
OPEN(6,FILE='DVINT.OUT')
WRITE (6,5)
5 FORMAT (1H1//10X,'IZRACUNAVANJE DVOSTRUKOG',
1' INTEGRALA'//)
DO 10 K=1,3
CALL DVINT(EF, K~ VRINT)
10 WRITE (6, 15) K, VRINT _ .
15 FORMAT (15X,I1,' PRIMER'// lOX,
1 'VREDNOST INTEGRALA =',F12.6//)
STOP
END
By using this program we calculated approximately values of the following integrals:
. 1G:r: 2 y 2
II
. .
G
1 + :r 2
•
+ y2
duly;

.// Jl + (1 + :r) 2 + y 2 ch:dy;

G

.!/.J2-
G
---;=2=4=;:r:o=2=::::;:;: d:r:dy.
:r;2- y2
134 Numerical Methods in Computational Engineering

Function subroutine EF and output listing are of form:

FUNCTION EF(X,Y,K)
GO TO (1Q,20,30),K
10 EF=(16.*X*X*Y*Y)/(1.+X•X+Y•Y)
RETURN
20 EF=SQRT(1.+Y•Y+(1.+X)••2)
RETURN
30 EF=(24.*X*X)/SQRT(2.-X•X-Y•Y)
RETURN
END

1
IZRACUNAVANJE DVOSTRUKOG INTEGRALA
1 PRIMER
VREDNOST INTEGRALA = 1.256637
2 PRIMER
VREDNOST INTEGRALA = 4.858376
3 PRIMER
VREDNOST INTEGRALA = 16.324200

8.2. 7. Packages for Numerical Integration

N nmerical intq;ration of both discrete data and known functions are needed iu
engineering practic<~. Tlw proce<lnres for first case are based on fitting approximating
polynomials to tlw <lata alHl integrating the approximating polynomials. The direct
fit polynomial method works wdl for both equally spaced data and non-equally spaced
data. Least squan~s fit. polynomi<tl:-; can be used for large sets of data or sets of rough
data. The Newton-Cot<-~s formula:-;, which are based on Newton forward-difference poly-
nomials, give simpl<~ integration formulas for equally spaced data. Romberg integration,
which is extrapolation of the trapezoid rule is of important practical use. An example
of multiple integration is pn~sent<~<l a.s illustrative cas<~.
Of presented :-;implc uwthod:-; it i:-; likely that Romberg integration is most effici<~nt.
Simpson's rules are <~kgant, lmt the first extrapolation of Romberg integration giw~s
comparable results. Sub:-;equent extrapolation of Romberg integration increase the order
at a very satisfactory rat<~. Simpson's rules coul<l he d<~velopecl into an extrapolation
proceclnre, hut with no a<lvant.age ow~r Romberg integration.
Many comm<~n:ial :-;oftwan~ packages contain solvers for numerical integration. Some
of tlH~ nHn·e pnnuiw~nt :-;y:-;tem:-; e-m~ Mat lab and Mathe ad. Mon~ sophi:-;ticated sy:-;t<~m:-;,
snch a:-; Mathematica, Macsyma (VAX UNIX MACSYMA, Refen~ncf~ Mmmal. Syml>ol-
ics Inc., Camhridg<~, MA), awl Maple (MAPLE V Library Refen~nce Manual, Sprin12;<~1·,
NY, 1991) also contain llllll1crical int<~gration solv<~rs.
Some organizations lmv<~ own package:-; - colkction of high-quality routiw~s, like
ACM (Collccte<l algoritlun:-;). IMSL (Houston, TX), NAG (Numerical Algorithms Group,
DowmT:-; Grove, IL), all< l :-;ouw fcmton:-; indivi<lual padmges are QUADPACK (R. Pies:-;<~11S,
d all.), QUADPACK, A Subr-outine Package for A'ntomat'ic Integration.
Springer. B<~din. 19<'\:J). CUBTRI (Cnlmtme Fonunlac Over Triangl<~), SSP (IBI\11 Nn-
merical Softwan~).
The hook Numer"ical Rec1:pes ([4], Chap. 4) contains several :-;nhrontine:-; for
integration of fmH·tion:-;. Som<~ al)..>,uritlnns, from which :-;ome an~ co<l<l<~d. an~ giwn iu
lJOok N'umer'ical Methods for Engineers and Scientists ([3], Chap. G).
On tlH~ <~nd. in onlcr to giv<~ :-;omc hint:-; for soft.wan~ own <lcvdopnwnt or usage of
:-;oftwan~ p<L.ckag<~s, w<~ will p;ivc•. st.awlanl t<~st <~xample:-; for tc:-;tini? or lwnchmarking.
 Lesson VIII - N nmcric:al Diffcn~utaiatiou aud Int(~~ratiou

Standard test examples (Incldinite integrals):

1° ;·sin :r d:r; 2° ;· )tan :t: rh:; :3° ;· .. :r: :I~ . rh; 5° ;· log :r rl:r::
rl:r; 4° ;· - _
. . . .1.
3
- 1 . sm :r: . JX+l
Go ;· .Jf+""X :r: ~ rl:r:; 7o ;· c-n:t:2 rl:r:; 8o ;· ~ rh:; go ;· siu :~: rl:J:;
. + 1 - :r . . log :1: . :t: 2

10 0 ;· 1 rl:r::
. 2 +cos :r: ·

Standard test cxmupl(~S (D(~finitc integrals):

1o ;·4rr
. 0 2 + cos :r
1 'o
d:r; 2
;·=
. _=
sin :J:
- - rl:r; 3
:r:
• o ;··XJ e-:r:
. 0
r;;: d:r:;
v :r
4
0
./ ()
•CXJ

1-
?
.'['~0-.'
' ,,
'I"

? ..
e.-~.~-
d:r:;

5° ;·=
. ()
e-:r;
2
log 2 :r d:r:; 6°
·1

./ -1 .J,
1
:-:2 d:r:;
·CXJ

/,
. 1
..
o-:r;,I,ll/3
.' rl·r
.....,

Bibliography (Cited references and further reading)

[1] Milova.noviC:, G.V., Numer·ical Analysis II, Nan(:na kl1jiga, Beograd, 1988 (Ser-
bian).
[2] MilovanoviC:, G.V. awl Djordjevi(:, Dj.R, Programiranje numerickih metoda
na FORTRAN jeziku. Institnt za cloktmwntaciju zastite na raclu "Edvanl
Karclelj" , Nis, 1981 (Scrhian).
[3] Hoffman, .J.D., Nurnerical Methods for Engineers and Scientists. Taylor
& Francis, Boca R.aton-Lond.ou-N(~W Ymk-Siugapore, 2001.
[4] Press, W.H., Flannery, B.P., Tcnkolsky, S.A., ancl Vetterling, W.T., Numerical
Recepies - The Art of Scientific Computing. Cambridge University Press,
1989.
[5] Kronrocl, A. S., Nodes and Weights of Quadrature Formulas. Consultants
Bureau, New York, 1~JGG.
[G] Clenshaw, C.W. and Curtis, A.R, A method for numerical integration on
an automatic computer Nnm. Math. 2(1~)60), pp. 197~205.
[7] Engels, H., Numerical Quadrature and Cubature. Academic Press, London,
1980. -
[8] Abramowitz, M., On the practical evaluation of integrals. SIAM .J. Appl.
Math. 2(1954)20-:35.
[9] Davis, P ..J., and P. Rabinowitz, P., Methods of Numerical Integration. Aca-
dmnic: Press, New York, 1975.
[10] Krylov,.V.I., Approximate Calculation of Integrals. MacMillan, New York,
1962 (Russian, tra.nsl. A.H. s-troud).
[11] Stroud, A.H., Approximat·ive Calculation of M·ultiple Integrals. PrelJtice-
Ha.ll, Englewood Cliffs, N.J. 1971.
[12] Mysovskih, I.P., Interpolyacionnye K ubaturnye Formuly. Nauka, Moskva,
1981.
[13] Stroud, A.H., Gaussian Quadrature Formulas. Prentice Hall, Englewood
Cliffs, N.J., 1966.
[14] Ralston,A., A First Co·urse in Numerical Analysis,
McGraw-Hill, New York, 19GG. ·
136 Numerical Methods in Comp\)-tational Engineering

[15) Hildebrand, F.B., Introduction to Numerical Analysis.

· Mc.Graw-Hill, New York, 1974.
[16) Acton, F.S., Numerical Methods That Work (corrected edition). Mathemat-
ical Association of America, Washington, D.C., 1990.
[17) Abramowitz, M., and Stegun, I.A., Handbook of Mathematical Functions.
National Bureau of Standards, Applied Mathematics Series, Washington,· 1964
(reprinted 1968 by Dover Publications, New York).
[18) Rice, .J.R., Numerical Methods, Software, and Analysis. McGraw-Hill, New
York, 1983.
[19) Forsythe, G.E., Malcolm, M.A., and Moler, C.B., Computer Methods for
Mathematical Computations. Englewood Cliffs, Prentice-Hall, N.J~ 1977.
[20) Kahaner, D., Moler, C., and Nash, S., 1989, Numerical Methods and Soft-
ware. Englewood Cliffs, Prentice Hall, N.J, 1989.
[21) Hamming, R.W., Numerical Methods for Engineers and Scientists. Dover,
New York, 1962 (reprinted 1986). · .
[22) Ferziger, .J.H., Numerical Methods for Engineering Applications. Stanford
University, .John Willey & Sons, Inc., New York, 1998.
[23) Pearson, C.E., Numerical Methods in Engineering and Science. University
of Washington, Van Nostrand Reinhold Company, New York, 1986.
[24) Stephenson, G. and Raclmore, P.M., Advanced Mathematical Methods for
Engineering and Science Students. Imperial College London, University Col-
lege, London Cambridge Univ. Press, 1999.
[25) Milovanovic, G.V. and Kovacevic, M.A., Zbirka resenih zadataka iz nu-
mericke analize. Nauc:na knjiga, Beograd, 1985. (Serbian).
[26) IMSL Math/Library Users Manual , IMSL Inc., 2500 City West Boulevard,
Houston TX 77042.
[27) NAG Fortran Library, Numerical Algorithms Group, 256 Banbury Road, Ox-
ford OX27DE, U.K.
[28) Piessens, R., de Doncker-Kapenga, E., Uberhuber, C.W., Kahaner, D.K., QUAD-
PACK, A Subroutine Package for Automatic Integration. Springer-
Verlag, Berlin, 1983.
[29) Laurie, D. P., Algorithm 584 CUBTRI: Automatic cubature over a tr'i-
angle. ACM Trans. Math. Software 8(1982), 210.-218.
[30) Stoer, .J., and Bulirsch . R., Introduction to Numerical Analysis. Springer-
Verlag, New York, 1980.
[31) .Johnson, L.W., and Riess, RD., Numerical Analysis, 2nd ed. Addison- Wesley,
Reading, MA, 1982.
[32) Ralston, A., and Rabinowitz, P., A First Course in Numerical Analysis,
2nd ed. McGraw-Hill, New York, 1978.
[33) Isaacson, E., ancl Keller, H.B., Analysis of Numerical Methods. Wiley, New
York, 1966.
Faculty of Civil Engiw~ering Fac:nlty of Civil Engineering and Architecture
Belgrade - Nis
Master Study Doctoral Stmly
COMPUTATIONAL ENGINEERING

LECTURES

LESSON IX

9. Ordinary Differential Equations

9.1. Introduction
Problems involving ordinary differential equations (ODEs) can always be reduced
to the set of first-order differential equations. For example the second order equation

(9.1.1)

can be rewritten as two first-order equations

dy
- - .'L
-Z ( ·)
(l:J;
(9.1.2)
rlz
- = T(:r.)- q(x)z(x),
(1,:J; .

where z is a new variable. This exemplifies the procedure for an arbitrary ODE. The
usual choice for the new variables is to let theni be just derivatives if each other, and, of
course, of original variable. Occasionally, it is useful to incorporate into their definition
some other factors in the equation, or some powers of the independent variable, for the
purpose of the mitigating singular behavior that could result in overflows or increased
roundoff error. Thus, involving new variables should be carefully chosen. The possibility
of a formal reduction of a differential equation system to an equivalent set of first-order
equations means that computer programs for the solution of differential equation sets
can be directed toward the general form

d:1h (:r)
(9.1.3) i
( :r:
= fi. (:c,
. - ( )
:U1, · · · , Yn) . i = 1, ... , n ,

where the .f,; functions are known and :Ul, y2, ... , Yn are dependent variables.
A problem involving ODEs is not completely specified by its equations. Even more
crucial in determining how to start solving problem numerically is the nature of the
problem's boundary conditions. Boundary conditions are algebraic conditions on the
values of the functions '.lh in (9.1.3). Generally, they can be satisfied at discrete specified
points, but do not hold between those points, i.e. are not preserved automatically by the
differential equations. Boundary conditions can be as simple as requiring that certain
variables have certain numerical values, or as complicated as a set of nonlinear algebraic
equations among the variables. Usually, it is the nature of the boundary conditions that
determines which nunH~rical methods will be applied. Boundary conditions divide into
two broad categories.

137
 1\1 umencallVletllbds m lJomputatwnal ~ngmeermg

• Initial value problems, where all th~ Yi are given at some starting value x 8 , and
it is desired to find the the Yi 's at some final point x f, or at some discrete list of
points (for example, to generate a table of results).
• Two-point boundary value problems, where the boundary conditions are spec-
ified at mote than one x. Usually some conditions are specified at X 8 and the
remainder at :rf. . .
In considering methods for numerical solution of Cauchy problem for differential
equations of first order, we will note two general classes of those methods:
a) Linear mul.ti-step methods,
b) Runge-Kutta methods.
The first class of methods has a property of linearity, in contrary to Runge-K utta
methods, where the increasing of method order is realized by involving nonlinearity.
The common "predecessor" of both classes is Euler's method, which belongs to both
classes.
In newer times there appeared a whole series of n1ethods, so known hybrid methods,
which use good characteristics of mentioned basic classes of methods ..

9.2. Euler's method

Euler's method is the simplest numerical method for solving Cauchy's problem

(9.2.1) y' = .f(:r, y), y(xa) =Yo

and is based on approximative equality

:y(:r:) = :y(xo) + (:1:- .To):u'(xo),

I.e.

(9.2.2) y(:r:) = y(.ro) + (:r:- xo)f(xo, Yo),

in regard to (9.2.1). If we denote with y 1 the approximate value for y(x1), based on
(9.2.2) we have
:Y1 =:Yo+ (x1- J:o).f(xo, Yo).
In general case, for arbitrary set of points :r: 0 < x 1 < x 2 < ... , the approximate values
for y(x.,), denoted as y.,, can be determined using

(9.2.3) :Yn+l = Yn + (:rn+l- Xn)f(xn, Yn) (n = 0, 1, · · .).

The last formula defines Euler's method, which geometric interpretation is given in the
Fig. 9.2.1.

y •

y(x)

y Q. r· - -
IX
.
I v
,
)I
1
\ 0 1 ""'0
I
I
I'
I
I
o' x,
<.
X
3 X

Figure ~.2.1
 Lessou IX - Ordinary Diff<~n=mtial Equations 139

Polygonal lim~ (io, :uo)- (:r1, Yl)- (:r2, :t/2)- ... is known as Euler's polygon.
The points :rn an~ usually chosen equidistantly, i.e. 1:n+l - :rn = h = canst.(>
0) (n = 0, 1, ... ) in which case (9.2.3) reduc<~s to

(n=0,1, ... ).

9.3. General linear multi-step method

In this and followiug s<~ctions a general method for solving Cauchy problem

(9.3.1) :1/ = .f (:r, y) , y (:r: o) = y n (:r 0 ::=; :r: ::=; b) .

will be c:onsid<~red. If we divide the segment [:r 0 , b] to N subsegments of length h

b -J:
N °,W<~ get a striug of points :1:n determined with

:rn =:Do+ n.h (n = 0, 1, ... , N).

Let Yn denote Hequenc<~ of approximate values of solutions of problem (9.3.1) in
points :rn and let fn =
f (:rn, y.11.). It is our task to determine a set Yn. In or elm· to
solve this problem a nmuber of methods have been developed. One of them is Euler's
method, which has been considered in previous section. At Euler's method series Yn is
computed recursively using

(9.3.2) (n = 0, 1, ... ),

whereby the linear relation among Yn, Yn+l and .f.n. exists. In general case, for evaluation
of series more complicated recunence relations than (9.3.2) can be used. Among the
methods originated from these relations, important role have the methods with linear
relation between Yn+i., fn.+i. ('i = 0, 1, ... k) and they form the class of linear multi-step
methods.
General linear multi-Htep method can be represented in form

A: A:
(9.3.3) L CV.-i.Yn+l = h L /Jfn.+i. (n = 0, 1, ... ),

where a and j] are constant coefficients determined by accuracy up to multiplicative

constant. In order to obtain their uniqueness we will take CYA: = 1.
If fJA: = 0, we say that method (9.3.3) is of open type or that is explicit; in coun-
terpart we say that it is of dosed type or implicit.
In general case (9.3.3) represents nonlinear difference equation, because of f.n+·i. =
.f(.Tn+i., Yn+'i.)· .
For determination of series Yn using method (9.3.3) it is necessary to know initial
values Yi ('i = 0, 1, ... , k- 1). Knowing in advance only value y 0 , a particular problem
in applic~ttion of multi-step methods (9.3.3) is determination of other initial values. A
special section will be devoted to this problem.
Supposing that initial values JJ.i. ('i = 0, 1, ... , l;;- 1) are known, at explicit methods
are directly comput~d ~Jh:, Yk+l, ... , :UN using
k~l A:-1
7/n+A: = hL fJdn+i - L (Y,i.Yn+i. (n = 0, 1, ... , N- k).
'i.=O

Nevertheless, at implicit methods for determination of values :Un+A: the equation

(9.3.4)
140 Numerical Methods in Computational Engineering

where
h:-1 h:-1

<P = h L fJifn+i - L a.iYn+i,

i=O

shell be solved. When (x, y) --+ .f (:r, y) is nonlinear function which satisfies Lipschitz
condition in y with constant L, the equation (9.3.4) can be solved by iterative process

, [s+1] _ l fJ j'( . [s] )

(9.3.5) Yn+k: - L k: .'l,n+h:' Yn+k: + ([)'
starting from arbitrary value Y.~~L: if

The condition given by this inequality ensures convergence of iterative process (9.3.5).
Let us for method (9.3.3) define difference operator Lh : C 1 [x 0 , b]--+ C[x 0 , b] by

h:
(9.3.6) Lh[y] = .L)a..iy(x + ih) - hf3iY 1 (x + ih)].
·i=O

Let function g E C 00 [:r 0 , b]. Then Lh[g] can be presented in form

(9.3.7)

where Cj (j = 0, 1, ... ) are constants not depending on h and g.

Definition 9.3.1. Linear multi-step method (9.3.3) is of order p if in development
(9.3.7)

Let .T --+ y(x) be exact solution of problem (9.3.1) and Yn series of approximate
values of this solution in points :rn = :r: 0 + nh (n = 0, 1, ... , N) obtained by method
(9.3.3), with initial values Yi = s.i.(h) ('i = 0, 1, ... , k- 1).
Definition 9.3.2. For linen.r multi-step method (9.3.3) one says to be convergent if for
every :r; E [:.co, b]
lim Yn = y(x)
:r:-+0
,,·-.r:o=nh

and for initial values hold

lim s.; (h) =Yo (i = 0, 1, ... , k- 1).

h-+0

Linear multi-step method (9.3.3) can be characterized by first and second charac-
teristic polynomials given by

k: h:

p(~) = I:nie and a-(~) = L f3i~i,

·i=O i=O

respectively.
Two important. classes of convergent multi-step methods, which are met in practice
are:
1. Methods at which p(O = e-
e:-\
2. Methods at which p(E) = e'- (":- 2 .
 Lesson IX - Ordinary Differential Equations 141

Explicit methocls of first class are called Adam-Bashfort.h methods, and the im-
plicit Adam-Moulton methods. Similarly, explicit methods of second class are called
Nystrom's methods and corresponding implicit methods Milrw-Simpson's.
Of course, there are methods that do not belong to neither of these classes.

9.4. Choice of initial values

As earlier mentioned, at application oflinear multi-step methods on solving problem

(9.3.1), it is necessary knowledge on initial values :Ui = si(h), such that

lim s.i(h) = :uo (i = 1, ... , k- 1).

h-tO

Certainly, this problem is stated when k > 1.

If method (9.3.3) is of order p, then initial values 8 7; (h) are obviously to be chosen
such that
s.i(h)- :y(:r.i) = O(hP+ 1 ) ('i = 1, ... , k- 1),
where :r---+ y(:r) is exact solution of problem (9.3.1).
In this section we will show one class of methods for determination of necessary
initial values.
Suppose that function f in differential equation (9.3.1) is enough times differen-
tiable. Than, based on Tailor's method we have

y(.ro +h)= y(:r:o) + hy'(:ro) + 12~~ y"(:r:o) + ... + h~ y(Pl(.ro) + O(hP+l).

0 p.

Last equation points out that it can be taken

72 hP
S-i (h) = y(:r:o) + hy' (:co) + 27,' y" (:ro)
0
+ ... + -,
• p.
y(P) (xo)'

because of 8-i.(h)- y(x: 1 ) = O(hP+ 1 ) (:r: 1 = x 0 +h). The same procedure can be applied
to determination of other initial values. Namely, in general case, we have

whereby for y(:r.i-l) we ta.ke s.i_ 1 (h).

9.5. Predictor-corrector methods

As mentioned in section 9.3., application of implicit methods is in connection with

solution of equation (9.3.4) in every integration step, whereby for this solution is used
iterative process (9.3.5). Regardless to this difficulty in implicit method, they are of-
ten used for solving Cauchy problem, because they have a number of advantages over
explicit methods (higher order, better numerical stability). The initial value y~~L: is
determined in practice using some explicit method, which. is then called predictor. Im~
plicit method (9.3.4) is then called corrector. Method obtained by such combination is
called predictor-corrector method.
For deterl:nination Yn+k:, the iterative procedure (9.3.5) should be applied until
fulfilment of the condition
[s+l] [s] I
IYn+k: - Yr1.+k: < E,

where E tolerable error, usually of order of local round-off error. Then for Yn+k: can be
taken y~::k~l.
142 Numerical Methods in Computational Engineering

Nevertheless, this method is usually not applied in practice, due to demanding large
number of function f evaluations by step of calculation and, in adaition, this number
is varying from step to step. In order to reduce this number of calculations, number of
iterations in (9.3.5) is fixed. Thus, one takes orily s = 0, 1, ... , m- 1.

9.6. Program realization of multi-step methods

In this section we will give program realization of explicit as well as implicit meth-
ods. The presented programs are tested on the example (with h = 0.1).

:y
1
= :r: 2 + :t}, y(1) = 1 (1 S X S 2).

The exact solution of this problem is y(:r) = 6e~J:-l.., x 2 - 2x- 2.

9.6.1. Euler's method

Euler's method is given by expression

'I'}
. n+l -'I'}
. n-- hf'. n (n = 0, 1, ... ),

of order p = 1, and Adams-Ba,shforth method of third order

Yn+3 - Yn+2 = h(?3f'

~ . n+2 - 1':t•
0. n+l + 0.
r:f')
n (n = 0, 1, ... ),
12
are realized by subroutines EULER i ADAMS, respectively.

SUBROUTINE EULER (XP,XK,H,Y,FUN)

DIMENSION Y(1)
N=(XK-XP+0.00001)/H
X=XP
DO 11 I=1,N
Y(I+1)=Y(I)+H*FUN(X,Y(I))
11 X=X+H
RETURN
END
c
FUNCTION FUN (X,Y)
FUN=X*X+Y
RETURN
END
c
SUBROUTINE ADAMS (XP, XK, H, Y, FUN)
DIMENSION Y(1)
N=(XK-XP+0.00001)/H
X=XP
FO=FUN (X,Y(1))
F1=FUN (X+H,Y(2))
N2=N-2
DO 11 I=1,N2
F2=FUN(X+2.*H,Y(I+2))
Y(I+3)=Y(I+2)+H*(23.*F2-16.*F1+5.*F0)/12.
FO=F1
F1=F2
11 X=X+H
RETURN
END
 Lesson IX - Ordinary Differential Equations 143

Parameter::; i11 ll::;t of subroutine parameters are of following meaning:

XP and XK - start a11d end point of integration interval;
H - ::;tep of integration;
Y - vector of approximate value::; of solution obtained by multi-step method, where
at Euler's method Y(1) represents given initial value, and at Adam's method initial
values are giveri by Y(1), Y(2) and Y(3);
FUN - name of function subroutine which defines right hand size of differential
equation .f(x, :y). Initial values for Adam's method we determine by using Taylor's
method for p = 3 (see section 9.3.4). Namely, beiug

y(1) = 1, :u'(l) = 2, :u"(l) = 4, :u"'(l) = 6, h = 0.1,

we get Y(1)=1., Y(2)=1.221, Y(3)=1.48836.
Main program and output listing are of form:

c
C==================================================
C RESAVANJE DIFERENCIJALNIH JEDNACINA
C EKSPLICITNIM METODIMA
C==================================================
EXTERNAL FUN
DIMENSION Y(100),Z(100)
F(X)=6.*EXP(X-1.)-X*X-2.*X-2.
OPEN(5,FILE='EULER.OUT')
WRITE (5,10)
10 FORMAT(3X,'RESAVANJE DIFERENCIJAL.JED. ',
1'EKSPLICITNIM METODIMA'//8X,'XN',8X,'YN(I)',
15X,'GRESKA(%)' ,3X,'YN(II)',4X,'GRESKA (%)'/)
XP=1.
XK=2.
H=0.1
Y(1)=1.
CALL EULER (XP,XK,H,Y,FUN)
Z(1)=Y(1)
Z(2)=1.221
z (3) =1. 48836
CALL ADAMS (XP,XK,a,Z,FUN)
N=(XK-XP+0.00001)/H
NN=N+1
X=XP
DO 22 I=1,NN
G1=ABS((Y(I)-F(X))/F(X))*100.
G2=ABS((Z(I)-F(X))/F(X))*100.
WRITE (5,20)X,Y(I),G1,Z(I),G2
22 X=X+H
20 FORMAT (8X,F3.1,2(4X,F9.5,4X,F5.2))
CLOSE(5)
·STOP
END
RESAVANJE DIFERENCIJAL.JED.EKSPLICITNIM METODIMA
· XN . YN(I) GRESKA(%) YN(II) GRESKA (%)
1. 0 1. 00000 . 00 1. 00000 . 00
1.1 1.20000 1.72 1.22100 .00
1.2 1.44100 3.19 1.48836 .00
1.3 1.72910 4.42 1.80883 .02
1.4 2.07101 5.47 2.19028 .03
1.5 2.47411 6.37 . 2.64126 .04
144 Nmn<~rical Md.hods in Computational Engineering

1.6 2.94652 7.13 3.17116 .05

1.7 3.49717 7.79 3.79040 .06
1.8 4.13589 8.36 4.51045 .06
1.9 4.87348 8.87 5.34403 .07
2.0 5.72183 9.32 6.30518 .07
9.6.2. Taking Eul<~r's metlw<l as predictor and trapezoid rule (p = 2)

h . . (
Yn+l- Yn = 2Un + fn+d n = 0, 1, ... ),

as conector (with nnmh<~r of iterations n1. = 2) the subroutine PREKOR is written. Main
progra.ru, subprogram, aU< l output results are of fmm:

C===================================================
C RESAVANJE DIF.JED. METODOM PREDIKTOR-KOREKTOR
C===================================================
EXTERNAL FUN
DIMENSION Y(100)
F(X)=6.*EXP(X-1.)-X*X-2.*X-2.
OPEN(5,FILE='PREKOR.OUT')
OPEN(8,FILE='PREKOR.TXT')
WRITE(5,10)
10 FORMAT(8X, 'RESAVANJE DIF. JED. METODOM',
1' PREDIKTOR-KOREKTOR'//15X,'XN' ,13X, 'YN'
2,10X, 'GRESKA(%)'/)
READ(8,5)XP,XK,YP,H
5 FORMAT(4F6.1)
CALL PREKOR(XP,XK,YP,H,Y,FUN)
N=(XK-XP+0.00001)/H
NN=N+1
X=XP
DO 11 I=1,NN
G=ABS((Y(I)-F(X))/F(X))*100.
WRITE(5,15)X,Y(I),G
15 FORMAT(15X,F3.1,8X,F9.5,8X,F5.2)
11 X=X+H
STOP
END
c
c
SUBROUTINE PREKOR(XP,XK,YP,H,Y,FUN)
DIMENSION Y(100)
N=(XK-XP+0.00001)/H
X=XP
Y(1)=YP
DO 10 I=1,N
C PROGNOZIRANJE VREDNOSTI
FXY=FUN (X, Y(I))
YP=Y (I) +H*FXY
C KOREKCIJA VREDNOSTI
DO 20 M=1,2
20 YP=Y(I)+H/2.*(FXY+FUN(X+H,YP))
Y(I+1) =YP
10 X=X+H
RETURN
END
c
 L<~ssm1 IX - Ordinary Diff<Tent.ial Equations 145

c
FUNCTION FUN(X,Y)
FUN=X*X+Y
RETURN
END
RESAVANJE DIF. JED. METOD OM PREDIKTOR-KOREKTOR
XN YN GRESKA(%)
1.0 1.00000 .00
1.1 1. 22152 .04
1.2 1.48952 .07
1.3 1.81097 .10
1.4 2.19363 .12
1.5 2.64602 .14
1.6 3.17760 .15
1.7 3.79881 .17
1.8 4.52118 .18
1.9 5.35747 .18
2.0 6.32177 .19

9.7. Runge-Kutta methods

Iu previous sections are cousickred lim~ar multi-step methods for solving Cauchy
problem (9.3.1). Th<~ md<~r of these methods can be enlarged by increasing nmnlwr
of steps. Nevertheless, by sacrifiu~ of linearity these methoclH posses, it is possible to
construct single-Htcp methods of arbitrary order.
For solving Cauchy problem of form (9.3.1) with enough times differentiable func-
tion f, it is possibl<-~ to coustruc:t single-step methods of higher order (e.g. Taylor's
method).
Consider general explicit sing-lc-st<-~p method

(9.7.1)

Definition 9. 7 .1. Method ( 9.7 .1) is of order p if p is greatest intege1· for which holds

y(:r +h)- y(:r)- h<.I>(:r, y(x:), h)= O(hP+ 1 ),

where :z:--+ :y(:r) is exact solution ofprol>len1 (9.3.1).

Definition 9. 7.2. Jvietlwd (!J.7.1) is consistent if <D(:r, :y, 0) = f(:r, :y).
Note that Taylor's method is special case of method (9.7.1). Namely, at Taylor's
method of order ]J wt~ have

]J-l 1, a D
(9.7.2) <D(:r,y,h) = <Dr(:r,y,h) = E ('i ~ l)!(o:r: + f oyr'f(:r:,y).
i,=O

In special case, at Eulerov's method is <D(:r:,y,h) = f(:r:,y).

In this section we will consider a special class of metho<ls of form (9.7.1), which
was proposed in 1895. year by C. Runge. Later on, this class of methods was developed
by W. Kutta i K. Henn.
As we will see later, all these methods c:ont~tin free parameters. Considering time iu
which these niethods <lppearecl, the free parameters have been chosen in such a way to
obtain as simple as possible formulas for practical calcnlation. Nevertheless, such values
of pal'ameters do not ensure optimal characteristics of observed methods. In ftuther text
these methods will be called classical. General explicit Runge-Kutta method is of form

(9.7.3)
146 Numerical Methods in Computational Engineering

where
m.
<D(x, y, h)= L c.;,ki,
·i~1

k1 = f(:r:,:y),
k.;, = f(.r; +a.;,, y + b.;,h) ('i = 2, ... , m).

·i-1 ·i-1
(9.7.4) a.;,= L CJ.ij, b.;,= I: aijk_i·
.i=1 .i=1

Note that from the condition of consistence of method (9.7.3) it follows

m.
Lc.;, = 1.
·i=1

Unknown coefficients which appear in this method are to be determined from the con-
dition that method has a maximal order. Here, we use the following fact: If <D(:r,, y, h),
developed by degrees of h, can be presented in form

<D(.r;, y, h)= <Dr(:r:, y, h)= O(hP),

where <Dr is defined by (9.7.2), then method (9.7.3) is of order p.

Find previously development <Dr(:r, y, h) by degrees of h. Using Monge's notations
for partial derivative, we have

( _!!__
cl +D
i_) .f' --.:z:+
t· ..t·t·v-
- F.
u:r: y
and

,~ + f uy 2 .f = ( ux
( ux ~ + f uy
!? ) ~ )F = G + fu F,
where we put G = f:c.'I: + 2.fJr:y + .f 2 f 1111 • Then from (9.7.2) it follows

(9.7.5)

Consider now only Runge-Kutta methods of order p :S 3. One shows that for obtaining
method of third order it is enough to take rn = 3. In this case, formulas (9.7.3) re<luce
to

<D(:r, y, h) = c1k1 + c2h:2 + c3k3

k1 = .f(:c, y)
h:2 = f(:r, + a2h, V + b2h),
/:.:3 = .f(:r + a3h, y + b3h)
and
o.2 = a21, b2 = a21k1,
(1.3 = a.11 + CY32, h3 = CY31h:1 + CY32k2.
 Lesson IX - Ordinary Diff'erential Equations 147

By developing of fmiction k2 in Taylor's series in neighborhood of point (:1:, :y), w<~ get

Because of

we have

and
h~ = o.V· 2 + O(h).
By developing of function /;:3 in neighborhood of point (x, :y) and by using last equalities
we have

Finally, by substitnting the obtained expressions for /;: 1 , k 2 , k3 in expression for <D(:r, y, h)
we get
<I>(:r, y, h) =(cl + c2 + c3).f + (c2a2 + c3a3)Fh
2
+ (c2o. 22G + 2c3a2n32F.fu. + c3a 32G ) 2h + 0 (h. 3) .
Last equality enables construction of methods for m. = 1, 2, 3.

Case m=l. Being c2 = c3 = 0, we have

By comparison with (9.7.G) we g<~t

wherefrom we conclude that for c 1 = 1 the method

of order p = 1 is obtained. Consi<lering that it is Euler's method, we see that it belongs

to the class of Rungc-Kutta methods too.

Case :rp=2. Here is c3 = 0 and

Because of
1
<I>(:r, :y, h)- <Dr(:J:, y, h)= (c1 + c2- 1).f + (c2a.2- 2)Fh
+ ~[(3c2a.~- l)G- f:yF]h? + O(h 3),
148 Numerical Methods in Computational Engineering

we conclude that under condition

(9.7.6) and

one obtainH method of second order with one free parameter. Namely, from t>ysteni of
equations (9. 7.6) it follows

2a2- 1
and c1 = - - -
2a2

where a 2 ( # 0) is fi:'c~e parameter. Thus, with 'In = 2 we have single-parametric family of

methods · ·
h
Yn+l- Yn = - ···· ((2a2- 1)11:1 + k2),
2(1,2
f.:1·= f(:E.n.,:Yn),
k2 = .f(:En + a2h, :Un + a2k1h).
In special cas<\ for a 2 = ~, W<~ get Eulcr-Gauchy method

Similarly, for a 2 = 1, w<~ get so kuown impnlV<)d Euler-Cauchy method

On geometric interpretntion of olJtailH~d methods Hee, e.g. [6 ].

Case m=3. Acconling to

we condwk that for obtaining of uwtho<ls of third onlcr the satisfactory ccmditious an~

c1 + c 2 + CJ = 1,
1
(;2(}·2 + C:30•.'3 = 2'
(~.7.7) 2 1
(;2(1,,)
-
+ CJ0•3'2 = -:-'
:3
1
(;3(/,2{\'3? = -
' - 6
0

Having fonr <~qnations with six mJknowm;, it follows that, in case m. = 3. we have two-
parametric family of Rm112;<~- K nt.t.a md.hods. Om~ can show that among nwthods of this
family does not Pxist:-; not singl<) md.lw<l with ordPr great<~r than~ three.
 Lesson IX - Ordinary Diffc-~n~ut.ial Equations 149

In special case, when n.2 = ~ aucl a 3 = ~, from (9. 7. 7) it follows c1 = i, c2 = 0, c3 =

~' a.32 = l Thus, we obtained the~ method
h
:Un+l - :Un = 4(k1 + 3k3),

k1 = f(.r.n, Y.n.),
h h )
1.
n:2 =
"(
.f :r:n + J':Un + 3k1'
2h 2h
k3 = f (.T.n + 3' :Un + 3k2),
which is known in bibliography as Helm's method.
For a2 = ~' a3 = 1(=* c1 = c3 = fi,
c2 = ~' a.32 = 2) we get the method

Yn+l - Yn = h( k1 +
6 4k2 + k3),
k1 = f(xn, :Un),

k2 = .f"( :r:n h ;t/n + 2k1

+ 2' h ),

k3 = .f(:Dn + h, :Un- hk1 + 2hk2),

which is most popular among the methods of third order from the point of view of hand
calculations.
In case when rn = 4, we get two-parameter family of methods of fourth order.
Namely, here, analogously to system (9.7.7), appears system of 11 equations in 13
uukuowns.
Now we quote~, without proof, Ruuge-Kutta method of fourth order.

Y.n+l- Yn = h( k1 + 2k2 +
6 2k3 + k4),
k1 = .f(:r:n, Yn),,
h h
(9~ 7.8) k2 = f(:rn + 2' :Yn + 2,k1),
h h
k3 =J(:rn + 2'Yn + 2k2),
k4 = .f(:I:n + h, Yn + hk3),
which is traditionally most used in applications.
From methods of fourth order. it is often used so known Gill's variant, which can
be expressed as the following recursive procedure:
n := 0, Q 0 := 0
(*) Yo := :Un ,
1
k1 := h.f(:r:n, Yo), Y1 :=Yo+ 2(k1- 2Qo),
3 1
Q1 := Qo + -(k1- 2Qo)- -k1,
2 2

k2 := hf(:r:n + ~' Y1), Y2 := Y1 + (1- Jl72)(k2- Q1),

Q2 Ql + 3(1- .Jl72)(k2- Ql)(1- Jl72)k2,

:=
. h .
k3 := hf(:r:n + 2' Y2), Y3 := Y2 + (1 + Jl72)(k3- Q2),

Ch := Ch + 3(1 + fi72)(k3 ~ Q2)- (1 + fi72)k3,

150 Numerical Methods in Computational E!lgineering

Yn+l := Y4,
n := n +1
skip to(*).

In contrary to linear multi-step methods, Runge-Kutta methods do ~ot demand

knowledge of initial values (except y(:r 0 ) = y0 , what, by the way, defines Cauchy prob-
lem), but for practical application are more complicated, because they demand rn cal-
culations of function .f values in every step.

9.8. Program realization of Runge-Kutta methods

In this section we presPnt program realization of Euler-Cauchy method, improved

Eueler-Cauchy method, a,s well as method of fourth order (9.1.8) and Gill's variant of
Runge-Kutta method. The obtained softwme will he tested on the example from section
9.3.6.

Program 9.8.1.
By subroutine EULCAU are realized Euler-Cauchy and improved Euler-Cauchy
method. Parameters in parameter list have the following meaning:
XP - start point of integration interval;
H - integration step;
N - integer, such that N + 1 is lenght of vector Y;
M - integer which ddines way of construction of vector Y. Namely, in vector Y is
stored in turn every M-th value of solution obtained during integration process.
Y- vector containing solutions of length N+1, whereby Y ( 1) is given initial condition
y 0 , Y(2) is value of solution obtained by integration in point XP + M*H, etc.
FUN - name of function subroutine, which defines right-hand side of differential
equation .f (:r, y);
K- integer with values K=1 awl K=2, which governs integration a,cc:onling to Euler-
Cauchy and improved Euler-Cauchy method, respectively.
Subroutine EULCAU is of form:

SUBROUTINE EULCAU(XP,H,N,M,Y,FUN,K)
DIMENSION Y (1)
X=XP
Y1=Y(1)
NN=N+1
DO 10 I=2,NN
DO 20 J=1,M
YO=Y1
Y1=FUN(X,YO)
GO TO (1 , 2) , K
1 Y1=YO+H*FUN(X+0.5*H,Y0+0.5*H*Y1)
GO TO 20
2 Y1=YO+H*(Y1+FUN(X+H,YO+H*Y1))/2.
20 X=X+H
10 Y(I)=Y1
RETURN
END
c
FUNCTION FUN(X,Y)
 Lesson IX - On linary Differential Equations lSI

FUN=X*,X+Y
RETURN
END
Main program and outpnt listing are given in further text. As input parameters for
integration w<-~ hav<~ taken H=O .1, N=10, M=1, and in second case H=O. 05, N=10,
M=2. Columns Y1N and Y2N in ontput listing give values for solution of given Cauchy
problem, according to regular and improved Euler-Cauchy method, respectively. Iu
additiou to those columns, in output listing are given columns with corresponding enors
(as relation to exact. solution, expressed in <7<1)
C===================================================
C RESAVANJE DIF. JED. EULER-CAUCHYEVIM
C I POBOLJSANIM METODOM
C===================================================
EXTERNAL FUN
DIMENSION Y(100), 2(100)
F(X)=6.*EXP(X-1.)-X*X-2.*X-2.
OPEN(5,FILE='EULCAU.OUT')
OPEN(8,FILE='EULCAU.IN')
WRITE(5,10)
10 FORMAT(10X,'RESAVANJE DIF.JED.EULER-CAUCHYEVIM'
1 ' I POBOLJSANIM METODOM')
20 READ(8,25,END=99)XP,Y(1),H,N,M
25 FORMAT(3F6.1,2I3)
CALL EULCAU(XP,H,N,M,Y,FUN,1)
Z(1)=Y(1)
CALL EULCAU(XP,H,N,M,Z,FUN,2)
WRITE(5,30)H
30 FORMAT(1H0,30X,'(H=',F6.4,')'//15X,'XN',8X,
1'Y1N',4X,'GRESKA(%)',5X,'Y2N',4X,'GRESKA(%)'/)
NN=N+1
X=XP
DO 11 I=1,NN
G1=ABS((Y(I)-F(X))/F(X))*100.
G2=ABS((Z(I)-F(X))/F(X))*100.
WRITE(5,15)X,Y(I) ,G1,Z(I),G2
15 FORMAT(15X,F3.1,3X,F9.6,2X,F7.5,3X,F9.6,2X,
1 F7.5)
11 X=X+H*M
GO TO 20
99 CLOSE(5)
CLOSE(8)
STOP
END

RESAVANJE. DIF.JED.EULER-CAUtHYEVIM I POBOLJSANIM METODOM

(H= .1000)
XN Y1N GRESKA(%) Y2N GRESKA(%)
1.0 1.000000 .00000 1.000000 .00000
1.1 1.220250 .06352 1.220500 ,04304
1.2 1.486676 .11693 1.487203 .08157
1.3 1.806227 .16173 1.807059 .11576
1.4 .2.186581 .19934 2.187750 .14599
. 1. 5 2.636222 .23109 2.637764 .17274
1.6 3.164526 .25808 3.166479 .19650
1.7 3.781851. -. 28125 3.784260 .21773
1.8 4.499645 .30138 4.502557 .23685
1.9 5.330558 .31907 5.334026 .25422
2.0 6.288567 .33483 6.292649 .27013
152 Numerical Methods in Computational Engineering

(H= .0500)
XN Y1N GRESKA(%) Y2N GRESKA(%)
1.0 1.000000 .00000 1.000000 .00000
1.1 1.220824 .01655 1.220888 .01130
1.2 1.487963 .03046 1.488098 .02140
1.3 1.808391 .04213 1.808604 .03034
1.4 2.189811 .05192 2.190111 .03824
1.5 2.640738 .06019 2.641133 .04523
1.6 3.170581 .06721 3.171082 .05143
1.7 3.789740 .07324 3.. 790357 .05696
1.8 4.509705 .07848 4.510451 .06195
1.9 5.343177 .08309 5.344066 .06647
2.0 6.304192 .08719 6.305238 .07061

Program 9.8.2.
According to fonnulas (9.7.8) for standard Runge-Kutta method of fourth degree,
the following subroutine RK4 is written:

SUBROUTINE RK4(XO,YO,H,M,N,YVEK,F)
C=============================================
C METOD RUNGE-KUTTA CETVRTOG REDA
C=============================================
DIMENSION YVEK(1)
T=H/2.
X=XO
Y=YO
DO 20 I=1,N
DO 10 J=1,M
A=F(X,Y)
B=F(X+T,Y+T*A)
C=F(X+T,Y+T*B)
D=F(X+H,Y+H*C)
X=X+H
10 Y=Y+H/6.*(A+2.*B+2.*C+D)
20 YVEK (I) =Y
RETURN
END

Parameters in list of snlmmtin<~ parameters are of following meaning:

XO, YO- <ldinc giv<~n initial condition (YO=y(XO)); .
H - step of integration:
M, N - integers with nwanings similar to ones in subroutine EULCAU;
YVEK - vector of length N which is ohtain<~<l as result of numerical integration,
whereby Y(1) is valnP olJtainc<l in point XO+M*H, Y(2) value in point X0+2M*H, <~tc.
F- name of fnn<:tion snhrontim~ which <ldin<~s right-hand side of differential equation
.f (:I:' y).
Main program is of form:

C==========================================
C RESAVANJE DIF.JED. METODOM RUGE-KUTTA
C==========================================
EXTERNAL FUN
DIMENSION Y (100)
F(X)=6.*EXP(X-1.)-X*X-2.*X-2.
OPEN(5,FILE='RK4.0UT')
 Lc~ssou IX - Onlinary Diff<'n~utial Eqnat.ious

OPEN(8;FILE='RK4.IN')
WRITE(5,10)
10 FORMAT (14X, 'RESAVANJE DIF.JED. METODOM',
1 ' RUNGE-KUTTA')
20 READ (8,5,END=99)XO,YO,H,N,M
5 FORMAT (3F6.1,2I3)
CALL RK4(XO,YO,H,M,N,Y,FUN)
G=O.
WRITE (5,25) H,XO,YO,G
25 FORMAT( 28X,'(H=' ,F6.4,') '//15X, 'XN',13X, 'YN',
110X,'GRESKA(%) '//15X,F3.1,8X,F9.6,7X,F7.5)
X=XO
DO 11 I=1,N
X=X+H*M
G=ABS((Y(I)-F(X))/F(X))*100.
11 WRITE (5,15)X,Y(I),G
15 FORMAT (15X,F3.1,8X,F9.6,7X,F7.5)
GO TO 20
99 CLOSE(5)
CLOSE(8)
STOP
END
c
FUNCTION FUN(X,Y)
FUN=X*X+Y
RETURN
END
Taking H=O. 1, N=10, M=1 tlw following; n~imlts are obtailH~d:

RESAVANJE DIF.JED. METODOM RUNG-KUTTA

(H= .1000)
XN YN GRESKA(%)
1.0 1.000000 .00000
1.1 1.221025 .00002
1.2 1.488416 .00005
1.3 1.809152 .00007
1.4 2.190946 .00009
1.5 2.642325 .00011
1.6 3.172709 .00012
1.7 3.792512 .00014
1.8 4.513240 .00015
1.9 5.347611 .00017
2.0 6.309682 .00018
Program 9.8.3.
The Gill's variant of Rm1ge-Kntta uH:tlwd is realized in double precision. Param-
eters in par;-imctcr list of snbrontiue GILL, XO, H, N, M, Y, FUN have the same
meaning as the parameters HP, H, N, M, Y, FUN in subroutine EULCAU, n~spcc­
tively. Note that this snbrontim: is realiz<:d in such a way that the optimiza.tiou of
number of variables has hc<:n p<-:rforuwcl. ·
Input panmH:ters arc taken like in program 9. 8 .1.

C~=================================================
C RESAVANJE DIF.JED. METODOM RUNGE~KUTTA
C (GILLOVA VARIJANTA)
C============================================~=====
154 Numerical Methods in Computational Engineering

EXTERNAL FUN
REAL*B Y(100),F,FUN,XO,X,H,G
F(X)=6.*DEXP(X-1.)-X*X-2.*X-2.
OPEN(8,FILE= 1 GILL.IN 1 )
OPEN(5,FILE= 1 GILL.OUT 1 )
WRITE(5,10)
10 FORMAT(8X, 1 RESAVANJE DIF.JED.METODOM 1
1 RUNGE-KUTTA (GILLOVA VARIJANTA)
I I ) .

20 READ(8,25,END=99)X,Y(1),H,N,M
25 FORMAT(3F6.1,2I3)
XO=X
CALL GILL(XO,H,N,M,Y,FUN)
WRITE(5,30)H
30 FORMAT(/28X, 1 (H= 1 ,F6.4, 1 ) 1 //15X, 1 XN 1 ,13X, 1 YN 1 ,

1 10X, 1 GRESKA(%) 1 / )
NN=N+1
DO 11 I=1,NN
G=DABS((Y(I)-F(X))/F(X))*100.
WRITE(5,15)X,Y(I),G
15 FORMAT(15X,F3.1,8X,F9.6,6X,D10.3)
11 X=X+H*M
GO TO 20
99 CLOSE(5)
CLOSE(8)
STOP
END
c
c-
SUBROUTINE GILL(XO;H,N,M,Y,FUN)
REAL*B Y(1),H,FUN,XO,YO,Q,K,A,B
B=DSQRT(0.5DO)
Q=O.DO
YO=Y(1)
NN=N+1
DO 10 I=2,NN
DO 20 J=1,M
K=H*FUN(XO,YO)
A=0.5*(K-2.*Q)
YO=YO+A
Q=Q+3.*A-0.5*K
K=H*FUN(XO+H/2. ,YO)
A=(1.-B)*(K-Q)
YO=YO+A
Q=Q+3.*A-(1.-B)*K
K=H*FUN(XO+H/2,YO)
A=(1.+B)*(K-Q)
YO=YO+A
Q=Q+3.*A-(1.+B)*K
K=H*FUN(XO+H,YO)
A=(K-2.*0)/6.
YO=YO+A
Q=Q+3.*A-K/2.
20 XO=XO+H
10 Y(I)=YO
RETURN
END -
c
FUNCTION FUN(X,Y)
REAL*B FUN,X,Y
 Lesson IX - Ordinary Differential Equations 155

FUN=X*X+Y
RETURN
END

RESAVANJE DIF.JED.METODOM RUNGE-KUTTA (GILLOVA VARIJANTA)

(H= .1000)
XN YN GRESKA(%)
1.0 1.000000 .OOOD+OO
1.1 1.221025 .2460-04
1.2 1.488416 .4600-04
1.3 1.809152 .6470-04
1.4 2.190946 .8080-04
1.5 2.642325 .9490-04
1.6 3.172709 .1070-03
1.7 3.792512 .1180-03
1.8 4.513240 .1280-03
1.9 5.347611 .1360-03
2.0 6.309682 .1440-03
(H= .0500)
XN YN GRESKA(%)
1.0 1.000000 .OOOD+OO
1.1 1.221025 .1620-05
1.2 1.488417 .3030-05
1.3 1.809153 .4250-05
1.4 2.190948 .5310-05
1.5 2.642327 .6230-05
1.6 3.172713 .7040-05
1.7 3.792516 .7750-05
1.8 4.513245 .8380-05
1.9 5.347618 .8940-05
2.0 6.309690 .9460-05

9.9. Solution of system of equations and equations of higher order

Methods considered in pn~vious sections can he generalized in that sense to be

applicable in solution of Cauchy problem for system of p equations of first order

(9.9.1) y;, = f;(:r:; :U1, · · ·, Yp), y.i(:r:o) = Y·iO ('i = 1, ... ,p).

In this case, system of equations (9.9.1) shell be represented in vector form

(9.9.2) :t7' = ,{(:r, Y), :l}(:t:o) =Yo,

I
where

y= Yl1 [ YlO
Y2 __, Y2o
. , Yo= . ,
[
~P . y~JO
It is of our interest the solution of Cauchy problem for differential equations of higher
order. Note, nevertheless, that this problem can be reduced to previous one. Namely,
let be given the differential equation of order p

(9.9.3) y(P) = f(:r:, y, y'' ... 'y(p-1))

 156 Numerical Methods in Computational Engineering

with initial conditions

. (9.9.4) Y(i)(:r:o) = Yio (i = 0,1, ... ,p- 1) .

Then, by substitution
Z1 = y,
_
Zz - y I , ... , Zp _,
- y (p-1) ,
equation (9.9.3) with conditions (9.9.4), reduces to system
I
z1 = zz
I
Zz = Z3

I
zp-1 = z,in
Z~ 1 = f(x; z1, zz, ... , zp),

with conditions z.i(:Eo) = z.io = Yio (1:= 1, ... ,p).

Linear multi-step methods considered up to now, can be formally generalized to
vector form
k: k

Laiffn+i = h LPd:+i,
·i=O

where .t:.+i = .{(:J;n+·i., Yn+·i.), and then as such can be applied to solution of Cauchy
___ problem (9.9.2).
· ~ " Also, the Runge-Kutta methods for solution of Cauchy problem (9.9.2) are of form

where rn

k1 = .((:r, m,
k.;. = .((:r: + o,)l., if+ b.;.h)
1:-1 ·i-1
a.i = L (Y.i,j, b.;.= L (Y.;,jkj (1: = 2, ... , m).
j=l .i=1

All analysis given in previous sections can formally be translated to noted vector
methods.
As an example, realize standard Runge-Kutta method of forth order (9.7.8) for
solving of system of two differential equations

:t/ = h (:r, y, z), 1

Z = h (:1:; y, z),
..•
'•··
with conditions y(:r 0 ) = Yo a11<l z(:ro) = zo.
.~ ~

The corresponding subroutiw~ is of form:

SUBROUTINE RKS(XP,XKRAJ,YP,ZP,H,N,YY,ZZ)
REAL KY1,KY2,KY3,KY4,KZ1,KZ2,KZ3,KZ4
DIMENSION YY(1),ZZ(1)
K=(XKRAJ-XP)/(H*FLOAT(N))
 Lesson IX - Ordinary Differential Equations 157

N1=N+1 ~
X=XP
Y=YP
Z=ZP
T=H/2.
YY(1)=Y
ZZ (1) =Z
DO 6 I=2,N1
DO 7 J=1,K
KY1=FUN(1,X,Y,Z)
KZ1=FUN(2,X,Y,Z)
KY2=FUN(1,X+T,Y+T*KY1,Z+T*KZ1)
KZ2=FUN(2,X+T,Y+T*KY1,Z+T*KZ1)
KY3=FUN(1,X+T,Y+T*KY2,Z+T*KZ2)
KZ3=FUN(2,X+T,Y+T*KY2,Z+T*KZ2)
KY4=FUN(1,X+H,Y+H*KY3,Z+H*KZ3)
KZ4=FUN(2,X+H,Y+H*KY3,Z+H*KZ3)
Y=Y+H*(KY1+2.*(KY2+KY3)+KY4)/6.
Z=Z+H*(KZ1+2.*(KZ2+KZ3)+KZ4)/6.
7 X=X+H
YY(I) =Y
6 ZZ(I)=Z
RETURN
END
Using this subroutine we solved system of equations

y' = :ryz, z' = :ryjz,

under conditions y(1) = 1/3 and z(1) = 1 on segment [1, 2.5] taking for integration step
h = 0.01, and printing on exit :c with step 0.1 and corresponding values of y, vr, z, zr,
where YT and zr are exact solutions of this system, given with

72 6
and ZT = .
7-x 2

The corresponding program and output listing are of form:

C====================================================
C RESAVANJE SISTEMA DIF. JED. METODOM RUNGE-KUTTA
C====================================================
DIMENSION YT(16),ZT(16),YY(16),ZZ(16),X(16)
YEG(P)=72./(7.-P*P)**3
ZEG(P)=6./(7.-P*P)
OPEN(8,FILE='RKS.IN')
OPEN(5,FILE='RKS.OUT') _
'READ(8,15)N,XP,YP,ZP,XKRAJ
15 FORMAT(I2,4F3.1)
YP=YP/3.
H=0.1
Nl=N+1
DO 5 I=1, N1
X(I)=XP+H*FLOAT(I-1)
YT(I)=YEG(X(I))
5 ZT(I)=ZEG(X(I))
WRITE(5,22)
H=0.01
 158 Numerical Methods in Computational Engineering

CALL RKS(XP,XKRAJ,YP,ZP,H,N,YY,ZZ)
WRITE(5,18)H,(X(I1,YY(I),YT(I),ZZ(I),ZT(I),
1 I=1,N1)
18 FORMAT(//7X,'KORAK INTEGRACIJE H=',F6.3//7X,
1'Xr,11X,'Y',10X~'TACN0',11X, 'Z'~10X,'ZTACNO'//
2(F10.2,4F14.7))
22 FORMAT(1H1,9X, 'RESAVANJE SISTEMA SIMULTANIH',
1' DIFERENCIJALNIH JEDNACINA'//33X,'Y''=XYZ'//
1 33X, 'Z' '=XY /Z')
CLOSE(5)
CLOSE(8)
STOP
END
c
FUNCTION FUN(J,X,Y,Z)
GO TO (50,60),J
50 FUN=X*Y*Z
RETURN
60 FUN=X*Y/Z
RETURN
END

1 RESAVANJE SISTEMA SIMULTANIHDIFERENCIJALNIH JEDNACINA

Y'=XYZ
Z'=XY/Z
KORAK INTEGRACIJE H= .010
X y TACNO z ZTACNO
1.00 .3333333 .3333333 1.0000000 1.0000000
1.10 .3709342 .3709342 1.0362690 1.0362690
1. 20 .4188979 .4188979 1.0791370 1. 0791370
1.30 .4808936 .4808935 1.1299430 1.1299430
1.40 .5623943 .5623943 1.1904760 1.1904760
1.50 .6718181 .6718181 1.2631580 1. 2631580
1.60 .8225902 .8225904 1. 3513510 1. 3513510
1.70 1.0370670 1.0370680 1.4598540 1.4598540
1. 80 1.3544680 1.3544680 1. 5957440 1.5957450
1. 90 1. 8481330 1. 8481340 1.7699110 1. 7699110
2.00 2.6666650 2.6666670 2.0000000 2.0000000
2.10 4.1441250 4.1441260 2.3166020 2.3166020
2.20 7.1444800 7.1444920 2.7777760 2.7777780
2.30 14.3993600 14.3993900 3.5087690 3.5087710
2.40 37.7628900 37.7631300 4.8387000 4.8387110
2.50 170.6632000 170.6667000 7.9999230 8.0000000

9.10. Boundary problems

In this section we will point out to <lifference method for solution boundary problem

(9.10.1) y" + JJ(:r):t/ + q(:r:)u = f(:r:); y(a) =A, y(/J) = B,

where functions p, 11, f an~ continuous on [o., b].
b- (],
Let us divide segnwnt [o., IJ] t.o N +1
subsegments of length h = N + , so that
1
:1:n = a+ nh (n = 0, 1, ... , N + 1). In points :r:n (n = 1, .... N) we approximate the
differential equation from (9.10.1) with

'lj
.n+1 - 2·t;.
,1/.
+ .n.-1+
'lj. 'lj. - .'lj n-1+
.n+1 . .
(9.10.2) 2 1)n ?/ qn.'lj n -- 1
. n (n=1, ... ,N),
I/, ~ /,
 L<~sson IX - Ordinary Differential Equations

wlH~re Vn. = ]J(:1:.".), rfn

If we involve substitnt.ions
=q(:r:."J fn = f(:~:.".).

(9.10. 2) can he represent(~d in form

2
(9.10.3) O.n:tfn.-1 + b";t}n + Cn.Jln+l = h fn (n = 1, ... ,N).

In reganls to honndary conditions y 0 =A ancl YN+l = B, we have the problem of

solving system of lim~ar <~qnations T:17 = r~ when~

l 0

,]
cl
J/2 ~ h'h;A"t
kh 1 l(},2
b, b2 c2 ()
:1]= l'll. 1' rl = . ' T= .

:UN h2 fN- BeN () () ()

Syst<~m matrix is tri-diagonal. For solving of this system it is convenient to perform

decomposition of matrix T as T=LR (see Chapter 2), whereby the problem is reduced
to successive solution of two triangnlar systems of linear equations. This procedun~ for
solntion boundary problem (9.1 0.1), is known as matrix factorization.
The following program is written in accordance to explained procedure.

DIMENSION A(100),B(100),C(100),D(100)
C===================================================
C MATRICNA FAKTORIZACIJA ZA RESAVANJE
C KONTURNIH PROBLEMA KOD LINEARNIH
C DIFERENCIJALNIH JEDNACINA II REDA
C Y''+ P(X)Y'+ Q(X)Y = F(X)
C Y(DG) = YA, Y(GG} = YB .
c ==================================================
OPEN(S,FILE='KONTUR.IN')
OPEN(7,FILE='KONTUR.OUT')
READ(8,5) DG,YA,GG,YB
5 FORMAT(4F10.5)
C UCITAVANJE BROJA MEDJUTACAKA
10 WRITE(*,14)
14 FORMAT(1X,'UNETI. BROJ MEDJUTACAKA'
1' U FORMATU I2'/ 5X,'(ZA N=O => KRAJ)')
READ(5,15) N
15 FORMAT (I2)
N1=N+1
IF(N.EQ.O) GO TO 60
H=(GG-DG)/FLOAT(N1)
· HH=H*H
X=DG
DO 20 I=1,N
X=X+H
Y=H/2.*PQF(X,1)
A(I)=1. -Y
C(I)='=1.+Y
B(I)=HH*PQF(X,2)-2.
20 D(I)=HH*PQF(X,3)
D(1)=D(1)-YA*A(1)
D(N)=D(N)-YB*C(N)
 160 Numerical Methods in Computational Engineering

C(1)=C(1)/B(1)
DO 25 I=2,N
B(I)=B(I)-A(I}*C(I-1)
25 C(I)=C(I)/B(I)
D(1}=D(1)/B(1)
DO 30 I=2,N
30 D(I)=(D(I)~A(I)*D(I-1))/B(I)
NM=N-1
DO 35 I=1,NM
J=NM-I+1
35 D(J)=D(J)-C(J)*D(J+1)
WRITE(7,40)N,(I,I=1,N1)
40 FORMAT(///5X,'BROJ MEDJUTACAKA N='
1 ,I3///5X, 'I' ,6X, '0' ,9I10)
DO 45 I=1,N
C(I)=DG+H*FLOAT(I)
45 B(I)=PQF(C(I),4)
WRITE(7,50)DG,(C(I),I=1,N),GG
WRITE(7,55)YA,(D(I),I=1,N),YB
WRITE(7,65)YA,(B(I),I=1,N),YB
50 FORMAT(/5X,'X(I)' ,10(F6.2,4X))
55 FORMAT(/5X,'Y(I)',10F10.6)
65 FORMAT(/5X, 'YEGZ',10F10.6)
GO TO 10
60 CLOSE(7)
CLOSE(8)
STOP
END
Note that this program is so realized that number of inner points N is read on
input. In case when N = 0 program ends. Also, in program is foreseen tabulating
of exact solution in observing points, as control. It is clear that last has meaning for
scholastic examples when solution is known. So, for example, for boundary problem

y" - 2:r:y' - 2y = -4:r; y(O) ;::= 1, y(1) = 1+e ~ 3. 7182818,

the exact solution is y = :r + e:r:p(.r 2 ).

For this contour problem function subroutine for defining functions p, q, .f, as for as
for exact solution, is named PQF. Iu case N=4, we got the results given in continuation.

FUNCTION PQF(X,M)
GO TO (10,20,30,40),M
10 PQF=-2.*X
RETURN
20 PQF=-2.
RETURN
30 PQF=-4.*X
RETURN
40 PQF=X+EXP(X*X)
RETURN
END

BROJ MEDJUTACAKA N= 4
I 0 1 2 3 4 5
X(I) .00 . 20 .40 .60 .80 1.00
Y(I) 1.000000 1.243014 1.576530 2.035572 2.695769 3.711828
YEGZ 1.000000 1.240811 1.573511 2.033329 2.696481 ~.711828
 Lesson IX - Ordinary Differential Equations 1o1

9.11. Packages for ODEs

Numerous libraries and software packages are available for integrating initial-value
ordinary differential equations. Many work stations and main frame computers have
such libraries attached to their operating systc~ms.
Many comnwrcial software packages contain routines for integrating initial-value
ODEs. One of the oldest and very known among senior scientist is SSP (Scientific
Subroutine Package) of IBM. For ODEs it has subroutines RK1 (integral of first-order
differential equation by Runge-K utta method), RK2 (integral of first-order differential
equation by Runge-Kutta method in tabulated form) using in both subroutines fourth
order Runge- K utta nwthod, and RKGS (solution of system of first-mcler differential equa-
tions with given initial valnc~s hy the Runge-Kutta method) using evaluation by mec:.ms
offonrth order Rnnge-Kutta formulae in the modification due to Gill. Some ofthe more
prominent packagc-~s are Matlab and Mathcad. More sophisticated packages, such as
IMSL, Mathematica, and Macsyma contain also algorithms for integrating initial-value
ODEs. The hook N'Umerical Recipes([12]) contains numerous subroutines for inte-
grating initial-value onlinary differential equations and the book N'Umerical M eth-
ods for Engineers and Scientists([3]) program code for solving single first-order
ODEs, higher order ODEs, and syst(~ms of first-order ODEs, by using single-point meth-
ods, extrapolation methods, auclmnlti-point methodt:> (see Chapter 7, One-Dimensional
Initial-Value Ordinary Difff~nmtial Equations).

Bibliography (Cited references and further reading)

[1] Milovanovic, G.V., Nv:merical Analysis III, Nauf:na knjiga, Beograd, 1988 (Ser-
bian).
[2] Milovanovi{:, G.V., N'Umerical Analys'iS I, Nauc:na knjiga, Beograd, 1988 (Ser-
bian).
[3] Hoffman, .J.D., N'Umerical Methods for Engineers and Scientists. Taylor
& Francis, Boca Raton-Lonclou-N(-~W York-Singapore, 2001.
[4] Milovanovi{:, G.V. and Djorcljevi(:, Dj.R., Programiranje n'Umer'ickih metoda
na FORTRAN jeZ'ik'U. Institnt za clokmnentaciju zastite na racln "Edvanl
Kanlelj", Nis, 1981 (Scrhian).
[5] Milovanovi(:, G.V., N'Umerical Analysis II, Nauc:na kujiga, Beograd, 1988 (Ser-
bian).
[6] Bertolino, M., N'Umer'icka anal'iza, B(~ograd, 1977.
[7] Runge, C., Uber d'ie numer'ische Avfios'Ung von Differentialgle,ich'Ungen,
Math. Ann. 4o(1985), 167-178.
[8] Kntta, W., Beitrag Z'UT Naher'Ungsweisen Integration totaler Differen-
tialgleich'Ungen, Z. Math. Phys. 46(1901), 435-453.
[9] Neuu, K., Ne'Ue Methode· Z'Ur approxirnat'iven Integration der Differ-
entialgleich'Ungen einer 1tnabhangigen Veranderlichen, Z. Math. Phyt:>.
45(1900), 23-38. .
[10] Gill, S., A Process for the step-by-step integration of differential eqtt,a-
tions in an a'Utornatic comp'Uting machine, Proc. Cambridge Phil. Soc.
47(1951), 9o-1os.
[11] Stoer, J., and Bulirsch, R., Introd'uct'ion to N'Umerical Analysis, Springer,
New York, 1980.
[12] Pret:>s, W.H., Flannery, B.P., Tenkolsky, S.A., and Vetterling, W.T., N'Umerical
Recepies --.The Art of Scientific Comp'Uting. Cambridge University Prest:>,
1989.
[13] Milovanovic:, G.V. and Kovai':evif:, M,A., Zbirka resenih zadataka iz n'U-
mericke analize. Nauc:ua knjiga, Beograd, 1985. (Serbian).
[14] R.alston,A., A First Co'Urse in N'Umerical Analysis.
McGraw-Hill, New York, 19GG,
1()2 Numerical Methods in Computational Engineering

[15] Hildebrand, F.B., Introduction to Numerical Analysis.

Mc.Graw-Hill, New York, 1974.
[16] Gear, C.W., Numerical Initial Value Problems in Ordinary Differential
Equations. Englewood Cliffs, N.J: Prentice-Hall, 1971.
[17] Acton, F.S., Numerical Methods That Work (corrected edition). Mathemat-
ical Association of America, Washington, D.C., 1990~
[18] Lambert, .J., Computational Methods in Ordinary Differential Equa-
tions, Wiley, New York, 1973.
[19] Lapidus, L., and Seinfelcl, J., Numerical Solution of Ordinary Differential
Equations, Academic Press. New York, 1971. ,
[20] Abramowitz, M., and Steg;un, LA., Handbook ofMathematical Functions.
National Bureau of St~mclanls, Applied Mathematics Series, Washington, 1964
(reprinted 1968 by Dover Publications, New York).
[21] Rice, .J.R., Numerical Methods, Software, and Analysis. McGraw-Hill, New
York, 1983.
[22] Forsythe, G.E., Malcolm, M.A., and Moler, C.B., Computer Methods for
Mathematical Computat'ions. Englewood Cliffs, Prentice-Hall, N.J, 1977.
[23] Kahaner, D., Moler, C., and Nash, 8., 1989, Numerical Methods and Soft-
ware. Englewood Cliffs, Prentice Hall, N.J, 1989.
[24] Hamming, R.W., N'umerical Methods for Engineers and Scientists. Dover,
New York, 1962 (reprint<~d 1980).
[25] IMSL Math/Library Users Manual , IMSL Inc., 2500 City West Boulevard,
Houston TX 77042
[26] NAG Fortran Library, Numerical Algorithms Group, 256 Banbury Road, Ox-
ford OX27DE, U.K., Chapter F02.
Faculty of Civil Eugim~<~riug Faculty of Civil Engineeriug aucl Architect.nn~
Belgrad<~ ~ . Nis
Master Study Doctoral Stndy
COMPUTATIONAL ENGINEERING

LECTURES

LESSON X

10. Partial Differential Equations - PDE

10.1. Introduction
Partial diffen~utial <~quations (PDEs) arise in all fields of engineering and sci<~nu~.
Most real physical proc<~sscs an~ gov<~rned by partial cliffereutial equations. Iu umuy
cases, simplifying approximations arc made to reduce the governing PDEs to ordinary
cliffen~utial equatious (ODEs) or even algebraic equations. However, because of the <·~ver
increasing requirem<~uts for mon~ accurate modelling of physical processes, engineers and
sc:ieutists are mor<~ anclmcm~ rcquin~d to solve the actual PDEs that govern the physical
problem being investigated. Physical problems are governed by many different PDEs. A
few problems are gov<~rued by a single first-order PDE. Numerous problems are governed
by a system of first order PDEs. Some problems are governed by a singl<~ second-cmler
PDE, and num<~rons prohl<~ms are governed by a system of second-order PDEs. A few
problems are governed by fourth order PDEs. Tlw two most frequent types of physical
problems described by PDEs are equilibrium and propagation problems.
The classification of PDEs is most easily <~xplained for a single second order linear
PDE of form
D2 ·n !.l')
02 ;) D
B C v, D u·u E v,
(10.1.1) A-+
2
D:r:
u~'.IL
--+
D:r:Uy
/-+
Dy 2
-.
D:r:
+ -Dy+ } ' i' l = G .'
where A., B, C, D, E, F. G an~ giv<~u fnuctions which are continuous in area S of plaue
:r:Oy. The areaS is nsnally <l<~fined as inside part of some curve r. Of course, tlw area
S can be as finite as well as infiuite. Typical problem is finding two times continu-
ous differentiable solutim1 (:~:, y) ---1 u(:r:, y) which satisfies equation ( 10 .1.1) and some
conditions on curve (contour) r.
Linear PDEs of second order can be classified as eliptic, parabolic and hyperbolic,
depending on the sign of the discriminant B 2 - 4AC in given area S, as follows:
1° B 2 . - 4AC < 0 Elliptic
2° B 2 - 4AC = 0 Parabolic
3° B 2 - 4AC < 0 Hyperbolic
The terminology elliptic, para:bolic, and hyperbolic chosen to classify PDEs n~flects
the analogy betwe<'~n the form of the discriminant, B 2 - 4AC, for PDEs and the form
of the disc:riminaut, B 2 - 4AC . which classifies conic sections, described by the general
seconcl-orcl0r algebraic equation

A:r: 2 + B:r:y + Cy 2 + D:r: + Ey + P = 0,

where we for neg<1.t.ive, ~ero, and positive value of discriminant have ellipse, parabola,
and hyperbola, n~spcctively. It is <~asy to check that the Laplace equation

D2 v.
-·+-· -=0
o2 v.
(10.1.2)
Dx 2 Dy 2 '

Hi3
1{)4 l~umencallVletllocis m Computational .l:!_;ngineering

is of elliptic type, heat conduction equation

~) >)2
U7L 2U '/1,
(10.1.3) ~ - a >), • = 0,
ut u.L 2

is of patabolic type, and wave eqnettion

(10.1.4)

of hyperbolic type. In this chapter we will show one way for numerical solution of PDEs,
for Laplace and wave equation by grid method. In the similar way can be solve heat
conduction equation, what we leave to the reader.

10.2. Grid method

Grid method or difference method, or finite~clifference grid method, is basic method
for solution of equations of mathematical physics (partial equations which appear in
physics and science)
Let be given linear PDE

(10.2.1) Lu= f

and let in area D, which is hounded by curve f(D = int; r), look for such its solution
on curve r that satisfi~~s given boundary condition

(10.2.2) Kv.=\]1 ((:r:,y)Ef).

In i.Lpplication of grid method, at first, one should chose discrete set of points Dh,
which belongs to area D (= D U r), called grid. Most frequently, in applications is for
grid taken family of parallel straight lines :r:.i = .1: 0 +ih, Y,j = Yo+jl ('i,j = 0, ±1, ±2, ... ).
Intersection points of these families are called nodes of grid, and h and l are steps of
grid. Two nodes of grid are called neighbored if the distance between them along :r: and
y axes is one step only. If all four neighbor nodes of some node belong to area D, then
this node is called interior or inner: in counterpart node of grid Dh is called boundary
node. In addition to rectangul<tr grids, in practice are also used other grid shapes.
Grid method consists of approximation of equations (10.2.1) and (10.2.2) using cor-
responding difference equations. Namely, we can approximate operator L by difference
operator very simple, l>y snbstituting derivative with corresponding differences in inner
nodes of grid. Then~by a.n~ nse<l tlw following formulas

Un(:r:.;, :tf:i) ~ ·u,·i.+l,.J- n.;.,_j

ih: h
a'IJ.(:r:;., Y.i) 0! '1/,i+l,.J- n·i-l,j
dy 2h
d 2 n(:r:;, Y.i) ~ '1/,;.+l,.i- 2n.;.,:i + n·i-l,j de.
Dx 2 h2
Formulas for partial dl~rivativcs in varia,blc y are absolutely symmetric. Approxi-
mation of boundary conditions can he in some cases very complicated probkm, what
depewls on form of opcratm J( awl contour r. At so known boundary conditions of first
kind, where Kn = ·u,, mu~ practical way for approximation was proposed hy L. Collatz
ancl cmuprises of tlw followiug:
 Lesson X - Partial Differential Equations 165

. Let the ~:losest /\)oint fmm contom f to boundary uocle A be point B and let their
chstance be r~ (see F1g. 10.2.1).

!I'
a/ A c
~~
~h .....

I
x.

Figure 10.2.1
Based on function vahws in points B and C, we get by linear interpolation

u(A) ~ h\J!(B) + r~n(C).

h + rl
Approximation of btnmdary condition (10.2.2) in this case comprises of defining (~qua­
tions of above form for every boundary node.
The (~quations obtained by approximation of equation (10.2.1) and boundary comli-
tion (10.2.2) form system of linear equations, by which solution are obtained murHTical
solutions of given problem.
In further consicl<mttion we will give two basic examples.

10.3. Laplace equation

Lc~t it be needfully to find solution of Laplace equation

which on the contour of square D = { (:r:, y) I0 < :r: < 1, 0 < y < 1} fulfills given condition
.. . 1
n(:r, y) = \J!(:r:, y) ((:r, y) E f). Let's chose the grid in Dh at which is l = h = N _ , so
1
that grid nods are points (:.r:.;,, y.;.) _:_ (('i ~ 1)h, (.j- 1)1) (i,j = 1, ... , N). The standard
difference approximation sdwme for solving Laplace equation is of form

1
~·u,i.+1 1·
h, £, '·
+ v,'i.-1 1· + 'IJ.i. 1·-1
'· ).
- 4v,;,. 1· = 0,
).

or
1
'1/,i.,j = 4('U.·i.,j+l + 'lJ.-i.,j-1 + 'U,'i-1,.i + 'U,·i+1,j)·
Taking i, _j = 2, ... , N -1 inlast equality we get the system of (N- 2) 2 linear equation.s.
For solving this system usually is used method of simple iterations, or, even more
simpler, Gauss-Seidel method.
The corresponding program for solving problem in consideration is of form
166 Numerical Methods in Computational Engineering

C====================================;=~==========
C RESAVANJE LAPLACE-OVE JEDNACINA
C=================================================
DIMENSION U(25,25)
OPE.N(8,FILE='LAPLACE.IN')
OPEN(5,FILE='LAPLACE.OUT')
READ(8,4)N
4 FORMAT (I2)
M=N-1
READ(8,1)(U(1,J),J=1,N),(U(N,J),J=1,N),
1(U(I,1),I=2,M),(U(I,N),I=2,M)
1 FORMAT(8F10.0)
DO 10 I=2,M
DO 10 J=2,M
10 U(I,J)=O.
IMAX=O
20 WRITE(*,5)
5 FORMAT(5X,'UNETI MAKSIMALNI BROJ ITERACIJA'/
110X, '(ZA MAX=O => KRAJ)')
READ(*,4)MAX .
IF(MAX.EQ.O) GOTO 100
DO 30 ITER=1,MAX
DO 30 I=2,M
DO 30 J=2,M
30 U(I,J)=(U(I,J+1)+U(I,J-1)+U(I-1,J)+U(I+1,J))/4.
IMAX=IMAX+MAX
WRITE(5,65) IMAX,(J,J=1,N)
65 FORMAT(//26X,'BROJ ITERACIJA JE',I3//17X~
14(5X,'J=',I2)) .
DO 60 I=1,N
60 WRITE(5,66) I,(U(I,J),J=1,N)
66 FORMAT(13X, 'I =',I2,6F10.4)
GO TO 20
100 CLOSE(8)
CLOSE(5)
STOP
END
For Rolving sy~t<~m of lill(~ar equations WP n~ed Gauss-S<~iclel method with initial
conditions 'IL;.,:j = 0 ('i.,j = 2 .... ,N -1), wlwn~by one can control number of iterations
on input. For N=4 and homHlary conditions

'1/.u = 0, ·u.1.2 = 30, ·u.13 =GO, v.1,4 = 90,

'1/.41 = 180, '/1.4,2 = 120, '1/.43 = 60, 'IL4,4 = 0,
·u. 21 = GO, ·u.3,1 = 120, n24 = GO, ·u.3,4 = :30,
the following results an~ ohtaiw~d:

BROJ ITERACIJA JE 2
J= 1 J= 2 J= 3 J= 4
I = 1 .0000 30.0000 60.0000 90.0000
I = 2 60.0000 47.8125 53.9063 60.0000
I = 3 120.0000 83.9063 56.9531 30.0000
I = 4 180.0000 120.0000 60.0000 .0000
BROJ ITERACIJA JE 7
J= 1 J= 2 J= 3 J= 4
I = 1 .0000 30.0000 60.0000 90.0000
 L<~ssml X - Partial Diff·(~n~ntial Eqnat.ions Hi7

I = 2 60.0000 59.9881 59.9940 60.0000

I = 3 120.0000 89.9940 59.9970 30.0000
I = 4 180.0000 120.0000 60.0000 .0000
BROJ ITERACIJA JE 9
J= 1 J= 2 J= 3 J= 4
I = 1 .0000 30.0000
60.0000 90.0000
I = 2 60.0000 59.9993
59.9996 60.0000
I = 3 120.0000 89.9996
59.9998 30.0000
I = 4 180.0000 120.0000
60.0000 .0000
BROJ ITERACIJA JE 10
J= 1 J= 2 J= 3 J= 4
I = 1 .0000 30.0000 60.0000 90.0000
I = 2 60.0000 59.9998 59.9999 60.0000
I = 3 120.0000 89.9999 60.0000 30.0000
I = 4 180.0000 120.0000 60.0000 .0000
BROJ ITERACIJA JE 21
J= 1 J= 2 J= 3 J= 4
I = 1 .0000 30.0000 60.0000 90.0000
I =2 60.0000 60.0000 60.0000 60.0000
I = 3 120.0000 90.0000 60.0000 30.0000
I = 4 180.0000 120.0000 60.0000 .0000

10.4. Wave equation

Consider wave equation

[)2'11.
(10.4.1)
(J:r:2 a 2 . o:r 2

with initial conditiom;

(10.4.2) ·u.(:r. 0) = .f(:E:), ·n 1 (:c, 0) = g(:1:) (0 < :1: <h)

ancl bonnclary cm1ditious

(10.4.3) u.(o,t) = <D(t), ·u.(b, t) = w(t) (t ~ o).

Using finite diff'creuccs, the (~quation (11.4.1) can be approximated by

1
(10.4.4) '/L.i+LJ.
..
- 2n.; ).
J. + '11..i.~1 '·J. ,,.. . (·n.; '·1.+1- 2n.; ' j
= -2 + n.; 1·-1),
),

where T = o.t_ (h ancll arc steps along :rand taxes respectively), and '/l..i.,.i ""'v.(:c.;, tj)·
Based on first equality in (10.4.2) we have

(10.4.5) 'IL;,n = f (:r.;) = .f.i.

By introducing fictive layer j = -1, secon<l initial condition in (10.4.2) can simple be
approximated using
'/J.,i 1 - 'LL·i -1
(10.4.6) v.i(:r:.i, 0) = q(:r.i) = .rh ~ ., z ., ·
2

If we put in (10.4.4) .i = 0 we get

los Numerical Methods in Computational Engineering

wherehom, in regard to (10.4.G) it follows

1
ni,l = l.rh + fi + ?T 2 (fi+1- 2fi + fi-1),
""'
Le.

(10.4.7)

On the ot.h<w hawl, fi:om (10.4.4) it follows

c 1 1
(10.4.8) 'll.i
'·
j+1 = -:)"(·u
'f'.:.J
..i+1 .; + '1/..i.-1 .;) -
1,/ ),/
'11..;. ·i-1
)./ '
+ 2(-2
,,.
- l)v.i 7·.
'·

If w<~ put h = b/N and :1:.; = ('i -l)h (-i = 1., 2, ... , N + 1), clue to hounclary
conditions (10.4.3) we have

(10.4.9)

where j = 0, 1, .... For ddcnnining of solution inside of rectangle P = {(:r:, t)(O < :r: <
b, 0 < t < Tmn:r:}, uwxinml valn<c~ of inclex j is int<~ger part of Tm.n:c/l i.e . .J.rn.o.:r: = M =
[Tnw.:r:/l].
Bas<~d on <~qualities (10.4.G), (10.4.7), (10.4.8), (10.4.9) the approximate solutious
of giveu problem in ~ri<l uoclcs of rectangle P, arc tlimple to obtain. This algorithm is
coclcd in the followin~ program.

C==================================================
C RESAVANJE PARCIJALNE DIF. JED. HIPERBOLICNOG TIPA
C==================================================
DIMENSION U(3,9) ,
OPEN(8,FILE='TALAS.IN')
OPEN(5,FILE='TALAS.OUT')
READ (8,5)N,A,B,R,TMAX
5 FORMAT(I2,4F5.2)
N1=N+1
WRITE (5,10) (I,I=1,N1)
10 FORMAT(10X,1HJ,<N+1>(4X,'U(' ,I1,' ,J) ')/)
H=B/FLOAT(N)
EL=R*H/A
M=TMAX/EL
T=O.
DO 15 K=1,2
U(K,1)=FF(T,B,3)
U(K,N1)=FF(T,B,4)
15 T=T+EL
X=O.
R2=R*R
DO 20 I=2,N
X=X+H
U(1,I)=FF(X,B,1)
20 U(2,I)=EL*FF(X,B,2)+(1.-R)*U(1,I)
DO 25 I=2,N
25 U(2,I)=U(2,I)+R2/2.*(U(1,I+1)+U(1,I-1))
J=O
30 WRITE(5,35)J,(U(1,I),I=1,N1)
35 FORMAT(7X,I5,<N1>F10.4)
 L!~ssou X - Partial Difh~n~ntial Equations I L()!)

IF(J.EQ.M)GO TO 50
J=J+1
U(3,1)=FF(T,B,3)
U(3,N1)=FF(T,B,4)
DO 40 I=2,N
40 U(3,I)=(U(2,I+1)+U(2,I-1))/R2-U(1,I)-2.
1*(1./R2-1.)*U(2,I)
T=T+EL
DO 45 I=1,N1
U(1,I)=U(2,I)
45 U(2,I)=U(3,I)
GO TO 30
50 CLOSE(5)
CLOSE(5)
STOP
END
. . Note that tlw vahws of solution iu thn~e successive layers .i - 1, j, _j + 1, a.re stored
m first, S<~cowl, awl thinl row of nw.trix U, respectively..
. Functions .f, q. <D, \]1 an~ d<~fiw~d hy hmction subroutine FF fm 1=1, 2, 3, 4, respPc-
tlvely. .
In considered case for a= 2, !J = 4, Tma.:r: = 6, f(:E) = :r(4- :r), g(:r) = 0, <D(t) = 0,
w(t) = 0, N = 4, awl,.= 1, subroutim~ FF and corresponding output listing with rc~mlt
have the following f(mn: -

FUNCTION FF(X,B,I)
GO T0(10,20,30,40),I
10 FF=X*(B-X)
RETURN
20 FF=O.
RETURN
30 FF=O.
RETURN
40 FF=O.
RETURN
END

J U(1,J) U(2,J) U(3,J) U(4,J) U(5,J)

0 .0000 3.0000 4.0000 3.0000 .0000
1 .0000 2.0000 3.0000 2;0000 .0000
2 .0000 .0000 .0000 .0000 .0000
3 .0000 -2.0000 73.0000 -2.0000 .. 0000
4 .0000 -3.0000 -4.0000 -3.0000 .0000
5 .0000 -2.0000 -3.0000 -2.0000 .0000
6 .0000 .0000 .0000 .0000 .0000
7 .0000 2.0000 3.0000 2.0000 .0000
8 .0000 3.0000 4.0000 3.0000 .0000
9 .0000 2.0000 3.0000 2.0000 .0000
10 .0000 .0000 .0000 .0000 .0000
11 .0000 -2.0000 -3.0000 -2.0000 .0000
12 .0000 -3.0000 -4.0000 -3.0000 .0000

10.5. Packages for PDEs

Elliptic PDEs govcm equilibrium problem, which have no preferred paths of infor-
mation propagation. The domain of dependence awl range of influence of every point is
the entire closed solution domain. Snell problems "arc solved nnnH:~ricaUy by relaxa.tion
methods. Finite difh~nmcc nH~tho<ls, as typified l >y five-point method, yield a system
of finite difference <~qnations, call!~<l the system equations, which have to be solved by
170 Numerical Methods in Computational Engineering
- -
relaxation methods. The successive-over-relaxation method (SOR)method is generally
method of choice. The multigrid method (Brandt, 1977) shows the best potential for
rapid convergence. Nonlinear PDEs yield nonlinear finite difference equations (FDE).
System of nonlinear FDEs can be very difficult to solve. The multigrid method can be
applied directly to nonlinear PDEs. Three-dimensional PDEs are approximated sir'nply
by including the finite difference approximations of the spatial derivatives in the third
direction. The relaxation techniques used to solve two-dimensional problems gener-
ally can be used to solve three-dimensional problems, at the expense of a considerable
increase of computational time.
Parabolic PDEs govern propagation problems which have an infinite physical infor-
mation propagation speed. They are usually solved numerically by marching metho<l.
Explicit finite difference methods, like FTCS (Forward- Time Centered-Space method,
see [3], pp. 633-635) are conditionally stable and require relatively small step size in
the marching direction to satisfy the stability criteria. Implicit methoqs, like BTCS
(Backward- Time Centered-Space method, see [3], pp. 635-637) are unconditionally sta-
ble. The marching step size is restricted by accuracy requirements, not stability require-
ments. For accurate solution of transient problems, the marching step-size for implicit
methods cannot be very much la,rger than the stable step size for explicit methods.
Consequently, explicit methods are generally preferred for obtaining; accurate transient
solutions. Asymptotic steady state H6lutions can be obtained very efficiently by BTCS
method with a large marching step size. Nonlinear PDEs can be solved directly by
explicit methods. When solved by implicit methods, system of nonlinear FDEs must be
solved. Multidimensional problems can be solved directly by explicit methods. When
solved by implicit methods, large banded systems of FDEs result.
Hyperbolic PDEs gowm propagation problems, which have a finite physical infor-
mation propagation speed. Tlwy are solved numerically. by marching method. Explicit
finite difference nwtho<hi are cowlitioually stable and require a relatively small step size
in marching direction to satisfy the stability criteria. Implicit methods, as typified by
the BTCS method, are Ull<:owliti<mally stable. The marching step size is restricted by
accuracy requin~ments, not stc=tbility requin~ments. For accurate solution of transient
problems, explicit methods are recommended. When steady state solutions are to he
obtained as the asymptotic solution in time of an appropriate unsteady propagation
problem, BTCS with alarg<~ step size is recommended.
Nonlim~ar PDEs can lH~ solve<l directly by explicit methods. When solved hy im-
plicit methods, system of nonlinear FDEs must b<~ solved. Multidimensional probl(~llls
can be solved directly by explicit mdlwds. When solved by implicit methods, large
banded systmns of FDEs result.
Numerous libraries awl software packages are available for integratiug the Laplace
and Poisson equations, diffusion type (i.e. parabolic) and convection type (i.e. hy-
perbolic) PDEs. Many work stations and main frame computers have such librari<~s
attached to their operating systems.
Mmw comnH~n:ial software packages coutain routim~s for integrating Laplau~ awl
Poisson equations. Dn<~ to the wide vari<~ty of dliptic, parabolic, aucl hyperbolic PDEs
governing physical prohlems, many PDE solvers (programs) have been developed.
The book N·umer'ical Recipes ([7]) contains <1 lot of algorithms for integrating
PDEs. For some of tlwm is given programming <:o<l<~ in Fortran (available also in C).
Survey of uwtho<ls for solving different dass<~s of PDEs accompani<~d with algorithms.
from which some an~ cod<lecl, is given in book Nv,merical NI ethods for Eng'ineers
and Scientists ([3], Ch<tpt<~r \J, 10 awl 11).

Bibliography (Cited references and further reading)

[1] Milovanovi<':, G.V., Numerical Analysis III, Nanc:na kl~jiga, Beogra<l, 1U1:l8 (Ser-
bian).
[2] Milovanovi( G.V., N'umerical Analys'is I. Nanc:na kn.iiga, BeognuL 1U88 (S<~r­
bian).
[:3] Hoffman, .T .D .. Numerical Methods for Eng'ineers and Scientists. Taylor
& Fmncis, Boca R.aton-Londo!l-Ncw Ymk-Sillgapm<~, 2001. ,
 L<~ssmJ X - Partial Diff<~n~utial Eqnatious 171

[4] Milovauovit, G .\r. aud D. ion li<~vit. D.i ,R., Program·iranje numer·ick·ih metoda
na FORTRAN _jez·iku. Iust.it.nt za doknuwutacijn zastitc ua radn ''Edvard
Km-dd.(. Nis, 19~1 (S<~rbiau).
[S] I\!Iilovauovit, G.V .. Numer·ical Analysis II. Nan(:ua lmjiga. B~ograd. 1988 (S<~r­
biau).
[G] Sto<T ..L awl Bnlirscl1. R., Introduct·ion to N·umerical Analysis, SpriHg<T.
N<~w York. 1~Jo0.
[7] Press, W.H .. F'l;mwT'>'- B.P., T<~nkolsky, S.A., ctll<l Vc~ttediug, vV.T., Nv.men:cal
Recepies - The A'l"i of Scientific Comput·ing. Cambridge University Pn~ss,
Elo9.
[8] lVIilovauovit, G.V. i\Jl(l Kova!:<~vit, M.A., Zb·irka resenih zadataka iz n'll.-
mericke ana.Z.iz('.. Nanhm ku.ii.u;a, B<~ogra(L 198S. (S(~rbiau).
[0] Ralstou.A., A P.iTst Co·u.rse in Numerical Analys·is.
McGraw-Hill, New Ymk, EJG:J.
[10] Hilddm\.lHL F.I3 .. Introduct·ion to N·nmerical Analysis.
Mc.Graw-Hill. New York. 197L1.
[11] Actou, F.S., Numer·ical Mdhods That vVork (conecterl e<litiou). Matlwmat.-
ical Associatiou of Auwrica, Washingtou, D.C., 1990.
[12] Abramowit:;,. M .. alHl St.cgm1, I.A., Handbook of Mathemat·ical Functions.
National Bmean of SL1.ndanls, Applif~d Matlwmatic:s Series, Washingtou, 19Cd
(n~printcd 19()8 hy Dover Pnhlicatiom;, N(~W York).
[1:3] Rice, J.R., Numer·ical Methods, Software, and Analysis. McGraw-Hill, New
York, 1 ~)83.
[14] Forsyth(~, G.E., Malcolm, M.A., and Moler, C.B., Comp·uter Methods for
Mathemat·ical Computat·ions. Englewood Cliffs, Prentice-Hall, NJ, 1977..
[1S] Kahauer, D., Mol(~r. C., aml Nash, S., HJ89, Numer·ical Methods and Soft-
ware. Englewood Clift"s, Preutiu~ HalL N.T, Em9.
[Hi] Hamming, R.vV., Nmner·ical Methods for Engineers and Sc'ient·ists. Dov(~L
New York, 1%2 (u~priut.ed lV~G).
[17] F'<~rzigcr, .T.H., Nv.mer·ica.l Methods for Eng-ineering Applicab:ons. St.anfoul
Uuivc-~rsit.y, John Will(~y & Sous, Inc., N<~w York, 1998
[18] Pearson, C.E .. Nv,mcn:cal A1cthods ·in Engineering and Science. University
of \iV;t!ihingtmL Van Nostraw l H.cinhold Company, New Ymk, 1D8G.
[19] St<~plwnson. G. awl n.arlmorc. P.M., Advanced Mathematical Methods for
Eng·ineer·ing and Science St'ILdents. Imp(~rial Collegf~ Loudon, University Col-
lege. Loudon Cmubri(lg<~ Uuiv. Press, 1999
[20] Anws, W.F., Numer·ico.l Methods for Partial Different·ial Eq·uations. 2rtd
(-~d. Acad(~lllic Press . N(~W York. 1977.
[21] Richtllly<~r. RD. awl l\!Imtou, K.vV ... D·i.fference Methods for In-itial Val·ue
Problem,s. 2n(l cd .. vVilcy-Iutcl·scienc:c, New York, 1907.
[22] Mitchell, A.R, and Griiiiths, D.F., The Finde Difference Method in Partial
D~fferent·ial Equations., \!\Tiley, New York, 1980.
[23] IJvfSL Math/Library Users Man·ual, IMSL Inc., 2500 City West Boulevard,
Houston TX 77042
[24] NAG Fortran Library. N1mt<Tic:al Algorithms Group, 2S6 Banbury Road. Ox-
ford OX27DE.. U.K., Chapter F02.
/
Facnlty of Civil Eugiw~<~riug Faculty of Civil Engineering ancl Archit<~ctme
BelgraclP " Nis
Master Study Doctoral Study
COMPUTATIONAL ENGINEERING

LECTURES

LESSON XI

11. Integral Equations

11.1. Introduction
In spite the fact that integral eqnations an~ i1hnost never treated in numerical <mal-
ysis textbook1->, there is a large awl growing literature on their numerical solution. Om~
reason for the she<~r vohmw of this activity is that there are many different kinds of
equations, each with many different possibk pitfallt~. Often many different algorithms
have been propm;<~<l to deal with a single case. There is a close corresponcknce between
linear integral equations, which specify linear, integral relation1-> among functions in au
infinite-dimensional fum:timJ space, awl plain old linear equations, which specify c-malo-
gous rdatiom; among V<-~ctors in a finite-clinwnsional vector space. This correspondence
lies at the heart of most cmuputational algorithms, as we will se<~ in program n~alization
of their nmm~rical solntiou.
The equation

I!

(11.1.1) y(:1:) = f(:r) +A./ K(:D, t)y(t) dt,

where f i J{ are known fnnctions, y nnknown function, ancl A numerical parameter, is

called a Freclhohu integral <~quation of second kind. Fredholm <~quations involve rldiuite
integrals with fixed upper and lower limits.
The function in two variables K is called hrnel of integral equation (11.1.1). In
our considerations w<~ will always snppos<~ that kernel is defined aud continuous 011
D={(x:,t)la:S:r:Sh, a::=;t::=;b}.
If .f(:c) 1:- 0, tlw equation (11.1.1) is called inhomogeneous, ancl in case when .f(:t:) =
0, eqnation is homogc~nous.
Integral (~qnation of form
/)

f(:r:) +A./ K(:r, t)y(t) dt =0

(],

is callecl Fredholm integntl equation of first kind. This equation can be written m
analogous fonn, as matr:ix equation

K·:l}=-/

which solution is __, K- 1 f_,

y =- .. '

173
 174 Numerical Methods in Computational Engineering

·and K- 1 is matrix inverse. Both equation are solvable when function f and [are
nonzero, respectively (the homogeneous case with f=O is almost never useful), and
K(K) is invertible.
The analogous matrix f()lm of Fredholm equation of second kind (11.1.1) is

1
( K - -I) · y =
~ f
~-.
). ).

Again, iff or [is zero, then theequation is said to be homogem~ous. If the kernel K(x, t)
is bounded, then, like in matrix form, the equation (11.1.1) has the property that its
homogeneous form has solutions for at most a denumerably infinite set >. . . :. . An, n =
1, 2 ... , the eigenvalues. The corresponding solutions Yn (x) are the eigenfunctions. The
eigenvalues are reaLif the kernel is symmetric. In the inhomogeneous case of nonzero f
or [, both equations are solvable except when ). or 1/). is an eigenvalue - because the
integral operator (or matrix) is singular then. In integral equations this dichotomy is
called the Fredholm alternative. ·
Fredholm equations of the first kind are often extremely ill-conditioned. Applying
the kernel to a function is generally a smoothing operation, so the solution, which
requires inverting the operator, will be extremely Hensitive to small changes or errors in
the input. Sinoothing often actually loses information, and there is no way to get it back
in an inverse operation. Specialized methods have been developed for such equations,
which are often called inverse problems. The idea is that method must augment the
information given with some prior knowledge of the nature of the solution. This prior
knowledge is then used, in some way, to restore lost information.
Volterra integral equations of first and second kind are of forms
.7:

f(:r:) + >. ./ K(:r, t)y(t) dt = 0

and ~r;

y(:r:) =f(:r) + >. ./ K(:z;, t)y(t) dt,

respectively. Volterra equations an~ a special case of Fredholm equations with K(:r, t) =
0 fort > :r;. Chopping off the unnecessary part of the integration, Volterra equations are
written in a form where the upper limit of integration is the independent variable :r. The
analogous matrix form of Volterra equation of first kind (written out in components) is
h:

2:.= Kk:.iY.i = .h,,

.i=l

wherefrom we see that Voltcna equation corresponds to a matrix K that is lower (left)
triangular. As we already know, such matrix <~quations are trivially soluble by forward
substitution. Techniques for solving Voltmra equations are similarly straightforward.
When experimental measm·<~nH~nt noise <loes not dominate, Volterra equations of the
first kind tend not to be ill-conditioned. The upper limit to !he integral introduces a
sharp step that conveniently spoils any smoothing properties of the kerw~l. The matrix
analog of Voltena (~quation of the second kind is

(K - I) . :t7 = .F,
with K lower triangular matrix. The reason there is no >. in these equations is that in
inhomogeneous cas<~ (nonzero f) it can be absorb<~d into K or K, while in the homog<~­
neous case (.f = 0), there is a t.lH~orem that Vol ten a equations of the second kind with
hounclecl kerw~ls have uo <~ig<~nvalnes with sqnare-integrahle eigenfunctions.
 Lesson XI - Integral Eqnations 175

We have c:ousicterecl only the case of linear integral equations. The integrand in
a nonlinear version of given equations of first kind (Fredholm and Volterra) would
be K(:r, t, y(t)) iustead of K(:r:, t)y(t), and a nonlinear versions of equations of second
kind would have an integrand K (:1:, t, y(x), y( t)). Nonlinear Fredholm equations are
considerably more complicated than their linear counterparts. Fortunately, they clo
not occur as frequently in practice. By contrast, solving nonlinear Volterra equations
usually involves only a slight. modification of the algorithm for linear equations. Almost
all methods for solving iutegral equations numerically make use of quadrature rules,
frequently Gaussian quaclratures.

11.2. Method of successive approximations

For solving Fredholm equation ( 11.1.1) it is often used method of successive ap-
proximations based on eqnality

(11.2.1) Yn(:c) = f(:~:) +AI K(:.c, t)Y·n.-l(t) dt (n = 1, 2, ... ),

whereby is taken :uo = f(:r). Namdy, if we define sequence of functions {vd by using
I!

y 0(x) = :Uo(:c) = f(:r:), !h,(1:) = ./ K(:I:, t)yk_ 1 (t) dt (k = 1, 2 ... ),

a.

then (11.2.1) can be presented in the form

'().

(11.2.2) :Un(:c) = L Ah:Yk(x) (n = 1, 2, ... ).

h:=O

One can show that sequence :Un converges to exact solution of equation (11.1.1) if fulfilled
1
the condition 1>.1 < M(b _a) where

JVJ = max IK(1:, t)l

:r:,tE[a.,b].

11.3. Application of quad~ature formulas

In order to solve Fredholm equation (11.1.1) let's take quadrature formula

b. n

(11.3.1) J
.
F(:c) rl:r: = L AjF(xj) + Rn(F),
j=l
1/.

where abscissas .T 1 , ... , Xn are from [a, b], Aj are weight coefficients not dependi~g on
F, and Rn(F) corresponding remainder tern1. .
· If we put in (11.1.1) successively x = X-t (i = 1, ... , n), we obtam

y(:ci) = f(:r:i) +). . / K(.T, t)y(t) dt (1 = 1, ... , n),

().
 176 Numerical Methods in Computational Engineering

wherefrom by using quadrature formula (11.3.1) it follows

'/1.

(11.3.2) y(:z:.i) = f(:r:.;,) +A L AjK(:r:i, Xj) y(:r:j) + Rn(F) (i = 1, ... , n),

A:=l

where F.;.(t) = K(:r:.iyt) y(t) (-i = 1, ... , n.). By discarding members R.~~,(Fi) (1:
1, ... , n), based on (11.3.2) we get system of linear equations

n
(11.3.3) Yi- A L AJ<i.i:IJ.i =.f.,: (i = 1, ... , n),
·i=l

where we put :th = y(:r:.,:), .{.,: = f(:~:.,;), Ki.i K(:D.,:, :J:.i)· System (11.3.3) can also be

l[
given in matrix form

1- AA1K11 -AA2K12
. -AA1K21 1- AA2K22
-AA.,,.Kln :1J1]
-AAnK2n h
[
:Y2
.ill
[)n .
-AAlKnl 1- AA.nKnn Y:n

By solving the obtained system of linear equations in y 1 , ... , :IJn, the approximative
solution of equation (11.1.1) can he presented in the form

(11.3.4) :1)(:1:) = f(:J:) +A L AiK(:r, :J:.i) Y.i·

j=l

11.4. Program realization

Method explained in previous section will be realized by using generalizc>.cl Simpson
quadrature formula, at which we have

b-a
h = -- n = 2m. + 1, :r:.,: = a + ('i - 1) h ('i = 1, ... , n.) ,
2nr '

2h
A3 = As = ... = A2w.-1 = 3·
Fm solving system of linear <~qnat.ions (11.3.:3) we will use subroutines LRFAK and RSTS
The coll<~ of subrontilH~s and d<~scription of subrontirws parameters are given in Chapter
2.
In snbrontiw~ FRED is formed system of <xpmtions (11.3.~:q. Parmnet<~rs in snhron-
tinc parameter list <l.l'!~ of following nH~aning:
X - vector of a.bsciksas of quadrature forumla;
A - vector of weight codiici!~uts of qnadratnrc formula;
FK - name of hmction snlmmtim~ with hmction f and kemel K;
PL - pa.ntmd.!~r A:
C - matrix of syst.1~m ( 11.:3.:3), stored as vcd.m iu columuwise way (cohmm by
cohmm):
F- vectm of fn~~~ uwmlH~rs in system of ~~qnat.ion (11.3.3).
Sulmmtine code is of fmm:
 Lesson XI - Integral Eqnations 177

SUBRObTINE FRED(X,A,N,FK,PL,C,F)
DIMENSION X(1), A(1),C(1).,F(1)
IND=-N
DO 15 J=1,N
IND=IND+N
DO 10 I=1,N
IJ=IND+I
C(IJ)=-PL*A(J)*FK(X(I),X(J),2)
IF(I-1)10,5,10
5 C(IJ)=1+C(IJ)
10 CONTINUE
15 F(J)=FK(X(J),X(I),1)
RETURN
END
Function subroutine FK has a following parameters in the parameter list:
X and T - values of arguments :r: and t respectively.
M- integer which governs calculation of function f (M=1) and kernel K (M=2) for
given values of arguments. Subroutine code is of form:

FUNCTION FK(X,T,M)
GO TO (10,20), M
10 FK=EXP(X)
RETURN
20 FK=X*EXP(X*T)
RETURN
END
Main program is orga.nized 1n such a way that at first in FRED is formed system of
equation, and then is matrix of system factorized by subroutine LRF AK, what enables
solving of system of equations by subroutine RSTS.
Taking as an exampl<~ equation

1 .

y(:1:) = e"; - ./ :u:ct :y(t) dt

0

and M=1, 2 (N=3, 5), the corresponding results are obtained and presented below main
program code. Note that exact solution of given equation is y(x) = 1.

EXTERNAL FK
DIMENSION X(10) , A(10) , C(iOO) , B(10) , IP (9)
OPEN(8,FILE='FREI).IN')
OPEN(5,FILE='FRED.OUT')
READ(8,5)PL,DG,GG
5 FORMAT(3F5.0)
10 READ(8,15,END=60) M
15 FORMAT(I2)
N=2*M+1
H=(GG-DG)/(2.*FLOAT(M))
X(1) =DG
DO 20 I=2,N
20 X(I)=X(I-1)+H
Q=H/3.
A(l) =Q
A(N)=Q
DO 25 I=1,M
178 Numerical Methods in Computational Engineering

25 A(2*I)=4.*Q
DO 30 I=2,M
30 A(2*I-1)=2.*Q
CALL FRED(X,A,N,FK,PL,C,B)
CALL LRFAK(C,N,IP,DET,KB)
IF(KB) 35,40,35
35 WRITE(5,45)
45 FORMAT(1HO,'MATRICA SISTEMA SINGULARNA'//)
GO TO 60
40 CALL RSTS(C,N,IP,B)
WRITE(5,50)(B(I),I=1,N)
50 FORMAT(/5X, 'RESENJE'//(10F10.5))
GO TO 10
60 CLOSE(5)
CLOSE(8)
STOP
END

RESENJE
1.00000 0.94328 0.79472
RESENJE
1.00000 1.00000 1.00000 1.00000 0.99998

Bibliography (Cited references and further reading)

[1] Milovanovic, G.V., Numer·ical Analysis III, Nauc:na kr~jiga, Beograd, 1988 (Ser-
bian). ,
[2] Milovanovi(, G.V., Numerical Analysis I, Naucna knjiga, Beograd, 1988 (Ser-
bian).
[3] Milovanovic, G.V. aud Djonl.i<-~vi(, Dj.R, Programiranje numerickih metoda
na FORTRAN jez'iku. Iustitut z;;-t, <loknmentaciju zastite na raclu "Edvanl
Kmdelj", Nis, 1981 (Serbian).
[4] Milovanovic:, G.V., Numerical Analys·is II, Nauc:na knjiga, Beograd, 1988 (Ser-
hian).
[5] Press, W.I-L Flmmcry, B.P., T<~nkolsky, S.A., and Vetterling, W.T., Numer·ical
Recepies - The Art of Scientific Comp·uting. Cambridge University Press,
1989.
[6] Ddv<~s, L.J\!I., awl Molmme<l,.l.L., Computational Methods for Integral
Equations. Camhridgc University Press, Cambridge, 1985.
[7] Linz, P., Analytical and N·umerical Methods for Volterra Eqv,ations.
S.I.A.M., Philadelphia:. El85.
[8] Atkinson, K.E., A Survey of Numerical Methods for the Solution of Fred-
holm Integral Equat·ions of the Second Kind. S.I.A.M., Philadelphia, El7u.
[9] Smithies, F., Integral Equations. Cambridge University Press, Cmnbriclge,1\J58.
[10] Kanwal, R.P., Linear Integral Equations. Academic Press, N<~w York, 1971.
[11] Gr<~en, C.D., Integral Equation Methods . Dames & Noble, New York, 1%9,
[12] Stocr, .L awl Bnlirsch, R., Introduction to N·umerical Analys1:s, Springer,
New Ymk. 1D80.
[13] Ralston,A., A First Co·urse in Numerical Analysis. McGraw-Hill, New Ymk
Elu5.
[14] Hilrkbran<l. F.B .. Introd'U,ction to Numerical Analysis. Mc.Graw-HilL New
York 1974.
[1GJ Ac:tou. F.S .. Num,en:cal Methods That Work (corrected edition). Matlwlllat-
ictl Association of Alll<~rica, Washiup;tou, D.C., 1990.
 Lc~ssou XI - Iutegral Equations 17!)

[Hi] Kahmwr, D., Niokr, C., aud Nash, S., 1~8!), Numerical Methods and Soft-
ware. Euglc~wood Cliffs, Pn~llticc~ Hall. N.J. 1~8~.
[17] IMSL Math/L-ibrary Users Manual. Il\IISL Inc., 2500 City Wc~st. Bonlc~vard,
Houston TX 77042
[18] NAG Fortran L·ibrary, Nnuwrical Algorithms Group, 2SG Baulmry Road. Ox-
ford OX27DE, U.K., Chapter F02.