Lecture Notes in

Mathematics
Edited by A Dold and B. Eckmann

630

Numerical Analysis
Proceedings of the Biennial Conference

Held at Dundee, June 28-July 1, 1977

Edited by G.A Watson

Springer-Verlag

Berlin Heidelberg New York 1978

32166

""'" "'" 11111 111111111111111111

Editor
G. A. Watson
University of Dundee
Department of Mathematics
Dundee, DD1 4HN/Scotland

AMS Subject Classifications (1970): 65-02, 65D10, 65F05, 65F20,

65 K05, 65 L05, 65 L10, 65 M 99, 65 N 30,65 R05

ISBN 3-540-08538-6 Springer-Verlag Berlin Heidelberg New York

ISBN 0-387-08538-6 Springer-Verlag New York Heidelberg Berlin

This work is subject to copyright. All rights are reserved, whether the whole
or part of the material is concerned, specifically those of translation, re-
printing, re-use of illustrations, broadcasting, reproduction by photocopying
machine or similar means, and storage in data banks. Under § 54 of the
German Copyright Law where copies are made for other than private use,
a fee is payable to the publisher, the amount of the fee to be determined by
agreement with the publisher.
© by Springer-Verlag Berlin Heidelberg 1978
Printed in Germany
Printing and binding: Beltz Offsetdruck, Hemsbach/Bergstr.
2141/3140-543210
 Preface

For the 4 days June 28 - July i, 1977, over 220 people attended the 7th
Dundee Biennial Conference on Numerical Analysis at the University of Dundee,
Scotland. The technical program consisted of 16 invited papers, and 63 short
submitted papers, the contributed talks being given in 3 parallel sessions. This
volume contains, in complete form, the papers given by the invited speakers, and
a list of all other papers presented.

I would like to take this opportunity of thanking the speakers, including the
after dinner speaker at the conference dinner, Professor oD S Jones, all chairmen
and participants for their contributions. I would also like to thank the many
people in the Mathematics Department of this University who assisted in various
ways with the preparation for, and running of, this conference. In particular, the
considerable task of typing the various documents associated with the conference,
and some of the typing in this volume has been done by Miss R Dudgeon; this work
is gratefully acknowledged.

G A Watson

Dundee, September 1977.

 CONTENTS

C T H BAKER: Runge-Kutta methods for Volterra integral equations

of the second kind ................................................ I

I BARRODALE: Best approximation of complex-valued data ................ 14

D BOLEY and G H GOLUB: Inverse eigenvalue problems for band matrices .. 23

J S R CHISHOLM: Multivariate approximants with branch points ............ 32

L COLLATZ: The numerical treatment of some singular boundary value

problems ............................................................ 41

M G COX: The incorporation of boundary conditions in spline approxi-

mation problems ..................................................... 51

J DOUGLAS JR, T DUPONT and P PERCELL: A time-stepping method for

Galerkin approximations for nonlinear parabolic equations ........... 64

A GEORGE: An automatic one-way dissection algorithm for irregular

f--~nite element problems ........................................... 76

A R MITCHELL and D F GRIFFITHS: Generalised G a l e r k i n m e t h o d s for second

order equations with significant first derivative terms ........... 90

J J MORE: The Levenberg-Marquardt algorithm: Implementation and theory 105

M R OSBORNE and G A WATSON: Nonlinear approximation problems in vector

117
n o r m s ° . . , , , . . . , , . . . , . . , ° . . , , . , , , , , . , . , . . . , . . , . . , . , , , . , . .... . . . , . , .

V PEREYRA: Finite difference solution of two-point boundary value

133
problems and symbolic manipulation ................................

M J D POWELL: A fast algorithm for nonlinearly constrained optimization

calculations ...................................................... 144

R W H SARGENT: The decomposition of systems of procedures and

algebraic equations ................................................. 158

179
H J STETTER: Global error estimation in ODE-solvers ...................
190
E L WACHSPRESS: Isojacobic crosswind differencing .....................
 INVITED SPEAKERS

C T H Baker Department of Mathematics, University of Manchester,

Oxford Road, Manchester MI3 9PL, England.

I Barrodale Department of Mathematics, University of Victoria,

P.O. Box 1700, Victoria, B.C., Canada.

J S R Chisholm Mathematical Institute, The University, Canterbury,

Kent CT2 7NF, England.

L Collatz Institut fur Angewandte Mathematik, Universitat Hamburg,

2 Hamburg 13, Bundesstr 55, W Germany.

M G Cox Division of Numerical Analysis and Computing,

National Physical Laboratory, Teddington, Middlesex
TWII 0LW, England.

J Douglas, Jnr Department of Mathematics, The University of Chicago,

5734 University Avenue, Chicago, Illinois 60637, USA.

J A George Department of Computer Science, University of Waterloo,

Ontario, Canada.

G H Golub Computer Science Department, Stanford University,

Stanford, California 94305, USA.

D S Jones Department of Mathematics, University of Dundee,

Dundee DDI 4HN, Scotland.

A R Mitchell Department of Mathematics, University of Dundee,

Dundee DD] 4HN, Scotland.

J J More Applied Mathematics Division, Argonne National Laboratory,

9700 South Cass Avenue, Argonne, Illinois 60439, USA.

M R Osborne Computer Centre, Australian National University,

Box 4 P.O., Canberra, A.C.T. 2600, Australia.

V Pereyra Applied Mathematics I01-50, California Institute of

Technology, Pasadena, California 91125, USA.

M J D Powell Department of Applied Mathematics and Theoretical Physics,

University of Cambridge, Silver Street, Cambridge CB3 9EW,
England.

R W H Sargent Department of Chemical Engineering and Chemical

Technology, Imperial College, London SW7, England.

H J Stetter Institut fur Numerische Mathematik, Technische Hochschule

Wien, A-1040 Wien, Gusshausstr, 27-29 Austria.

E L Wachspress General Electric Company, P.O. Box I072, Schenectady,

New York 12301, USA.
 SUBMITTED PAPERS

Z Aktas: Computer Science Dept, Middle East Technical University, Turkey.

An accuracy improvement for the method of lines.

P Alfeld: Mathematics Dept, University of Dundee, Scotland.

CDS - A new technique for certain stiff systems of ordinary differential equations.

K Balla: Computer and Automation Institute, Hungarian Academy of Science.

On error estimates of the substitution of the boundedness condition on solutions of
systems of linear ordinary differential equations with regular singularity.

K E Barrett: Mathematics Dept, Lanchester Polytechnic, England.

The finite integral method for partial differential equations.

D G Bettis: Institute for Mathematics, Technical University of Munich, Germany.

An efficient embedded Runge-Kutta method.

Jean Beuneu: University of Lille I, France.

The rebalancing method for solving linear systems and eigenproblems.

Ake Bj~rck: Mathematics Dept, Linkoping University, Sweden.

Iterative solution of under- and overdetermined linear systems.

Klaus W A Bohmer: Mathematics Institute, University of Karlsruhe, Germany.

Defect corrections via neighbouring problems.

Claude Brezinski: University of Lille I, France.

Rational approximants to power series.

Hermann Brunner: Mathematics Department, Dalhousie University, Canada.

Volterra integral equations and their discretizations.

J P Coleman: Mathematics Dept, University of Durham, England.

Evaluation of the Bessel Functions J0 and Jl of complex argument.

I D Coope: Mathematics Dept, University of Dundee, Scotland.

Global convergence results for augmented Lagrangian methods.

G J Cooper: School of Math. and Physical Science, University of Sussex, England.

The order of convergence of linear methods for ordinary differential equations.

L J Cromme: Mathematics Dept, University of Bonn, Germany.

Numerical methods for nonlinear maximum norm approximations.

L M Delves: Dept of Comp and Statistical Science, University of Liverpool, England.

A global element method for the solution of elliptic partial differential equations.

P M Dew: Centre for Computer Studies, University of Leeds, England.

Numerical solution of quasi-linear heat problems with error estimates.

I S Duff: A.E.R.E. Harwell, England.

MA28 - a set of subroutines for solving sparse unsymmetric linear equations.

S Ellacott: Mathematics Dept, Brighton Polytechnic, England.

Practical complex best approximation: The state of the art.

C M Elliott: Computing Laboratory, Oxford University, England.

On the numerical solution of an electrochemical machining problem via a variational
inequality formulation.
G Elliott: Mathematics Dept, Portsmouth Polytechnic, England.
The construction of Chebyshev approximations in the complex plane.

R England and J P Hennart: Universidad Nacional de Mexico.

Fractional steps finite element techniques for strongly anisotropic diffusion
problems.

R Fletcher: Mathematics Dept, University of Dundee, Scotland.

The reduced Hessian in variable metric methods.

T L Freeman: Mathematics Dept., University of Manchester, England.

A method for computing the zeros of a polynomial with real coefficients.

Nima Geffen and Sara Yaniv: Tel-Aviv University, Israel.

Isoparametric characteristic elements for the Tricomi equation.

B Germain-Bonne, University of Lille I, France.

Shape and variation diminishing properties of spline curves.

Michael Ghil and Remesh Balgovind: Courant Institute of Mathematical Sciences,

New York University, USA.
A fast Cauchy-Riemann solver with nonlinear applications.

lan Gladwell: Mathematics Dept, University of Manchester, England.

The NAG library chapter for the solution of ordinary differential equations.

Moshe Goldberg: Mathematics Dept, University of California, USA.

Dissipative schemes for hyperbolic problems and boundary extrapolation.

R Gorenflo: Mathematics Dept, Freie Universitat Berlin, Germany.

Conservative difference schemes for diffusion problems.

Myron S Henry: Mathematics Dept, Montana State University, USA.

Numerical comparisons of algorithms for polynomial and rational multivariate
approximations.

J N Holt: Mathematics Dept, University of Queensland, Australia.

Free-knot cubic spline inversion of a Fredholm integral equation.

M K Horn: Institute for Mathematics, Technical University of Munich, Germany.

Developments in high-order Runge-Kutta-NystrSmmethods.

W D Hoskins, D S Meek, D J Walton: Dept of Computer Science, University of

Manitoba, Canada.
An alternative method for the solution of Poisson-type equations on Rectangular
Regions in two or three space dimensions.

K Jittorntrum, M R Osborne: Computer Centre, Australian National University.

Trajectory analysis and extrapolation in barrier function methods.

D C Joyce: Mathematics Dept, Massey University, New Zealand.

Extrapolation to the limit - algorithms and applications.

Bo Kagstrom: Dept of Information Processing and Numerical Analysis, University of

Umea, Sweden.
On the nnmerical computation of matrix functions.

Malcolm S Keech: Mathematics Dept, University of Manchester, England.

Semi-explicit methods in the numerical solution of first kind Volterra integral
equations.
 XI

R Kress: Universit~t G~ttingen and University of Strathclyde, Scotland.

On improving the rate of convergence of successive approximation for integral
equations of potential theory.

D P Laurie: National Research Institute for Mathematical Sciences, South Africa.

Exponentially fitted multipoint methods for two-polnt boundary value problems.

J D Lawson and J LI Morris: Computer Science Dept, University of Waterloo, Canada.

The extrapolation of first order methods for parabolic partial differential equations.

A V Levy*and A Montalvo+: *Universidad Nacional Aut~noma de MExico, +Universidad

Iberoamericana, MExico.
The tunneling algorithm for the global minimization of functions.

I M Longman: Dept of Geophysics and Planetary Sciences, Tel-Aviv University, Israel.

A method of Laplace transform inversion by exponential series.

Jens Lorenz: Institute for Numerical Mathematics, University of Munster, Germany.

Stability inequalities for discrete boundary value problems.

J T Marti: Mathematics Dept, Swiss Federal Institute of Technology.

An algorithm for the computation of Fourier coefficients of non-analytic functions
using B-splines of arbitrary order.

J C Mason: Mathematics Branch, Royal Military College of Science, Shrivenham,

England.
A one-dimensional spline approximation method for the numerical solution of heat
conduction problems.

S McKee: University of Oxford, England.

Multistep methods for solving linear Volterra integro-differential equations.

G Moore and A Spence: School of Mathematics, University of Bath, England.

Newton's method near a bifurcation point.

P Moore: Mathematics Dept, University of Aston in Birmingham, England.

Finite element multistep multiderivative schemes for linear parabolic equations.

Gerhard Opfer: Mathematics Dept, University of Hamburg, Germany.

Numerical solution of certain nonstandard approximation problems.

I Riddell: Dept of Computational and Statistical Science, University of Liverpool,

England.
On comparing integral equation routines.

A Robinson and A Prothero: Shell Research Limited, Chester, England.

Global error estimates for solutions to stiff systems of ordinary differential
equations.

J Barkley Rosser: Mathematics Research Center, University of Wisconsin-Madison, US~

Harmonic functions on regions with reentrant corners.

A Sayfy: School of Maths. and Physical Sciences, University of Sussex, England.

Additive numerical methods for ordinary differential equations.

J Sinclair: Mathematics Dept, University of Dundee, Scotland.

A variable metric method generating orthogonal directions.

H J J te Riele: Mathematical Centre, Amsterdam, Holland.

Computation of zeros of partial sums of the Riemann ~-function with real part > I.
 XIt

Per Grove Thomsen and Zahari Zlatev: Institute for Numerical Analysis, Technical
University of Denmark.
The use of Backward Differentiation methods in the solution of non-stationary heat
conduction problems.

Ph L Toint: F.N.D.P. Belguim.

On sparse and symmetric matrix updating subject to a linear equation.

J M Watt: Dept of Computational Science, University of Liverpool, England.

The convergence of deferred and defect corrections.

Richard Weiss: Technische Universitat, Wien, Austria.

On the eigenvalue problem for singular systems of ordinary differential equations.

B Werner: Mathematics Dept., University of Hamburg, Germany.

About a connection between complementary and nonconforming finite elements.

Ragnar Winther: Institute of Informatics, University of Oslo, Norway.

A Galerkin method for a parabolic control problem.

G Woodford: Mathematics Dept, University of Dundee, Scotland.

Isoparametric cubic triangles in the finite element method.

K Wright: Computing Laboratory, University of Newcastle upon Tyne, England.

Asymptotic properties of quadrature weig1~sbased on zeros of orthogonal polynomials
over partial and full ranges.
 THE LEVE_~BERG-MARQUARDT ALGORITHM:
IMPLEMENTATION AND THEORY

Jorge J. Mor~

i. Introduction

Let F: R n ÷ R m be continuously differentiable, and consider the nonlinear least

squares problem of finding a local minimizer of

(1.1) ~(x) = 7I f (x) = 71 IIFx)IL2

i=l
Levenberg [1944] and Marquardt [1963] proposed a very elegant algorithm for the
numerical solution of (i.i). However, most implementations are either not robust,
or do not have a solid theoretical justification. In this work we discuss a robust
and efficient implementation of a version of the Levenberg-Marqnardt algorithm, and
show that it has strong convergence properties. In addition to robustness, the main
features of this implementation are the proper use of implicitly scaled variables,
and the choice of the Levenberg-Marquardt parameter via a scheme due to Hebden
[1973]. Numerical results illustrating the behavior of this implementation are also
presented.

Notation. in all cases If" II refers to the £2 vector norm or to the induced operator
norm. The Jacobian matrix of F evaluated at x is denoted by F' (x), but if we have a
sequence of vectors {Xk} , then Jk and fk are used instead of F'(x k) and F(x k)
respectively.

2. Derivation

The easiest way to derive the Levenberg-Marquardt algorithm is by a lineariza-

tion argument. If, given x ~ R n, we could minimize

~(P) = IIF(x+p) II

as a function of p, then x+p would be the desired solution. Since P is usually a

nonlinear function of p, we linearize F(x+p) and obtain the linear least squares
problem

~(p) = IIF(x) + F'(x)pl I •

Of course, this linearization is not valid for all values of p, and thus we con-
sider the constrained linear least squares problem

Work performed under the auspices of the U.S. Energy Research and Development
Administration
 106

(2.1) min{~(p): [IDpll! A} .

In theory D is any given nonsingular matrix, but in our implementation D is a diago-

nal matrix which takes into account the scaling of the problem. In either case, p
lies in the hyperellipsoid

(2.2) E = {p: I!Dplli A} ,

but if D is diagonal, then E has axes along the coordinate directions and the length
of the ith semi-axis is A/d..
i

We now consider the solution of (2.1) in some generality, and thus the problem

(2.3) min{IIf+Jpll : IIDpll~ A}

where f ~ Rm and J is any m by n matrix. The basis for the Levenberg-Marquardt

method is the result that if p is a solution to (2.3), then p = p(l) for some
I > 0 where

(2.4) p(l) = _(jTj + IDTD)-IjTf .

If J is rank deficient and i = 0, then (2.4) is defined by the limiting process

Dp(0) E lim Dp(l) = -(jD-l)#f .

l÷0 +
There are two possibilities: Either % = 0 and IIDp(0)ll ! A, in which case p(0) is
the solution to (2.3) for which liNeN is least, or % > 0 and IIDp(%)II = A, and then
p(%) is the unique solution to (2.3).

The above results suggest the following iteration.

(2.5) Al$orithm

(a) Given Ak > 0, find %k ~ 0 such that if

(JJJk + %kDkTDk)Pk = -JkTfk '

then either Xk = 0 and IIDkPklI ~ A k, or %k > 0 and llDkPkll = A k •

(b) If IIF(Xk+P~I 1 < IIF(Xk)II set Xk+ I = Xk+P k and evaluate Jk+l; otherwise
set Xk+ I = x k and Jk+l = Jk"

(c) Choose gk+ I and Dk+ I.

In the next four sections we elaborate on how (2.5) leads to a very robust and
efficient implementation of the Levenberg-Marquardt algorithm.
 107

3. Solution of a Structured Linear Least Squares Problem

The s ~ p l e s t way to obtain the correction p is to use Cholesky decomposition on

the linear system

(3.1) (jTj + %DTD)p = _jTf .

Another method is to recognize that (3.1) are the normal equations for the least
squares problem

(3.2) p ~ - ,
0

and to solve this structured least squares problem using QR decomposition with
column pivoting.

The main advantage of the n o d a l equations is speed; it is possible to solve

(3.1) twice as fast as (3.2). On the other hand, the normal equations are particu-
larly unreliable when % = 0 and J is nearly rank deficient. Moreover, the f o ~ a t i o n
of jTj or DTD can lead to unnecessary underflows and overflows, while this is not
the case with (3.2). We feel that the loss in speed is more than made up by the
gain in reliability and robustness.

The least squares solution of (3.2) proceeds in two stages. These stages are
the same as those suggested by Golub (Osborne [1972]), but modified to take into
account the pivoting.

In the first stage, compute the QR decomposition of J with column pivoting.

This produces an orthogonal matrix Q and a permutation ~ of the columns of J such
that

where T is a nonsingular upper triangular matrix of rank (J) order. If X = 0, then

a solution of (3.2) is

p=~
I-°l
0 0
Qf~J-f

where J- refers to a particular symmetric generalized inverse of J in the sense

that JJ- is symmetric and JJ-J = J. To solve (3.2) when X > 0 first note that (3.3)
implies that

4 ilI] I
where D~ = x½~TD~ is still a diagonal matrix and R is a (possibly singular) upper

triangular matrix of order n.

 108

In the second stage, compute the QR decomposition of the matrix on the right of
(3.4). This can be done with a sequence of n(n+l)/2 Givens rotations. The result
is an orthogonal matrix W such that

(3.5)

where R x is a nonsingular upper triangular matrix of order n. The solution to (3.2)

is then

p = -~R~lu

where u ~ R n is determined from

W = .
0 V

It is important to note that if X is changed, then only the second stage must be

redone.

4. Updating the Step Bound

The choice of A depends on the ratio between the actual reduction and the pre-
dicted reduction obtained by the correction. In our case, this ratio is given by

(4.1) p(p) = L!F(x)ll2 -tIF(x+P)II2

ilF(x) II2 - lIF(x)+F'(x)P112 "

Thus (4.1) measures the agreement between the linear model and the (nonlinear) func-
tion. For example, if F is linear then p(p) = i for all p, and if F'(x)TF(x) # 0,
then p(p) + 1 as lien ~ o. Moreover, if !IF(x+p)II t IIF(x)II then p(p) ! 0.

The scheme for updating A has the objective of keeping the value of (4.1) at a
reasonable level. Thus, if p(p) is close to unity (i.e. p(p) ~ 3/4), we may want to
increase &, but if p(p) is not close to unity (i.e. p(p) ! 1/4), then A must be
decreased. Before giving more specific rules for updating A, we discuss the compu-
tation of (4.1). For this, write

IIfN2 -Ilf+II 2
(4.2) p =
[If[l2 - Hf+jpll 2

with an obvious change in notation. Since p satisfies (3.1),

(4.3) Hfll2 -[If+JpN 2 = HJpI[2 + 2XnDp[I 2 ,

and hence we can rewrite (4.2) as

 109

11f+l[] 2
1- ~i-F~J
(4.4) p = IllJPIll2 [~ll] 2
t~j + 2

Since (4.3) implies that

llJPll <_ llfll, Z~IDpll <_ Ilfll,

the computation of the denominator will not generate any overflows, and moreover,
the denominator will be non-negative regardless of roundoff errors. Note that this
is not the case with (4.2). The numerator of (4.4) may generate overflows if llf+[I
is much larger than llfII, but since we are only interested in positive values of p,
if Ilf+II > Ilfl] we can just set p = 0 and avoid (4.4).

We now discuss how to update A. To increase A we simply multiply A by a con-

stant factor not less than one. To decrease A we follow Fletcher [1971] and fit a
quadratic to 8(0), ~'(0) and 8(1) where

8(0) = ~1 llF(x+ep)Ii2

If ~ is the minimizer of the resulting quadratic, we decrease A by multiplying A by

#.
D, but if ~ # I~0' ~I' we replace p by the closest endpoint. To compute safely,
first note that (3.1) implies that

y _= pTjTf [[ l[fllj j '

and that y ~ [-i,0]. It is now easy to verify that

1
~Y
(4.5) ~ =
+½[1 [Hf421
Urn-F] J
If IIf+lI <_ llfll we set N = 1/2. Also note that we only compute U by (4.5) if say,
llf+lJ < lOIIf]], for otherwise, u < i/i0.

5. The Levenberg-Marquardt Parameter

In our implementation ~ > 0 is accepted as the Levenberg-Marquardt parameter if

(5.1)
where

(5.2) $(~) = I[D(jTj+~DTD)-IjTfll- A ,

and o e (0,i) specifies the desired relative error in IIDp(~)II. Of course, if

 110

¢(0) ! 0 then ~ = 0 is the required parameter, so in the remainder of this section

we assume that ¢(0) > 0. Then # is a continuous, strictly decreasing function on
[0,+~o) and ¢(~) approaches -A at infinity. It follows that there is a unique
> 0 such that ¢(6 ) = 0. To determine the L e v e n b e r g - M a r q u a r d t parameter we
assume that an initial estimate 60 > 0 is available, and generate a sequence {~k}
which converges to ~ .

Since ~ is a convex function, it is very tempting to use Newton's method to

generate {Ok} , but this turns out to be very inefficient -- the particular structure
of this problem allows us to derive a m u c h more efficient iteration due to Hebden
[1973]. To do this, note that

(5.3) ¢(6) = II(~TG+6n-Z~Tfll A, ~ = JD -I

and let G = UEV T be the singular value decomposition of ]. Then

2 2 ~½
° i zi
,

where z = uTf and oi,...,o n are the singular values of ~. Hence, it is very natural
to assume that

• a A ~ 7(6)
¢ (~) b + 6

and to choose a and b so that ¢ ( ~ k ) = ¢(~k ) and 7'(~ k) = ¢'(~k ). Then ~(~k+l ) = 0
if

(5.4) k+l "

This iterative scheme must be safeguarded if it is to converge. Hebden [1973] pro-

posed using upper and lower bounds u k and £k' and that (5.4) be applied with the
restriction that no iterate may be within (Uk-q)/10 of either endpoint. It turns
out that this restriction is very detrimental to the progress of the iteration since
in a lot of cases u k is much larger than £k" A m u c h more efficient algorithm can
be obtained if (5.4) is only modified when 6k+ 1 is outside of ( ~ + l , U k + l ) . To
specify this algorithm we firs't follow Hebden [1973] and note that (5.3) implies
that
= [I(JD-I)Tfll
Uo A

is a suitable upper bound. If J is not rank deficient, then ~'(0) is defined and
the convexity of ¢ implies that
 111

is a lower bound; otherwise let £0 = O.

(5.5) Algorithm

(a) If ~k ~ (~'Uk) let ~k = max{0.001 Uk, (£kUk)½}.

(b) Evaluate ~(0~k) and ~'(ek). Update u k by letting uk# 1 = ~k if ~(ak) < 0
and Uk+ 1 = u k otherwise. Update £k by
~(o k)
Ik+l = maxllk' ~k ~'(~k) } "

(c) Obtain ak+l from (5.4).

The role of (5.5)(a) is to replace ak by a point in (£k,Uk) which is biased

towards £k; the factor 0.001 u k was added to guard against exceedingly small values
of £k' and in particular, ~ = O. In (5.5)(b), the convexity of ~ guarantees that
the Newton iterate can be used to update ~ .

It is not too difficult to show that algorithm (5.5) always generates a

sequence which converges quadratically to a . In practice, less than two iterations
(on the average) are required to satisfy (5.1) when o = 0.i.

To complete the discussion of the Hebden algorithm, we show how to evaluate

~'(~). From (5.2) it follows that

(DT~(~))T(jTj+aDTD)-I(DTq(~))
~'(~) ............ II~(~)ll

where q(a) = Dp(a) and p(.) is defined by (2.4). From (3.4) and (3.5) we have

~T(jTj+~DTD)~ = R TR
and hence,
oTI 112
6. Scaling

Since the purpose of the matrix D k in the Levenberg-Marquardt algorithm is to

take into account the scaling of the problem, some authors (e.g. Fletcher [1971])
choose

(6.1) D k = diag(dl(k) ..... dn(k) )

where

(6.2) di(k) = ]l~iF(x0)II, k ~ 0 ,

This choice is usually adequate as long as II~iF(Xk)II does not increase with k. How-
ever, if ll~iF(Xk)II increases, this requires a decrease in the length (= ~/di) of the
.th
z semi-axis of the hyperellipsoid (2.2), since F is now changing faster along the
 "i12

.th .th
l variable, and therefore, steps which have a large I component tend to be un-
reliable. This argument leads to the choice

(6.3) di(O) = II$iF(xo) II

di(k) = maxldi(k-l), ll3iF(Xk)H} , k I>_ .

Note that a decrease in II$iF(Xk)!I only implies that F is not changing as fast along
the i th variable, and hence does not require a decrease in d.. In fact, the choice
i

(6.4) d~l (k) = II~iF(Xk) II , k t 0 ,

is computationally inferior to both (6.2) and (6.3). Moreover, our theoretical re-
sults support choice (6.3) over (6.4), and to a lesser extent, (6.2).

It is interesting to note that (6.2), (6.3), and (6.4) make the Levenberg-
Marquardt algorithm scale invariant. In other words, for all of the above choices,
if D is a diagonal matrix with positive diagonal elements, then algorithm (2.5) gen-
erates the same iterates if either it is applied to F and started at x 0, or if it is
applied to F(x) = F(D-Ix) and started at x0 = Dx0" For this result it is assumed
that the decision to change A is only based on (4.1), and thus is also scale
invariant.

7. Theoretical Results

It will be sufficient to present a convergence result for the following version

of the Levenberg-Marquardt algorithm.

(7.1) Al$orithm

(a) Let o ~ (0,i). If IIDkJkfklI ~ (l+o)A k, set X k = 0 and Pk = -Jkfk"

Otherwise determine %k > 0 such that if

then

(I-o)A k ~ HDkPkll ~ (I+o)A k

(b) Compute the ratio Ok of actual to predicted reduction.

(c) If O k ! 0.0001, set Xk+ 1 = x k and Jk+l = Jk"

If Pk > 0.0001, set Xk÷ 1 = xk+P k and compute Jk+l"
1
(d) If O k ! 1/4, set Ak+l ~ [i~ Ak' ~ Ak]"

If Pk ~ [¼, ¼J and Xk = 0, or if pk 3>/ 4 , _ set Ak+ I = 211mkPkll •

 113

(e) Update Dk+ I by (6.1) and (6.3).

The proof of our convergence result is somewhat long and will therefore be pre-
sented elsewhere.

Theorem. Let F: R n + R TM be continuously differentiable on R n, and let {Xk} be the

sequence generated by algorithm (7.1). Then

(7.2) lim inf '"ll(JkDk-l)Tfkl1 = 0 i

k÷+~

This result guarantees that eventually a scaled gradient will be small enough.
Of course, if {Jk} is bounded then (7.2) implies the more standard result that

(7.3) lim inf IIJkTfk H = 0 I

k÷+~

Furthermore, we can also show that if F' is uniformly continuous then

(7.4) lim IIJkTfkll = 0 i

k÷q~o

Powell [1975] and Osborne [1975] have also obtained global convergence results
for their versions of the Levenberg-Marquardt algorithm. Powell presented a general
algorithm for unconstrained minimization which as a special case contains (7.1) with
o = 0 and {Dk} constant. For this case Powell obtains (7.3) under the assumption
that {Jk} is bounded. Osborne's algorithm directly controls {Ik} instead of {Ak},
and allows {Dk} to be chosen by (6.1) and (6.3). For this case he proves (7.4)
under the assumptions that {Jk} and {%k} are bounded.

8. Numerical Results

In our numerical results we would like to illustrate the behavior of our algo-
rithm with the three choices of scaling mentioned in Section 6. For this purpose,
we have chosen four functions.

1) Fletcher and Powell [1963] n=3, m=3

fl(x) = 10[x 3 - 108(Xl,X2) ]

f2(x ) = lO[(Xl~+X 2 )2 _ i]

f3(x) = x 3
where
f! (x2/xl), Xl > 0
e(Xl,X2) = ~2~ arctan

/--[~ arctan (x2/xl) + 0.5, Xl < 0

x 0 = (-i,0,0) T
 114

2. Kowalik and Osborne [1968] n=4, m=ll

2
Xl[U i + x2u i]
fi(x) Yi
(ui 2 + x3u i + x 4)

where u i and Yi are specified in the original paper.

x 0 = (0.25, 0.39, 0.415, 0.39) T

3. Bard [1970] n=3, m=15

f i ( x ) = Yi - 1 + x 2 v i + x3w i

where u i = i, v i = 16-i, w.l = min~ui,vi},~, and Yi is specified in the

original paper.

x o = (i,i,i) T

4. Brown and Dennis [1971] n=4, m=20

fi(x) = [x I + x2t i - exp(ti)]2 + [x 3 + x4sin(t i) - cos(ti)] 2

where t. = (0.2)i.
1
x 0 = (25, 5, -5, i) T

These problems have very interesting features. Problem 1 is a helix with a

zero residual at x = (i,0,0) and a discontinuity along the plane x I = 0; note that
the algorithm must cross this plane to reach the solution. Problems 2 and 3 are
data fitting problems with small residuals, while Problem 4 has a large residual.
The residuals are given below.

i. lIF(x*)II = 0.0
2. IIF(x~)II 0.0175358
3. IIF(x~)II 0.0906359
4. llF(x~) I] 292. 9542

Problems 2 and 3 have other solutions. To see this, note that for Kowalik and
Osborne 's function,

(8.1) lim fi(~,x2,~,~) = Yi - (x2+ui) '

~->oo
while for Bard's function,

(8.2) lim fi(xl,e,~) = Yi - Xl "

These are now linear least squares problems, and as such, the parameter x 2 in (8.1)
and x I in (8.2) are completely determined. However, the remaining parameters only
need to be sufficiently large.

In presenting numerical results one must be very careful about the convergence
criteria used. This is particularly true of the Levenberg-Marquardt method since,
unless F(x*) = 0, the algorithm converges linearly. In our implementation, an
approximation x to x is acceptable if either x is close to x or IIF(x)II is close
 115

to IIF(x*)II . We attempt to satisfy these criteria by the convergence tests

(8.3) A ~ XTOL IIDxll ,

and
Ij~~ H I 2 [ 2
(8.4) IIfIU + 2 %½ ~ I <__ FTOL .

An important aspect of these tests is that they are scale invariant in the sense of
Section 6. Also note that the work of Section 4 shows that (8.4) is just the rela-
tive error between IIf+Jpll2 and IIf~2.

The problems were run on the IBM 370/195 of Argonne National Laboratory in dou-
ble precision (14 hexadecimal digits) and under the FORTRAN H (opt=2) compiler. The
tolerances in (8.3) and (8.4) were set at FTOL = 10 -8 and XTOL = 10 -8 . Each problem
is run with three starting vectors. We have already given the starting vector x 0
which is closest to the solution; the other two points are 10x 0 and lOOx 0. For each
starting vector, we have tried our algorithm with the three choices of {Dk}. In the
table below, choices (6.2), (6.3) and (6.4) are referred to as initial, adaptive,
and continuous sealing, respectively. Moreover, NF and NJ stands for the number of
function and Jacobian evaluations required for convergence.

x0 10x 0 100x 0
L
PROBLEM SCALING I NF NJ I NF NJ NF NJ

Initial 12 9 34 29 FC FC
1 Adaptive ii 8 20 15 19 16
Continuous 12 9 14 12 176 141

Initial 19 17 81 71 365 315

2 Adaptive 18 16 79 71 348 307
Continuous 18 16 63 54 FC FC

t Initial 8 7 37 36 14 13
3 Adaptive 8 7 37 36 14 13
Continuous 8 7 FC FC FC FC

Initial 268 242 423 400 FC FC

4 Adaptive 268 242 57 47 229 207
Continuous FC FC FC FC FC FC

Interestingly enough, convergence to the minimizer indicated by (8.1) only

occurred for starting vector 10x 0 of Problem 2, while for Problem 3 starting vec-
tors 10x 0 and lOOx 0 led to (8.2). Otherwise, either the global minimizer was ob-
tained, or the algorithm failed to converge to a solution; the latter is indicated
by FC in the table.

it is clear from the table that the adaptive strategy is best in these four
examples. We have run other problems, but in all other cases the difference is not
as dramatic as in these cases. However, we believe that the above examples ade-
quately justify our choice of scaling matrix.
 116

Acknowledgments. This work benefited from interaction with several people. Beverly
Arnoldy provided the numerical results for several versions of the Levenberg-
Marquardt algorithm, Brian Smith showed how to use pivoting in the two-stage process
of Section 3, and Danny Sorensen made many valuable comments on an earlier draft of
this paper. Finally, I would like to thank Judy Beumer for her swift and beautiful
typing of the paper.

References

i. Bard, Y. [1970]. Comparison of gradient methods for the solution of nonlinear

parameter estimation problem, SIAM J. Numer. Anal. 7, 157-186.

2. Brown, K. M. and Dennis, J. E. [1971]. New computational algorithms for mini-

mizing a sum of squares of nonlinear functions, Department of Computer Science
report 71-6, Yale University, New Haven, Connecticut.

3. Fletcher, R. [1971]. A modified Marquardt subroutine for nonlinear least

squares, Atomic Energy Research Establishment report R6799, Harwell, England.

4. Fletcher, R. and Powell, M.J°D. [1963]. A rapidly convergent descent method

for minimization, Comput. J. 6, 163-168.

5. Hebden, M. D. [1973]. An algorithm for minimization using exact second deriva-

tives, Atomic Energy Research Establishment report TP515, Harwell, England.

6. Kowalik, J. and Osborne, M. R. [1968]. Methods for Unconstrained Optimization

Problems, American Elsevier.

7. Levenberg, K. [1944]. A method for the solution of certain nonlinear problems

in least squares, Quart. Appl. Math. 2, 164-168.

8. Marquardt, D. W. [1963]. An algorithm for least squares estimation of non-

linear parameters, SIAM J. Appl. Math. ii, 431-441.

9. Osborne, M. R. [1972]. Some aspects of nonlinear least squares calculations,

in Numerical Methods for Nonlinear optimization, F. A. Lootsma, ed., Academic
Press.

i0. Osborne, M. R. [1975]. Nonlinear least squares - the Levenberg algorithm re-
visited, to appear in Series B of the Journal of the Australian Mathematical
Society.

ii. Powell, M. J. D. [1975]. Convergence properties of a class of minimization

algorithms, in NonlinearProgramming 2, O. L. Mangasarian, R. R. Meyer, and
S. M. Robinson, eds., Academic Press.