Downloaded from https://iranpaper.ir http://www.master-thesis.

ir/Employment
[email protected]

Applied Mathematics and Computation 189 (2007) 1902–1915

www.elsevier.com/locate/amc

A new method for optimal control

of Volterra integral equations
S.A. Belbas
Mathematics Department, University of Alabama, Tuscaloosa, AL 35487-0350, USA

Abstract

We formulate and analyze a new method for solving optimal control problems for systems governed by Volterra inte-
gral equations. Our method utilizes discretization of the original Volterra controlled system and a novel type of dynamic
programming in which the Hamilton–Jacobi function is parametrized by the control function (rather than the state, as in
the case of ordinary dynamic programming). We also derive estimates for the computational cost of our method.
Ó 2007 Elsevier Inc. All rights reserved.

Keywords: Optimal control; Volterra integral equation; Discrete approximation

1. Introduction

The classical theory of optimal control was originally developed to deal with systems of controlled ordinary
differential equations. It has been understood that many physical, technological, biological, and socio-eco-
nomic problems cannot be adequately described by ordinary differential equations, and other mathematical
models, including systems with memory, distributed systems, and other types of systems, have been added
to the arsenal of the theory of optimal control. A broad category of systems can be described by Volterra inte-
gral equations.
The simplest form of a controlled Volterra integral equation is
Z t
xðtÞ ¼ x0 ðtÞ þ f ðt; s; xðsÞ; uðsÞÞ ds: ð1:1Þ
0

In this system, x(t) is the n-dimensional state function, and u(t) is the m-dimensional control function. For the
purposes of this exposition, we postulate that f be continuous with respect to all variables and uniformly Lips-
chitz with respect to x. For the purposes of describing the necessary conditions that are briefly reviewed in this
section, the admissible control functions are continuous functions with values in a compact set U, U  Rm . In
certain parts of the overall theory of optimal control for Volterra integral equations, the class of admissible
control functions can be more general, for example it may consist of bounded measurable or p-integrable

E-mail address: [email protected]

0096-3003/$ - see front matter Ó 2007 Elsevier Inc. All rights reserved.
doi:10.1016/j.amc.2006.12.077
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1903

functions. On the other hand, as it will be explained below, for some of the results of the present paper it is
necessary to further restrict the class of admissible controls and postulate Lipschitz continuity.
Volterra integral equations arise in a wide variety of applications. In fact, it seems that, with the exception
of the simplest physical problems, practically every situation that can be modelled by ordinary differential
equations can be extended to a model with Volterra integral equations. For example, a general ODE system
of interacting biological populations, of the form
dxi ðtÞ X X
¼ aijk xj ðtÞxk ðtÞ þ bij xj ðtÞ; xi ð0Þ ¼ xi;0
dt j;k j

can be extended to an integro-differential system

dxi ðtÞ X X
¼ aijk xj ðtÞxk ðtÞ þ bij xj ðtÞ
dt j;k j
(
Z t X )
X
þ Aijk ðt; sÞxj ðsÞxk ðsÞ þ Bij ðt; sÞxj ðsÞ ds; xi ð0Þ ¼ xi;0 :
0 j;k j

Indeed, some related extensions have already been considered in [3] and in other works. In turn, every inte-
grodifferential system of the form
 Z t 
dxðtÞ
¼ g t; xðtÞ; f ðt; s; xðsÞÞ ds ; xð0Þ ¼ x0
dt 0
Rt
can be reduced to a system of Volterra integral Rt equations by setting yðtÞ ¼ 0 f ðt; s; xðsÞ ds; then the integro-
differential equation becomes xðtÞ ¼ x0 þ 0 gðs; xðsÞ; yðsÞÞ ds, so that we get a system of Volterra integral equa-
tions in the unknowns (x(t), y(t)).
Problems in mathematical economics also lead to Volterra integral equations. The relationships among dif-
ferent quantities, for example between capital and investment, include memory effects (e.g. the present stock of
capital depends on the history of investment strategies over a period of time, cf. [4]), and the simplest way to
describe such memory effects is through Volterra integral operators.
Now we return to the general model of state dynamics (1.1).
An optimal control problem for (1.1) concerns the minimization of a cost functional
Z T
J :¼ F 0 ðxðT ÞÞ þ F ðt; xðtÞ; uðtÞÞ dt: ð1:2Þ
0

The theory of optimal control of ordinary differential equations has two main methods: the extremum (usually
called ‘‘maximum’’) principle of Pontryagin and his coworkers, and the method of dynamic programming. The
method of dynamic programming is particularly useful as it provides sufficient conditions for optimality. How-
ever, the nature of controlled Volterra equations is not, at first glance, conducive to the application of dynamic
programming methods. If the state x(t) is known at some particular time t, and a control function is specified
over an interval ðt; t þ dt, these two bits of information are not enough for the determination of the solution of
(1.1) over the interval ðt; t þ dt. By contrast, for ordinary differential equations, it is always true that, given x(t)
and a control function over ðt; t þ dt, the trajectory over ðt; t þ dt can be determined by solving an initial value
problem for an ordinary differential equation with initial time t. For these reasons, optimal control problems
for Volterra integral equations have been traditionally treated by extensions of Pontryagin’s extremum princi-
ple. The related results are found in a number of papers, including [5,7,8,6]; an approach based on direct var-
iational methods, but still utilizing necessary conditions for optimality, may be found in [2].
The co-state w(t) for the problem consisting of (1.1) and (1.2) satisfies the following adjoint equation, which
is the counterpart of Hamiltonian equations (see, e.g., [7,6]):
Z T
wðtÞ ¼ rx F ðt; xðtÞ; uðtÞÞ þ ðrx F 0 ðxðT ÞÞÞrx f ðT ; t; xðtÞ; uðtÞÞ þ wðsÞrx f ðs; t; xðtÞ; uðtÞÞ ds: ð1:3Þ
t

The state x(Æ) is an n-dimensional column vector; the function f takes values that are n-dimensional column
vectors; the co-state w(Æ) is an n-dimensional row vector. The gradient, with respect to x, of a scalar-valued
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

1904 S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915

ofi ofi
function is an n-dimensional row vector; $xf is a matrix with elements oxj
(each oxj
is the element in the ith row
and jth column of $xf). It should be noted that the adjoint equations in [7 and 6] are not exactly identical to
(1.3), due to the fact that these authors do not use exactly the same cost functional as (1.2); however, these
differences do not require substantially different proofs, and for that reason we present (1.3) without proof.
The Hamiltonian is defined by
Z T
H ðt; x; wðÞ; uÞ :¼ F ðt; x; uÞ þ ðrx F 0 ðxðT ÞÞÞf ðT ; t; x; uÞ þ wðsÞf ðs; t; x; uÞ ds: ð1:4Þ
t

The extremum principle takes the following form [6]: under certain smoothness conditions, an optimal control
u*(Æ) and the corresponding trajectory and co-state, x*(Æ) and w*(Æ), respectively, satisfy, for almost all t,
H ðt; x ðtÞ; w ðÞ; u ðtÞÞ 6 H ðt; x ðtÞ; w ðÞ; uÞ 8u 2 U: ð1:5Þ
In this paper, we shall need the concept of what we shall term the relevant values of the state x(Æ). This is not
standard terminology, but is useful for our purposes. Under certain conditions, it is possible to estimate the
range of the solution x(t), without relying on actually solving (1.1). For example, if the function f has linear
growth rate with respect to x, i.e. if
jf ðt; s; x; uÞj 6 G1 jxj þ G0 ; for all t; s : 0 6 s 6 t 6 T; for all u 2 U: ð1:6Þ
At first, we do not specify the set of values of x for which (1.6) should hold; we tentatively carry out the cal-
culations as if (1.6) were true for all x in Rn , until we find an estimate for the set Xrel of relevant values of x(Æ),
then we go back and postulate that (1.6) should hold for all x in Xrel.
Then it follows from (1.1) that every solution x(Æ) satisfies:
Z t
jxðtÞj 6 kx0 k1 þ TG0 þ G1 jxðsÞj ds; kx0 k1 :¼ sup jx0 ðtÞj ð1:7Þ
0 06t6T

and therefore, by Gronwall’s inequality,

jxðtÞj 6 ðkx0 k1 þ TG0 Þ expðG1 T Þ 8 2 ½0; T : ð1:8Þ
In this case, the set Xrel of relevant values of x(Æ) is
X rel :¼ fx 2 Rn : jxj 6 ðkx0 k1 þ TG0 Þ expðG1 T Þg: ð1:9Þ
Thus the conclusion is that, if (1.6) holds for all x in the set Xrel given by (1.9), then the solution x(t) always lies
in that Xrel.
It should be emphasized that (1.9) is an example of one case of a set of relevant values, and not the defi-
nition of Xrel in general.
Another example of estimating Xrel is the case in which f satisfies a Lipschitz condition
jf ðt; s; x1 ; uÞ  f ðt; s; x2 ; uÞj 6 Lf ;1 jx1  x2 j; for all t; s : 0 6 s 6 t 6 T; for all u 2 U ð1:10Þ
and for a particular value of x, which we may take, without loss of generality, to be 0, the function f remains
bounded:
jf ðt; s; 0; uÞj 6 G2 ; for all t; s : 0 6 s 6 t 6 T; for all u 2 U: ð1:11Þ
n
As before, we work at first as if (1.10) were true for all x1 ; x2 in R , then we determine a suitable set Xrel, and
then we return and postulate that (1.10) should hold for all x1 ; x2 in Xrel.
Now, we have
Z t Z t Z t
jxðtÞj  jf ðt; s; 0; uðsÞÞj ds 6 jxðtÞ  f ðt; s; 0; uðsÞÞ dsj 6 jx0 ðtÞj þ Lf ;1 jxðsÞj ds ð1:12Þ
0 0 0

thus
Z t
jxðtÞj 6 G2 T þ kx0 k1 þ Lf ;1 jxðsÞj ds ð1:13Þ
0
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1905

and, again by Gronwall’s inequality,

jxðtÞj 6 ðG2 T þ kx0 k1 Þ expðLf ;1 T Þ 8t 2 ½0; T  ð1:14Þ
so that, in this case, we can take
X rel :¼ fx 2 Rn : jxj 6 ðG2 T þ jjx0 jj1 Þ expðLf ;1 T Þg: ð1:15Þ
Thus, if (1.10) and (1.11) hold for all x1 ; x2 in the set Xrel given by (1.15), then all values of x(t) lie in that set
Xrel.
It should be noted that there are many other possibilities of finding examples of Xrel under suitable assump-
tions, but, in this paper, we are not interested in exhausting this topic.
In the rest of this paper, we shall assume the existence of a bounded set X rel  Rn , without specifying how
that set has been determined.

2. The discrete Volterra control problem

Our approach will be to approximate the original Volterra control problem by a sequence of analogous
control problems for discrete Volterra equations. For this reason we need to have a method for solving opti-
mal control problems for discrete Volterra equations. Thus we consider, in this section, the controlled Volterra
equation in discrete time:
X
i1
xðiÞ ¼ x0 ðiÞ þ uði; j; xðjÞ; uðjÞÞ; 1 6 i 6 N : ð2:1Þ
j¼0

The discrete optimal control problem concerns the minimization of a functional J given by
X
N 1
J :¼ Uði; xðiÞ; uðiÞÞ þ U0 ðxðN ÞÞ: ð2:2Þ
i¼0

In order to apply a suitable variant of the dynamic programing method to the problem consisting of (2.1) and
(2.2), we need to build a parametrization of this optimal control problem. The expression ‘‘suitable variant’’
refers to the fact that, for the problem under consideration, the value function needs to be parametrized by
current time and history of the control up to the current time, whereas in classical dynamic programming
the value function is parametrized by current time and current value of the state. The memory effect of Vol-
terra equations necessitates this seemingly unorthodox parametrization.
We set
uðiÞ :¼ ðuðjÞ : 0 6 j 6 i  1Þ:
~ ð2:3Þ
uðiÞ Þ; 0 6 k 6 i is the solution of the discrete Volterra equation
If xðk; i; ~
X
k1
xðkÞ ¼ x0 ðkÞ þ uðk; j; xðjÞ; uðjÞÞ; 0 6 k 6 i  1; ð2:4Þ
j¼0

we define
x½i  x½i ði; ~
uðiÞ Þ :¼ ðxðk; i; ~
uðiÞ Þ : 0 6 k 6 iÞ: ð2:5Þ
The concatenation of b ¼ ðbðjÞ : 0 6 j 6 i  1Þ and c ¼ ðcðjÞ : i 6 j 6 N  1Þ is defined as
k  b  c; kðjÞ ¼ bðjÞ for j ¼ 0; 1; . . . ; i  1; kðjÞ ¼ cðjÞ for j ¼ i; i þ 1; . . . ; N  1: ð2:6Þ
The restriction of a control to indices that exceed i  1 will be denoted by uhii:
uhii ¼ ðuðjÞ : i 6 j 6 N  1Þ: ð2:7Þ
The cost functional J is parametrized as
X
N 1
J i;b ðuhii Þ :¼ Uðj; xðj; i; b  uhii Þ; uðjÞÞ þ U0 ðxðN ; i; b  uhii ÞÞ; ð2:8Þ
j¼i
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

1906 S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915

where b ¼ ðbðjÞ : 0 6 j 6 i  1Þ, uhii ¼ ðuðjÞ : i 6 j 6 N  1Þ, and xðk; i; b  uhii Þ; i 6 k 6 N  1 solves
Xi X
k1
xðkÞ ¼ x0 ðkÞ þ uðk; j; xðj; i; bÞ; bðjÞÞ þ uðk; j; xðjÞ; uðjÞÞ: ð2:9Þ
j¼0 j¼iþ1

In particular, we note that xði; i; b  uhii Þ ¼ xði; i; bÞ.

The value function V ði; bÞ, b ¼ ðbðjÞ : 0 6 j 6 i  1Þ, is defined in terms of the parametrization (2.8):
V ði; bÞ :¼ min J i;b ðuhii Þ: ð2:10Þ
uhii

For i ¼ 0, the collection ðbðjÞ : 0 6 j 6 i  1Þ is empty; we therefore use a symbolic ‘‘empty set’’ ; in the func-
tion V, and that function becomes, when i ¼ 0, V(0,;).
For n 2 U , we identify b  n with the control b ^ :¼ ðbð0Þ; bð1Þ; . . . ; bði  1Þ; nÞ so that bðiÞ
^ ¼ n. With this
notational convention, the dynamic programming equations for V are
V ði; bÞ ¼ minfV ði þ 1; b  nÞ þ Uði; xði; i; bÞ; nÞg; V ðN ; bÞ ¼ U0 ðxðN ; N ; bÞÞ: ð2:11Þ
n2U

Next, we prove the necessity and sufficiency of (2.10).

We have:
Theorem 2.1. Eq. (2.11) is necessary for optimality, i.e. if V ði; bÞ is defined by (2.10), then it satisfies (2.11).

Proof. According to (2.10), we have, for every control uhi+1i,

V ði þ 1; b  nÞ 6 J iþ1;bn ðuhiþ1i Þ: ð2:12Þ
Therefore, for every n 2 U, we have
minfV ði þ 1; b  nÞ þ Uði; xði; i; bÞ; nÞg 6 V ði þ 1; b  nÞ þ Uði; xði; i; bÞ; nÞ
n2U

6 J iþ1;bn ðuhiþ1i Þ þ Uði; xði; i; bÞ; nÞ ¼ J i;b ðn  uhiþ1i Þ: ð2:13Þ

Since every uhii can be represented as n  uhiþ1i for some n 2 U, we have
V ði; bÞ ¼ min J i;b ðuhii Þ ¼ min min J i;b ðn  uhiþ1i Þ
uhii n2U uhiþ1i

¼ minfV ði þ 1; b  nÞ þ Uði; xði; i; bÞ; nÞg:  ð2:14Þ

n2U

Theorem 2.2. Eq. (2.11) is sufficient for optimality, i.e. the solution of (2.11) satisfies (2.10). If a control function
uðiÞ Þ ¼ J i;~uðiÞ ðuhii Þ for all i 2 f0; 1; 2; . . . ; N g, then u*(Æ) is an optimal control, i.e. J(u*(Æ)) 6 J(u(Æ))
u*(Æ) satisfies V ði; ~
for every admissible control function u(Æ).

Proof. The proof that (2.11) implies (2.10) uses backward induction. If the statement is true for i þ 1, we shall
show that it must be true for i. By the induction hypothesis, we have
V ði þ 1; b  nÞ ¼ min J iþ1;bn ðuhiþ1i Þ: ð2:15Þ
uhiþ1i

As in the proof of Theorem 2.1, every uhii can be represented as uhii ¼ n  uhiþ1i . Then we have
J i;b ðn  uhiþ1i Þ ¼ J iþ1;bn ðuhiþ1i Þ þ Uði; xði; i; bÞ; nÞ: ð2:16Þ
Consequently
min J i;b ðn  uhiþ1i Þ ¼ min fJ iþ1;bn ðuhiþ1i Þ þ Uði; xði; i; bÞ; nÞg
nuhiþ1i nuhiþ1i

¼ min minfJ iþ1;bn ðuhiþ1i Þ þ Uði; xði; i; bÞ; nÞg

n uhiþ1i

¼ minffmin J iþ1;bn ðuhiþ1i Þg þ Uði; xði; i; bÞ; nÞg

n uhiþ1i

¼ minfV ði þ 1; b  nÞ þ Uði; xði; i; bÞ; nÞg ¼ V ði; bÞ ð2:17Þ

n
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1907

thus
V ði; bÞ ¼ min J i;b ðuhii Þ: ð2:18Þ
uhii

For i ¼ N , the statement is true since J N ;b ðuhN i Þ  J N ;b ð;Þ ¼ U0 ðxðN ; N ; bÞÞ ¼ V ðN ; bÞ.
Thus the backward induction is complete.
For the second assertion of this theorem, suppose u*(Æ) satisfies V ði; ~uðiÞ Þ ¼ J i;~uðiÞ ðuhii Þ for all
i 2 f0; 1; 2; . . . ; N g. Then, in particular, for i = 0, we have V ð0; ;Þ ¼ J 0;; ðuh0i Þ. It is a simple consequence of
our notational convention that, for every admissible control function u(Æ), we have J 0;; ðuh0i Þ  JðuÞ. Therefore,
V ð0; ;Þ ¼ J ðu ðÞÞ. At the same time, since V satisfies (2.9), we have V ð0; ;Þ 6 J 0;; ðuh0i Þ  J ðuðÞÞ for every
admissible control function u(Æ). Therefore J ðu ðÞÞ 6 J ðuðÞÞ for every admissible control function u(Æ). h
The next question is how to find a control function u*(Æ) that satisfies V ði; u~ðiÞ Þ ¼ J i;~uðiÞ ðuhii Þ for all
i 2 f0; 1; 2; . . . ; N g. This is done by forward recursion.
For i ¼ 0, u*(0) is found as a value of n that is a minimizer of V ð1; nÞ þ Uð0; x0 ð0Þ; nÞ, or, equivalently, as a
solution of V ð0; ;Þ ¼ V ð1; u ð0ÞÞ þ Uð0; x0 ð0Þ; u ð0ÞÞ. Inductively, if uðiÞ has been determined, then u*(i) is
found as a value of n that minimizes V ði þ 1; uðiÞ  nÞ þ Uði; xði; i; uðiÞ Þ; nÞ, or, equivalently, as a solution of
V ði; uðiÞ Þ ¼ V ði þ 1; uðiÞ  u ðiÞÞ þ Uði; xði; i; uðiÞ Þ; u ðiÞÞ. We have the following:
Theorem 2.3. If a control function u*(Æ) is constructed so as to satisfy
V ð0; ;Þ ¼ V ð1; u ð0ÞÞ þ Uð0; x0 ð0Þ; u ð0ÞÞ;
ð2:19Þ
V ði; uðiÞ Þ ¼ V ði þ 1; uðiÞ  u ðiÞÞ þ Uði; xði; i; uðiÞ Þ; u ðiÞÞ for 1 6 i 6 N  1
then it also satisfies
uðiÞ Þ ¼ J i;~uðiÞ ðuhii Þ
V ði; ~ for all i 2 f0; 1; 2; . . . ; N g: ð2:20Þ

Proof. We denote by x*(Æ) the solution of (2.1) that corresponds to the control function u*(Æ). We use (2.11)
and the fact that uhN i  ; to obtain

V ðN ; uðN Þ Þ ¼ U0 ðx ðN ÞÞ  J N ;uðN Þ ð;Þ  J N ;uðNÞ ðuhN i Þ ð2:21Þ

thus the wanted assertion is true for i ¼ N . We use backward induction: assuming that the wanted assertion is
true for i þ 1, we shall show that it must be true for i. By the induction hypothesis, we have
V ði þ 1; uðiþ1Þ Þ ¼ J iþ1uðiþ1Þ ðuhiþ1i Þ: ð2:22Þ

Since V ði; uðiÞ Þ ¼ V ði þ 1; uðiÞ  u ðiÞÞ þ Uði; xði; i; uðiÞ Þ; u ðiÞÞ and uhii ¼ u ðiÞ  uhiþ1i , we have
V ði; uðiÞ Þ ¼ J iþ1uðiþ1Þ ðuhiþ1i Þ þ Uði; xði; i; uðiÞ Þ; u ðiÞÞ ¼ J i;uðiÞ ðuhii Þ ð2:23Þ

thus the induction is complete. h

Remark 2.1. The construction of an optimal control for our variant of dynamic programming for discrete
Volterra equations differs in a substantial way from the ‘‘feedback’’ or ‘‘closed loop’’ controls that are
obtained in ordinary dynamic programming (i.e. in dynamic programming for ordinary differential equations
or finite-difference equations). In our case, each optimal value u*(i) depends on, among other things, the future
optimal control policy uhii . This additional complication further contributes to Bellman’s ‘‘curse of dimension-
ality’’. It is, of course, natural that, due to the memory effect of Volterra equations, the construction of opti-
mal controls will be more complicated than in the case of ordinary differential equations or finite-difference
equations.

Remark 2.2. The method of dynamic programming developed above can be modified to include the possibility
of constraints of the type
uðiÞ 2 Nði; ~
uðiÞ ; ~x½i Þ; ð2:24Þ
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

1908 S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915

uðiÞ ; ~x½i Þ is a closed subset of U. In that case, the dynamic programming equations take the
where each set Nði; ~
form
V ði; bÞ ¼ min fV ði þ 1; b  nÞ þ Uði; xði; i; bÞ; nÞg; V ðN ; bÞ ¼ U0 ðxðN ; N ; bÞÞ: ð2:25Þ
n2Nði;b;~x½i ði;bÞÞ

The proof that these dynamic programming equations are necessary and sufficient for optimality under the
constraints uðiÞ 2 Nði; ~
uðiÞ ; ~x½i Þ can be carried out as in the unconstrained case, and therefore we omit the
details.

3. Results on discretization of controlled Volterra equations and cost functionals

We consider an Euler discretization of the original Volterra controlled system. If h  NT is the step size of the
Euler discretization, we set ti :¼ ih, i ¼ 0; 1; 2; . . . ; N ; xh0 ðiÞ :¼ x0 ðti Þ; f h ði; j; x; uÞ :¼ f ðti ; tj ; x; uÞ, uh ðiÞ :¼ uðti Þ.
The discretized controlled Volterra equation is
X
i1
xh ðiÞ ¼ xh0 ðiÞ þ h f h ði; j; xh ðjÞ; uh ðjÞÞ; 0 6 i 6 N : ð3:1Þ
j¼0

The cost functional J is discretized as

X
N 1
J h ðxðÞ; uðÞÞ :¼ h F ðti ; xðti Þ; uh ðiÞÞ þ F 0 ðxðT ÞÞ: ð3:2Þ
i¼0

We also consider the functional

X
N 1
J h ðxh ðÞ; uðÞÞ :¼ h F ðti ; xh ðiÞ; uh ðiÞÞ þ F 0 ðxh ðN ÞÞ: ð3:3Þ
i¼0

In (3.2), x(Æ) is the solution of the continuous Volterra integral equation (1.1), whereas in (3.3) xh(Æ) is the solu-
tion of the discretized Volterra equation (3.1). We have chosen the simplest numerical integration schemes in
order to minimize the regularity assumptions that we need for the error estimates.
The existing literature on numerical solution of Volterra integral equations deals with simple (i.e. non-con-
trolled) Volterra equations. The approximation of controlled Volterra equations involves additional ingredi-
ents, and for this reason we cannot simply invoke existing results, but instead we must prove all the results we
need.
For the purpose of obtaining error estimates, it becomes necessary to restrict the class of admissible control
functions to functions that satisfy a Lipschitz condition.
Definition 3.1. The set of admissible control functions Uad;Lip ðLÞ is defined as the set of all functions u(Æ) from
[0, T] into U that satisfy a Lipschitz condition with fixed Lipschitz constant L: juðt1 Þ  uðt2 Þj 6 Ljt1  t2 j for all
t1 ; t2 in [0, T].
We postulate:

(i) There is a compact subset Xrel of Rn such that all values x(t) of solutions of the continuous problem (1.1)
and also all values xh(ti) of solutions of the discrete Volterra equation fall into the set Xrel. (In other
words, Xrel is a common set of relevant values for both the continuous and the discretized problem.Also,
we postulate the following properties for the functions f, F, and F0, in addition to the previous
conditions:
(ii) The function f is jointly Lipschitz in x and u, with Lipschitz constant Lf, uniformly in s and t:
jf ðt; s; x1 ; u1 Þ  f ðt; s; x2 ; u2 Þj 6 Lf fjx1  x2 j þ ju1  u2 jg for all x1 ; x2 in R and all
u1 ; u2 in U; for all ðs; tÞ that satisfy 0 6 s 6 t 6 T :
(iii) The functions f, ft ; fs are bounded: maxfjf ðt; s; x; uÞj; jft ðt; s; x; uÞj; jfs ðt; s; x; uÞjg 6 C 0 , for all
x 2 R; u 2 U, and (s, t) that satisfy 0 6 s 6 t 6 T .
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1909

(iv) The function F is jointly Lipschitz in x and u, and has bounded derivative with respect to t:
jF ðt; x1 ; u1 Þ  F ðt; x2 ; u2 Þj 6 LF fjx1  x2 j þ ju1  u2 jg; 8x1 ; x2 in X rel ; 8u1 ; u2 in U;

jF t ðt; x; uÞj 6 M F ; 8x 2 X rel ; 8u 2 U:

(v) The function F0 is Lipschitz: jF 0 ðx1 Þ  F 0 ðx2 Þj 6 LF 0 jx1  x2 j 8x1 ; x2 in X rel .

Some explanations are in order about condition (i). When the set Xrel is found either from (1.9) or from
(1.15), under the appropriate conditions in each case, then the same set Xrel contains also all values xh(ti)
of all solutions of the discretized Volterra equation.
In the case of (1.9), under condition (1.6), we have, for the discretized problem
X
i1 X
i1
jxh ðiÞj 6 jjx0 jj1 þ h ðG0 þ G1 jxh ðjÞjÞ ¼ jjx0 jj1 þ TG0 þ hG1 jxh ðjÞj
j¼0 j¼0

from which, by the discrete Gronwall inequality,

i T
jxh ðiÞj 6 ðkx0 k1 þ TG0 Þð1 þ hG1 Þh 6 ðkx0 k1 þ TG0 Þð1 þ hG1 Þ h :
T
Since ð1 þ hG1 Þ h # expðG1 T Þ as h ! 0þ , we conclude that every xh(i) is in the set Xrel given by (1.9).
In the case of (1.15) under conditions (1.10) and (1.11), it can be proved, in a similar way, that every xh(i) is
in the set Xrel given by (1.15).
Consequently, condition (i), in its general form, is a reasonable condition that can be satisfied in specific
cases.
We have
Theorem 3.1. Under the conditions of Section 2 and the conditions (i) and (ii) above, for every uðÞ 2 Uad;Lip ðLÞ ,
with h ¼ NT , the solution of (3.1) satisfies

jxðihÞ  xh ðiÞj 6 C 1 h for all i ¼ 1; 2; . . . ; N ; uniformly in N and uðÞ 2 Uad;Lip ðLÞ; ð3:4Þ

where C1 is a constant that can be expressed in terms of L, Lf, and C0.

Proof. We set
M 1 :¼ supfjfs ðt; s; x; uÞj : 0 6 s 6 t 6 T ; x 2 X rel ; u 2 Ug;
ð3:5Þ
M 2 :¼ supfjft ðt; s; x; uÞj : 0 6 s 6 t 6 T ; x 2 X rel ; u 2 Ug:
The conditions we have postulated lead to, among other things, a uniform bound on the time-derivative of
x(Æ). We use a dot to denote the time-derivative of x. We have
Z t
x_ ðtÞ ¼ x_ 0 ðtÞ þ f ðt; t; xðtÞ; uðtÞÞ þ ft ðt; s; xðsÞ; uðsÞÞ ds
0

thus
Z t
j_xðtÞj 6 j_x0 ðtÞj þ jf ðt; t; xðtÞ; uðtÞÞj þ jft ðt; s; xðsÞ; uðsÞÞj ds 6 M 0 þ C 0 þ M 1 t 6 M 0 þ C 0 þ M 1 T  M x :
0
ð3:6Þ

We note the following fact from analysis: if u is a differentiable function from [0, T] into Rd , with components
ui ; i ¼ 1; 2; . . . ; d, with derivative that satisfies juðtÞj
_ 6 M u 8t 2 ½0; T , where jÆj denotes one of the ‘p norms on
d d
R , 1 6 p 6 1, then for every two points t1 ; t2 in R we have juðt1 Þ  uðt2 Þj 6 kd M u jtP 1  t 2 j, where
1
the coef-
ficient kd depends on the dimension d and the norm that we use. For the norm jvjp :¼ ð di¼1 jvi jp Þp ; 1 6 p < 1,
1
we have kd ¼ d p , and for the norm jvj1 :¼ max16i6d jvi j we have kd ¼ 1.
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

1910 S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915

Next, we estimate the error of approximation jxðihÞ  xh ðiÞj as follows:

i1 Z ðjþ1Þh
X
h
jxðihÞ  x ðiÞj 6 jf ðih; s; xðsÞ; uðsÞÞ  f ðih; jh; xh ðjÞ; uðjhÞÞj ds
j¼0 jh
i1 Z
X ðjþ1Þh
6 fjf ðih; s; xðsÞ; uðsÞÞ  f ðih; jh; xðsÞ; uðsÞÞj
j¼0 jh

þ jf ðih; jh; xðsÞ; uðsÞÞ  f ðih; jh; xðjhÞ; uðjhÞÞj þ jf ðih; jh; xðjhÞ; uðjhÞÞ
 f ðih; jh; xh ðjÞ; uðjhÞÞjg ds
Xi1 Z ðjþ1Þh
6 fM 1 kn js  jhj þ Lf ½jxðsÞ  xðjhÞj þ juðsÞ  uðjhÞj þ Lf jxðjhÞ  xh ðjÞjg ds:
j¼0 jh

ð3:7Þ
The various terms on the right-hand side of the last inequality in (3.7) are estimated as follows:
Xi1 Z ðjþ1Þh X
i1
M 1 kn js  jhj ds 6 M 1 kn h2 ¼ ih2 M 1 kn 6 ThM 1 kn ;
j¼0 jh i¼0
X
i1 Z ðjþ1Þh i1 Z
X ðjþ1Þh i1 Z
X ðjþ1Þh
Lf jxðsÞ  xðjhÞj ds 6 Lf kn M x js  jhj ds 6 Lf kn M x h ds
j¼0 jh j¼0 jh j¼0 jh
X
i1
2
¼ L f kn M x h ¼ ih2 Lf kn M x 6 ThLf kn M x ; ð3:8Þ
i¼0
i1 Z ðjþ1Þh
X i1 Z
X ðjþ1Þh i1 Z
X ðjþ1Þh
Lf juðsÞ  uðjhÞj ds 6 Lf M 2 km js  jhj ds 6 Lf M 2 km h ds
j¼0 jh j¼0 jh j¼0 jh
X
i1
¼ Lf M 2 km h2 ¼ ih2 Lf M 2 km 6 ThLf M 2 km :
i¼0

We set
M :¼ T ½M 1 kn þ Lf kn M x þ Lf M 2 km : ð3:9Þ
Then (3.7) and (3.9) give
X
i1
jxðihÞ  xh ðiÞj 6 Mh þ Lf h jxðjhÞ  xh ðjÞj ð3:10Þ
j¼0

from which we obtain, via the discrete Gronwall inequality,

i N T
jxðihÞ  xh ðiÞj 6 Mhð1 þ hLf Þ 6 Mhð1 þ hLf Þ ¼ Mhð1 þ hLf Þ h 6 Mh expðLf T Þ ð3:11Þ
which proves the assertion of the theorem, with C 1 ¼ M expðLf T Þ. h

Theorem 3.2. Under the above conditions, we have

jJ ðxðÞ; uðÞ  J h ðX h ðÞ; uðÞÞj 6 C 2 h uniformly for uðÞ 2 Uad;Lip ðLÞ ð3:12Þ
for a constant to be calculated in the proof of this theorem.

Proof. We have
h h
jJ ðxðÞ;
ZuðÞ  J ðX ðÞ; uðÞÞj 
 T X
N 1 
 
¼ F ðt; xðtÞ; uðtÞ dt  hF ðih; xðihÞ; uðihÞÞ
 0 i¼1

X
N 1 Z ðiþ1Þh
6 fjF ðt; xðtÞ; uðtÞ  f ðih; xðtÞ; uðtÞ þ jF ðih; xðtÞ; uðtÞ  F ðihÞ; xðihÞ; uðihÞjgdt
i¼1 ih
6 fM F þ LF M x kn þ LF Lgh  C 3 h ð3:13Þ
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1911

X
N 1
jJ ðxðÞ; uðÞ  J h ðX h ðÞ; uðÞÞj 6 hjF ðih; xðihÞ; uðihÞÞ
i¼0

 F ðih; sh ðihÞ; U ðihÞÞj þ jF 0 ðxðT ÞÞ  F 0 ðxh ðNhÞÞj 6 LF C 1 h þ LF 0 C 1 h  C 4 h: ð3:14Þ

Thus

jJ ðxðÞ; uðÞÞ  J h ðxh ðÞÞj 6 jJ ðxðÞ; uðÞÞ  J h ðxðÞ; uðÞÞj þ jJ h ðxðÞ; uðÞÞ  J h ðxh ðÞ; uðÞÞj
6 C 3 h þ C 4 h  C 2 h: ð3:15Þ

This proves the assertion of this theorem. h

h
If u (Æ) is a discrete-time control that satisfies

juh ðði þ 1ÞhÞj 6 Lh 8i ¼ 0; 1; . . . ; N  1:

We can construct a continuous-time control, which we shall denote by using linear interpolation u~h ðÞ, as
follows:

uh ðði þ #ÞhÞ :¼ #~
~ uh ðði þ 1ÞhÞ þ ð1  #Þ~
uh ðihÞ; 0 6 # 6 1; 0 6 i 6 N  1: ð3:16Þ

We shall need the following:

~h ðÞ, constructed as in (3.16), is a member of Uad;Lip ðLÞ.
Lemma 3.1. The function u
   
 1 u3 u2 
Proof. We start with the following proposition: if t1 < t2 < t3 and u1 ; u2 ; u3 are such that ut22 u
t1  6 L;  t3 t2 
6 L,
if
s1 ¼ #t1 þ ð1  #Þt2 ; s2 ¼ lt2 þ ð1  lÞt3 ; # 2 ½0; 1; l 2 ½0; 1;
~1 ¼ #u1 þ ð1  #Þu2 ; ~
u u2 ¼ lu2 þ ð1  lÞu3
 
~u2 ~u1 
then  s2 s1  6 L . This is proved by direct calculation:
u2  ~
~ u1 ¼ lu2 þ ð1  lÞu3  #u1  ð1  #Þu2 ¼ ð1  lÞðu3  u2 Þ þ #ðu2  u1 Þ; and similarly
s2  s1 ¼ ð1  lÞðt3  t2 Þ þ #ðt2  t1 Þ; thus
j~
u2  ~
u1 j 6 ð1  lÞju3  u2 j þ #ju2  u1 j 6 L½ð1  lÞðt3  t2 Þ þ #ðt2  t1 Þ ¼ Łðs2  s1 Þ
¼ Ljs2  s1 j
(the last equality is due to the fact s2  s1 ¼ ð1  lÞðt3  t2 Þ þ #ðt2  t1 Þ that implies that s2  s1 P 0Þ.
This proposition can be extended, by induction, as follows: If t1 < t2 <    < tk < tkþ1 and
u1 ; u2 ; . . . ; uk ; ukþ1 are such that ,
 
ujþ1  uj 
 
 t  t  6 L 8j ¼ 1; 2; . . . k; if
jþ1 j

s1 ¼ #1 t1 þ ð1  #1 Þt2 ; sk tk þ ð1  #k Þtkþ1 ;
~1 ¼ #1 u1 þ ð1  #1 Þu2 ; uk tk þ ð1  #k Þukþ1 ;
u
 
 u1 
then ~uskk ~s1 
6 L . The inductive step is this: if the wanted conclusion is true for k ¼ m, we shall show that it
must be true for k ¼ m þ 1; to that effect, we apply the proposition proved above, with the points
s 1 ; sm ; tmþ1 in lieu of t1 ; t2 ; t3 , and the values ~ u1 ; ~um ; ~um þ 1 instead of u1 ; u2 ; u3 , and we conclude that
umþ1 ~u1 
 tmþ1 s1  6 L; using this last inequality, we apply again the same proposition   to points s1 ; tmþ1 ; tmþ2 in lieu of
 ~u1 
t1 ; t2 ; t3 , and values ~ u1 ; umþ1 ; umþ2 instead of u1 ; u2 ; u3 , and we obtain ~usmþ1
mþ1 s1
 6 L.
The assertion of the theorem is a particular case, with uniformly spaced points tj, of the proposition that we
just proved by induction for arbitrarily spaced points. h
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

1912 S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915

4. Approximate solution of the Volterra control problem with Lipschitz controls

We consider the problem of minimizing the functional J given by (1.2) subject to the Volterra integral equa-
tion (1.1) and the constraint uðÞ 2 Uad;Lip ðLÞ. We also consider the approximate problem of minimizing the
functional given Jh by (3.2) subject to the discretized Volterra equation (3.1) and the constraint.
juh ðði þ 1ÞhÞ  uh ðihÞj 6 Lh 8i ¼ 0; 1; . . . ; N  2. A solution of the discretized optimal control problem can
be found by using the discrete dynamic programming equations of Section 2.
We denote by uh,*(Æ) a solution of the discretized optimal control problem of this section. Our goal is to
prove that uh,*(Æ) is close to an optimal control for the continuous optimal control problem of this section,
in the following sense: if we construct a continuous-time control function by linear interpolation from the val-
ues of uh,*(Æ) and then use that continuous-time control function in the continous-time Volterra equation and
the continuous-time functional J, then the value of J will be close to the infimum of J under the constraints
stated above. Now, we make all this precise.
We denote by ~ uh; ðÞ the continuous-time control obtained through linear interpolation from uh,*(Æ):
 s s 
uh; ðsÞ :¼ 1 þ i  uh; ðihÞ þ
~  i uh; ðði þ 1ÞhÞ for ih 6 s 6 ði þ 1Þh; 0 6 i 6 N  2: ð4:1Þ
h h
~h; ðÞ is in Uad;Lip ðLÞ. We denote by xh,*(Æ) the solution of the discrete
According to the results of Section 3, u
Volterra equation obtained by using control function uh,*, (Æ) and by x*(Æ) the solution of the controlled Vol-
terra integral equation (1.1) obtained by using the control function ~uh; ðÞ. We shall prove
Theorem 4.1. We have
uh; ðÞÞ ¼
lim J ðx  ðÞ; ~ inf J ðxðÞ; uðÞÞ ð4:2Þ
h!0þ uðÞ2Uad;Lip ðLÞ

with linear rate of convergence, i.e.

uh; ðÞÞ 
lim J ðx  ðÞ; ~ inf J ðxðÞ; uðÞÞ ¼ OðhÞ as h ! 0þ :
h!0þ uðÞ2Uad;Lip ðLÞ

Proof. We have, by the optimality of uh,*(Æ),

J h ðxh; ðÞ; ~
uh; ðÞÞ 6 J h ðÞ; uðÞ 8uðÞ 2 Uad;Lip ðLÞ: ð4:3Þ
By combining (4.3) with Theorem 3.2, we obtain
J h ðxh; ðÞ; ~
uh; ðÞÞ 6 J h ðÞ; uðÞÞ 6 J ðxðÞ; uðÞÞ þ C 2 h 8uðÞ 2 Uad;Lip ðLÞ
and
J ðx ; ~
uh; ðÞÞ 6 J h ðxh; ðÞ; ~
uh; ðÞÞ þ C 2 h 6 J ðxðÞ; uðÞÞ þ 2C 2 h8uðÞ 2 Uad;Lip ðLÞ
thus, if we set J  :¼ inf uðÞ2Uad;Lip ðLÞ J ðxðÞ; uðÞÞ, we have
J  6 J ðx ~
uh; ðÞÞ 6 J  þ 2C 2 h ð4:4Þ
which proves the assertion of the theorem. h

5. Estimation of the computational cost of our version of dynamic programming, and design of parallel
implementation

The question of computational cost is important for every numerical algorithm; comparisons among dif-
ferent algorithms are generally based on their computational costs, although other aspects may also become
relevant in specific cases.
We now present the calculation of the (approximate) computational cost of the variant of dynamic pro-
gramming described in Section 2.
First, at each step of numerically evaluating V ði; bÞ by using (2.11), the variable b has to be quantized. If M
is the number of quantized values of each b(j), then the number of quantized values of at the ith stage of our
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1913

dynamic programming is Mi . The number M depends on the dimension m of the range of the control and on
m
the nature of the set U. If U is a cube, i.e. U ¼ ½a; b , and if Q is the quantized points in the interval [a, b] in
m m
each coordinate of R , then M ¼ Q .
At the ith stage of (2.11), and for each quantized value of b, the solution xði; i; bÞ; we denote the cost of
evaluating xði; i; bÞ by usol ðiÞ. In turn, usol ðiÞ ¼ iueval ðf ; nÞ þ ueval ðx0 ; nÞ þ 1, where ueval ðf ; nÞ is the cost of eval-
uating the values of the Rn -valued function f, and likewise ueval ðx0 ; nÞ is the cost of evaluating x0(i); in order to
evaluate xði; i; bÞ by using (2.1), i evaluations of f, one evaluation of, and one multiplication by h, are required.
The form of usol(i) reflects the fact that the total number of evaluations of x0, over the range 0 6 i 6 N , is
N þ 1. (As usual for work of this type, we do not include the operations of addition and subtraction in the
estimation of the computational cost.) The optimization shown on the right-hand side of (2.11) has a compu-
tational cost which we denote by uopt(i). In order to produce a result that can be useful for actual calculations,
a minimizer n*(i) of the expression on the right-hand side of (2.11) would have to be approximately expressed,
for instance via linear interpolation, as a function of quantized values of the control; we denote this cost of
interpolation by uint(n*(i)). Thus far we have expressed these costs in connection with the evaluation of
V ði; bÞ. We denote by ueval ðV ; iÞ the cost of evaluating V ði; bÞ over all quantized values of b, and
ueval ðU; usol ðiÞ; n) by the cost of evaluating U for each b and n, then
ueval ðV ; iÞ ¼ ueval ðV ; i þ 1Þ þ M iþ1 ueval ðueval ðU; usol ðiÞ; nÞ þ M i ½uopt ðiÞ þ uint ðn ðiÞÞ: ð5:1Þ
Eq. (5.1) is supplemented by the final-time condition
ueval ðV ; N Þ ¼ M nþ1 ueval ðU; usol ðN Þ; nÞ: ð5:2Þ
In order to arrive at closed-form estimates, we make certain simplifying assumptions. First, we assume that
uopt(i) and uint(n*(i)) are constants, and we set A :¼ uopt ðiÞ þ uint ðn  ðiÞÞ; second, we assume that the cost
of evaluating U is proportional to the cost of evaluating xði; i; bÞ, thus we assume that
ueval ðU; usol ðiÞ; nÞ ¼ C eval ½iueval ðf ; nÞ þ ueval ðx0 ; nÞ  C U;1 i þ C U;0 :
Then the system consisting of (5.1) and (5.2) becomes
ueval ðV ; iÞ ¼ ueval ðV ; i þ 1Þ þ M iþ1 ðC 0 þ iC 1 Þ þ M i A;
ð5:3Þ
ueval ðV ; N Þ ¼ M N þ1 ðC 0 þ NC 1 Þ:
The total cost of evaluating V is
X
N
utotal ðV Þ ¼ ueval ðV ; iÞ: ð5:4Þ
i¼1

The summations indicated in (5.3) and (5.4) can be carried out by elementary methods, and the answer is
NM N þ2  ðN þ 1ÞM N þ1 þ M
utotal ðV Þ ¼ C 0
ðM  1Þ2
2
N 2 M N þ3  ð2N 2 þ 2N  1ÞM N þ2 þ ðN þ 1Þ M N þ1  M 2  M
þ C1 3
ðM  1Þ
ðN  1ÞM N þ1  NM N þ M
þ A0 2
: ð5:5Þ
ðM  1Þ
The important conclusion follows from the highest-degree term in (5.5): under the stated assumptions, the
computational cost of our dynamic programming algorithm is of the order of N 2 M N ; for a rectangular set
U, the cost is of the order of N 2 QmN .
Next, we examine the possibilities of parallel implementation of the discrete dynamic programming equa-
tions for Volterra control problems. At each stage (discrete time) i, the evaluation of xði; i; bÞ and the evalu-
ation of the right-hand side of the discrete dynamic programming equation (2.25) can be carried out in parallel
for each of the quantized values of b. Thus, this type of calculation is suited to SIMD (single instruction,
multiple data) parallel processors. At each stage, and for each quantized value of b, all values of
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

1914 S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915

V ði þ 1; b  nÞ over all quantized values of n, are needed. These values, over all quantized values of b and n,
have to be stored in a shared memory unit with which all processors can communicate and select those values
of b that correspond to the appropriate processor. The number of processors that are needed at each stage i is
a function of i, since the dimensionality of b depends on i. This leads to an adaptive requirement: the number
of active processors is time-varying, it depends on the discrete time i at which the set of parallel computations
needs to be performed. This adaptivity is a normal feature of parallel computing, cf. [1]. The cost, in terms of
computing time rather than number of operations, at each stage i, is ueval ðV ; iÞ þ ucomm ðiÞ þ usel ðiÞ, where
ucomm(i) is the cost of the processors’ communicating with the memory unit at stage i, and usel(i) the cost asso-
ciated with selecting the values of b, out of all V ði þ 1; b  nÞ, that correspond to each processor. In general,
the number of processors will be smaller that the number of quantized values of the control function, thus a set
of values of the control would need to be assigned to each processor.

6. Remarks on continuous-time dynamic programming for Volterra control

This section concerns a conceptual question that arises naturally from the results of the previous section,
namely: what, if any, would be the form of continuous-time dynamic programming equations for optimal con-
trol of systems governed by Volterra integral equations? This question is, as far as we can judge, of only con-
ceptual value: any actual computational solution of Volterra control problems will require some sort of
approximation, such as the method we have developed above. It is nevertheless a question that a reader might
reasonably ask.
It is expressly stated that this section does not contain rigorous results; in our assessment, a rigorous devel-
opment would be useless for actually solving the related optimal control problems. Our purpose here is to for-
mally discern the possible nature of continuous-time dynamic programming equations for systems governed
by Volterra integral equations, not to prove theorems about such equations.
The continuous-time analogue of (2.11) is not a straightforward matter. In the continuous case, if we use, as
a variable in the Hamilton–Jacobi function, the restriction uð:Þj½0;t of a control function to the interval [0, t],
then the pairs ðt; uðÞj½0;t Þ form a vector bundle, each uðÞj½0;t being an element of the space Cð½0; t7!RÞ, the
space of continuous functions from [0, t] into the real numbers. Because the spaces Cð½0; t7!RÞ depend on
t, differentiation with respect to the variables t and uðÞj½0;t , in the Hamilton–Jacobi function, cannot be carried
separately with respect to each variable, and thus there is no straightforward way to obtain, even formally, a
continuous-time dynamic programming equation.
We have devised a roundabout way to obtain a framework that allows differentiation, by using a transfor-
mation that changes the variable vector spaces into a fixed vector space, and we carry out the calculations in
these transformed spaces.
For each function v(Æ) in Cð½0; t7!RÞ, we define the corresponding function v# t ðÞ in Cð½0; 17!RÞ by
#
vt ðsÞ ¼ vðtsÞ; 0 6 s 6 1. Similarly, for each continuous function of two variables, say wðt1 ; s1 Þ, defined for
0 6 s1 6 t1 6 t, we define the corresponding function w# t ð; Þ in CðDð½0; 1Þ7!RÞ, where Dð½0; 1Þ :¼
fðs; rÞ : 0 6 r 6 s 6 1g, by w# t ðs; rÞ :¼ wðts; trÞ.
Now, the integral equation
Z t
xðtÞ ¼ x0 ðtÞ þ f ðt; s; xðsÞ; uðsÞÞ ds
0
can be written in the form
Z s
xðtsÞ ¼ x0 ðtsÞ þ f ðts; tr; xðtrÞ; uðtrÞÞt dr ð6:1Þ
0
which is the same as
Z s
#
x#
t ðsÞ ¼ x0;t ðsÞ þ t ft# ðs; r; x# #
t ðrÞ; ut ðrÞÞ dr; 0 6 s 6 1: ð6:2Þ
0
The cost functional
Z T
J¼ F ðt; xðtÞ; uðtÞÞ dt þ F 0 ðxðT ÞÞ
0
 Downloaded from https://iranpaper.ir http://www.master-thesis.ir/Employment
[email protected]

S.A. Belbas / Applied Mathematics and Computation 189 (2007) 1902–1915 1915

can also be written as

Z 1
J ¼ Wðx# #
T ; uT Þ :¼ F ðT s; x# # #
T ðsÞ; uT ðsÞÞT ds þ F 0 ðxT ð1ÞÞ: ð6:3Þ
0

We denote by R the solution operator associated with (6.2), i.e.

x# #
t ¼ Rðt; ut Þ: ð6:4Þ
#
We define the Hamilton–Jacobi function V ðt; a Þ by
V ðt; a# Þ :¼ inffWðx# # # #
T ; uT Þ : ut ¼ a g: ð6:5Þ
ou#
We consider controls u# # # #
t that satisfy ot ¼ wt for some continuous function wt , and we interpret wt as a new
t

control taking values in a set W of continuous functions. Then the dynamic programming equation is,
formally,
 
oV ðt; u# Þ oV ðt; u# Þ #
þ inf ;w ¼ 0; V ðT ; u# Þ ¼ WðRðT ; u# Þ; u# Þ: ð6:6Þ
ot w# 2W ou#
oV ðt;u# Þ
The brackets h; i are used to signify the action of the linear functional oU #
on w#.

References

[1] S.G. Akl, The Design and Analysis of Parallel Algorithms, Prentice Hall, Englewood Cliffs, NJ, 1989.
[2] S.A. Belbas, Iterative schemes for optimal control of Volterra integral equations, Nonlinear Analysis 37 (1999) 57–79.
[3] J.M. Cushing, Integrodifferential Equations and Delay Models in Population Dynamics, Lecture Notes in Biomathematics, vol. 20,
Springer-Verlag, Berlin, Heidelberg, New York, 1977.
[4] M.I. Kamien, E. Muller, Optimal control with integral state equations, The Review of Economic Studies 43 (1976) 469–473.
[5] N.G. Medhin, Optimal processes governed by integral equations, Journal of Mathematical Analysis and Applications 120 (1986) 1–12.
[6] L.W. Neustadt, J. Warga, Comments on the paper ‘Optimal control of processes described by integral equations’ by V.R. Vinokurov,
SIAM Journal of Control 8 (1970) 572.
[7] W.H. Schmidt, Durch Integralgleichungen beschrienbene optimale Prozesse mit Nebenbedingungen in Banachräumen – notwendige
Optimalitätsbedingungen, Zeitschrift für angewandte Mathematik und Mechanik 62 (1982) 65–75.
[8] V.R. Vinokurov, Optimal control of processes described by integral equations, Parts I, II, and III, SIAM Journal of Control 7 (1969)
324–355.