Stochastic Processes and their Applications 94 (2001) 95–103

www.elsevier.com/locate/spa

An adaptive simulated annealing algorithm

Guanglu Gonga; 1 , Yong Liub; c; ∗ , Minping Qianb; 2
a Department of Mathematical Sciences, Tsinghua University, Beijing, 100084,
People’s Republic of China
b Department of Probability and Statistics, Peking University, Beijing, 100871,

People’s Republic of China

c Institute of Applied Mathematics, Chinese Academy of Sciences, Beijing, 100080,

People’s Republic of China

Received 23 November 1999; received in revised form 3 January 2001; accepted 3 January 2001

Abstract
In this paper, inspired by the idea of Metropolis algorithm, a new sample adaptive simulated
annealing algorithm is constructed on 0nite state space. This new algorithm can be considered as
a substitute of the annealing of iterative stochastic schemes. The convergence of the algorithm
is shown.
c 2001 Elsevier Science B.V. All rights reserved.

MSC: 60G99; 65C05

Keywords: Simulated annealing; Adaptive algorithm; Recognition of handwriting Chinese
characters; Estimate of spectral gap

1. Introduction

Adaptive algorithms with stochastics appear frequently in various applications, such

as self-organizing learning algorithms (see Kohonen, 1984), optimization (see Kir-
patrick et al., 1983), neural networks (see Hertz et al., 1991), system identi0cation,
adaptive controls (see Michalwicz, 1992) and so on. The function of these algorithms
is to adjust a state vector (or monitored parameter vector) Xn for specifying the system
considered, where n refers to the time of observation of the system. In most cases of
applications mentioned above, the rule used to update X: will typically be of the form

Xn+1 = Xn + rn b(Xn ; n ); (1.1)

∗ Corresponding author. Institute of Applied Mathematics, Chinese Academy of Science, Beijing, 100080,
People’s Republic of China.
E-mail addresses: [email protected] (G. Gong), [email protected], liuyong71@
yahoo.com (Y. Liu), [email protected] (M. Qian).
1 Supported by NSFC of China 79970120.
2 Supported by NSFC of China 19971005, the Doctoral program Foundation of Institution of Higher

Education and 863 program.

0304-4149/01/$ - see front matter  c 2001 Elsevier Science B.V. All rights reserved.
PII: S 0 3 0 4 - 4 1 4 9 ( 0 1 ) 0 0 0 8 2 - 5
96 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103

where rn is a sequence of small gains and n is the input of the system at time n, either
deterministic or stochastic. This model can be illustrated by practical setups. Let us
take the recognition of oI-line handwriting Chinese characters as an example. In this
case, samples of handwriting Chinese characters are read in one by one, denoted by
1 ; : : : ; n ; : : : , submitted the population of Chinese handwriting characters. Assume that
this population is described by an random vector , i.e. 1 ; : : : ; n ; : : : are i.i.d. copies
of . The bias of a candidate point x from the samples is measured by the following
objective function:


U (x) = E min
x − 
; x = (x(1) ; : : : ; x(m) ):
(i) 2
i6m

Here m is the number of “standard patterns” of the handwriting Chinese characters

desired to be established, e.g. m = 5000 or 20,000. Then the set of standard patterns of
size m will be represented by x∗ = (x(1)∗ ; : : : ; x(m)∗ ) – a minimizing site of the objective
function U (x), i.e.

U (x∗ ) = min U (x):

x
(1)∗ (m)∗
Thus x ; : : : ; x are the optimal representations of handwriting Chinese characters
within these samples. Of course, maybe there are several x(i)∗ s standing for the same
character because of the diIerent styles of writing. Here we see that U (x) takes the
form of Eg(x; ) with a function g(x; ), of which the derivative with respect to x is
not continuous. The clustering problems are much like this. And they are focused to
minimize an expectation

U (x) = Eg(x; ):

A stochastic approach to treat the above model, well known as the self-organizing
algorithm, was suggested by Kohonen (1984), where the iterative formula (1.1) is
designed to 0nd the minimum of U (x).
Algorithm (1.1) has been studied broadly (see, e.g. Benveniste et al., 1990; Fang
et al., 1997 and references therein). However, even in the case of dg(x; )=d x be-
ing Lipschitz continuous, it is not always successful in 0nding an element in the set
S = {z ∈ Rd ; U (z) = minx ∈ Rd U (x)}. To avoid getting trapped in local minima, stochas-
tic perturbation is added as follows:

Xn+1 = Xn + rn b(Xn ; n ) + hn n ; (1.2)

where n is a sequence of independently and identically distributed (i.i.d.) random
variables and b(x; ) takes the role of dg(x; )=d x. As it borrows the idea of annealing
process from statistical mechanics, it is well known as the simulated annealing algo-
rithm. In various special cases, the asymptotic behavior of (1.1), (1.2) as n → ∞ has
been studied by Kushner (1987), Gelfand and Mitter (1991; 1993), Ljung et al. (1992),
MMetivier and Priouret (1987), Fang et al. (1997) and so on. Using the generalized large
deviation theory of Wentzell (1990), under mild conditions on n ; rn ; hn , Fang et al.
(1997) give a uniform treatment and convergence theorem of (1.2) which includes
the results in Gelfand and Mitter (1991; 1993), Ljung et al. (1992). Their main result
P ≡ Eb(x; n ) is Lipschitz continuous and has
can be roughly stated as follows: If b(x)
 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103 97

a potential function U (x) and En = 0, then under some restrictions on the behavior
P
of b(x) and b(x; n ) at in0nity, for {rn } in a wide class, one can always choose {hn }
such that for any  ¿ 0,
 
P0; y U (Xn ) ¡ min U (z) +  → 1;
z ∈ Rd

uniformly for y in an arbitrary compact set F.

The above iterative algorithm (1.2) has given a satisfactory answer if Xn belongs to
the Euclidean space and dg(x; )=d x is Lipschitz continuous. However, in the applica-
tion of the image pattern recognition, especially in the Kohonen’s self-organizing al-
gorithm, dg(x; )=d x is always discontinuous, which almost makes (1.2) useless, since
for (1.1), the convergence behavior is only known extremely roughly even for the
2-dimensional case, while (1.2) is more complicated. Fortunately, in the image pattern
recognition, images are actually discretized by pixels, and we can avoid the Kohonen’s
model and let the state vector take values in a 0nite set X instead.
How to design an adaptive algorithm in this situation such that it converges to the
global minima of U (x) = E(b(x; )), when x takes value in a 0nite set? This situation
often happens in the stochastic approximation and neural network framework. In gen-
eral, an adaptive algorithm for a 0nite state space can be built in the two following
equivalent forms: either by putting

Xn+1 = fn (Xn ; n ); (1.3)

where fn is a mapping from X × R to X, or by de0ning {X0 ; X1 ; X2 : : : Xn : : :} as

a Markov chain with a prescribed transition matrix. Similar to (1.1), {Xn } does not
always converge to a global minimum.
Inspired by the idea of Metropolis algorithm, we construct a sample adaptive algo-
rithm as following.
Let X be a 0nite set, and N = card(X ). Assume {Un (x); x ∈ X; : : : ; n = 1; 2 : : :} be
a family of random variables on probability space (; F; P), satisfying:

(1) for any x ∈ X; {Un (x); n = 1; 2 : : :} is a sequence of independently, identically dis-

tributed random variables, standing for the sample read in;
(2) EUn (x) = U (x), and maxx ∈ X Var(Un (x)) = D ¡ ∞.
n
Un (x) is the observation value at time n. Let Sn (x) = 1=n k =1 Uk (x). By strong law
of large numbers, we have
 
P lim Sn (x) = U (x) = 1:
n→∞

Let !0 be a probability measure on X satisfying !0 (x) ¿ 0 for any x ∈ X, and q0 (x; y)

be an irreducible reversible probability transition matrix on X × X such that !0 is the
invariant probability measure of q0 (x; y), i.e.

#(x; y) ≡ !0 (x)q0 (x; y) = #(y; x); x; y ∈ X:

98 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103

Next, we de0ne a sequence of random probability measures and random probability

transition matrices: for any ! ∈ ; %n ¿ 0; %n → ∞
e−%n SK(n); ! (x) 
!%n ; ! (x) ≡ !0 (x); Z%n ; ! = e−%n SK(n); ! (x) !0 (x);
Z%n ; !
x∈X
 +
 exp[ − %n (SK(n); ! (y) − SK(n); ! (x)) ]q0 (x; y); y
= x;

q%n ; ! (x; y) ≡ 
1−
 q%n ; ! (x; z); y = x;
z=x

where K(n) is a function from N to N (the set of natural numbers) satisfying K(n) ¿
K(n − 1) and limn → ∞ K(n) = ∞. Denote B ≡ {( : X → R}. For any ( ∈ B, let

!%n ; ! (() ≡ x ∈ X !%n ; ! (x)((x) and
(
∞ = supx ∈ X |((x)|.
P = !P n ; !P ∈ X∞ on the
We consider the coordinate process {Yn ; n = 1; 2 : : :}: Yn (!)
coordinate space X , and a random probability measure Q! on X∞ such that Yn is a
∞

Markov chain under Q! with probability transition matrix

Q! [Yn = y|Yn−1 = x] = q%n ; ! (x; y):
We de0ne the above random Markov chain to be the adaptive simulated annealing
algorithm. Let us give some preliminary propositions, before we state and prove
the main theorem. First, we de0ne the random operator L%n ; ! and the Dirichlet
form corresponding L%n ; ! as follows:

L%n ; ! ((x) ≡ (((y) − ((x))q%n ; ! (x; y); x ∈ X;
y∈X

E%n ; ! ((; ) ≡ − (; L%n ; ! ( )L2 (!%n ; ! )

1 
= exp[ − %n (SK(n); ! (x) ∨ SK(n); ! (y))]
2Z%n ; !
x;y ∈ X

×(((x) − ((y))( (x) − (y))#(x; y);

where (; ∈ B. Clearly, −L%n ; ! is a non-negative de0nite operator on L2 (!%n ; ! ). Let
-%n ; ! ≡ inf {E%n ; ! ((; ():
(
L2 (!%n ; ! ) = 1; !%n ; ! (() = 0}
then -%n ; ! is the gap between 0 and the rest of the spectrum of −L%n ; ! .
For any x; y ∈ X, we de0ne a path from x to y to be any sequence of points joining
x to y: x = x0 ; x1 ; : : : ; xk = y satisfying
q0 (xi−1 ; xi ) ¿ 0 (i:e: q%n (xi−1 ; xi ) ¿ 0); i = 1; : : : ; k:
Let .(x; y) denote the set of all the paths from x to y de0ned above. Let p = (x0 ;
x1 ; : : : ; xk ) be an element of .(x; y). For p ∈ .(x; y), we de0ne
Elev(p)(!; n) ≡ max{Sn; ! (xi ); xi ∈ p};

Elev(p) ≡ max{U (xi ); xi ∈ p};

Hn; ! (x; y) ≡ min{Elev(p)(!; n); p ∈ .(x; y)};

H (x; y) ≡ min{Elev(p); p ∈ .(x; y)};

 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103 99

 
mn; ! ≡ Hn; ! (x; y) − Sn; ! (x) − Sn; ! (y) + min Sn; ! (z); x; y ∈ X ;
z∈X
 
m ≡ H (x; y) − U (x) − U (y) + min U (z); x; y ∈ X :
z∈X

Obviously, if
Sn; ! − U
∞ ¡ #; then |mn; ! − m|¡4#. Due to the results of Holley and
Stroock (1988), LUowe (1995) and Diaconis and Stroock (1991), we have the following
proposition

Proposition 1.1. There exists a constant C¿0 independent of n and !, for any %n ¿
0 such that

-%n ; ! ¿ Ce−%n mn; ! :

We de0ne  = minx ∈ X U (x). Let x satisfy U (x ) = . Then for any  ¿ 0, we have

Lemma 1.2. There is a constant C1 such that

lim P{!: !%n ; ! {x; U (x) ¿  + } 6 C1 e−%n =4 } = 1:

n→∞

Proof. If ! ∈ {! ∈ ;
SK(n); ! (x) − U (x)
∞ ¡ 4 }, then

 −%n SK(n); ! (x)
{x: U (x) ¿ +} e !0 (x)
!%n ; ! (x; U (x) ¿  + ) 6 −% S (x) 
e n K(n); ! !0 (x )
 −%n SK(n); ! (x)
{x: SK(n); ! (x) ¿ =2+} e !0 (x)
6 −% S (x) 
e n K(n); ! !0 (x )
1
6 e−%n =4 :
!0 (x )

And then the lemma follows from the law of large numbers.

Main Theorem. We choose %n → ∞ and 0 6 %n −%n−1 6 1=cn; c¿m, take 3 satisfy-

ing m=c ¡ 3 ¡ 1 and choose K(n) satisfying: (1) [K(n) − K(n − 1)]=K(n) ¡ M=n3 ; n =
∞ 3
1; 2 : : : , where M is a constant; (2) n = 1 n =K(n) ¡ ∞. Then for any  ¿ 0 and
5 ¿ 0, there exists n0 , and for any n ¿ n0 there exist constants C1 ; C2; 5 and 6 ¡ 0
such that
 
P Q! (U (Yn ) ¿  + min U (x)) ¡ C1 n−=4c + C2; 5 n6 ¿ 1 − 5:
x∈X

Remark 1.1. It is necessary in practical computation that c; 6; C1 ; C2; 5 , and %n are in-
dependent of !.

Remark 1.2. For convenience, we assume that Q! [Y0 = x0 ] = 1; x0 ∈ X, for any ! ∈ .

In fact, this assumption does not inVuence the convergence of our algorithm.
100 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103

Remark 1.3. For a given 3, we can take a # ¿ 0 satisfying (m + 4#)=c ¡ 3 ¡ 1 since

m=c ¡ 3 ¡ 1. Thus by the following proof of the main theorem, we are able to take
6 ∈ ((m + 4#)=c − 3; 0).

Remark 1.4. Actually, we can choose K(n) = n2 , then K(n) satis0es the conditions of
the theorem.

Our algorithm is designed in view of discrete time and 0nite state space. It may be
considered as a substitute of the annealing of iterative stochastic schemes (1.2) in case
of 0nite state space and can be hopefully extended to the denumerable state case with
some necessary modi0cations.
Comparing our SA algorithm with (1.2), the function of probability transition matrix
q0 (x; y) is analogous to the distribution of the arti0cial noise n in (1.2), and that
exp[ − %n (SK(n); ! (y) − SK(n); ! (x))+ ] is analogous to hn in (1.2).
In order to bring the information into full play, we use the mean of observation values
Sn . Because our algorithm converges 0nally to some set in X while mean values even
do not belong to the sample space or the state space X, that is diIerent from some
adaptive learning algorithms (see Kohonen, 1984), in which the mean values are often
considered as the cluster points.

2. Proof of the Main Theorem

Just because we borrow the ideas of proof of Holley and Stroock (1988), GUotze
(1992) and especially quote a general framework of Frigerio and Grillo (1993), we
only give a short sketch of our proof.
K(n)
Lemma 2.1. Let An ≡ {!; 1=K(n) j = K(n−1)+1 Uj ¡ 1=n3 }, then limk → ∞ P{ n¿k
An (x)} = 1.

Proof. Because of the hypotheses on K(n), we have

 
 
lim P An (x) ¿ 1 − lim P{Acn (x)}
k →∞ k →∞
n¿k n¿k

K(n)
 n23 
¿ 1 − 2 lim EUn2 (x)
k →∞ K(n)2
n¿k j = K(n−1)+1

 n3
¿ 1 − 2MEU12 (x) lim = 1:
k →∞ K(n)
n¿k

For # and for any 5 ¿ 0, by Hajek–Renyi inequality, we can take K#; 5 ¿ 0 such that
∞
for any k ¿ K#; 5 ; x ∈ X; P{!; supj¿k |Sj (x) − U (x)| 6 #} 6 #12 (D=k + D j = k+1 1=j 2 )
6 5=4N .
5
Now, we take K#; 5 ¿ K#; 5 such that for any K(k) ¿ K#; 5 ; P{ n¿k An (x)} ¿ 1 − 4N .
 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103 101

Lemma 2.2. If ! ∈ x ∈ X {( K(n)¿K#;
5 An (x)) ∩ {!; supK(n)¿K#;
5 |SK(n); ! (x) − U (x)|
¡ #}}, then there exists a constant M1; 5 ¿ 0 such that for any K(n) ¿ K#; 5
M1; 5

SK(n); !
∞ 6 M1; 5 and
SK(n); ! − SK(n−1); !
∞ ¡ :
n3

Proof. Obviously,
SK(n); !
∞ ¡ M  , where M  is a constant. Furthermore,

|SK(n); ! (x) − SK(n−1); ! (x)|

   

 K(n−1)   K(n) 
 1 1    1  
6  − Uj (x) +  Uj (x)
 K(n) K(n − 1)   K(n) 
j=1 j = K(n−1)+1
 
   K(n) 
 K(n − 1) − K(n)   1  
6  SK(n−1); ! (x) +  Uj (x)
K(n)  K(n) 
j = K(n−1)+1

M M 1 M1; 5
6 + 3 6 3 :
n3 n n

Using Hajek–Renyi inequality again, we obtain for J5 large enough,

   
 K(n)   ∞
1  
 ¿ J5 6 D
 1 5
P max
 (U j (x) − U (x))  ¡ :
K(n)6K#; 5 K(n)    2
J5 j 2 4N
j=1 j=1

1
K(n)
If ! ∈ x ∈ X {maxK(n) ¡ K#;
5 K(n) | j = 1 (Uj (x) − U (x))| 6 J5 }, then there exists a
constant M2; 5 ¿ 0 such that
SK(n); ! (x)
∞ ¡ M2; 5 , for K(n) 6 K#; 5 . Moreover, it is
easy to show that there exist the constants C; P M3; 5 satisfying 0 ¡ CP 6 C and M3; 5 ¿ 0
such that

P −%n (m+4#) M3; 5

-%n ; ! ¿ Ce and
SK(n); ! − SK(n−1); !
¡ for K(n) 6 K#; 5 :
n3
We denote
   
K(n) 
  1     
B(K#; 5 ) ≡ !: max  (Uj (x)−U (x)) 6J5
 K(n)¡K#;
5 K(n)   
x∈X j=1
     
 K(n)   
1     
∩ !: sup  (Uj (x)−U (x)) 6# ∩  An (x) :
 K(n)¿K#;
5 K(n)    
j=1 K(n)¿K#;
5

Summing up the Lemma 2.2 and the above formula, we have

Lemma 2.3. If ! ∈ B(K#; 5 ), then there exists a constant M4; 5 (independent of !) such
that
(1)
SK(n); !
∞ ¡ M4; 5 ; n = 1; 2 : : : ;
P −%n (m+4#) ; n = 1; 2 : : : ;
(2) -%n ; ! ¿ Ce
(3)
SK(n); ! − SK(n−1); !
¡ M4; 5 =n3 ; n = 1; 2 : : :.
102 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103

For ( ∈ B, we denote
 
Qn; ! (y) = q%n ; ! (x; y)Qn−1; ! (x); Qn; ! (() = Qn; ! (x)((x); n = 1; 2 : : : ;
x∈X x∈X

where Q0 is similar to that in Remark 1.2.

By the Theorem 2:5 in Frigerio and Grillo (1993), we have

Proposition 2.4. If ! ∈ B(K#; 5 ), then for any f ∈ B we have

 
  
 
 f(x)Qn; ! (x) − f(x)!%n ; ! (x) 6 C2; 5
f
∞ n6 ;
 
x∈X x∈X

where C2; 5 is a constant.

Proof of the Main Theorem. Let A ≡ {x: U (x) ¿ +}. Due to Lemma 2.2, it implies
for any 5 ¿ 0, there exists n0 ¿ K#; 5 , such that for any n ¿ n0
5
P{!%n ; ! (A ) ¡ C1 n−=4c } ¿ 1 − :
4
Let B̂n (K#; 5 ) ≡ {!; !%n ; ! (A ) ¡ kn−=4c } ∩ B(K#; 5 ). If ! ∈ B̂n (K#; 5 ), then

Qn; ! (A ) 6 !%n ; ! (A ) + |Qn; ! (A ) − !%n ; ! (A )| ¡ C1 n−=4c + C2; 5 n6 :

Hence we have

 
−=4c 6
P Q! U (Yn ) ¿  + min U (x) ¡ C1 n + C2; 5 n
x∈X

¿ P{B̂n (K#; 5 )} ¿ 1 − 5:

If Un (x); x ∈ X; n = 1; 2; : : : are bounded random variables, i.e. for any n; x and

! ∈ , there exists a constant MP , such that |Un; ! (x) − U (x)| ¡ MP , then we have the
following corollary.

Corollary 2.5. For any 5 ¿ 0 and 5 ¿ 0; there exists n0 , if n ¿ n0 , then

P{!; Qn; ! (A ) ¡ 5 } ¿ 1 − 5:

3. Unsolved problems

According to the referee’s suggestions, we use SK(n) at the nth iteration of the SA
algorithm rather than Sn in our original manuscript. This idea leads to the expected
condition c ¿ m on the speed of decrease of the temperature in the main theorem,
which become more delicate. As the referee pointed out, the question of the optimal
choice of K(n) should be observed. How can we choose K(n) such that the speed of
convergence is as fast as possible for given 5; . Another question is how to judge
whether this SA algorithm with the random energy has suXciently closed to a required
set in a limited time and determine when it should be stopped.
 G. Gong et al. / Stochastic Processes and their Applications 94 (2001) 95–103 103

These two questions are both valuable to be considered in practical computation.

However, in practical applications, it is diXcult to obtain a satisfactory answer to the
above problems by the ideas and tools used in the present paper. Although we guess
that some large deviation results of Markov chains could be applied to solve this sort
of problem, we are not sure how to apply them to this kind of random energy model.

Acknowledgements

We would like to thank the referee for his valuable comments, which were a great
incentive to improve our paper.

References

Benveniste, A., MMetiviet, M., Priouret, P., 1990. Adaptive Algorithms and Stochastic Approximations.
Springer, Berlin.
Diaconis, P., Stroock, D., 1991. Geometric bounds for eigenvalues of Markov chains. Ann. Appl. Probab. 1
(1), 36–61.
Frigerio, A., Grillo, G., 1993. Simulated annealing with time-dependent energy function. Math. Z. 213,
97–116.
Fang, H.T., Gong, G.L., Qian, M.P., 1997. Annealing of iterative stochastic schemes. SIAM J. Control
Optim. 35 (6), 1886–1907.
Gelfand, S.B., Mitter, S.K., 1991. Recursive stochastic algorithm for global optimization in Rd . SIAM
J. Control Optim. 29, 999–1018.
Gelfand, S.B., Mitter, S.K., 1993. Metropolis-type annealing algorithms for global optimization in Rd . SIAM
J. Control Optim. 31, 111–131.
GUotze, F., 1992. Rate of convergence of simulated annealing processes, preprint paper.
Hertz, J., Krogh, A., Palmar, G.R., 1991. Introduction to the Theory of Neural Computation. Santa Fa Inst.
Studies in the Science of Complexity. Addison-Wesley, Reading, MA.
Holley, R.A., Stroock, D.W., 1988. Simulated annealing via Sobolev inequalities. Comm. Math. Phys. 115,
553–569.
Kirpatrick, S., Gelatt, C.D., Vecchi, M.P., 1983. Optimization by simulated annealing. Science 220, 671–680.
Kohonen, T., 1984. Self-organization and Associative Memory. Springer, New York.
Kushner, H.J., 1987. Asymptotic global behavior for stochstic approximation and diIusions with slowly
decreasing noise eIects: global minimization via Monte Carlo. SIAM J. Appl. Math. 47, 169–185.
Ljung, L., PVug, G., Walk, H., 1992. Stochastic approximation and optimization of random system.
BirhUauser-verlag, Basel.
LUowe, M., 1995. Simulated annealing with time-dependent energy function via Sobolev inequalities.
Stochastic Process. Appl. 63, 221–233.
MMetivier, M., Priouret, P., 1987. ThMerMemes de convergence presque sûre pour une classe d’algorithmes
stochastiques aM pas dMecroissant. Probab. Theory Related Fields 74, 403–428.
Michalwicz, Z., 1992. Genetic Algorithms + Datastuctuers = Evolution Programs. Springer, Berlin.
Wentzell, A.D., 1990. Limit Theorems on Large Deviations for Markov Stochastic Processes. Kluwer
Academic Publishers, Dordrecht.