Measures for Performance Evaluation of Genetic Algorithms

(Extended Abstract)

Kazuo Sugihara

Dept. of ICS, Univ. of Hawaii at Manoa

1 Introduction ploy it for motion planning of an underwater vehicle

In recent years, genetic algorithms (GAs) [1] have being developed at Univ. of Hawaii. The measures
been widely recognized as an eective solving tech- enable us to compare dierent GAs for an optimiza-
nique for complex problems in the real world. GAs tion problem and dierent choices of their parameter
can be regarded as a paradigm of algorithms in the values and to develop a systematic way for parameters
sense that the GAs are parameterized and applicable setting. First, we dene three measures for the perfor-
to a variety of problems by instantiating the paradigm. mance of GAs based on observations in experiments
There are three major components to be designed in by simulation: The likelihood of optimality, the aver-
GAs. The rst one is the coding which is a mapping age tness value and the likelihood of evolution leap.
scheme from a problem to the GA paradigm and rep- Second, we present a case study in which parameters
resents potential solutions. The second one is a tness of a GA for robot path planning was tuned and its
function which quanties quality of solutions and en- performance was optimized through performance eval-
ables us to dierentiate \good" solutions from \bad" uation by using the measures. Third, we propose per-
solutions. The third one is a set of parameters includ- formance evaluation by means of techniques for design
ing population size, population structure, a sequence of experiments and brie y discuss a systematic way of
of genetic operators, the operators' parameters, ter- tuning through the performance evaluation.
mination condition, etc.
As GAs are getting a larger spectrum of applica- 2 Preliminaries
tions, it becomes more crucial to develop a methodol- 2.1 Genetic Algorithms

ogy for the design of GAs. In particular, practition- A genetic algorithm (GA) [1] consists of a popula-
ers need a systematic way to design and implement tion and an evolutionary mechanism. The population
\good" GAs for their particular problems quickly. To is a collection of individuals which represent potential
design \good" algorithms, we must be able to compare solutions through a mapping called a coding. The evo-
algorithms for the same problem and choose the best lutionary mechanism repeatedly transforms the popu-
one with respect to a certain criterion. lation by executing the following steps.
The performance of GAs have been studied in terms (1) Fitness Evaluation: The tness (i.e., an objective
of resultant tness values and convergence primarily function) is calculated for each individual.
(e.g., correlations between convergence speed and the (2) Selection: Individuals are chosen from the current
population size). In many applications of optimiza- population as parents to be involved in recombination.
tion in practice, however, a goal is to nd a solution (3) Recombination: New individuals (called ospring)
as good as possible \within a reasonable amount of are produced from the parents by applying genetic op-
time." With such a constraint on computational cost, erators such as crossover and mutation.
it is not much critical to seek for convergence of a pop- (4) Replacement: Some of the ospring are replaced
ulation unless the problem to be solved has multiple with some individuals (usually with their parents).
objective functions and seeks for Pareto optimal solu- One cycle of transforming a population is called a
tions. In addition, there may be a trade-o between generation. In each generation, a fraction of the pop-
solution quality and convergence, since a higher pres- ulation is replaced with ospring and its proportion
sure to convergence tends to increase the possibility of to the entire population is called the generation gap
premature convergence at a local optimum. (between 0 and 1). There are two extremes. One is
This paper proposes measures for performance eval- a generational replacement GA where the generation
uation of GAs and presents a case study in which a gap is large. Another is a steady-state GA where only
GA was tuned by using the measures in order to em- a few (typically two) individuals are involved.
2.2 Performance Measures 3 A Case Study
Assume that we observe simulation runs of a GA 3.1 Robot Path Planning
for an optimization problem and all the runs are in- As a case study, we present how the performance
dependent. We dene the following performance mea- measures have been used to design a GA for path plan-
sures from a viewpoint of solution quality regardless ning of a mobile robot [5]. Consider an N 2 N grid
of convergence. and obstacles (which are a collection of cells) on the
Likelihood of Optimality grid. Suppose that there are two types of obstacles.
Suppose that a GA was executed for k generations One is a solid obstacle such that a path cannot cross
in each of n runs. Let m be the number of runs which it. Another is a hazardous obstacle such that a path
produced an optimal solution within k generations. can cross it, but at the expense of an extra path length
The likelihood of optimality Lopt(k) at the kth gener- for each cell in proportion to the obstacle's weight. In
ation is the estimated probability m=n. this case study, weights are limited to 1, 2 or 3.
Average Fitness Value A path between the start and destination is dened
Suppose that a GA was executed for k generations as a sequence of adjacent cells on the grid. For simplic-
in each of n runs. The average tness value f(k) at ity, the start and destination of a path are assumed to
the kth generation is the average of the best tness be located at the left upper and right lower corners,
values obtained within k generations in the n runs. respectively. The path length is the total weighted
distance, where the distance between horizontally or
Likelihood of Evolution Leap vertically adjacent cells is 1pand the distance between
A generation is said to be a leap if a solution pro- diagonally adjacent cells is 2. The problem of robot
duced at the generation is better than the best solu- path planning is to nd a shortest path between the
tion obtained before the generation. Suppose that a start and destination, subject to the constraint that
GA was executed for k generations in each of n runs the path does not cross any solid obstacle.
and ` is the average number of leaps within k gener- An example of a 16 2 16 grid is shown in Figure 1. A
ations. The likelihood of evolution leap Lel(k) at the sequence of dots on the grid shows the optimum path
kth generation is the estimated probability `=n.1 whose tness value is 999.86. This example is fairly
Based on the above measures (especially, Lopt), we complex, non-trivial and general enough to use as a
can decide a cut-o generation K , i.e., how many gen- benchmark for the performance evaluation presented
erations a GA should be executed in each run. Let below. Note that there are at least 6 local optima
C = kr be the total computation cost given to exe- within 0.4% tness dierence. Besides this input in-
cute the GA, where r is the number of repeated runs. stance, we conducted simulation of GAs on a few more
The best cut-o generation is the number k of gener- dierent input instances and observed that tendency
ations which maximizes the performance with respect in the performance of GAs is similar.
to a particular measure. If C is xed, we want to nd The following brie y describe the GA to be inves-
k maximizing (1 0 p(k))r , where p(k) denotes the prob- tigated and optimized by using the performance mea-
ability that a GA produces an optimal solution within sures. A path is represented by a binary string con-
k generations. If the value of (1 0 p(k))r is xed, we sisting of blocks each of which denotes the direction
want to nd k minimizing C = kr. and distance of a segment on the path (See details in
It is well-known that diversity in a population plays [5]). To transform minimization into maximization, a
a key role to reach an optimal solution . On the other tness function is dened as (4N 2 0 the path length)
hand, approaching to convergence decreases the diver- for a valid path and 1 for an invalid path, where N is
sity. Hence, there may be a trade-o between conver- the size of a grid.
gence speed and solution quality. Furthermore, com- Assuming that the coding and tness function are
putational time is often bounded a priori. In such xed as mentioned above, various instances of the GA
cases, convergence of a population is not appropriate for the path planning problem were examined and
to decide the termination of a GA's execution. After their parameters were optimized.
all, convergence is not the primary goal of optimiza-
tion. Therefore, we focus on direct correlations be- 3.2 Simulation Results

tween solution quality and computational cost without We conducted simulation of GAs for the path plan-
regard to convergence. ning problem by using our GA Toolkit on the Web
[4]. 2 This paper presents preliminary results of the
1 Although Lel does not explicitly represent solution quality,
it may be useful to determine how many generations should be
2 The toolkit is available at http://www.ics.hawaii.edu/
executed without using convergence of a population. sugihara/research/ga-updates.html
simulation. Each of the investigated GAs consists of a
combination of the following operators, where parents
are always replaced with their ospring.
(a) Selection: Roulette selection, tournament selection
or roulette tournament selection
(b) Crossover: 1-point crossover, 2-point crossover or
uniform crossover Figure 3 Figure 4
(c) Mutation: Multi-point mutation
For each parameter setting of the GAs, simula-
tion results are presented below with 100 runs (un-
less stated otherwise) on the same input shown above.
Thus, we have 100 samples in Bernoulli trials with
binomial distribution. Note that a 95% condence in-
terval [2] is about 0.096 when the probability of success Figure 5 Figure 6
in a Bernoulli trial is 0.4. C. Mutation Rate 
The rst combination of operators to be investi- Figure 6 shows Lopt when  varies from 0.005 to
gated consists of roulette tournament selection, 1- 0.1, assuming that P opSize = 30, k = 1000, ! = 0:95
point crossover and mutation. The following are and = 0:8 are all xed. The simulation results imply
results which show correlations between the perfor- that a choice of the mutation rate is very critical in
mance of the GA and each of its parameters. terms of Lopt.
D. Win Rate !
Figure 7 shows Lopt when ! varies from 0.5 to 1.0,
assuming that P opSize = 30, k = 1000, = 0:8 and
 = 0:04 are all xed.

Figure 1 Figure 2
A. Population Size P opSize
Figure 2 shows the likelihood of optimality Lopt
when P opSize varies from 10 to 300, assuming that
the number k of generations is 1000 and the following Figure 7 Figure 8
parameters of the 3 operators are xed at 0.95, 0.8 E. Crossover Rate
and 0.04, respectively. Figure 8 shows the likelihood of optimality Lopt
(a) Win Rate !: The probability that a better indi- when varies from 0.1 to 1.0, assuming that
vidual wins in each binary tournament P opSize = 30, k = 1000, ! = 0:95 and  = 0:04 are
(b) Crossover Rate : The proportion to the entire all xed. If computational cost for each generation
population whose individuals are involved in crossover is linearly proportional to , k should be normalized
(c) Mutation Rate : The probability that each bit is where the standard is = 0:8 and k = 1000. Note
ipped that potential parallelism in a GA decreases as de-
Since computational cost is linearly proportional to creases. Hence, even if the normalized likelihood of
P opSize, the number k of generations should be nor- optimality for a steady-state GA is larger than that
malized. Let P opSize = 30 and k = 1000 be the for a generational replacement GA, the former may
standard. Then, Lopt(1000 2 30=P opSize) is the nor- not necessarily be better in a parallel computing en-
malized likelihood of optimality. vironment.
B. The Number k of Generations The above simulation results suggest that the best
Figure 3 shows Lopt when k varies from 10 to 5000, parameter setting is P opSize = 30, ! = 0:95, = 0:8
assuming that P opSize = 30, ! = 0:95, = 0:8 and and  = 0:04. This is used as the standard cong-
 = 0:04 are all xed. Figures 4 and 5 shows the uration when we compare dierent combinations of
average tness value f(k) and the likelihood of evo- operators below.
lutionary leap Lel(k) in the same condition, respec- In order to estimate Lopt(1000) for this parameter
tively. Both the results clearly indicate saturation as setting more accurately, we observed 400 runs in total
the number of generations increases. (i.e., with a 95% condence interval at most 0.049)
and found that Lopt(1000) = 0:45. Thus, the proba- The self-tuning is to apply a self-adaptive mecha-
bility that the GA with this parameter setting nds an nism which dynamically changes components, struc-
optimal solution is at least 0.4 with 95% condence. tures and their attributes of a GA and attempts to
If we are given the computational cost equivalent to converge at the best GA or at least a locally optimal
5000 generations in total, we can repeat execution of GA. One of the methods for self-tuning which have
the GA for 1000 generations 5 times. As a result, been investigated is a meta-GA or hierarchical GA,
Lopt is increased to at least 1 0 (1 0 0:4)5 = 0:92 while where dierent GAs themselves are coded and another
Lopt(5000) is only 0.87 in our simulation. GA manipulates them based on the performance of
Next, consider 4 other options for tness remapping the GAs competing to each other. However, the def-
in the roulette procedure: (1) Fitness scaling given in inition of tness is not given a priori nor obvious in
[1]; (2) linear ranking which assigns P , P 01, P 02, 1 1 1, this context. The performance measures discussed in
1 to individuals in nondecreasing order of their tness this paper can be used as a tness function for GAs.
values, where P is the population size; (3) geometric In practice, the empirical tuning is commonly used
ranking which assigns 1, 1/2, 1/3, 1 1 1 , 1=P to in- so far. The best parameter setting is determined
dividuals in nondecreasing order; and (4) exponential through extensive experiments on the performance of
ranking which assigns 1, 1/2, 1/4, 1 1 1 , 1=2P 01 to indi- GAs by simulation. However, \blind" experiments
viduals in nondecreasing order. Among the 4 options, waste time and resources.
the performance of the linear ranking was comparable There are two issues to be addressed in order to
with that of the standard GA. However, for another conduct experiments eciently with sucient accu-
input instance, it was not as good as the standard one. racy. One issue is the number of dierent combina-
The rst and last options were poor. Lopt(1000) for tions (called treatments in experiments) to be exam-
the geometric option was 0.23. ined. Since there are more than several parameters
All the other combinations of operators were also (called factors in experiments) to be controlled simul-
examined. If we use roulette selection, the geometric taneously, exhaustive experiments of all combinations
ranking was the best option giving Lopt(1000) = 0:28 of the parameters' values result in a huge number of
and other options were poor. The performance of a treatments to be observed. Another issue is the un-
GA with tournament selection, 1-point crossover and certainty in experiments. Since execution of a GA
mutation was very poor. The performance of a GA includes processes of random nature, the outcome of
with roulette tournament selection, 2-point crossover experiments always has uncertainty to some extent.
and mutation was comparable with that of the stan- The less uncertainty we want, the more experiments
dard GA, where Lopt(1000) was 0.39. The perfor- we need.
mance of a GA with uniform crossover was 0.24. We propose to use techniques for design of exper-
When no mutation was used (i.e., only roulette iments [2,3] in order to reduce the total number of
tournament selection and crossover), we observed samples in experiments eectively without loosing the
quite few evolution leaps after the rst dozen of gen- creditability of their results. The existing techniques
erations and the performance was very poor. When [2,3] such as blocking, randomization and factorial de-
no crossover was used (i.e., only roulette tourna- sign are useful.
ment selection and mutation), Lopt(1000) was 0.19. References
These simulation results indicate that a GA without
crossover (called the naive evolution [1]) works to some [1] David Beasley et al., \An overview of genetic algo-
extent while a GA without mutation does not work at rithms," Part 1 & 2, University Computing, Vol. 15,
all. Therefore, they imply that mutation plays a cru- No. 2 & 4, pp.58{69 & 170{181, 1993.
cial role in this optimization problem, although it does [2] William J. Diamond, Practical Experiment De-
not work best without crossover. signs for Engineers and Scientists, 2nd Edition, Van
Norstrand Reinhold, 1989.
4 Systematic Tuning
[3] Thomas J. Lorenzen et al., Design of Experiments:
It has been shown by empirical study that the per- A No-Name Approach, Marcel Dekker, 1993.
formance of a GA strongly depends on choices of pa-
rameter values. Thus, one of the major tasks in the [4] John Smith and Kazuo Sugihara, \GA toolkit on
design of a GA is to solve another optimization prob- the Web," Proc. 1st Online Workshop on Soft Com-
lem, called tuning, which seeks for the best parameter puting, Aug. 1996, pp.93{98.
setting to maximize the performance of the GA. There [5] Kazuo Sugihara and John Smith, \A genetic algo-
are two approaches to the tuning: Self-tuning and em- rithm for 3-D path planning of a mobile robots," Tech.
pirical tuning. Rep., Univ. of Hawaii at Manoa, Sept. 1996.