• Gamma variable is the time factor, which is depicting the delay in

giving reward. Its value in in between 0 & 1. If its near to 0 then

weightage is given to immediate rewards & if its value is near to 1
then weightage is given to delayed rewards. It should be small.
• In manufacturing optimization, cost in manufacaturing &
value of the product are optimized.
• In cab service, wait time of customer & fuel requirement are
reduced.
Reinforcement Learning
 UNIT 5
REINFORCEMENT LEARNING
• Gamma = 0.8
• Q(3,1)=0+0.8 max(0,100) = 80
• Q(1,5)=100+0.8 max(0,0) = 100
• Repeat the above process for every state & fill
the Q matrix.
 Introduction
 After scientists became disillusioned
with classical and neo-classical attempts
at modeling intelligence, they looked in
other directions.
 Two prominent fields arose,
connectionism (neural networking,
parallel processing) and evolutionary
computing.
 It is the latter that this essay deals with -
genetic algorithms and genetic
programming.
 What is GA
 A genetic algorithm (or GA) is a search
technique used in computing to find true or
approximate solutions to optimization and
search problems.
 Genetic algorithms are categorized as global
search heuristics.
 Genetic algorithms are a particular class of
evolutionary algorithms that use techniques
inspired by evolutionary biology such as
inheritance, mutation, selection, and crossover
(also called recombination).
 What is GA
 The evolution usually starts from a population
of randomly generated individuals and
happens in generations.

 In each generation, the fitness of every

individual in the population is evaluated,
multiple individuals are selected from the
current population (based on their fitness),
and modified (recombined and possibly
mutated) to form a new population.
 What is GA
 Genetic algorithms are implemented as a
computer simulation in which a population of
abstract representations (called chromosomes
or the genotype or the genome) of candidate
solutions (called individuals, creatures, or
phenotypes) to an optimization problem
evolves toward better solutions.

 Traditionally, solutions are represented in

binary as strings of 0s and 1s, but other
encodings are also possible.
 What is GA
 The new population is then used in the
next iteration of the algorithm.
 Commonly, the algorithm terminates when
either a maximum number of generations
has been produced, or a satisfactory
fitness level has been reached for the
population.
 If the algorithm has terminated due to a
maximum number of generations, a
satisfactory solution may or may not have
been reached.
 Basics of GA
 The most common type of genetic algorithm works like
this:
 a population is created with a group of individuals
created randomly.
 The individuals in the population are then evaluated.
 The evaluation function is provided by the programmer
and gives the individuals a score based on how well they
perform at the given task.
 Two individuals are then selected based on their fitness,
the higher the fitness, the higher the chance of being
selected.
 These individuals then "reproduce" to create one or
more offspring, after which the offspring are mutated
randomly.
 This continues until a suitable solution has been found or
a certain number of generations have passed,
 General Algorithm for GA
 Initialization
 Initially many individual solutions are randomly
generated to form an initial population. The
population size depends on the nature of the
problem, but typically contains several hundreds
or thousands of possible solutions.
 Traditionally, the population is generated
randomly, covering the entire range of possible
solutions (the search space).
 Occasionally, the solutions may be "seeded" in
areas where optimal solutions are likely to be
found.
 General Algorithm for GA
 Selection
 During each successive generation, a proportion of the
existing population is selected to breed a new
generation.
 Individual solutions are selected through a fitness-based
process, where fitter solutions (as measured by a fitness
function) are typically more likely to be selected.
 Certain selection methods rate the fitness of each
solution and preferentially select the best solutions.
Other methods rate only a random sample of the
population, as this process may be very time-consuming.
 Most functions are stochastic and designed so that a
small proportion of less fit solutions are selected. This
helps keep the diversity of the population large,
preventing premature convergence on poor solutions.
Popular and well-studied selection methods include
roulette wheel selection and tournament selection.
 General Algorithm for GA
 In roulette wheel selection, individuals are
given a probability of being selected that is
directly proportionate to their fitness.

 Two individuals are then chosen randomly

based on these probabilities and produce
offspring.
 General Algorithm for GA
 Reproduction
 The next step is to generate a second generation
population of solutions from those selected through
genetic operators:
crossover (also called recombination), and/or mutation.
 For each new solution to be produced, a pair of "parent"
solutions is selected for breeding from the pool selected
previously.
 By producing a "child" solution using the above methods
of crossover and mutation, a new solution is created
which typically shares many of the characteristics of its
"parents". New parents are selected for each child, and
the process continues until a new population of solutions
of appropriate size is generated.
 General Algorithm for GA
 These processes ultimately result in the
next generation population of
chromosomes that is different from the
initial generation.

 Generally the average fitness will have

increased by this procedure for the
population, since only the best organisms
from the first generation are selected for
breeding, along with a small proportion of
less fit solutions, for reasons already
 Crossover
 the most common type is single point crossover. In
single point crossover, you choose a locus at which you
swap the remaining alleles from on parent to the other.
This is complex and is best understood visually.
 As you can see, the children take one section of the
chromosome from each parent.
 The point at which the chromosome is broken depends
on the randomly selected crossover point.
 This particular method is called single point crossover
because only one crossover point exists. Sometimes
only child 1 or child 2 is created, but oftentimes both
offspring are created and put into the new population.
 Crossover does not always occur, however. Sometimes,
based on a set probability, no crossover occurs and the
parents are copied directly to the new population. The
probability of crossover occurring is usually 60% to 70%.
 Mutation
 After selection and crossover, you now have a new
population full of individuals.
 Some are directly copied, and others are produced by
crossover.
 In order to ensure that the individuals are not all exactly
the same, you allow for a small chance of mutation.
 You loop through all the alleles of all the individuals, and
if that allele is selected for mutation, you can either
change it by a small amount or replace it with a new
value. The probability of mutation is usually between 1
and 2 tenths of a percent.
 Mutation is fairly simple. You just change the selected
alleles based on what you feel is necessary and move
on. Mutation is, however, vital to ensuring genetic
diversity within the population.
 General Algorithm for GA
 Termination
 This generational process is repeated until a
termination condition has been reached.
 Common terminating conditions are:
 A solution is found that satisfies minimum criteria
 Fixed number of generations reached
 Allocated budget (computation time/money) reached
 The highest ranking solution's fitness is reaching or
has reached a plateau such that successive iterations
no longer produce better results
 Manual inspection
 Any Combinations of the above
• A genetic algorithm is a search heuristic that is inspired by Charles
Darwin’s theory of natural evolution. This algorithm reflects the process
of natural selection where the fittest individuals are selected for
reproduction in order to produce offspring of the next generation.
• Rather than search from general-to-specific hypotheses, or from simple-
to-complex, GAS generate successor hypotheses by repeatedly mutating
and recombining parts of the best currently known hypotheses.
• At each step, a collection of hypotheses called the current population is
updated by replacing some fraction of the population by offspring of the
most fit current hypotheses.
• The popularity of GAS is motivated by a number of factors including.
Genetic Operators
• Crossover
• The crossover operator produces two new offspring from two parent strings, by
copying selected bits from each parent. The bit at position i in each offspring is
copied from the bit at position i in one of the two parents. The choice of which
parent contributes the bit for position i is determined by an additional string
called the crossover mask.
• Single point crossover offspring takes its first five bits from the first parent and
its remaining six bits from the second parent, because the crossover mask 11 11
1000000 specifies these choices for each of the bit positions.
• In two-point crossover, offspring are created by substituting intermediate
segments of one parent into the middle of the second parent string.
• Uniform Crossover is the case the crossover mask is generated as a random bit
string with each bit chosen at random and independent of the others.
Mutation
• The mutation operator produces small random
changes to the bit string by choosing a single bit at
random, then changing its value. Mutation is often
performed after crossover.
• Fitness Function & Selection
• The fitness function defines the criterion for ranking
potential hypotheses and for probabilistically
selecting them for inclusion in the next generation
population.
• The probability that a hypothesis will be selected is
given by the ratio of its fitness to the fitness of other
members of the current population
 Symbolic AI VS. Genetic
Algorithms
 Most symbolic AI systems are very static.
 Most of them can usually only solve one given
specific problem, since their architecture was
designed for whatever that specific problem was
in the first place.
 Thus, if the given problem were somehow to be
changed, these systems could have a hard time
adapting to them, since the algorithm that would
originally arrive to the solution may be either
incorrect or less efficient.
 Genetic algorithms (or GA) were created to
combat these problems; they are basically
algorithms based on natural biological evolution.
 Symbolic AI VS. Genetic
Algorithms
 The architecture of systems that implement genetic
algorithms (or GA) are more able to adapt to a wide
range of problems.
 A GA functions by generating a large set of possible
solutions to a given problem.
 It then evaluates each of those solutions, and decides on
a "fitness level" (you may recall the phrase: "survival of
the fittest") for each solution set.
 These solutions then breed new solutions.
 The parent solutions that were more "fit" are more likely
to reproduce, while those that were less "fit" are more
unlikely to do so.
 In essence, solutions are evolved over time. This way
you evolve your search space scope to a point where
you can find the solution.
 Genetic algorithms can be incredibly efficient if