Reinforcement learning(Unit V)

Introduction
Reinforcement learning is an area of Machine Learning. It is about taking suitable
action to maximize reward in a particular situation. It is employed by various software
and machines to find the best possible behavior or path it should take in a specific
situation. Reinforcement learning differs from supervised learning in a way that in
supervised learning the training data has the answer key with it so the model is trained
with the correct answer itself whereas in reinforcement learning, there is no answer
but the reinforcement agent decides what to do to perform the given task. In the
absence of a training dataset, it is bound to learn from its experience.
Introduction(continued)

Reinforcement Learning (RL) is the science of decision making. It is about learning the
optimal behavior in an environment to obtain maximum reward. In RL, the data is
accumulated from machine learning systems that use a trial-and-error method. Data is not
part of the input that we would find in supervised or unsupervised machine learning.
Reinforcement learning uses algorithms that learn from outcomes and decide which action
to take next. After each action, the algorithm receives feedback that helps it determine
whether the choice it made was correct, neutral or incorrect. It is a good technique to use
for automated systems that have to make a lot of small decisions without human guidance.
Reinforcement learning is an autonomous, self-teaching system that essentially learns by
trial and error. It performs actions with the aim of maximizing rewards, or in other words, it
is learning by doing in order to achieve the best outcomes.
 Example:

The problem is as follows: We have an agent and a

reward, with many hurdles in between. The agent is
supposed to find the best possible path to reach the
reward. The following problem explains the problem
more easily.
The above image shows the robot, diamond, and fire.
The goal of the robot is to get the reward that is the
diamond and avoid the hurdles that are fired. The
robot learns by trying all the possible paths and then
choosing the path which gives him the reward with
the least hurdles. Each right step will give the robot
a reward and each wrong step will subtract the
reward of the robot. The total reward will be
calculated when it reaches the final reward that is
the diamond.
Main points in Reinforcement learning –

•Input: The input should be an initial state from which the model will start
•Output: There are many possible outputs as there are a variety of solutions to a
particular problem
•Training: The training is based upon the input, The model will return a state and the
user will decide to reward or punish the model based on its output.
•The model keeps continues to learn.
•The best solution is decided based on the maximum reward.
Difference between Reinforcement learning and
Supervised learning:

Reinforcement learning Supervised learning

Reinforcement learning is all about making decisions sequentially.

In simple words, we can say that the output depends on the state of In Supervised learning, the decision is made on the initial input or
the current input and the next input depends on the output of the the input given at the start
previous input

In Reinforcement learning decision is dependent, So we give labels In supervised learning the decisions are independent of each other
to sequences of dependent decisions so labels are given to each decision.

Example: Chess game,text summarization Example: Object recognition,spam detetction

Types of Reinforcement:

There are two types of Reinforcement:

1.Positive: Positive Reinforcement is defined as when an event, occurs due to a
particular behavior, increases the strength and the frequency of the behavior. In other
words, it has a positive effect on behavior. Advantages of reinforcement learning are:
1. Maximizes Performance
2. Sustain Change for a long period of time
3. Too much Reinforcement can lead to an overload of states which can diminish the results
Types of Reinforcement(continued)

2. Negative: Negative Reinforcement is defined as strengthening of behavior because

a negative condition is stopped or avoided.
Advantages of reinforcement learning:
1. Increases Behavior
2. Provide defiance to a minimum standard of performance
3. It Only provides enough to meet up the minimum behavior
Elements of Reinforcement Learning

Reinforcement learning elements are as follows:

1.Policy
2.Reward function
3.Value function
4.Model of the environment
Policy: Policy defines the learning agent behavior for given time period. It is a mapping from perceived
states of the environment to actions to be taken when in those states.
Reward function: Reward function is used to define a goal in a reinforcement learning problem.A
reward function is a function that provides a numerical score based on the state of the environment
Value function: Value functions specify what is good in the long run. The value of a state is the total
amount of reward an agent can expect to accumulate over the future, starting from that state.
Model of the environment: Models are used for
planning.
Credit assignment problem: Reinforcement learning algorithms learn to generate an internal value
for the intermediate states as to how good they are in leading to the goal. The learning decision maker
is called the agent. The agent interacts with the environment that includes everything outside the
agent.
The agent has sensors to decide on its state in the environment and takes action that modifies its
state.
The reinforcement learning problem model is an agent continuously interacting with an environment.
The agent and the environment interact in a sequence of time steps. At each time step t, the agent
receives the state of the environment and a scalar numerical reward for the previous action, and then
the agent then selects an action. Reinforcement learning is a technique for solving Markov decision
problems.
Reinforcement learning uses a formal framework defining the interaction between a learning agent
and its environment in terms of states, actions, and rewards. This framework is intended to be a
simple way of representing essential features of the artificial intelligence problem.
Various Practical Applications of
Reinforcement Learning –
•RL can be used in robotics for industrial automation.
•RL can be used in machine learning and data processing
•RL can be used to create training systems that provide custom instruction and
materials according to the requirement of students.
Application of Reinforcement Learnings
1. Robotics: Robots with pre-programmed behavior are useful in structured environments, such as the
assembly line of an automobile manufacturing plant, where the task is repetitive in nature.
2. A master chess player makes a move. The choice is informed both by planning, anticipating possible
replies and counter replies.
3. An adaptive controller adjusts parameters of a petroleum refinery’s operation in real time.
RL can be used in large environments in the following situations:

1.A model of the environment is known, but an analytic solution is not available;
2.Only a simulation model of the environment is given (the subject of simulation-based optimization)
3.The only way to collect information about the environment is to interact with it.
Advantages and Disadvantages of Reinforcement
Learning

Advantages of Reinforcement learning

1. Reinforcement learning can be used to solve very complex problems that cannot be solved by
conventional techniques.
2. The model can correct the errors that occurred during the training process.
3. In RL, training data is obtained via the direct interaction of the agent with the environment
4. Reinforcement learning can handle environments that are non-deterministic, meaning that the outcomes
of actions are not always predictable. This is useful in real-world applications where the environment may
change over time or is uncertain.
5. Reinforcement learning can be used to solve a wide range of problems, including those that involve
decision making, control, and optimization.
6. Reinforcement learning is a flexible approach that can be combined with other machine learning
techniques, such as deep learning, to improve performance.
Advantages and Disadvantages of
Reinforcement Learning

Disadvantages of Reinforcement learning

1. Reinforcement learning is not preferable to use for solving simple problems.
2. Reinforcement learning needs a lot of data and a lot of computation
3. Reinforcement learning is highly dependent on the quality of the reward function. If
the reward function is poorly designed, the agent may not learn the desired behavior.
4. Reinforcement learning can be difficult to debug and interpret. It is not always clear
why the agent is behaving in a certain way, which can make it difficult to diagnose and
fix problems.
Markov Decision Process

Reinforcement Learning :
Reinforcement Learning is a type of Machine Learning. It allows machines and software agents to automatically determine
the ideal behavior within a specific context, in order to maximize its performance. Simple reward feedback is required for the
agent to learn its behavior; this is known as the reinforcement signal.
There are many different algorithms that tackle this issue. As a matter of fact, Reinforcement Learning is defined by a
specific type of problem, and all its solutions are classed as Reinforcement Learning algorithms. In the problem, an agent is
supposed to decide the best action to select based on his current state. When this step is repeated, the problem is known as
a Markov Decision Process.
A Markov Decision Process (MDP) model contains:

•A set of possible world states S.

•A set of Models.
•A set of possible actions A.
•A real-valued reward function R(s,a).
•A policy the solution of Markov Decision Process.
What is a State?
A State is a set of tokens that represent every state that the agent can be in.
What is a Model?
A Model (sometimes called Transition Model) gives an action’s effect in a state. In particular, T(S, a, S’)
defines a transition T where being in state S and taking an action ‘a’ takes us to state S’ (S and S’ may
be the same). For stochastic actions (noisy, non-deterministic) we also define a probability P(S’|S,a)
which represents the probability of reaching a state S’ if action ‘a’ is taken in state S. Note Markov
property states that the effects of an action taken in a state depend only on that state and not on the
prior history.

What are Actions?

An Action A is a set of all possible actions. A(s) defines the set of actions that can be taken being in
state S.

What is a Reward?
A Reward is a real-valued reward function. R(s) indicates the reward for simply being in the state S.
R(S,a) indicates the reward for being in a state S and taking an action ‘a’. R(S,a,S’) indicates the reward
for being in a state S, taking an action ‘a’ and ending up in a state S’.
What is a Policy?
A Policy is a solution to the Markov Decision
Process. A policy is a mapping from S to a. It
indicates the action ‘a’ to be taken while in state
S.
Let us take the example of a grid world:
An agent lives in the grid. The above example is
a 3*4 grid. The grid has a START state(grid no
1,1). The purpose of the agent is to wander
around the grid to finally reach the Blue
Diamond (grid no 4,3). Under all circumstances,
the agent should avoid the Fire grid (orange
color, grid no 4,2). Also the grid no 2,2 is a
blocked grid, it acts as a wall hence the agent
cannot enter it.
First Aim: To find the shortest sequence getting from START to the Diamond. Two such
The agent can take any one of these sequences can be found:
actions: UP, DOWN, LEFT, RIGHT
•RIGHT RIGHT UP UPRIGHT
Walls block the agent path, i.e., if there is a wall •UP UP RIGHT RIGHT RIGHT
in the direction the agent would have taken, the Let us take the second one (UP UP RIGHT RIGHT RIGHT) for the subsequent discussion.
agent stays in the same place. So for example, The move is now noisy. 80% of the time the intended action works correctly. 20% of the time
the action agent takes causes it to move at right angles. For example, if the agent says UP the
if the agent says LEFT in the START grid he probability of going UP is 0.8 whereas the probability of going LEFT is 0.1, and the probability
would stay put in the START grid. of going RIGHT is 0.1 (since LEFT and RIGHT are right angles to UP).
The agent receives rewards each time step:-

•Small reward each step (can be negative when can also be term as punishment, in the above
example entering the Fire can have a reward of -1).
•Big rewards come at the end (good or bad).
•The goal is to Maximize the sum of rewards.
Q-Learning
Q-learning is a kind of reinforcement learning algorithm that enables machines to
discover via trial and error the best behaviors to adopt in a given environment. The
quality value, also known as the Q-value or quality, is an estimate of the expected
reward for doing a certain action in a specific condition and is the “Q” in Q-learning.

Finding the best course of action that accelerates the long-term benefit is the aim of Q-
learning. Starting with a database of Q-values for each state-action combination, the Q-
learning algorithm operates. These parameters are initially set at random or to zero. The
agent then investigates the surroundings, acting and earning rewards.

A mathematical formula that considers the present Q-value, the reward received, and
the anticipated value of the following state-action combination is used to update the Q-
values based on these rewards.

The Q-values, which reflect the ideal actions to perform in each state as the agent
continues to investigate the environment, converge to their optimal values. By doing so,
the agent may make choices in complicated contexts with a wide range of alternative
behaviors that will optimize its long-term value.
Why do we need Q-Learning?
In the field of Machine Learning, machines may learn the best course of action in
challenging circumstances with the help of the potential technique known as Q-learning.
But why do we actually require Q-learning? There are various factors that make Q-
learning crucial:

First, without explicit programming, computers may learn from new settings and adapt
to them. This is called Q-learning. Explicit instructions would need to be written for each
vital circumstance the computer may experience in traditional programming. The
computer is more versatile and adaptive to new scenarios because of Q-learning, which
allows it to learn independently via trial and error.

Second, a variety of decision-making processes, including those in robotics, game

theory, and finance, may be optimized using Q-learning. Q-learning can assist
computers in making more useful judgments in complicated contexts by identifying the
best course of action or collection of actions that maximize long-term reward.

Last but not least, Q-learning has the power to change a wide range of industries,
including manufacturing, transportation, and healthcare. Automating various operations
using Q-learning may boost productivity and cut costs by allowing robots to learn and
adapt on their own to make work more swift and seamless.
How Q-Learning Works?
Q-learning is a form of reinforcement learning algorithm that enables an agent to discover the best course of action by
maximizing a reward signal. Here’s how it functions:
•Q-values: The algorithm creates a table of Q-values, which indicate the anticipated reward for doing a certain action in
a specific condition. These Q-values are first chosen at random.
•State: The agent keeps track of the environment’s condition, which reveals details about the scenario as it stands.
•Action: Depending on the situation, the agent decides which action to take. This can be accomplished via an
exploration strategy that chooses a random action with some probability or a straightforward greedy policy that
chooses the action with the greatest Q-value for the current state.
•Reward: In the current state, the agent is given a reward for the activity it took.
•Update Q-value: Using the Bellman equation, the agent changes the Q-value for the current state-action pair.
According to this equation, the immediate reward received plus the discounted expected future reward, which is
calculated using the Q-values for the following state-action pairs, equals the expected Q-value for a state-action pair.
•Repeat: As the agent accumulates experience with the environment, it repeats processes 2 through 5, progressively
updating the Q-values. The objective is to discover the best course of action or the one that maximizes the predicted
cumulative benefit over time.
•Converge: The agent learns the best behaviors to perform in each state as it explores more of the environment,
causing the Q-values to converge to the ideal values.
Bellman Equation in Q-Learning

A key idea in reinforcement learning, including Q-learning, is the Bellman equation. Based on the
incentives received and the anticipated Q-values for the subsequent state-action pairs, the
Bellman equation is employed in Q-learning to update the Q-values for state-action pairings. The
following is the Bellman equation:

Q(s, a) = Q(s, a) + α [R + γ max Q(s', a') - Q(s, a)]

Where:
•The Q-value for performing an action in state s is Q(s, a).
•α is the learning rate, which establishes how much weight to place on new vs old experiences.
•R is the reward received for taking action in-state s.
•γ is the discount factor that establishes the relative importance of present benefits and future
rewards.
•The greatest Q-value for the next state s’ and all the actions a’ that could be conducted in s’ is
expressed as max Q(s’, a’).
•Q(s’, a’) is the Q-value for taking action a’ in state s’.
Applications of Q-Learning
Q-learning is a versatile algorithm that can be applied in various fields to optimize
decision-making processes based on data and real-time feedback. Let us see the various
applications to have a better understanding:
•Recommendation systems: In order to determine the best suggestion approach
based on user input and historical data, recommendation systems can apply Q-learning.
•Marketing: Based on consumer behavior and industry trends, Q-learning may be used
in marketing to optimize pricing tactics and product positioning.
•Supply chain management: Based on real-time demand and supply data, Q-learning
may be utilized in supply chain management to optimize inventory management and
distribution methods.
•Energy management: Q-learning may be used to optimize energy management
systems, such as smart grids, by figuring out the best control strategies based on data
on energy output and consumption.
•Air traffic control: Based on real-time traffic data, Q-learning may be deployed in air
traffic control to optimize routing and scheduling choices.
Q-learning algorithm
The Q-learning algorithm is commonly used in reinforcement
learning to find the best policy for an agent to make decisions
within an environment to obtain the highest possible long-term
rewards.
Reinforcement learning is a type of machine learning in which
an agent is taught to make decisions based on feedback from its
environment, such as rewards or penalties. The goal of the
agent is to determine the best action to take in each state of
the environment to maximize its cumulative reward. Q-learning
algorithm is a type of machine learning algorithm that is used
to help an agent learn the best action to take in a certain
situation to receive the maximum reward. It is model-free,
meaning it doesn't require prior knowledge of how the
environment works. It's also off-policy, which means it can
explore different ways of acting before ultimately learning the
optimal policy. The value function in Q-learning is represented
as Q(s,a) wheres represents the current state and a represents
the action taken.
Q-learning algorithm
Key terminologies in Q-learning
Understanding the parameters used in the Q-learning algorithm is essential before diving into the algorithm itself. To help with this, let's take a look at an
explanation of each parameter:
•Q-values or action-values: This represents the anticipated reward that an agent can obtain by taking a specific action in a given state and subsequently
following the optimal path.
•Episode: An episode refers to a sequence of actions taken by the agent in the environment until it reaches a terminal state.
•Starting state: This is the state from which the agent begins an episode.
•Step: This is a single action taken by the agent in the environment.
•Epsilon-greedy policy: This is a way for the agent to decide whether to explore new actions or exploit actions that have worked well in the past. The epsilon-
greedy policy in the Q-learning algorithm helps the agent make decisions by either exploiting the current best action or exploring other actions. By
balancing exploration and exploitation, the agent can learn and adapt its behavior to achieve optimal long-term rewards in a reinforcement learning setting.
•Chosen action: This is the action selected by the agent based on the epsilon-greedy policy.
•Q-learning update rule: This mathematical formula updates the Q-value of a particular state-action pair. This update is based on the reward that is received
and the maximum Q-value of the next state-action pair.
•New state: It refers to the state that an agent transitions to after taking an action in the current state.
•Goal state: This is a terminal state in the environment where the agent receives the highest reward.
•Alpha (�α): This is a learning rate parameter that controls the degree of weight given to newly acquired information when updating the Q-values.
•Gamma (�γ): This is a discount factor parameter that controls the degree of weight given to future rewards when calculating the expected cumulative
reward.
Algorithm pseudocode
The pseudocode for the Q-Learning algorithm is given below:
What is Deep Q-Learning?
Deep Q-Learning is a reinforcement learning technique that combines Q-Learning, an
algorithm for learning optimal actions in an environment, with deep neural networks. It
aims to enable agents to learn optimal actions in complex, high-dimensional
environments. By using a neural network to approximate the Q-function, which
estimates the expected cumulative reward for each action in a given state, Deep Q-
Learning can handle environments with large state spaces. The network is updated
iteratively through episodes, using a combination of exploration and exploitation
strategies. However, care must be taken to mitigate instability caused by non-
stationarity and divergence issues, typically addressed by experience replay and target
networks. Deep Q-Learning has proven effective in training agents for various tasks,
including video games and robotic control.
Why ‘Deep’ Q-Learning?
Q-learning is a simple yet quite powerful algorithm to create a cheat sheet for our agent.
This helps the agent figure out exactly which action to perform.
But what if this cheatsheet is too long? Imagine an environment with 10,000 states and
1,000 actions per state. This would create a table of 10 million cells. Things will quickly
get out of control!
It is pretty clear that we can’t infer the Q-value of new states from already explored
states. This presents two problems:
•First, the amount of memory required to save and update that table would increase as
the number of states increases
•Second, the amount of time required to explore each state to create the required Q-
table would be unrealistic
Genetic Algorithm in Machine Learning
A genetic algorithm is an adaptive heuristic search algorithm inspired by "Darwin's theory of
evolution in Nature." It is used to solve optimization problems in machine learning. It is one of
the important algorithms as it helps solve complex problems that would take a long time to
solve. Genetic Algorithms are being widely used in different real-world applications, for
example, Designing electronic circuits, code-breaking, image processing, and artificial
creativity.
In this topic, we will explain Genetic algorithm in detail, including basic terminologies used in
Genetic algorithm, how it works, advantages and limitations of genetic algorithm, etc.
What is a Genetic Algorithm?
•Before understanding the Genetic algorithm, let's first understand basic terminologies to better understand this
algorithm:
• Population: Population is the subset of all possible or probable solutions, which can solve the given problem.
•Chromosomes: A chromosome is one of the solutions in the population for the given problem, and the
collection of gene generate a chromosome.
•Gene: A chromosome is divided into a different gene, or it is an element of the chromosome.
•Allele: Allele is the value provided to the gene within a particular chromosome.
•Fitness Function: The fitness function is used to determine the individual's fitness level in the population. It
means the ability of an individual to compete with other individuals. In every iteration, individuals are evaluated
based on their fitness function.
•Genetic Operators: In a genetic algorithm, the best individual mate to regenerate offspring better than parents.
Here genetic operators play a role in changing the genetic composition of the next generation.
•Selection
After calculating the fitness of every existent in the population, a selection process is used to determine which of
the individualities in the population will get to reproduce and produce the seed that will form the coming
generation.
Types of selection styles available

•Roulette wheel selection

•Event selection
•Rank- grounded selection
So, now we can define a genetic algorithm as a heuristic search algorithm to solve optimization
problems. It is a subset of evolutionary algorithms, which is used in computing. A genetic
algorithm uses genetic and natural selection concepts to solve optimization problems.
How Genetic Algorithm Work?

The genetic algorithm works on the evolutionary generational cycle to generate high-quality
solutions. These algorithms use different operations that either enhance or replace the population to
give an improved fit solution.
It basically involves five phases to solve the complex optimization problems, which are given as
below:
•Initialization
•Fitness Assignment
•Selection
•Reproduction
•Termination
1. Initialization
The process of a genetic algorithm starts by
generating the set of individuals, which is
called population. Here each individual is the
solution for the given problem. An individual
contains or is characterized by a set of
parameters called Genes. Genes are
combined into a string and generate
chromosomes, which is the solution to the
problem. One of the most popular
techniques for initialization is the use of
random binary strings.
2. Fitness Assignment
3. Selection
Fitness function is used to determine how fit an individual is? It means the ability of an
individual to compete with other individuals. In every iteration, individuals are evaluated based
on their fitness function. The fitness function provides a fitness score to each individual. This
score further determines the probability of being selected for reproduction. The high the fitness
score, the more chances of getting selected for reproduction.
The selection phase involves the selection of individuals for the reproduction of offspring. All
the selected individuals are then arranged in a pair of two to increase reproduction. Then these
individuals transfer their genes to the next generation.
There are three types of Selection methods available, which are:
•Roulette wheel selection
•Tournament selection
•Rank-based selection
4. Reproduction
After the selection process, the creation of a child occurs in the reproduction step. In this step, the genetic algorithm uses two variation operators that
are applied to the parent population. The two operators involved in the reproduction phase are given below:
•Crossover: The crossover plays a most significant role in the reproduction phase of the genetic algorithm. In this process, a crossover point is selected
at random within the genes. Then the crossover operator swaps genetic information of two parents from the current generation to produce a new
individual representing the offspring. The genes of parents are exchanged among themselves until the crossover point is met. These newly generated
offspring are added to the population. This process is also called or crossover. Types of crossover styles available:One point crossover
•Two-point crossover
•Livery crossover
•Inheritable Algorithms crossover
Mutation
The mutation operator inserts random genes in the offspring (new child) to maintain the diversity in the
population. It can be done by flipping some bits in the chromosomes.
Mutation helps in solving the issue of premature convergence and enhances diversification. The below image
shows the mutation process:
Types of mutation styles available,
•Flip bit mutation
•Gaussian mutation
•Exchange/Swap mutation
Termination

After the reproduction phase, a stopping criterion is applied as a base for termination. The
algorithm terminates after the threshold fitness solution is reached. It will identify the final
solution as the best solution in the population.
General Workflow of a Simple Genetic Algorithm
Advantages of Genetic Algorithm
•The parallel capabilities of genetic algorithms are best.
•It helps in optimizing various problems such as discrete functions, multi-objective problems,
and continuous functions.
•It provides a solution for a problem that improves over time.
•A genetic algorithm does not need derivative information.
Limitations of Genetic Algorithms
•Genetic algorithms are not efficient algorithms for solving simple problems.
•It does not guarantee the quality of the final solution to a problem.
•Repetitive calculation of fitness values may generate some computational challenges
Difference between Genetic Algorithms and Traditional
Algorithms
•A search space is the set of all possible solutions to the problem. In the traditional algorithm, only one set of solutions
is maintained, whereas, in a genetic algorithm, several sets of solutions in search space can be used.
•Traditional algorithms need more information in order to perform a search, whereas genetic algorithms need only one
objective function to calculate the fitness of an individual.
•Traditional Algorithms cannot work parallelly, whereas genetic Algorithms can work parallelly (calculating the fitness of
the individualities are independent).
•One big difference in genetic Algorithms is that rather of operating directly on seeker results, inheritable algorithms
operate on their representations (or rendering), frequently appertained to as chromosomes.
•One of the big differences between traditional algorithm and genetic algorithm is that it does not directly operate on
candidate solutions.
•Traditional Algorithms can only generate one result in the end, whereas Genetic Algorithms can generate multiple
optimal results from different generations.
•The traditional algorithm is not more likely to generate optimal results, whereas Genetic algorithms do not guarantee
to generate optimal global results, but also there is a great possibility of getting the optimal result for a problem as it
uses genetic operators such as Crossover and Mutation.
•Traditional algorithms are deterministic in nature, whereas Genetic algorithms are probabilistic and stochastic in
nature.