UNIT 1: Introduction (3 Hours) Concept of AI, history, current status,

scope, agents, environments, Problem Formulations, Review of tree

and graph structures, State space representation, Search graph and
Search tree.
Solution:

Concept of AI
Definition: Artificial Intelligence (AI) is when machines or computers are designed to
think and act like humans. This means they can learn, solve problems, make decisions, or
even understand language.

Advantages of AI:

 Saves time, works 24/7, handles big data.

 Reduces human error.

Disadvantages of AI:

 Expensive to build and maintain.

 Can replace jobs or be misused.

Types:
 Narrow AI: Built for one task, like translating languages or playing chess; it’s fast
and focused but can’t adapt beyond its job.

 Example: Siri (voice assistant) or a spam email filter.

 Advantage: Very good at its job, fast, and accurate.
 Disadvantage: Can’t do anything outside its task.

 General AI: Aims to think like a human across any task, such as solving problems or
learning new skills; it’s flexible but doesn’t fully exist yet.
 Example: Doesn’t fully exist yet, but think of a robot that can cook, drive, and write
music.
 Advantage: Super flexible and smart.
 Disadvantage: Hard to create, might be unpredictable

 Super AI: Imagined as smarter than humans in everything, from creativity to science;
it could revolutionize the world but raises control and safety fears.

 Example: Sci-fi movies like "Terminator."

 Advantage: Could solve huge problems (e.g., cure diseases).
 Disadvantage: Could be dangerous if not controlled.

1|Page
History of AI

1. 1950s – Early Beginnings: Alan Turing proposed the "Turing Test" to evaluate a
machine’s ability to think.
2. 1956 – Birth of AI: The Dartmouth Conference introduced the term "Artificial
Intelligence."
3. 1960s-70s – Growth and Challenges: Early AI programs solved algebra, played
chess, and proved theorems but faced limitations due to hardware.
4. 1980s – Expert Systems: AI research saw success with expert systems used in
medicine and industry.
5. 1990s – Machine Learning: AI shifted towards data-driven learning approaches.
6. 2000s-Present – Deep Learning & Modern AI: AI powers self-driving cars, voice
assistants, and recommendation systems.

Current Status of AI
 AI in Everyday Life: Chatbots, voice assistants (Siri, Alexa), and recommendation
systems (Netflix, YouTube).
 AI in Industries: Healthcare (disease detection), Finance (fraud detection), and
Automation (robotic process automation).
 Advanced AI: Deep learning, computer vision, and natural language processing
(NLP).
Advantages:

 Solves real problems (e.g., predicting weather, diagnosing diseases).

 Improves daily life (e.g., Netflix recommendations).

Disadvantages:

 Privacy concerns (e.g., too much data collection).

 Bias in AI (e.g., unfair decisions if trained on bad data).

Scope of AI
 Healthcare: AI-based diagnosis, robotic surgery.
 Finance: AI-powered fraud detection, automated trading.

 Education: Personalized learning, automated grading.

 Robotics: AI-powered robots in industries, military.
 Entertainment: AI-generated content, game playing.
 Autonomous Systems: Self-driving cars, smart cities.
Advantages:

 Endless possibilities to improve life.

 Can tackle huge, complex problems.

2|Page
Disadvantages:
 Needs lots of money, data, and skilled people.

 Ethical risks (e.g., AI weapons).

Agents:
Definition: An AI agent is something (software or robot) that senses its surroundings and
takes actions to achieve a goal.
Types of AI Agents
1. Simple Reflex Agents: Act only based on current conditions (e.g., thermostat).

2. Model-Based Reflex Agents: Use memory to make decisions (e.g., self-driving cars).
3. Goal-Based Agents: Act to achieve specific goals (e.g., chess-playing AI).
4. Utility-Based Agents: Choose actions based on maximizing overall benefit (e.g.,
recommendation systems).
5. Learning Agents: Improve over time using experience (e.g., AI chatbots).
Advantages & Disadvantages of AI Agents

Advantages:
 Automation of tasks
 Faster decision-making
 Reduction of human error
Disadvantages:

 High cost
 Lack of common sense
 Ethical concerns
Types of AI Environments
1. Fully Observable vs. Partially Observable: Whether the agent has complete or
partial data.
2. Deterministic vs. Stochastic: Whether the environment follows fixed rules or
includes randomness.
3. Static vs. Dynamic: Whether the environment changes over time.

3|Page
 4. Discrete vs. Continuous: Whether the environment consists of distinct steps or a
continuous range.
5. Single-Agent vs. Multi-Agent: Whether one or multiple agents interact.
Advantages & Disadvantages of AI Environments
Advantages:

 Helps define AI problem-solving techniques.

 Determines the complexity of AI tasks.
Disadvantages:
 Complex environments require high computational power.
 Hard to predict real-world changes.

Problem Formulation in AI
AI problem formulation involves defining a problem clearly so that an AI agent can solve it.
Components of Problem Formulation
1. Initial State: Starting point of the problem.
2. Actions: Possible moves AI can take.

3. Transition Model: Rules of how actions affect the state.

4. Goal Test: Determines if the goal is reached.
5. Path Cost: Cost of taking actions.
Advantages & Disadvantages
Advantages:

 Helps in efficient problem-solving.

 Reduces unnecessary computations.
Disadvantages:
 Requires well-defined problems.
 Some problems may be too complex to formulate.

Tree and Graph Structures in AI

Both trees and graphs are used in AI for problem-solving and search algorithms.
Tree Structure
A tree is a hierarchical data structure with nodes connected by edges.
Types of Trees

4|Page
 1. Binary Tree: Each node has up to two children.
2. Binary Search Tree (BST): Sorted binary tree for fast searching.
3. Decision Tree: Used in AI for decision-making.

Advantages & Disadvantages of Trees

Advantages:
 Helps in structured problem-solving.
 Efficient for search operations.
Disadvantages:

 Can grow large, consuming memory.

 Requires balancing for efficiency.
Graph Structure
A graph consists of nodes (vertices) connected by edges.
Basic Structure of a Tree

 Root Node: The top-most node of the tree.

 Parent Node: A node that has child nodes.
 Child Node: A node that comes from a parent node.
 Leaf Node: A node with no children.
 Edges: Connections between nodes.

Types of Graphs
1. Directed Graph: Edges have direction.
2. Undirected Graph: Edges have no direction.
3. Weighted Graph: Edges have weights (costs).
Advantages & Disadvantages of Graphs

Advantages:
 Represents complex relationships.
 Useful for AI pathfinding.
Disadvantages:
 Requires significant memory.
 Algorithms can be computationally expensive.

5|Page
State Space Representation
State space is the collection of all possible states a problem can have.
Components of State Space Representation

 States: Different configurations of the problem.

 Operators: Actions that transition states.
 Goal State: The desired outcome.
Advantages & Disadvantages
Advantages:

 Provides a structured way to model problems.

 Helps in efficient search and problem-solving.
Disadvantages:
 Large state spaces increase complexity.
 Some problems require an enormous number of states.

Search Graph and Search Tree

 Search Graph: A graph where nodes represent states and edges represent actions.
 Search Tree: A tree is a hierarchical data structure where each node is connected to
one or more child nodes. Trees are widely used in search algorithms, decision-
making processes, and classification tasksTypes of Search Trees
1. Uninformed Search (Blind Search): Does not use additional information.
o Breadth-First Search (BFS)
o Depth-First Search (DFS)

2. Informed Search (Heuristic Search): Uses extra information to improve efficiency.

o A Algorithm*
o Greedy Best-First Search
Advantages & Disadvantages
Advantages:

 Search trees make problem-solving systematic.

 Graphs help in solving complex AI problems efficiently.
Disadvantages:

 Large graphs increase time and space complexity.

6|Page
  Some searches may lead to infinite loops.

Unit 02: Search Algorithms Random search, Search with closed and
open list, Depth first and Breadth first search, Heuristic search, Best
first search, A* algorithm, Game Search.
Solution:
1. Random Search
Definition
Random search is an uninformed search algorithm that explores possible solutions
randomly without using any prior knowledge. It does not follow a specific strategy or
heuristic, making it inefficient for large problem spaces.
Example

Imagine a robot in a maze that moves randomly in different directions until it finds the exit.
It does not remember where it has been before.

Advantages & Disadvantages

Advantages Disadvantages

Simple to implement Very inefficient

Works in unknown environments May take a long time to find a solution

Can sometimes find solutions by Does not guarantee an optimal or even a valid
chance solution

2. Search with Closed and Open List

Definition
This search technique keeps two lists:

 Open List: Stores nodes that need to be explored.

 Closed List: Stores nodes that have already been visited.
By using these lists, the algorithm avoids revisiting the same nodes, making the
search more efficient.

Example
In A search*, the open list contains paths that are still being explored, while the closed list
keeps track of paths that have already been checked.

7|Page
Advantages & Disadvantages

Advantages Disadvantages

Prevents redundant searches Requires more memory to store lists

Speeds up search by avoiding revisits Can be slow in large graphs

Ensures completeness and correctness Not always the best approach for simple problems

3. Depth-First Search (DFS)

Definition
DFS explores the deepest possible nodes first before backtracking when no further moves
are possible. It uses a stack (LIFO - Last In, First Out) for storing nodes.
Example
Consider a file system where you explore the deepest folder first before coming back up.
Advantages & Disadvantages

Advantages Disadvantages

Requires less memory than BFS May get stuck in infinite loops if cycles exist

Can find solutions quickly in deep search spaces Does not guarantee the shortest path

Useful for solving puzzles like mazes Inefficient for wide search spaces

4. Breadth-First Search (BFS)

Definition
BFS explores all nodes at the current level before moving deeper into the search space. It
uses a queue (FIFO - First In, First Out) to store nodes.
Example
Finding the shortest path in an unweighted road network, where all roads have equal
weight.

Advantages & Disadvantages

Advantages Disadvantages

Guarantees the shortest path in an unweighted graph Requires more memory than DFS

8|Page
Advantages Disadvantages

Always finds a solution if one exists Can be slow in deep search spaces

Useful for applications like web crawling Not efficient for very large graphs

5. Heuristic Search
Definition
Heuristic search uses domain-specific knowledge (heuristics) to estimate which paths are
most promising, improving search efficiency.
Example
In a GPS navigation system, the straight-line distance to the destination is used as a
heuristic to guide the search.
Advantages & Disadvantages

Advantages Disadvantages

Accuracy depends on the quality of the

Reduces unnecessary exploration
heuristic

Faster than uninformed searches May not always find the best path

Commonly used in AI and optimization Complex heuristics can be computationally

problems expensive

6. Best-First Search

Definition
This algorithm expands the most promising node first based on a heuristic function. It uses a
priority queue to always explore the best candidate next.
Example
Google Maps prioritizing highways over smaller roads when suggesting a route.
Advantages & Disadvantages

Advantages Disadvantages

Faster than blind searches May not always find the optimal solution

Efficient in large search spaces Requires a well-defined heuristic

9|Page
Advantages Disadvantages

Used in AI planning and robotics Can get stuck in local optima

7. A Algorithm*
Definition
A* is an informed search algorithm that combines BFS and Best-First Search. It uses the
formula:
f(n)=g(n)+h(n)f(n) = g(n) + h(n)f(n)=g(n)+h(n)

where:
 g(n)g(n)g(n) is the cost from the start node to node nnn.
 h(n)h(n)h(n) is the estimated cost from node nnn to the goal.
Example
Finding the shortest route in a GPS navigation system, considering both distance traveled
and estimated remaining distance.
Advantages & Disadvantages

Advantages Disadvantages

Guarantees the shortest path if h(n)h(n)h(n) is Requires more computation than simpler
admissible searches

Very efficient in many real-world problems Uses more memory

Performance depends on the heuristic

Used in AI, robotics, and pathfinding
function

8. Game Search
Definition
Game search algorithms are used in AI to find optimal strategies in competitive
environments like chess or tic-tac-toe. These algorithms simulate possible moves and
counter-moves.
Example
The Minimax Algorithm evaluates all possible moves for both players and selects the best
one. Alpha-Beta Pruning optimizes this process by ignoring unnecessary calculations.
Advantages & Disadvantages

10 | P a g e
Advantages Disadvantages

Helps AI make optimal decisions in

Computationally expensive
games

May not always find the best move in complex

Alpha-Beta Pruning improves efficiency
games

Used in Chess, Go, and Poker AI Needs a well-defined evaluation function

11 | P a g e
UNIT 3: Probabilistic Reasoning
Probability, conditional probability, Bayes Rule, Bayesian Networks-
representation, construction and inference, temporal model,
hidden Markov model.
Solution:
1. Probability
Definition

Probability quantifies the likelihood of an event occurring, ranging from 0 (impossible) to 1

(certain). It provides a mathematical framework for reasoning under uncertainty.

Types
 Classical Probability: Assumes equally likely outcomes. For example, the probability
of rolling a 3 on a fair six-sided die is 16\frac{1}{6}61.
 Empirical Probability: Based on observed data. For instance, if it rained 30 out of 100
days, the empirical probability of rain on any given day is 0.3.
 Subjective Probability: Based on personal judgment or belief. For example,
estimating a 70% chance of a project’s success based on expert opinion.
Advantages
 Provides a structured approach to quantify uncertainty.
 Facilitates decision-making under uncertainty.

 Forms the foundation for statistical inference and modeling.

Disadvantages
 Requires accurate probability assignments, which can be challenging.
 Assumes that outcomes are well-defined and measurable.
 Can be misinterpreted without proper understanding.

Example
In a deck of 52 cards, the probability of drawing an Ace is 452=113\frac{4}{52} =
\frac{1}{13}524=131.
2. Conditional Probability
Definition

12 | P a g e
Conditional Probability is the probability of an event occurring given that another event has
already occurred. It is denoted as P(A∣B)P(A|B)P(A∣B), representing the probability of event
A occurring given that event B has occurred.
Mathematically:
P(A∣B)=P(A∩B)P(B)P(A|B) = \frac{P(A \cap B)}{P(B)}P(A∣B)=P(B)P(A∩B)
Advantages

 Allows updating probabilities based on new information.

 Essential for understanding dependencies between events.
 Forms the basis for advanced probabilistic models.
Disadvantages
 Requires accurate joint probability distributions.

 Can be counterintuitive in complex scenarios.

Example
If 1% of the population has a certain disease (Event D), and a test correctly identifies the
disease 99% of the time (Event T), the probability of having the disease given a positive test
result is calculated using Bayes' Theorem.
For more details, refer to GFG's article on Conditional Probability vs Bayes Theorem.
3. Bayes' Rule
Definition
Bayes' Rule (or Bayes' Theorem) relates the conditional and marginal probabilities of
random events. It provides a way to update the probability estimate of an event based on
new evidence.
The formula is:

P(A∣B)=P(B∣A)×P(A)P(B)P(A|B) = \frac{P(B|A) \times P(A)}{P(B)}P(A∣B)=P(B)P(B∣A)×P(A)

Advantages
 Facilitates the updating of beliefs with new data.
 Integral to Bayesian inference and decision-making.
 Applicable in various fields, including medicine and machine learning.

Disadvantages
 Requires prior probabilities, which may be subjective.
 Computationally intensive for complex models.

13 | P a g e
Example
In medical diagnostics, Bayes' Rule helps calculate the probability of a disease given a
positive test result, considering the test's accuracy and the disease's prevalence.
For a comprehensive understanding, see GFG's article on Bayes' Theorem.
4. Bayesian Networks

Definition
Bayesian Networks are graphical models that represent probabilistic relationships among a
set of variables. They consist of nodes (variables) and directed edges (dependencies),
forming a Directed Acyclic Graph (DAG).
Representation
Each node is associated with a Conditional Probability Table (CPT) that quantifies the effect
of the parent nodes. The joint probability distribution is factored into these conditional
probabilities.
Construction
1. Identify Variables: Determine relevant variables for the domain.
2. Establish Dependencies: Define directed edges based on causal or influential
relationships.
3. Assign Probabilities: Populate CPTs with appropriate conditional probabilities.

Inference
Inference involves computing the posterior probabilities of certain variables given observed
evidence. Techniques include:
 Exact Inference: Methods like Variable Elimination and Belief Propagation.
 Approximate Inference: Methods like Monte Carlo simulations.
Advantages
 Compact representation of joint probability distributions.

 Facilitates reasoning under uncertainty.

 Incorporates prior knowledge and observed data.
Disadvantages
 Construction can be complex for large systems.
 Exact inference is computationally intensive in dense networks.
 Requires accurate probability assessments.
Example

14 | P a g e
In a medical diagnosis system, a Bayesian Network can represent diseases and symptoms,
where edges denote the influence of diseases on symptoms. Given observed symptoms, the
network can infer the probabilities of various diseases.
For more information, refer to GFG's articles on Understanding Bayesian Networks and Exact
Inference in Bayesian Networks.
5. Temporal Models
Definition
Temporal Models represent systems that evolve over time, capturing temporal
dependencies between variables. They are essential for modeling time-series data and
sequential processes.
Types

 State-Space Models: Represent systems with hidden states evolving over time,
observed through noisy measurements.

 Hidden Markov Models (HMMs): A specific type of state-space model with discrete
hidden states and observable outputs.

Advantages
 Capture temporal dynamics and dependencies.
 Useful for forecasting and sequential decision-making.
 Applicable in various domains, including finance and speech recognition.
Disadvantages

 Model complexity increases with the number of states.

 Parameter estimation can be challenging.
 Assumptions (e.g., Markov property) may not hold in all scenarios.
Example
In speech recognition, temporal models capture the sequence of spoken words over time,
facilitating accurate transcription.

For a deeper understanding, see GFG's article on Probabilistic Notation in AI.

6. Hidden Markov Models (HMMs)
Definition
Hidden Markov Models are temporal models where the system is modeled as a Markov
process with hidden states. Each state produces an observable output with a certain
probability.
Components

15 | P a g e
  States: Hidden conditions of the system.
 Observations: Visible outputs influenced by hidden states.
 Transition Probabilities: Probabilities of moving between states.

 Emission Probabilities: Probabilities of observing a particular output from a state.

Advantages
 Effective for modeling sequential data with hidden structures.
 Well-established algorithms for training and inference.
 Applicable in various fields, including bioinformatics and finance.

Disadvantages
 Assumes the Markov property, which may not always hold.
 Requires large datasets for accurate parameter estimation.
 Computationally intensive for large state spaces.
Example

In natural language processing, HMMs can model

UNIT 4 Markov Decision process MDP formulation, utility theory,

utility functions, value iteration, policy iteration and partially
observable MDPs.
Solution:
1. Markov Decision Process (MDP)
Definition

A Markov Decision Process (MDP) is a mathematical framework for modeling decision-

making situations where outcomes are partly random and partly under the control of a
decision-maker. An MDP is characterized by:
 States (S): Possible situations in which an agent can be.

 Actions (A): Possible actions the agent can take.

 Transition Model (T): Probability of moving from one state to another, given a
specific action.

16 | P a g e
  Reward Function (R): Immediate reward received after transitioning from one state
to another due to an action.
The goal in an MDP is to find a policy (a mapping from states to actions) that maximizes the
expected sum of rewards over time.
For a detailed explanation, refer to GFG's article on Markov Decision Process.
Advantages

 Structured Framework: Provides a clear structure for modeling complex decision-

making problems.

 Optimal Policy Derivation: Facilitates the computation of optimal policies that

maximize expected rewards.

 Applicability: Widely applicable in various fields such as robotics, economics, and

automated control systems.
Disadvantages
 Computational Complexity: Solving MDPs can be computationally intensive,
especially with large state and action spaces.
 Assumption of Markov Property: Assumes that future states depend only on the
current state and action, which may not hold in all real-world scenarios.
Example
Consider a robot navigating a grid where each cell represents a state. The robot can move in
four directions (actions). Each move has a probability (transition model) of success or failure,
and certain cells provide rewards or penalties (reward function). The robot's goal is to find a
path (policy) that maximizes its total reward.
2. Utility Theory
Definition
Utility Theory is a framework in economics and decision theory that assesses the
preferences of individuals or agents over a set of goods or outcomes. It assigns a numerical
value (utility) to each outcome, reflecting the satisfaction or value derived from it.
For more information, refer to GFG's article on Decision Theory in AI.

Advantages
 Quantitative Decision-Making: Provides a numerical basis for comparing and making
decisions between different outcomes.
 Captures Preferences: Effectively models the preferences and risk attitudes of
agents.
Disadvantages

17 | P a g e
  Subjectivity: Utility assignments can be subjective and vary between individuals.
 Simplification: May oversimplify complex preferences and ignore factors like
emotions or irrational behaviors.
Example
In investment decisions, a risk-averse investor may have a utility function that reflects a
preference for certain outcomes over uncertain ones with the same expected return.

3. Value Iteration
Definition
Value Iteration is an algorithm used to compute the optimal policy and value function in an
MDP. It iteratively updates the value of each state based on the expected utility of available
actions until convergence.
For a practical implementation, refer to GFG's article on Implement Value Iteration in
Python.
Advantages
 Convergence to Optimal Policy: Guarantees finding the optimal policy if the process
converges.
 Simplicity: Conceptually straightforward and easy to implement.
Disadvantages

 Computational Intensity: Can be computationally expensive, especially for large

state spaces.

 Slow Convergence: May require many iterations to converge, particularly in complex

MDPs.
Example
In a grid-world navigation problem, value iteration can be used to determine the optimal
path by iteratively updating the value of each cell based on possible movements and their
associated rewards.
4. Policy Iteration
Definition
Policy Iteration is an algorithm used to find the optimal policy in an MDP. It involves two
main steps:
1. Policy Evaluation: Calculate the value function for a given policy.

2. Policy Improvement: Update the policy by choosing actions that maximize the
expected utility based on the current value function.

18 | P a g e
These steps are repeated until the policy converges to the optimal policy.
For a detailed comparison between value iteration and policy iteration, refer to GFG's article
on Difference Between Value Iteration and Policy Iteration.
Advantages
 Efficiency: Often converges faster than value iteration in practice.

 Policy-Focused: Directly improves policies, which can be more intuitive in certain

applications.
Disadvantages
 Complexity in Policy Evaluation: Policy evaluation can be computationally intensive,
especially with large state spaces.
 Dependence on Initial Policy: The quality of the initial policy can affect the
convergence speed.
Example
In a robot navigation scenario, policy iteration can be used to iteratively improve the robot's
path-planning strategy by evaluating and updating its movement policy based on expected
rewards.
5. Partially Observable MDPs (POMDPs)
Definition

A Partially Observable Markov Decision Process (POMDP) extends the MDP framework to
situations where the agent cannot fully observe the current state. Instead, the agent
receives observations that provide partial information about the state. A POMDP is defined
by:
 States (S): Possible situations in which an agent can be.

 Actions (A): Possible actions the agent can take.

 Transition Model (T): Probability of moving from one state to another, given a
specific action.
 Reward Function (R): Immediate reward received after transitioning from one state
to another due to an action.
 Observations (O): Possible observations the agent can receive.
 Observation Model (Ω): Probability of receiving a particular observation given a state
and action.
The objective in a POMDP is to find a policy that maximizes the expected sum of rewards
over time, considering the uncertainty in state observations.

19 | P a g e
For a detailed explanation, refer to GFG's article on Partially Observable Markov Decision
Process (POMDP) in AI.
Advantages
 Realistic Modeling: More accurately represents real-world scenarios where agents
have limited or noisy perceptions.
 Robust Decision-Making: Enables the development of policies that account for
uncertainty in observations.
Disadvantages

 Increased Complexity: Solving POMDPs is significantly more complex than solving

fully observable MDPs.

 Computational Challenges: Requires sophisticated algorithms and substantial

computational resources.
Example
In autonomous vehicle navigation, the vehicle may not have complete information about its
environment due to sensor

UNIT 5 Reinforcement Learning Passive reinforcement learning,

direct utility estimation, adaptive dynamic programming, temporal
difference learning, active reinforcement learning- Q learning.
Solution:
1. Passive Reinforcement Learning

Definition
Passive Reinforcement Learning refers to scenarios where an agent follows a fixed policy
and learns the value of states or state-action pairs without influencing the environment
through its actions. The agent observes the outcomes of the policy and updates its value
estimates accordingly.

Types
 Direct Utility Estimation: The agent estimates the utility (value) of each state directly
by averaging the observed rewards obtained from that state.
 Adaptive Dynamic Programming (ADP): The agent uses a model of the
environment's dynamics to compute the value function, often employing techniques
like value iteration or policy iteration.

20 | P a g e
Advantages
 Simplicity: Easier to implement as the policy is fixed, and the agent only needs to
evaluate it.
 Stability: Since the policy doesn't change, the learning process is stable and
predictable.
Disadvantages

 Lack of Exploration: The agent cannot explore alternative actions that might lead to
higher rewards.

 Suboptimal Policies: The fixed policy may not be optimal, limiting the agent's
performance.

Example
Consider a robot following a predetermined path in a maze. It learns the value of each
position based on the rewards received (e.g., reaching the exit) but doesn't deviate from its
path to explore potentially better routes.
2. Direct Utility Estimation
Definition
Direct Utility Estimation involves the agent estimating the utility (value) of each state by
averaging the rewards observed from that state under a fixed policy. This method doesn't
require a model of the environment's dynamics.
Advantages
 Model-Free: Doesn't require knowledge of the environment's transition probabilities.

 Simplicity: Straightforward to implement as it involves averaging observed rewards.

Disadvantages
 High Variance: Estimates can have high variance, especially with limited data.
 Slow Convergence: May require a large number of observations to achieve accurate
estimates.
Example
In a game where an agent always moves right, it estimates the value of each position by
averaging the scores obtained from that position over multiple episodes.
3. Adaptive Dynamic Programming (ADP)

Definition

21 | P a g e
Adaptive Dynamic Programming (ADP) is a method where the agent uses a model of the
environment's dynamics to compute the value function. Techniques like value iteration or
policy iteration are employed to find the optimal policy based on the estimated model.
Advantages
 Efficiency: Utilizes a model to compute value functions, potentially speeding up
learning.
 Optimality: Can converge to the optimal policy if the model is accurate.
Disadvantages

 Model Dependence: Requires an accurate model of the environment, which may not
always be available.

 Computational Complexity: Solving the model can be computationally intensive,

especially in large state spaces.
Example
A self-driving car uses a simulated model of traffic dynamics to compute the optimal driving
policy, adjusting its behavior based on the model's predictions.
4. Temporal Difference Learning
Definition
Temporal Difference (TD) Learning is a model-free reinforcement learning approach that
updates value estimates based on the difference (temporal difference) between consecutive
estimates. It combines ideas from Monte Carlo methods and dynamic programming.
Advantages

 Online Learning: Updates can be made after each step, allowing for real-time
learning.

 Efficiency: Generally requires fewer samples to learn accurate value functions

compared to Monte Carlo methods.
Disadvantages
 Bias: Estimates can be biased, especially with function approximation.
 Stability Issues: Can be unstable under certain conditions, particularly with nonlinear
function approximators.
Example
A chess-playing agent updates its evaluation of board positions after each move based on
the difference between its predicted value and the actual outcome.
5. Active Reinforcement Learning

22 | P a g e
Definition
Active Reinforcement Learning involves an agent that not only learns from the environment
but also actively makes decisions to influence it. The agent explores different actions to
discover which yield the highest rewards, balancing exploration and exploitation.
Types
 Q-Learning: A model-free algorithm where the agent learns the value of action-state
pairs (Q-values) and derives a policy by selecting actions that maximize these values.
Advantages

 Policy Optimization: Capable of learning optimal policies through exploration.

 Flexibility: Adapts to changing environments by continually updating its knowledge.
Disadvantages
 Exploration Challenges: Balancing exploration and exploitation can be complex.
 Computational Demands: Requires significant computational resources, especially in
large state-action spaces.
Example

A robotic vacuum learns to navigate a room efficiently by trying different cleaning paths and
learning which ones cover the area most effectively.
6. Q-Learning
Definition
Q-Learning is a model-free reinforcement learning algorithm that seeks to learn the value of
state-action pairs (Q-values). The agent updates its Q-values based on the reward received
and the maximum expected future rewards, following the Bellman equation:
Q(s,a)←Q(s,a)+α[R+γmax⁡a′Q(s′,a′)−Q(s,a)]Q(s, a) \leftarrow Q(s, a) + \alpha \left[ R +
\gamma \max_{a'} Q(s', a') - Q(s, a) \right]Q(s,a)←Q(s,a)+α[R+γmaxa′Q(s′,a′)−Q(s,a)]
Where:
 sss = current state

 aaa = action taken

 s′s's′ = next state
 RRR = reward received
 α\alphaα = learning rate
 γ\gammaγ = discount factor

23 | P a g e
For a detailed explanation and Python implementation, refer to GFG's article on Q-Learning
in Python.
Advantages
 Model-Free: Doesn't require a model of the environment, making it widely
applicable.
 Convergence: Proven to converge to the optimal policy given sufficient exploration
and learning time.
Disadvantages

 Exploration-Exploitation Trade-off: Needs a strategy to balance exploration of new

actions and exploitation of known rewarding actions.

 Scalability Issues: Can be inefficient in environments with large state or action

spaces.
Example
In a grid-world environment, an agent uses Q-learning to determine the optimal path to a
goal by updating its Q-values based on the rewards received for moving between grid cells.
For further reading on reinforcement learning and its various aspects, explore GFG's
comprehensive articles on Reinforcement Learning and Types of Reinforcement Learning.

24 | P a g e