Introduction to Machine Learning

a r
d d
H u
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Syllabus
Module – 1 10 Hours
What is artificial intelligence? Problems, problem spaces and search, Heuristic search techniques. Textbook 1: Chapter 1, 2 and 3
RBT: L1, L2
Module – 2 10 Hours
Knowledge representation issues, Predicate logic, Representation knowledge using rules. Concept Learning: Concept learning
r
task, Concept learning as search, Find-S algorithm, Candidate Elimination Algorithm, Inductive bias of Candidate Elimination
a
d
Algorithm. Textbook 1: Chapter 4, 5 and 6 Texbook2: Chapter 2 (2.1-2.5, 2.7) RBT: L1, L2, L3
d
Module – 3
H u 08 Hours

s h
Decision Tree Learning: Introduction, Decision tree representation, appropriate problems, ID3 algorithm. Artificial Neural

he
Network: Introduction, NN representation, appropriate problems, Perceptions, Back propagation algorithm.

M a
Texbook2: Chapter 3 (3.1-3.4), Chapter 4 (4.1-4.5) RBT: L1, L2, L3
Module – 4 10 Hours
Bayesian Learning: Introduction, Bayes theorem, Bayes theorem and concept learning, ML and LS error hypothesis, ML for
predicting, MDL principle, Bates optimal classifier, Gibbs algorithm, Naive Bayes classifier, BBN, EM Algorithm
Texbook2: Chapter 6 RBT: L1, L2, L3
Module – 5 12 Hours
Instance-Base Learning: Introduction, k-Nearest Neighbor Learning, Locally weighted regression, Radial basis function, Case-
Based reasoning. Reinforcement Learning: Introduction, The learning task, Q-Learning.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
Textbook 1: Chapter 8 (8.1-8.5), Chapter 13 (13.1 – 13.3) RBT: L1, L2, L3
 Syllabus - Lab
1. Implement A* Search algorithm.
2. Implement AO* Search algorithm.
3. For a given set of training data examples stored in a .CSV file, implement and demonstrate the Candidate-
Elimination algorithm to output a description of the set of all hypotheses consistent with the training examples.

4.
a r
Write a program to demonstrate the working of the decision tree based ID3 algorithm. Use an appropriate data
d d
set for building the decision tree and apply this knowledge to classify a new sample.
5. H u
Build an Artificial Neural Network by implementing the Backpropagation algorithm and test the same using
appropriate data sets. s h
6.
a he
Write a program to implement the naïve Bayesian classifier for a sample training data set stored as a .CSV file.
M
Compute the accuracy of the classifier, considering few test data sets.
7. Apply EM algorithm to cluster a set of data stored in a .CSV file. Use the same data set for clustering using k-
Means algorithm. Compare the results of these two algorithms and comment on the quality of clustering. You
can add Java/Python ML library classes/API in the program.
8. Write a program to implement k-Nearest Neighbour algorithm to classify the iris data set. Print both correct and
wrong predictions. Java/Python ML library classes can be used for this problem.
9. Implement the non-parametric Locally Weighted
Watch Video Tutorial at
Regression algorithm in order to fit data points. Select
https://www.youtube.com/@MaheshHuddar
appropriate data set for your experiment and draw graphs
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A Few Quotes
• “A breakthrough in machine learning would be worth ten Microsofts” (Bill Gates,
Chairman, Microsoft)

ar Director, DARPA)
• “Machine learning is the next Internet” (Tony Tether,
d
u d
H Hennessy, President, Stanford)
• Machine learning is the hot new thing” (John
h
s
e of machine learning” (Prabhakar Raghavan,
•
a h
“Web rankings today are mostly a matter
Dir. Research, Yahoo)
M
• “Machine learning is going to result in a real revolution” (Greg Papadopoulos, CTO,
Sun)

• “Machine learning is today’s discontinuity” (Jerry Yang, CEO, Yahoo)

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Introduction to Machine Learning
• We have seen Machine Learning as a buzzword for the past few years, the
reason for this might be the high amount of data production by applications,
the increase of computation power in the pastarfew years and the development
d d
of better algorithms. H u
s h
e
hor application that utilizes it.
• a
You may already be using a device
M
• For example, GMAIL, WhatsApp, E-Commerce Website, Video Sharing
Platforms, a wearable fitness tracker like Fitbit, or an intelligent home assistant
like Google Home.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 History of Machine Learning

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 What is Machine Learning..?
• “Learning is any process by which a system improves performance from
experience.” - Herbert Simon
a r
d d
H u
s h concerned with the design and
• A branch of artificial intelligence,
e h
M a
development of algorithms that allow computers to evolve behaviors
based on empirical data.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 What is Machine Learning..?
• Definition by Tom Mitchell (1998):

• Machine Learning is the study of algorithms that

a r
– improve their performance P d d
H u
s h
he
– at some task T
M a
– with experience E.

• A well-defined Machine Learning task is given by <P, T, E>.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

A classic example of a task that requires machine learning

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Handwritten Digit Recognition Problem
• Task T: Recognizing and Classifying handwritten words within
images
a r
d d
• Performance P: percent of wordsHucorrectly classified
s h
• Experience E: a database ofh e
M a handwritten words with given
classifications

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A robot driving learning problem
• Task T: driving on public four-lane highways using vision sensors

• Performance P: average distance traveled before an error (as

a r
d d
judged by human overseer) H u
s h
h e
• Experience E: a sequence aof images and steering commands
M
recorded while observing a human driver

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A checkers learning problem
• Task T: playing checkers

• Performance P: percent of games won against opponents

a r
d d
• Experience E: playing practice games
H u against itself
s h
h e
M a

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 How Machine Learning Works?

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Types of Learning Algorithms
• Supervised learning

• Unsupervised learning
a r
• Reinforcement learning d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Supervised Learning
• In Supervised learning, an AI system is presented with data which is labeled, which
means that each data tagged with the correct label.

arwell that when you have new

• The goal is to approximate the mapping function so
d
d
input data (x) that you can predict the outputuvariables (Y) for that data.
h H
s
e taken some data and marked them as
• h
As shown in the example, we have initially
a
Mdata is used by the training supervised model, this
‘Spam’ or ‘Not Spam’. This labeled
data is used to train the model.

• Once it is trained we can test our model by testing it with some test new mails and
checking of the model is able to predict the right output.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Types of Supervised learning
• Classification: A classification problem is when the output variable is a
category, such as “red” or “blue” or “disease” and “no disease”.
a r
d d
H u
• Regression: A regression problem s
ish when the output variable is a real
e h
M a
value, such as “dollars” or “weight”.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Types of Algorithms

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Unsupervised Learning Algorithm
• In unsupervised learning, an AI system is presented with unlabeled, uncategorized
data and the system’s algorithms act on the data without prior training.

r model which are ‘Ducks’ and

• In the example, we have given some characters to our
a
d d
‘Not Ducks’.
H u
• In our training data, we don’t provide e
h
s label to the corresponding data.
any
a h
M
• The unsupervised model is able to separate both the characters by looking at the type
of data and models the underlying structure or distribution in the data in order to
learn more about it.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Types of Unsupervised learning
• Clustering: A clustering problem is where you want to discover the
inherent groupings in the data, such as grouping customers by
purchasing behavior. a r
d d
H u
s h
he
a learning problem is where you want to
• Association: An associationMrule
discover rules that describe large portions of your data, such as people
that buy X also tend to buy Y.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Types of Algorithms

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement Learning
• A reinforcement learning algorithm, or agent, learns by interacting with
its environment.
a r
• The agent receives rewards by performing d d correctly and penalties for
H u
performing incorrectly. s h
h e
M a
• The agent learns without intervention from a human by maximizing its
reward and minimizing its penalty.

• It is a type of dynamic programming that trains algorithms using a system

of reward and punishment.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Reinforcement Learning
• In the example, we can see that the agent is given 2 options i.e. a path with water or
a path with fire.

r i.e. if the agent uses the fire

• A reinforcement algorithm works on reward a system
a
d
d tries to learn that it should avoid the
path then the rewards are subtracted and agent
H u
s h
fire path.
h e
a
• If it had chosen the water path orMthe safe path then some points would have been
added to the reward points, the agent then would try to learn what path is safe and
what path isn’t.

• It is basically leveraging the rewards obtained, the agent improves its environment
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
knowledge to select the next action.
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Applications of Machine Learning
• Recognizing patterns:

– Facial identities or facial expressions

– Handwritten or spoken words

a r
– Medical images d d
H u
• Generating patterns:
s h
h e
– a
Generating images or motion sequences
M
• Recognizing anomalies:

– Unusual sequences of credit card transactions

– Unusual patterns of sensor readings in a nuclear power plant or unusual sound in your car engine.

• Prediction:
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
– Future stock prices or currency exchange rates
 Influence of Disciplines on Machine Learning
Artificial intelligence

• Learning symbolic representations of concepts. Machine learning as a search problem. Learning as an approach to
improving problem solving. Using prior knowledge together with training data to guide learning.

a r
Bayesian methods
d d
•
H u
Bayes' theorem as the basis for calculating probabilities of hypotheses. The naive Bayes classifier. Algorithms for

s h
he
estimating values of unobserved variables.

Computational complexity theory

M a
• Theoretical bounds on the inherent complexity of different learning tasks, measured in terms of the
computational effort, number of training examples, number of mistakes, etc. required in order to learn.

Control theory

• Procedures that learn to control processes in order to optimize predefined objectives and that learn to predict the
next state of the process they are controlling.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Influence of Disciplines on Machine Learning
Information theory

• Measures of entropy and information content. Minimum description length approaches to learning. Optimal
codes and their relationship to optimal training sequences for encoding a hypothesis.

Philosophy
a r
d d
•
H u
Occam's razor, suggesting that the simplest hypothesis is the best. Analysis of the justification for generalizing
beyond observed data.
s h
Psychology and neurobiology a he
•
M
The power law of practice, which states that over a very broad range of learning problems,

• people's response time improves with practice according to a power law. Neurobiological studies motivating
artificial neural network models of learning.

Statistics

• Characterization of errors (e.g., bias and variance) that occur when estimating the accuracy of a hypothesis based
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
on a limited sample of data. Confidence intervals, statistical tests.
 CONCEPT LEARNING
• The problem of inducing general functions from specific training
examples is central to learning.
a r
• Concept learning can be formulated as a dproblem of searching through a
u d
h H
predefined space of potential hypotheses for the hypothesis that best fits
es
a h
the training examples. M
• What is Concept Learning…?

• “A task of acquiring potential hypothesis (solution) that best fits the

given training examples.”
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 CONCEPT LEARNING

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK
• Consider the example task of learning the target concept "days on which
XYZ enjoys his favorite water sport."
a r
• Table describes a set of example days,deach represented by a set of
u d
attributes. The attribute EnjoySport h H
indicates whether or not XYZ enjoys
es
a h
M day.
his favorite water sport on this

• The task is to learn to predict the value of EnjoySport for an arbitrary day,
based on the values of its other attributes.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK - Notation

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK
• What hypothesis representation shall we provide to the learner in this case?

• Let us begin by considering a simple representation in which each hypothesis consists

r
of a conjunction of constraints on the instance attributes.
a
d d
• In particular, let each hypothesis be a vectoruof six constraints, specifying the values
h H
es Wind, Water, and Forecast.
h
of the six attributes Sky, AirTemp, Humidity,
M a
• For each attribute, the hypothesis will either

– indicate by a "?' that any value is acceptable for this attribute,

– specify a single required value (e.g., Warm) for the attribute, or

– indicate by a "ø" that no value is acceptable.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 A CONCEPT LEARNING TASK
• If some instance x satisfies all the constraints of hypothesis h,
then h classifies x as a positive example (h(x) = 1).
a r
d d
• To illustrate, the hypothesis thatu Prabhas enjoys his favorite
h H
s
ehigh humidity (independent of the
sport only on cold days with
a h
M
values of the other attributes) is represented by the expression

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK

a r
d d
H u
s h
a he
M
1. (Sunny, Warm, Normal, Strong, Warm, Same)

2. (Rainy, Warm, High, Strong, Warm, Same)

3. (Rainy, Cold, High, Strong, Warm, Change)

4.Watch(?,
Video Tutorial at
Cold, High, ?, ?, ?) https://www.youtube.com/@MaheshHuddar
 A CONCEPT LEARNING TASK
• The most general hypothesis-that every day is a positive example-is
represented by
a r
d
(?, ?, ?, ?, ?, ?)
d
H u
s h
h e
a
M hypothesis-that no day is a positive
• and the most specific possible
example-is represented by

(ø, ø, ø, ø, ø, ø)

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK - Notation

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK - Notation

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK - Search
• Concept learning can be viewed as the task of searching through a large
space of hypotheses implicitly defined by the hypothesis representation.
a r
d d
• The goal of this search is to find the hypothesis that best fits the training
H u
examples. s h
h e
a
• It is important to note that Mby selecting a hypothesis representation, the
designer of the learning algorithm implicitly defines the space of all
hypotheses that the program can ever represent and therefore can ever
learn.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 A CONCEPT LEARNING TASK - Search
Instance Space:
• Consider, for example, the instances X and hypotheses H in the EnjoySport
a r
learning task. d d
H u
sh possible values, and that AirTemp,
• Given that the attribute Sky has ethree
a h
Humidity, Wind, Water, and M Forecast each have two possible values, the
instance space X contains exactly 3 . 2 . 2 . 2 . 2 . 2 = 96 distinct instances.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK - Search
F1 - > A, B
F2 - > X, Y
3 * 3 = 9 + 1 = 10
a r
(?, ø) d d
H u
s h
he
a Y) – 4 Instances
Instance Space: (A, X), (A, Y), (B, X), (B,
M
Hypothesis Space: (A, X), (A, Y), (A, ø), (A, ?), (B, X), (B, Y), (B, ø), (B, ?), (ø, X), (ø, Y),
(ø, ø), (ø, ?), (?, X), (?, Y), (?, ø), (?, ?) - 16

Hypothesis Space: (A, X), (A, Y), https://www.youtube.com/@MaheshHuddar

Watch Video Tutorial at
(A, ?), (B, X), (B, Y), (B, ?), (?, X), (?, Y (?, ?) (ø, ø)- 10
 A CONCEPT LEARNING TASK – Instance Space

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A CONCEPT LEARNING TASK - Search
Hypothesis Space
• Similarly there are 5 . 4 . 4 . 4 . 4 . 4 = 5120 syntactically distinct hypotheses within H.
a r
• Notice, however, that every hypothesis containing
d d one or more "ø" symbols represents
H u
s h
the empty set of instances; that is, it classifies every instance as negative.
e
h distinct hypotheses is only 1 + (4 . 3 . 3 . 3 . 3 . 3) =
•
M
Therefore, the number of semanticallya
973.

• Our EnjoySport example is a very simple learning task, with a relatively small, finite
hypothesis space.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 A CONCEPT LEARNING TASK – Hypothesis Space

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 General-to-Specific Ordering of Hypotheses
• To illustrate the general-to-specific ordering, consider the two hypotheses
h1 = (Sunny, ?, ?, Strong, ?, ?)
a r
d
d ?, ?, ?)
h2 = (Sunny, ?,u?,
h H
esthat are classified positive by hl and by
• Now consider the sets of instances
h
M a
h2. Because h2 imposes fewer constraints on the instance, it classifies
more instances as positive.
• In fact, any instance classified positive by h1 will also be classified positive
by h2. Therefore, we say that
Watch Video Tutorial at
h2 is more general than h1.
https://www.youtube.com/@MaheshHuddar
 More General Than hypothesis

• For any instance x in X and hypothesis h in H, we say that x satisjies h

if and only if h(x) = 1.
a r
d d
H u
• We define the more_general_than_or_equale_to relation in terms of
s h
h
the sets of instances that satisfye the two hypotheses:
M a

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 More General Than hypothesis

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 FIND-S: FINDING A MAXIMALLY SPECIFIC HYPOTHESIS

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 FIND-S Algorithm
Finding A Maximally Specific a r Hypothesis
d d
H u
Solved Example
sh -1
h e
M a

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 FIND-S: Step-1

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

FIND-S: Step-2

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

FIND-S: Step-2

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

FIND-S: Step-2

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

FIND-S: Step-2

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 FIND-S: Step-2

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 FIND-S Algorithm
Finding A Maximally Specific r Hypothesis
d a
u d
Solved Example
H -2
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 FIND-S Algorithm Solved Example - 2

a r
d d
H u
s h
h e this instance space?
1. a
How many concepts are possible
M
for

2. How many hypotheses can be expressed by the hypothesis language?

3. Apply the FIND-S algorithm by hand on the given training set. Consider the

examples in the specified order and write down your hypothesis each time after
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
observing an example.
 a r
d d
H u
shinstance space?
1. How many concepts are possible for this
e
a h
Solution: 2 * 3 * 2 * 2 * 3 = 72 M
2. How many hypotheses can be expressed by the hypothesis language?

Solution: 4 * 5 * 4 * 4 * 5 = 1600

Semantically Distinct Hypothesishttps://www.youtube.com/@MaheshHuddar

Watch Video Tutorial at = ( 3 * 4 * 3 * 3 * 4 ) + 1 = 433
Step 1:

a r
h0 = (ø, ø, ø, ø, ø)
d d
H u
Step 2:
s h
he
a many) – No
X1 = (some, small, no, expensive,
M
Negative Example Hence Ignore

h1 = (ø, ø, ø, ø, ø)

X2 = (many, big, no, expensive, one) – Yes

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
h2 = (many, big, no, expensive, one)
Step 2:

a r
h2 = (many, big, no, expensive, one)
d d
H u
X3 = (some, big, always, expensive, h
few) – No
es
a
Negative example hence Ignore h
M
h3 = (many, big, no, expensive, one)

X4 = (many, medium, no, expensive, many) – Yes

h4 = (many, ?, no, expensive, ?)

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
Step 2:

a r
h4 = (many, ?, no, expensive, ?)
d d
H u
h – Yes
X5 = (many, small, no, affordable, many)
s
h e
h5 = (many, ?, no, ?, ?)
M a
Step 3:

Final Hypothesis is:

h5 = (many, ?, no, ?, ?)
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 FIND-S Algorithm
r
d a
UnansweredHQuestions
u d
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Unanswered Questions of FIND-S Algorithm
1. Has the learner converged to the correct target concept? Although FIND-S will find
a hypothesis consistent with the training data, it has no way to determine whether
a r with the data (i.e., the correct
d
it has found the only hypothesis in H consistent
d
H
target concept), or whether there are many
u other consistent hypotheses as well.
s h
h e
2. a
Why prefer the most specific hypothesis?
M
In case there are multiple hypotheses
consistent with the training examples, FIND-S will find the most specific. It is
unclear whether we should prefer this hypothesis over the most general, or some
other hypothesis of intermediate generality.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Unanswered Questions of FIND-S Algorithm
3. Are the training examples consistent? In most practical learning problems there is some

chance that the training examples will contain at least some errors or noise. Such inconsistent

a r given the fact that it ignores negative

d
sets of training examples can severely mislead FIND-S,
d
examples. We would prefer an algorithm thatH
u
s h could at least detect when the training data is
h e such errors.
a
inconsistent and, preferably, accommodate
M
4. What if there are several maximally specific consistent hypotheses? In the hypothesis

language H for the EnjoySport task, there is always a unique, most specific hypothesis

consistent with any set of positive examples. However, for other hypothesis spaces there can

be several
Watch maximally
Video Tutorial at specific hypotheses consistent with the data.
https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Consistent Hypothesis and Version Space
▪ The idea: output a description of the set of all hypotheses consistent with the
training examples (correctly classify training examples).
a r
d hypotheses which are consistent
▪ Version Space: a representation of the setdof
H u
with D s h
h e
M a
1. an explicit list of hypotheses (List-Then-Eliminate)

2. a compact representation of hypotheses which exploits the

more_general_than partial ordering (Candidate-Elimination)
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Consistent Hypothesis and Version Space
An hypothesis h is consistent with a set of training examples D iff h(x) = c(x) for each

example in D
Consistent(h, D)  ( x, c(x)  D) h(x) = c(x))r
d a
u d
h H
Example Citations Size
s
InLibrary Price Editions Buy
1 Some Small a he No Affordable One No
2 Many BigM No Expensive Many Yes

h1 = (?, ?, No, ?, Many) – Consistent

h2 = (?, ?, No, ?, ?) – Not Consistent

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Consistent Hypothesis and Version Space

• The version space VSH,D is the subset of the hypothesis from H

consistent with the training example in D
a r
d d
H u
VSH,D {h  H | Consistent(h, D)}
s h
h e
M a

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 List-Then-Eliminate algorithm
Version space as list of hypotheses

1. VersionSpace  a list containing

a r every hypothesis in H
d d
H u
2. s h x, c(x) Remove from
For each training example,
e h
M a
VersionSpace any hypothesis h for which h(x)  c(x)

3. Output the list of hypotheses in VersionSpace

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Consistent Hypothesis and Version Space
• F1 - > A, B
• F2 - > X, Y
a r
• Instance Space: (A, X), (A, Y), (B, X), (B, Y) – d4dExamples
H u
s h
h eø), (A, ?), (B, X), (B, Y), (B, ø), (B, ?), (ø, X), (ø,
a
• Hypothesis Space: (A, X), (A, Y), (A,
M
Y), (ø, ø), (ø, ?), (?, X), (?, Y), (?, ø), (?, ?) - 16 Hypothesis

• Semantically Distinct Hypothesis : (A, X), (A, Y), (A, ?), (B, X), (B, Y), (B, ?), (?, X),
(?, Y (?, ?), (ø, ø) – 10
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Consistent Hypothesis and Version Space

• Version Space: (A, X), (A, Y), (A, ?), (B, X), (B, Y), (B, ?), (?, X), (?, Y) (?, ?), (ø, ø),

• Training Instances
a r
d
F1 F2
H ud
Target
Xs h
A
h e Yes
M a
A Y Yes

• Consistent Hypothesis are: (A, ?), (?, ?)

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 List-Then-Eliminate algorithm

Problems

• a r
The hypothesis space mustddbe finite
H u
s h
• h
Enumeration of all the e hypothesis, rather inefficient
M a

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

Candidate Elimination Algorithm
r
Solved Example
Hu
d d a - 1
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
S0: , , , , . 

S1: Sunny,Warm, Normal, Strong, Warm, Same

a r
d d
H u
S2: S3:
s h
Sunny,Warm, ?, Strong, Warm, Same

h e
S4 Sunny, Warm,
M a ?, Strong, ?, ?

G4: Sunny, ?, ?, ?, ?, ? ?, Warm, ?, ?, ?, ?

G3: Sunny,?,?,?,?,? ?,Warm,?,?,?,? ?,?,Normal,?,?,? ?, ?,?,?,Cool,? ?,?,?,?,?,Same

G0:Watch G
Video Gat2:
1: Tutorial ?, ?, ?, ?, ?, ?
https://www.youtube.com/@MaheshHuddar
Learned Version Space by Candidate Elimination Algorithm
S Sunny, Warm, ?, Strong, ?, ?

a r
d d
H u
Sunny, ?, ?, Strong, ?, ? s h
Sunny, Warm, ?, ?, ?, ? ?, Warm, ?, Strong, ?, ?

a he
M

G Sunny, ?, ?, ?, ?, ? ?, Warm, ?, ?, ?, ?

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

Learned Version Space by Candidate Elimination Algorithm

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 New instances to be classified

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

Candidate Elimination Algorithm
r
Solved Example
Hu
d d a - 2
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 S0: (0, 0, 0)
Candidate Elimination Algorithm Solved Example - 2
S1: (0, 0, 0)

S2: (0, 0, 0)

S3: (Small, Red, Circle)

S4: (Small, Red, Circle)

a r
d d
S5: (Small, ?, Circle)
H u
S: G: (Small, ?, Circle)
s h
G5: (Small, ?, Circle) a he
M
G4: (Small, ?, Circle)

G3: (Small, ?, Circle)

G2: (Small, Blue, ?) (Small, ?, Circle) (?, Blue, ?) (Big, ?, Triangle) (?, Blue, Triangle)

G1: (Small, ?, ?) (?, Blue, ?) (?, ?, Triangle)

Watch
G0:Video
(?, Tutorial
?, ?) at https://www.youtube.com/@MaheshHuddar
Candidate Elimination Algorithm
Solved Example
d a r - 3
u d
h H
s
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
• S0: (0, 0, 0, 0, 0) Candidate Elimination Algorithm Solved Example - 3
• S1: (0, 0, 0, 0, 0)
• S2: (Many, Big, No, Exp, One)
• S3: (Many, Big, No, Exp, One)
• S4: (Many, ?, No, Exp, ?)
a r
• S5: (Many, ?, No, ?, ?)
d d
• (Many, ?, No, ?, ?)
H u
s h
Final Hypothesis Set: (Many, ?, No, ?, ?) (Many, ?, ?, ?, ?)
he
• G5: (Many, ?, ?, ?, ?)
• M a
G4: (Many,?,?,?,?) (Many,?,?,Exp,?) (?,?,No,exp,?)
• G3: (Many,?,?,?,?) (Many, big,?,?,?) (?,Big,no,?,?) (?,Big,?,Aff,?) (?,Big,?,?,Many) (?,Big,?,?,One) (Many,?,?,Exp,?)
(?,Small,?,Exp,?) (?,Medium,?,Exp,?) (?,?,No,exp,?) (?,?,?,Exp,one) (?,?,?,Exp,many) (?,?,?,?,One)
• G2: (Many,?,?,?, ?) (?, Big,?,?,?) (?,?,?,Exp,?) (?,?,?,?,One)
• G1: (Many,?,?,?, ?) (?, Big,?,?,?) (?,Medium,?,?,?) (?,?,Always,?,?) (?,?,?,Exp,?) (?,?,?,?,One) (?,?,?,?,Few)
• G0: Video
Watch (?, ?,Tutorial
?, ?, at?) https://www.youtube.com/@MaheshHuddar
Candidate Elimination Algorithm
Solved Example
d a r - 4
u d
h H
s
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Candidate Elimination Algorithm - Solved Example - 4
Target
S0: (0, 0, 0, 0, 0) Example Shape Size Color Surface Thickness
Concept
S1: (Circular, Large, Light, Smooth, Thick) Malignant
1 Circular Large Light Smooth Thick
(+)
S2: (Circular, Large, Light, ?, Thick) Malignant
2 Circular Large Light Irregular Thick
a r (+)
Oval d Large
d
S3: (Circular, Large, Light, ?, Thick) 3 Dark Smooth Thin Benign (-)
H u Malignant
S4: (?, Large, Light, ?, Thick) 4
s h Oval Large Light Irregular Thick
(+)
h e
G4: (?, ?, Light, ?, ?) (?, ?, ?, ?, Thick)
M a
G3: (Circular, ?, ?, ?, ?) (?, ?, Light, ?, ?) (?, ?, ?, ?, Thick)

G2: (?, ?, ?, ?, ?)

G1: (?, ?, ?, ?, ?)

G0: (?, ?, ?, ?, ?)
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
Candidate Elimination Algorithm
r
Solved Example
Hu
d d a - 5
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Candidate Elimination Algorithm - Solved Example - 5
S0: (0, 0, 0, 0, 0) Ex Eyes Nose Head Fcolor Hair Smile

S1: (Round, Triangle, Round, Purple, Yes) 1 Round Triangle Round Purple Yes Yes

S2: (Round, Triangle, Round, Purple, Yes) 2 Square Square Square Green Yes No

S3: (?, Triangle, Round, ?, Yes)

ar
3 Square Triangle Round Yellow Yes Yes
S4: (?, Triangle, Round, ?, Yes)
d d
S5: (?, ?, Round, ?, Yes)
4 Round
H uTriangle Round Green No No

5
s h Square
Square Round Yellow Yes Yes
G5: (?, ?, Round, ?, Yes)
h e
M a
G4: (Square, Triangle, ?, ?, ?) (?, Triangle, Square, ?, ?) (?, Triangle, ?, Yellow, ?) (?, Triangle, ?, Purple, ?) (?, Triangle, ?, ?, yes)
(Square, ?, Round, ?, ?) (?, Square, Round, ?, ?) (?, ?, Round, Yellow, ?) (?, ?, Round, Purple, ?) (?, ?, Round, ?, Yes)
G3: (?, Triangle, ?, ?, ?) (?, ?, Round, ?, ?)
G2: (Round, ?, ?, ?, ?) (?, Triangle, ?, ?, ?) (?, ?, Round, ?, ?) (?, ?, ?, Purple, ?)
G1: (?, ?, ?, ?, ?)
G0: (?, ?,Watch
?, ?,Video
?) Tutorial at https://www.youtube.com/@MaheshHuddar
 Candidate Elimination Algorithm - Solved Example - 5
S0: (0, 0, 0, 0, 0) Ex Eyes Nose Head Fcolor Hair Smile

S1: (Round, Triangle, Round, Purple, Yes) 1 Round Triangle Round Purple Yes Yes

S2: (Round, Triangle, Round, Purple, Yes) 2 Square Square Square Green Yes No

S3: (?, Triangle, Round, ?, Yes)

ar
3 Square Triangle Round Yellow Yes Yes
S4: (?, Triangle, Round, ?, Yes)
d d
S5: (?, ?, Round, ?, Yes)
4 Round
H uTriangle Round Green No No

5
s h Square
Square Round Yellow Yes Yes
G5: (?, ?, Round, ?, Yes)
h e
M a
G4: (Square, Triangle, ?, ?, ?) (?, Triangle, Square, ?, ?) (?, Triangle, ?, Yellow, ?) (?, Triangle, ?, Purple, ?) (?, Triangle, ?, ?, yes)
(Square, ?, Round, ?, ?) (?, Square, Round, ?, ?) (?, ?, Round, Yellow, ?) (?, ?, Round, Purple, ?) (?, ?, Round, ?, Yes)
G3: (?, Triangle, ?, ?, ?) (?, ?, Round, ?, ?)
G2: (Round, ?, ?, ?, ?) (?, Triangle, ?, ?, ?) (?, ?, Round, ?, ?) (?, ?, ?, Purple, ?)
G1: (?, ?, ?, ?, ?)
G0: (?, ?,Watch
?, ?,Video
?) Tutorial at https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

Candidate Elimination Algorithm
a r
d d
H u
s h
a he
M

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Candidate elimination algorithm
For each training example d, do:
If d is positive example
Remove from G any hypothesis h inconsistent with d
For each hypothesis s in S not consistent with d:
a r
• Remove s from S
d d
• H u
Add to S all minimal generalizations of s consistent with d and having a generalization in G
• Remove from S any hypothesis witheasmoreh specific h in S
a h
If d is negative example M
Remove from S any hypothesis h inconsistent with d
For each hypothesis g in G not consistent with d:
• Remove g from G
• Add to G all minimal specializations of g consistent with d and having a specialization in S
• Remove from G any hypothesis having a more general hypothesis in G
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Remarks On Version Spaces And Candidate-Elimination Algorithm

1. Will the CANDIDATE-ELIMINATION Algorithm Converge to the

Correct Hypothesis?
a r
d d
H u
s h
2. How Can Partially Learned Concepts Be Used?
h e
M a
3. What Training Example Should the Learner Request Next?

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Remarks On Version Spaces And Candidate-Elimination Algorithm

Will the CANDIDATE-ELIMINATION Algorithm Converge to the Correct Hypothesis?

• The learned Version Space correctly describes the target concept, provided:

a r
d
1. There are no errors in the training examples
d
H u
s h
2. There is some hypothesis that correctly describes the target concept
h e
M a
• If S and G converge to a single hypothesis the concept is exactly learned

• An empty version space means no hypothesis in H is consistent with training

examples
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Remarks On Version Spaces And Candidate-Elimination Algorithm
How Can Partially Learned Concepts Be Used?

• The learner is required to classify new instances that it has not yet
observed. a r
d d
u
H contains multiple hypotheses,
• Even though the version space ofshFigure
h e
M a
indicating that the target concept has not yet been fully learned, it is
possible to classify certain examples with the same degree of
confidence as if the target concept had been uniquely identified.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Remarks On Version Spaces And Candidate-Elimination Algorithm
How Can Partially Learned Concepts Be Used?

• Note that although instance A was not among the training examples, it is classified as a
positive instance by every hypothesis in the current version space (shown in Figure).
a r
•
d
Because the hypotheses in the version space unanimouslyd agree that this is a positive
H u
instance, the learner can classify instance A has positive with the same confidence it would
es
h
have if it had already converged to theasingle, correct target concept.
M
• Regardless of which hypothesis in the version space is eventually found to be the correct
target concept, it is already clear that it will classify instance A as a positive example.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Remarks On Version Spaces And Candidate-Elimination Algorithm
How Can Partially Learned Concepts Be Used?

• Similarly, instance B is classified as a negative instance by every hypothesis in the version space. This
instance can therefore be safely classified as negative, given the partially learned concept.
r hypotheses classify it as positive
• Instance C presents a different situation. Half of the versionaspace
d d
H
and half classify it as negative. Thus, the learner cannotu classify this example with confidence until
s h
further training examples are available.
h e
•
a
Finally, instance D is classified as positiveMby two of the version space hypotheses and negative by the
other four hypotheses. In this case we have less confidence in the classification than in the
unambiguous cases of instances A and B. Still, the vote is in favor of a negative classification, and one
approach we could take would be to output the majority vote, perhaps with a confidence rating
indicating how close the vote was.
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Remarks On Version Spaces And Candidate-Elimination Algorithm
How Can Partially Learned Concepts Be Used?

• The learner is required to classify new instances that it has not yet observed.

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Remarks On Version Spaces And Candidate-Elimination Algorithm
How Can Partially Learned Concepts Be Used?

a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Remarks On Version Spaces And Candidate-Elimination Algorithm
What Training Example Should the Learner Request Next?

• Up to this point we have assumed that training examples are provided to the

a r
learner by some external teacher.
d d
H u
sh to conduct experiments in which it
• Suppose instead that the learner is allowed
e
a h
chooses the next instance, thenM
obtains the correct classification for this instance

from an external nature.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Remarks On Version Spaces And Candidate-Elimination Algorithm
What Training Example Should the Learner Request Next?

• Consider the version space learned from the four training examples of the

a r
Enjoysport concept.
d d
H u
sh to pose at this point?
• What would be a good query for the learner
e
a h
• What is a good query strategy inMgeneral?

• The learner should choose an instance that would be classified positive by some

of these hypotheses, but negative by others.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 a r
d d
H u
s h
a he
M

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Unbiased Learner
a r
Candidate Elimination Algorithm d
ud H
s h
h e
M a

Dr. Mahesh G Huddar

Dept. of Computer Science and Engineering
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Inductive Bias of Candidate-Elimination Algorithm
• The CANDIDATE-ELIMINATION Algorithm will converge toward the true target concept

provided

– it is given accurate training examples and a r

d d
H u
h concept.
– its initial hypothesis space contains the target
s
h e
M a
• What if the target concept is not contained in the hypothesis space?

• Can we avoid this difficulty by using a hypothesis space that includes every possible

hypothesis?

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Inductive Bias of Candidate-Elimination Algorithm
A Biased Hypothesis Space
• Suppose we wish to assure that the hypothesis space contains the unknown
target concept.
a r
d
ud space to include every possible
• The obvious solution is to enrich the hypothesis
H
s h
hypothesis.
h e
a
M example in which we restricted the
• To illustrate, consider the EnjoySport
hypothesis space to include only conjunctions of attribute values.
<Sunny, High, Normal, Strong, Cool, Same>

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Inductive Bias of Candidate-Elimination Algorithm
A Biased Hypothesis Space

a r
d d
H u
S0: <, , , , . >
s h
S1: <Sunny, Warm,h e Strong, Cool, Change>
a Normal,
S2: <?, Warm,M
Normal, Strong, Cool, Change>
S3: <?, Warm, Normal, Strong, Cool, Change>
G2: <?, ?, ?, ?, ?, ?>
G1: <?, ?, ?, ?, ?, ?>
G0: <?, ?, ?, ?, ?, ?>
Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Inductive Bias of Candidate-Elimination Algorithm
A Biased Hypothesis Space

a r
d d
H u
s h
h e space is unable to represent even
• Because of the restriction, the hypothesis
M a
simple disjunctive target concepts such as "Sky = Sunny or Sky = Cloudy.“

• < “Sky = Sunny or Sky = Cloudy”, Warm, Normal, Strong, Cool, Change>

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Inductive Bias of Candidate-Elimination Algorithm
Unbiased Learner

• The obvious solution to the problem of assuring that the target concept is in the

hypothesis space H is to provide a hypothesis d ar capable of representing every

space
u d
H
teachable concept; that is, it is capableshof representing every possible subset of
h e
M a
the instances X.

• In general, the set of all subsets of a set X is called the powerset of X.

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Inductive Bias of Candidate-Elimination Algorithm
Unbiased Learner

• In the EnjoySport learning task, for example, the size of the instance space X of

days described by the six available attributes isda r

96.
u d
h H
• |H'| = 2|X|, or 296 (vs |H| = 973, a strongsbias)
h e
a
• < “Sky = Sunny or Sky = Cloudy”,M?, ?, ?, ?>
Sunny, ?, ?, ?, ?, ? V <Cloudy, ?, ?, ?, ?, ?
• We are guaranteed that the target concept exists

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Inductive Bias of Candidate-Elimination Algorithm
Inductive System

a r
d d
H u
s h
a he
Equivalent M
Deductive System

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 The Inductive Learning Hypothesis

The Inductive Learning Hypothesis

a r
Any hypothesis found to approximate dthe d target function well over a
H u
h
sufficiently large set of training sexamples will also approximate the
h e
a
M unobserved examples.
target function well over other

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Issues in Machine Learning
1. What algorithms exist for learning general target functions from specific training
examples? In what settings will particular algorithms converge to the desired

r
function, given sufficient training data? Which algorithms perform best for which
a
d d
types of problems and representations?
H u
s h general bounds can be found to relate
h e
2. How much training data is sufficient? What
a
M to the amount of training experience and the
the confidence in learned hypotheses
character of the learner's hypothesis space?

3. When and how can prior knowledge held by the learner guide the process of
generalizing from examples? Can prior knowledge be helpful even when it is only
Watch Video Tutorial at
approximately correct? https://www.youtube.com/@MaheshHuddar
 Issues in Machine Learning
4. What is the best strategy for choosing a useful next training experience, and
how does the choice of this strategy alter the complexity of the learning
problem?
a r
d d
5. What is the best way to reduce the H u
learning task to one or more function
s h
h e way, what specific functions should the
a
approximation problems? Put another
M
system attempt to learn? Can this process itself be automated?

6. How can the learner automatically alter its representation to improve its
ability to represent and learn the target function?

Watch Video Tutorial at https://www.youtube.com/@MaheshHuddar