MODULE 1 – CONCEPT LEARNING

CONCEPT LEARNING
1. Define concept learning and discuss with example.
Solution:
▪ Learning involves acquiring general concepts from specific training examples. Example: People
continually learn general concepts or categories such as "bird," "car," "situations in which I should
study more in order to pass the exam," etc.
▪ Each such concept can be viewed as describing some subset of objects or events defined over a
larger set
▪ Alternatively, each concept can be thought of as a Boolean-valued function defined over this larger
set. (Example: A function defined over all animals, whose value is true for birds and false for other
animals).
Concept learning - Inferring a Boolean-valued function from training examples of its input and
output

Concept Learning Task :

Consider the example task of learning the target concept
"Days on which my friend Aldo enjoys his favorite water sport."

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

Sunny Warm Normal Strong Warm Same Yes

Sunny Warm High Strong Warm Same Yes

Rainy Cold High Strong Warm Change No

Sunny Warm High Strong Cool Change Yes

The attribute EnjoySport indicates whether or not a Person enjoys his favorite water sport on this day.

The task is to learn to predict the value of EnjoySport for an arbitrary day, based on the values of its otherattributes ?

What hypothesis representation is provided to the learner?

For each attribute, the hypothesis will either
• Indicate by a "?' that any value is acceptable for this attribute,
• Specify a single required value (e.g., Warm) for the attribute, or
• Indicate by a "Φ" that no value is acceptable

If some instance x satisfies all the constraints of hypothesis h, then h classifies x as a positive example (h(x)
= 1).

The hypothesis that PERSON enjoys his favorite sport only on cold days with high humidity (independent
of the values of the other attributes) is represented by the expression
(?, Cold, High, ?, ?, ?)
 The most general hypothesis-that every day is a positive example-is represented by
(?, ?, ?, ?, ?, ?)

The most specific possible hypothesis-that no day is positive is represented by

(Φ , Φ, Φ, Φ, Φ, Φ)

Example: X is the set of all possible days, each represented by the attributes: Sky, AirTemp, Humidity,
Wind, Water, and Forecast

Concept learning as Search:

• Concept learning can be viewed as the task of searching through a large space of hypotheses
implicitly defined by the hypothesis representation.
• The goal of this search is to find the hypothesis that best fits the training examples.

Example, the instances X and hypotheses H in the EnjoySport learning task.

The attribute Sky has three possible values, and AirTemp, Humidity, Wind, Water
Forecast each have two possible values, the instance space X contains exactly

3.2.2.2.2.2 = 96 Distinct instances

5.4.4.4.4.4 = 5120 Syntactically distinct hypotheses within H.
Every hypothesis containing one or more " Φ" symbols represents the empty set of
instances; that is, it classifies every instance as negative.
• 1 + (4.3.3.3.3.3) = 973. semantically distinct hypothesis

2.Write FIND-S algorithm and explain with example given below

Solution:

FIND-S Algorithm
1. Initialize h to the most specific hypothesis in H
2. For each positive training instance x
For each attribute constraint ai in h
If the constraint ai is satisfied by x
Then do nothing
Else replace ai in h by the next more general constraint that is satisfied by x
3. Output hypothesis h

For Example refer Q16.

The key property of the FIND-S algorithm is

• FIND-S is guaranteed to output the most specific hypothesis within H

that is consistent with the positive training examples
 • FIND-S algorithm’s final hypothesis will also be consistent with the negative examples provided the
correct target concept is contained in H, and provided the training examples are correct.

Find S – Drawback

Questions still left unanswered

▪ Has the learner converged to the correct target concept?
▪ Why prefer the most specific hypothesis?

If multiple hypotheses consistent with the training examples, FIND-S will find the most specific. It is
unclear whether we should prefer this hypothesis

Are the training examples consistent?

Training examples may contain at least some errors or noise. Such inconsistent sets of training examples
can severely mislead FIND-S, since it ignores negative examples.

▪ What if there are several maximally specific consistent hypotheses?

There can be several maximally specific hypotheses consistent with the data. Find S finds only one.

3.Define Consistent Hypothesis and Version Space.

Solution:

Definition: A hypothesis h is consistent with a set of training examples D if and only if

h(x) = c(x) for each example (x, c(x)) in D.

Consistent (h, D) = (for all <x, c(x)> € D) h(x) = c(x))

Note difference between definitions of consistent and satisfies

• an example x is said to satisfy hypothesis h when h(x) = 1, regardless of whether x is a positive or
negative example of the target concept.
• an example x is said to consistent with hypothesis h iff h(x) = c(x)

Version Space:
The version space, denoted VSH,D with respect to hypothesis space H and training examples D, is the subset
of hypotheses from H consistent with the training examples in D

VSH,D ={h € H | Consistent(h, D)}

LIST-THEN-ELIMINATE Algorithm:

The LIST-THEN-ELIMINATE algorithm first initializes the version space to contain all hypotheses in H
and then eliminates any hypothesis found inconsistent with any training example.

1. VersionSpace c a list containing every hypothesis in H

2. For each training example, (x, c(x))
remove from VersionSpace any hypothesis h for which h(x) ≠ c(x)
3. Output the list of hypotheses in VersionSpace

List-Then-Eliminate works in principle, so long as version space is finite.

However, since it requires exhaustive enumeration of all hypotheses in practice it is not feasible.
 4.Write the candidate elimination algorithm and illustrate with example.
Solution:

The CANDIDATE-ELIMINTION algorithm computes the version space containing all hypotheses from H
that are consistent with an observed sequence of training examples.

Algorithm:
1. Initialize G to the set of maximally general hypotheses in H
2. Initialize S to the set of maximally specific hypotheses in H
3. For each training example d, do
a. If d is a positive example
▪ Remove from G any hypothesis inconsistent with d,
▪ For each hypothesis s in S that is not consistent with d,
• Remove s from S
• Add to S all minimal generalizations h of s such that h is consistent with d,
and some member of G is more general than h
• Remove from S, hypothesis that is more general than another hypothesis in S
b. If d is a negative example
▪ Remove from S any hypothesis inconsistent with d
▪ For each hypothesis g in G that is not consistent with d
• Remove g from G
• Add to G all minimal specializations h of g such that h is consistent with d,
and some member of S is more specific than h
• Remove from G any hypothesis that is less general than another in G

For example refer Q16.

5. Explain in detail the Inductive Bias of Candidate Elimination algorithm.

Solution:

1. A Biased Hypothesis Space

▪ Suppose we wish to assure that the hypothesis space contains the unknown target concept.
▪ The obvious solution is to enrich the hypothesis space to include every possible hypothesis.
▪ Consider EnjoySport example in which we restricted the hypothesis space to include only
conjunctions of attribute values.

▪ Most specific hypothesis consistent with the first two examples

▪ It incorrectly covers the third (negative) training example
▪ The problem is that we have biased the learner to consider only conjunctive hypotheses.
 ▪ In this case we require a more expressive hypothesis space.

2. An unbiased learner

▪ The obvious solution to be a unbiased learner– design hypothesis space H to represent every
teachable concept;
▪ It should capable of representing every possible subset of the instances X. In general, the set of all
subsets of a set X is called the power-set of X.
▪ In general, number of distinct subsets is 2|X|.
▪ Thus, there are 296, or approximately distinct target concepts that could be defined over this instance
space and that our learner might be called upon to learn.

Our conjunctive hypothesis space is able to represent only 973 of these-a very biased hypothesisspace indeed!
▪ Let us reformulate the Enjoysport learning task
▪ Let H’ represent every subset of instances; that is, let H’ correspond to the power set of X.
▪ One way to define such an H' is to allow arbitrary disjunctions, conjunctions, and negations of our
earlier hypotheses.
▪ For instance, the target concept "Sky = Sunny or Sky = Cloudy" could then be described as

3. The Futility of Bias-Free Learning

▪ CEA generalizes observed training examples because it was biased by the implicit assumption that
the target concept could be represented by a conjunction of attribute values.
• If this assumption is correct (and the training examples are error-free), its classification of new
sample will also be correct.
• If this assumption is incorrect, however, it is certain that the CEA will mis-classify at least
some instances from X.

The input-output behavior of the CANDIDATE-ELIMINATION algorithm using a hypothesis space H is

identical to that of a deductive theorem prover utilizing the assertion “H contains the target concept." This
assertion is therefore called the inductive bias of the CANDIDATE ELIMINATION algorithm
characterizing inductive systems by their inductive bias allows modeling them by their equivalent
deductive systems. This provides a way to compare inductive systems according to their policies for
generalizing beyond the observed training data.

Problems w.r.t FIND-S and CANDIDATE ELIMINATION

FIND-S

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

Sunny Warm Normal Strong Warm Same Yes
Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
Sunny Warm High Strong Cool Change Yes

The first step of FIND-S is to initialize h to the most specific hypothesis in H

h - (Ø, Ø, Ø, Ø, Ø, Ø)

x1 = <Sunny Warm Normal Strong Warm Same>, +

Observing the first training example, it is clear that our hypothesis is too specific. In particular, none of the
"Ø" constraints in h are satisfied by this example, so each is replaced by the next more general constraint
that fits the example

h1 = <Sunny Warm Normal Strong Warm Same>

This h is still very specific; it asserts that all instances are negative except for the single positive training
example

x2 = <Sunny, Warm, High, Strong, Warm, Same>, +

The second training example forces the algorithm to further generalize h, this time substituting a "?' in place
of any attribute value in h that is not satisfied by the new example

h2 = <Sunny Warm ? Strong Warm Same>

x3 = <Rainy, Cold, High, Strong, Warm, Change>, -

Upon encountering the third training the algorithm makes no change to h. The FIND-S algorithm simply
ignores every negative example.

h3 = < Sunny Warm ? Strong Warm Same>

x4 = <Sunny Warm High Strong Cool Change>, +

The fourth example leads to a further generalization of h
h4 = < Sunny Warm ? Strong ? ? >
 CANDIDATE ELIMINATION:

Example-1:
Sky Airtemp Humidity Wind Water Forecast EnjoySport
Sunny Warm Normal Strong Warm Same Yes
Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
SUnny Warm High Strong Cool Change Yes

Step1: S0= {'0', '0', '0', '0', '0', '0'}

G0 = {'?', '?', '?', '?', '?', '?'}

Step2: <Sunny, Warm, Normal, Strong, Warm, Same> +

Our first training ex is positive so make changes in specific hypothesis and generalize it. There is no
Change in general hypothesis.

S1 = {Sunny, Warm, Normal, Strong, Warm, Same}

G1= {'?', '?', '?', '?', '?', '?'}

Consider the second training instance : <Sunny, Warm, High, Strong, Warm, Same> +
Compare each instance value with specific hypothesis value and generalize it. There is no change in
general hypothesis

G after removing inconsistent hypothesis with d:

{('?', '?', '?', '?', '?', '?')}
S generalization
Consider s: ('sunny', 'warm', 'normal', 'strong', 'warm', 'same')
s after min generalization
{('sunny', 'warm', '?', 'strong', 'warm', 'same')}
S2: {('sunny', 'warm', '?', 'strong', 'warm', 'same')}
G2: {('?', '?', '?', '?', '?', '?')}

Consider the third training instance : <Rainy, Cold, High, Strong, Warm, Change> -
-ve instance
S after removing consistent hypothesis with d
{('sunny', 'warm', '?', 'strong', 'warm',
'same')} G specialization
Consider g: ('?', '?', '?', '?', '?', '?')
g after min specialization: Compare g('?', '?', '?', '?', '?', '?') with all the instances of example and try
to make each ? with some value pair.
{('?', '?', '?', '?', 'cool', '?'), ('?', '?', '?', '?', '?', 'same'), ('?', 'warm', '?', '?', '?', '?'),
('sunny', '?', '?', '?', '?', '?'), ('?', '?', 'normal', '?', '?', '?')}
G after retaining g where g is consistent with d & some member of S is more specific than g
{('?', '?', '?', '?', '?', 'same'), ('?', 'warm', '?', '?', '?', '?'), ('sunny', '?', '?', '?', '?', '?')}

S3: {('sunny', 'warm', '?', 'strong', 'warm', 'same')}

G3: {('?', '?', '?', '?', '?', 'same'), ('?', 'warm', '?', '?', '?', '?'), ('sunny', '?', '?', '?', '?', '?')}

Consider the Fourth Training Instance d: ('sunny', 'warm', 'high', 'strong', 'cool', 'change', 'Y')
+ve instance
G after removing inconsistent hypothesis with d:
{('?', 'warm', '?', '?', '?', '?'), ('sunny', '?', '?', '?', '?', '?')}
S generalization
Consider s: ('sunny', 'warm', '?', 'strong', 'warm', 'same')
s after min generalization
{('sunny', 'warm', '?', 'strong', '?', '?')}

S4: {('sunny', 'warm', '?', 'strong', '?', '?')}

G4: {('?', 'warm', '?', '?', '?', '?'), ('sunny', '?', '?', '?', '?', '?')}
Example-2:

Step1: S0= {'0', '0', '0'}

G0 = {'?', '?', '?' }

Step2: Training Instance d: ('big', 'red', 'circle', 'N') -ve instance

S after removing consistent hypothesis with d
S1={('0','0', '0')}

Consider g: ('?', '?', '?')

g after min specialization:
Replace each ? with opposite pair in –ve instance. (First ? can be either small or big
but we already have big in our –ve example so replace with Small, next ? is either red or blue and
we have red already so replace with blue and third ? can be either triangle or circle and we have
circle already so replace with triangle)

S1: {('0', '0', '0')}

G1: {('?', '?', 'triangle'), ('small', '?', '?'), ('?', 'blue', '?')}

Next Instance d: {small, red, triangle} –ve instance

S2={0,0,0}
G: {('?', '?', 'triangle'), ('small', '?', '?'), ('?', 'blue', '?')}

First two pairs are matching with my –ve instance (i.e. triangle and small) which should be opposite
so try to make each ? in the first two pairs with specific ones. {?,blue,?} is opposite to –ve instance d so
keep it as it is.
Replace each pair ? with the opposite pair in the negative instance. (In the first pair ? should be replaced by
{big,?,triangle} bcz small is there in my –ve instance and second ? can be replaced by {?,blue,triangle}
Second pair second ? can be replaced by {small,blue,?} and third ? can be replaced by {small,?,circle}

My final G will be as below:

G[2]: {{big,?, triangle} {?,blue, triangle}{small,blue,?} {small,?,circle} ('?', 'blue', '?')

Compare G[2] with d and if it is not consistent then remove that pair.
{big,?,triangle} {small, ?, circle} ,– consistant bcz it is –ve and my instance also negative so consider this one.
{?,blue,triangle} and {small,blue,?} - These two are specific to {?,blue,?} so ignore it.
{?,blue,?} – This is –ve as opposite to red is blue but my instance also –ve so consistant.

My final generic hypothesis after removing less consistant ones are :

G : {{big,?,triangle}{small,?,circle}{?,blue,?}

Next Instance d: {small, red, circle} +ve instance

G after removing inconsistent hypothesis : {small,?,circle}

bcz {big,?,triangle} and {?,blue,?} are not consistant with d so ignore it.
S[3] = {small,red,circle}
G[3] = {small,?,circle}

Next Instance d: {big, blue, circle} -ve instance

S after removing inconsistent hypothesis : {small,red,circle}
G: {small,?,circle} which is negative and my d also –ve so consistant so keep as it is.
G[4] : {small,?,circle}

Next Instance d: {small, blue, circle} +ve instance

G after removing inconsistent hypothesis : {small,?,circle}
S[5] = {small,?,circle}
G[5] = {small,?,circle}

Example-3 :

Size Color Shape Label

small blue circle Y
big red circle N
small red triangle N
small red circle Y
big blue circle N

1. S0 ={0,0,0}
G0 ={?,?,?}
 2. Consider the first instance d:{small,blue,circle} +ve
S1={small,blue,circle}
G1={?,?,?}
Consider the second instance d:{big,red,circle} –ve
S2={small,blue,circle}
G : Each ? can be replaced by specific pairs
G2: {small,?,?}{?,blue,?}{?,?,triangle}
Try to make G with consistant pairs there by removing less consistant ones.Here first two
pairs are consistant with specific hypothesis but third pair is not consistant so ignore it.
G2={small,?,?}{?,blue,?}
Consider the third instance d:{small,red,triangle} –ve
S3={small,blue,circle}
G after specialization : Make second and third ? in the first pair specific and second pair is
consistant with d so keep as it is.
{small,blue,?}{small,?,circle}{?,blue,?}
Try to make G with consistant pairs there by removing less consistant ones.
Among first two pairs {small,blue,?} is less consistant and matches with {small,?,circle} so
ignore it.
G3={small,?,circle}{?,blue,?}
Consider the fourth instance d:{small,red,circle} +ve
S4 = {small,?,circle}
G after specialization : G4 ={small,?,circle} because second pair is less consistant so
ignore.
Consider the fifth instance d:{big,blue,circle} –ve
S5 = {small,?,circle}
G5={small,?,circle}

Example-4 :

Origin Manufacturer Color Decade Type Target Value

Japan Honda Blue 1980 Economy Y
Japan Toyota Green 1970 Sports N
Japan Toyota Blue 1990 Economy Y
USA Chrysler Red 1980 Economy N
Japan Honda White 1980 Economy Y
Japan Toyota Green 1980 Economy Y
Japan Honda Red 1990 Economy N

1. S0={0,0,0,0,0}
G0={?,?,?,?,?}

2. First example d: {Japan,Honda,Blue,1980,Economy} +ve

S1= {Japan,Honda,Blue,1980,Economy}
G1={?,?,?,?,?}

Second example d: {Japan,Toyota,Green,1970,Sports} –ve

S2={Japan,Honda,Blue,1980,Economy}
Try to make each ? with different possible pairs
G2 :
{USA,?,?,?,?}{?,Honda,?,?,?}{?,Chrysler,?,?,?}{?,?,Blue,?,?}{?,?,Red,?,?}{?,?,White,?,?}
{?,?,?,1980,?}{?,?,?,1990,?}{?,?,?,?,Economy}
 Try to remove inconsistent pairs and keep only consistant ones.Pairs 2,4,7 and 9 are
consistant with specific hypothesis S so keep only this and remaining pairs ignore it.
G2 : {?,Honda,?,?,?}{?,?,Blue,?,?}{?,?,?,1980,?}{?,?,?,?,Economy}

Third example d: {Japan,Toyota,Blue,1990,Economy} +ve

S3 ={Japan,?,Blue,?,Economy}
Make G3 more consistant pairs by removing less consistant pairs with Specific
hypothesis.
G3 ={?,?,blue,?,?}{?,?,?,?,Economy} because first and 3rd pair is not consistant with
specific hypothesis.

Fourth example d: {USA,Chrysler,Red,1980,Economy} -ve

S4 ={Japan,?,blue,?,Economy}
For every pair and ? in G make it specific
G4= {Japan,?,?,?,Economy} {?,Toyota,?,?,Economy} {?,Honda,?,?,Economy}
{?,?,Blue,?,Economy}{?,?,Green,?,Economy}{?,?,White,?,Economy}
{?,?,?,1970,Economy}{?,?,?,1990,Economy}
G after removing less general hypothesis : {Japan,?,?,?,Economy}{?,?,blue,?,?}
{?,?,blue,?,?} – This is consistant with S so keep as it is..
G4 = {Japan,?,?,?,Economy}{?,?,blue,?,?}

Fifth example d: {Japan,Honda,White,1980,Economy} +ve

S5 = {Japan,?,?,?,Economy}
Among the two G pairs try to keep consistant one.
G5={Japan,?,?,?,Economy}

Sixth example d: {Japan, Honda, Red, 1990, Economy} -ve

S6={Japan,?,?,?,Economy}
Here my S6 is matching with d hence S6 is +ve where as d is –ve there by in-consistant so
make S6={ } and G6={}