Learning agents

Performance standard

Critic Sensors
Learning from Observations
feedback

Environment
changes
Chapter 18, Sections 1–3 Learning Performance
element element
knowledge
learning
goals
experiments
Problem
generator

Agent Effectors

Chapter 18, Sections 1–3 1 Chapter 18, Sections 1–3 4

Outline Learning element

♦ Learning agents Design of learning element is dictated by
♦ what type of performance element is used
♦ Inductive learning
♦ which functional component is to be learned
♦ Decision tree learning ♦ how that functional compoent is represented
♦ what kind of feedback is available
♦ Measuring learning performance Example scenarios:
Performance element Component Representation Feedback

Alpha−beta search Eval. fn. Weighted linear function Win/loss

Logical agent Transition model Successor−state axioms Outcome

Utility−based agent Transition model Dynamic Bayes net Outcome

Simple reflex agent Percept−action fn Neural net Correct action

Supervised learning: correct answers for each instance

Reinforcement learning: occasional rewards

Chapter 18, Sections 1–3 2 Chapter 18, Sections 1–3 5

Learning Inductive learning (a.k.a. Science)

Learning is essential for unknown environments, Simplest form: learn a function from examples (tabula rasa)
i.e., when designer lacks omniscience
f is the target function
Learning is useful as a system construction method,
i.e., expose the agent to reality rather than trying to write it down O O X
An example is a pair x, f (x), e.g., X , +1
Learning modifies the agent’s decision mechanisms to improve performance X
Problem: find a(n) hypothesis h
such that h ≈ f
given a training set of examples
(This is a highly simplified model of real learning:
– Ignores prior knowledge
– Assumes a deterministic, observable “environment”
– Assumes examples are given
– Assumes that the agent wants to learn f —why?)

Chapter 18, Sections 1–3 3 Chapter 18, Sections 1–3 6

 Inductive learning method Inductive learning method
Construct/adjust h to agree with f on training set Construct/adjust h to agree with f on training set
(h is consistent if it agrees with f on all examples) (h is consistent if it agrees with f on all examples)
E.g., curve fitting: E.g., curve fitting:
f(x) f(x)

x x

Chapter 18, Sections 1–3 7 Chapter 18, Sections 1–3 10

Inductive learning method Inductive learning method

Construct/adjust h to agree with f on training set Construct/adjust h to agree with f on training set
(h is consistent if it agrees with f on all examples) (h is consistent if it agrees with f on all examples)
E.g., curve fitting: E.g., curve fitting:
f(x) f(x)

x x

Chapter 18, Sections 1–3 8 Chapter 18, Sections 1–3 11

Inductive learning method Inductive learning method

Construct/adjust h to agree with f on training set Construct/adjust h to agree with f on training set
(h is consistent if it agrees with f on all examples) (h is consistent if it agrees with f on all examples)
E.g., curve fitting: E.g., curve fitting:
f(x) f(x)

x x

Ockham’s razor: maximize a combination of consistency and simplicity

Chapter 18, Sections 1–3 9 Chapter 18, Sections 1–3 12
 Attribute-based representations Hypothesis spaces
Examples described by attribute values (Boolean, discrete, continuous, etc.) How many distinct decision trees with n Boolean attributes??
E.g., situations where I will/won’t wait for a table:

Example Attributes Target

Alt Bar F ri Hun P at P rice Rain Res T ype Est WillWait
X1 T F F T Some $$$ F T French 0–10 T
X2 T F F T Full $ F F Thai 30–60 F
X3 F T F F Some $ F F Burger 0–10 T
X4 T F T T Full $ F F Thai 10–30 T
X5 T F T F Full $$$ F T French >60 F
X6 F T F T Some $$ T T Italian 0–10 T
X7 F T F F None $ T F Burger 0–10 F
X8 F F F T Some $$ T T Thai 0–10 T
X9 F T T F Full $ T F Burger >60 F
X10 T T T T Full $$$ F T Italian 10–30 F
X11 F F F F None $ F F Thai 0–10 F
X12 T T T T Full $ F F Burger 30–60 T

Classification of examples is positive (T) or negative (F)

Chapter 18, Sections 1–3 13 Chapter 18, Sections 1–3 16

Decision trees Hypothesis spaces

One possible representation for hypotheses How many distinct decision trees with n Boolean attributes??
E.g., here is the “true” tree for deciding whether to wait:
= number of Boolean functions
Patrons?

None Some Full

F T WaitEstimate?

>60 30−60 10−30 0−10

F Alternate? Hungry? T
No Yes No Yes

Reservation? Fri/Sat? T Alternate?

No Yes No Yes No Yes

Bar? T F T T Raining?
No Yes No Yes

F T F T

Chapter 18, Sections 1–3 14 Chapter 18, Sections 1–3 17

Expressiveness Hypothesis spaces

Decision trees can express any function of the input attributes. How many distinct decision trees with n Boolean attributes??
E.g., for Boolean functions, truth table row → path to leaf:
= number of Boolean functions
A B A xor B
A = number of distinct truth tables with 2n rows
F T
F F F
B B
F T T
F T F T
T F T
T T F F T T F

Trivially, there is a consistent decision tree for any training set

w/ one path to leaf for each example (unless f nondeterministic in x)
but it probably won’t generalize to new examples
Prefer to find more compact decision trees

Chapter 18, Sections 1–3 15 Chapter 18, Sections 1–3 18

 Hypothesis spaces Hypothesis spaces
How many distinct decision trees with n Boolean attributes?? How many distinct decision trees with n Boolean attributes??
= number of Boolean functions = number of Boolean functions
n n
= number of distinct truth tables with 2n rows = 22 = number of distinct truth tables with 2n rows = 22
E.g., with 6 Boolean attributes, there are 18,446,744,073,709,551,616 trees
How many purely conjunctive hypotheses (e.g., Hungry ∧ ¬Rain)??
Each attribute can be in (positive), in (negative), or out
⇒ 3n distinct conjunctive hypotheses
More expressive hypothesis space
– increases chance that target function can be expressed
– increases number of hypotheses consistent w/ training set
⇒ may get worse predictions

Chapter 18, Sections 1–3 19 Chapter 18, Sections 1–3 22

Hypothesis spaces Decision tree learning

How many distinct decision trees with n Boolean attributes?? Aim: find a small tree consistent with the training examples
= number of Boolean functions Idea: (recursively) choose “most significant” attribute as root of (sub)tree
n
= number of distinct truth tables with 2n rows = 22
function DTL(examples, attributes, default) returns a decision tree
E.g., with 6 Boolean attributes, there are 18,446,744,073,709,551,616 trees
if examples is empty then return default
else if all examples have the same classification then return the classification
else if attributes is empty then return Mode(examples)
else
best ← Choose-Attribute(attributes, examples)
tree ← a new decision tree with root test best
for each value vi of best do
examplesi ← {elements of examples with best = vi }
subtree ← DTL(examplesi, attributes − best, Mode(examples))
add a branch to tree with label vi and subtree subtree
return tree

Chapter 18, Sections 1–3 20 Chapter 18, Sections 1–3 23

Hypothesis spaces Choosing an attribute

How many distinct decision trees with n Boolean attributes?? Idea: a good attribute splits the examples into subsets that are (ideally) “all
positive” or “all negative”
= number of Boolean functions
n
= number of distinct truth tables with 2n rows = 22
E.g., with 6 Boolean attributes, there are 18,446,744,073,709,551,616 trees
Patrons? Type?
How many purely conjunctive hypotheses (e.g., Hungry ∧ ¬Rain)??
None Some Full French Italian Thai Burger

P atrons? is a better choice—gives information about the classification

Chapter 18, Sections 1–3 21 Chapter 18, Sections 1–3 24

 Information Performance measurement
Information answers questions How do we know that h ≈ f ? (Hume’s Problem of Induction)
The more clueless I am about the answer initially, the more information is 1) Use theorems of computational/statistical learning theory
contained in the answer
2) Try h on a new test set of examples
Scale: 1 bit = answer to Boolean question with prior h0.5, 0.5i (use same distribution over example space as training set)
Information in an answer when prior is hP1, . . . , Pni is Learning curve = % correct on test set as a function of training set size
1
n
H(hP1, . . . , Pni) = Σ − Pi log2 Pi

% correct on test set

i=1 0.9
(also called entropy of the prior) 0.8
0.7
0.6
0.5
0.4
0 10 20 30 40 50 60 70 80 90 100
Training set size
Chapter 18, Sections 1–3 25 Chapter 18, Sections 1–3 28

Information contd. Performance measurement contd.

Suppose we have p positive and n negative examples at the root Learning curve depends on
⇒ H(hp/(p+n), n/(p+n)i) bits needed to classify a new example – realizable (can express target function) vs. non-realizable
E.g., for 12 restaurant examples, p = n = 6 so we need 1 bit non-realizability can be due to missing attributes
or restricted hypothesis class (e.g., thresholded linear function)
An attribute splits the examples E into subsets Ei, each of which (we hope) – redundant expressiveness (e.g., loads of irrelevant attributes)
needs less information to complete the classification
% correct
Let Ei have pi positive and ni negative examples
1 realizable
⇒ H(hpi/(pi +ni), ni/(pi +ni)i) bits needed to classify a new example
⇒ expected number of bits per example over all branches is
redundant
pi + ni
Σi H(hpi/(pi + ni), ni/(pi + ni)i) nonrealizable
p+n
For P atrons?, this is 0.459 bits, for T ype this is (still) 1 bit
⇒ choose the attribute that minimizes the remaining information needed
# of examples

Chapter 18, Sections 1–3 26 Chapter 18, Sections 1–3 29

Example contd. Summary

Decision tree learned from the 12 examples: Learning needed for unknown environments, lazy designers

Patrons?
Learning agent = performance element + learning element
None Some Full Learning method depends on type of performance element, available
F T Hungry?
feedback, type of component to be improved, and its representation
Yes No
For supervised learning, the aim is to find a simple hypothesis
Type? F that is approximately consistent with training examples
French Italian Thai Burger Decision tree learning using information gain
T F Fri/Sat? T
No Yes Learning performance = prediction accuracy measured on test set
F T

Substantially simpler than “true” tree—a more complex hypothesis isn’t jus-
tified by small amount of data

Chapter 18, Sections 1–3 27 Chapter 18, Sections 1–3 30