Introduction to Machine Learning (67577)

Lecture 2
Shai Shalev-Shwartz
School of CS and Engineering,
The Hebrew University of Jerusalem

PAC learning

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

1 / 45

Outline
1

The PAC Learning Framework

No Free Lunch and Prior Knowledge

PAC Learning of Finite Hypothesis Classes

The Fundamental Theorem of Learning Theory

The VC dimension

Solving ERM for Halfspaces

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

2 / 45

Recall: The Game Board

Example

Domain set, X : This is the set of

objects that we may wish to label.
Label set, Y: The set of possible labels.
A prediction rule, h : X Y: used to
label future examples. This function is
called a predictor, a hypothesis, or a
classifier.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

X = R2 representing color
and shape of papayas.
Y = {1} representing
tasty or non-tasty.
h(x) = 1 if x is within the
inner rectangle

PAC learning

3 / 45

Batch Learning
The learners input:
Training data, S = ((x1 , y1 ) . . . (xm , ym )) (X Y)m

The learners output:

A prediction rule, h : X Y

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

4 / 45

Batch Learning
The learners input:
Training data, S = ((x1 , y1 ) . . . (xm , ym )) (X Y)m

The learners output:

A prediction rule, h : X Y

What should be the goal of the learner?

Intuitively, h should be correct on future examples

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

4 / 45

Correct on future examples

Let f be the correct classifier, then we should find h s.t. h f

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

5 / 45

Correct on future examples

Let f be the correct classifier, then we should find h s.t. h f
One way: define the error of h w.r.t. f to be
LD,f (h) = P [h(x) 6= f (x)]
xD

where D is some (unknown) probability measure over X

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

5 / 45

Correct on future examples

Let f be the correct classifier, then we should find h s.t. h f
One way: define the error of h w.r.t. f to be
LD,f (h) = P [h(x) 6= f (x)]
xD

where D is some (unknown) probability measure over X

More formally, D is a distribution over X , that is, for a given A X ,
the value of D(A) is the probability to see some x A. Then,
def

LD,f (h) =

def

P [h(x) 6= f (x)] = D ({x X : h(x) 6= f (x)}) .

xD

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

5 / 45

Correct on future examples

Let f be the correct classifier, then we should find h s.t. h f
One way: define the error of h w.r.t. f to be
LD,f (h) = P [h(x) 6= f (x)]
xD

where D is some (unknown) probability measure over X

More formally, D is a distribution over X , that is, for a given A X ,
the value of D(A) is the probability to see some x A. Then,
def

LD,f (h) =

def

P [h(x) 6= f (x)] = D ({x X : h(x) 6= f (x)}) .

xD

Can we find h s.t. LD,f (h) is small ?

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

5 / 45

Data-generation Model
We must assume some relation between the training data and D, f
Simple data generation model:
Independently Identically Distributed (i.i.d.): Each xi is sampled
independently according to D
Realizability: For every i [m], yi = f (xi )

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

6 / 45

Can only be Approximately correct

Claim: We cant hope to find h s.t. L(D,f ) (h) = 0

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

7 / 45

Can only be Approximately correct

Claim: We cant hope to find h s.t. L(D,f ) (h) = 0
Proof: for every  (0, 1) take X = {x1 , x2 } and D({x1 }) = 1 ,
D({x2 }) = 

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

7 / 45

Can only be Approximately correct

Claim: We cant hope to find h s.t. L(D,f ) (h) = 0
Proof: for every  (0, 1) take X = {x1 , x2 } and D({x1 }) = 1 ,
D({x2 }) = 
The probability not to see x2 at all among m i.i.d. examples is
(1 )m em

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

7 / 45

Can only be Approximately correct

Claim: We cant hope to find h s.t. L(D,f ) (h) = 0
Proof: for every  (0, 1) take X = {x1 , x2 } and D({x1 }) = 1 ,
D({x2 }) = 
The probability not to see x2 at all among m i.i.d. examples is
(1 )m em
So, if   1/m were likely not to see x2 at all, but then we cant
know its label

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

7 / 45

Can only be Approximately correct

Claim: We cant hope to find h s.t. L(D,f ) (h) = 0
Proof: for every  (0, 1) take X = {x1 , x2 } and D({x1 }) = 1 ,
D({x2 }) = 
The probability not to see x2 at all among m i.i.d. examples is
(1 )m em
So, if   1/m were likely not to see x2 at all, but then we cant
know its label
Relaxation: Wed be happy with L(D,f ) (h) , where  is
user-specified

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

7 / 45

Can only be Probably correct

Recall that the input to the learner is randomly generated

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

8 / 45

Can only be Probably correct

Recall that the input to the learner is randomly generated
Theres always a (very small) chance to see the same example again
and again

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

8 / 45

Can only be Probably correct

Recall that the input to the learner is randomly generated
Theres always a (very small) chance to see the same example again
and again
Claim: No algorithm can guarantee L(D,f ) (h)  for sure

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

8 / 45

Can only be Probably correct

Recall that the input to the learner is randomly generated
Theres always a (very small) chance to see the same example again
and again
Claim: No algorithm can guarantee L(D,f ) (h)  for sure
Relaxation: Wed allow the algorithm to fail with probability , where
(0, 1) is user-specified
Here, the probability is over the random choice of examples

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

8 / 45

Probably Approximately Correct (PAC) learning

The learner doesnt know D and f .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

9 / 45

Probably Approximately Correct (PAC) learning

The learner doesnt know D and f .
The learner receives accuracy parameter  and confidence parameter

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

9 / 45

Probably Approximately Correct (PAC) learning

The learner doesnt know D and f .
The learner receives accuracy parameter  and confidence parameter
The learner can ask for training data, S, containing m(, ) examples
(that is, the number of examples can depend on the value of  and ,
but it cant depend on D or f )

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

9 / 45

Probably Approximately Correct (PAC) learning

The learner doesnt know D and f .
The learner receives accuracy parameter  and confidence parameter
The learner can ask for training data, S, containing m(, ) examples
(that is, the number of examples can depend on the value of  and ,
but it cant depend on D or f )
Learner should output a hypothesis h s.t. with probability of at least
1 it holds that LD,f (h) 

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

9 / 45

Probably Approximately Correct (PAC) learning

The learner doesnt know D and f .
The learner receives accuracy parameter  and confidence parameter
The learner can ask for training data, S, containing m(, ) examples
(that is, the number of examples can depend on the value of  and ,
but it cant depend on D or f )
Learner should output a hypothesis h s.t. with probability of at least
1 it holds that LD,f (h) 
That is, the learner should be Probably (with probability at least
1 ) Approximately (up to accuracy ) Correct

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

9 / 45

No Free Lunch
Suppose that |X | =
For any finite C X take D to be uniform distribution over C
If number of training examples is m |C|/2 the learner has no
knowledge on at least half the elements in C
Formalizing the above, it can be shown that:

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

10 / 45

No Free Lunch
Suppose that |X | =
For any finite C X take D to be uniform distribution over C
If number of training examples is m |C|/2 the learner has no
knowledge on at least half the elements in C
Formalizing the above, it can be shown that:

Theorem (No Free Lunch)

Fix (0, 1),  < 1/2. For every learner A and training set size m, there
exists D, f such that with probability of at least over the generation of a
training data, S, of m examples, it holds that LD,f (A(S)) .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

10 / 45

No Free Lunch
Suppose that |X | =
For any finite C X take D to be uniform distribution over C
If number of training examples is m |C|/2 the learner has no
knowledge on at least half the elements in C
Formalizing the above, it can be shown that:

Theorem (No Free Lunch)

Fix (0, 1),  < 1/2. For every learner A and training set size m, there
exists D, f such that with probability of at least over the generation of a
training data, S, of m examples, it holds that LD,f (A(S)) .
Remark: LD,f (random guess) = 1/2, so the theorem states that you cant
be better than a random guess

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

10 / 45

Prior Knowledge
Give more knowledge to the learner: the target f comes from some
hypothesis class, H Y X
The learner knows H
Is it possible to PAC learn now ?
Of course, the answer depends on H since the No Free Lunch
theorem tells us that for H = Y X the problem is not solvable ...

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

11 / 45

Outline
1

The PAC Learning Framework

No Free Lunch and Prior Knowledge

PAC Learning of Finite Hypothesis Classes

The Fundamental Theorem of Learning Theory

The VC dimension

Solving ERM for Halfspaces

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

12 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Use the Consistent learning rule:

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Use the Consistent learning rule:

Input: H and S = (x1 , y1 ), . . . , (xm , ym )

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Use the Consistent learning rule:

Input: H and S = (x1 , y1 ), . . . , (xm , ym )
Output: any h H s.t. i, yi = h(xi )

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Use the Consistent learning rule:

Input: H and S = (x1 , y1 ), . . . , (xm , ym )
Output: any h H s.t. i, yi = h(xi )

This is also called Empirical Risk Minimization (ERM)

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Use the Consistent learning rule:

Input: H and S = (x1 , y1 ), . . . , (xm , ym )
Output: any h H s.t. i, yi = h(xi )

This is also called Empirical Risk Minimization (ERM)

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Assume that H is a finite hypothesis class
E.g.: H is all the functions from X to Y that can be implemented
using a Python program of length at most b

Use the Consistent learning rule:

Input: H and S = (x1 , y1 ), . . . , (xm , ym )
Output: any h H s.t. i, yi = h(xi )

This is also called Empirical Risk Minimization (ERM)

ERMH (S)
Input: training set S = (x1 , y1 ), . . . , (xm , ym )
Define the empirical risk: LS (h) =

1
m |{i

: h(xi ) 6= yi }|

Output: any h H that minimizes LS (h)

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

13 / 45

Learning Finite Classes

Theorem
then for every D, f , with probability of at least
Fix , . If m log(|H|/)

1 (over the choice of S of size m), LD,f (ERMH (S)) .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

14 / 45

Proof
Let S|x = (x1 , . . . , xm ) be the instances of the training set

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

15 / 45

Proof
Let S|x = (x1 , . . . , xm ) be the instances of the training set
We would like to prove:
Dm ({S|x : L(D,f ) (ERMH (S)) > })

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

15 / 45

Proof
Let S|x = (x1 , . . . , xm ) be the instances of the training set
We would like to prove:
Dm ({S|x : L(D,f ) (ERMH (S)) > })
Let HB be the set of bad hypotheses,
HB = {h H : L(D,f ) (h) > }

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

15 / 45

Proof
Let S|x = (x1 , . . . , xm ) be the instances of the training set
We would like to prove:
Dm ({S|x : L(D,f ) (ERMH (S)) > })
Let HB be the set of bad hypotheses,
HB = {h H : L(D,f ) (h) > }
Let M be the set of misleading samples,
M = {S|x : h HB , LS (h) = 0}

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

15 / 45

Proof
Let S|x = (x1 , . . . , xm ) be the instances of the training set
We would like to prove:
Dm ({S|x : L(D,f ) (ERMH (S)) > })
Let HB be the set of bad hypotheses,
HB = {h H : L(D,f ) (h) > }
Let M be the set of misleading samples,
M = {S|x : h HB , LS (h) = 0}
Observe:
{S|x : L(D,f ) (ERMH (S)) > } M =

{S|x : LS (h) = 0}

hHB

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

15 / 45

Proof (Cont.)
Lemma (Union bound)
For any two sets A, B and a distribution D we have
D(A B) D(A) + D(B) .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

16 / 45

Proof (Cont.)
Lemma (Union bound)
For any two sets A, B and a distribution D we have
D(A B) D(A) + D(B) .
We have shown:
S
{S|x : L(D,f ) (ERMH (S)) > } hHB {S|x : LS (h) = 0}

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

16 / 45

Proof (Cont.)
Lemma (Union bound)
For any two sets A, B and a distribution D we have
D(A B) D(A) + D(B) .
We have shown:
S
{S|x : L(D,f ) (ERMH (S)) > } hHB {S|x : LS (h) = 0}
Therefore, using the union bound
Dm ({S|x :L(D,f ) (ERMH (S)) > })
X

Dm ({S|x : LS (h) = 0})

hHB

|HB | max Dm ({S|x : LS (h) = 0})

hHB

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

16 / 45

Proof (Cont.)
Observe:
Dm ({S|x : LS (h) = 0}) = (1 LD,f (h))m

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

17 / 45

Proof (Cont.)
Observe:
Dm ({S|x : LS (h) = 0}) = (1 LD,f (h))m
If h HB then LD,f (h) >  and therefore
Dm ({S|x : LS (h) = 0}) < (1 )m

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

17 / 45

Proof (Cont.)
Observe:
Dm ({S|x : LS (h) = 0}) = (1 LD,f (h))m
If h HB then LD,f (h) >  and therefore
Dm ({S|x : LS (h) = 0}) < (1 )m
We have shown:
Dm ({S|x : L(D,f ) (ERMH (S)) > }) < |HB | (1 )m

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

17 / 45

Proof (Cont.)
Observe:
Dm ({S|x : LS (h) = 0}) = (1 LD,f (h))m
If h HB then LD,f (h) >  and therefore
Dm ({S|x : LS (h) = 0}) < (1 )m
We have shown:
Dm ({S|x : L(D,f ) (ERMH (S)) > }) < |HB | (1 )m
Finally, using 1  e and |HB | |H| we conclude:
Dm ({S|x : L(D,f ) (ERMH (S)) > }) < |H| e m

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

17 / 45

Proof (Cont.)
Observe:
Dm ({S|x : LS (h) = 0}) = (1 LD,f (h))m
If h HB then LD,f (h) >  and therefore
Dm ({S|x : LS (h) = 0}) < (1 )m
We have shown:
Dm ({S|x : L(D,f ) (ERMH (S)) > }) < |HB | (1 )m
Finally, using 1  e and |HB | |H| we conclude:
Dm ({S|x : L(D,f ) (ERMH (S)) > }) < |H| e m
The right-hand side would be if m
Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

log(|H|/)
.

PAC learning

17 / 45

Illustrating the use of the union bound

Each point is a possible sample S|x . Each colored oval represents

misleading samples for some h HB . The probability mass of each
such oval is at most (1 )m . But, the algorithm might err if it
samples S|x from any of these ovals.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

18 / 45

Outline
1

The PAC Learning Framework

No Free Lunch and Prior Knowledge

PAC Learning of Finite Hypothesis Classes

The Fundamental Theorem of Learning Theory

The VC dimension

Solving ERM for Halfspaces

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

19 / 45

PAC learning
Definition (PAC learnability)
A hypothesis class H is PAC learnable if there exists a function
mH : (0, 1)2 N and a learning algorithm with the following property:
for every , (0, 1)
for every distribution D over X , and for every labeling function
f : X {0, 1}
when running the learning algorithm on m mH (, ) i.i.d. examples
generated by D and labeled by f , the algorithm returns a hypothesis h
such that, with probability of at least 1 (over the choice of the
examples), L(D,f ) (h) .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

20 / 45

PAC learning
Definition (PAC learnability)
A hypothesis class H is PAC learnable if there exists a function
mH : (0, 1)2 N and a learning algorithm with the following property:
for every , (0, 1)
for every distribution D over X , and for every labeling function
f : X {0, 1}
when running the learning algorithm on m mH (, ) i.i.d. examples
generated by D and labeled by f , the algorithm returns a hypothesis h
such that, with probability of at least 1 (over the choice of the
examples), L(D,f ) (h) .

mH is called the sample complexity of learning H

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

20 / 45

PAC learning
Leslie Valiant, Turing award 2010
For transformative contributions to the theory of computation,
including the theory of probably approximately correct (PAC)
learning, the complexity of enumeration and of algebraic
computation, and the theory of parallel and distributed
computing.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

21 / 45

What is learnable and how to learn?

We have shown:

Corollary
Let H be a finite hypothesis class.
H is PAC learnable with sample complexity mH (, )

log(|H|/)


This sample complexity is obtained by using the ERMH learning rule

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

22 / 45

What is learnable and how to learn?

We have shown:

Corollary
Let H be a finite hypothesis class.
H is PAC learnable with sample complexity mH (, )

log(|H|/)


This sample complexity is obtained by using the ERMH learning rule

What about infinite hypothesis classes?
What is the sample complexity of a given class?
Is there a generic learning algorithm that achieves the optimal sample
complexity ?

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

22 / 45

What is learnable and how to learn?

The fundamental theorem of statistical learning:
The sample complexity is characterized by the VC dimension
The ERM learning rule is a generic (near) optimal learner

Chervonenkis

Shai Shalev-Shwartz (Hebrew U)

Vapnik

IML Lecture 2

PAC learning

23 / 45

Outline
1

The PAC Learning Framework

No Free Lunch and Prior Knowledge

PAC Learning of Finite Hypothesis Classes

The Fundamental Theorem of Learning Theory

The VC dimension

Solving ERM for Halfspaces

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

24 / 45

The VC dimension Motivation

if someone can explain every phenomena, her explanations are worthless.

Example: http://www.youtube.com/watch?v=p_MzP2MZaOo
Pay attention to the retrospect explanations at 5:00

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

25 / 45

The VC dimension Motivation

Suppose we got a training set S = (x1 , y1 ), . . . , (xm , ym )

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

26 / 45

The VC dimension Motivation

Suppose we got a training set S = (x1 , y1 ), . . . , (xm , ym )
We try to explain the labels using a hypothesis from H

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

26 / 45

The VC dimension Motivation

Suppose we got a training set S = (x1 , y1 ), . . . , (xm , ym )
We try to explain the labels using a hypothesis from H
Then, ooops, the labels we received are incorrect and we get the same
0 )
instances with different labels, S 0 = (x1 , y10 ), . . . , (xm , ym

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

26 / 45

The VC dimension Motivation

Suppose we got a training set S = (x1 , y1 ), . . . , (xm , ym )
We try to explain the labels using a hypothesis from H
Then, ooops, the labels we received are incorrect and we get the same
0 )
instances with different labels, S 0 = (x1 , y10 ), . . . , (xm , ym
We again try to explain the labels using a hypothesis from H

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

26 / 45

The VC dimension Motivation

Suppose we got a training set S = (x1 , y1 ), . . . , (xm , ym )
We try to explain the labels using a hypothesis from H
Then, ooops, the labels we received are incorrect and we get the same
0 )
instances with different labels, S 0 = (x1 , y10 ), . . . , (xm , ym
We again try to explain the labels using a hypothesis from H
If this works for us, no matter what the labels are, then something is
fishy ...

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

26 / 45

The VC dimension Motivation

Suppose we got a training set S = (x1 , y1 ), . . . , (xm , ym )
We try to explain the labels using a hypothesis from H
Then, ooops, the labels we received are incorrect and we get the same
0 )
instances with different labels, S 0 = (x1 , y10 ), . . . , (xm , ym
We again try to explain the labels using a hypothesis from H
If this works for us, no matter what the labels are, then something is
fishy ...
Formally, if H allows all functions over some set C of size m, then
based on the No Free Lunch, we cant learn from, say, m/2 examples

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

26 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X
Let HC be the restriction of H to C, namely, HC = {hC : h H}
where hC : C {0, 1} is s.t. hC (xi ) = h(xi ) for every xi C

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X
Let HC be the restriction of H to C, namely, HC = {hC : h H}
where hC : C {0, 1} is s.t. hC (xi ) = h(xi ) for every xi C
Observe: we can represent each hC as the vector
(h(x1 ), . . . , h(x|C| )) {1}|C|

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X
Let HC be the restriction of H to C, namely, HC = {hC : h H}
where hC : C {0, 1} is s.t. hC (xi ) = h(xi ) for every xi C
Observe: we can represent each hC as the vector
(h(x1 ), . . . , h(x|C| )) {1}|C|
Therefore: |HC | 2|C|

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X
Let HC be the restriction of H to C, namely, HC = {hC : h H}
where hC : C {0, 1} is s.t. hC (xi ) = h(xi ) for every xi C
Observe: we can represent each hC as the vector
(h(x1 ), . . . , h(x|C| )) {1}|C|
Therefore: |HC | 2|C|
We say that H shatters C if |HC | = 2|C|

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X
Let HC be the restriction of H to C, namely, HC = {hC : h H}
where hC : C {0, 1} is s.t. hC (xi ) = h(xi ) for every xi C
Observe: we can represent each hC as the vector
(h(x1 ), . . . , h(x|C| )) {1}|C|
Therefore: |HC | 2|C|
We say that H shatters C if |HC | = 2|C|
VCdim(H) = sup{|C| : H shatters C}

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

The VC dimension Formal Definition

Let C = {x1 , . . . , x|C| } X
Let HC be the restriction of H to C, namely, HC = {hC : h H}
where hC : C {0, 1} is s.t. hC (xi ) = h(xi ) for every xi C
Observe: we can represent each hC as the vector
(h(x1 ), . . . , h(x|C| )) {1}|C|
Therefore: |HC | 2|C|
We say that H shatters C if |HC | = 2|C|
VCdim(H) = sup{|C| : H shatters C}
That is, the VC dimension is the maximal size of a set C such that H
gives no prior knowledge w.r.t. C

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

27 / 45

VC dimension Examples
To show that VCdim(H) = d we need to show that:
1

There exists a set C of size d which is shattered by H.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

28 / 45

VC dimension Examples
To show that VCdim(H) = d we need to show that:
1

There exists a set C of size d which is shattered by H.

Every set C of size d + 1 is not shattered by H.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

28 / 45

VC dimension Examples
Threshold functions: X = R, H = {x 7 sign(x ) : R}
Show that {0} is shattered

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

29 / 45

VC dimension Examples
Threshold functions: X = R, H = {x 7 sign(x ) : R}
Show that {0} is shattered
Show that any two points cannot be shattered

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

29 / 45

VC dimension Examples
Intervals: X = R, H = {ha,b : a < b R}, where ha,b (x) = 1 iff x [a, b]
Show that {0, 1} is shattered

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

30 / 45

VC dimension Examples
Intervals: X = R, H = {ha,b : a < b R}, where ha,b (x) = 1 iff x [a, b]
Show that {0, 1} is shattered
Show that any three points cannot be shattered

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

30 / 45

VC dimension Examples
Axis aligned rectangles: X = R2 ,
H = {h(a1 ,a2 ,b1 ,b2 ) : a1 < a2 and b1 < b2 }, where h(a1 ,a2 ,b1 ,b2 ) (x1 , x2 ) = 1
iff x1 [a1 , a2 ] and x2 [b1 , b2 ]

Show:
Shattered

Not Shattered
c1
c4

c5

c2

c3

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

31 / 45

VC dimension Examples
Finite classes:
Show that the VC dimension of a finite H is at most log2 (|H|).

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

32 / 45

VC dimension Examples
Finite classes:
Show that the VC dimension of a finite H is at most log2 (|H|).
Show that there can be arbitrary gap between VCdim(H) and
log2 (|H|)

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

32 / 45

VC dimension Examples
Halfspaces: X = Rd , H = {x 7 sign(hw, xi) : w Rd }
Show that {e1 , . . . , ed } is shattered

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

33 / 45

VC dimension Examples
Halfspaces: X = Rd , H = {x 7 sign(hw, xi) : w Rd }
Show that {e1 , . . . , ed } is shattered
Show that any d + 1 points cannot be shattered

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

33 / 45

The Fundamental Theorem of Statistical Learning

Theorem (The Fundamental Theorem of Statistical Learning)
Let H be a hypothesis class of binary classifiers. Then, there are absolute
constants C1 , C2 such that the sample complexity of PAC learning H is
C1

d log(1/) + log(1/)
d + log(1/)
mH (, ) C2



Furthermore, this sample complexity is achieved by the ERM learning rule.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

34 / 45

Proof of the lower bound main ideas

Suppose VCdim(H) = d and let C = {x1 , . . . , xd } be a shattered set

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

35 / 45

Proof of the lower bound main ideas

Suppose VCdim(H) = d and let C = {x1 , . . . , xd } be a shattered set
Consider the distribution D supported on C s.t.
(
1 4
if i = 1
D({xi }) =
4/(d 1) if i > 1

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

35 / 45

Proof of the lower bound main ideas

Suppose VCdim(H) = d and let C = {x1 , . . . , xd } be a shattered set
Consider the distribution D supported on C s.t.
(
1 4
if i = 1
D({xi }) =
4/(d 1) if i > 1
If we see m i.i.d. examples then the expected number of examples
from C \ {x1 } is 4m

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

35 / 45

Proof of the lower bound main ideas

Suppose VCdim(H) = d and let C = {x1 , . . . , xd } be a shattered set
Consider the distribution D supported on C s.t.
(
1 4
if i = 1
D({xi }) =
4/(d 1) if i > 1
If we see m i.i.d. examples then the expected number of examples
from C \ {x1 } is 4m
d1
If m < d1
8 then 4m < 2 and therefore, we have no information
on the labels of at least half the examples in C \ {x1 }

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

35 / 45

Proof of the lower bound main ideas

Suppose VCdim(H) = d and let C = {x1 , . . . , xd } be a shattered set
Consider the distribution D supported on C s.t.
(
1 4
if i = 1
D({xi }) =
4/(d 1) if i > 1
If we see m i.i.d. examples then the expected number of examples
from C \ {x1 } is 4m
d1
If m < d1
8 then 4m < 2 and therefore, we have no information
on the labels of at least half the examples in C \ {x1 }

Best we can do is to guess, but then our error is

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

1
2

2 = 

PAC learning

35 / 45

Proof of the upper bound main ideas

Recall our proof for finite class:

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

36 / 45

Proof of the upper bound main ideas

Recall our proof for finite class:
For a single hypothesis, weve shown that the probability of the event:
LS (h) = 0 given that L(D,f ) >  is at most em

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

36 / 45

Proof of the upper bound main ideas

Recall our proof for finite class:
For a single hypothesis, weve shown that the probability of the event:
LS (h) = 0 given that L(D,f ) >  is at most em
Then we applied the union bound over all bad hypotheses, to obtain
the bound on ERM failure: |H| em

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

36 / 45

Proof of the upper bound main ideas

Recall our proof for finite class:
For a single hypothesis, weve shown that the probability of the event:
LS (h) = 0 given that L(D,f ) >  is at most em
Then we applied the union bound over all bad hypotheses, to obtain
the bound on ERM failure: |H| em

If H is infinite, or very large, the union bound yields a meaningless

bound

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

36 / 45

Proof of the upper bound main ideas

The two samples trick: show that
P [h HB : LS (h) = 0]

SDm

S,T Dm

Shai Shalev-Shwartz (Hebrew U)

[h HB : LS (h) = 0 and LT (h) /2]

IML Lecture 2

PAC learning

37 / 45

Proof of the upper bound main ideas

The two samples trick: show that
P [h HB : LS (h) = 0]

SDm

S,T Dm

[h HB : LS (h) = 0 and LT (h) /2]

Symmetrization: Since S, T are i.i.d., we can think on first sampling

2m examples and then splitting them to S, T at random

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

37 / 45

Proof of the upper bound main ideas

The two samples trick: show that
P [h HB : LS (h) = 0]

SDm

S,T Dm

[h HB : LS (h) = 0 and LT (h) /2]

Symmetrization: Since S, T are i.i.d., we can think on first sampling

2m examples and then splitting them to S, T at random
If we fix h, and S T , the probability to have LS (h) = 0 while
LT (h) /2 is em/4

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

37 / 45

Proof of the upper bound main ideas

The two samples trick: show that
P [h HB : LS (h) = 0]

SDm

S,T Dm

[h HB : LS (h) = 0 and LT (h) /2]

Symmetrization: Since S, T are i.i.d., we can think on first sampling

2m examples and then splitting them to S, T at random
If we fix h, and S T , the probability to have LS (h) = 0 while
LT (h) /2 is em/4
Once we fixed S T , we can take a union bound only over HST !

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

37 / 45

Proof of the upper bound main ideas

Lemma (Sauer-Shelah-Perles2 -Vapnik-Chervonenkis)
Let H be a hypothesis class with VCdim(H) d < . Then, for all
C X s.t. |C| = m > d + 1 we have
|HC |

Shai Shalev-Shwartz (Hebrew U)

 em d
d

IML Lecture 2

PAC learning

38 / 45

Outline
1

The PAC Learning Framework

No Free Lunch and Prior Knowledge

PAC Learning of Finite Hypothesis Classes

The Fundamental Theorem of Learning Theory

The VC dimension

Solving ERM for Halfspaces

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

39 / 45

ERM for halfspaces

Recall:
H = {x 7 sign(hw, xi) : w Rd }

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

40 / 45

ERM for halfspaces

Recall:
H = {x 7 sign(hw, xi) : w Rd }
ERM for Halfspaces:
given S = (x1 , y1 ), . . . , (xm , ym ) find w s.t. for all i,
sign(hw, xi i) = yi .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

40 / 45

ERM for halfspaces

Recall:
H = {x 7 sign(hw, xi) : w Rd }
ERM for Halfspaces:
given S = (x1 , y1 ), . . . , (xm , ym ) find w s.t. for all i,
sign(hw, xi i) = yi .
Cast as a Linear Program:
Find w s.t.
i,

Shai Shalev-Shwartz (Hebrew U)

yi hw, xi i > 0 .

IML Lecture 2

PAC learning

40 / 45

ERM for halfspaces

Recall:
H = {x 7 sign(hw, xi) : w Rd }
ERM for Halfspaces:
given S = (x1 , y1 ), . . . , (xm , ym ) find w s.t. for all i,
sign(hw, xi i) = yi .
Cast as a Linear Program:
Find w s.t.
i,

yi hw, xi i > 0 .

Can solve efficiently using standard methods

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

40 / 45

ERM for halfspaces

Recall:
H = {x 7 sign(hw, xi) : w Rd }
ERM for Halfspaces:
given S = (x1 , y1 ), . . . , (xm , ym ) find w s.t. for all i,
sign(hw, xi i) = yi .
Cast as a Linear Program:
Find w s.t.
i,

yi hw, xi i > 0 .

Can solve efficiently using standard methods

Exercise: show how to solve the above Linear Program using the
Ellipsoid learner from the previous lecture

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

40 / 45

ERM for halfspaces using the Perceptron Algorithm

Perceptron
initialize: w = (0, . . . , 0) Rd
while i s.t. yi hw, xi i 0
w w + yi xi
Dates back at least to Rosenblatt 1958.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

41 / 45

Analysis
Theorem (Agmon54, Novikoff62)
Let (x1 , y1 ), . . . , (xm , ym ) be a sequence of examples such that there
exists w Rd such that for every i, yi hw , xi i 1. Then, the
Perceptron will make at most
kw k2 max kxi k2
i

updates before breaking with an ERM halfspace.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

42 / 45

Analysis
Theorem (Agmon54, Novikoff62)
Let (x1 , y1 ), . . . , (xm , ym ) be a sequence of examples such that there
exists w Rd such that for every i, yi hw , xi i 1. Then, the
Perceptron will make at most
kw k2 max kxi k2
i

updates before breaking with an ERM halfspace.

The condition would always hold if the data is realizable by some
halfspace
However, kw k might be very large
In many practical cases, kw k would not be too large

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

42 / 45

Proof
Let w(t) be the value of w at iteration t

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

hw(t) ,w i
kw(t) k kw k

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

hw(t) ,w i
kw(t) k kw k

By the Cauchy-Schwartz inequality, this is always 1

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

hw(t) ,w i
kw(t) k kw k

By the Cauchy-Schwartz inequality, this is always 1

We will show:

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

hw(t) ,w i
kw(t) k kw k

By the Cauchy-Schwartz inequality, this is always 1

We will show:
1

hw(t+1) , w i t

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

hw(t) ,w i
kw(t) k kw k

By the Cauchy-Schwartz inequality, this is always 1

We will show:
1
2

hw(t+1) , w i t
kw(t+1) k R t

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

hw(t) ,w i
kw(t) k kw k

By the Cauchy-Schwartz inequality, this is always 1

We will show:
1
2

hw(t+1) , w i t
kw(t+1) k R t

This would yield

Shai Shalev-Shwartz (Hebrew U)

t
hw(t) , w i

1
kw(t) k kw k
t kw k

IML Lecture 2

PAC learning

43 / 45

Proof
Let w(t) be the value of w at iteration t
Let (xt , yt ) be the example used to update w at iteration t
Denote R = maxi kxi k
The cosine of the angle between w and w(t) is

hw(t) ,w i
kw(t) k kw k

By the Cauchy-Schwartz inequality, this is always 1

We will show:
1
2

hw(t+1) , w i t
kw(t+1) k R t

This would yield

t
hw(t) , w i

1
kw(t) k kw k
t kw k

Rearranging the above would yield t kw k2 R2 as required.

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

43 / 45

Proof (Cont.)
Showing hw(t+1) , w i t
Initially, hw(1) , w i = 0

Showing kw(t+1) k2 R2 t

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

44 / 45

Proof (Cont.)
Showing hw(t+1) , w i t
Initially, hw(1) , w i = 0
Whenever we update, hw(t) , w i increases by at least 1:
hw(t+1) , w i = hw(t) + yt xt , w i = hw(t) , w i + yt hxt , w i
| {z }
1

Showing kw(t+1) k2 R2 t

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

44 / 45

Proof (Cont.)
Showing hw(t+1) , w i t
Initially, hw(1) , w i = 0
Whenever we update, hw(t) , w i increases by at least 1:
hw(t+1) , w i = hw(t) + yt xt , w i = hw(t) , w i + yt hxt , w i
| {z }
1

Showing kw(t+1) k2 R2 t
Initially, kw(1) k2 = 0

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

44 / 45

Proof (Cont.)
Showing hw(t+1) , w i t
Initially, hw(1) , w i = 0
Whenever we update, hw(t) , w i increases by at least 1:
hw(t+1) , w i = hw(t) + yt xt , w i = hw(t) , w i + yt hxt , w i
| {z }
1

Showing kw(t+1) k2 R2 t
Initially, kw(1) k2 = 0
Whenever we update, kw(t) k2 increases by at most 1:
kw(t+1) k2 = kw(t) + yt xt k2 = kw(t) k2 + 2yt hw(t) , xt i +yt2 kxt k2
{z
}
|
0

(t) 2

kw k + R .

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

44 / 45

Summary
The PAC Learning model
What is PAC learnable?
PAC learning of finite classes using ERM
The VC dimension and the fundmental theorem of learning
Classes of finite VC dimension

How to PAC learn?

Using ERM

Learning halfspaces using: Linear programming, Ellipsoid, Perceptron

Shai Shalev-Shwartz (Hebrew U)

IML Lecture 2

PAC learning

45 / 45