Artificial Intelligence II (CS4442 & CS9542)

Overfitting, Cross-Validation, and Regularization

Boyu Wang
Department of Computer Science
University of Western Ontario
Motivation examples: polynomial regression

I As the degree of the polynomial increases, there is more degrees of

freedom, and the (training) error approaches to zero.

Figure credit: Christopher Bishop

1
Motivation examples: polynomial regression

I Minimizing the training/empirical loss does NOT indicate a good

test/generalization performance.
I Overfitting: Very low training error, very high test error.

Figure credit: Christopher Bishop

2
Overfitting – general phenomenon

I Too simple (e.g., small M) → underfitting

I Too complex (e.g., large M) → overfitting

Figure credit: Ian Goodfellow

3
Overfitting

I Training loss and test loss are different

I Larger the hypothesis class, easier to find a hypothesis that fits
the training data
- but may have large test error (overfitting)
I Prevent overfitting:
- Large data set
- Throw away useless hypothesis class (model selection)
- Control model complexity (regularization)

4
Larger data set

I Overfitting is mostly due to sparseness of data.

I Same model complexity: more data ⇒ less overfitting. With more data,
more complex (i.e. more flexible) models can be used.

Figure credit: Christopher Bishop

5
Model selection

I How to choose the optimal model complexity/hyper-parameter

(e.g., choose the best degree for polynomial regression)
I Cannot be done by training data alone

6
Model selection

I How to choose the optimal model complexity/hyper-parameter

(e.g., choose the best degree for polynomial regression)
I Cannot be done by training data alone

I We can use our prior knowledge or expertise (e.g., somehow we

know that the degree should not exceed 4)
I Create held-out data to approximate the test error (i.e., mimic
the test data)
I called validation data set

6
Model selection: cross-validation

For each order of polynomial M

1. Randomly split the training data into K groups, and following procedure
K times:
i. Leave out the k -th group from the training set as a validation set
ii. Use the other other K − 1 to find best parameter vector wk
iii. Measure the error of wk on the validation set; call this Jk

1
PK
2. Compute the average errors: J = K k =1 Jk

Choose the order of polynomial M with the lowest error J.

7
Model selection: cross-validation

Figure: K -fold cross-validation for the case of K = 4

Figure credit: Christopher Bishop

8
General learning procedure

Given a training set and a test set

1. Use cross-validation to choose the hyper-parameter/hypothesis

class.

2. Once the hyper-parameter is selected, use the entire training set

to find the best model parameters w.

3. Evaluate the performance of w on the test set.

These sets must be disjoint! – you should never touch the test data
before you evaluate your model.

9
Summary of cross-validation

I Can also used for selecting other hyper-parameters for

model/algorithm (e.g., number of hidden layers of neural
networks, learning rate of gradient descent, or even different
machine learning models)
I Very straightforward to implement algorithm
I Provides a great estimate of the true error of a model
I Leave-one-out cross-validation: number of groups = number of
training instances
I Computationally expensive; even worse when there are more
hyper-parameters

10
Regularization

I Intuition: complicated hypotheses lead to overfitting

I Idea: penalize the model complexity (e.g., large values of wj ):

L(w) = J(w) + λR(w)

where J(w): training loss, R(w): regularization

function/regularizer, and λ ≥ 0: regularization parameter to
control the tradeoff between data fitting and model complexity.

11
`2 -norm regularization for linear regression
Objective function:
m n 2 λ X n
1 XX
L(w) = w0 + wj · xi,j − yi + wj2
2 2
i=1 j=1 j=1

I No regularization on w0 !

Equivalently, we have
1 λ
L(w) = ||Xw − y||22 + w > Îw
2 2
where w = [w0 , w1 , . . . , wn ]>
 
0 0 ··· 0
0 1 ··· 0
Î =  .
 
.. .. .. 
 .. . . .
0 0 ··· 1
12
`2 -norm regularization for linear regression

Objective function:

1 λ
L(w) = ||Xw − y||22 + w > Îw
2 2
1 > > 
= w (X X + λÎ)w − w > X > y − y > Xw + y > y
2
Optimal solution (by solving ∇L(w) = 0):

w = (X > X + λÎ)−1 X > y

13
More on `2 -norm regularization

1 λ
arg min ||Xw − y||22 + w > Îw = (X > X + λÎ)−1 X > y
w 2 2
I `2 -norm regularization pushes the parameters towards to 0.

I λ = 0 ⇒ same as in the regular linear regression

I λ→∞⇒w →0

I 0 < λ < ∞ ⇒ magnitude of the weights will be smaller than in the

regular linear regression

Figure credit: Christopher Bishop 14

Another view of `2 -norm regularization

I From the optimization theory1 , we know that

min J(w) + λR(w)

w

is equivalent to
min J(w)
w

such that R(w) ≤ η

for some η ≥ 0.
I Hence, `2 -regularized linear regression can be re-formulated as (we
only consider wj , j > 0 here)

min ||Xw − y ||22

w

such that ||w||22 ≤ η

1
e.g., Boyd and Lieven. Convex Optimization. 2004.
15
Visualizing `2 -norm regularization (2 features)

Figure: w ∗ = (X > X + λI)−1 Xy

Figure credit: Christopher Bishop

16
`1 -norm regularization

I Instead of using `2 -norm, we use `1 -norm to control the model

complexity:
m n n
1 XX 2 X
min w0 + wj · xi,j − yi + λ |wj |
w 2
i=1 j=1 j=1

which is equivalent to
m n
1 XX 2
min w0 + wj · xi,j − yi
w 2
i=1 j=1
n
X
such that |wj | ≤ η
j=1

I Also called LASSO (least absolute shrinkage and selection operator).

I No analytical solution anymore!

17
Visualizing `1 -norm regularization (2 features)
I If λ is large enough , the circle is very likely to intersect the diamond at
one of the corners.
I This makes `1 -norm regularization much more likely to make some
weights exactly 0.
I In other words, we essentially perform feature selection!

18
Comparison of `2 and `1

Figure credit: Bishop; Hastie, Tibshirani & Friedman 19

Summary of regularization

I Both are commonly used approaches to avoid overfitting.

I Both push the weights towards 0.

I `2 produces small, but non-zero weights, while `1 is likely to make some

weights exactly 0.
I `1 optimization is computationally more expensive than `2 .

I Choose appropriate λ: cross-validation is often used.

20