0

50

100

200
TV

300

25
5

10

15

Sales

20

25
20
15

Sales

10

15
5

10

Sales

20

25

What is Statistical Learning?

10

20

30

40

50

20

Radio

40

60

80

100

Newspaper

Shown are Sales vs TV, Radio and Newspaper, with a blue

linear-regression line fit separately to each.
Can we predict Sales using these three?
Perhaps we can do better using a model
Sales f (TV, Radio, Newspaper)
1 / 30

Notation
Here Sales is a response or target that we wish to predict. We
generically refer to the response as Y .
TV is a feature, or input, or predictor; we name it X1 .
Likewise name Radio as X2 , and so on.
We can refer to the input vector collectively as

X1

X = X2
X3
Now we write our model as
Y = f (X) + 
where  captures measurement errors and other discrepancies.
2 / 30

What is f (X) good for?

With a good f we can make predictions of Y at new points

X = x.

We can understand which components of

X = (X1 , X2 , . . . , Xp ) are important in explaining Y , and

which are irrelevant. e.g. Seniority and Years of
Education have a big impact on Income, but Marital
Status typically does not.

Depending on the complexity of f , we may be able to

understand how each component Xj of X affects Y .

3 / 30

6
4
2
0

Is there an ideal f (X)? In particular, what is a good value for

f (X) at any selected value of X, say X = 4? There can be
many Y values at X = 4. A good value is
f (4) = E(Y |X = 4)

E(Y |X = 4) means expected value (average) of Y given X = 4.

This ideal f (x) = E(Y |X = x) is called the regression function.

4 / 30

The regression function f (x)

Is also defined for vector X; e.g.

f (x) = f (x1 , x2 , x3 ) = E(Y |X1 = x1 , X2 = x2 , X3 = x3 )

5 / 30

The regression function f (x)

Is also defined for vector X; e.g.

f (x) = f (x1 , x2 , x3 ) = E(Y |X1 = x1 , X2 = x2 , X3 = x3 )

Is the ideal or optimal predictor of Y with regard to

mean-squared prediction error: f (x) = E(Y |X = x) is the

function that minimizes E[(Y g(X))2 |X = x] over all
functions g at all points X = x.

5 / 30

The regression function f (x)

Is also defined for vector X; e.g.

f (x) = f (x1 , x2 , x3 ) = E(Y |X1 = x1 , X2 = x2 , X3 = x3 )

Is the ideal or optimal predictor of Y with regard to

mean-squared prediction error: f (x) = E(Y |X = x) is the

function that minimizes E[(Y g(X))2 |X = x] over all
functions g at all points X = x.

 = Y f (x) is the irreducible error i.e. even if we knew

f (x), we would still make errors in prediction, since at each

X = x there is typically a distribution of possible Y values.

5 / 30

The regression function f (x)

Is also defined for vector X; e.g.

f (x) = f (x1 , x2 , x3 ) = E(Y |X1 = x1 , X2 = x2 , X3 = x3 )

Is the ideal or optimal predictor of Y with regard to

mean-squared prediction error: f (x) = E(Y |X = x) is the

function that minimizes E[(Y g(X))2 |X = x] over all
functions g at all points X = x.

 = Y f (x) is the irreducible error i.e. even if we knew

f (x), we would still make errors in prediction, since at each

X = x there is typically a distribution of possible Y values.
For any estimate f(x) of f (x), we have
E[(Y f(X))2 |X = x] = [f (x) f(x)]2 + Var()
|
{z
}
| {z }
Reducible

Irreducible

5 / 30

How to estimate f
Typically we have few if any data points with X = 4

exactly.
So we cannot compute E(Y |X = x)!
Relax the definition and let
f(x) = Ave(Y |X N (x))

where N (x) is some neighborhood of x.

4
x

6 / 30

 Nearest neighbor averaging can be pretty good for small p

i.e. p 4 and large-ish N .

We will discuss smoother versions, such as kernel and

spline smoothing later in the course.

7 / 30

 Nearest neighbor averaging can be pretty good for small p

i.e. p 4 and large-ish N .

We will discuss smoother versions, such as kernel and

spline smoothing later in the course.

Nearest neighbor methods can be lousy when p is large.

Reason: the curse of dimensionality. Nearest neighbors

tend to be far away in high dimensions.

We need to get a reasonable fraction of the N values of yi

to average to bring the variance downe.g. 10%.

A 10% neighborhood in high dimensions need no longer be

local, so we lose the spirit of estimating E(Y |X = x) by

local averaging.

7 / 30

The curse of dimensionality

1.0

p= 5

1.5

0.5

0.0
x1

p= 2
p= 1

0.5

p= 3

1.0

p= 10

Radius

1.0

x2

0.0

0.5

0.5

0.0

1.0

10% Neighborhood

0.5

1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Fraction of Volume

8 / 30

Parametric and structured models

The linear model is an important example of a parametric
model:
fL (X) = 0 + 1 X1 + 2 X2 + . . . p Xp .
A linear model is specified in terms of p + 1 parameters

0 , 1 , . . . , p .

We estimate the parameters by fitting the model to

training data.

Although it is almost never correct, a linear model often

serves as a good and interpretable approximation to the

unknown true function f (X).

9 / 30

A linear model fL (X) = 0 + 1 X gives a reasonable fit here

2

A quadratic model fQ (X) = 0 + 1 X + 2 X 2 fits slightly

better.
2

10 / 30

Incom
of

Se

rs

ni
or

ity

e
Ye
a

Ed

uc
a

tio
n

Simulated example. Red points are simulated values for income

from the model
income = f (education, seniority) + 
f is the blue surface.
11 / 30

Incom
of

Se

rs

ni

or

ity

e
Ye
a

Ed

uc
a

tio
n

Linear regression model fit to the simulated data.

fL (education, seniority) = 0 +1 education+2 seniority

12 / 30

of

Se

rs

ni

or

ity

Incom
Ye
a

Ed

uc
a

tio
n

More flexible regression model fS (education, seniority) fit to

the simulated data. Here we use a technique called a thin-plate
spline to fit a flexible surface. We control the roughness of the
fit (chapter 7).
13 / 30

of

Se

rs

ni

or

ity

Incom
Ye
a

Ed

uc
a

tio
n

Even more flexible spline regression model

fS (education, seniority) fit to the simulated data. Here the
fitted model makes no errors on the training data! Also known
as overfitting.
14 / 30

Some trade-offs

Prediction accuracy versus interpretability.

Linear models are easy to interpret; thin-plate splines

are not.

15 / 30

Some trade-offs

Prediction accuracy versus interpretability.

Linear models are easy to interpret; thin-plate splines

are not.

Good fit versus over-fit or under-fit.

How do we know when the fit is just right?

15 / 30

Some trade-offs

Prediction accuracy versus interpretability.

Linear models are easy to interpret; thin-plate splines

are not.

Good fit versus over-fit or under-fit.

How do we know when the fit is just right?

Parsimony versus black-box.

We often prefer a simpler model involving fewer

variables over a black-box predictor involving them all.

15 / 30

High

Subset Selection
Lasso

Interpretability

Least Squares
Generalized Additive Models
Trees

Bagging, Boosting
Low

Support Vector Machines

Low

High

Flexibility

FIGURE 2.7. A representation of the tradeoff between flexibility and inte

pretability, using different statistical learning methods. In general, as the16flexib
/ 30

Assessing Model Accuracy

Suppose we fit a model f(x) to some training data
Tr = {xi , yi }N
1 , and we wish to see how well it performs.

We could compute the average squared prediction error

over Tr:

MSETr = AveiTr [yi f(xi )]2

This may be biased toward more overfit models.

Instead we should, if possible, compute it using fresh test

data Te = {xi , yi }M
1 :

MSETe = AveiTe [yi f(xi )]2

17 / 30

2.5
0.5

1.0

Mean Squared Error

1.5

2.0

12
10
8
Y
6
4

0.0

2
0

20

40

60
X

80

100

10

20

Flexibility

Black curve is truth. Red curve on right is MSETe , grey curve is

FIGURE
2.9. Left: Data simulated from f , shown in black. Three estimates of
MSE
Tr . Orange, blue and green curves/squares correspond to fits of
f are shown: the linear regression line (orange curve), and two smoothing spline
different flexibility.
fits (blue and green curves). Right: Training MSE (grey curve), test MSE (red
curve), and minimum possible test MSE over all methods (dashed line). Squares
represent the training and test MSEs for the three fits shown in the left-hand

18 / 30

2.5
0.0

0.5

1.0

Mean Squared Error

1.5

2.0

12
10
8
6

20

40

60
X

80

100

10

20

Flexibility

Here the truth is smoother, so the smoother fit and linear model do
FIGURE 2.10. Details are as in Figure 2.9, using a different true f that is
really
well. to linear. In this setting, linear regression provides a very good fit to
much closer
the data.

19 / 30

15
10
5
0

10

10

Mean Squared Error

20

20

the data.

20

40

60
X

80

100

10

20

Flexibility

Here the truth is wiggly and the noise is low, so the more flexible fits
FIGURE 2.11. Details are as in Figure 2.9, using a different f that is far from
do
the best.
linear. In this setting, linear regression provides a very poor fit to the data.

20 / 30

Bias-Variance Trade-off
Suppose we have fit a model f(x) to some training data Tr, and
let (x0 , y0 ) be a test observation drawn from the population. If
the true model is Y = f (X) +  (with f (x) = E(Y |X = x)),
then

2
E y0 f(x0 ) = Var(f(x0 )) + [Bias(f(x0 ))]2 + Var().
The expectation averages over the variability of y0 as well as
the variability in Tr. Note that Bias(f(x0 ))] = E[f(x0 )] f (x0 ).
Typically as the flexibility of f increases, its variance increases,
and its bias decreases. So choosing the flexibility based on
average test error amounts to a bias-variance trade-off.

21 / 30

Bias-variance trade-off for the three examples

20
15

2.0

MSE
Bias
Var

0.5
2

10
Flexibility

20

0.0

0.0

0.5

1.0

1.0

10

1.5

1.5

2.0

2.5

2. Statistical Learning

2.5

36

10
Flexibility

20

10

20

Flexibility

FIGURE 2.12. Squared bias (blue curve), variance (orange curve), Var()
(dashed line), and test MSE (red curve) for the three data sets in Figures 2.92.11.
The vertical dashed line indicates the flexibility level corresponding to the smallest
test MSE.
22 / 30

Classification Problems

Here the response variable Y is qualitative e.g. email is one

of C = (spam, ham) (ham=good email), digit class is one of
C = {0, 1, . . . , 9}. Our goals are to:

Build a classifier C(X) that assigns a class label from C to

a future unlabeled observation X.

Assess the uncertainty in each classification

Understand the roles of the different predictors among

X = (X1 , X2 , . . . , Xp ).

23 / 30

1.0

| | | || || |||||| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
||||||||||||||||||||||||||
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||| |||| ||| | | ||

0.0

0.2

0.4

0.6

0.8

||

||

| | || |||| |||||||| |||||||||||||||||||||||||||||||||||||||||||

||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| | ||||||| | | |
1

Is there an ideal C(X)? Suppose the K elements in C are

numbered 1, 2, . . . , K. Let
pk (x) = Pr(Y = k|X = x), k = 1, 2, . . . , K.
These are the conditional class probabilities at x; e.g. see little
barplot at x = 5. Then the Bayes optimal classifier at x is
C(x) = j if pj (x) = max{p1 (x), p2 (x), . . . , pK (x)}
24 / 30

1.0

| |

||

||

|| || || ||

|| | | || ||| | | ||||||||| || || | ||||| || | | | |

0.0

0.2

0.4

0.6

0.8

| || |

||

|| | |

| |||| || ||| | | || | | ||| | || |

|
5

|
6

Nearest-neighbor averaging can be used as before.

Also breaks down as dimension grows. However, the impact on

C(x)
is less than on pk (x), k = 1, . . . , K.
25 / 30

Classification: some details

Typically we measure the performance of C(x)

using the
misclassification error rate:

i )]
ErrTe = AveiTe I[yi 6= C(x
The Bayes classifier (using the true pk (x)) has smallest

error (in the population).

26 / 30

Classification: some details

Typically we measure the performance of C(x)

using the
misclassification error rate:

i )]
ErrTe = AveiTe I[yi 6= C(x
The Bayes classifier (using the true pk (x)) has smallest

error (in the population).

Support-vector machines build structured models for C(x).

We will also build structured models for representing the

pk (x). e.g. Logistic regression, generalized additive models.

26 / 30

2. Statistical Learning

Example: K-nearest neighbors in two dimensions

oo o
o
o
o
o
oo oo o
o
o
o
o ooo
o
oo ooo o
oo o
oo
oo o o
o
o ooo oo
o oo
oo
o
o o o oo
o
o
o
oo
o
o o
o
o
o o
o oo o
o o
o o o
o o
o oo
o o
o
o o oooo o
ooo o o o o
ooo
o
o
oo o o o ooooo o o
o o
o
oo o o
oo o
o o oo o
o
o
o
o oo
o o o oo o ooo o
o
o
oo o
o ooooo oooo
oo
o
o oo o o
o
o o
oo oo o
o
oo oo
oo
o
o
o
o
o
o
o
oo o
o
o
o o

X2

oo

X1
27 / 30

KNN: K=10

oo o
o
o
o
o
oo oo o
o
o
o
o
o
o o
o
oo oo
oo o
oo
o oo o
oo o o
o
o
o
o
o
o oo
oo
o
o o o oo
o
o
o
oo
o
o o
o
o
o o
o oo o
o o
o
o o o
o
o oo
o o
o
o o oooo o
ooo o o o o
o
o
o
o
o o oo o o o
o
oo
o
o o
o
oo o oo o
oo o
o o oo o
o
o
o
o oo
o
o o o oo o oo o
o
o
oo o
o ooooo oo
oo
o
o
o
o oo o o
o
o o
oo oo o
o
o
ooo
o
o
o ooo
o
o
o
oo o
o
o
o o

X2

oo

X1

FIGURE 2.15. The black curve indicates the KNN decision boundary on the

28 / 30

FIGURE 2.15. The black curve indicates the KNN decision boundary on the
data from Figure 2.13, using K = 10. The Bayes decision boundary is shown as
a purple dashed line. The KNN and Bayes decision boundaries are very similar.
KNN: K=1

oo

o
o
o
oo

o
o

o
o
o
o
o
oo oo o o
o
o
oo o o
o
oo
o oo
oo o o
o
o
oo o
o
oo
oo
o
o
o
o
o
o
o o
o
o
o
o
o o
o
o
o o
o oo o
o o
o
o o
o
o oo
o
o
o
o o
o
o o ooo o
ooo o
o
o
oo
o
o
o o oo o o
o
o
oo
o ooo
o
o
o o o
oo o
o o oo o
oo o
o
o
o
o
o
o
o o o o o oo
o
o
ooo o
o oooo oo
oo
o
oo
o
o
o oo o
o
o o
oo oo o
o
oo o
o
o
oooo
oo
o
o
oo o
o
o

oo

KNN: K=100

oo

o
o
o
oo

o
o

o
o
o
o
oo oo o o
o
o
oo o o
o
oo
o oo
oo o o
o
o
oo o
o
oo
oo
o
o
o
o
o
o
o o
o
o
o
o
o o
o
o
o o
o oo o
o o
o
o o
o
o oo
o
o
o
o o
o
o o ooo o
ooo o
o
o
oo
o
o
o o oo o o
o
o
oo
o ooo
o
o
o o o
oo o
o o oo o
oo o
o
o
o
o
o
o
o o o o o oo
o
o
ooo o
o oooo oo
oo
o
oo
o
o
o oo o
o
o o
oo oo o
o
oo o
o
o
oooo
oo
o
o
oo o
o
o

oo

FIGURE 2.16. A comparison of the KNN decision boundaries (solid black

curves) obtained using K = 1 and K = 100 on the data from Figure 2.13. With
K = 1, the decision boundary is overly flexible, while with K = 100 it is not
29 / 30
sufficiently flexible. The Bayes decision boundary is shown as a purple dashed

0.20
0.15
Error Rate

0.10
0.05
0.00

Training Errors
Test Errors
0.01

0.02

0.05

0.10

0.20

0.50

1.00

1/K
30 / 30