Statistical Methods for Bioinformatics

II-2: Variable Selection, Ridge & Lasso

1/33
Statistical Methods for Bioinformatics
Agenda

Framing Variable Selection and High Dimensionality

Variable Selection
Best Subset Selection
Forward Step-wise Selection
Backward Step-wise Selection
Predictor Set Size Penalized Performance Measures
Shrinkage/regularization methods
Ridge
Lasso
Details and interpretations of the optimization functions

2/33
Statistical Methods for Bioinformatics
High dimensional datasets are common in Bioinformatics

It is quite common to have many measured features

103 − 105 genomic markers (GWAS)
Agilent Human Genome CGH1 +SNP Microarray:
292,097 probes for CGH and 119,091 probes to identify SNPs
525 (mycoplasma) - 28,354 (Norway Spruce) expression of all
genes (RNA-seq)
50 − 5 · 103 metabolite levels (Metabolomics)
physiological parameters

1
Comparative Genome Hybridisation to identify copy number changes 3/33
Statistical Methods for Bioinformatics
High dimensional datasets are a challenge

Here we will talk about the standard linear model:

Y = β0 + β1 X1 + . . . + βp Xp + ϵ
In general: more observations are needed to accurately fit the
parameters
As the number of parameters (p) approaches the number of
observations (n)
The Ordinary Least Squares fit may have have little bias, but
will have large variability
When p > n Ordinary Least Squares gives no unique solution.
The interpretability is diminished

4/33
Statistical Methods for Bioinformatics
The perennial trade-off: bias vs variance

One wants a model that captures the regularities in training

data, but also generalizes well to unseen data.

E (MSE ) = Var (fˆ(x0 )) + Bias(fˆ(x0 ))2 + Var (ϵ)

With Var (fˆ) = E ((E (fˆ) − fˆ)2 ) & Bias(fˆ)2 = E ((f − E (fˆ))2 )
and the irreducible error ϵ.
With OLS there may be little bias, but as n approaches p
variability increases.

5/33
Statistical Methods for Bioinformatics
Choose the Optimal Model
The Principle of Parsimony (Occam’s Razor)
An explanation is better if it is simple.
e.g. models should be pared down until they are minimal and
adequate

Adding more predictors will always improve or match

performance on a training set.
Estimate the test error using a validation set or a
cross-validation approach.
How do we select the best set of features?

6/33
Statistical Methods for Bioinformatics
Linear Model Selection and Regularization
Subset Selection
Identifying a subset of all p predictors X that we believe to be
related to the response Y, and then fitting the model using
this subset
Best subset selection, forward and backward step-wise
selection
Shrinkage
Shrinking the estimates coefficients towards zero.
This shrinkage reduces the variance. Some of the coefficients
may shrink to exactly zero, and hence shrinkage methods can
also perform variable selection
E.g. Ridge regression and the Lasso
Dimension Reduction (next class)
Involves projecting all p predictors into an M-dimensional
space where M < p, and then fitting linear regression model
Principle Components Regression and Partial Least Squares 7/33
Statistical Methods for Bioinformatics
Subset selection algorithm
In this approach, we run a model (e.g. a linear regression) for
each possible combination of the X predictors
For k=1,2,...p:
 
p
Fit all models that contain k predictors.
k
Pick the best among these, best is defined as having the best
performance, e.g. smallest RSS, or largest R2.
Select a single best model across sizes taking into account the
number of parameters: Cp, AIC, BIC, and adjusted R2 ;
Cross-Validated prediction error.

8/33
Statistical Methods for Bioinformatics
Refresher: definitions

Residual Sum of Squares (RSS)

X
RSS = (y − ŷ )2
Total Sum of Squares (TSS)
X
TSS = (y − ȳ )2

R 2 measure of fit / explained variance

RSS
1−
TSS

9/33
Statistical Methods for Bioinformatics
Subset Selection and Step-wise Selection

The number of models to consider increases rapidly with the

number of predictors (2p )
The method is not computationally feasible for large numbers
of predictors
The larger the search space, the likelier you have over-fitting
on the training/higher variance.
Alternatives: step-wise selection
Forward Step-wise Selection: Begins with the model containing
no predictor, and then adds one predictor at a time that
improves the model the most until no further improvement is
possible
Backward Step-wise Selection: Begins with the model
containing all predictors, and then deleting one predictor at a
time that improves the model the most until no further
improvement is possible

10/33
Statistical Methods for Bioinformatics
Forward Step-wise Selection
1 Start from an empty model
2 For k = 0, . . . , p − 1
1 Consider all p − k models that augment the predictor set with
one
2 Choose the best according to a performance measure (e.g.
RSS)
3 Compare the models of different sizes using Cp, AIC, BIC, and
adjusted R2 ; Cross-Validated prediction error.

Considerations
1 Considerable less models fit (1 + p2 (p + 1) vs 2p )
2 Works well in practice, but may be sub-optimal due to
correlation and interaction between variables
1 additions of a new variable may make already included
variables “non-significant”
2 an optimal pair or triple of parameters may be missed in the
early phases by the progressive procedure
11/33
Statistical Methods for Bioinformatics
Backward Step-wise Selection
1 Start from an full model with all predictors
2 For k = p, p − 1, . . . , 1
1 Consider all k models that remove one predictor from the
predictor set
2 Choose the best according to a performance measure (e.g.
RSS)
3 Compare the models of different sizes using Cp, AIC, BIC, and
adjusted R2 ; Cross-Validated prediction error.
Considerations
1 Considerable less models fit than “Best Subset” (1 + p2 (p + 1)
vs 2p )
2 Works well in practice, avoids missing successful combinations
between variables
Only possible when p<n (without further constraints and with
Ordinary Least Squares)
No guarantee optimal solution
12/33
Statistical Methods for Bioinformatics
Predictor Set Size Penalized Performance Measures

The following measures for a linear model with Gaussian error!

Adjusted R2
RSS/(n−d−1)
Ra2 = 1 −
TSS/(n−1)

Bayesian information criterion (BIC)

BIC = −2 ln L̂ + ln(n)d)
with L the maximized likelihood for the model. For a least squares
lm fit
1
BIC = (RSS + ln(n)d σ̂ 2 )
n
d - # of predictors; n - # of observations; σˆ2 - estimate of error variance

13/33
Statistical Methods for Bioinformatics
Predictor Set Size Penalized Performance Measures

Akaike information criterion (AIC)AIC = 2d − 2 ln(L)

1
AIC = (RSS + 2d σ̂ 2 )
nσ̂ 2
Mallow’s Cp (proportional to AIC for linear regression)

1
Cp = (RSS + 2d σ̂ 2 )
n
d - # of predictors; n - # of observations; σˆ2 - estimate of error variance

14/33
Statistical Methods for Bioinformatics
(Cross)-Validation

More direct way of evaluating the effect of larger prediction

models
Validation: learn on one set, test on another normally smaller
set
Cross-Validation: similar to validation, the data set is split in
training and test sets, but now repeatedly (typically 5-10
times) and the resulting performance measures are averaged.
When comparing models for feature selection:
does not depend on knowing d, the fitted parameters, more
difficult e.g. in regularization approaches (e.g. LASSO/Ridge,
see later)
or σˆ2 , which can be challenging to estimate, esp. with n ∼ p

15/33
Statistical Methods for Bioinformatics
Credit data example (from the book)

16/33
Statistical Methods for Bioinformatics
Choose the Optimal Model

All approaches suggest that the 4-7-variable models are

roughly equivalent in terms of their test errors.
One way to select a model relies on the 1-Standard-Error
Rule. Estimate the standard error of the test performance
statistic for each model, and then select the -smallest- model
with a score within one standard error of the lowest point on
the curve. (Why?)

17/33
Statistical Methods for Bioinformatics
Shrinkage/regularization methods

In high dimensional datasets, regression with least squares can

face over-defined systems and inflated variance (as in
trade-variance trade-off)
Subset selection methods use least squares to fit a linear
model with a subset of the predictors.
An alternative approache is to fit a model containing all p
predictors but constraining or regularizes the coefficient
estimates, or equivalently, that shrinks the coefficient
estimates towards zero.
It may not be immediately obvious why such a constraint
should improve the fit, but it turns out that shrinking the
coefficient estimates can significantly reduce their variance.

18/33
Statistical Methods for Bioinformatics
Shrinkage methods

Ordinary least squares minimizes:

X n p
X
RSS = (yi -β0 − βj xij )2
i=1 j=1

Ridge regression minimizes:

p
X
RSS + λ βj2
j=1

Lasso minimizes:
p
X
RSS + λ |βj |
j=1

19/33
Statistical Methods for Bioinformatics
Shrinkage adds a penalty on coefficients

The penalty acts against a large β vector

20/33
Statistical Methods for Bioinformatics
Figure Explanation

In the left-hand panel, each curve corresponds to the ridge

regression coefficient estimate for one of the ten variables,
plotted as a function of λ.
The right-hand panel displays the same ridge coefficient
estimates as the left-hand panel, but instead of displaying λ
on the x-axis, we now display βλR 2 / ∥β∥2 where β denotes
the vector of least squares coefficient estimates, βλR the
reduced coefficients for a λ.
The notation ∥β∥2 denotes the l2 norm
qP(pronounced “ell 2”) of
p 2
a vector, and is defined as ∥β∥2 = j=1 βj

21/33
Statistical Methods for Bioinformatics
The strength of Ridge/Lasso vs Ordinary Least Squares
Ideally the penalty in the fit reduces variance at the cost of a
small increase in bias.
In some cases variance can seriously hamper Least Squares
fits. In the simulated dataset analysed below there are 45
predictors and 50 datapoints. Variance is large with n close to
p. Least Squares matches the right extreme of the right panel.
The shrinkage methods can work with n<p.

22/33
Statistical Methods for Bioinformatics
The strength of Ridge/Lasso vs Ordinary Least Squares
In the simulated dataset analysed below there are 45
predictors and 50 datapoints, but only 2 predictors are
associated with the response

23/33
Statistical Methods for Bioinformatics
Selecting the Tuning Parameter λ

For optimal performance of ridge regression/lasso, finding the

right λ is essential.
Cross-validation
Select a grid of potential values, use cross validation to
estimate the error rate on test data (for each value of λ) and
select the value that gives the least error rate, or use the
1-Standard-Error Rule.

24/33
Statistical Methods for Bioinformatics
Some nuts and bolts
No penalty is imposed on the intercept.
The penalty is formulated on the size of the coefficients, this
implies that the scale of the variables is important! Therefore
the variables should be standardized to make them
comparable:
xi,j
x̃i,j = q P
1 n 2
2 i=1 (xi,j − x¯j )

If p is large, then using the best subset selection approach

requires searching through enormous numbers of possible
models
With Ridge Regression/lasso, the computations required to
solve the optimization can be very efficient. A fit,
simultaneously for all values of λ, requires almost identical
number of computations to those for fitting a model using
least squares.
25/33
Statistical Methods for Bioinformatics
Lasso vs ridge regression
Lasso uses a different penalty on the coefficients, which has
the effect of tending to set some coefficients to zero.
With Lasso, we can produce a model that has high predictive
power and it simpler to interpret
Automated feature selection (what could possibly go wrong).

26/33
Statistical Methods for Bioinformatics
Lasso vs ridge regression

We can rewrite the optimization problems as follows:

Definitions
Lasso: nP Pp o
n 2 subject to
minimize(β) i=1 (yi − β0 − j=1 βj xij )
Pp
j=1 |β j | ≤ s
Ridge: nP o
n Pp 2 subject to
minimize(β) (y
i=1 i − β0 − β x
j=1 j ij )
Pp 2
j=1 βj ≤ s

In words: For every value of λ there is a value of s so that the

outcome of the above defitions match our earlier definitions

27/33
Statistical Methods for Bioinformatics
Lasso vs ridge regression

28/33
Statistical Methods for Bioinformatics
Lasso vs ridge regression

1 Lasso produces simpler and more interpretable models

2 However predictive performance depends on the data. Are
there (many) variables with no (independent) association to
the response? If there are Lasso may work better, if there are
not, Ridge may work better.

29/33
Statistical Methods for Bioinformatics
A generalization of the penalty

Contours of constant value of j |βj |q , with lasso at q=1 and

P
ridge q=2. There are alternative forms, with as q → 0
corresponding to subset selection.
There are also hybrid approaches: the Elastic Net combines
the ridge and lasso penalties. Minimizes:
p
X p
X
RSS + λ2 βj2 + λ1 |βj |
j=1 j=1

And the penalized graphical lasso, used for estimating a sparse

undirected graph.
30/33
Statistical Methods for Bioinformatics
Bayesian interpretation of Ridge Regression and the Lasso
One way of deriving the Ridge Regression and Lasso formulations is by
assuming a probability distribution g that yields the coefficients β,
Qp
p(β) = j=1 g (βj ). The figure below shows the double exponential
density (solid line), which is the implicit prior used by lasso. The dotted
line shows a normal density, used by ridge regression.

31/33
Statistical Methods for Bioinformatics
Recap

Variable Selection Approaches

Best Subset Selection
Forward Step-wise Selection
Backward Step-wise Selection
Measures to compare Models with different numbers of
Variables
Definition and use of Ridge and Lasso
How they compare to each other and other approaches
Interpretation of the optimization functions

32/33
Statistical Methods for Bioinformatics
To do:

Preparation for next class

Read remainder chapter 6
Send in any questions day before class

Exercises
Exercise 6.8.1 and 6.8.2
Do Lab 6.5 and 6.6
Exercises 6.8.5, 6.8.8, 6.8.10

33/33
Statistical Methods for Bioinformatics