GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE

SELECTION

MOHAMMAD REZA KARIMI

Abstract. In the class, the SVMs where introduced as a variation to the Perceptron al-
gorithm: SVM uses the Hinge loss instead of the Perceptron loss. In the first part of this
handout, we give a geometrical interpretation to what SVM does, and return to the Hinge
loss from geometry. For interested readers, we also bring the concept of support vectors, as
well as the dual of the SVM problem, which can be considered as a prelude to kernels. In the
second part, we bring some examples for the problem of feature selection in machine learning.
Sections marked with a * are considered as optional.

Contents
Notations 1

Part 1. Geometry of SVM 2

0. Preliminaries 2
1. Linear Classification 2
2. Maximum Margin Separator 3
3. Soft-SVM 4
4. From Soft-SVM to Hinge Loss 5
5. Support Vectors* 6
6. Dual of the SVM Problem* 6

Part 2. Feature Selection 7

7. Linear Models 7
8. Importance of a Global Look 7
9. LASSO Regularisation 8
References 8

Notations. We do not use bold font to distinguish vectors from scalars, as it is easily understood
from the context. If x is a vector, its ith component is denoted by xi . We index different vectors
by lowerPindices, e.g., x1 and xj . We denote by hu, vi the inner product of u and v, defined by
hu, vi = ui v i . The Euclidean norm of a vector x is denoted by kxk.

Learning and Adaptive Systems, ETH Zürich

E-mail address: [email protected].
Date: March 2019.
1
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 2

Part 1. Geometry of SVM

0. Preliminaries
Before starting off, we bring some facts about (affine) hyperplanes. A hyperplane in Rd is
defined by the set of solutions to a single linear equation. That is, it is the set of all x ∈ Rd
satisfying
hw, xi = w1 x1 + · · · + wd xd = 0,
where w = (w1 , . . . , wd ) ∈ Rd is a nonzero vector, and is sometimes called the normal vector,
as it is orthogonal to all points on the hyperplane. Note that the dimension of a hyperplane is
d − 1, as it is the kernel of a linear transformation with one dimensional image. See the recap
slides to learn more about kernels and images of linear transformations.
For b ∈ R, the set {x ∈ Rd | hw, xi = b} is a translated hyperplane, which is sometimes called
an affine hyperplane, or simply a hyperplane. This hyperplane is parallel to {x | hw, xi = 0}.
Increasing b will move the hyperplane in the direction of w, and decreasing b will move the
hyperplane in the direction of −w. Sometimes b is called the intercept of the hyperplane.
For a point z ∈ Rd , the orthogonal projection of z on the hyperplane hw, xi = b can be
computed (geometrically) as follows: one moves z along the line defined by w, until it hits the
hyperplane. Concretely, one is looking for t ∈ R such that
hw, z + twi = b.
b−hw,zi
Expanding the inner product gives hw, zi+tkwk2 = b, or t = kwk2 . One is also able to compute
|b−hw,zi|
the Euclidean distance, which is ktwk = |t| · kwk = kwk . A special case is demonstrated in
the following lemma.
Lemma 1. Let w ∈ Rd and H = {x ∈ Rd | hw, xi = 0} be a hyperplane. The orthogonal
distance of a point z ∈ Rd to H can be computed as
| hw, zi |
.
kwk
Specifically, if w is a unit vector, the inner product hw, zi directly gives the distance of z to H.
The last reminder is about separation. A hyperplane H = {x ∈ Rd | hw, xi = b} divides the
space into three regions: the points on H, the half-space where w is pointing to, {x | hw, xi − b >
0}, and the half-space on the other side of H, which is {x | hw, xi − b < 0}.

1. Linear Classification
Let d ∈ N and D be an (unknown, but fixed) distribution over Rd × {−1, 1}. Assume that
we are given n i.i.d. samples (x1 , y1 ), . . . , (xn , yn ) ∼ D, where xi ∈ Rd are the features, and
yi ∈ {−1, 1} are the labels. For now, we make the following assumption (we will relax this
assumption in later sections):
Assumption 2 (Linear Separability). There exists a hyperplane that separates the data points
with y = 1 from those with y = −1. That is, there exists w ∈ Rd and b ∈ R such that
yi (hw, xi i − b) ≥ 0, ∀i ∈ {1, . . . , n}.
Any such hyperplane is called a separating hyperplane. Note that w is pointing towards the
datapoints with y = 1.
Our goal is to find (among all possible separating hyperplanes) the hyperplane H with maxi-
mum margin, where the margin is the distance of the closest point among x1 , . . . , xn to H, see
Figure 1. The reason behind this choice of a hyperplane is two-fold: firstly, the intuitive picture
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 3

suggests that this hyperplane is the correct separating hyperplane. Secondly, it is related to
generalisation power and stability, which we do not cover in this handout.

Figure 1. Different separating hyperplanes for a dataset. The green line de-
notes the largest margin separator.

2. Maximum Margin Separator

In the previous section, we worked with affine hyperplanes. To make the computations easier,
we augment each xi with a 1, and augment w with −b, i.e.,
xi ← (x1i , . . . , xdi , 1),
w ← (w1 , . . . , wd , −b).
Using this new xi and w, we only need to consider hyperplanes passing through the origin in Rd+1
and forget about the intercept. For ease of notation, let us also define D = {(x1 , y1 ), . . . , (xn , yn )}
to be the complete dataset.
Definition 3 (Margin). Let H be any separating hyperplane of D, defined via w as above. The
margin of H with respect to D is defined as
| hw, xi |
γD (w) := min .
(x,y)∈D kwk
By the results so far, one way to write down the problem of finding the maximum margin
separator is the following optimisation problem:
maximise γD (w)
w
(2.1)
s.t. yi · hw, xi i ≥ 0 for all i ∈ {1, . . . , n}.
The constraint in the problem above is simply stating that w is a separating hyperplane of D.
We need another assumption to proceed (this assumption will be removed in the following
sections).
Assumption 4. There exists a separating hyperplane with a nonzero margin.
The reader is encouraged to create an example that this assumption does not hold. Making
this assumption is necessary; otherwise, the optimisation problem (2.1) has the trivial solution
of 0.
Let w be such a separating hyperplane. Define a := min | hw, xi i | > 0. This means that
| hw, xi i | ≥ a,
which implies, by the fact that w is a separating hyperplane, that
yi · hw, xi i ≥ a.
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 4

Dividing w by a, gives rise to a separator w̃ with the same margin, but with the new constraints
yi · hw̃, xi i ≥ 1.
Note that in this case, γD (w̃) = 1/kw̃k. Thus, maximising the margin is equivalent to minimising
kw̃k, or similarly, minimising kw̃k2 . The final optimisation problem can be stated as follows:
Definition 5 (Hard-SVM). Let D be a linearly separable dataset, where Assumption 4 holds.
The following optimization problem (called the Hard-SVM) yields the maximum margin separa-
tor for D:
minimise 12 kwk2
(2.2) w
s.t. yi · hw, xi i ≥ 1 for all i ∈ {1, . . . , n}.

3. Soft-SVM
In the last section, we had two assumptions (Assumptions 2 and 4) about the dataset D,
which may not hold in a general dataset. Note that when these assumptions are violated, the
optimisation problem (2.2) has no solutions. In this section, we drop these assumptions, and
derive a soft version of SVM.
The key idea of this derivation is letting some constraints in 2.2 to be violated. One way to
achieve this is to introduce some slack variables for each constraint. Let ξ1 , . . . , ξn ≥ 0, and
define a new optimization problem
minimise 12 kwk2
w,ξ1 ,...,ξn
(3.1) s.t. yi · hw, xi i ≥ 1 − ξi , for all i ∈ {1, . . . , n},
ξi ≥ 0, for all i ∈ {1, . . . , n}.
The variables ξi denote the amount that the ith constraint is violated: ξi = 0 denotes no violation,
and larger ξi shows larger violation, see Figure 2.

Figure 2. Slack variables in Soft-SVM.

Although this optimisation problem has solutions for every dataset D (even the ones that are
not linearly separable), it is easy to see that its solution is always w = 0. This is not surprising,
as we have not defined any cost for violation of constraints. The simplest way to define such a
cost, is as follows:
Xn
minimise 12 kwk2 + C ξi
w,ξ1 ,...,ξn
i=1
(3.2)
s.t. yi · hw, xi i ≥ 1 − ξi , for all i ∈ {1, . . . , n},
ξi ≥ 0, for all i ∈ {1, . . . , n},
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 5

where C > 0 denotes the softness of the problem: larger C goes for larger violation penalties.
The optimisation problem (3.2) is called Soft-SVM. One can observe that when C → ∞, the
Soft-SVM problem gets similar to Hard-SVM problem.

4. From Soft-SVM to Hinge Loss

Let us start from optimisation problem (3.2). For a fixed w, the optimisation problem is
equivalent to
n
X
minimise ξi
ξ1 ,...,ξn
i=1
(4.1)
s.t. ξi ≥ 1 − yi · hw, xi i , for all i ∈ {1, . . . , n},
ξi ≥ 0, for all i ∈ {1, . . . , n}.
In a more compact way, the set of constraints is the same as
ξi ≥ max{0, 1 − yi · hw, xi i} = `hinge (yi · hw, xi i),
where `hinge is the Hinge loss defined as
`hinge (t) = max{0, 1 − t}.
P
As the objective of the optimisation is ξi , and the constraints are independent of eachother,
we find the optimal values of ξi by setting them to their lower bounds:
ξi? (w) = `hinge (yi · hw, xi i).
Note that this optimal value is evaluated for a fixed w, thus we used the notation ξi? (w). This
derivation suggests the following optimisation problem:
n
X
(4.2) minimise 21 kwk2 + C `hinge (yi · hw, xi i).
w
i=1

The following lemma states that this new optimisation problem has the same solution as in (3.2).
Lemma 6. The optimisation problems (4.2) and (3.2) have the same solution.
Proof. Let w? and ξ1? , . . . , ξn? be the optimal solutions to (3.2). Feasibility implies that for all
i ∈ {1, . . . , n},
ξi∗ ≥ `hinge (yi · hw? , xi i).
Optimality implies that the inequality above is indeed equality. Thus, the optimisation problem
gets reduced to (4.2). Reduction of (4.2) to (3.2) is similar. 
This completes our discussion about the equivalence of regularized Hinge loss minimisation
and the geometric approach to SVM. We conclude with the following remark.
Remark 7. There is a hidden issue when we do the transformation described in the begining
of Section 2. As seen in optimisation problems (2.2) and (4.2), the last coordinate of vector w
is the intercept of the separating hyperplane. As we are penalising the norm of w, we are also
penalising the intercept as well. This means (roughly) that we are looking for hyperplanes that
are closer to the origin.
The “correct” way to write the optimisation problem for SVM is to include b (the intercept)
and write the problem in terms of b and w, having in mind that b shall not be penalised. Although
this is possible, the clarity of our current derivation and little difference in experiments, made
us choosing the former approach. The reader is encouraged to derive the latter derivation and
see the difference.
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 6

5. Support Vectors*
Note. For this section, we need material discussed in the recap slides related to
KKT Optimality Conditions. This section is not related to the exam and is only
here for the interested reader.
We start with the Hard-SVM problem (2.2). Notice that the objective function is convex, and
all the conditions are linear inequalities. Thus, the KKT conditions apply. First, we have the
stationarity condition:
n
X
(5.1) w− λi yi xi = 0,
i=1

where λi ≥ 0. Let us denote by T the set of tight constraints of (2.2), i.e.,

T = {i ∈ {1, . . . , n} | yi · hw, xi i = 1} .
In SVM literature, we refer to these points as support vectors. The complementary slackness of
KKT theorem states that for i 6∈ T , we have λi = 0. Thus, one can rewrite (5.1) as
X
w− λi yi xi = 0,
i∈T

or equivalently
X
w= λi yi xi .
i∈T
We have thus proved the following lemma:
Lemma 8. The solution to Hard-SVM can be expressed (or represented) as a linear combination
of the support vectors, i.e., those points of the dataset which are exactly on the margin.
The good thing about support vectors is that they can be much smaller than the whole dataset,
and predictions (in the dual SVM) will get less expensive.

6. Dual of the SVM Problem*

Let us move a bit further and study the dual of the Hard-SVM problem (2.2). The first thing
that we do is to put the constraints in the objective function using the following trick. Let us
define the function
Xn
g(w) = maxn αi (1 − yi · hw, xi i).
α∈R
α≥0 i=1

This function has a closed form as follows


0 if yi · hw, xi i ≥ 1 for all i ∈ {1, . . . , n},
g(w) =
∞ otherwise.
That is, g(w) = 0 if w satisfies all the constraints of (2.2), and is infinity otherwise. Thus, the
optimization problem (2.2) is equivalent to the following (unconstrained) optimisation problem
min 21 kwk2 + g(w),
w

or the following minimax problem

n
X
min maxn 12 kwk2 + αi (1 − yi · hw, xi i).
w α∈R
α≥0 i=1
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 7

We note that in this situation we can exchange min and max.1 Thus, we arrive to the following
equivalent problem:
n
X
(6.1) maxn min 12 kwk2 + αi (1 − yi · hw, xi i).
α∈R w
α≥0 i=1

The inner minimisation problem can be solved analytically, and one gets
Xn
w= αi yi xi ,
i=1

which is again the result of Lemma 8. Note that for constraints which are not tight, αi = 0.
Replacing the value of w in (6.1) gives
n n n
X 1 XX i j
(6.2) maxn αi − α α yi yj hxi , xj i .
α∈R
i=1
2 i=1 j=1
α≥0

Note that solving (6.2) only needs the inner products of xi ’s. This is helpful in situations where
the dimensions of the problem is high (or even infinite!). You will learn in the course that the
inner product hxi , xj i can be replaced with a so-called kernel k(xi , xj ), which enables one to
solve nonlinear separation problems.

Part 2. Feature Selection

We bring three examples of feature selection in ML. The first two examples emphasise on
the fact that feature selection is not an obvious procedure, and one needs to be careful in
certain situations. The next example focuses on the Lasso regression and gives a picture for one
dimensional case. All examples are borrowed from [1].

7. Linear Models
Assume a regression problem, where features (x) are in R2 and responses (y) are real values.
Define the following distribution over the features and responses:
x1 ∼ Uniform(−1, 1)
y = (x1 )2
x2 = y + z, where z ∼ Uniform(−0.01, 0.01).
The question is, “which feature to choose? x1 or x2 ?”
Since the response is a function of x1 , one may say that x1 is the best selected feature.
However, in a linear model, x2 performs much better than x1 (why?).

8. Importance of a Global Look

An intuitive procedure for feature selection is as follows: for each feature, individually, we
assign a score (of how good the feature is), and we pick the features with highest score. While
this method seems promising, it can fail dramatically.
For linear regression, as an example, one can define the score of each feature to be the goodness
of fit for that single feature, i.e., the empirical squared loss of the prediction using only that
feature. It turns out (why?) that this score is proportional to the correlation of the feature and
the response: the higher the correlation, the higher the score. Thus, one shall pick the features
with highest correlations (see Pearson’s Correlation Coefficient).
1This is called Strong Duality.
 GEOMETRY OF SUPPORT VECTOR MACHINES AND FEATURE SELECTION 8

The following example shows how this method fails. Consider a linear regression problem on
R2 . Define the distribution over features (x) and response (y) as follows:
y = x1 + 2x2 ,
x1 ∼ Uniform({±1})
1 1
x2 = − x1 + z, where z ∼ Uniform({±1}).
2 2
Notice that E[yx ] = 0, meaning that x1 is not a good predictor alone! But one can see that any
1

good predictor should take x1 in to account to be able to predict well.

9. LASSO Regularisation
Consider a one dimensional linear regression problem with Lasso regularisation. The data
points are x1 , . . . , xn ∈ R and the responses are y1 , . . . , yn ∈ R. The problem is equivalent to
solving the following optimisation problem
n
1 X
(9.1) minimise (xi w − yi )2 + λ|w|,
w∈R 2n i=1
where λ is the regularisation parameter. Rewriting the objective function, we get the following
equivalent problem
n n
!
1 1 X 1 X
minimise w2 · x2 − w · xi yi + λ|w|.
w∈R 2 n i=1 i n i=1
By normalising the data, we can assume that n1
P 2
xi = 1, and get
 
1 2 1
minimise w − w · hx, yi + λ|w|.
w∈R 2 n
The following lemma then gives us the optimal solution
 
? | hx, yi |
w = sign(hx, yi) −λ ,
n +
which basically means that if the correlation between x and y is below the threshold λ, the
solution is w? = 0. The reader is encouraged to derive the same solution for Ridge regression
and see that there is no thresholding effect in that case.
Lemma 9. The solution to the optimisation problem
1
minimise w2 − xw + λ|w|
w∈R 2
for λ ≥ 0 is
w? = sign(x)[|x| − λ]+ ,
where [a]+ = max{a, 0}.
Proof. A simple proof proceeds with a case by case analysis and is left to the reader. 

References
[1] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory
to Algorithms. Cambridge University Press, 2014.