Support Vector Machines

Dr Sridhar R
Dept. of ISE, GAT
 Support Vector Machines
Support Vector Machine (SVM) is a supervised machine learning algorithm that
can be used for both classification or regression challenges. However, it is
mostly used in classification problems.
In the SVM algorithm, we plot each data item as a point in
n-dimensional space (where n is a number of features you have) with the value of
each feature being the value of a particular coordinate. Then, we perform
classification by finding the hyper-plane that differentiates the two classes very
well.
 Support Vector Machines
Support Vectors are simply the coordinates of individual observation. The SVM classifier is a
frontier that best segregates the two classes (hyper-plane/ line).
Example
Support Vector Machines
Support Vector Machines
Support Vector Machines
Support Vector Machines
Support Vector Machines - separable
case Support Vector Machines -
non-separable case
 Linearly Separable Case
Dataset is linearly separable and the classification problem is Binary
classification or two class classification.

Binary classification can be viewed as the task of separating classes in

feature space:
Hyperplane f(x) = (wTx + b) – w is a weight vector
– x is input vector
– b is bias
 α
Linear Classifiers
x f yest
f(x,w,b) = sign(w. x + b)
denotes +1
denotes -1

Any of these
would be fine..

..but which is
best?
Examples of Bad Decision Boundaries

Class 2 Class 2

Class 1 Class 1
Mehryar Mohri - Foundations of Machine Learning page 13
 Linearly Separable Case
Define the hyperplanes H such that:
w•xi+b ≥ +1 when yi =+1 H1
w•xi+b ≤ -1 when yi = –1 H0
H2
H1 and H2 are the margins: d
+

H1: w•xi+b = +1 d
-
H
H2: w•xi+b = –1
The points on the margins H1 and
H2 are the tips of the Support
Vectors.The plane H0 is the median
in between, where w•xi+b =0 d+ = the shortest distance to the closest positive point
d- = the shortest distance to the closest negative point

The margin (gutter) of a separating hyperplane is d+ + d–.

 Maximizing the margin
We want a classifier with as big margin as possible.
Recall the distance from a point(x0,y0) to a line:
Ax+By+c = 0 is |A x0 +B y0 +c|/sqrt(A2+B2)
The distance between H and H1 is:
|w•x+b|/||w||=1/||w||
The distance between H1 and H2 is: 2/||w||

In order to maximize the margin, we need to minimize ||w||. With the condition that there are no
datapoints between H1 and H2:

xi•w+b ≥ +1 when yi =+1

xi•w+b ≤ -1 when yi =-1 Can be combined into yi(xi•w) ≥ 1
constrained optimization problem
 Definitions
• Margin of Separation (d): the separation between the hyperplane and the closest
data point for a given weight vector w and bias b.

• Optimal Hyperplane (maximal margin): the particular hyperplane for which

the margin of separation d is maximized.
• Support vectors are the elements of the training set that would change the
position of the dividing hyperplane if removed. Support vectors are the critical
elements of the training set
Support Vector Machines - separable
case Support Vector Machines -
non-separable case
 Non Separable Case
Support Vector Machine (SVM) is one of the most popular classification
techniques which aims to minimize the number of misclassification errors
directly. Suppose we have some data and it can be depicted as following in the
2D space:
Non Separable Case
Non Separable Case
 Non Separable Case

Mehryar Mohri - Foundations of Machine Learning page 22

 Implementing
Loading The Data SVM
from sklearn import datasets

cancer_data = datasets.load_breast_cancer()
print(cancer_data.data[5])

Output:

[1.245e+01 1.570e+01 8.257e+01 4.771e+02 1.278e-01 1.700e-01

1.578e-01 8.089e-02 2.087e-01 7.613e-02 3.345e-01 8.902e-01
2.217e+00 2.719e+01 7.510e-03 3.345e-02 3.672e-02 1.137e-02
2.165e-02 5.082e-03 1.547e+01 2.375e+01 1.034e+02 7.416e+02
1.791e-01 5.249e-01 5.355e-01 1.741e-01 3.985e-01 1.244e-01]

Mehryar Mohri - Foundations of Machine Learning page 23

 Implementing
Explore Data SVM
print(cancer_data.data.shape)
#target set
print(cancer_data.target)

Output:
(569, 30) [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 1 0 1 1 1 1 1 0 0 1 0 0 1 1 1 1 0 1 0 0 1 1 1 1 0 1 0 0 1 0 1 0 0 1 1 1 0 0 1
0 0 0 1 1 1 0 1 1 0 0 1 1 1 0 0 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 0 0 0 1 0 0 1 1 1 0 0 1 0
1 0 0 1 0 0 1 1 0 1 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 0 1 0 1 1 0 0 1 1 0 0 1 1
1 1 0 1 1 0 0 0 1 0 1 0 1 1 1 0 1 1 0 0 1 0 0 0 0 1 0 0 0 1 0 1 0 1 1 0 1 0 0 0 0 1 1 0 0
1 1 1 0 1 1 1 1 1 0 0 1 1 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
0 1 1 1 1 1 1 0 1 0 1 1 0 1 1 0 1 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1 1 1
1 1 1 1 1 1 1 0 1 1 1 0 1 0 1 1 1 1 0 0 0 1 1 1 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 0 0 0 1
1 1 1 1 1 1 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 1 1 0 1 1 1 1 1 0 1 1 1 0 1 1 0 0 1 1 1 1 1 1
0 1 1 1 1 1 1 1 0 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 1 0 1 1 1 1 1 0 1 1 0
1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 1 0
1 1 0 1 1 1 1 1 0 0 1 0 1 0 1 1 1 1 1 0 1 1 0 1 0 1 0 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1
0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1]

Mehryar Mohri - Foundations of Machine Learning page 24

 Implementing
Splitting Data
SVM
from sklearn.model_selection import train_test_split

cancer_data = datasets.load_breast_cancer()

X_train, X_test, y_train, y_test = train_test_split(cancer_data.data,

cancer_data.target, test_size=0.4,random_state=109)

Mehryar Mohri - Foundations of Machine Learning page 25

 Implementing
Generating The Model SVM
from sklearn import svm
#create a classifier
cls = svm.SVC(kernel="linear")
#train the model
cls.fit(X_train,y_train)
#predict the response
pred = cls.predict(X_test)

Evaluating the Model

from sklearn import metrics

#accuracy
print("acuracy:", metrics.accuracy_score(y_test,y_pred=pred))
#precision score
print("precision:", metrics.precision_score(y_test,y_pred=pred))
#recall score
print("recall" , metrics.recall_score(y_test,y_pred=pred))
print(metrics.classification_report(y_test, y_pred=pred))

Mehryar Mohri - Foundations of Machine Learning page 26

 Implementing
Output SVM

Mehryar Mohri - Foundations of Machine Learning page 27