Support Vector Machine

Support Vector Machine

 SVM is a supervised algorithm which can be
used for both classification or regression
problems. It separates data by using concept
of hyperplanes.
Support Vector Machine
Support Vectors are simply the co-ordinates
of individual observation. These are data
points that are closer to the hyperplane and
influence the position and orientation of the
hyperplane.

The distance between the vectors and the

hyperplane is called as margin.

The main objective is to maximize this

margin. The hyperplane with maximum
margin is called the optimal hyperplane.
 Support Vector Machine

.
SVM takes the training data as an input and
outputs a line/hyperplane that separates
those classes if possible.

For a space of n dimensions, we have a

hyperplane of n-1 dimensions separating it
into two or more parts.
 Types of SVM

Linear SVM: Linear SVM is used for linearly separable data,

which means if a dataset can be classified into two classes by
using a single straight line, then such data is termed as
.linearly separable data, and classifier is used called as Linear
SVM classifier.

Non-linear SVM: Non-Linear SVM is used for non-linearly

separated data, which means if a dataset cannot be classified
by using a straight line, then such data is termed as non-
linear data and classifier used is called as Non-linear SVM
classifier.
Support Vector Machine
Support Vector Machine
 Support Vector Machine

 This data is simple to classify and one can see

that the data is clearly separated into two
segments.
 Any line that separates the red and blue items
can be used to classify the above data.
 Support Vector Machine

 Had this data been multi-dimensional data,

any plane can separate and successfully
classify the data.
 However, we want to find the “most optimal”
solution. What will then be the characteristic
of this most optimal line?
 Support Vector Machine

 We have to remember that this is just the

training data and we can have more data
points which can lie anywhere in the
subspace. If our line is too close to any of the
data points, noisy test data is more likely to
get classified in a wrong segment.
 We have to choose the line which lies
between these groups and is at the farthest
distance from each of the segments.
 Support Vector Machine

 To solve this classifier line, we draw the line

as y=ax+b and make it equidistant from the
respective data points closest to the line. So
we want to maximize the margin (m).
Support Vector Machine
 Support Vector Machine

How does it work?

The process of segregating the two classes with
a hyper-plane.

“How can we identify the right hyper-plane?”.

 Support Vector Machine

Identify the right hyper-plane (Scenario-

1): Here, we have three hyper-planes (A, B and
C). Now, identify the right hyper-plane to classify
star and circle.
Support Vector Machine
 Support Vector Machine

Identify the right hyper-plane (Scenario-

2): Here, we have three hyper-planes (A, B and
C) and all are segregating the classes well. Now,
How can we identify the right hyper-plane?
Support Vector Machine
 Support Vector Machine

Here, maximizing the distances between nearest

data point (either class) and hyper-plane
will help us to decide the right hyper-plane. This
distance is called as Margin. Let’s look at the
below snapshot:
Support Vector Machine
 Support Vector Machine

 You can see that the margin for hyper-plane C

is high as compared to both A and B. Hence,
we name the hyper-plane C as best.
 Another lightning reason for selecting the
hyper-plane with higher margin is robustness.
If we select a hyper-plane having low margin
then there is high chance of miss-
classification.
 Support Vector Machine

Identify the right hyper-plane (Scenario-

3):Hint: Use the rules as discussed in previous
section to identify the right hyper-plane
Support Vector Machine
 Support Vector Machine

 Some of you may have selected the hyper-

plane B as it has higher margin compared
to A. But, here is the catch, SVM selects the
hyper-plane which classifies the classes
accurately prior to maximizing margin. Here,
hyper-plane B has a classification error and A
has classified all correctly. Therefore, the right
hyper-plane is A.
Support Vector Machine
 Support Vector Machine

 One star at other end is like an outlier for star

class. SVM has a feature to ignore outliers
and find the hyper-plane that has maximum
margin. Hence, we can say, SVM is robust to
outliers.
Support Vector Machine
 Support Vector Machine

 Find the hyper-plane to segregate to

classes (Scenario-5): In this scenario, we
can’t have linear hyper-plane between the
two classes, so how does SVM classify these
two classes? (Till now, we have only looked at
the linear hyper-plane)
Support Vector Machine
 Support Vector Machine

 SVM can solve this problem. Easily! It solves

this problem by introducing additional feature.
Here, we will add a new feature z=x^2+y^2.
Now, let’s plot the data points on axis x and z
Support Vector Machine
 Support Vector Machine

In this plot, points to be considered are:

 All values for z would be positive always
because z is the squared sum of both x and y.

 In the original plot, red circles appear close to

the origin of x and y axes, leading to lower
value of z and star relatively away from the
origin result to higher value of z.
 Support Vector Machine
 SVM, the function which converts non-separable
problem to separable problem, is called kernel.
 It is mostly useful in non-linear separation
problem.
 It does some extremely complex data
transformations to separate the data based on
the labels or outputs.
 SVM Implementation
import pandas as pd
import numpy as np
from sklearn.model_selection import
train_test_split
bankdata = pd.read_csv("D:/Machine
Learning/Lab/bill_authentication.csv")
print(bankdata.shape)
print(bankdata.head())
X = bankdata.drop('Class', axis=1)
y = bankdata['Class']
34
X_train, X_test, y_train, y_test =
train_test_split(X, y, test_size = 0.20)

print(X_train.shape)
from sklearn.svm import SVC

svclassifier = SVC(kernel='linear')

svclassifier.fit(X_train, y_train)

35
y_pred = svclassifier.predict(X_test)
from sklearn.metrics import
classification_report, confusion_matrix

print(confusion_matrix(y_test,y_pred))

print(classification_report(y_test,y_pred))

36
 SVM Kernels

1. Polynomial Kernel
In the case of polynomial kernel, you also
have to pass a value for the degree parameter
of the SVC class. This basically is the degree
of the polynomial.

svclassifier = SVC(kernel='poly', degree=8)

37
2. Gaussian Kernel
Take a look at how we can use polynomial
kernel to implement kernel SVM:

svclassifier = SVC(kernel=‘rbf')

Radial basis function

38
 SVM Kernels

Sigmoid Kernel

svclassifier = SVC(kernel=‘sigmoid')

39
 Parameters Tuning
Most of the machine learning and deep learning algorithms
have some parameters that can be adjusted which are
called hyperparameters.
 We need to set hyperparameters before we train the
models. Hyperparameters are very critical in building
robust and accurate models.
 They help us find the balance between bias and variance
and thus, prevent the model from overfitting or
underfitting.

40
 Consider the data points in 2D space as

Result: Model is not generalized and overfit.

41
To overcome this issue, in 1995, Cortes and Vapnik, came up with the
idea of “soft margin” SVM which allows some examples to be
misclassified or be on the wrong side of decision boundary.

Soft margin SVM often result in a better generalized model.

42
When determining the decision boundary, a soft margin
SVM tries to solve an optimization problem with the
following:
 Increase the distance of decision boundary to classes (or
support vectors)
 Maximize the number of points that are correctly classified
in the training set

43
There is a trade-off between these two goals.

Decision boundary might have to be very close to one

particular class to correctly label all data points in training set.
However, in this case, accuracy on test dataset might be
lower because decision boundary is too sensitive to noise and
to small changes in the independent variables.

On the other hand, a decision boundary might be placed as

far as possible to each class with the expense of some
misclassified exceptions. This trade-off is controlled by c
parameter.

44
C parameter adds a penalty for each misclassified data
point.
 If c is small, the penalty for misclassified points is low so a
decision boundary with a large margin is chosen at the
expense of a greater number of misclassifications.

 If c is large, SVM tries to minimize the number of

misclassified examples due to high penalty which results in
a decision boundary with a smaller margin.

 Penalty is not same for all misclassified examples. It is

directly proportional to the distance to decision boundary.

45
 Parameters Tuning
Tuning parameters value for machine learning
algorithms effectively improves the model
performance.
 kernel:Here, we have various options
available with kernel like, “linear”,
“rbf”,”poly” and others (default value is
“rbf”). Here “rbf” and “poly” are useful for
non-linear hyper-plane.

46
 Programming Example

# import some data to play with

iris = datasets.load_iris()
X = iris.data[:, :2]
# we only take the first two features.
y = iris.target

C = 1.0 # SVM regularization parameter

svc = svm.SVC(kernel='linear', C=1,gamma=0).fit(X, y)

47
48
Change the kernel type to rbf

svc = svm.SVC(kernel='rbf', C=1,gamma=0).fit(X, y)

49
gamma: Kernel coefficient for ‘rbf’, ‘poly’ and
‘sigmoid’. Higher the value of gamma, will try to
exact fit the model as per training data set i.e.
generalization error and cause over-fitting
problem.

50
C: Penalty parameter C of the error term. It also
controls the trade off between smooth decision
boundary and classifying the training points
correctly.

51
 Gamma vs C parameter
 For a linear kernel, we just need to optimize the c
parameter.
 However, if we want to use an RBF kernel, both c and
gamma parameter need to optimized simultaneously.
 If gamma is large, the effect of c becomes negligible.
 If gamma is small, c affects the model just like how it
affects a linear model.

52