Scribd
Upload
182 views

A Introduction To SVM PDF

This document provides an overview of support vector machines (SVMs) for data classification. It begins with an introduction to machine learning and data classification problems. It then covers high-level concepts of SVMs, including maximizing the margin between classes, mapping data to higher dimensions for nonlinear separation, kernels, soft margins, and support vectors. The document also discusses interpreting SVM models and results, such as using decision values and weights. It provides a case study example and discusses resources for using SVM software libraries like LIBSVM.

Uploaded by

Hau Nguyen
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
182 views

A Introduction To SVM PDF

This document provides an overview of support vector machines (SVMs) for data classification. It begins with an introduction to machine learning and data classification problems. It then covers high-level concepts of SVMs, including maximizing the margin between classes, mapping data to higher dimensions for nonlinear separation, kernels, soft margins, and support vectors. The document also discusses interpreting SVM models and results, such as using decision values and weights. It provides a case study example and discusses resources for using SVM software libraries like LIBSVM.

Uploaded by

Hau Nguyen
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 48

Practical Guide to Support

Vector Machines
Tingfan Wu
MPLAB, UCSD

Outline

Data Classification
High-level Concepts of SVM
Interpretation of SVM Model/Result
Use Case Study

What does it mean to learn?


Acquire new skills?

Make predictions about the world?

Making predictions is
fundamental to survival
Will that bear eat me?

Is there water in that canyon?

Is that person a good mate?

These are all examples of classification problems

Boot Camp Related


Motion classification

face recognition / speaker identification

Brain Computer Interface / Spikes Classification

Driver Fatigue Detection from Facial Expression

Data Classification
Sensor

Data
Preprocessing
features

Classifier
SVM
Adaboost
Neural Network

Prediction

Given training data (class labels known)


Predicts test data (class labels unknown)
Not just fitting generalization

Generalization

Many possible classification models


Which one generalize better ?
8

Generalization

Why SVM ? (my opinion)


With careful data preprocessing, and properly
use of SVM or NN similar performance.
SVM is easier to use properly.

SVM provides a reasonable good baseline


performance.

10

Outline

Data Classification
High-level Concepts of SVM
Interpretation of SVM Model/Result
Use case study

11

A Simple Dilemma

Who do I invite to my
birthday party?

12

Problem Formulation
training data as vectors: xi
binary labels [ +1, -1]
Name

Gift?

Income

Fondness

John
Mary

Yes
No

3k
5k

3/5
1/5

class

feature vector

y1=+1 x1 = [3000, 0.6]


y2= -1 x2 = [5000, 0.2]

13

x2
(Disposable Income)

Vector space

+ +
+

No Gift

+ Gift

+
+ ++
+
+

x1(Fondness)
14

A Line
The line : w T x

+ b= 0

x2(second feature)

Normal: w

++
+
+
+
+ ++
+
+
+
x1(first feature)

Hyperplane in high dimensional space

15

The inequalities and regions


T

w x + b= 0
+ + wT x + b> 0
i
- +
+
+
+
+
+
wT xi + b < 0
+
+
+
-

model

D eci si on funct i on f ( x ) = si gn( w T x n ew + b)

16

Large Margin

17

Maximal Margin

18

Data not linearly separable

Case 1

Case 2

19

Trick 1: Soft-Margin
These points are usually outliers. The hyperplane should not bias too much.

Penalty of
violating data

20

Soft-margin

21

[Ben-Hur & Weston 2005]

Support vectors

More important data that support (define) the hyperplane

22

Trick2: Map to Higher Dimension


x2

x 2 = x 21

x2

x 2 = x 21

M appi ng:
x1

x 21

23

Mapping to Infinite Dimension


Is it possible to create a universal mapping ?
What if we can map to infinite dimension ? Every problem is separable!
Consider Radial Basis Function (RBF):

=Kernel(x,y)

w : infinite number of variables!

24

Dual Problem
Primal

Dual

finite calculation

25

Gaussian/RBF Kernel

~ linear kernel

Overfitting
nearest neighbor?

26

27

[Ben-Hur & Weston 2005]

Recap

Soft-ness

Nonlinearity

28

Checkout the SVMToy


http://www.csie.ntu.edu.tw/~cjlin/libsvm/
-c (cost control softness of the margin/#SV)
-g (gamma controls the curvature of the
hyperplane)

29

Cross Validation
What is the best (C, ) ? Date dependent
Need to be determined by testing performance
Split training data into pseudo training, testing sets
Training

Split: training

Testing

Split: test

determine the best (C, )

Exhausted grid search for best (C, )

30

Outline

Machine Learning Classification


High-level Concepts of SVM
Interpretation of SVM Model/Result
Use Case Study

31

(1)Decision value as strength


D eci si on funct i on f ( x ) = si gn( w T x n ew + b)

32

Facial Movement Classification


Classes: brow up(+) or down(-)
Features: pixels of Gabor filtered image

33

Decision value as strength

Probability estimates from decision values also available

34

(2)Weight as feature importance


Magnitude of weight : feature importance
Similar to regression

35

(3)Weights as profiles
Fluorescent image of cells of
various dosage of certain drug
Various image-based features

Clustering the weights shows the


primal and secondary effect of the
drug
37

Outline

Machine Learning Classification


High-level Concepts of SVM
Interpretation of SVM Model/Result
User Case Study

38

The Software
SVM requires an constraint quadratic
optimization solver
not easy to implement.
Off-the-shelf Software
libsvm by Chih-Jen Lin et. al.
svmlight by Thorsten Joachims

Incorporated into many ML software


matlab / pyML / R
39

Beginners may
1. Convert their data into the format of a
SVM software.
2. May not conduct scaling
3. Randomly try few parameters and
without cross validation
4. Good result on training data, but poor in
testing.
40

Data scaling
Without scaling
feature of large dynamic range may dominate
separating hyperplane.

X Height Gender
x1 150 2

y2=1
y3=1

x2 180
x3 185

1
1

Gender

label
y1=0

Height
41

Parameter Selection
Contour of cross validation accuracy.

Good area

42

User case : Astroparticle scientist


User:
I am using libsvm in a astroparticle physics
application .. First, let me congratulate you to
a really easy to use and nice package.
Unfortunately, it gives me astonishingly bad
test results...
OK. Please send us your data
We are able to get 97% test accuracy. Is that
good enough for you ?
User:
You earned a copy of my PhD thesis
43

Dynamic Range Mismatch


A problem from astroparticle physics
<label> <index>:<value> <index>:<value>
1 1:2.6173e+01 2:5.88670e+01 3:-1.89469e-01 4:1.25122e+02
1 1:5.7073e+01 2:2.21404e+02 3:8.60795e-02 4:1.22911e+02
1 1:1.7259e+01 2:1.73436e+02 3:-1.29805e-01 4:1.25031e+02
1 1:2.1779e+01 2:1.24953e+02 3:1.53885e-01 4:1.52715e+02
1 1:9.1339e+01 2:2.93569e+02 3:1.42391e-01 4:1.60540e+02
1 1:5.5375e+01 2:1.79222e+02 3:1.65495e-01 4:1.11227e+02
1 1:2.9562e+01 2:1.91357e+02 3:9.90143e-02 4:1.03407e+02

#Training set 3,089 and #testing set 4,000


Large dynamic range of some features.

44

Overfitting
Training
$./svm-train train.1 (default parameter used)
optimization finished, #iter = 6131
nSV = 3053, nBSV = 724
Total nSV = 3053
Training Accuracy
$./svm-predict train.1 train.1.model o
Accuracy = 99.7734% (3082/3089)
Testing Accuracy
$./svm-predict test.1 train.1.model test.1.out
Accuracy = 66.925% (2677/4000)
nSV and nBSV: number of SVs and bounded SVs (i = C).
Without scaling. One feature may dominant the value overfitting

3053/3089 training data become support vectorOverfitting


Training accuracy high, but low testing accuracy Overfitting
45

Suggested Procedure
Data pre-scaling
scale range [0 1] or unit variance

Using (default) Gaussian(RBF) kernel


Use cross-validation to find the best parameter (C, )
Train your model with best parameter
Test!

All above done automatically in easy.py script provided with libsvm.

46

Large Scale SVM


(#training data >> #feature ) and linear kernel
Use primal solvers (eg. liblinear)
To approximated result in short time
Allow inaccurate stopping condition
svm-train e 0.01
Use stochastic gradient descent solvers

24
47

Resources

LIBSVM: http://www.csie.ntu.edu.tw/~cjlin/libsvm
LIBSVM Tools: http://www.csie.ntu.edu.tw/~cjlin/libsvmtools
Kernel Machines Forum: http://www.kernel-machines.org
Hsu, Chang, and Lin: A Practical Guide to Suppor t Vector
Classification
my email: [email protected]

Acknowledgement
Many slides from Dr. Chih-Jen Lin , NTU

48

You might also like