27/02/2016

ChooseaClassifierMATLAB&Simulink

ChooseaClassifier
ChooseaClassifierType
InClassificationLearner,youcanexploreseveraltypesofclassifiers.Toseeallavailableclassifieroptions,
ontheClassificationLearnertab,clickthearrowonthefarrightoftheClassifiersectiontoexpandthelistof
classifiers.TheoptionsintheClassifiergalleryarepresetstartingpointswithdifferentsettings,suitablefora
rangeofdifferentclassificationproblems.
Forhelpchoosingaclassifiertype,seethetableshowingtypicalcharacteristicsofdifferentsupervised
learningalgorithms.Usethetableasaguideforyourinitialchoiceofalgorithms.Decideonthetradeoffyou
wantinspeed,memoryusage,flexibility,andinterpretability.Thebestclassifiertypedependsonyourdata.
TipTryadecisiontreeordiscriminantfirst,becausetheseclassifiersarefastandeasytointerpret.Ifthemodels
arenotaccurateenoughpredictingtheresponse,tryotherclassifierswithhigherflexibility.
Tocontrolflexibility,seethedetailsforeachclassifiertype.Toavoidoverfitting,lookforamodeloflowerflexibility
thatprovidessufficientaccuracy.

ChooseaClassifierType
Classifier

PredictionSpeed

MemoryUsage

Interpretability

DecisionTrees

Fast

Small

Easy

DiscriminantAnalysis

Fast

Smallforlinear,large
forquadratic

Easy

SupportVectorMachines

Mediumforlinear.
Slowforothers.

Mediumforlinear.
Allothers:medium
formulticlass,large
forbinary.

EasyforLinearSVM.
Hardforallotherkernel
types.

NearestNeighborClassifiers

Slowforcubic.
Mediumforothers.

Medium

Hard

EnsembleClassifiers

Fasttomediumdepending
onchoiceofalgorithm

Lowtohigh
dependingonchoice
ofalgorithm.

Hard

Thetablesonthispagedescribegeneralcharacteristicsofspeedandmemoryusageforallthepreset
classifiers.Theclassifiersweretestedwithvariousdatasets(upto7000observations,80predictors,and50
classes),andtheresultsdefinethefollowinggroups:
Speed

Fast0.01second

Medium1second

Slow100seconds

Memory

Small1MB

Medium4MB

Large100MB

Thesetablesprovideageneralguide.Yourresultsdependonyourdataandthespeedofyourmachine.
http://www.mathworks.com/help/stats/chooseaclassifier.html

1/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

ToreadadescriptionofeachclassifierinClassificationLearner,switchtothedetailsview.

TipAfteryouchooseaclassifiertype(e.g.,decisiontrees),trytrainingusingeachoftheclassifiers.Theoptionsin
theClassifiergalleryarestartingpointswithdifferentsettings.Trythemalltoseewhichoptionproducesthebest
modelwithyourdata.

Forworkflowinstructions,seeExploreClassificationModelsInteractively.
CategoricalPredictorSupport
InClassificationLearner,theclassifiergalleryonlyshowsclassifiertypesthatsupportyourselecteddata.
Classifier

Allpredictorsnumeric

Allpredictorscategorical

Somecategorical,some
numeric

DecisionTrees

Yes

Yes

Yes

DiscriminantAnalysis

Yes

No

No

SVM

Yes

Yes

Yes

NearestNeighbour

Euclideandistanceonly

Hammingdistanceonly

No

Ensembles

Yes

Yes,exceptSubspace
Discriminant

Yes,exceptanySubspace

DecisionTrees
Decisiontreesareeasytointerpret,fastforfittingandprediction,andlowonmemoryusage,buttheycan
havelowpredictiveaccuracy.Trytogrowsimplertreestopreventoverfitting.Controlthedepthwiththe
Maximumnumberofsplitssetting.
TipModelflexibilityincreaseswiththeMaximumnumberofsplitssetting.
PredictionSpeed
http://www.mathworks.com/help/stats/chooseaclassifier.html

ModelFlexibility
2/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

ClassifierType

MemoryUsage

Interpretability

SimpleTree

Fast

Small

Easy

Low.
Fewleavestomake
coarsedistinctions
betweenclasses
(maximumnumberof
splitsis4).

MediumTree

Fast

Small

Easy

Medium
Mediumnumberof
leavesforfiner
distinctionsbetween
classes(maximum
numberofsplitsis
20).

ComplexTree

Fast

Small

Easy

High
Manyleavestomake
manyfinedistinctions
betweenclasses
(maximumnumberof
splitsis100).

TipTrytrainingeachofthedecisiontreeoptionsintheClassifiergallery.Trainthemalltoseewhichsettings
producethebestmodelwithyourdata.SelectthebestmodelintheHistorylist.Totrytoimproveyourmodel,try
featureselection,andthentrychangingsomeadvancedoptions.

Youtrainclassificationtreestopredictresponsestodata.Topredictaresponse,followthedecisionsinthe
treefromtheroot(beginning)nodedowntoaleafnode.Theleafnodecontainstheresponse.Statisticsand
MachineLearningToolboxtreesarebinary.Eachstepinapredictioninvolvescheckingthevalueofone
predictor(variable).Forexample,hereisasimpleclassificationtree:

Thistreepredictsclassificationsbasedontwopredictors,x1andx2.Topredict,startatthetopnode.Ateach
decision,checkthevaluesofthepredictorstodecidewhichbranchtofollow.Whenthebranchesreachaleaf
node,thedataisclassifiedeitherastype0or1.
Youcanvisualizeyourdecisiontreemodelbyexportingthemodelfromtheapp,andthenentering:
view(trainedClassifier.ClassificationTree,'Mode','graph')
Thefigureshowsanexamplecomplextreetrainedwiththefisheririsdata.

http://www.mathworks.com/help/stats/chooseaclassifier.html

3/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

TipForanexample,seeExploreDecisionTreesInteractively.

AdvancedTreeOptions
ClassificationtreesinClassificationLearnerusethefitctreefunction.Youcansettheseoptions:

Maximumnumberofsplits
Specifythemaximumnumberofsplitsorbranchpointstocontrolthedepthofyourtree.Whenyougrowa
decisiontree,consideritssimplicityandpredictivepower.Tochangethenumberofsplits,clickthebuttons
orenterapositiveintegervalueintheMaximumnumberofsplitsbox.

Acomplextreewithmanyleavesisusuallyhighlyaccurateonthetrainingdata.However,thetree
mightnotshowcomparableaccuracyonanindependenttestset.Aleafytreetendstoovertrain,andits
validationaccuracyisoftenfarlowerthanitstraining(orresubstitution)accuracy.

Incontrast,asimpletreedoesnotattainhightrainingaccuracy.Butasimpletreecanbemorerobustin
thatitstrainingaccuracycanapproachthatofarepresentativetestset.Also,asimpletreeiseasyto
interpret.

Splitcriterion
Specifythesplitcriterionmeasurefordecidingwhentosplitnodes.Tryeachofthethreesettingstoseeif
theyimprovethemodelwithyourdata.
SplitcriterionoptionsareGini'sdiversityindex,Twoingrule,orMaximumdeviancereduction
(alsoknownascrossentropy).
Theclassificationtreetriestooptimizetopurenodescontainingonlyoneclass.Gini'sdiversityindex(the
default)andthedeviancecriterionmeasurenodeimpurity.Thetwoingruleisadifferentmeasurefor
decidinghowtosplitanode,wheremaximizingthetwoingruleexpressionincreasesnodepurity.
Fordetailsofthesesplitcriteria,seeClassificationTreeDefinitions.

SurrogatedecisionsplitsOnlyformissingdata.

http://www.mathworks.com/help/stats/chooseaclassifier.html

4/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

Specifysurrogateusefordecisionsplits.Ifyouhavedatawithmissingvalues,usesurrogatesplitsto
improvetheaccuracyofpredictions.
WhenyousetSurrogatedecisionsplitstoOn,theclassificationtreefindsatmost10surrogatesplitsat
eachbranchnode.Tochangethenumber,clickthebuttonsorenterapositiveintegervalueinthe
Maximumsurrogatespernodebox.
WhenyousetSurrogatedecisionsplitstoFindAll,theclassificationtreefindsallsurrogatesplitsat
eachbranchnode.TheFindAllsettingcanuseconsiderabletimeandmemory.

DiscriminantAnalysis
Discriminantanalysisisapopularfirstclassificationalgorithmtotrybecauseitisfast,accurateandeasyto
interpret.Discriminantanalysisisgoodforwidedatasets.
DiscriminantanalysisassumesthatdifferentclassesgeneratedatabasedondifferentGaussiandistributions.
Totrainaclassifier,thefittingfunctionestimatestheparametersofaGaussiandistributionforeachclass.
ClassifierType

PredictionSpeed

MemoryUsage

Interpretability

ModelFlexibility

LinearDiscriminant

Fast

Small

Easy

Low.
Createslinear
boundariesbetween
classes.

Quadratic
Discriminant

Fast

Large

Easy

Low.
Createsnonlinear
boundariesbetween
classes(ellipse,
parabolaor
hyperbola).

AdvancedDiscriminantOptions
DiscriminantanalysisinClassificationLearnerusesthefitcdiscrfunction.Foreitherlinearorquadratic
discriminants,youcanchangetheRegularizationoption.Ifyouchangetheregularizationoption,trainingcan
failifyouhavepredictorswithzerovarianceoranyofthecovariancematricesofyourpredictorsaresingular.

Lineardiscriminants:Ifyourpredictorsareindependentorifyourdatasetissmall,thenleavethedefault
DiagonalCovariance.Ifyourpredictorsaredependent,tryselectingtheAutooptiontosetregularization
automaticallyandseeifthatimprovesyourmodel.Iftrainingfails,useDiagonalCovarianceinstead.

Quadraticdiscriminants:YoucantryselectingNonetoremoveregularizationandseeifthatimprovesyour
model.TheNoneoptioncancausetrainingtofailifanyofthecovariancematricesofyourpredictorsare
singular.Tryremovingpredictors,selecttheDiagonalCovarianceoptioninstead,ortryanother
classifier.

SupportVectorMachines
InClassificationLearner,youcantrainSVMswhenyourdatahastwoormoreclasses.
TipModelflexibilitydecreaseswiththeKernelscalesetting.

ClassifierType

Prediction
Speed

MemoryUsage

Interpretability ModelFlexibility

LinearSVM

Binary:Fast

Medium

Easy

Low
Makesasimplelinearseparationbetween
classes.

Binary:Medium

Hard

Medium

Multiclass:
Medium
QuadraticSVM

Binary:Fast

Multiclass:Large
http://www.mathworks.com/help/stats/chooseaclassifier.html

5/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

Multiclass:
Slow
CubicSVM

Binary:Fast

Binary:Medium

Multiclass:
Slow

Multiclass:Large

FineGaussian
SVM

Binary:Fast

Binary:Medium

Multiclass:
Slow

Multiclass:Large

Medium
GaussianSVM

Binary:Fast

Binary:Medium

Multiclass:
Slow

Multiclass:Large

Coarse
GaussianSVM

Binary:Fast

Binary:Medium

Multiclass:
Slow

Multiclass:Large

Hard

Medium

Hard

HighdecreaseswithKernelscale
setting.
Makesfinelydetaileddistinctionsbetween
classes,withkernelscalesetto
sqrt(P)/4.

Hard

Medium
Mediumdistinctions,withkernelscaleset
tosqrt(P).

Hard

Low
Makescoarsedistinctionsbetween
classes,withkernelscalesetto
sqrt(P)*4,wherePisthenumberof
predictors.

TipTrytrainingeachofthesupportvectormachineoptionsintheClassifiergallery.Trainthemalltoseewhich
settingsproducethebestmodelwithyourdata.SelectthebestmodelintheHistorylist.Totrytoimproveyour
model,tryfeatureselection,andthentrychangingsomeadvancedoptions.

AnSVMclassifiesdatabyfindingthebesthyperplanethatseparatesdatapointsofoneclassfromthoseof
theotherclass.ThebesthyperplaneforanSVMmeanstheonewiththelargestmarginbetweenthetwo
classes.Marginmeansthemaximalwidthoftheslabparalleltothehyperplanethathasnointeriordatapoints.
Thesupportvectorsarethedatapointsthatareclosesttotheseparatinghyperplanethesepointsareonthe
boundaryoftheslab.Thefollowingfigureillustratesthesedefinitions,with+indicatingdatapointsoftype1,
andindicatingdatapointsoftype1.

SVMscanalsouseasoftmargin,meaningahyperplanethatseparatesmany,butnotalldatapoints.
Foranexample,seeExploreSupportVectorMachinesInteractively.
AdvancedSVMOptions
Ifyouhaveexactlytwoclasses,ClassificationLearnerusesthefitcsvmfunctiontotraintheclassifier.Ifyou
havemorethantwoclasses,theappusesthefitcecocfunctiontoreducethemulticlassclassification
problemtoasetofbinaryclassificationsubproblems,withoneSVMlearnerforeachsubproblem.Toexamine

http://www.mathworks.com/help/stats/chooseaclassifier.html

6/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

thecodeforthebinaryandmulticlassclassifiertypes,youcangeneratecodefromyourtrainedclassifiersin
theapp.
Youcansettheseoptionsintheapp:

Kernelfunction
SpecifytheKernelfunctiontocomputetheGrammatrix.

Linearkernel,easiesttointerpret

GaussianorRadialBasisFunction(RBF)kernel

Quadratic

Cubic

Boxconstraintlevel
SpecifytheboxconstrainttokeeptheallowablevaluesoftheLagrangemultipliersinabox,abounded
region.
TotuneyourSVMclassifier,tryincreasingtheboxconstraintlevel.Clickthebuttonsorenterapositive
scalarvalueintheBoxconstraintlevelbox.Increasingtheboxconstraintlevelcandecreasethenumber
ofsupportvectors,butalsocanincreasetrainingtime.
TheBoxConstraintparameteristhesoftmarginpenaltyknownasCintheprimalequations,andisahard
"box"constraintinthedualequations.

Kernelscalemode
Specifymanualkernelscalingifdesired.
WhenyousetKernelscalemodetoAuto,thenthesoftwareusesaheuristicproceduretoselectthescale
value.Theheuristicprocedureusessubsampling.Therefore,toreproduceresults,setarandomnumber
seedusingrngbeforetrainingtheclassifier.
WhenyousetKernelscalemodetoManual,youcanspecifyavalue.Clickthebuttonsorenterapositive
scalarvalueintheManualkernelscalebox.Thesoftwaredividesallelementsofthepredictormatrixby
thevalueofthekernelscale.Then,thesoftwareappliestheappropriatekernelnormtocomputetheGram
matrix.

Multiclassmethod
Onlyfordatawith3ormoreclasses.Thismethodreducesthemulticlassclassificationproblemtoasetof
binaryclassificationsubproblems,withoneSVMlearnerforeachsubproblem.OnevsOnetrainsone
learnerforeachpairofclasses.Itlearnstodistinguishoneclassfromtheother.OnevsAlltrainsone
learnerforeachclass.Itlearnstodistinguishoneclassfromallothers.

Standardizedata
Specifywhethertoscaleeachcoordinatedistance.Ifpredictorshavewidelydifferentscales,standardizing
canimprovethefit.

NearestNeighborClassifiers
Nearestneighborclassifierstypicallyhavegoodpredictiveaccuracyinlowdimensions,butmightnotinhigh
dimensions.Theyhavehighmemoryusage,andarenoteasytointerpret.
TipModelflexibilitydecreaseswiththeNumberofneighborssetting.
ClassifierType

PredictionSpeed

MemoryUsage

Interpretability

ModelFlexibility

FineKNN

Medium

Medium

Hard

Finelydetailed
distinctionsbetween
classes.Thenumber
ofneighborsissetto
1.

MediumKNN

Medium

Medium

Hard

http://www.mathworks.com/help/stats/chooseaclassifier.html

7/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

Mediumdistinctions
betweenclasses.
Thenumberof
neighborsissetto
10.
CoarseKNN

Medium

Medium

Hard

Coarsedistinctions
betweenclasses.
Thenumberof
neighborsissetto
100.

CosineKNN

Medium

Medium

Hard

Mediumdistinctions
betweenclasses,
usingaCosine
distancemetric.The
numberofneighbors
issetto10.

CubicKNN

Slow

Medium

Hard

Mediumdistinctions
betweenclasses,
usingacubic
distancemetric.The
numberofneighbors
issetto10.

WeightedKNN

Medium

Medium

Hard

Mediumdistinctions
betweenclasses,
usingadistance
weight.Thenumber
ofneighborsissetto
10.

TipTrytrainingeachofthenearestneighboroptionsintheClassifiergallery.Trainthemalltoseewhichsettings
producethebestmodelwithyourdata.SelectthebestmodelintheHistorylist.Totrytoimproveyourmodel,try
featureselection,andthen(optionally)trychangingsomeadvancedoptions.

WhatiskNearestNeighborclassification?Categorizingquerypointsbasedontheirdistancetopoints(or
neighbours)inatrainingdatasetcanbeasimpleyeteffectivewayofclassifyingnewpoints.Youcanuse
variousmetricstodeterminethedistance.GivenasetXofnpointsandadistancefunction,knearest
neighbor(kNN)searchletsyoufindthekclosestpointsinXtoaquerypointorsetofpoints.kNNbased
algorithmsarewidelyusedasbenchmarkmachinelearningrules.

http://www.mathworks.com/help/stats/chooseaclassifier.html

8/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

Foranexample,seeExploreNearestNeighborClassificationInteractively.
AdvancedKNNOptions
NearestNeighborclassifiersinClassificationLearnerusethefitcknnfunction.Youcansettheseoptions:

Numberofneighbors
Specifythenumberofnearestneighborstofindforclassifyingeachpointwhenpredicting.Specifyafine
(lownumber)orcoarseclassifier(highnumber)bychangingthenumberofneighbors.Forexample,afine
KNNusesoneneighbor,andacoarseKNNuses100.Manyneighborscanbetimeconsumingtofit.

Distancemetric
Youcanusevariousmetricstodeterminethedistancetopoints.Fordefinitions,seetheclass
ClassificationKNN.

Distanceweight
Specifythedistanceweightingfunction.YoucanchooseEqual(noweights),Inverse(weightis
1/distance),orSquaredInverse(weightis1/distance2).

Standardizedata
Specifywhethertoscaleeachcoordinatedistance.Ifpredictorshavewidelydifferentscales,standardizing
canimprovethefit.

EnsembleClassifiers
Ensembleclassifiersmeldresultsfrommanyweaklearnersintoonehighqualityensemblepredictor.Qualities
dependonthechoiceofalgorithm.
TipModelflexibilityincreaseswiththeNumberoflearnerssetting.
Allensembleclassifierstendtobeslowtofitbecausetheyoftenneedmanylearners.

ClassifierType

Prediction
Speed

Memory
Usage

Interpretability

EnsembleMethod

ModelFlexibility

BoostedTrees

Fast

Low

Hard

AdaBoost,with
DecisionTree
learners

Mediumtohigh
increaseswithNumberof
learnersorMaximum
numberofsplitssetting.

http://www.mathworks.com/help/stats/chooseaclassifier.html

9/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

TipBoostedtreescan
usuallydobetterthan
bagged,butmight
requireparametertuning
andmorelearners

BaggedTrees

Medium

High

Hard

Randomforest
Bag,withDecision
Treelearners

Highincreaseswith
Numberoflearners
setting.
TipTrythisclassifier
first.

Subspace
Discriminant

Medium

Low

Hard

Subspace,with
Discriminantlearners

Mediumincreaseswith
Numberoflearners
setting.
Goodformanypredictors

SubspaceKNN

Medium

Medium

Hard

Subspace,withNearest Mediumincreaseswith
Neighborlearners
Numberoflearners
setting.
Goodformanypredictors

RUSBoostTrees

Fast

Low

Hard

RUSBoost,with
DecisionTree
learners

Mediumincreaseswith
Numberoflearnersor
Maximumnumberof
splitssetting.
Goodforskeweddata
(withmanymore
observationsof1class)

GentleBoostor
LogitBoost
notavailablein
Classifier
gallery.
Ifyouhave2
classdata,
selectmanually.

Fast

Low

Hard

GentleBoostor
LogitBoost,with
DecisionTree
learners
ChooseBoosted
Treesandchangeto
GentleBoostmethod.

Mediumincreaseswith
Numberoflearnersor
Maximumnumberof
splitssetting.
Forbinaryclassification
only

BaggedtreesuseBreiman's'randomforest'algorithm.Forreference,seeBreiman,L.RandomForests.
MachineLearning45,pp.532,2001.
Tips

Trybaggedtreesfirst.Boostedtreescanusuallydobetterbutmightrequiresearchingmanyparameter
values,whichistimeconsuming.

TrytrainingeachoftheensembleclassifieroptionsintheClassifiergallery.Trainthemalltoseewhich
settingsproducethebestmodelwithyourdata.SelectthebestmodelintheHistorylist.Totrytoimprove
yourmodel,tryfeatureselection,PCA,andthen(optionally)trychangingsomeadvancedoptions.

Forboostingensemblemethods,youcangetfinedetailwitheitherdeepertreesorlargernumbersof
shallowtrees.Aswithsingletreeclassifiers,deeptreescancauseoverfitting.Youneedtoexperimentto
choosethebesttreedepthforthetreesintheensemble,inordertotradeoffdatafitwithtreecomplexity.
UsetheNumberoflearnersandMaximumnumberofsplitssettings.

Foranexample,seeExploreEnsembleClassificationInteractively.
AdvancedEnsembleOptions
http://www.mathworks.com/help/stats/chooseaclassifier.html

10/11

27/02/2016

ChooseaClassifierMATLAB&Simulink

EnsembleclassifiersinClassificationLearnerusethefitensemblefunction.Youcansettheseoptions:

ForhelpchoosingEnsemblemethodandLearnertype,seetheEnsembletable.Trythepresetsfirst.

Maximumnumberofsplits
Forboostingensemblemethods,specifythemaximumnumberofsplitsorbranchpointstocontrolthe
depthofyourtreelearners.Manybranchestendtooverfit,andsimplertreescanbemorerobustandeasy
tointerpret.Experimenttochoosethebesttreedepthforthetreesintheensemble.

Numberoflearners
Trychangingthenumberoflearnerstoseeifyoucanimprovethemodel.Manylearnerscanproducehigh
accuracy,butcanbetimeconsumingtofit.Startwithafewdozenlearners,andtheninspectthe
performance.Anensemblewithgoodpredictivepowercanneedafewhundredlearners.

Learningrate
Specifythelearningrateforshrinkage.Ifyousetthelearningratetolessthan1,theensemblerequires
morelearningiterationsbutoftenachievesbetteraccuracy.0.1isapopularchoice.

Subspacedimension
Forsubspaceensembles,specifythenumberofpredictorstosampleineachlearner.Theappchoosesa
randomsubsetofthepredictorsforeachlearner.Thesubsetschosenbydifferentlearnersareindependent.

Fornextstepstrainingmodels,seeExploreClassificationModelsInteractively.

RelatedExamples

ExploreClassificationModelsInteractively

SelectDataandValidationforClassificationProblem

FeatureSelectionandFeatureTransformation

AssessClassifierPerformance

ExportClassificationModeltoPredictNewData

ExploreDecisionTreesInteractively

http://www.mathworks.com/help/stats/chooseaclassifier.html

11/11