Scribd
Upload
121 views

Lecture 9 PDF

This document discusses random forests, an ensemble learning method for classification and regression. Random forests grow multiple decision trees with randomness, which reduces variance and helps prevent overfitting. Trees are built on bootstrap samples of the data, and at each node, the best split is chosen from a random subset of features. The predictions from all trees are averaged to improve accuracy. Out-of-bag error estimates can be used for model selection and provide an unbiased estimate of test set error without a separate test set.

Uploaded by

Sachin singh
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
121 views

Lecture 9 PDF

This document discusses random forests, an ensemble learning method for classification and regression. Random forests grow multiple decision trees with randomness, which reduces variance and helps prevent overfitting. Trees are built on bootstrap samples of the data, and at each node, the best split is chosen from a random subset of features. The predictions from all trees are averaged to improve accuracy. Out-of-bag error estimates can be used for model selection and provide an unbiased estimate of test set error without a separate test set.

Uploaded by

Sachin singh
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 28

Random Forests

Dr. Jainendra Shukla


[email protected]
B-403, R & D Block
Computer Science and Engineering
Bias/Variance Tradeoff
Reduce Variance Without Increasing Bias
● Variance: How much does prediction change if we change
the training set?

● Averaging reduces variance:

● Average models to reduce model variance


● One problem:
○ Only one training set
○ where do multiple models come from?
Ensemble learning

● Combining weak classifiers (of the same type)...


○ ... in order to produce a strong classifier
○ Condition: diversity among the weak classifiers
● Example: Boosting
○ Train each new weak classifier focusing on samples
misclassified by previous ones
○ Popular implementation: AdaBoost
● Weak classifiers: only need to be better than random
guess
Decision Trees
● Easy to
○ Build
○ Use
● Non-flexible -> Inaccuracy
○ Performance on unseen data

● Random Forests
Pruning
Random Forests
● Purpose
○ Improve prediction accuracy
● Principle
○ Encouraging diversity among the tree
● Definition
○ Collection of unpruned CARTs
○ Rule to combine individual tree decisions
● Solution: randomness
○ Bagging
○ Random decision trees (rCART)
Random Forests: Example
Bootstrapping
● L: original learning set composed of n samples
● Generate K learning sets Lk…
○ ... composed of m samples, m ≤ n,...
○ ... obtained by uniform sampling with replacement
from L
○ In consequences, Lk may contain repeated samples
0.632 Bootstrap
● m=n
● A particular training data has a probability of (1-1/n) of
not being picked
● Thus its probability of ending up in the test data (not
selected) is:

● -> Training data will contain approximately 63.2% of the


instances
● Out of Bag (OOB) samples-> Out-Of-Bag Error
Bagging
● Bagging: Bootstrap aggregation
● Technique of ensemble learning
○ To avoid over-fitting
■ Important since trees are unpruned
○ to improve stability and accuracy
● Idea
○ Create 𝑁 bootstrap samples {𝑆1, … , 𝑆𝑁} of 𝑆 as follows:
○ For each 𝑆𝑖 randomly draw |𝑆| examples from 𝑆 with
replacement
○ For each 𝑖 = 1, … , 𝑁, ℎ𝑖 = Learn(Si)
○ Output 𝐻 =<{ℎ1, … , hN}, 𝑚𝑎𝑗𝑜𝑟𝑖𝑡𝑦𝑉𝑜𝑡𝑒 >
Bagging Example -1

Goal: Find a collection of 10 simple thresholding classifiers that


collectively can classify correctly.
Bagging Example - I
Bagging Example - II
Bagging Example - III
Bagging Example - IV
Bagging Example - 2
Bagging
● Resample data points
● Weight of each classifier is the same
● Only variance reduction
● Robust to noise and outliers
Random Forests
● Ensemble method specifically designed for decision tree
classifiers
● Introduce two sources of randomness:
○ “Bagging” and “Random input vectors”
■ Bagging method: each tree is grown using a
bootstrap sample of training data
■ Random vector method: At each node, best split is
chosen from a random sample of m attributes
instead of all attributes
Method for Growing the tree
● Fix a m <= M. At each node
● Method 1
○ Choose m attributes randomly, compute their
information gains, and choose the attribute with the
largest gain to split
● Method 2: (When M is not very large):
○ Select L of the attributes randomly. Compute a linear
combination of the L attributes using weights
generated from [-1,+1] randomly. That is, new A =
Sum(Wi*Ai), i=1..L.
Method for Growing the tree
● Method 3:
○ Compute the information gain of all M attributes.
Select the top m attributes by information gain.
Randomly select one of the m attributes as the
splitting node.
The Random Forest Algorithm
Value of m
● For classification, the default value for m is ⌊√M⌋ and the
minimum node size is one.
● For regression, the default value for m is ⌊M/3⌋ and the
minimum node size is five.
Out-of-bag error

● For each sample S of the OOB set


○ Predict the class of S with OOB classifiers that does
not contain the sample S
○ Error = is prediction correct?
● Out-of-bag error = average over all samples of S
● Provides an estimation of the generalization error
○ Can be used to decide when to stop adding trees to the
forest
○ As accurate as using a test set of the same size as the
training set.
■ Therefore, using the out-of-bag error estimate
removes the need for a set aside test set.
Reducing Bias and Variance
● Bagging reduces variance by averaging
● BUT Bagging has little effect on bias
● Can we average and reduce bias?
○ Yes
● Boosting
Boosting
● Records that are wrongly classified will have their weights
increased
● Records that are classified correctly will have their
weights decreased
● Adaboost – popular boosting algorithm
References
1. The Elements of Statistical Learning, Chapter 15
2. https://perso.math.univ-toulouse.fr/motimo/files/2013/07/random-for
est.pdf
3. https://stat.ethz.ch/education/semesters/ss2012/ams/slides/v10.2.pdf
4. StatQuest: Random Forests Part 1 - Building, Using and Evaluating
5. Look at Part 2 also.

You might also like