Decision Trees.

Random Forests.
 Definition
• A tree-like model that illustrates series of events leading to certain decisions
• Each node represents a test on an attribute and each branch is an outcome of
that test
Who to loan?

• Not a student
• 45 years old
• Medium income
• Fair credit record

• Student
• 27 years old
• Low income
• Excellent credit record
 Definition
• A tree-like model that illustrates series of events leading to certain decisions
• Each node represents a test on an attribute and each branch is an outcome of
that test
Who to loan?

• Not a student
• 45 years old
• Medium income
• Fair credit record
 Yes

• Student
• 27 years old
• Low income
• Excellent credit record
 Definition
• A tree-like model that illustrates series of events leading to certain decisions
• Each node represents a test on an attribute and each branch is an outcome of
that test
Who to loan?

• Not a student
• 45 years old
• Medium income
• Fair credit record
 Yes

• Student
• 27 years old
• Low income
• Excellent credit record
 No
 Decision Tree Learning
• We use labeled data to obtain a suitable decision tree for future predictions
 We want a decision tree that works well on unseen data, while asking as few
questions as possible
 Decision Tree Learning
• Basic step: choose an attribute and, based on its values, split the data into
smaller sets
 Recursively repeat this step until we can surely decide the label
 Decision Tree Learning
• Basic step: choose an attribute and, based on its values, split the data into
smaller sets
 Recursively repeat this step until we can surely decide the label
 Decision Tree Learning
• Basic step: choose an attribute and, based on its values, split the data into
smaller sets
 Recursively repeat this step until we can surely decide the label
 Decision Tree Learning
• Basic step: choose an attribute and, based on its values, split the data into
smaller sets
 Recursively repeat this step until we can surely decide the label
 Decision Tree Learning
• Basic step: choose an attribute and, based on its values, split the data into
smaller sets
 Recursively repeat this step until we can surely decide the label
Decision Tree Learning (Python)
def make_tree(X):
node = TreeNode(X)
if should_be_leaf_node(X):
node.label = majority_label(X)
else:
a = select_best_splitting_attribute(X)
for v in values(a):
𝑣 ={ | [ ] == }
node.children.append(make_tree( 𝑣))
return node
 What is a good attribute?

• Which attribute provides better splitting?

• Why?
 Because the resulting subsets are more pure
 Knowing the value of this attribute gives us more information about the label
(the entropy of the subsets is lower)
Information Gain
 Entropy
• Entropy measures the degree of randomness in data

• For a set of samples with classes:

where 𝑖 is the proportion of elements of class

• Lower entropy implies greater predictability!

 Information Gain
• The information gain of an attribute a is the expected
reduction in entropy due to splitting on values of a:

where is the subset of for which

Best attribute = highest information gain

mammal 2 mammal bird 2 bird 2 2

Best attribute = highest information gain

mammal 2 mammal bird 2 bird 2 2

2 2
Best attribute = highest information gain

mammal 2 mammal bird 2 bird 2 2

2 2
Best attribute = highest information gain

mammal 2 mammal bird 2 bird 2 2

2 2
Best attribute = highest information gain

mammal 2 mammal bird 2 bird 2 2

2 2
Best attribute = highest information gain

mammal 2 mammal bird 2 bird 2 2

2 2

2 2
 Best attribute = highest information gain
In practice, we compute 𝑒𝑛𝑡𝑟𝑜𝑝𝑦(𝑋) only once!

mammal 2 mammal bird 2 bird 2 2

2 2

2 2
 ID3 Algorithm (Python)
# ID = Iterative Dichotomiser
def ID3(X):
node = TreeNode(X)
if all_points_have_same_class(X):
node.label = majority_label(X)
else:
a = select_attribute_with_highest_information_gain(X)
if gain(X, a) == 0:
node.label = majority_label(X)
else:
for v in values(a):
𝑣
node.children.append(ID3( 𝑣 ))
return node
Gini Impurity
 Gini Impurity
• Gini impurity measures how often a randomly chosen example would be
incorrectly labeled if it was randomly labeled according to the label distribution

• For a set of samples with classes:

where 𝑖 is the proportion of elements of class

• Can be used as an alternative to entropy for selecting attributes!

 Best attribute = highest impurity decrease
In practice, we compute 𝑔𝑖𝑛𝑖(𝑋) only once!
 Entropy versus Gini Impurity
• Entropy and Gini Impurity give similar results in practice
 They only disagree in about 2% of cases
“Theoretical Comparison between the Gini Index and Information Gain
Criteria” [Răileanu & Stoffel, AMAI 2004]
 Entropy might be slower to compute, because of the log
Pruning
 Pruning
• Pruning is a technique that reduces the size of a decision tree by
removing branches of the tree which provide little predictive power
• It is a regularization method that reduces the complexity of the
final model, thus reducing overfitting
 Decision trees are prone to overfitting!

• Pruning methods:
 Pre-pruning: Stop the tree building algorithm before it fully
classifies the data
 Post-pruning: Build the complete tree, then replace some non-
leaf nodes with leaf nodes if this improves validation error
 Pre-pruning Minimum
𝜽𝒆𝒏𝒕 = 𝟎. 𝟒 threshold
• Pre-pruning implies early stopping: on entropy
 If some condition is met, the current node will
not be split, even if it is not 100% pure
 It will become a leaf node with the label of the
majority class in the current set
(the class distribution could be used as prediction
𝑒𝑛𝑡
confidence)
Stop, even if node
can be split
• Common stopping criteria include setting a
threshold on:
 Entropy (or Gini Impurity) of the current set
 Number of samples in the current set
 Gain of the best-splitting attribute
 Depth of the tree
91.52% confidence
 Post-pruning

• Prune nodes in a bottom-up manner, if it

decreases validation error
 Post-pruning

• Prune nodes in a bottom-up manner, if it

decreases validation error
 Post-pruning

• Prune nodes in a bottom-up manner, if it

decreases validation error
Handling Missing Values
Handling missing values at training time
• Data sets might have samples with missing values for some
attributes
• Simply ignoring them would mean throwing away a lot of
information
• There are better ways of handling missing values:
Handling missing values at training time
• Data sets might have samples with missing values for some
attributes
• Simply ignoring them would mean throwing away a lot of
information
• There are better ways of handling missing values:
 Set them to the most common value
 Handling missing values at training time
• Data sets might have samples with missing values for some
attributes
• Simply ignoring them would mean throwing away a lot of
information
• There are better ways of handling missing values:
 Set them to the most common value
 Set them to the most probable value given the label
2
𝑃 𝑌𝑒𝑠 𝐵𝑖𝑟𝑑 = = 0.66
3
1
𝑃 𝑁𝑜 𝐵𝑖𝑟𝑑 = = 0.33
3
𝑃 𝑊ℎ𝑖𝑡𝑒 𝑀𝑎𝑚𝑚𝑎𝑙 = 1

𝑃 𝐵𝑟𝑜𝑤𝑛 𝑀𝑎𝑚𝑚𝑎𝑙 = 0
Handling missing values at training time
• Data sets might have samples with missing values for some
attributes
• Simply ignoring them would mean throwing away a lot of
information
• There are better ways of handling missing values:
 Set them to the most common value
 Set them to the most probable value given the label
 Add a new instance for each possible value
 Handling missing values at training time
• Data sets might have samples with missing values for some
attributes
• Simply ignoring them would mean throwing away a lot of
information
• There are better ways of handling missing values:
 Set them to the most common value
 Set them to the most probable value given the label
entropy(𝑋 )=0  Add a new instance for each possible value
entropy(𝑋 )=1  Leave them unknown, but discard the sample when
evaluating the gain of that attribute
gain 𝑋 𝑐𝑜𝑙𝑜𝑟 = 0.985 − 0− 1 (if the attribute is chosen for splitting, send the instances
= 0.318 with unknown values to all children)
Handling missing values at training time
• Data sets might have samples with missing values for some
attributes
• Simply ignoring them would mean throwing away a lot of
information
• There are better ways of handling missing values:
 Set them to the most common value
 Set them to the most probable value given the label
 Add a new instance for each possible value
 Leave them unknown, but discard the sample when
evaluating the gain of that attribute
(if the attribute is chosen for splitting, send the instances
with unknown values to all children)
 Build a decision tree on all other attributes (including label)
to predict missing values
(use instances where the attribute is defined as training data)
 Handling missing values at inference time
• When we encounter a node that checks an attribute with a missing value, we
explore all possibilities

Loan?
• Not a student
• 49 years old
• Unknown income
• Fair credit record
 Handling missing values at inference time
• When we encounter a node that checks an attribute with a missing value, we
explore all possibilities
• We explore all branches and take the final prediction based on a (weighted)
vote of the corresponding leaf nodes

Loan?
• Not a student
• 49 years old
• Unknown income
• Fair credit record
 Yes
 Decision Boundaries
• Decision trees produce non-linear decision boundaries

Support Vector Machines Decision Tree

Decision Trees: Training and Inference

Training

Inference
 History of Decision Trees
• The first regression tree algorithm
 “Automatic Interaction Detection (AID)” [Morgan & Sonquist, 1963]
• The first classification tree algorithm
 “Theta Automatic Interaction Detection (THAID)” [Messenger & Mandel,
1972]
• Decision trees become popular
 “Classification and regression trees (CART)” [Breiman et al., 1984]
• Introduction of the ID3 algorithm
 “Induction of Decision Trees” [Quinlan, 1986]
• Introduction of the C4.5 algorithm
 “C4.5: Programs for Machine Learning” [Quinlan, 1993]
 Summary
• Decision trees represent a tool based on a tree-like graph of decisions
and their possible outcomes
• Decision tree learning is a machine learning method that employs a
decision tree as a predictive model
• ID3 builds a decision tree by iteratively splitting the data based on the
values of an attribute with the largest information gain (decrease in
entropy)
 Using the decrease of Gini Impurity is also a commonly-used option in
practice
• C4.5 is an extension of ID3 that handles attributes with continuous
values, missing values and adds regularization by pruning branches likely
to overfit
 Random Forests
(Ensemble learning with decision trees)
 Random Forests
• Random Forests:
 Instead of building a single decision tree and use it to make
predictions, build many slightly different trees and combine their
predictions
• We have a single data set, so how do we obtain slightly different trees?
1. Bagging (Bootstrap Aggregating):
 Take random subsets of data points from the training set to create N
smaller data sets
 Fit a decision tree on each subset
2. Random Subspace Method (also known as Feature Bagging):
 Fit N different decision trees by constraining each one to operate on a
random subset of features
 Bagging at training time
N subsets (with
replacement)

Training set
 Bagging at inference time

A test sample

75% confidence
 Random Subspace Method at training time

Training data
Random Subspace Method at inference time

A test sample

66% confidence
 Random Forests

Tree 1 Tree 2
Random Forest Tree N
 History of Random Forests

• Introduction of the Random Subspace Method

 “Random Decision Forests” [Ho, 1995] and “The Random Subspace
Method for Constructing Decision Forests” [Ho, 1998]

• Combined the Random Subspace Method with Bagging. Introduce the

term Random Forest (a trademark of Leo Breiman and Adele Cutler)
 “Random Forests” [Breiman, 2001]
 Ensemble Learning
• Ensemble Learning:
 Method that combines multiple learning algorithms to obtain
performance improvements over its components
• Random Forests are one of the most common examples of ensemble
learning
• Other commonly-used ensemble methods:
 Bagging: multiple models on random subsets of data samples
 Random Subspace Method: multiple models on random subsets of
features
 Boosting: train models iteratively, while making the current model
focus on the mistakes of the previous ones by increasing the weight
of misclassified samples
 Boosting
All samples have
the same weight
 Boosting
All samples have
the same weight

Reweight based on
model’s mistakes
 Boosting

Next model sees

weighted samples
Boosting

Reweight based
on current
model’s mistakes
Boosting
Boosting
 Summary
• Ensemble Learning methods combine multiple learning algorithms to
obtain performance improvements over its components
• Commonly-used ensemble methods:
 Bagging (multiple models on random subsets of data samples)
 Random Subspace Method (multiple models on random subsets of
features)
 Boosting (train models iteratively, while making the current model
focus on the mistakes of the previous ones by increasing the weight
of misclassified samples)
• Random Forests are an ensemble learning method that employ decision
tree learning to build multiple trees through bagging and random
subspace method.
 They rectify the overfitting problem of decision trees!
 Decision Trees and Random Forest (Python)
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

clf = DecisionTreeClassifier(criterion = "entropy", min_samples_leaf = 3)

# Lots of parameters: criterion = "gini" / "entropy";
# max_depth;
# min_impurity_split;

clf.fit(X, y) # It can only handle numerical attributes!

# Categorical attributes need to be encoded, see LabelEncoder and OneHotEncoder

clf.predict([x]) # Predict class for x

clf.feature_importances_ # Importance of each feature

clf.tree_ # The underlying tree object

clf = RandomForestClassifier(n_estimators = 20) # Random Forest with 20 trees

 Acknowledgement
Radu Ionescu, Prof. PhD.
[email protected]
Faculty of Mathematics and Computer Science
University of Bucharest
64