UNIT-4

Classification Basic Concepts: Basic concepts, Decision tree induction: Decision treeinduction
Algorithm,Attribute selection measures, tree pruning. Bayes classification methods.
Classification Basic Concepts: Basic concepts
Classification is a fundamental concept in data mining and machine learning used to categorize data into
predefined classes or groups. It is a supervised learning approach, where the model is trained on a labeled
dataset to learn how to assign labels to new, unseen instances based on their features.
Basic Concepts of Classification:
1. Supervised Learning:
o Classification operates under supervised learning, where a model is trained on a dataset
containing input features (predictors) and corresponding output labels (class labels). The goal
is to predict the class labels of new data.
2. Target (Class) Variable:
o The target or dependent variable is a categorical variable that indicates the class or category to
which each data instance belongs. For example, in a customer churn dataset, the target variable
might be "Churn" with classes like "Yes" or "No."
3. Features (Attributes):
o The independent variables or features are the predictors used to determine the class. For
instance, customer characteristics like age, income, and account duration can be features in a
classification model.
4. Training Phase:
o The model is trained using a dataset where both the input features and the target labels are
known. During training, the algorithm identifies patterns and relationships between the features
and the target class.
5. Test Phase:
o After the model is trained, its performance is tested on new, unseen data. The goal is to assess
how well the model generalizes and predicts the correct class labels for new instances.
6. Decision Boundary:
o A classification model tries to find the optimal decision boundary that separates different
classes based on the features. This boundary can be linear or more complex, depending on the
classification algorithm.
7. Classification Algorithms:
o Various algorithms are used for classification, including:
 Decision Trees: Build tree structures where each node represents a decision based on a
feature.
 Naive Bayes: A probabilistic classifier based on Bayes' theorem.
 K-Nearest Neighbors (KNN): Classifies instances based on the class of the nearest
neighbors.
 Support Vector Machines (SVM): Finds the hyperplane that best separates classes.
  Logistic Regression: Used for binary classification, where the output is the probability
of belonging to a certain class.
 Neural Networks: Model complex relationships using layers of interconnected nodes.
8. Evaluation Metrics:
o The performance of classification models is evaluated using various metrics:
 Accuracy: The percentage of correct predictions out of all predictions.
 Precision and Recall: Precision is the fraction of relevant instances among the
retrieved instances, while recall is the fraction of relevant instances retrieved.
 F1 Score: The harmonic mean of precision and recall.
 Confusion Matrix: A matrix that shows the true positives, false positives, true
negatives, and false negatives.
9. Overfitting and Underfitting:
o Overfitting: When the model learns noise or details specific to the training data, making it
perform poorly on new data.
o Underfitting: When the model is too simple and fails to capture the underlying patterns in the
data.
10. Applications of Classification:
o Classification has broad applications, such as:
 Spam email detection (Spam vs. Not Spam)
 Credit scoring (Good Risk vs. Bad Risk)
 Medical diagnosis (Disease vs. No Disease)
 Image recognition (Cat vs. Dog)
Classification is a key technique in predictive modeling, helping to categorize data points into meaningful
groups based on input features
Decision Tree Induction Algorithm
The Decision Tree Induction Algorithm follows these steps:
1. Start with the entire dataset at the root node.
2. Select the best attribute to split the data at each node using a selection criterion (e.g., Information
Gain, Gini Index).
3. Split the data into subsets based on the selected attribute.
4. Repeat recursively for each subset, treating each as a new node, until:
o All instances belong to the same class.
o No further splitting improves classification.
o Some other stopping condition is met (e.g., tree depth limit or minimum node size).
5. Assign a class label to each leaf node based on the majority class in the subset at that node.
Example of Decision Tree Induction
Assume we have a dataset with attributes like "Weather" and "Temperature," and we want to predict whether
people will play tennis. The target class is "PlayTennis" (Yes/No).
1. Start with the entire dataset at the root.
2. Check if all records have the same value for "PlayTennis." If yes, make a leaf with that class. If no,
continue.
3. Choose the best attribute to split on (e.g., "Weather" based on Information Gain).
4. Split the dataset based on "Weather" (e.g., Sunny, Rainy, Overcast).
5. Recursively apply the process to the subsets created from each "Weather" value.
6. Assign "PlayTennis" = Yes or No to the final leaf nodes based on the majority class in those subsets.
Attribute Selection Measures
Attribute selection measures (ASMs) help identify the best feature to split the data at each node. Common
ASMs include:
1. Information Gain (IG):
o Based on the concept of entropy from information theory.
o Measures how much uncertainty (or entropy) is reduced by a split.
o The attribute with the highest information gain is chosen for the split.

Where:
o S is the current dataset.
o A is the attribute being considered.
o Sv is the subset of S for a particular value of A.
2. Gini Index:
o A measure of impurity used in CART (Classification and Regression Trees).
o A Gini index of 0 indicates pure subsets (all instances belong to the same class), while a higher
value indicates more impurity.
o The attribute with the lowest Gini index is selected.
Gini(S)=1-∑ (𝑃𝑖 )2
Where pi is the probability of class i in subset S.
3. Gain Ratio:
o Addresses the bias of information gain towards attributes with many distinct values.
o Normalizes the information gain by the intrinsic information of the split.
o The attribute with the highest gain ratio is chosen.
𝐼𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 𝐺𝑎𝑖𝑛
Gain Ratio = 𝐼𝑛𝑡𝑟𝑖𝑛𝑖𝑐 𝑣𝑎𝑙𝑢𝑒
 Tree Pruning
Pruning is used to simplify decision trees by removing parts of the tree that may cause overfitting (i.e.,
capturing noise in the data). It helps improve the tree's ability to generalize to unseen data.
1. Pre-Pruning (Early Stopping):
o Stop the tree-building process early, based on certain conditions (e.g., maximum depth,
minimum number of samples per node).
o Prevents the tree from growing too complex during the initial building phase.
2. Post-Pruning:
o Grow the full tree, then prune back by removing branches that add little value.
o Pruning is based on evaluating performance on a validation set or using statistical tests to
remove non-significant branches.
Types of Post-Pruning:
 Reduced Error Pruning: Simplify the tree by removing branches that do not reduce error on a
validation dataset.
 Cost-Complexity Pruning: Reduce the complexity of the tree by balancing the depth of the tree with
its performance.
Decision Tree Induction Algorithm:
Input: Training dataset, feature set
Output: Decision tree
1. If all data points belong to the same class, return a leaf node with that class.
2. If no features are left to split, return a leaf node with the majority class.
3. Otherwise:
a. Select the best feature using an attribute selection measure (e.g., information gain, Gini index).
b. Split the dataset based on the selected feature.
c. Create a child node for each split.
d. Recursively call the algorithm on each child node.
4. Return the decision tree.
Advantages and Disadvantages of Decision Tree Induction
Advantages:
 Simple and Intuitive: Easy to interpret and visualize, especially for small trees.
 Handles both Numerical and Categorical Data: Decision trees can work with different types of data.
 No Need for Normalization: Decision trees don’t require the data to be scaled or normalized.
Disadvantages:
 Prone to Overfitting: Decision trees can become overly complex, fitting noise in the data.
 Unstable: Small changes in the data can result in significantly different trees.
  Bias Toward Features with More Levels: Splits based on attributes with many distinct values (like
ID numbers) can dominate the tree unless measures like gain ratio are used
BAYESIAN CLASSIFICATION METHODS
Bayes classification methods play a significant role in data mining, leveraging probabilistic reasoning to
classify data based on statistical principles. They are especially useful for tasks like spam detection, sentiment
analysis, and document classification. Here’s an overview of the key Bayes classification methods used in
data mining:
1. Naive Bayes Classifier
The Naive Bayes Classifier is the most widely used Bayes classification method. It applies Bayes’ theorem
with the assumption that the features are conditionally independent given the class label. Despite its simplicity,
it performs surprisingly well in various applications, particularly in text classification.
Key Characteristics:
 Independence Assumption: Assumes all features are independent, which simplifies the calculation of
the posterior probabilities.
 Types:
o Gaussian Naive Bayes: Assumes that continuous features follow a Gaussian distribution. Used
for datasets with continuous numerical features.

o Multinomial Naive Bayes: Suitable for discrete data, especially for text classification. It
models the frequency of words in documents.

o Bernoulli Naive Bayes: Works with binary/boolean features, focusing on the presence or
absence of features.

Use Cases:
 Spam detection in emails
 Sentiment analysis in reviews
 Document classification and categorization
2. Gaussian Naive Bayes
Gaussian Naive Bayes is a specific instance of the Naive Bayes classifier that is particularly effective when
the features are continuous and assumed to be normally distributed.
Key Characteristics:
 Uses the probability density function of the Gaussian distribution to model the likelihood of continuous
features.
 Computes the mean and variance for each feature in each class during training.
Use Cases:
 Classification problems in domains where feature values are continuous (e.g., medical diagnosis, stock
price prediction).
3. Multinomial Naive Bayes
Multinomial Naive Bayes is particularly effective for text classification tasks where features represent word
counts or frequencies. This method assumes that features follow a multinomial distribution, making it suitable
for scenarios where the data is represented as counts.
Key Characteristics:
 Calculates probabilities based on the frequency of words or tokens.
 Handles the sparsity of data well, especially in high-dimensional feature spaces.
Use Cases:
 Text categorization (e.g., news articles, product reviews)
 Sentiment analysis of social media posts
4. Bernoulli Naive Bayes
Bernoulli Naive Bayes is a variant that is suitable for binary/boolean features. It assumes that features are
binary indicators (i.e., a feature is either present or absent).
Key Characteristics:
 Focuses on whether certain features exist in the instance.
 Useful when the frequency of occurrences is not as important as the presence or absence of features.
Use Cases:
 Document classification where features represent the presence or absence of specific words
 Spam filtering, where emails are classified based on the presence of certain keywords
5. Bayesian Networks
Bayesian Networks are directed acyclic graphs that represent a set of variables and their conditional
dependencies via a directed graph. While not strictly a classification method, they can be used for classification
tasks by modeling the joint probability distribution of the features and the class.
Key Characteristics:
 Allows for modeling complex relationships between variables, capturing dependencies that are not
assumed to be independent.
 Provides a way to reason about uncertainty in the data.
Use Cases:
 Medical diagnosis, where symptoms and diseases are interconnected
 Risk assessment in finance and insurance
6. Bayesian Model Averaging (BMA)
Bayesian Model Averaging is a statistical technique that accounts for model uncertainty by averaging over
multiple models instead of selecting a single best model. It uses the posterior probabilities of different models
to make predictions.
Key Characteristics:
 Combines predictions from multiple models, weighing them by their posterior probabilities.
 Can improve predictive performance by considering model uncertainty.
Use Cases:
 Any classification problem where model uncertainty is significant, such as in complex datasets with
many features.
Advantages of Bayes Classification Methods in Data Mining
1. Simple and Efficient: Bayes classification methods are computationally efficient and easy to
implement, making them suitable for large datasets.
2. Good Performance with Small Data: They can perform well even with limited training data, as they
rely on probabilistic reasoning rather than large amounts of training data.
3. Scalability: Bayes classifiers can handle high-dimensional data effectively, which is crucial in fields
like text mining.
4. Robustness to Irrelevant Features: The independence assumption helps mitigate the impact of
irrelevant features, which may not significantly affect the outcome.
Disadvantages
1. Independence Assumption: The naive assumption that all features are independent given the class
label can lead to poor performance if this assumption is violated.
2. Zero Probability Problem: If a feature does not appear in the training set for a particular class, it can
lead to zero probabilities, which can be mitigated through techniques like Laplace smoothing.
3. Limited Expressiveness: Bayes classifiers may not capture complex relationships between features,
which can limit their performance on certain tasks.
Conclusion
Bayes classification methods are powerful tools in data mining, particularly for tasks involving text
classification, spam detection, and sentiment analysis. Their probabilistic nature, coupled with various
adaptations (like Naive Bayes, Gaussian Naive Bayes, and Multinomial Naive Bayes), allows them to perform
well in diverse applications, making them essential in the toolkit of data scientists and machine learning
practitioners