Bayes Classification

Bayes Classification is a Supervised machine learning approach for classification. It works on a probabilistic method which uses Bayes Theorem to be implemented. It predicts the data point label or assigns the class on the basis of heuristic and statistical data. Let's dive deep into Bayes Classification and its Differences from Naive Bayes Classification.

The above image denotes the relationship representation in a Bayes Classifier. The nodes represent features or variables in a Bayesian network. These can either be discrete or continuous. The links, on the other hand, denote the relationships between the nodes, along with a probability distribution over variables.

Key Features of Bayes Classifier

1. Probabilistic Model: Based on conditional probability using Bayes' Theorem.
2. Supervised Learning Approach: Requires labeled training data.
3. Classifies Using Prior and Likelihood: Combines prior class probabilities with data likelihood.
4. Model Interpretability: Easy to interpret due to mathematical formulation.
5. Scalability: Efficient for larger datasets.
6. Handles Continuous/Discrete Features: Works with both types depending on implementation.

Mathematical Representation

Bayes Theorem

Bayes’ Theorem is a fundamental theorem in probability and machine learning that describes how to update the probability of an event when given new evidence. It is used as the basis of Bayes Classification.

P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}

Where:

• P(C∣X): Posterior probability of class C given data X
• P(X∣C): Likelihood of data X given class C
• P(C): Prior probability of class C
• P(X): Marginal probability of data X

Assumptions in Bayes Classification

1. Well-defined prior probabilities.
2. Correct conditional probability distributions.
3. Independence of training and test samples.
4. Stationarity of features over time.
5. Classes are mutually exclusive and exhaustive.

Bayes Classification Workflow

Terminologies

• Prior: Initial belief before evidence (e.g., % of spam emails). It is an Input to the Classifier.
• Likelihood: Evidence corresponding to a particular given class (e.g., frequency of “free” in spam). It is an Input to the Classifier.
• Posterior: Updated belief after evidence. It is an Output from the Classifier.
• Evidence: Overall probability of observed features.

The above image demonstrates that Likelihood, Data, and Prior probabilities are used as Input to the model. Bayes Theorem is used as the mathematical principle. The resultant is Posterior Distribution.

Steps Involved in Classification

1. Collect Data: From training set, extract features X and target C.
2. Estimate Priors: Calculate P(C) for each class.
3. Estimate Likelihoods: For every feature X, compute P(X∣C).
4. Calculate Posterior: Use Bayes’ Theorem to get P(C∣X).
5. Predict Class: Assign the class with the highest posterior probability. It calculates the posterior probability for each class and assigns the class with the highest probability.

Bayes Classifier vs. Naive Bayes Classifier

Bayes Classifier is often misunderstood by its simplified version Naive Bayes Classifier. Here are some differences between them:

Why is it called a Bayes Classifier?

It's based on Bayes’ Theorem, named after Thomas Bayes, an 18th-century statistician. The theorem helps update beliefs based on evidence, which is the core idea of classification here: updating class probability based on observed data.

Applications of Bayes Classification

• Stock Market Prediction: Analyzes time-varying relationships and Financial indicators.
• Fraud Detection and Credit Risk Modelling: Analyzes Fraud probability based on transaction patterns, contextual data, timestamps, etc.
• Medical Diagnosis: Predicts the chances or probability of occurrence of a disease based on medical history.
• Email Spam Detection and Phishing: Classifies emails and messages as "Spam" or "Not Spam" based on previous probabilities.

Advantages of Bayes Classifier

1. Handles small data well.
2. Can incorporate domain knowledge.
3. Probabilistic output allows threshold tuning.
4. Can model non-linear decision boundaries.
5. Adaptable to both discrete and continuous data.

Disadvantages of Bayes Classifier

1. Computationally expensive for high-dimensional data.
2. Complex to model feature dependencies.
3. Requires large data to estimate joint distributions.
4. Not scalable for many features or classes.
5. Poor performance if assumptions are violated.