Continuous Probability Distributions for Machine Learning

Machine learning relies heavily on probability distributions because they offer a framework for comprehending the uncertainty and variability present in data. Specifically, for a given dataset, continuous probability distributions express the chance of witnessing continuous outcomes, like real numbers.

What are Continuous Probability Distributions (CPDs)?

A probability distribution is a mathematical function that describes the likelihood of different outcomes for a random variable. Continuous probability distributions (CPDs) are probability distributions that apply to continuous random variables. It describes events that can take on any value within a specific range, like the height of a person or the amount of time it takes to complete a task.

In continuous probability distributions, two key functions describe the likelihood of a variable taking on specific values:

Probability Density Function (PDF):

The PDF gives the probability density at a specific point or interval for a continuous random variable. It indicates how likely the variable is to fall within a small interval around a particular value.

• The height of the PDF curve at any point represents the probability density at that value.
• Higher density implies a higher probability of the variable taking on values around that point.

Cumulative Distribution Function (CDF):

The CDF gives the probability that a random variable is less than or equal to a specific value.It provides a cumulative view of the probability distribution, starting at 0 and increasing to 1 as the value of the random variable increases.

• The CDF starts at 0 for the smallest possible value of the random variable (since there is no probability below this value) and approaches 1 as the value approaches infinity (since the probability of the variable being less than or equal to infinity is 1).

CDF is the integral of the PDF, and the PDF is the derivative of the CDF.

Why are Continuous Probability Distribution important in machine learning?

Imagine trying to build a model to predict the price of a car. You have data on various factors like mileage, year, and brand. But how do you account for the fact that prices can vary continuously? This is where continuous distributions come to the rescue! By fitting a suitable distribution to the price data, you can estimate the probability of a car with specific features falling within a certain price range.

Types of Continuous Probability Distributions

Here are some common types used in Machine learning,

Normal Distribution (Bell Curve) or Gaussian Distribution:

The Normal Distribution, sometimes referred to as the Gaussian Distribution, is a bell-shaped, symmetrical basic continuous probability distribution. Two factors define it: the standard deviation (σ), which indicates the distribution's spread or dispersion, and the mean (μ), which establishes the distribution's

• Parameters: Mean (μ) and standard deviation (σ)

For a random variable x, it is expressed as,

f(x) =\frac{1}{\sqrt{2 \pi \sigma^2}} \exp \left( -\frac{1}{2} \left( \frac{x - \mu}{\sigma} \right)^2 \right)

Note: The shape of the Normal Distribution is such that about 68% of the values fall within one standard deviation of the mean (μ ± σ), about 95% fall within two standard deviations (μ ± 2σ), and about 99.7% fall within three standard deviations (μ ± 3σ).

Uniform Distribution:

The Uniform Distribution is a continuous probability distribution where all values within a specified range are equally likely to occur.

• Parameters: Lower bound (a) and upper bound (b).
• The mean of a uniform distribution is \mu = \frac{a + b}{2} and the variance is \sigma^2 = \frac{(b - a)^2}{12}

It is expressed as:

f(x) = \frac{1}{b - a} \quad \text{for } a \leq x \leq b

Exponential Distribution:

The exponential distribution is a continuous probability distribution that represents the duration between occurrences in a Poisson process, which occurs continuously and independently at a constant average rate.

• Parameter: Rate parameter (λ).
• The mean of the exponential distribution is 1/λ, and the variance is 1/λ^2.

For a random variable x, it is expressed as

f(x) = \lambda e^{-\lambda x} \quad \text{for } x \geq 0

Chi-Squared Distribution:

The Chi-Squared Distribution is a continuous probability distribution that arises in statistics, particularly in hypothesis testing and confidence interval estimation.

• It is characterized by a single parameter, often denoted as k ,which represents the degrees of freedom.
• The mean of the Chi-Squared Distribution is k and the variance is 2k.

For a random variable x, it is expressed as

f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} \left( \frac{x}{2} \right)^{k/2 - 1} e^{-x/2}

Determining the distribution of a variable

Example :

Consider the iris dataset and let us try to understand how the petal length is distributed, here are the steps to be considered

Execute on jupyter notebook or any other ide that supports libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm  # loading normal distribution

# Step 1: Load the Iris dataset
url = "https://raw.githubusercontent.com/uiuc-cse/data-fa14/gh-pages/data/iris.csv"
iris_data = pd.read_csv(url)

# Step 2: Select the feature for analysis (e.g., petal length)
selected_feature = 'petal_length'
selected_data = iris_data[selected_feature]

# Step 3: Plot the histogram of the selected feature
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.hist(selected_data, bins=30, density=True, color='skyblue', alpha=0.6)
plt.title('Histogram of {}'.format(selected_feature))
plt.xlabel(selected_feature)
plt.ylabel('Density')
plt.grid(True)
# Step 4: Fit a Gaussian distribution to the selected feature
estimated_mean, estimated_std = np.mean(selected_data), np.std(selected_data)

# Step 5: Plot the histogram along with the fitted Gaussian distribution
plt.subplot(1, 2, 2)
plt.hist(selected_data, bins=30, density=True, color='skyblue', alpha=0.6)

x = np.linspace(np.min(selected_data), np.max(selected_data), 100)
pdf = norm.pdf(x, estimated_mean, estimated_std)
plt.plot(x, pdf, color='red', linestyle='--', linewidth=2)

plt.title('Histogram and Fitted Gaussian Distribution of {}'.format(selected_feature))
plt.xlabel(selected_feature)
plt.ylabel('Density')
plt.legend(['Fitted Gaussian Distribution', 'Histogram'])
plt.grid(True)

plt.tight_layout()
plt.show()

Output:

Explanation for the output:

• Histogram: The left side of the image shows a histogram, which is a graphical representation of the distribution of petal lengths. Each bar in the histogram represents a range of petal lengths, and the height of the bar shows how many petals fall within that range. The histogram appears to be roughly bell-shaped, suggesting that the petal lengths are normally distributed.
• Gaussian Distribution: The right side of the image shows a fitted Gaussian distribution, also known as a normal distribution. This is a theoretical curve that represents the probability distribution of a normally distributed variable. We see that there are two blocks of hostogram, It's possible that the data comes from two distinct populations with different petal lengths, resulting in two overlapping normal distributions. This could happen if the sample includes flowers from different varieties or grown under different conditions.

These graphs provide insights into the distribution of petal lengths in the Iris dataset and help us assess whether a Gaussian distribution is a suitable model for representing this data.