Fuzzy Clustering is a type of clustering algorithm in machine learning that allows a data point to belong to more than one cluster with different degrees of membership. Unlike traditional clustering (like K-Means) where each data point belongs to only one cluster.
Imagine you're at a party and guests are casually forming groups based on shared interests like music lovers, food enthusiasts and sports fans. Some people clearly fit into one group like guitarist who only talks about music. But others might belong to multiple groups like a person who enjoys both music and food could be partially in both groups instead of being forced into just one.
If you're new to clustering, you may want to check out our Introduction to Clustering in Machine Learning
How Does Fuzzy Clustering Work?
Fuzzy clustering follows an iterative optimization process where data points are assigned membership values instead of hard cluster labels. Here’s a step-by-step breakdown of how it works:
Step 1: Initialize Membership Values Randomly
- Each data point is assigned a membership degree for all clusters. These values indicate the probability of the data point belonging to each cluster. Unlike hard clustering where a point strictly belongs to one cluster fuzzy clustering allows partial membership.
- Let us assume there are 2 clusters in which the data is to be divided, initializing the data point randomly. Each data point lies in both clusters with some membership value which can be assumed anything in the initial state.
The table below represents the values of the data points along with their membership (gamma) in each cluster.
Cluster (1, 3) (2, 5) (4, 8) (7, 9)
1) 0.8 0.7 0.2 0.1
2) 0.2 0.3 0.8 0.9
Step 2: Compute Cluster Centroids
- The centroids of the clusters are calculated based on the weighted sum of all data points where weights are determined by membership values. This ensures that points with higher membership contribute more to the centroid.
- The formula for finding out the centroid (V) is:
V_{ij} = ( \sum \limits_1^n ( \gamma_{ik}^m * x_k) / \sum \limits_1^n \gamma_{ik}^m
Where
- µ is fuzzy membership value of the data point
- m is the fuzziness parameter (generally taken as 2)
- x_k is the data point.
Here
V11 = (0.8^2 *1 + 0.7^2 * 2 + 0.2^2 * 4 + 0.1^2 * 7) / ( (0.8^2 + 0.7^2 + 0.2^2 + 0.1^2 ) = 1.568
V12 = (0.8^2 *3 + 0.7^2 * 5 + 0.2^2 * 8 + 0.1^2 * 9) / ( (0.8^2 + 0.7^2 + 0.2^2 + 0.1^2 ) = 4.051
V21 = (0.2^2 *1 + 0.3^2 * 2 + 0.8^2 * 4 + 0.9^2 * 7) / ( (0.2^2 + 0.3^2 + 0.8^2 + 0.9^2 ) = 5.35
V22 = (0.2^2 *3 + 0.3^2 * 5 + 0.8^2 * 8 + 0.9^2 * 9) / ( (0.2^2 + 0.3^2 + 0.8^2 + 0.9^2 ) = 8.215
Centroids are: (1.568, 4.051) and (5.35, 8.215)
Step 3: Calculate Distance Between Data Points and Centroids
- The Euclidean distance or any another distance metric between each data point and the centroids is computed. This helps in updating the membership values.
D11 = ((1 - 1.568)2 + (3 - 4.051)2)0.5 = 1.2
D12 = ((1 - 5.35)2 + (3 - 8.215)2)0.5 = 6.79
Similarly the distance of all other points is computed from both the centroids.
Step 4: Update Membership Values
The membership values are recalculated based on how close a point is to a centroid relative to the other centroids. The farther a point is from a centroid, the lower its membership value for that cluster.
\gamma = \sum \limits_1^n {(d_{ki}^2 /d_{kj}^2)}^{1/m-1} ]^{-1}
For point 1 new membership values are:
- \gamma_{11} = [{ [(1.2)2 / (1.2)2] + [(1.2)2 / (6.79)2]} ^ {(1 / (2 - 1))} ] -1 = 0.96
- \gamma_{12} = [{ [(6.79)2 / (6.79)2] + [(6.79)2 / (1.2)2]} ^ {(1 / (2 - 1))} ] -1 = 0.04
Alternatively
- \gamma_{12} = 1- \gamma_{11} = 0.04
Similarly compute all other membership values and update the matrix.
Step 5: Repeat Until Convergence
Steps 2–4 are repeated until the membership values stabilize meaning there are no significant changes from one iteration to the next. This indicates that the clustering has reached an optimal state.
Step 6: Defuzzification (Optional)
In some cases we may want to convert fuzzy memberships into crisp cluster assignments by assigning each data point to the cluster where it has the highest membership
Implementation of Fuzzy Clustering in Python
The fuzzy scikit learn library has a pre-defined function for fuzzy c-means which can be used in Python. For using fuzzy c-means you need to install the skfuzzy library.
pip install sklearn
pip install scikit-fuzzy
Example :
Python
import numpy as np
import skfuzzy as fuzz
import matplotlib.pyplot as plt
np.random.seed(0)
data = np.random.rand(2, 100)
n_clusters = 3
cntr, u, _, _, _, _, fpc = fuzz.cluster.cmeans(
data, c=n_clusters, m=2, error=0.005, maxiter=1000, init=None
)
hard_clusters = np.argmax(u, axis=0)
print("Cluster Centers:\n", cntr)
print("\nFuzzy Membership Matrix (first 5 data points):\n", u[:, :5])
fig, ax = plt.subplots(figsize=(8, 6))
for i in range(n_clusters):
ax.scatter(data[0], data[1], c=u[i], alpha=0.6, label=f'Cluster {i+1}')
ax.scatter(cntr[:, 0], cntr[:, 1], c='red', marker='X', s=200, label='Cluster Centers')
ax.set_title('Fuzzy C-Means Clustering')
ax.set_xlabel('Feature 1')
ax.set_ylabel('Feature 2')
ax.legend()
plt.show()
Output:
Cluster Centers and Fuzzy Membership Matrix
Fuzzy C means Clustering The plot shows soft clustering meaning a point can belong to multiple clusters with different probabilities rather than being assigned to just one cluster. This makes it useful when boundaries between clusters are not well-defined and all the Red "X" markers indicate the cluster centers computed by the algorithm.
Advantages of Fuzzy Clustering
- Flexibility: It allows for overlapping clusters which can be useful when the data has a complex structure or when there are ambiguous or overlapping class boundaries.
- Robustness: It can be more robust to outliers and noise in the data as it allows for a more gradual transition from one cluster to another.
- Interpretability: It provides better understanding of the structure of the data as it allows for a more detailed representation of the relationships between data points and clusters.
Disadvantages of Fuzzy Clustering
- Complexity: It can be computationally more expensive than traditional clustering algorithms as they require optimization over multiple membership degrees.
- Model selection: Choosing the right number of clusters and membership functions can be challenging and may require expert knowledge or trial and error.
Similar Reads
ML | Fuzzy Clustering
Fuzzy Clustering is a type of clustering algorithm in machine learning that allows a data point to belong to more than one cluster with different degrees of membership. Unlike traditional clustering (like K-Means) where each data point belongs to only one cluster.Imagine you're at a party and guests
5 min read
Fuzzy Clustering in R
Clustering is an unsupervised machine-learning technique that is used to identify similarities and patterns within data points by grouping similar points based on their features. These points can belong to different clusters simultaneously. This method is widely used in various fields such as Custom
15+ min read
Clustering in R Programming
Clustering is an unsupervised learning technique where a dataset is divided into groups, or clusters, based on similarities among data points. It helps identify natural groupings within the data without prior labeling. Each cluster has data points that are closer to one another than to other cluster
5 min read
K means Clustering - Introduction
K-Means Clustering is an Unsupervised Machine Learning algorithm which groups unlabeled dataset into different clusters. It is used to organize data into groups based on their similarity. Understanding K-means ClusteringFor example online store uses K-Means to group customers based on purchase frequ
4 min read
ML | Active Learning
Active Learning is a special case of Supervised Machine Learning. This approach is used to construct a high-performance classifier while keeping the size of the training dataset to a minimum by actively selecting the valuable data points. Active Learning in Machine Learning A subset of machine learn
9 min read
Image Segmentation Using Fuzzy C-Means Clustering
This article delves into the process of image segmentation using Fuzzy C-Means (FCM) clustering, a powerful technique for partitioning images into meaningful regions. We'll explore the fundamentals of FCM, its advantages over traditional methods, and provide a step-by-step guide to implementing FCM
7 min read
ML | Classification vs Clustering
Prerequisite: Classification and Clustering As you have read the articles about classification and clustering, here is the difference between them. Both Classification and Clustering is used for the categorization of objects into one or more classes based on the features. They appear to be a similar
2 min read
Mean Shift Clustering using Sklearn
Clustering is a fundamental method in unsupervised device learning, and one powerful set of rules for this venture is Mean Shift clustering. Mean Shift is a technique for grouping comparable data factors into clusters primarily based on their inherent characteristics, with our previous understanding
9 min read
Using Learning Curves - ML
A learning model of a Machine Learning model shows how the error in the prediction of a Machine Learning model changes as the size of the training set increases or decreases. Before we continue, we must first understand what variance and bias mean in the Machine Learning model. Bias: It is basically
4 min read
Fuzzy Logic applications in AI
Fuzzy logic, a concept introduced by Lotfi Zadeh in the 1960s, extends classical logic to handle the concept of partial truth, where truth values range between completely true and completely false. This approach is especially useful in dealing with uncertainty and vagueness in complex systems. Fuzzy
8 min read