Hierarchical Clustering in Machine Learning

Clustering is an unsupervised learning technique that organizes data

based on its resemblance to other data sets. There are numerous types of
clustering methods in machine learning.

 Connectivity-based clustering: This type of clustering algorithm

creates a cluster based on the connection of data points. Examples
include hierarchical clustering.
 Centroid-based clustering: This clustering algorithm clusters
data points around their centroids. Examples include K-Means and K-
Mode clustering.
 Distribution-based clustering: Statistical distributions are used
to model this clustering process. It assumes that the data points in a
cluster are created from a particular probability distribution, and the
method seeks to estimate the parameters of the distribution in
order to group comparable data points into clusters. Example:
Gaussian Mixture Models (GMM)
 Density-based clustering: With this kind of clustering method,
data points in high-density concentrations are grouped together,
while points in low-density concentrations are separated. The main
concept is that it finds high density data point locations in the data
space and clusters those points together. DBSCAN (Density-Based
Spatial Clustering of Applications with Noise) is one example.

Hierarchical clustering

A connectivity-based clustering methodology called hierarchical clustering

puts nearby data points in groups according to similarity or distance. Data
points that are closer together are thought to be more similar or
connected than those that are farther apart.

The hierarchical links between groups are shown by a dendrogram, a tree-

like picture created via hierarchical clustering. The dendrogram shows
that the largest clusters, which contain all of the data points, are at the
top, while individual data points are at the bottom. It is possible to slice
the dendrogram at different heights to produce varying numbers of
clusters.
 Dendogram

What is a Dendrogram?

A tree diagram showing the arrangement of clusters produced by

hierarchical clustering.

 Vertical lines represent the merging of clusters.

 Horizontal lines indicate the distance between clusters.
 The height at which clusters merge can guide in determining the
optimal number of clusters.

Clusters are iteratively merged or broken according to a distance or

similarity metric between data points to form the dendrogram. Until every
data point is contained in a single cluster or until the target number of
clusters is reached, clusters are split or combined again.

To determine the ideal number of clusters, we can examine the

dendrogram and assess the height at which the branches form distinct
clusters. At this height, the dendrogram can be cut to find the number of
clusters.
Types of Hierarchical Clustering

Basically, there are two types of hierarchical Clustering

1. Agglomerative Clustering
2. Divisive clustering

Hierarchical Agglomerative Clustering

It is sometimes referred to as hierarchical agglomerative clustering (HAC)

or the bottom-up technique. An organized set of clusters that yields more
information than the unorganized group obtained from flat clustering. The
number of clusters does not need to be predetermined when using this
clustering procedure. From the beginning, bottom-up algorithms treat
each data set as a singleton cluster. They subsequently group pairs of
clusters together until all of the clusters are combined into a single cluster
that contains all of the data.

Algorithm :

given a dataset (d1, d2, d3, ....dN) of size N

# compute the distance matrix

for i=1 to N:

# as the distance matrix is symmetric about

# the primary diagonal so we compute only lower

# part of the primary diagonal

for j=1 to i:

dis_mat[i][j] = distance[di, dj]

each data point is a singleton cluster

repeat

merge the two cluster having minimum distance

update the distance matrix

until only a single cluster remains

Hierarchical Agglomerative Clustering
Steps:

 Consider each alphabet as a single cluster and calculate the

distance of one cluster from all the other clusters.
 In the second step, comparable clusters are merged together to
form a single cluster. Let’s say cluster (B) and cluster (C) are very
similar to each other therefore we merge them in the second step
similarly to cluster (D) and (E) and at last, we get the clusters [(A),
(BC), (DE), (F)]
 We recalculate the proximity according to the algorithm and merge
the two nearest clusters([(DE), (F)]) together to form new clusters as
[(A), (BC), (DEF)]
 Repeating the same process; The clusters DEF and BC are
comparable and merged together to form a new cluster. We’re now
left with clusters [(A), (BCDEF)].
 At last, the two remaining clusters are merged together to form a
single cluster [(ABCDEF)].

Python implementation of the above algorithm using the scikit-learn

library:

from sklearn.cluster import AgglomerativeClustering

import numpy as np

# randomly chosen dataset

X = np.array([[1, 2], [1, 4], [1, 0],

[4, 2], [4, 4], [4, 0]])

# here we need to mention the number of clusters

# otherwise the result will be a single cluster

# containing all the data

clustering = AgglomerativeClustering(n_clusters=2).fit(X)

# print the class labels

print(clustering.labels_)

Output :

[1, 1, 1, 0, 0, 0]

Hierarchical Divisive Clustering

Another name for it is a top-down approach. It is also not necessary to

predetermine the number of clusters using this algorithm. The process of
breaking a cluster that includes all of the data must be done recursively in
order to divide each data into singleton clusters, which is necessary for
top-down clustering.

Algorithm :

given a dataset (d1, d2, d3, ....dN) of size N

at the top we have all data in one cluster

the cluster is split using a flat clustering method eg. K-Means etc

repeat

choose the best cluster among all the clusters to split

split that cluster by the flat clustering algorithm

until each data is in its own singleton cluster

Hierarchical Divisive clustering

Computing Distance Matrix

We measure the distance between each pair of clusters while merging

them, combining the ones that have the greatest similarity and the least
distance. How that distance is calculated is the question, though.
Distance/similarity between clusters can be defined in a variety of ways.
Some of them are:

1. Min Distance: Find the minimum distance between any two points of
the cluster.
2. Max Distance: Find the maximum distance between any two points
of the cluster.
3. Group Average: Find the average distance between every two points
of the clusters.
4. Ward’s Method: The similarity of two clusters is based on the
increase in squared error when two clusters are merged.

For example, if we group a given data using different methods, we may

get different results:

Distance Matrix Comparison in Hierarchical Clustering

Implementations code

import numpy as np

from scipy.cluster.hierarchy import dendrogram, linkage

import matplotlib.pyplot as plt

# randomly chosen dataset

X = np.array([[1, 2], [1, 4], [1, 0],

[4, 2], [4, 4], [4, 0]])

# Perform hierarchical clustering

Z = linkage(X, 'ward')

# Plot dendrogram

dendrogram(Z)

plt.title('Hierarchical Clustering Dendrogram')

plt.xlabel('Data point')

plt.ylabel('Distance')

plt.show()

Output:

Hierarchical Clustering Dendrogram

Hierarchical Agglomerative vs Divisive Clustering

 Divisive clustering is more complex as compared to agglomerative
clustering, as in the case of divisive clustering we need a flat
clustering method as “subroutine” to split each cluster until we have
each data having its own singleton cluster.
 Divisive clustering is more efficient if we do not generate a complete
hierarchy all the way down to individual data leaves. The time
complexity of a naive agglomerative clustering is O(n3) because we
exhaustively scan the N x N matrix dist_mat for the lowest distance
in each of N-1 iterations. Using priority queue data structure we can
reduce this complexity to O(n2logn). By using some more
optimizations it can be brought down to O(n2). Whereas for divisive
clustering given a fixed number of top levels, using an efficient flat
algorithm like K-Means, divisive algorithms are linear in the number
of patterns and clusters.
 A divisive algorithm is also more accurate. Agglomerative clustering
makes decisions by considering the local patterns or neighbour
points without initially taking into account the global distribution of
data. These early decisions cannot be undone. Whereas divisive
clustering takes into consideration the global distribution of data
when making top-level partitioning decisions.