Clustering

Course: Artificial Intelligence

Fundamentals

Instructor: Marco Bonzanini

Machine Learning Tasks

Supervised Unsupervised
Discrete Data

Classification Clustering
(predict a label) (group similar items)

Continuous Data
Dimensionality
Regression Reduction
(predict a quantity) (reduce n. of variables)
Machine Learning Tasks

Supervised Unsupervised
Discrete Data

Classification Clustering
(predict a label) (group similar items)

Continuous Data
Dimensionality
Regression Reduction
(predict a quantity) (reduce n. of variables)
 Agenda
• Introduction to Clustering

• Clustering Algorithms

• Centroid-based

• Optional: Connectivity-based

• Optional: Density-based

• Evaluation
Introduction to
Clustering
 Clustering

• Place similar items in the same group

• (Place dissimilar items in different groups)

• How to define “similarity”?

• Clustering cannot give a comprehensive

description of an object
Clustering Applications

• Customer Segmentation

• Fraud Detection

• Social Network Analysis

• Search engines (navigation, indexing, …)

• … Your use case?

Clustering vs Classification

• Classification:
- supervised
- requires a set of labelled training samples

• Clustering:
- unsupervised
- learns without labels
Classification Example

Training:
items with labels
Classification Example
New, unseen item
?
Training:
items with labels
Classification Example

Training:
items with labels
Prediction:
assign item to class
Clustering Example

Training: no labels
Clustering Example

Training: no labels
Prediction: group items
 More Definitions

• “Learn from raw data”

• “Find structure in data”

• “Unsupervised classification”
Flat vs Hierarchical
• Flat approach:

• There’s a number of clusters, and the relation

between clusters is undetermined

• Often start with a random partial partitioning

• Refine it iteratively (e.g. K-Means)

• Measurement: error minimisation

Flat vs Hierarchical
• Hierarchical approach:

• Bottom-up, agglomerative

• Top-down, divisive

• A hierarchy of clusters (i.e. tree structure)

• Measurement: similarity of instances

 Hard vs Soft

• Cluster assignment

• Hard clustering: each item belongs to one and only

one cluster (more common)

• Soft clustering: items can belong to more than one

cluster (e.g. a pair of sneakers can be in “sports”
and “shoes”)
 Common Issues
• Item representation (e.g. vector)

• Need a notion of distance / similarity

• Ideal: semantic similarity

• Practical: Euclidean distance, cosine similarity

• How many clusters?

• Fixed a priori? Data-driven?

Clustering
Algorithms
Different Approaches

• Centroid-based clustering:
K-Means

• Connectivity-based clustering:
Hierarchical Agglomerative

• Density-based clustering:
DBSCAN
Centroid-based Clustering

• Clusters are represented by their centroid

• Centroid: central vector, centre of gravity, arithmetic

mean

• Centroid: not necessarily a member of the cluster

• Based on distance between items and centroids

• Most common algorithm: K-Means

 K-Means Overview
• Input:
- Set of items
- Desired n. of clusters K

• Output:
- A partition of the input set into K clusters

• Assumption:
- Input items are real-valued vectors
- Notion of distance / similarity
 K-Means Algorithm

1.Initialise K centroids randomly

2.For each point: assign to closest centroid

3.Update centroids

4.Repeat 2-3 until convergence

K-Means Example

K=2
K-Means Example

1. Centroids: random init

K-Means Example

2. Assign items to
closest centroid
K-Means Example

3. Update centroids
K-Means Example

4. Repeat:
- assign items to centroids
- update centroids
K-Means Example

4. Repeat:
- assign items to centroids
- update centroids
K-Means Example

4. Repeat:
- assign items to centroids
- update centroids
K-Means Example

Convergence!
 Assignment Step

• Assign to “closest” centroid

• Closest = least squared Euclidean distance

 Update Step

• New centroid is the mean of the cluster

K-Means Discussion

• Pros: intuitive, quite good in practice

• Cons: requires to know (or find out) K

• Elbow method to find out K

 Elbow Method

• Intrinsic metric: within-cluster Sum of Squared Error

• a.k.a. Distortion

• If K increases
the distortion decreases

X
Notebook Intermezzo
Clustering - KMeans
 Connectivity-based
Clustering
• a.k.a. Hierarchical Clustering

• It needs a notion of pairwise dissimilarity between

groups (clusters), called linkage

• Top-down (divisive): start with all items in one group,

then divide them to maximise within-group similarity

• Bottom-up (agglomerative): start with items in

individual groups, then aggregate the most similar
ones (until one cluster containing all objects is formed)
Agglomerative Clustering
Example
7 2
1

Steps:
6
{1}, {2}, {3}, {4}, {5}, {6}, {7}
{1, 2}, {3}, {4}, {5}, {6}, {7}
{1, 2}, {3, 4}, {5}, {6}, {7}
4
{1, 2}, {3, 4, 5}, {6}, {7}
5
{1, 2, 6}, {3, 4, 5}, {7}
3 {1, 2, 6, 7}, {3, 4, 5}
{1, 2, 3, 4, 5, 6, 7}
 Dendrogram

• Used to illustrate the output of hierarchical

clustering

• The algorithm builds a tree-based hierarchical

taxonomy
Dendrogram Example
7 2
1

5 4

3
1 2 6 7 5 3 4
Dendrogram Example on
Iris dataset
From Dendroid to Clusters

• Cutting the dendrogram horizontally partitions the

data points into clusters

• Choice of distance

• Choice of number of clusters

 Linkage
• The notion of dissimilarity is described with a distance
function d(G, H), with G and H groups of nodes (cluster
assignments at any level)

• Single linkage: smallest dissimilarity between two points in

opposite groups, i.e. nearest neighbour interpretation

• Complete linkage: largest dissimilarity between two points

in opposite groups, i.e. furthest neighbour interpretation

• Average linkage: average dissimilarity between all points

in opposite groups
 Single Linkage
Distance

0.7
• For each point x
there is a point y
in its cluster
where d(x, y) ≤ 0.7
1 2 6 7 5 3 4
 Complete Linkage
Distance

0.7
• For each point x
all points y
in its cluster
satisfy d(x, y) ≤ 0.7
1 2 6 7 5 3 4
 Average Linkage
Distance

0.7

• Cut interpretation:
there isn’t a good one!

1 2 6 7 5 3 4
 Linkage Issues
• Single linkage suffers from chaining.
Only one pair of points needs to be close in order to
merge two groups, i.e. clusters can be spread out and
not very compact

• Complete linkage suffers from crowding.

Score based on worst-case dissimilarity between pairs
of points, i.e. clusters are compact but not far apart

• Average linkage strikes a balance, but doesn’t have a

clear interpretation
 More on Linkage
• Centroid linkage (new centroid is avg of all group items)

• Median linkage: like Centroid, but new centroid

calculated as avg of the two old centroids

• Ward linkage (Ward’s variance minimisation)

 Hierarchical Clustering
Discussion

• Pros: repeatability (why?), no prior knowledge of K

is required (can choose cut-off threshold, or
number of clusters)

• Cons: complexity (why?)

• No silver bullet
 Notebook Intermezzo:
Clustering - Hierarchical
Density-based Clustering
• Clusters are regions in the data space with higher
density, separated by lower density regions

• Objects in sparse areas are considered noise or

borders between clusters

• A cluster is defined as a maximal set of density-

connected points

• Shape of clusters: arbitrary

• Most common algorithm: DBSCAN

 DBSCAN Overview

• Density-Based Spatial Clustering of Applications

with Noise

• Given an input set of points, it groups together

points that are closely packed together, i.e. with
many neighbour points.
 DBSCAN Concepts
• ε-Neighbourhood
N(p) = {q | d(p, q) ≤ ε }

• High density points (core points)

p is core point if N(p) contains at least minPts objects

• Density-reachable points
q is directly reachable from p if it’s in N(p) and p is a
core point
q is reachable from p if p is a core point and there’s a
path of directly reachable core points between them
DBSCAN Concepts
7 minPts = 2

2
6 1

8
- 1 is reachable from 6
- 3 and 5 are
5 4
ε directly reachable from 4
3 - 7 is an outlier
 DBSCAN Algorithm
• Find the ε-neighbour of all points

• Identify the core points (at least minPts)

• Find the connected components of core points,

ignoring the non-core points

• Assign non-core points to nearby cluster, if the

cluster is an ε-neighbourhood, otherwise assign to
noise
DBSCAN Example
DBSCAN Discussion

• Pros: no prior knowledge of K is required, can find

arbitrarily shaped cluster, robust to outliers (notion
of noise)

• Cons: complexity (why?), sensitive to data sets with

large differences in densities
Notebook Intermezzo:
Clustering - DBSCAN
Clustering
Evaluation
 Cluster Quality

• How good is the clustering result?

• What is its interpretation?

• What is the purpose of the clustering task?

 Internal Evaluation
• Evaluation based on the data set itself

• No external gold standard

• Idea: good clustering produces clusters with high

within-cluster similarity and low between-cluster
similarity

• Drawback: is the evaluation biased?

• e.g. Sum of Squared Errors (SSE)

External Evaluation
• Requires externally supplied labels

• Relationship with classification evaluation

• Metrics: Precision, Recall, F-Measure, Jaccard

Index, Dice Index, …

• Drawback: are we missing out on knowledge

discovery?
Questions?