DATA SCIENCE

LECTURE 8
Clustering
 2

Basic Data Model: Sets

• Document: A document is represented as a set
shingles (more accurately, hashes of shingles)

• Document similarity: Jaccard similarity of the sets

of shingles.
• Common shingles over the union of shingles
• Sim (C1, C2) = |C1C2|/|C1C2|.

• Applicable to any kind of sets.

• E.g., similar customers or items.
What is a Clustering?
• In general a grouping of objects such that the objects in a
group (cluster) are similar (or related) to one another and
different from (or unrelated to) the objects in other groups

Inter-cluster
Intra-cluster distances are
distances are maximized
minimized
 Applications of Cluster Analysis
Discovered Clusters Industry Group
• Understanding Applied-Matl-DOWN,Bay-Network-Down,3-COM-DOWN,

• Group related documents for

1 Cabletron-Sys-DOWN,CISCO-DOWN,HP-DOWN,
DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN,
Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down,
Technology1-DOWN
Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN,
browsing, group genes and Sun-DOWN
Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN,

proteins that have similar 2 ADV-Micro-Device-DOWN,Andrew-Corp-DOWN,

Computer-Assoc-DOWN,Circuit-City-DOWN,
Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN,
Technology2-DOWN
functionality, or group stocks Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN
Fannie-Mae-DOWN,Fed-Home-Loan-DOWN,
with similar price fluctuations 3 MBNA-Corp-DOWN,Morgan-Stanley-DOWN Financial-DOWN
Baker-Hughes-UP,Dresser-Inds-UP,Halliburton-HLD-UP,

4 Louisiana-Land-UP,Phillips-Petro-UP,Unocal-UP,
Schlumberger-UP
Oil-UP

• Summarization
• Reduce the size of large data
sets

Clustering precipitation
in Australia
Early applications of cluster analysis
• John Snow, London 1854
Notion of a Cluster can be Ambiguous

How many clusters? Six Clusters

Two Clusters Four Clusters

Types of Clusterings
• A clustering is a set of clusters

• Important distinction between hierarchical and

partitional sets of clusters
• Partitional Clustering
• A division data objects into subsets (clusters) such
that each data object is in exactly one subset
• Hierarchical clustering
• A set of nested clusters organized as a hierarchical
tree
Partitional Clustering

Original Points A Partitional Clustering

Hierarchical Clustering

p1
p3 p4
p2
p1 p2 p3 p4

Traditional Hierarchical Traditional Dendrogram

Clustering

p1
p3 p4
p2
p1 p2 p3 p4

Non-traditional Hierarchical Non-traditional Dendrogram

Clustering
Other types of clustering
• Exclusive (or non-overlapping) versus non-
exclusive (or overlapping)
• In non-exclusive clusterings, points may belong to
multiple clusters.
• Points that belong to multiple classes, or ‘border’ points

• Fuzzy (or soft) versus non-fuzzy (or hard)

• In fuzzy clustering, a point belongs to every cluster
with some weight between 0 and 1
• Weights usually must sum to 1 (often interpreted as probabilities)

• Partial versus complete

• In some cases, we only want to cluster some of the
data
Types of Clusters: Well-Separated
• Well-Separated Clusters:
• A cluster is a set of points such that any point in a cluster is
closer (or more similar) to every other point in the cluster than
to any point not in the cluster.

3 well-separated clusters
Types of Clusters: Center-Based
• Center-based
• A cluster is a set of objects such that an object in a cluster is
closer (more similar) to the “center” of a cluster, than to the
center of any other cluster
• The center of a cluster is often a centroid, the minimizer of
distances from all the points in the cluster, or a medoid, the
most “representative” point of a cluster

4 center-based clusters
Types of Clusters: Contiguity-Based
• Contiguous Cluster (Nearest neighbor or
Transitive)
• A cluster is a set of points such that a point in a cluster is
closer (or more similar) to one or more other points in the
cluster than to any point not in the cluster.

8 contiguous clusters
Types of Clusters: Density-Based
• Density-based
• A cluster is a dense region of points, which is separated by
low-density regions, from other regions of high density.
• Used when the clusters are irregular or intertwined, and when
noise and outliers are present.

6 density-based clusters
Types of Clusters: Conceptual Clusters
• Shared Property or Conceptual Clusters
• Finds clusters that share some common property or represent
a particular concept.
.

2 Overlapping Circles
Types of Clusters: Objective Function
• Clustering as an optimization problem
• Finds clusters that minimize or maximize an objective function.
• Enumerate all possible ways of dividing the points into clusters
and evaluate the `goodness' of each potential set of clusters by
using the given objective function. (NP Hard)
• Can have global or local objectives.
• Hierarchical clustering algorithms typically have local objectives
• Partitional algorithms typically have global objectives
• A variation of the global objective function approach is to fit the
data to a parameterized model.
• The parameters for the model are determined from the data, and they
determine the clustering
• E.g., Mixture models assume that the data is a ‘mixture' of a number of
statistical distributions.
Clustering Algorithms
• K-means and its variants

• Hierarchical clustering

• DBSCAN
K-MEANS
K-means Clustering
• Partitional clustering approach
• Each cluster is associated with a centroid
(center point)
• Each point is assigned to the cluster with the
closest centroid
• Number of clusters, K, must be specified
• The objective is to minimize the sum of
distances of the points to their respective
centroid
K-means Clustering

• Problem: Given a set X of n points in a d-

dimensional space and an integer K group the
points into K clusters C= {C1, C2,…,Ck} such that

is minimized, where ci is the centroid of the points

in cluster Ci
K-means Clustering
• Most common definition is with euclidean distance,
minimizing the Sum of Squares Error (SSE)
function
• Sometimes K-means is defined like that

• Problem: Given a set X of n points in a d-

dimensional space and an integer K group the
points into K clusters C= {C1, C2,…,Ck} such that

is minimized, where ci is the mean of the points in

Sum of Squares Error (SSE)
cluster Ci
Complexity of the k-means problem
• NP-hard if the dimensionality of the data is at
least 2 (d>=2)
• Finding the best solution in polynomial time is infeasible

• For d=1 the problem is solvable in polynomial

time (how?)

• A simple iterative algorithm works quite well in

practice
K-means Algorithm
• Also known as Lloyd’s algorithm.
• K-means is sometimes synonymous with this
algorithm
K-means Algorithm – Initialization
• Initial centroids are often chosen randomly.
• Clusters produced vary from one run to another.
Two different K-means Clusterings
3

2.5

1.5
Original Points

y
1

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Optimal Clustering Sub-optimal Clustering

Importance of Choosing Initial Centroids
Iteration 6
1
2
3
4
5
3

2.5

1.5
y

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x
 Importance of Choosing Initial Centroids
Iteration 1 Iteration 2 Iteration 3
3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

Iteration 4 Iteration 5 Iteration 6

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x
Importance of Choosing Initial Centroids
Iteration 5
1
2
3
4
3

2.5

1.5
y

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x
 Importance of Choosing Initial Centroids …
Iteration 1 Iteration 2
3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Iteration 3 Iteration 4 Iteration 5

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x
Dealing with Initialization
• Do multiple runs and select the clustering with the
smallest error

• Select original set of points by methods other

than random . E.g., pick the most distant (from
each other) points as cluster centers (K-means++
algorithm)
K-means Algorithm – Centroids
• The centroid depends on the distance function
• The minimizer for the distance function
• ‘Closeness’ is measured by Euclidean distance
(SSE), cosine similarity, correlation, etc.
• Centroid:
• The mean of the points in the cluster for SSE, and cosine
similarity
• The median for Manhattan distance.

• Finding the centroid is not always easy

• It can be an NP-hard problem for some distance functions
• E.g., median form multiple dimensions
K-means Algorithm – Convergence
• K-means will converge for common similarity
measures mentioned above.
• Most of the convergence happens in the first few
iterations.
• Often the stopping condition is changed to ‘Until
relatively few points change clusters’
• Complexity is O( n * K * I * d )
• n = number of points, K = number of clusters,
I = number of iterations, d = dimensionality
• In general a fast and efficient algorithm
Limitations of K-means
• K-means has problems when clusters are of
different
• Sizes
• Densities
• Non-globular shapes

• K-means has problems when the data contains

outliers.
Limitations of K-means: Differing Sizes

Original Points K-means (3 Clusters)

Limitations of K-means: Differing Density

Original Points K-means (3 Clusters)

Limitations of K-means: Non-globular
Shapes

Original Points K-means (2 Clusters)