Analysis of

High-Dimensional Data
Leif Kobbelt

Motivation

Given: n samples in d-dimensional space

X x1 , , x n R

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

d n

Motivation

Given: n samples in d-dimensional space

X x1 , , x n R

Decrease d
PCA
MDS

d n

dimensionality reduction:

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Principal Component Analysis

Idea: Compute orthorgonal linear transformation

that transforms the data into a new coordinate
system s.t.
greatest variance on first coordinate axis
second greatest variance on second axis
etc.

Optimal transform for a given data set in the least

squares sense
Dimensionality reduction: project data into lower
dimensional space spanned by first principal
components
4

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Principal Component Analysis

Given: n samples scattered in d-dimensional space,

written as a matrix

X x1 , x 2 , , x n R

d n

compute the centered covariance matrix:

C ( X X )( X X ) R
T

(interpretation as map from Rd to Rd)

5

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

d d

Principal Component Analysis

computation of C with the centering matrix:

C XJ XJ X J J X
T

1 T
J I n 11
n
principal component(s):
eigenvector(s) vi to largest eigenvalue(s) i of C
(low rank approximation)
6

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Principal Component Analysis

C V DV

v1 v d diag1 d v1 v d

v1 v q diag 1 q v1 v q

X : v1 v q
*

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

X J R

q n

Relation to SVD

singular value decomposition

XJ V U

C XJ ( XJ ) V U U V
T

V V
2

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

 for very large dimension d

d d

C XJ ( XJ ) R
~
T
n n
C ( XJ ) XJ R
T

w XJ v

Cv v

~
T
T
T
C w XJ XJ XJ v XJ v w
9

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Example

10 points in

10

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

10

Example

10 points in

0.617 0.615

C
0.615 0.717

0.74

e1
0.68
0.68

e2
0.74
11

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

11

Multi-Dimensional Scaling

Given: For n unknown samples X R

dimensional space

X x1 , , x n , x i R

we are given a matrix D R

(squared) distances:

n n

Di , j xi x j

12

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

12

d n

in high-

of pairwise

Multi-Dimensional Scaling

samples X in some abstract space:

X x1 , , x n , x i A
matrix

DR

n n

of pairwise abstract distances:

Di , j

13

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

13

Multi-Dimensional Scaling

Goal:find an embedding of X in a low-dimensional

space such that the pairwise (variations of)
distances D are preserved.

( D, D ) J ( D D ) J
T

2
F

) are possible
other measures ( D, D
but they cannot be solved easily.
14

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

14

Multi-Dimensional Scaling

closed form solution:

first q eigenvectors v1 , , v q of the matrix

J DJ R
1
2

n n

define the coordinates of a q-dimensional

embedding

vq
v1
X' 1
, , q

v1
v q

15

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

15

R q n

Multi-Dimensional Scaling

16

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Motivation

Given: n samples in d-dimensional space

X x1 , , x n R

Decrease n

clustering:

k-means
EM
Mean shift
Spectral clustering
Hierarchical clustering

17

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

17

d n

Cluster Analysis

Task: Given a set of observations / data samples,

assign them into clusters so that observations in
the same cluster are similar.

18

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

18

Cluster Analysis

Task: Given a set of observations / data samples,

assign them into clusters so that observations in
the same cluster are similar.

19

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

19

k-means Clustering

Idea: partition n observations into k clusters in

which each observation belongs to the cluster with
the nearest mean.
d
x
,

,
x
x

R
Given: data samples 1
n
i
Goal: partition the n samples into k sets (k n)
S1, S2, , Sk such that k

arg min
S

i 1 x j S i

is minimized, where i is the mean of points in Si.

20

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

20

k-means Clustering

Two step algorithm:

Assignment step: Assign each sample to the cluster with

the closest mean (Voronoi Diagram)

Sit x j : x j m ti x j m ti* , i * 1, , k

Update step: Calculate the new means to be the centroid

of the observations in the cluster.

t 1
i

1
t
Si

x j S it

Iterate until convergence (assignments change no

longer)
21

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

21

k-means Clustering

22

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

22

k-means Clustering

23

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

23

k-means Clustering

24

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

24

k-means Clustering

25

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

25

k-means Clustering

26

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

26

k-means Clustering

27

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

27

k-means Clustering

28

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

28

k-means Clustering

29

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

29

k-means Clustering

30

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

30

k-means Clustering

31

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

31

k-means Clustering

32

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

32

k-means Clustering

33

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

33

k-means Clustering

34

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

34

k-means Clustering

35

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

35

k-means Clustering - Comments

Advantages:

Efficient
Always converges to a solution

Drawbacks:

Not necessarily globally optimal solution

#clusters k is an input parameter
Sensitive to initial clusters
Cluster model: data is split halfway between cluster
means

36

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

36

Clustering Results

37

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

37

EM Algorithm

Expectation Maximization (EM)

Probabilistic assignments to clusters instead of
deterministic assignments
Multivariate Gaussian distributions instead of
means

38

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

38

EM Algorithm
d

x
,

,
x
,
x

R
Given: data samples
1
n
i
Assumption: data was generated by k Gaussians
Goal: Fit Gaussian mixture model (GMM) to data X
Find j 1,, k

means
covariances
j of the Gaussians j
probabilities (weights) j that the samples come from
the Gaussian j

39

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

39

EM Algorithm Example (1D)

Three samples drawn from each mixture component

means: 1 2, 2 2
40

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

40

EM Algorithm Example (2D)

41

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

41

EM Algorithm Example (2D)

42

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

42

EM Algorithm Example (2D)

43

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

43

EM Algorithm

1. Initialization: Choose initial estimates

0j , 0j , 0j , j 1, , k and compute the initial
log-likelihood

n
k

1
L0 log 0j x i 0j , 0j
n i 1
j 1

2. E-step: Compute

m
ij

mj x i mj , mj

m
m
m

l
i l
l
l 1

and

, i 1, , n, j 1, , k

n ijm , j 1, , k
m
j

44

i 1

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

44

EM Algorithm

3. M-step: Compute new estimates (j=1,...,k)

mj 1

n mj

n
1
m 1
j m
nj

m 1
j

1
m
nj

ij xi
i 1

x
n

i 1

m
ij

m 1
j

m 1 T
j

4. Convergence check: Compute new loglikelihood

m 1

45

k m 1
1 n
log j x i mj 1 , mj 1
n i 1
j 1

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

45

Example (2D)
Ground truth:
Means:

Covariance matrices:

Weights:

Input to EM-algorithm:
1000 samples

46

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

46

Initial Estimate

Initial density estimation:

(centroids of k-means result)

47

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

47

1st Iteration

48

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

48

2nd Iteration

49

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

49

3rd Iteration

Estimates after three iterations:

50

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

50

Mean Shift Clustering

Non-parametric clustering technique

No prior knowledge of #clusters
No constraints on shape of clusters

51

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

51

Mean Shift Clustering - Idea

Interprete points in feature space as empirical probability

density function
Dense regions in feature space correspond to local
maxima of the underlying distribution
For each sample: run gradient ascent procedure on local
estimated density until convergence
Stationary points = maxima of distribution
Samples associted with the same stationary point are
considered to be in the same cluster

52

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

52

Mean Shift Clustering

d
x
,

,
x
x

R
Given: data samples 1
n
i
Multi-variate kernel density estimate with radially
symmetric kernel K(x) and window radius h

1
f x d
nh

x xi
K

h
i 1
n

The radially symmetric kernel is defined as

K x ck ,d k x

where ck ,d is a normalization constant

Modes of density function are located at zeros of
gradient function f x 0
53

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

53

Mean Shift Clustering

Gradient of density estimator

2c k , d n
f x d 2 g
nh i 1

x xi 2

x
g

i
2

h
x x i i 1

n
h

x xi

g
h

i 1

'

x denotes the derivative of the

g
x

k
where

kernel profile k x

54

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

54

Mean Shift Clustering

Gradient of density estimator

2c k , d n
f x d 2 g
nh i 1

x xi 2

x
g

i
2

h
x x i i 1

n
h

x xi

g
h

i 1

proportional to density
estimate at x

mh x

mean shift vector mh x points toward direction of

maximum increase in the density.
55

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

55

Mean Shift Clustering

Mean shift procedure for sample x i :

1.
2.

m x ti

Compute mean shift vector

Translate density estimation window

x ti 1 x ti m x ti

Iterate 1. and 2. until convergence, i.e.,

f x i 0

56

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

56

Mean Shift Clustering

57

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

57

Mean Shift Clustering

x i0

58

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

58

Mean Shift Clustering

x1i

59

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

59

Mean Shift Clustering

x i2

60

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

60

Mean Shift Clustering

x 3i

61

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

61

Mean Shift Clustering

x in

62

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

62

Mean Shift - Comments

Advantages:

No prior knowledge of #clusters

No constraints on shape of clusters

Drawbacks:

Computationally expensive:

Run algorithm for every sample

Identification of sample neighborhood requires multi-dimensional
range search

How to choose the bandwidth parameter h ?

63

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Summary

Given: n samples in d-dimensional space

X x1 , , x n R

d n

Decrease d

dimensionality reduction:

Decrease n

clustering:

PCA
MDS

k-means
EM
Mean shift
Spectral clustering
Hierarchical clustering

64

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

64

Spectral Clustering
Model similarity between data points as graph

Clustering: Find connected components in graph

65

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Model similarity between data points as graph

(weighted) Adjacency Matrix W:

Degree Matrix D:

66

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Graphs:

Similarity graph: fully connected, model local neighborhood relations

Gaussian kernel similarity function:

K-nearest neighbour graph

-neighbourhood graph

67

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Model similarity between data points as graph

(weighted) Adjacency Matrix W:

Degree Matrix D:

Graph Laplacian L = D W:

68

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Properties of the Graph Laplacian L:
For every vector
L is symmetric and positive semi-definite
The smallest eigenvalue of L is 0
The corresonding eigenvector is the constant one vector
L has n non-negative, real-valued eigenvalues

69

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
The multiplicity k of the eigenvalue 0 of L equals the number of connected
components in the graph
Consider k = 1. Assume f is eigenvector with eigenvalue 0:

The sum only vanishes if all terms

vanish

If two vertices are connected (their edge weight > 0)

f needs to be constant for all vertices which can be connected by a path
All vertices of a connected component in an undirected graph can be connected by a
path:
f needs to be constant on the whole connected component

70

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Laplacian of graph with 1 connected component has one constant vector
with eigenvalue 0
For k > 1: Wlog. assume that vertices are ordered according to connected
components

Each

Each

is a graph Laplacian of a fully connected graph:

has one eigenvalue 0 with constant one vector on the i-th connected comp.

Spectrum of L is given by union of the spectra of

71

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Graph:

Graph Laplacian

Eigenvectors for eigenvalues

72

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Graph:

Project vertices into subspace spanned by k eigenvectors

Projected vertices:
K-means clustering recovers the connected components
Embedding is the same regardless of data ordering

73

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Similarity Graph:

W=

74

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Similarity Graph:

L=

Eigenvalues : 0, 0.4, 2, 2
Eigenvectors :

75

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Similarity Graph:

For fully connected graph we want to find the Min-Cut:

Partition graph into 2 sets of vertices such that the weight of edges connecting them
is minimal:
Vertices in each set should be similar to vertices in the same set, but dissimilar to
vertices from the other set
Partitions often not balanced: isolated vertices

76

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Similarity Graph:

For fully connected graph we want to find the Normalized Cut:

Partition graph into 2 sets of vertices such that the weight of edges connecting them
is minimal
Partitions should have similar size

77

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Min-Cut: minimize
Normalized Cut: minimize

78

minimal if

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Reformulate with Graph Laplacian
Construct f:

79

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Reformulate Ncut:
Minimize

subject to

Partition (cluster) assignment by thresholding f at 0

NP hard to compute since f is discrete
Relax problem by allowing f to take arbitrary real values

Solution: second eigenvector of

(normalized Graph Laplacian)

For k > 2 we can similarily construct indicator vectors like f and relax the
problem for minimization:
Project the vertices into the subspace spanned by the first k eigenvectors of L
Clustering the embedded vertices yields the solution

Spectral clustering (with normalized Graph Laplacian) approximates Ncut

80

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Spectral Clustering
Mean Shift

81

Spectral Clustering

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

K-Means

Spectral Clustering
Summary:

Useful for non-convex clustering problems

Computation intensive because of eigenvalue computation (for large matrices)
Choice of k necessary:
A heuristic can be used that tries to find jumps in the eigenvalues (eigengap)
Similarity has to be defined for graph construction:
Size of Gaussian kernel?
Size of neighbourhood?

82

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Hierarchical Clustering
Bottom up:

Each data point is its own cluster

Greedily merge clusters according to some criteria

83

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Hierarchical Clustering
Requirements:

Metric: distance between data points

Linkage: distance between data point sets:

84

Maximum linkage:

Average linkage:

Ward linkage:

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Hierarchical Clustering
Algorithm:

Start out with a cluster for each data point

Merge two clusters that result in the least increase in linkage criteria
Repeat until k clusters remain

Maximum linkage:

Minimizes maximimal distance of data points in each cluster

Average linkage:

Minimizes average distance of data points in each cluster

Ward linkage:

Minimizes inter-cluster variance

85

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Hierarchical Clustering
We can add connectivity constraints that enforce which clusters can be
merged

86

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization

Hierarchical Clustering
Summary:

Flexibel: any pairwise distance can be used

Choice of k, distance and linkage necessary
Instead of specifying k we can use a heuristic which stops cluster merging if the
linkage increases too much
Given connectivity constraints hierarchical clustering scales well for large number of
data points
How do we choose connectivity constraints?
K-nearest neighbour graph
-neighbourhood graph

87

Visual Computing Institute | Prof. Dr. Leif Kobbelt

Computer Graphics and Multimedia
Data Analysis and Visualization