A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 1

A Comparative Study of Data Clustering

Techniques
Khaled Hammouda
Prof. Fakhreddine Karray
University of Waterloo, Ontario, Canada

(two or three dimensions at maximum.) This is

why some methods in soft computing have been
Abstract Data clustering is a process of putting similar proposed to solve this kind of problem. Those
data into groups. A clustering algorithm partitions a
data set into several groups such that the similarity
methods are called Data Clustering Methods
within a group is larger than among groups. This paper and they are the subject of this paper.
reviews four of the most representative off-line Clustering algorithms are used extensively
clustering techniques: K-means clustering, Fuzzy C- not only to organize and categorize data, but are
means clustering, Mountain clustering, and Subtractive also useful for data compression and model
clustering. The techniques are implemented and tested
against a medical problem of heart disease diagnosis.
construction. By finding similarities in data, one
Performance and accuracy of the four techniques are can represent similar data with fewer symbols for
presented and compared. example. Also if we can find groups of data, we
can build a model of the problem based on those
groupings.
Index Termsdata clustering, k-means, fuzzy c-means,
mountain, subtractive.
Another reason for clustering is to discover
relevance knowledge in data. Francisco Azuaje et
I. INTRODUCTION al. [2] implemented a Case Based Reasoning
(CBR) system based on a Growing Cell Structure
(GCS) model. Data can be stored in a knowledge

D ATA CLUSTERING is considered an

interesting approach for finding similarities
in data and putting similar data into groups.
base that is indexed or categorized by cases; this
is what is called a Case Base. Each group of cases
is assigned to a certain category. Using a
Clustering partitions a data set into several groups Growing Cell Structure (GCS) data can be added
such that the similarity within a group is larger or removed based on the learning scheme used.
than that among groups [1]. The idea of data Later when a query is presented to the model, the
grouping, or clustering, is simple in its nature and system retrieves the most relevant cases from the
is close to the human way of thinking; whenever case base depending on how close those cases are
we are presented with a large amount of data, we to the query.
usually tend to summarize this huge number of In this paper, four of the most
data into a small number of groups or categories representative off-line clustering techniques are
in order to further facilitate its analysis. reviewed:
Moreover, most of the data collected in many K-means (or Hard C-means) Clustering,
problems seem to have some inherent properties Fuzzy C-means Clustering,
that lend themselves to natural groupings. Mountain Clustering, and
Nevertheless, finding these groupings or trying to Subtractive Clustering.
categorize the data is not a simple task for
humans unless the data is of low dimensionality These techniques are usually used in conjunction
with radial basis function networks (RBFNs) and
Fuzzy Modeling. Those four techniques are
K. M. Hammouda, implemented and tested against a medical
Department of Systems Design Engineering, diagnosis problem for heart disease. The results
University of Waterloo, Waterloo, Ontario, Canada N2L 3G1
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 2

are presented with a comprehensive comparison Some of the clustering techniques rely on
of the different techniques and the effect of knowing the number of clusters apriori. In that
different parameters in the process. case the algorithm tries to partition the data into
the given number of clusters. K-means and Fuzzy
The remainder of the paper is organized as
C-means clustering are of that type. In other cases
follows. Section II presents an overview of data
it is not necessary to have the number of clusters
clustering and the underlying concepts. Section
known from the beginning; instead the algorithm
III presents each of the four clustering techniques
starts by finding the first large cluster, and then
in detail along with the underlying mathematical
goes to find the second, and so on. Mountain and
foundations. Section IV introduces the
Subtractive clustering are of that type. In both
implementation of the techniques and goes over
cases a problem of known cluster numbers can be
the results of each technique, followed by a
applied; however if the number of clusters is not
comparison of the results. A brief conclusion is
known, K-means and Fuzzy C-means clustering
presented in Section V. The MATLAB code
cannot be used.
listing of the four clustering techniques can be
Another aspect of clustering algorithms is
found in the appendix.
their ability to be implemented in on-line or off-
line mode. On-line clustering is a process in
II. DATA CLUSTERING OVERVIEW
which each input vector is used to update the
cluster centers according to this vector position.
As mentioned earlier, data clustering is The system in this case learns where the cluster
concerned with the partitioning of a data set into centers are by introducing new input every time.
several groups such that the similarity within a In off-line mode, the system is presented with a
group is larger than that among groups. This training data set, which is used to find the cluster
implies that the data set to be partitioned has to centers by analyzing all the input vectors in the
have an inherent grouping to some extent; training set. Once the cluster centers are found
otherwise if the data is uniformly distributed, they are fixed, and they are used later to classify
trying to find clusters of data will fail, or will lead new input vectors. The techniques presented here
to artificially introduced partitions. Another are of the off-line type. A brief overview of the
problem that may arise is the overlapping of data four techniques is presented here. Full detailed
groups. Overlapping groupings sometimes reduce discussion will follow in the next section.
the efficiency of the clustering method, and this The first technique is K-means clustering
reduction is proportional to the amount of overlap [6] (or Hard C-means clustering, as compared to
between groupings. Fuzzy C-means clustering.) This technique has
Usually the techniques presented in this been applied to a variety of areas, including
paper are used in conjunction with other image and speech data compression, [3, 4] data
sophisticated neural or fuzzy models. In preprocessing for system modeling using radial
particular, most of these techniques can be used basis function networks, and task decomposition
as preprocessors for determining the initial in heterogeneous neural network architectures
locations for radial basis functions or fuzzy if- [5]. This algorithm relies on finding cluster
then rules. centers by trying to minimize a cost function of
The common approach of all the clustering dissimilarity (or distance) measure.
techniques presented here is to find cluster The second technique is Fuzzy C-means
centers that will represent each cluster. A cluster clustering, which was proposed by Bezdek in
center is a way to tell where the heart of each 1973 [1] as an improvement over earlier Hard C-
cluster is located, so that later when presented means clustering. In this technique each data
with an input vector, the system can tell which point belongs to a cluster to a degree specified by
cluster this vector belongs to by measuring a a membership grade. As in K-means clustering,
similarity metric between the input vector and al Fuzzy C-means clustering relies on minimizing a
the cluster centers, and determining which cluster cost function of dissimilarity measure.
is the nearest or most similar one.
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 3

The third technique is Mountain clustering, The partitioned groups are defined by a
proposed by Yager and Filev [1]. This technique c n binary membership matrix U , where the
builds calculates a mountain function (density element uij is 1 if the jth data point x j belongs
function) at every possible position in the data
to group i , and 0 otherwise. Once the cluster
space, and chooses the position with the greatest
centers ci are fixed, the minimizing uij for
density value as the center of the first cluster. It
then destructs the effect of the first cluster Equation (1) can be derived as follows:
mountain function and finds the second cluster 2 2
1 if x j ci x j ck , for each k i ,
center. This process is repeated until the desired uij = (2)
number of clusters have been found. 0 otherwise.
The fourth technique is Subtractive
clustering, proposed by Chiu [1]. This technique Which means that x j belongs to group i if
is similar to mountain clustering, except that
ci is the closest center among all centers.
instead of calculating the density function at
every possible position in the data space, it uses On the other hand, if the membership
the positions of the data points to calculate the matrix is fixed, i.e. if uij is fixed, then the optimal
density function, thus reducing the number of
center ci that minimize Equation (1) is the mean
calculations significantly.
of all vectors in group i :
III. DATA CLUSTERING TECHNIQUES 1
ci = xk , (3)
Gi k , x k Gi
In this section a detailed discussion of each
technique is presented. Implementation and where Gi is the size of Gi , or Gi =
n
u .
results are presented in the following sections. j =1 ij

The algorithm is presented with a data set

A. K-means Clustering xi , i = 1, , n ; it then determines the cluster
centers ci and the membership matrix U
The K-means clustering, or Hard C-means iteratively using the following steps:
clustering, is an algorithm based on finding data
clusters in a data set such that a cost function (or Step 1: Initialize the cluster center ci , i = 1, , c .
an objection function) of dissimilarity (or This is typically done by randomly
distance) measure is minimized [1]. In most cases selecting c points from among all of the
this dissimilarity measure is chosen as the data points.
Euclidean distance. Step 2: Determine the membership matrix U by
A set of n vectors x j , j = 1, , n , are to be Equation (2).
partitioned into c groups Gi , i = 1, , c . The cost Step 3: Compute the cost function according to
Equation (1). Stop if either it is below a
function, based on the Euclidean distance
certain tolerance value or its
between a vector x k in group j and the improvement over previous iteration is
corresponding cluster center ci , can be defined below a certain threshold.
by: Step 4: Update the cluster centers according to
c c Equation (3). Go to step 2.
2
J= Ji = x k ci , (1)
i =1 i =1 k , x k Gi
The performance of the K-means algorithm
depends on the initial positions of the cluster
2 centers, thus it is advisable to run the algorithm
where J i = k , x k Gi x k ci is the cost function several times, each with a different set of initial
within group i . cluster centers. A discussion of the
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 4

implementation issues is presented later in this The algorithm works iteratively through the
paper. preceding two conditions until the no more
improvement is noticed. In a batch mode
B. Fuzzy C-means Clustering operation, FCM determines the cluster centers ci
and the membership matrix U using the
Fuzzy C-means clustering (FCM), relies on following steps:
the basic idea of Hard C-means clustering
(HCM), with the difference that in FCM each Step 1: Initialize the membership matrix U with
data point belongs to a cluster to a degree of random values between 0 and 1 such that
membership grade, while in HCM every data the constraints in Equation (4) are
point either belongs to a certain cluster or not. So satisfied.
FCM employs fuzzy partitioning such that a Step 2: Calculate c fuzzy cluster centers
given data point can belong to several groups ci , i = 1, , c , using Equation (6).
with the degree of belongingness specified by Step 3: Compute the cost function according to
membership grades between 0 and 1. However, Equation (5). Stop if either it is below a
FCM still uses a cost function that is to be certain tolerance value or its
minimized while trying to partition the data set. improvement over previous iteration is
The membership matrix U is allowed to below a certain threshold.
have elements with values between 0 and 1. Step 4: Compute a new U using Equation (7).
However, the summation of degrees of Go to step 2.
belongingness of a data point to all clusters is
As in K-means clustering, the performance
always equal to unity:
c
of FCM depends on the initial membership matrix
uij = 1, j = 1, ,n . (4) values; thereby it is advisable to run the algorithm
i =1 for several times, each starting with different
values of membership grades of data points.
The cost function for FCM is a
generalization of Equation (1): C. Mountain Clustering
c c n
J (U , c1 , , cc ) = Ji = uijm dij2 , (5)
i =1 i =1 j =1 The mountain clustering approach is a
simple way to find cluster centers based on a
where uij is between 0 and 1; ci is the cluster density measure called the mountain function.
This method is a simple way to find approximate
center of fuzzy group i ; d ij = ci x j is the cluster centers, and can be used as a preprocessor
Euclidean distance between the ith cluster center for other sophisticated clustering methods.
and the jth data point; and m [1, ) is a The first step in mountain clustering
involves forming a grid on the data space, where
weighting exponent.
the intersections of the grid lines constitute the
The necessary conditions for Equation (5)
potential cluster centers, denoted as a set V .
to reach its minimum are
n
The second step entails constructing a
umx
j =1 ij j mountain function representing a data density
ci = n m
, (6)
measure. The height of the mountain function at a
u
j =1 ij point v V is equal to
2
and
N v xi
m( v ) = exp , (8)
i =1 2 2
1
uij = 2 /( m 1)
. (7)
c d ij where xi is the ith data point and is an
k =1
d kj application specific constant. This equation states
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 5

that the data density measure at a point v is Since each data point is a candidate for
affected by all the points xi in the data set, and cluster centers, a density measure at data point xi
this density measure is inversely proportional to is defined as
the distance between the data points xi and the n xi x j
2

point under consideration v . The constant Di = exp , (10)

j =1 (ra / 2) 2
determines the height as well as the smoothness
of the resultant mountain function.
The third step involves selecting the cluster where ra is a positive constant representing a
centers by sequentially destructing the mountain neighborhood radius. Hence, a data point will
function. The first cluster center c1 is determined have a high density value if it has many
by selecting the point with the greatest density neighboring data points.
measure. Obtaining the next cluster center The first cluster center x c1 is chosen as the
requires eliminating the effect of the first cluster.
point having the largest density value Dc1 . Next,
This is done by revising the mountain function: a
new mountain function is formed by subtracting a the density measure of each data point xi is
scaled Gaussian function centered at c1 : revised as follows:
2 2
v c1 xi x c1
mnew ( v) = m( v) m(c1 ) exp (9) Di = Di Dc1 exp (11)
2 2 (rb / 2) 2

The subtracted amount eliminates the effect of the where rb is a positive constant which defines a
first cluster. Note that after subtraction, the new
neighborhood that has measurable reductions in
mountain function mnew ( v) reduces to zero at
density measure. Therefore, the data points near
v = c1 . the first cluster center x c1 will have significantly
After subtraction, the second cluster center reduced density measure.
is selected as the point having the greatest value After revising the density function, the next
for the new mountain function. This process cluster center is selected as the point having the
continues until a sufficient number of cluster greatest density value. This process continues
centers is attained. until a sufficient number of clusters is attainted.
D. Subtractive Clustering
IV. IMPLEMENTATION AND RESULTS
The problem with the previous clustering
method, mountain clustering, is that its Having introduced the different clustering
computation grows exponentially with the techniques and their basic mathematical
dimension of the problem; that is because the foundations, we now turn to the discussion of
mountain function has to be evaluated at each these techniques on the basis of a practical study.
grid point. Subtractive clustering solves this This study involves the implementation of each of
problem by using data points as the candidates for the four techniques introduced previously, and
cluster centers, instead of grid points as in testing each one of them on a set of medical data
mountain clustering. This means that the related to heart disease diagnosis problem.
computation is now proportional to the problem The medical data used consists of 13 input
size instead of the problem dimension. However, attributes related to clinical diagnosis of a heart
the actual cluster centers are not necessarily disease, and one output attribute which indicates
located at one of the data points, but in most cases whether the patient is diagnosed with the heart
it is a good approximation, especially with the disease or not. The whole data set consists of 300
reduced computation this approach introduces. cases. The data set is partitioned into two data
sets: two-thirds of the data for training, and one-
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 6

Performance Test Runs

measure 1 2 3 4 5 6 7 8 9 10
No. of iterations 7 6 9 6 5 5 3 7 11 7
RMSE 0.469 0.469 0.447 0.469 0.632 0.692 0.692 0.447 0.447 0.469
Accuracy 78.0% 78.0% 80.0% 78.0% 60.0% 52.0% 52.0% 80.0% 80.0% 78.0%
Regression Line
0.564 0.564 0.6 0.564 0.387 0.066 0.057 0.6 0.6 0.564
Slope
Table 1. K-means Clustering Performance Results

third for evaluation. The number of clusters into Equations (2) and (3), respectively, until no
which the data set is to be partitioned is two further improvement in the cost function is
clusters; i.e. patients diagnosed with the heart noticed. Since the algorithm initializes the cluster
disease, and patients not diagnosed with the heart centers randomly, its performance is affected by
disease. Because of the high number of those initial cluster centers. So several runs of the
dimensions in the problem (13-dimensions), no algorithm is advised to have better results.
visual representation of the clusters can be Evaluating the algorithm is realized by
presented; only 2-D or 3-D clustering problems testing the accuracy of the evaluation set. After
can be visually inspected. We will rely heavily on the cluster centers are determined, the evaluation
performance measures to evaluate the clustering data vectors are assigned to their respective
techniques rather than on visual approaches. clusters according to the distance between each
As mentioned earlier, the similarity metric vector and each of the cluster centers. An error
used to calculate the similarity between an input measure is then calculated; the root mean square
vector and a cluster center is the Euclidean error (RMSE) is used for this purpose. Also an
distance. Since most similarity metrics are accuracy measure is calculated as the percentage
sensitive to the large ranges of elements in the of correctly classified vectors. The algorithm was
input vectors, each of the input variables must be tested for 10 times to determine the best
normalized to within the unit interval [ 0,1] ; i.e. performance. Table 1 lists the results of those
runs. Figure 1 shows a plot of the cost function
the data set has to be normalized to be within the
over time for the best test case.
unit hypercube.
Each clustering algorithm is presented with
the training data set, and as a result two clusters
K-means clustering cost function history
450

are produced. The data in the evaluation set is

then tested against the found clusters and an 400

analysis of the results is conducted. The

following sections present the results of each 350
Cost Function

clustering technique, followed by a comparison of

the four techniques. 300

MATLAB code for each of the four

techniques can be found in the appendix. 250

A. K-means Clustering
200
1 2 3 4 5 6 7 8 9 10 11
Iteration

As mentioned in the previous section, K-

means clustering works on finding the cluster Figure 1. K-means clustering cost function plot
centers by trying to minimize a cost function J . It
alternates between updating the membership
matrix and updating the cluster centers using
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 7

Performance Weighting exponent m

measure 1.1 1.2 1.5 2 3 5 8 12
No. of iterations 18 19 17 19 26 29 33 36
RMSE 0.469 0.469 0.479 0.469 0.458 0.479 0.479 0.479
Accuracy 78.0% 78.0% 77.0% 78.0% 79.0% 77.0% 77% 77%
Regression Line
0.559 0.559 0.539 0.559 0.579 0.539 0.539 0.539
Slope
Table 2. Fuzzy C-means Clustering Performance Results

1
Best Linear Fit: A = (0.604) T + (0.214)
Data Points B. Fuzzy C-means Clustering
A=T
Best Linear Fit

FCM allows for data points to have

different degrees of membership to each of the
clusters; thus eliminating the effect of hard
0.5 membership introduced by K-means clustering.
A

This approach employs fuzzy measures as the

basis for membership matrix calculation and for
R = 0.6
cluster centers identification.
As it is the case in K-means clustering,
0
FCM starts by assigning random values to the
membership matrix U , thus several runs have to
0 0.5 1
T

Figure 2. Regression Analysis of K-means Clustering be conducted to have higher probability of getting
good performance. However, the results showed
no (or insignificant) variation in performance or
To further measure how accurately the
accuracy when the algorithm was run for several
identified clusters represent the actual
times.
classification of data, a regression analysis is
For testing the results, every vector in the
performed of the resultant clustering against the
evaluation data set is assigned to one of the
original classification. Performance is considered
clusters with a certain degree of belongingness
better if the regression line slope is close to 1.
(as done in the training set). However, because
Figure 2 shows the regression analysis of the best
the output values we have are crisp values (either
test case.
1 or 0), the evaluation set degrees of membership
As seen from the results, the best case
are defuzzified to be tested against the actual
achieved 80% accuracy and an RMSE of 0.447.
outputs.
This relatively moderate performance is related to
The same performance measures applied in
the high dimensionality of the problem; having
K-means clustering will be used here; however
too much dimensions tend to disrupt the coupling
only the effect of the weighting exponent m is
of data and introduces overlapping in some of
analyzed, since the effect of random initial
these dimensions that reduces the accuracy of
membership grades has insignificant effect on the
clustering. It is noticed also that the cost function
final cluster centers. Table 2 lists the results of
converges rapidly to a minimum value as seen
the tests with the effect of varying the weighting
from the number of iterations in each test run.
exponent m . It is noticed that very low or very
However, this has no effect on the accuracy
measure. high values for m reduces the accuracy;
moreover high values tend to increase the time
taken by the algorithm to find the clusters. A
value of 2 seems adequate for this problem since
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 8

Performance Test Runs

measure 1 2 3 4 5 6 7 8 9 10
RMSE 0.566 0.469 0.566 0.49 0.548 0.566 0.566 0.529 0.693 0.469
Accuracy 68.0% 78.0% 68.0% 76.0% 70.0% 68.0% 68.0% 72.0% 52.0% 78.0%
Regression Line
0.351 0.556 0.343 0.515 0.428 0.345 0.345 0.492 0.026 0.551
Slope

Table 3. Mountain Clustering Performance Results

it has good accuracy and requires less number of points, and a grid size of g per dimension, the
iterations. Figure 3 shows the accuracy and required number of calculations is:
number of iterations against the weighting factor. N = m g n + (c 1) g n (12)
1st cluster remainder clusters
Performance of Fuzzy C-means Clustering
100

So for the problem at hand, with input data of 13-

Accuracy
Iterations
90

80
dimensions, 200 training inputs, and a grid size of
70
10 per dimension, the required number of
60
mountain function calculation is approximately
50
2.01 1015 calculations. In addition the value of
40
the mountain function needs to be stored for
30
every grid point for later calculations in finding
20
subsequent clusters; which requires g n storage
10 locations, for our problem this would be 1013
0
0 1 2 3 4 5 6 7 8 9 10 11 12
storage locations. Obviously this seems
m
impractical for a problem of this dimension.
In order to be able to test this algorithm, the
Figure 3. Fuzzy C-means Clustering Performance
dimension of the problem have to be reduced to a
reasonable number; e.g. 4-dimensions. This is
In general, the FCM technique showed no achieved by randomly selecting 4 variables from
improvement over the K-means clustering for this the input data out of the original 13 and
problem. Both showed close accuracy; moreover performing the test on those variables. Several
FCM was found to be slower than K-means tests involving differently selected random
because of fuzzy calculations. variables are conducted in order to have a better
understanding of the results. Table 3 lists the
C. Mountain Clustering results of 10 test runs of randomly selected
variables. The accuracy achieved ranged between
Mountain clustering relies on dividing the 52% and 78% with an average of 70%, and
data space into grid points and calculating a average RMSE of 0.546. Those results are quite
mountain function at every grid point. This discouraging compared to the results achieved in
mountain function is a representation of the K-means and FCM clustering. This is due to the
density of data at this point. fact that not all of the variables of the input data
The performance of mountain clustering is contribute to the clustering process; only 4 are
severely affected by the dimension of the chosen at random to make it possible to conduct
problem; the computation needed rises the tests. However, with only 4 attributes chosen
exponentially with the dimension of input data to do the tests, mountain clustering required far
because the mountain function has to be much more time than any other technique during
evaluated at each grid point in the data space. For the tests; this is because of the fact that the
a problem with c clusters, n dimensions, m data
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 9

Performance Neighborhood radius ra

measure 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
RMSE 0.67 0.648 0.648 0.5 0.5 0.5 0.5 0.5 0.648
Accuracy 55.0% 58.0% 58.0% 75.0% 75.0% 75.0% 75.0% 75.0% 58.0%
Regression Line
0.0993 0.1922 0.1922 0.5074 0.5074 0.5074 0.5074 0.5074 0.1922
Slope

Table 4. Subtractive Clustering Performance Results

number of computation required is varying ra . Figure 4 shows a plot of accuracy and

exponentially proportional to the number of RMSE against ra .
dimensions in the
problem, as stated in Equation (12). So
apparently mountain clustering is not suitable for
Subtractive Clustering Performance
100
Accuracy

problems of dimensions higher than two or three. 90

RMSE

80

D. Subtractive Clustering 70

60

This method is similar to mountain

Accuracy
50

clustering, with the difference that a density 40

function is calculated only at every data point, 30

instead of at every grid point. So the data points 20

themselves are the candidates for cluster centers. 10

This has the effect of reducing the number of

0
computations significantly, making it linearly 0.1 0.2 0.3 0.4 0.5
ra
0.6 0.7 0.8 0.9 1

proportional to the number of input data instead

of being exponentially proportional to its Figure 4. Subtractive Clustering Performance
dimension. For a problem of c clusters and m
data points, the required number of calculations It is clear from the results that choosing ra
is: very small or very large will result in a poor
N = m2 + (c 1)m (13) accuracy because if ra is chosen very small the
1st cluster
remainder clusters
density function will not take into account the
As seen from the equation, the number of effect of neighboring data points; while if taken
calculations does not depend on the dimension of very large, the density function will be affected
the problem. For the problem at hand, the number account all the data points in the data space. So a
of computations required is in the range of few value between 0.4 and 0.7 should be adequate for
ten thousands only. the radius of neighborhood.
Since the algorithm is fixed and does not As seen from table 4, the maximum
rely on any randomness, the results are fixed. achieved accuracy was 75% with an RMSE of
However, we can test the effect of the two 0.5. Compared to K-means and FCM, this result
is a little bit behind the accuracy achieved in
variables ra and rb on the accuracy of the
those other techniques.
algorithm. Those variables represent a radius of
neighborhood after which the effect (or E. Results Summary and Comparison
contribution) of other data points to the density
function is diminished. Usually the rb variable is According to the previous discussion of the
taken to be as 1.5ra . Table 4 shows the results of implementation of the four data clustering
techniques and their results, it is useful to
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 10

summarize the results and present some

comparison of performances. As stated earlier in this paper, clustering
A summary of the best achieved results for each algorithms are usually used in conjunction with
of the four techniques is presented in Table 5. radial basis function networks and fuzzy models.
The techniques described here can be used as
Comparison Aspect preprocessors for RBF networks for determining
Algorithm Regression Time the centers of the radial basis functions. In such
RMSE Accuracy
Line Slope (sec) cases, more accuracy can be gained by using
K-means 0.447 80.0% 0.6 0.9 gradient descent or other advanced derivative-
FCM 0.469 78.0% 0.559 2.2 based optimization schemes for further
Mountain 0.469 78.0% 0.556 118.0 refinement. In fuzzy modeling, the clustering
Subtractive 0.5 75.0% 0.5074 3.6 cluster centers produced by the clustering
techniques can be modeled as if-then rules in
Table 5. Performance Results Comparison ANFIS for example; where a training set
(including inputs and outputs) to find cluster
From this comparison we can conclude some centers (xi , yi ) via clustering first and then
remarks: forming a zero-order Sugeno fuzzy modeling in
K-means clustering produces fairly higher which the ith rule is expressed as
accuracy and lower RMSE than the other If X is close to xi then Y is close to yi
techniques, and requires less computation Which means that the ith rule is based on the ith
time. cluster center identified by the clustering method.
Mountain clustering has a very poor Again, after the structure is determined,
performance regarding its requirement for backpropagation-type gradient descent and other
huge number of computation and low optimization schemes can be applied to proceed
accuracy. However, we have to notice that with parameter identification.
tests conducted on mountain clustering were
done using part of the input variables in order
to make it feasible to run the tests. V. CONCLUSION
Mountain clustering is suitable only for
problems with two or three dimensions.
Four clustering techniques have been
FCM produces close results to K-means
reviewed in this paper, namely: K-means
clustering, yet it requires more computation
clustering, Fuzzy C-means clustering, Mountain
time than K-means because of the fuzzy
clustering, and Subtractive clustering. These
measures calculations involved in the
approaches solve the problem of categorizing
algorithm.
data by partitioning a data set into a number of
In subtractive clustering, care has to be taken
clusters based on some similarity measure so that
when choosing the value of the neighborhood
the similarity in each cluster is larger than among
radius ra , since too small radii will result in clusters. The four methods have been
neglecting the effect of neighboring data implemented and tested against a data set for
points, while large radii will result in a medical diagnosis of heart disease. The
neighborhood of all the data points thus comparative study done here is concerned with
canceling the effect of the cluster. the accuracy of each algorithm, with care being
Since non of the algorithms achieved enough taken toward the efficiency in calculation and
high accuracy rates, it is assumed that the other performance measures. The medical
problem data itself contains some overlapping problem presented has a high number of
in some of the dimensions; because of the high dimensions, which might involve some
number of dimensions tend to disrupt the complicated relationships between the variables
coupling of data and reduce the accuracy of in the input data. It was obvious that mountain
clustering.
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 11

clustering is not one of the good techniques for

problems with this high number of dimensions
due to its exponential proportionality to the
dimension of the problem. K-means clustering
seemed to over perform the other techniques for
this type of problem. However in other problems
where the number of clusters is not known, K-
means and FCM cannot be used to solve this type
of problem, leaving the choice only to mountain
or subtractive clustering. Subtractive clustering
seems to be a better alternative to mountain
clustering since it is based on the same idea, and
uses the data points as cluster centers candidates
instead of grid points; however, mountain
clustering can lead to better results if the grid
granularity is small enough to capture the
potential cluster centers, but with the side effect
of increasing computation needed for the larger
number of grid points.
Finally, the clustering techniques discussed
here do not have to be used as stand-alone
approaches; they can be used in conjunction with
other neural or fuzzy systems for further
refinement of the overall system performance.

VI. REFERENCES

[1] Jang, J.-S. R., Sun, C.-T., Mizutani, E., Neuro-

Fuzzy and Soft Computing A Computational
Approach to Learning and Machine Intelligence,
Prentice Hall.
[2] Azuaje, F., Dubitzky, W., Black, N., Adamson, K.,
Discovering Relevance Knowledge in Data: A
Growing Cell Structures Approach, IEEE
Transactions on Systems, Man, and Cybernetics-
Part B: Cybernetics, Vol. 30, No. 3, June 2000 (pp.
448)
[3] Lin, C., Lee, C., Neural Fuzzy Systems, Prentice
Hall, NJ, 1996.
[4] Tsoukalas, L., Uhrig, R., Fuzzy and Neural
Approaches in Engineering, John Wiley & Sons,
Inc., NY, 1997.
[5] Nauck, D., Kruse, R., Klawonn, F., Foundations of
Neuro-Fuzzy Systems, John Wiley & Sons Ltd.,
NY, 1997.
[6] J. A. Hartigan and M. A. Wong, A k-means
clustering algorithm, Applied Statistics, 28:100--
108, 1979.
[7] The MathWorks, Inc., Fuzzy Logic Toolbox For
Use With MATLAB, The MathWorks, Inc., 1999.
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 12

Appendix

K-means Clustering (MATLAB script)

% K-means clustering

% ------------------- CLUSTERING PHASE -------------------

% Load the Training Set
TrSet = load('TrainingSet.txt');
[m,n] = size(TrSet); % (m samples) x (n dimensions)
for i = 1:m % the output (last column) values (0,1,2,3) are mapped to (0,1)
if TrSet(i,end)>=1
TrSet(i,end)=1;
end
end

% find the range of each attribute (for normalization later)

for i = 1:n
range(1,i) = min(TrSet(:,i));
range(2,i) = max(TrSet(:,i));
end

x = Normalize(TrSet, range); % normalize the data set to a hypercube

x(:,end) = []; % get rid of the output column
[m,n] = size(x);

nc = 2; % number of clusters = 2

% Initialize cluster centers to random points

c = zeros(nc,n);
for i = 1:nc
rnd = int16(rand*m + 1); % select a random vector from the input set
c(i,:) = x(rnd,:); % assign this vector value to cluster (i)
end

% Clustering Loop

delta = 1e-5;
n = 1000;
iter = 1;
while (iter < n)

% Determine the membership matrix U

% u(i,j) = 1 if euc_dist(x(j),c(i)) <= euc_dist(x(j),c(k)) for each k ~= i
% u(i,j) = 0 otherwise

for i = 1:nc
for j = 1:m
d = euc_dist(x(j,:),c(i,:));
u(i,j) = 1;
for k = 1:nc
if k~=i
if euc_dist(x(j,:),c(k,:)) < d
u(i,j) = 0;
end
end
end
end
end

% Compute the cost function J

J(iter) = 0;
for i = 1:nc
JJ(i) = 0;
for k = 1:m
if u(i,k)==1
JJ(i) = JJ(i) + euc_dist(x(k,:),c(i,:));
end
end
J(iter) = J(iter) + JJ(i);
end
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 13

% Stop if either J is below a certain tolerance value,

% or its improvement over previous iteration is below a certain threshold
str = sprintf('iteration: %.0d, J=%d', iter, J(iter));
disp(str);
if (iter~=1) & (abs(J(iter-1) - J(iter)) < delta)
break;
end

% Update the cluster centers

% c(i) = mean of all vectors belonging to cluster (i)

for i = 1:nc
sum_x = 0;
G(i) = sum(u(i,:));
for k = 1:m
if u(i,k)==1
sum_x = sum_x + x(k,:);
end
end
c(i,:) = sum_x ./ G(i);
end

iter = iter + 1;
end % while
disp('Clustering Done.');

% ----------------- TESTING PHASE --------------------------

% Load the evaluation data set

EvalSet = load('EvaluationSet.txt');
[m,n] = size(EvalSet);
for i = 1:m
if EvalSet(i,end)>=1
EvalSet(i,end)=1;
end
end

x = Normalize(EvalSet, range);
x(:,end) = [];
[m,n] = size(x);

% Assign evaluation vectors to their respective clusters according

% to their distance from the cluster centers

for i = 1:nc
for j = 1:m
d = euc_dist(x(j,:),c(i,:));
evu(i,j) = 1;
for k = 1:nc
if k~=i
if euc_dist(x(j,:),c(k,:)) < d
evu(i,j) = 0;
end
end
end
end
end

% Analyze results
ev = EvalSet(:,end)';
rmse(1) = norm(evu(1,:)-ev)/sqrt(length(evu(1,:)));
rmse(2) = norm(evu(2,:)-ev)/sqrt(length(evu(2,:)));

subplot(2,1,1);
if rmse(1) < rmse(2)
r = 1;
else
r = 2;
end

str = sprintf('Testing Set RMSE: %f', rmse(r));

disp(str);
ctr = 0;
for i = 1:m
if evu(r,i)==ev(i)
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 14

ctr = ctr + 1;
end
end
str = sprintf('Testing Set accuracy: %.2f%%', ctr*100/m);
disp(str);
[m,b,r] = postreg(evu(r,:),ev); % Regression Analysis
disp(sprintf('r = %.3f', r));

Fuzzy C-means Clustering (MATLAB script)

% Fuzzy C-means clustering

% ------------------- CLUSTERING PHASE -------------------

% Load the Training Set
TrSet = load('TrainingSet.txt');
[m,n] = size(TrSet); % (m samples) x (n dimensions)
for i = 1:m % the output (last column) values (0,1,2,3) are mapped to (0,1)
if TrSet(i,end)>=1
TrSet(i,end)=1;
end
end

% find the range of each attribute (for normalization later)

for i = 1:n
range(1,i) = min(TrSet(:,i));
range(2,i) = max(TrSet(:,i));
end

x = Normalize(TrSet, range); % normalize the data set to a hypercube

x(:,end) = []; % get rid of the output column
[m,n] = size(x);

nc = 2; % number of clusters = 2

% Initialize the membership matrix with random values between 0 and 1

% such that the summation of membership degrees for each vector equals unity
u = zeros(nc,m);
for i = 1:m
u(1,i) = rand;
u(2,i) = 1 - u(1,i);
end

% Clustering Loop

m_exp = 12;
prevJ = 0;
J = 0;
delta = 1e-5;
n = 1000;
iter = 1;
while (iter < n)

% Calculate the fuzzy cluster centers

for i = 1:nc
sum_ux = 0;
sum_u = 0;
for j = 1:m
sum_ux = sum_ux + (u(i,j)^m_exp)*x(j,:);
sum_u = sum_u + (u(i,j)^m_exp);
end
c(i,:) = sum_ux ./ sum_u;
end

% Compute the cost function J

J(iter) = 0;
for i = 1:nc
JJ(i) = 0;
for j = 1:m
JJ(i) = JJ(i) + (u(i,j)^m_exp)*euc_dist(x(j,:),c(i,:));
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 15

end
J(iter) = J(iter) + JJ(i);
end

% Stop if either J is below a certain tolerance value,

% or its improvement over previous iteration is below a certain threshold
str = sprintf('iteration: %.0d, J=%d', iter, J);
disp(str);
if (iter~=1) & (abs(J(iter-1) - J(iter)) < delta)
break;
end

% Update the membership matrix U

for i = 1:nc
for j = 1:m
sum_d = 0;
for k = 1:nc
sum_d = sum_d + (euc_dist(c(i,:),x(j,:))/euc_dist(c(k,:),x(j,:)))^(2/(m_exp-1));
end
u(i,j) = 1/sum_d;
end
end

iter = iter + 1;
end % while
disp('Clustering Done.');

% ----------------- TESTING PHASE --------------------------

% Load the evaluation data set

EvalSet = load('EvaluationSet.txt');
[m,n] = size(EvalSet);
for i = 1:m
if EvalSet(i,end)>=1
EvalSet(i,end)=1;
end
end

x = Normalize(EvalSet, range);
x(:,end) = [];
[m,n] = size(x);

% Assign evaluation vectors to their respective clusters according

% to their distance from the cluster centers

for i = 1:nc
for j = 1:m
sum_d = 0;
for k = 1:nc
sum_d = sum_d + (euc_dist(c(i,:),x(j,:))/euc_dist(c(k,:),x(j,:)))^(2/(m_exp-1));
end
evu(i,j) = 1/sum_d;
end
end

% defuzzify the membership matrix

for j = 1:m
if evu(1,j) >= evu(2,j)
evu(1,j) = 1; evu(2,j) = 0;
else
evu(1,j) = 0; evu(2,j) = 1;
end
end

% Analyze results
ev = EvalSet(:,end)';
rmse(1) = norm(evu(1,:)-ev)/sqrt(length(evu(1,:)));
rmse(2) = norm(evu(2,:)-ev)/sqrt(length(evu(2,:)));

subplot(2,1,1);
if rmse(1) < rmse(2)
r = 1;
else
r = 2;
end
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 16

str = sprintf('Testing Set RMSE: %f', rmse(r));

disp(str);
ctr = 0;
for i = 1:m
if evu(r,i)==ev(i)
ctr = ctr + 1;
end
end
str = sprintf('Testing Set accuracy: %.2f%%', ctr*100/m);
disp(str);
[m,b,r] = postreg(evu(r,:),ev); % Regression Analysis
disp(sprintf('r = %.3f', r));

Mountain Clustering (MATLAB script)

% Mountain Clustering

%------------------------------------------------------------------
% Setup the training data
%------------------------------------------------------------------

% Load the Training Set

TrSet = load('TrainingSet.txt');
[m,n] = size(TrSet); % (m samples) x (n dimensions)
for i = 1:m % the output (last column) values (0,1,2,3) are mapped to (0,1)
if TrSet(i,end)>=1
TrSet(i,end)=1;
end
end

% find the range of each attribute (for normalization later)

for i = 1:n
range(1,i) = min(TrSet(:,i));
range(2,i) = max(TrSet(:,i));
end

x = Normalize(TrSet, range); % normalize the data set to a hypercube

x(:,end) = []; % get rid of the output column
[m,n] = size(x);

% Due to memory and speed limitations, the number of attributes

% will be set to a maximum of 4 attributes. Extra attributes will
% be dropped at random.
n_dropped = 0;
if n>4
for i = 1:(n-4)
attr = ceil(rand*(n-i+1));
x(:,attr) = [];
dropped(i) = attr; % save dropped attributes positions
n_dropped = n_dropped+1;
end
end
[m,n] = size(x);

%------------------------------------------------------------------
% First: setup a grid matrix of n-dimensions (V)
% (n = the dimension of input data vectors)
% The gridding granularity is 'gr' = # of grid points per dimension
%------------------------------------------------------------------

gr = 10;
% setup the dimension vector [d1 d2 d3 .... dn]
v_dim = gr * ones([1 n]);
% setup the mountain matrix
M = zeros(v_dim);
sigma = 0.1;

%------------------------------------------------------------------
% Second: calculate the mountain function at every grid point
%------------------------------------------------------------------
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 17

% setup some aiding variables

cur = ones([1 n]);
for i = 1:n
for j = 1:i
cur(i) = cur(i)*v_dim(j);
end
end
max_m = 0; % greatest density value
max_v = 0; % cluster center position

disp('Finding Cluster 1...');

% loop over each grid point
for i = 1:cur(1,end)

% calculate the vector indexes

idx = i;
for j = n:-1:2
dim(j) = ceil(idx/cur(j-1));
idx = idx - cur(j-1)*(dim(j)-1);
end
dim(1) = idx;
% dim is holding the current point index vector
% but needs to be normalized to the range [0,1]
v = dim./gr;

% calculate the mountain function for the current point

M(i) = mnt(v,x,sigma);
if M(i) > max_m
max_m = M(i);
max_v = v;
max_i = i;
end

% report progress
if mod(i,5000)==0
str = sprintf('vector %.0d/%.0d; M(v)=%.2f', i, cur(1,end), M(i));
disp(str);
end

end

%------------------------------------------------------------------
% Third: select the first cluster center by choosing the point
% with the greatest density value
%------------------------------------------------------------------

c(1,:) = max_v;
c1 = max_i;
str = sprintf('Cluster 1:');
disp(str);
str = sprintf('%4.1f', c(1,:));
disp(str);
str = sprintf('M=%.3f', max_m);
disp(str);

%------------------------------------------------------------------
% CLUSTER 2
%------------------------------------------------------------------

Mnew = zeros(v_dim);
max_m = 0;
max_v = 0;
beta = 0.1;

disp('Finding Cluster 2...');

for i = 1:cur(1,end)

% calculate the vector indexes

idx = i;
for j = n:-1:2
dim(j) = ceil(idx/cur(j-1));
idx = idx - cur(j-1)*(dim(j)-1);
end
dim(1) = idx;
% dim is holding the current point index vector
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 18

% but needs to be normalized to the range [0,1]

v = dim./gr;

% calculate the REVISED mountain function for the current point

Mnew(i) = M(i) - M(c1)*exp((-euc_dist(v,c(1,:)))./(2*beta^2));

if Mnew(i) > max_m

max_m = Mnew(i);
max_v = v;
max_i = i;
end

% report progress
if mod(i,5000)==0
str = sprintf('vector %.0d/%.0d; Mnew(v)=%.2f', i, cur(1,end), Mnew(i));
disp(str);
end

end

c(2,:) = max_v;
str = sprintf('Cluster 2:');
disp(str);
str = sprintf('%4.1f', c(2,:));
disp(str);
str = sprintf('M=%.3f', max_m);
disp(str);

%------------------------------------------------------------------
% Evaluation
%------------------------------------------------------------------

% Load the evaluation data set

EvalSet = load('EvaluationSet.txt');
[m,n] = size(EvalSet);
for i = 1:m
if EvalSet(i,end)>=1
EvalSet(i,end)=1;
end
end

x = Normalize(EvalSet, range);
x(:,end) = [];
[m,n] = size(x);

% drop the attributes corresponding to the ones dropped in the training set
for i = 1:n_dropped
x(:,dropped(i)) = [];
end
[m,n] = size(x);

% Assign every test vector to its nearest cluster

for i = 1:2
for j = 1:m
d = euc_dist(x(j,:),c(i,:));
evu(i,j) = 1;
for k = 1:2
if k~=i
if euc_dist(x(j,:),c(k,:)) < d
evu(i,j) = 0;
end
end
end
end
end

% Analyze results
ev = EvalSet(:,end)';
rmse(1) = norm(evu(1,:)-ev)/sqrt(length(evu(1,:)));
rmse(2) = norm(evu(2,:)-ev)/sqrt(length(evu(2,:)));

if rmse(1) < rmse(2)

r = 1;
else
r = 2;
end
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 19

str = sprintf('Testing Set RMSE: %f', rmse(r));

disp(str);
ctr = 0;
for i = 1:m
if evu(r,i)==ev(i)
ctr = ctr + 1;
end
end
str = sprintf('Testing Set accuracy: %.2f%%', ctr*100/m);
disp(str);
[m,b,r] = postreg(evu(r,:),ev);
disp(sprintf('r = %.3f', r));

Subtractive Clustering (MATLAB script)

% Subtractive Clustering

%------------------------------------------------------------------
% Setup the training data
%------------------------------------------------------------------

% Load the Training Set

TrSet = load('TrainingSet.txt');
[m,n] = size(TrSet); % (m samples) x (n dimensions)
for i = 1:m % the output (last column) values (0,1,2,3) are mapped to (0,1)
if TrSet(i,end)>=1
TrSet(i,end)=1;
end
end

% find the range of each attribute (for normalization later)

for i = 1:n
range(1,i) = min(TrSet(:,i));
range(2,i) = max(TrSet(:,i));
end

x = Normalize(TrSet, range); % normalize the data set to a hypercube

x(:,end) = []; % get rid of the output column
[m,n] = size(x);

%------------------------------------------------------------------
% First: Initialize the density matrix and some variables
%------------------------------------------------------------------

D = zeros([m 1]);
ra = 1.0;

%------------------------------------------------------------------
% Second: calculate the density function at every data point
%------------------------------------------------------------------

% setup some aiding variables

max_d = 0; % greatest density value
max_x = 0; % cluster center position

disp('Finding Cluster 1...');

% loop over each data point
for i = 1:m

% calculate the density function for the current point

D(i) = density(x(i,:),x,ra);
if D(i) > max_d
max_d = D(i);
max_x = x(i,:);
max_i = i;
end

% report progress
if mod(i,50)==0
str = sprintf('vector %.0d/%.0d; D(v)=%.2f', i, m, D(i));
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 20

% disp(str);
end

end

%------------------------------------------------------------------
% Third: select the first cluster center by choosing the point
% with the greatest density value
%------------------------------------------------------------------

c(1,:) = max_x;
c1 = max_i;
str = sprintf('Cluster 1:');
disp(str);
str = sprintf('%4.1f', c(1,:));
disp(str);
str = sprintf('D=%.3f', max_d);
disp(str);

%------------------------------------------------------------------
% CLUSTER 2
%------------------------------------------------------------------

Dnew = zeros([m 1]);

max_d = 0;
max_x = 0;
rb = 1.5*ra;

disp('Finding Cluster 2...');

for i = 1:m

% calculate the REVISED density function for the current point

Dnew(i) = D(i) - D(c1)*exp((-euc_dist(x(i,:),c(1,:)))./((rb/2)^2));

if Dnew(i) > max_d

max_d = Dnew(i);
max_x = x(i,:);
max_i = i;
end

% report progress
if mod(i,50)==0
str = sprintf('vector %.0d/%.0d; Dnew(v)=%.2f', i, m, Dnew(i));
disp(str);
end

end

c(2,:) = max_x;
str = sprintf('Cluster 2:');
disp(str);
str = sprintf('%4.1f', c(2,:));
disp(str);
str = sprintf('D=%.3f', max_d);
disp(str);

%------------------------------------------------------------------
% Evaluation
%------------------------------------------------------------------

% Load the evaluation data set

EvalSet = load('EvaluationSet.txt');
[m,n] = size(EvalSet);
for i = 1:m
if EvalSet(i,end)>=1
EvalSet(i,end)=1;
end
end

x = Normalize(EvalSet, range);
x(:,end) = [];
[m,n] = size(x);

% Assign every test vector to its nearest cluster

for i = 1:2
for j = 1:m
A COMPARATIVE STUDY OF DATA CLUSTERING TECHNIQUES 21

dd = euc_dist(x(j,:),c(i,:));
evu(i,j) = 1;
for k = 1:2
if k~=i
if euc_dist(x(j,:),c(k,:)) < dd
evu(i,j) = 0;
end
end
end
end
end

% Analyze results
ev = EvalSet(:,end)';
rmse(1) = norm(evu(1,:)-ev)/sqrt(length(evu(1,:)));
rmse(2) = norm(evu(2,:)-ev)/sqrt(length(evu(2,:)));

if rmse(1) < rmse(2)

r = 1;
else
r = 2;
end

str = sprintf('Testing Set RMSE: %f', rmse(r));

disp(str);
ctr = 0;
for i = 1:m
if evu(r,i)==ev(i)
ctr = ctr + 1;
end
end
str = sprintf('Testing Set accuracy: %.2f%%', ctr*100/m);
disp(str);
[m,b,r] = postreg(evu(r,:),ev);
disp(sprintf('r = %.3f', r));