Syracuse University

SUrface
L.C. Smith College of Engineering and Computer
Electrical Engineering and Computer Science
Science

1-1-1997

An efficient k-means clustering algorithm

Khaled Alsabti
Syracuse University

Sanjay Ranka
University of Florida

Vineet Singh
Hitachi America, Ltd.

Recommended Citation
Alsabti, Khaled; Ranka, Sanjay; and Singh, Vineet, "An efficient k-means clustering algorithm" (1997). Electrical Engineering and
Computer Science. Paper 43.
http://surface.syr.edu/eecs/43

This Working Paper is brought to you for free and open access by the L.C. Smith College of Engineering and Computer Science at SUrface. It has been
accepted for inclusion in Electrical Engineering and Computer Science by an authorized administrator of SUrface. For more information, please
contact [email protected].
 An Efficient K-Means Clustering Algorithm

Khaled Alsabti Sanjay Ranka Vineet Singh

Syracuse University University of Florida Hitachi America, Ltd.

Abstract parable (due to the round-off errors) clustering results to the

direct k-means algorithm. It has significantly superior per-
In this paper, we present a novel algorithm for perform- formance than the direct k-means algorithm in most cases.
ing k-means clustering. It organizes all the patterns in a k-d The rest of this paper is organized as follows. We review
tree structure such that one can find all the patterns which previously proposed approaches for improving the perfor-
are closest to a given prototype efficiently. The main intu- mance of the k-means algorithms in Section 2. We present
ition behind our approach is as follows. All the prototypes our algorithm in Section 3. We describe the experimental
are potential candidates for the closest prototype at the root results in Section 4 and we conclude with Section 5.
level. However, for the children of the root node, we may be
able to prune the candidate set by using simple geometrical 2 k-means Clustering
constraints. This approach can be applied recursively until
the size of the candidate set is one for each node.
Our experimental results demonstrate that our scheme In this section, we briefly describe the direct 
k-means
can improve the computational speed of the direct k-means algorithm [9, 8, 3]. The number of clusters  is assumed
algorithm by an order to two orders of magnitude in the to be fixed in k-means clustering. Let the prototypes
 



total number of distance calculations and the overall time 



be initialized to one of the  input patterns
of computation. .1 Therefore,
 "!$#&%'


 )( +*,!-#%'


. (


1. Introduction Figure 1 shows a high level description of the direct k-

 
means clustering algorithm. / is the th cluster whose
Clustering is the process of partitioning or grouping a value is a disjoint subset of input patterns. The quality of
given set of patterns into disjoint clusters. This is done such the clustering is determined by the following error function:
that patterns in the same cluster are alike and patterns be- 

longing to two different clusters are different. Clustering 0  1 1 )=>

has been a widely studied problem in a variety of applica- 32 
46587'9;:< <?
tion domains including neural networks, AI, and statistics.
Several algorithms have been proposed in the literature 
The appropriate choice of is problem and domain 
de-
for clustering: ISODATA [8, 3], CLARA [8], CLARANS
pendent and generally a user tries several values of . As-
[10], Focusing Techniques [5] P-CLUSTER [7]. DBSCAN
suming that there are  patterns, each of dimension @ , the
[4], Ejcluster [6], BIRCH [14] and GRIDCLUS [12].
computational cost of a direct k-means algorithm per itera-
tion (of the repeat loop) can be decomposed into three parts:
The k-means method has been shown to be effective in
producing good clustering results for many practical appli- 1. The time required for the first for loop in Figure 1 is
A   
cations. However, a direct algorithm of k-means method  @ .
requires time proportional to the product of number of pat-
terns and number of clusters per iteration. This is computa- 2. The time required for calculating
A  
the centroids (second
tionally very expensive especially for large datasets. for loop in Figure 1) is B@ .
We propose a novel algorithm for implementing the k-
means method. Our algorithm produces the same or com- 3. The
A 
time

required for calculating the error function is
B@ .


This work was supported by the Information Technology Lab (ITL) of
Hitachi America, Ltd. while K. Alsabti and S. Ranka were visiting ITL. 1 Our strategy is independent of this initialization.
 The number of iterations required can vary in a wide pattern becomes more efficient [11, 13]. This prob-
range from a few to several thousand depending on the lem reduces to finding the nearest neighbor problem
number of patterns, number of clusters, and the input data for a given pattern in the prototype space. The number
distribution. Thus, a direct implementation of the k-means of distance calculations
   
using this approach is propor-
method can be computationally very intensive. This is es- tional to GFIH @ per iteration. For many applica-
pecially true for typical data mining applications with large tions such as vector quantization, the prototype vectors
number of pattern vectors. are fixed. This allows for construction of optimal data
structures to find the closest vector for a given input
function Direct-k-means() test pattern [11]. However, these optimizations are not
applicable to the k-means algorithm as the prototype
Initialize
 prototypes
  
   such
 that 
  
 "!#$ 
 &%
vectors will change dynamically. Further, it is not clear
Each cluster '" is associated with prototype ( how these optimizations can be used to reduce the time
for calculation of the error function (which becomes a
Repeat
)  substantial component after reduction in the number of
for each input vector , where !*+ 
&% ,
do distance calculations).

Assign to the cluster ',
with near-
est prototype
 /. *
  .
(i.e., - /  
-102-   - 34 3 Our Algorithm
 
  ) /
for each cluster '" , where 45
   , do
Update the prototype * to be the The main intuition behind our approach is as follows.
centroid of all samples currently > All the prototypes are potential candidates for the closest
in '" , so that *687:9 5<;5=: -
prototype at the root level. However, for the children of the
'"?-
root node, we may be able to prune the candidate set by
Compute the error function:
using simple geometrical constraints. Clearly, each child

@
1 1
 . node will potentially have different candidate sets. Further,
 - C- D
5<;5=:
a given prototype may belong to the candidate set of several

AB 9
child nodes. This approach can be applied recursively till
@
Until does not change significantly or cluster mem- the size of the candidate set is one for each node. At this
bership no longer changes stage, all the patterns in the subspace represented by the
subtree have the sole candidate as their closest prototype.
Using this approach, we expect that the number of distance
Figure 1. Direct k-means clustering algorithm calculation for the    
first loop (in  Figure
 
1) will be propor-
tional
   
to JFLK @ where K @ is much smaller than
H @ . This is because the distance calculation has to be

There are two main approaches described in the litera- performed only with internal nodes (representing many pat-
ture which can be used to reduce the overall computational terns) and not the patterns themselves in most cases. This
requirements of the k-means clustering method especially approach can also be used to significantly reduce the time
for the distance calculations: requirements for calculating the prototypes for the next it-
eration (second for loop in Figure 1). We also expect the
1. Use the information from the previous iteration to time requirement for the second for loop to be proportional
   
reduce the number of distance calculations. P- to LFMK @ .
CLUSTER is a k-means-based clustering algorithm The improvements obtained using our approach are cru-
which exploits the fact that the change of the assign- cially dependent on obtaining good pruning methods for ob-
ment of patterns to clusters are relatively few after the taining candidate sets for the next level. We propose to use
first few iterations [7]. It uses a heuristic which deter- the following strategy.
mines if the closest prototype of a pattern E has been 
N
changed or not by using a simple check. If the assign- For each candidate 4 , find the minimum and maxi-
ment has not changed, no further distance calculations mum distances to any point in the subspace
are required. It also uses the fact that the movement of
N
the cluster centroids is small for consecutive iterations Find

the minimum of maximum distances, call it
O OQP/R
(especially after a few iterations). 

N
2. Organize the prototype vectors in a suitable data struc- Prune out all  candidates with minimum distance
O OQP/R
ture so that finding the closest prototype for a given greater than 
The above strategy guarantees that no candidate is pruned These results show that splitting along the longest dimen-
if it can potentially be closer than any other candidate pro- sion and choosing a midpoint-based approach for splitting
totype to a given subspace. is preferable [1].
Our algorithm is based on organizing the pattern vectors In the second phase of the k-means algorithm, the ini-
so that one can find all the patterns which are closest to a tial prototypes are derived. Just as in the direct k-means
given prototype efficiently. In the first phase of the algo- algorithm, these initial prototypes are generated randomly
rithm, we build a k-d tree to organize the pattern vectors. or drawn from the dataset randomly.4
The root of such a tree represents all the patterns, while the
children of the root represent subsets of the patterns com- function TraverseTree(% ,  ,! , )
pletely contained in subspaces (Boxes). The nodes at the  !   = Pruning(% ,  ,! , )
lower levels represent smaller boxes. For each node of the    - J then
if - ! ?
/* All the points in %
tree, we keep the following information:
belong to the alive
cluster */
1. The number of points ( ) Update the centroid’s statistics based on the in-
2. The linear sum of the points (

 ), i.e. 4 2  
7 4
formation stored in the node
return
 4
2   4
if % is a leaf then
for each point in %
3. The square sum of the points ( ), i.e. 7
?
Find the nearest prototype  9
Assign point to  9
Let the number of dimensions be @ and the depth of the
k-d tree be  . The extra time and space requirements for Update the centroid’s statistics
maintaining A 
the above information at each node is propor- return
B@ . Computing the medians at 
2
tionalA to levels takes
time  

[2]. These set of medians are needed in per-
for each child node do
   
TraverseTree(  !! , !  , - !  - , )


forming the splitting of the internal nodes of the tree. There-

fore, the total time requirement for building this tree such
that each internal nodeA at a  given level

represents the same
number of elements is  @
 .3 Figure 2. Tree traversal algorithm
For building the k-d tree, there are several competing
choices which affect the overall structure. In the third phase, the algorithm performs a number of it-
erations (as in the direct algorithm)

until a termination con-
1. Choice of dimension used for performing the split: dition is met. For each cluster , we maintain the number of
points / , the linear4 sum of the points / "# and the square
4 4
One option is to choose a common dimension across
all the nodes at the same level of the tree. The dimen- sum of the points / #$# .
sions are chosen in a round-robin fashion for different In each iteration, we traverse the k-d tree using a depth-
levels as we go down the tree. The second option is to first strategy (Figure

2) as follows.5 We start from the root
use the splitting dimension with the longest length. node with all candidate prototypes. At each node of the
tree, we apply a pruning function on the candidates proto-
2. Choice of splitting point along the chosen dimension: types. A high level description of the pruning algorithm is
We tried two approaches based on choosing the central given in Figure 3. If the number of candidate prototypes is
splitting point or median splitting point. The former di- equal to one, the traversal below that internal node is not
vides the splitting dimensions into two equal parts (by pursued. All the points belonging to this node have the sur-
width) while the latter divides the dimensions such that viving candidate as the closest prototype. The cluster statis-
there are equal number of patterns on either side. We tics are updated based on the information about the number
will refer to these approaches as midpoint-based and of points, linear sum, and square sum stored for that inter-
median-based approaches respectively. Clearly, the nal node. A direct k-means algorithm is applied on the leaf
cost of the median-based approach is slightly higher node if there is more than one candidate prototype. This di-
as it requires calculation of the median. rect algorithm performs one iteration of the direct algorithm
on the candidate prototypes and the points of the leaf node.
We have empirically investigated the effect of the two An example of the pruning achieved by using our algo-
choices on the overall performance of our algorithms. rithm is shown in Figure 4. Our approach is a conservative
 
2 The same information ( , , ) has also been used in [14] and is 4 The k-d tree structure can potentially be used to derive better ap-

called Clustering Feature (CF). However, as we will see later, we use CF proaches for this choice. However, these have not been investigated in
in a different way. this paper.
3 For the rest of the paper, patterns and points are used interchangeably. 5 Note that our approach is independent of the traversal strategy.
 
="! 9#
 %$'& 
function Pruning( 
   , ,!, ) 2. The error function is: 7 4
2  
/ #4 # 9 
 !    The leaf size is an important parameter for tuning the
for each prototype  9   do
 overall performance of our algorithm. Small leaf size re-
Compute the minimum ( % 9 ) and maximum sults in larger cost for constructing the tree, and increases
9
( 
 ) distances for any point in the box
representing the 
   the overall cost of pruning as the pruning may have to be
Find  the minimum of 
 9   0 ! , call it continued to lower levels. However, a small leaf size de-

%
 9  creases the overall cost for distance calculations for finding
for each prototype  9 

if  % 9 .


 
do

 9  then
 !  the closest prototype.
 % 

 !   )
!    9
Calculating the Minimum and Maximum Distances
return(
The pruning algorithm requires calculation of the minimum
as well as maximum distance to any given box from a given
prototype. It can be easily shown that the maximum dis-
Figure 3. Pruning algorithm box. Let H)(+*-,/.+0-12, 4
tance will be to one of the corners
be that corner for prototype

(
 of4 ).the The coordinates of
  



4 476
H)(+*-,/.+0%13, ( H)()*-,/.40%12, 4 H)(+*-,/.+0-12, 4 H)()*5,/.+0%12, ) can
approach and may miss some of the pruning opportunities. be computed as follows: ?
For example, the candidate shown as an x with a square
around it could be pruned with a more complex pruning 98;:

=<
:


=  4
 
:@
=  4

4 <
H)()*5,/.+0%12,  A<  <?> < <
strategy. However, our approach is relatively inexpensive :A@ ,/.405* 150

and can be shown to require time proportional to . Choos-  (1)
 
ing a more expensive pruning algorithm may decrease the where : and :@ are the
lower and upper coordinates of
overall number of distance calculations. This may, how- the box along dimension .
ever, be at the expense of higher overall computation time The maximum distance can be computed as follows:
due to an offsetting increase in cost of pruning. @

12,
B
7
6
32    4
 =
H)(+*-,/.+0%13,
4
 

A naive approach for calculating maximum and mini-

X mum distances for each prototype will perform the above
X
calculations for each node (box) of the tree indpenedently;
A  
X
d which will require @ time. The coordinates of the box
X
X
of the child node is exactly the same as its parent except
for one dimension which has been used for splitting at the
parent node. This information can be exploited to reduce
the time to constant time. This requires the use of the max-
imum distance of the prototype to the parent node. This
can be used to express the maximum square distance for the
child node in terms of its parent. The computation cost of
A % 
X
the above approach is for each candidate prototype.

The overall computational A   
requirement for a node with
candidate prototypes is . The value of minimum dis-
Figure 4. Example of pruning achieved by our tance can be obtained similarly. For more details the reader
algorithm. X represents the candidate set. @ is referred to a detailed version of this paper [1].
is the MinMax distance. All the candidates
which are circled get pruned. The candidate 4 Experimental Results
with a square around it is not pruned by our
algorithm We have evaluated our algorithm on several datasets. We
have compared our results with direct k-means algorithm in
terms of the number of performed distance calculations and
At the end of each iteration, the new set of centroids is the total execution time. A direct comparison with other al-
derived and the error function is computed as follows. gorithms (such as the P-Cluster [7] and [13] ) is not feasible
9 due to unavailability of their datasets and software. How-
  
1. The new centroid for cluster is: 9  ever, we present some qualitative comparisons. All the ex-
perimental results reported are on a IBM RS/6000 running Dataset k Direct Alg Our Algorithm
Total Time FRT FRD ADC
AIX version 4. The clock speed of the processor is 66 MHz
DS1 16 6.140 1.510 4.06 26.69 0.64
and the memory size is 128 MByte.
DS2 16 6.080 1.400 4.34 34.47 0.49

 and K 
of reduction in distance

For each dataset and the number of clusters, we com- DS3 16 6.010 1.370 4.38 35.68 0.48
pute the factors K R1 16 8.760 1.890 4.63 17.82 0.95
calculations and overall execution time over the direct algo- R2 16 17.420 3.130 5.56 27.38 0.62
R3 16 7.890 1.290 6.11 98.66 0.17
rithm respectively as well as the average number of distance
calculations per pattern   / . The number
R4 16 16.090 2.750 5.85 53.02 0.32
  %
of distance cal- R5 16 15.560 2.510 6.19 14.62 1.16
culations for the direct algorithm is
 per iteration.
6
R6 16 31.200 4.480 6.96 17.77 0.96
All time measurements are in seconds. R7 16 15.340 6.630 2.31 9.08 1.87
R8 16 22.200 6.800 3.26 9.08 1.87
Our main aim in this paper is to study the computational
R9 16 16.120 7.300 2.20 3.75 4.53
aspects of the k-means method. We used several datasets all R10 16 33.330 11.340 2.93 4.96 3.43
of which have been generated synthetically. This was done R11 16 14.200 11.260 1.26 2.21 7.68
to study the scaling properties of our algorithm for different R12 16 28.410 22.110 1.28 2.21 7.68

values of  and respectively. Table 1 gives a description DS1 64 23.020 2.240 10.27 54.72 1.19
DS2 64 22.880 2.330 9.81 43.25 1.50
for all the datasets. The datasets used are as follows: DS3 64 23.180 2.340 9.90 52.90 1.23
R1 64 38.880 5.880 6.61 10.61 6.12
1. We used three datasets (DS1, DS2 and DS3). These R2 64 141.149 11.770 11.99 10.64 6.11
are described in [14]. R3 64 32.080 1.780 18.02 99.66 0.65
R4 64 64.730 3.090 20.94 139.80 0.46
 R5 64 60.460 7.440 8.12 10.49 6.20
2. For the datasets R1 through R12, we have generated R6 64 121.200 14.670 8.26 10.65 6.10
 + 
points randomly in a cube of appropriate
 dimensional- R7 64 59.410 8.820 6.73 24.15 2.69

ity. For the th point we generate ! points around R8 64 89.750 8.810 10.18 24.15 2.69
$ R9 64 81.740 14.060 5.81 6.13 10.61
?
it using uniform distribution. These result in clusters R10 64 164.490 28.640 5.74 5.82 11.17
with non-uniform number of points. R11 64 58.280 15.340 3.79 5.92 10.97
R12 64 117.060 29.180 4.01 5.85 11.10
We experimented with leaf sizes of 4, 16, 64 and 256.
For most of our datasets, we found that choosing a leaf size
Table 2. The overall results for 10 iterations
of 64 resulted in optimal or near optimal performance. Fur-
ther, the overall performance was not sensitive to the leaf
size except when the leaf size was very small.
Tables 2 and 3 present the performance of our algorithms 5 Conclusions
for different number of clusters and iterations assuming a
leaf size of 64. For each combination used, we present
the factor reduction in overall time (FRT) and the time of
the direct k-means algorithm. We also present the factor In this paper, we presented a novel algorithm for per-
reduction in distance calculations (FRD) and the average forming k-means clustering. Our experimental results
number of distance calculations per pattern (ADC). These demonstrated that our scheme can improve the direct k-
results show that our algorithm can improve the overall per- means algorithm by an order to two orders of magnitude
formance of k-means clustering by an order to two orders in the total number of distance calculations and the overall
of magnitude. The average number of distance calculations time of computation.
required is very small and can vary anywhere from 0.17 to
11.17 depending on the dataset and the number of clusters There are several improvements possible to the basic
required. strategy presented in this paper. One approach will be to
The results presented in [7] show that their methods re- restructure  the tree

every few iterations to further reduce the

sult in factor of 4 to 5 improvements in overall compu- value of K @ . The intuition here is that the earlier itera-

tational time. Our improvements are substantially better. tions provide some partial clustering information. This in-
However, we note that the datasets used are different and a formation can potentially be used to construct the tree such
direct comparison may not be accurate. that the pruning is more effective. Another possibility is
to add the optimizations related to incremental approaches

6 This includes the distance calculations for finding the nearest pro-
presented in [7]. These optimizations seem to be orthogo-

totype and the equivalent of distance calculations for computing he new nal and can be used to further reduce the number of distance
set of centroids. calculations.
 Dataset Size Dimensi- No. of Characteristic Range
onality Clusters
DS1 100,000 2 100 Grid [-3,41]
DS2 100,000 2 100 Sine [2,632],[-29,29]
DS3 100,000 2 100 Random [-3,109],[-15,111]
R1 128k 2 16 Random [0,1]
R2 256k 2 16 Random [0,1]
R3 128k 2 128 Random [0,1]
R4 256k 2 128 Random [0,1]
R5 128k 4 16 Random [0,1]
R6 256k 4 16 Random [0,1]
R7 128k 4 128 Random [0,1]
R8 256k 4 128 Random [0,1]
R9 128k 6 16 Random [0,1]
R10 256k 6 16 Random [0,1]
R11 128k 6 128 Random [0,1]
R12 256k 6 128 Random [0,1]

Table 1. Description of the datasets. The range along each dimension is the same unless explicitly
stated

Dataset Direct Alg Our Algorithm Class Identification. Proc. of the Fourth Int’l. Symposium on
Total Time FRT FRD ADC
Large Spatial Databases, 1995.
DS1 115.100 6.830 16.85 64.65 1.01 [6] J. Garcia, J. Fdez-Valdivia, F. Cortijo, and R. Molina. Dy-
DS2 114.400 7.430 15.39 50.78 1.28
namic Approach for Clustering Data. Signal Processing,
DS3 115.900 6.520 17.77 66.81 0.97
44:(2), 1994.
R1 194.400 24.920 7.80 10.81 6.01
R2 705.745 49.320 14.30 10.80 6.02
[7] D. Judd, P. McKinley, and A. Jain. Large-Scale Parallel Data
R3 160.400 3.730 43.00 133.27 0.49 Clustering. Proc. Int’l Conference on Pattern Recognition,
R4 323.650 5.270 61.41 224.12 0.29 August 1996.
R5 302.300 32.430 9.32 10.72 6.06 [8] L. Kaufman and P. J. Rousseeuw. Finding Groups in Data:
R6 606.00 63.330 9.56 10.83 6.00 an Introduction to Cluster Analysis. John Wiley & Sons,
R7 297.050 32.100 9.25 26.66 2.44 1990.
R8 448.750 31.980 14.03 26.66 2.44 [9] K. Mehrotra, C. Mohan, and S. Ranka. Elements of Artificial
R9 408.700 63.920 6.39 6.25 10.41 Neural Networks. MIT Press, 1996.
R10 822.450 132.880 6.18 5.86 11.09 [10] R. T. Ng and J. Han. Efficient and Effective Clustering Meth-
R11 291.400 67.850 4.29 6.30 10.32
ods for Spatial Data Mining. Proc. of the 20th Int’l Conf.
R12 585.300 133.580 4.38 6.07 10.72
on Very Large Databases, Santiago, Chile, pages 144–155,
1994.
[11] V. Ramasubramanian and K. Paliwal. Fast K-Dimensional
Table 3. The overall results for 50 iterations
Tree Algorithms for Nearest Neighbor Search with Applica-
and 64 clusters
tion to Vector Quantization Encoding. IEEE Transactions
on Signal Processing, 40:(3), March 1992.
[12] E. Schikuta. Grid Clustering: An Efficient Hierarchical
References Clustering Method for Very Large Data Sets. Proc. 13th
Int’l. Conference on Pattern Recognition, 2, 1996.
[1] K. Alsabti, S. Ranka, and V. Singh. An Efficient K-Means [13] J. White, V. Faber, and J. Saltzman. United States Patent No.
Clustering Algorithm. http://www.cise.ufl.edu/ ranka/, 5,467,110. Nov. 1995.
1997. [14] T. Zhang, R. Ramakrishnan, and M. Livny. BIRCH: An Ef-
[2] T. H. Cormen, C. E. Leiserson, and R. L. Rivest. Introduc- ficient Data Clustering Method for Very Large Databases.
tion to Algorithms. McGraw-Hill Book Company, 1990. Proc. of the 1996 ACM SIGMOD Int’l Conf. on Management
[3] R. C. Dubes and A. K. Jain. Algorithms for Clustering Data. of Data, Montreal, Canada, pages 103–114, June 1996.
Prentice Hall, 1988.
[4] M. Ester, H. Kriegel, J. Sander, and X. Xu. A Density-
Based Algorithm for Discovering Clusters in Large Spatial
Databases with Noise. Proc. of the 2nd Int’l Conf. on Knowl-
edge Discovery and Data Mining, August 1996.
[5] M. Ester, H. Kriegel, and X. Xu. Knowledge Discovery in
Large Spatial Databases: Focusing Techniques for Efficient