Pattern Recognition 34 (2001) 2259}2281

Color image segmentation: advances and prospects

H.D. Cheng*, X.H. Jiang, Y. Sun, Jingli Wang
Department of Computer Science, Utah State University, Logan, UT 84322-4205, USA
Received 23 February 2000; accepted 12 September 2000

Abstract

Image segmentation is very essential and critical to image processing and pattern recognition. This survey provides a
summary of color image segmentation techniques available now. Basically, color segmentation approaches are based on
monochrome segmentation approaches operating in di!erent color spaces. Therefore, we "rst discuss the major
segmentation approaches for segmenting monochrome images: histogram thresholding, characteristic feature clustering,
edge detection, region-based methods, fuzzy techniques, neural networks, etc.; then review some major color representa-
tion methods and their advantages/disadvantages; "nally summarize the color image segmentation techniques using
di!erent color representations. The usage of color models for image segmentation is also discussed. Some novel
approaches such as fuzzy method and physics-based method are investigated as well.  2001 Pattern Recognition
Society. Published by Elsevier Science Ltd. All rights reserved.

Keywords: Color image segmentation; Color representations; Color space transformations; Neural networks; Thresholding; Clustering;
Edge detection; Region-based approach; Physics based approach; Fuzzy logic

1. Introduction (S , S ,2, S ) such that

  L
Image segmentation is the "rst step in image analysis L

S "F with S S " (iOj).
and pattern recognition. It is a critical and essential G G H
component of image analysis and/or pattern recognition G
system, is one of the most di$cult tasks in image process- The uniformity predicate P(S )"true for all regions,
G
ing, and determines the quality of the "nal result of S , and P(S S )"false, when iOj and S and S are
analysis. Image segmentation is a process of dividing an G G H G H
neighbors.
image into di!erent regions such that each region is, but According to [2], `the image segmentation problem is
the union of any two adjacent regions is not, homogene- basically one of psychophysical perception, and therefore
ous. A formal de"nition of image segmentation is as not susceptible to a purely analytical solutiona. There are
follows [1]: If P( ) is a homogeneity predicate de"ned on many papers and several surveys on monochrome image
groups of connected pixels, then segmentation is a parti- segmentation techniques. Color image segmentation
tion of the set F into connected subsets or regions attracts more and more attention mainly due to the
following reasons: (1) color images can provide more
information than gray level images; (2) the power of
personal computers is increasing rapidly, and PCs can be
used to process color images now. The segmentation
* Corresponding author. Tel.: #1-435-797-2054; fax: #1- techniques for monochrome images can be extended to
435-797-3265. segment color images by using R, G and B or their
E-mail addresses: [email protected] (H.D. Cheng), transformations (linear/non-linear). However, compre-
[email protected] (X.H. Jiang), [email protected] hensive surveys on color image segmentation are few.
(Y. Sun), [email protected] (J. Wang). Ref. [3] analyzed the problem when applying edge-based

0031-3203/01/$20.00  2001 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.
PII: S 0 0 3 1 - 3 2 0 3 ( 0 0 ) 0 0 1 4 9 - 7
2260 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

and region-based segmentation techniques to color im- erator [9,10], functional approximations [11,12], heuristic
ages with complex texture, and Ref. [4] discussed the search and dynamic programming, relaxation, and line
properties of several color representations, the segmenta- and curve "tting, while the sequential techniques make
tion methods and color spaces. decision based on the results of the previously examined
This paper provides a summary of color image seg- points. A brief description of the major component of
mentation techniques available at present, and describes a sequential edge detection procedure was given. It also
the properties of di!erent kinds of color representation brie#y introduced region merging, region splitting and
methods and problems encountered when applying the combination of region merging and splitting approaches.
color models to image segmentation. Some novel ap- Ref. [5] classi"ed image segmentation techniques into
proaches such as fuzzy and physics-based approaches six major groups: (1) measurement space guided spatial
will be discussed as well. clustering, (2) single linkage region growing schemes,
Section 2 brie#y introduces the major segmentation (3) hybrid linkage region growing schemes, (4) centroid
approaches for processing monochrome images: histo- linkage region growing schemes, (5) spatial clustering
gram thresholding, characteristic feature clustering, edge schemes, and (6) split-and-merge schemes. These groups
detection, region-based methods, fuzzy techniques and are compared on the problem of region merge error,
neural networks. Section 3 reviews some major color blocky region boundary and memory usage. The hybrid
representations and their advantages/disadvantages. Sec- linkage region growing schemes appear to o!er the best
tion 4 investigates the segmentation techniques applied compromise between having smooth boundaries and few
to color images using di!erent color representations, and unwanted region merges. One of the drawbacks of fea-
the summary is given in Section 5. ture space clustering is that the cluster analysis does not
utilize any spatial information. The survey presented
some spatial clustering approaches which combine clus-
2. Monochrome image segmentation tering in feature space with region growing or spatial
linkage techniques. It gives a good summary of kinds of
Monochrome image segmentation approaches are linkage region growing schemes. The problem of high
based on either discontinuity and/or homogeneity of correlation and spatial redundancy of multi-band image
gray level values in a region. The approach based on histograms and the di$culty of clustering using multi-
discontinuity tends to partition an image by detecting dimensional histograms are also discussed.
isolated points, lines and edges according to abrupt Ref. [6] surveyed segmentation algorithms based on
changes in gray levels. The approaches based on homo- thresholding and attempted to evaluate the performance
geneity include thresholding, clustering, region growing, of some thresholding techniques using uniformity and
and region splitting and merging. Several survey papers shape measures. It categorized global thresholding tech-
on monochrome image segmentation [1,2,5}8] cover the niques into two classes: point-dependent techniques (gray
major image segmentation techniques available. level histogram based) and region-dependent techniques
Ref. [2] discussed segmentation from the view point of (modi"ed histogram or co-occurrence based). Discussion
cytology image processing. It categorized various exist- on probabilistic relaxation and several methods of
ing segmentation techniques before 1980s into three multi-thresholding techniques was also given.
classes: (1) characteristic feature thresholding or cluster- Ref. [7] regarded image segmentation as the bridge
ing, (2) edge detection, and (3) region extraction. The in a machine vision system between a low-level vision
segmentation techniques were summarized and com- subsystem including image processing operations (such
ments were given on the advantages and disadvantages as noise reduction, edge extraction) to enhance the input
of each approach. The threshold selection schemes based image, and a high-level vision subsystem including object
on gray level histogram and local properties, and based recognition and scene interpretation. After segmentation,
on structural (textural) and syntactic techniques were the enhanced input image is mapped into a description
described. Clustering techniques were regarded as `the involving regions with some common features for the
multidimensional extension of the concept of thre- high-level vision tasks. The segmentation techniques are
sholdinga. Some clustering schemes utilizing di!erent categorized into three main classes: pixel-based, edge-
kinds of features (multi-spectral information, mean/vari- based, and region-based schemes. Some common image
ation of gray level, texture, color) were discussed. Various segmentation approaches are studied, such as Gaussian
edge detection techniques were presented, which were "ltering, Otsu's thresholding method, Chow}Kaneko's
categorized into two classes: parallel and sequential tech- adaptive thresholding, Yanowitz and Bruckstein's adap-
niques. The parallel edge detection technique implies that tive edge-based thresholding, Parker's local intensity
the decision of whether a set of points are on an edge or gradient approach, and Horowitz and Pavlidis's split,
not is dependent on the gray level of the set and some set merge, and group (SMG) approach [7]. Ref. [1] reviewed
of its neighbors, which includes high-emphasis spatial gray level thresholding, edge detection and many other
frequency "ltering, gradient operators, adaptive local op- approaches such as fuzzy set segmentation approaches,
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2261

neural network based approaches, Markov Random

Field (MRF) based approaches, and surface based ap-
proaches. It also considers range image and magnetic
resonance image (MRI) besides the most common light
intensity image. A brief introduction to color image seg-
mentation and fuzzy segmentation approach is discussed.
The development based on the applications of fuzzy
operators, properties and mathematics are presented,
and segmentation based on IF}THEN rules is predicted
as a promising research area in the near future. After the
discussion of these segmentation approaches, the authors
made a comparison of six histogram based methods and
two iterative pixel classi"cation methods: relaxation and
MAP (maximum a posteriori) estimation. Finally, the
attempts for objective evaluation of segmentation results
are studied.
For detail descriptions of various monochrome image Fig. 1. RGB color space represented in a 3-dimensional
segmentation approaches, readers may refer to Refs. cube [14].
[1,2,5}7].

3. Color features the three components by the same number; (3) the
luminance of a mixture of colors is equal to the sum of
Color is perceived by humans as a combination of the luminance of each color. The tristimulus values that
tristimuli R (red), G (green), and B (blue) which are served as the color basis are: 425.8 nm for blue, 546.1 nm
usually called three primary colors. From R, G, B re- for green, and 700.0 nm for red. Any color can be ex-
presentation, we can derive other kinds of color repres- pressed by these three color bases.
entations (spaces) by using either linear or nonlinear RGB is the most commonly used model for the televi-
transformations. Several color spaces, such as RGB, HSI, sion system and pictures acquired by digital cameras.
CIE ¸*u*v* are utilized in color image segmentation, but Video monitors display color images by modulating the
none of them can dominate the others for all kinds of intensity of the three primary colors (red, green, and blue)
color images. Selecting the best color space still is one of at each pixel of the image [16,17]. RGB is suitable for
the di$culties in color image segmentation [13]. color display, but not good for color scene segmentation
Red, green, and blue components can be represented and analysis because of the high correlation among the
by the brightness values of the scene obtained through R, G, and B components [18,19]. By high correlation, we
three separate "lters (red, green, and blue "lters) based on mean that if the intensity changes, all the three compo-
the following equations: nents will change accordingly. Also, the measurement of
a color in RGB space does not represent color di!erences
R"
 H
E()S () d,
0
in a uniform scale, hence, it is impossible to evaluate the
similarity of two colors from their distance in RGB space.

G"
 H
E()S () d,
%
3.1. Linear transformations
B"
 H
E()S () d,
3.1.1. YIQ
>IQ is used to encode color information in TV signal
where S , S , S are the color "lters on the incoming for American system. It is obtained from the RGB model
0 %
light or radiance E(), and  is the wavelength. by a linear transformation:
The RGB color space can be geometrically represented




in a 3-dimensional cube (Fig. 1) [14]. The coordinates of
> 0.299 0.587 0.114 R
each point inside the cube represent the values of red,
green and blue components, respectively. I " 0.596 !0.274 !0.322 G
The laws of colorimetry are [15]: (1) any color can be Q 0.211 !0.253 !0.312 B
created by these three colors and the combination of the
three colors is unique; (2) if two colors are equivalent,
they will be again equivalent after multiplying or dividing where 0)R)1, 0)G)1, 0)B)1.
2262 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

The > component, is a measure of the luminance of the Since r#g#b"1, when two components are given,
color, and is a likely candidate for edge detection in the third component can be determined. We may use
a color image. The I and Q components jointly describe only two out of three [22].
the hue and saturation of the image [20]. The >IQ space Ref. [23] created another normalized color representa-
can partly get rid of the correlation of the red, green and tion space which was de"ned as
blue components in an image. The linear transformation
>"c R#c G#c B,
needs less computation time than nonlineal ones, which   
makes the >IQ space more preferable to nonlinear sys- R
tems. ¹ " ,
 R#G#B

3.1.2. YUV G
¹ " .
>;< is also a kind of TV color representation suitable  R#G#B
for European TV system. The transformation is:
where c #c #c "1, c , c , and c are constants and
     




can be combined to produce the illumination of the
> 0.299 0.587 0.114 R image pixel. We can see that ¹ and ¹ are decided only
 
; " !0.147 !0.289 0.437 G by the percentage of the RGB components, therefore,
they can represent the real color information of an image,
< 0.615 !0.515 !0.100 B
and they are independent of the brightness of the image
which is one of the advantages of the normalized RGB
where 0)R)1, 0)G)1, 0)B)1. color system [24]. It is more convenient to represent the
color plane since the color values are set to a narrow
3.1.3. I I I limitation.
  
Ref. [21] performed systematic experiments of region Normalization reduces the sensitivity of the distribu-
segmentation to derive a set of e!ective color features. At tion to the color variability [25]. It is relatively robust to
each step of the recursive region splitting, new color the change of the illumination. But an obvious shortcom-
features are calculated by Karhunen}Loeve transforma- ing of normalized RGB is that the normalized colors are
tion of R, G, and B. It applied eight kinds of color pic- very noisy if they are under low intensities. This is due to
tures, analyzed over 100 color features, and found a set of the nonlinear transformation from the RGB space to the
e!ective color features as follows: normalized RGB space.

I "(R#G#B)/3, 3.2.2. HSI


The HSI (hue}saturation}intensity) system is another
I "(R!B)/2,
 commonly used color space in image processing, which is
more intuitive to human vision [26}29]. There are some
I "(2G!R!B)/4.
 variants of HSI systems, such as HSB (hue}satura-
Comparing I I I with 7 other color spaces tion}brightness), HS¸ (hue}saturation}lightness), and
   HS< (hue}saturation}value) [30}32].
(RGB, >IQ, HSI, Nrgb, CIE, X>Z, CIE(¸HuHvH), and
CIE(¸HaHbH)), Ref. [21] claimed that I I I was more The HSI system separates color information of an
   image from its intensity information. Color information
e!ective in terms of the quality of segmentation and the
computational complexity of the transformation. is represented by hue and saturation values, while inten-
sity, which describes the brightness of an image, is deter-
mined by the amount of the light. Hue represents basic
3.2. Nonlinear transformations
colors, and is determined by the dominant wavelength in
the spectral distribution of light wavelengths. It is the
3.2.1. Normalized RGB(Nrgb)
location of the peak in the spectral distribution. The
When performing color image segmentation, we need
saturation is a measure of the purity of the color, and
to make colors not dependent on the change in lighting
signi"es the amount of white light mixed with the hue. It
intensities. An e$cient method is to get the variations of
is the height of the peak relative to the entire spectral
intensities uniformly across the spectral distribution. The
distribution.
normalized color space is formulated as:
The HSI color space can be described geometrically as
r"R/(R#G#B), in Fig. 2 [14]. Generally, hue is considered as an angle
between a reference line and the color point in RGB
g"G/(R#G#B), space. The range of the hue value is from 03 to 3603, for
example, blue is 2403, yellow is 603, green is 1203, and
b"B/(R#G#B). magenta is 3003. The saturation component represents
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2263

growing algorithms. Hue is particularly useful in the

cases where the illumination level varies from point-to-
point or image-to-image. If the integrated white condi-
tion holds, hue is invariant to certain types of highlights,
shading, and shadows. Besides, segmentation in the 1-D
hue space is computationally less expensive than in the
3-D RGB space. One of the disadvantages of hue is that it
has a nonremovable singularity near the axis of the color
cylinder, where a slight change of input R, G, and
B values can cause a large jump in the transformed
values. The singularities may create discontinuities in the
representation of colors [15]. The hue values near the
singularity are numerically unstable. That is why pixels
having low saturation are left unassigned to any regions
in many segmentation algorithms. Also, if the intensity of
Fig. 2. HSI color space [14].
the color lies close to white or black, hue and saturation
play little role in distinguishing colors.
the radial distance from the cylinder center. The nearer
the point is to the center, the lighter is the color. Intensity 3.2.3. CIE spaces
is the height in the axis direction. The axis of the cylinder CIE (Commission International de l'Eclairage) color
describes the gray levels, for instance, zero (minimum) system was developed to represent perceptual uniformity,
intensity is black, full (maximum) intensity is white. Each and thus meets the psychophysical need for a human
slice of the cylinder perpendicular to the intensity axis is observer. It has three primaries denoted as X, >, and Z.
a plane with the same intensity. Any color can be speci"ed by the combination of X, >
The HSI color system has a good capability of repres- and Z. The values of X, >, and Z can be computered by
enting the colors of human perception, because human a linear transformation from RGB tristimulus coordi-
vision system can distinguish di!erent hues easily, where- nates. In particular, the transformation matrix for the
as the perception of di!erent intensity or saturation does NTSC (National Television System Commission, United
not imply the recognition of di!erent colors. States) receiver primary system is:




The HSI coordinates can be transformed from the
RGB space. The formulas for hue, saturation, and inten- X 0.607 0.174 0.200 R
sity are: > " 0.299 0.587 0.114 G .
Z 0.000 0.066 1.116 B


(3(G!B)
H"arctan ,
(R!G)#(R!B) There are a number of CIE spaces can be created once
the X>Z tristimulus coordinates are known. CIE
(R#G#B)
Int" , (¸HaHbH) space and CIE (¸HuHvH) space are two typical
3 examples. They can all be obtained through nonlinear
min(R, G, B) transformations of X, >, and Z values.
Sat"1! . CIE (¸HaHbH) is de"ned as:
I

 
>
The hue is unde"ned if the saturation is zero, and the ¸H"116  !16,
>
saturation is unde"ned when the intensity is zero. 

  
We may use gray-level algorithms to operate on the X >
intensity component of HSI description. To segment ob- aH"500  ! ,
X >
jects with di!erent colors, we may apply the segmenta-  

  
tion algorithms to the hue component only. For example, > Z
bH"200  ! .
we may set thresholds on the range of hues that separate > Z
di!erent objects easily, but it is di$cult to transform  
where frac >> '0.01, X/X '0.01, and Z/Z '0.01.
these thresholds into RGB values, since hue, saturation,   
(X , > , Z ) are X, >, Z values for the standard white.
and intensity values are all encoded into RGB values. It is   
especially e$cient when the images have non-uniform CIE (¸HuHvH) is de"ned as:


illumination such as shade, since hue is independent on >
intensity values. We may also e$ciently apply thresholds ¸H"116  !16, uH"13¸H(u!u ),
> 
to hue, saturation, and intensity components, respective- 
ly, to form some regions that can be "t for various region vH"13¸H(v!v ).

2264 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

where >/> '0.01, > , u , and v are the values for the
   
standard white, and

4X 6>
u" , v" .
X#15>#3Z X#15>#3Z

Each point in these two color spaces can be regarded

as a point in the (¸H, aH, bH) or (¸H, uH, vH) three-dimen-
sional color space, so that the di!erence of two colors can
be calculated as the Euclidean distance between two
color points. The formulas for the color di!erence are
described below:
For CIE (¸HaHbH) space:

E "((¸H)#(aH)#(bH).
?@
For CIE (¸HuHvH) space:

E "((¸H)#(uH)#(vH).
ST
The ability to express color di!erence of human per-
ception by Euclidean distance is very important to color
Fig. 3. Munsell color space [34].
segmentation. (¸H, aH, bH) or (¸H, uH, vH) is approximately
uniform chromaticity scale. That is, they match the sensi-
tivity of human eyes with computer processing [33],
whereas the RGB or X>Z color space does not have such 3.2.4. Munsell
a property. Therefore, we can derive the perceptual color Munsell color system was created in 1969 [34]. It is
attributes such as intensity, hue and saturation conve- one of the earliest methods to specify colors. The Munsell
niently. We may use one of the two CIE color spaces and color system uses three attributes of color perception,
the associated color di!erence formulas to de"ne HSI which are Munsell Hue, Value, and Chroma. Fig. 3
space which is mapped to the cylindrical coordinates of describes the Munsell color space.
(¸H, uH, vH) or (¸H, aH, bH) space, and also consistent to the There are "ve colors regarded as the major hues, they
de"nition of HSI space. are red (R), yellow (>), green (G), blue (B) and purple (P).
For CIE (¸HaHbH) space: The combinations of colors: >R, G>, BG, PB, and RP are
"ve half-way hues. The hue circle is divided equally into
I"¸H, 100 parts with the "ve major hues and the "ve half-way
hues located on the circle with equal spaces. All hue
H"arctan(aH/bH), values are arranged on the hue circle, and can be de-
scribed in two ways. One is to indicate their relative
S"((aH)#(bH). associations with a speci"ed hue, for example, R, 2R, and
10R. The other way is to assign a number to a hue:
For CIE (¸HuHvH) space: 5R"0, 6R"1, and increase the number clockwise
along the hue circle to get a number for each hue.
I"¸H,
Munsell value (<) describes the lightness of a color. It
H"arctan(uH/vH), de"nes the value of black as 0 and the value of white as
10. An equation describes the relationship between < and
S"((uH)#(vH). Luminance >:

The two CIE spaces share same ¸H value, which de- >"1.2219<!0.23111<#0.23951<!0.021009<
"nes the lightness, or the intensity, of a color. #0.0008404<,
CIE spaces can control color and intensity informa-
tion more independently and simply than RGB primary Chroma (C) is like the saturation component in the
colors. Direct color comparison can be performed based CIE representation. It describes the purity of a color.
on geometric separation within the color space, therefore, When C is equal to zero, it is an achromatic color.
it is especially e$cient in the measurement of small color Similar to the HSI space, the Munsell color space can
di!erence. However, it still has the same problem of also be represented by a cylindrical coordinate system
singularity as other nonlinear transformations. consisting of H, < and C. However, we cannot convert
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2265

the Munsell color system into CIE standard descriptions Based upon a dichromatic re#ection model, Ref. [36]
by a formula. Therefore, a method must be created to used a method to determine the amount of interface
map the real color signals into the Munsell space [34]. re#ection and body re#ection in a color image pixel by
Three levels of image representation were demonstrated in pixel. It also presented an algorithm for analyzing color
the experiment. The lowest level, density, can be achieved values in an image, which is very useful in color image
by Black/White component of the images; the next level, understanding.
three primary colors, can be implemented on RGB images;
and the highest level, three perceptual attributes can be 3.5. Discussion on color space
con"rmed with the images of Munsell H<C system.
Ref. [38] showed that nonlinear color transformations
3.3. Hybrid color space such as HSI and the normalized color space had essen-
tial, nonremovable singularities and there were spurious
Ref. [35] proposed a method to classify the pixels in modes in the distribution of values. Ref. [38] suggested
Hybrid Color Space (HCS) which is composed by a set of that linear spaces, such as >IQ, be used, rather than
three color features. Taking into account the K available nonlinear spaces.
color features (R, G, B, R , G , B , I, H, S, X, >, Z, X , > , The major problem of linear color spaces is the high
L L L L L
Z , ¸H, aH, bH, uH, vH, I , I , A, C , C , >, I, Q,;, <, correlation of the three components, which makes the
L    
CH , h , SH , CH , h ). It used a speci"c informational three components dependent upon each other and asso-
ST ST ST ?@ ?@
criterion to select a set of three most discriminating color ciate strongly with intensity. Hence, linear spaces are
features. Experimental results show that: I , A and very di$cult to discriminate highlights, shadows and

CH color features are selected. Finally, the classi"ed shading in color images. Besides, if a linear color space is
ST
¸;< and HCS images are compared. This method used, image segmentation has to be performed in a 3-D
depends on the application. It does not give the optimal space, usually on one component at a time, because it is
solution but consumes less computational time. di$cult to combine the information inherent in these
components. However, nonlinear color spaces do not
3.4. Physics-based models have such problems. In HSI space, hue can be used for
segmentation in 1-D space if the saturation is not low,
The traditional methods of color image segmentation where certain types of highlights, shadows and shading
su!er from too many erroneous regions because they can be discounted [30].
have not accounted for the in#uence of optical e!ects on
object colors. Image segmentation should base on mater-
ial surface changes that include the material changes and 4. Color image segmentation
variation due to shading, shadows and highlights. Be-
sides objects receive the lights directly from illumination It has long been recognized that human eye can dis-
sources, they re#ect lights from other objects. If the cern thousands of color shades and intensities but only
physical models were included at the segmentation stage, two-dozen shades of gray. It is quite often when the
many of these regions could be classi"ed correctly. objects cannot be extracted using gray scale but can be
Physical models in color image processing are quite extracted using color information. Compared to gray
di!erent from the conventional color information repres- scale, color provides information in addition to intensity.
entations. These models aim at eliminating the e!ect of Color is useful or even necessary for pattern recognition
highlights, shadows and shading, and segmenting a color and computer vision. Also the acquisition and processing
image at boundaries of objects. Shafer's `dichromatic hardwares for color images have become more available
re#ection modela [36] and Healey's `approximate and accessible to deal with the computational complexity
color-re#ectance model (ACRM)a [37] are two typical caused by the high-dimensional color space. Hence, color
examples. image processing has become increasingly more practi-
Re#ection is highly related to the nature of the mater- cal.
ials. Ref. [37] divided materials into di!erent classes: one As mentioned before, the literature on color image
includes optically homogeneous materials like metals, segmentation is not as extensively present as that on
glass and crystals, and another includes the optically monochrome image segmentation. Most published re-
inhomogeneous materials such as plastics, paper, textiles sults of color image segmentation are based on gray level
and paints. Usually it is very helpful to identify or classify image segmentation approaches with di!erent color rep-
the material in the scene of an image before an algorithm resentations, as shown in Fig. 4.
is applied. For example, we should distinguish metals Ref. [1] gives a brief introduction to color image seg-
from dielectrics since they interact with lights in di!erent mentation, and mentions that color images can be con-
ways. Thus, they require di!erent algorithms in image sidered as a special case of multi-spectral images and any
understanding. segmentation method for multi-spectral images can be
2266 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

Fig. 4. Commonly used color image segmentation approaches.

applied to color images. Ref. [3] analyzed the complexi- tion has its advantages and disadvantages. There is no
ties encountered in segmenting color images with com- single color representation that can surpasses others for
plex texture. Only two color spaces, RGB and HSI, are segmenting all kinds of color images.
discussed. Texture is considered to be the major problem In most of the existing color image segmentation ap-
for all segmentation techniques, thus much more dis- proaches, the de"nition of a region is based on similarity
cussion was made on texture analysis than on color of color. This assumption often makes it di$cult for any
representation, and the problems of feature extraction in algorithms to separate the objects with highlights,
images with textural variations are discussed parti- shadows, shadings or texture which cause inhomogeneity
cularly. The paper focuses on two major groups of seg- of colors of the objects' surface. Using HSI can solve this
mentation techniques: boundary formation and region problem to some extent except that hue is unstable at low
formation. The problem of how and to what degree the saturation. Some physics based models have been pro-
semantic information should be employed in image seg- posed to solve this problem [36,37,40}52]. These models
mentation is discussed brie#y. Experimental results are take into account the color image formation [17], but
also presented. Ref. [4] discussed several color spaces they have too many restrictions which prevent them from
(RGB, normalized RGB space, HSI, CIE ¸HuHvH, being extensively applied.
>IQ, >;<) and their properties. Segmentation ap- Segmentation may be also viewed as image classi"ca-
proaches are categorized into four classes: pixel based tion problem based on color and spatial features. There-
segmentation, area based segmentation, edge based seg- fore, segmentation methods can be categorized as
mentation and physics based segmentation. A brief con- supervised or unsupervised learning/classi"cation pro-
clusion is drawn based on the analysis of the literature cedures. Ref. [53] compared di!erent color spaces (RGB,
available. normalized RGB, HIS, hybrid color space) and super-
Most gray level image segmentation techniques can vised learning algorithms for segmenting fruit images.
be extended to color images, such as histogram thre- Supervised algorithms include: maximum likelihood, de-
sholding, clustering, region growing, edge detection, cision tree, K nearest neighbor, neural networks, etc. Ref.
fuzzy approaches and neural networks. Gray level seg- [54] explored six unsupervised color image segmentation
mentation methods can be directly applied to each com- approaches: adaptive thresholding, fuzzy C-means, SC¹/
ponent of a color space, then the results can be combined center split, PC¹ (principal components transform)/me-
in some way to obtain a "nal segmentation result [39]. dian cut, split and merge, multiresolution segmentation.
However, one of the problems is how to employ the color Ref. [54] showed that combining di!erent methods re-
information as a whole for each pixel. When the color is sults in further improvement in the number of correctly
projected onto three components, the color information identi"ed tumor borders, and by incorporating addi-
is so scattered that the color image becomes simply tional heuristics in merging the segmented object in-
a multispectral image and the color information that formation, the success rate can be further increased.
humans can perceive is lost. Another problem is how to Some algorithms combine unsupervised with supervised
choose the color representation for segmentation. As methods to segment color images. Ref. [55] used un-
discussed in the previous section, each color representa- supervised learning and classi"cation based on the FCM
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2267

algorithm and nearest prototype rule. The classi"ed obtained from three di!erent color spaces (RGB, >IQ,
pixels are used to generate a set of prototypes which are and HSI). This method uses a mask for region splitting
optimized using a multilayer neural network. The super- and the initial mask includes all pixels in the image. For
vised learning is utilized because the optimized proto- any mask, histograms of the nine redundant features
types are subsequently used to classify other image pixels. (R, G, B, >, I, Q, H, S, and I) of the masked image are
Ref. [56] employed a neural network for supervised seg- computed, all peaks in these histograms are located, the
mentation and a fuzzy clustering algorithm for unsuper- histogram with the best peak is selected and a threshold
vised segmentation. The detail description of various is determined in this histogram to split the masked image
unsupervised and supervised approaches to color image into two subregions for which two new masks are gener-
segmentation can be found in Refs. [124,130,131]. ated for further splitting. This operation is repeated until
no mask left unprocessed, which means none of the nine
4.1. Histogram thresholding (mode method) and color histograms of existing regions can be further thresholded
space clustering and each region is homogeneous. Ref. [63] segments
a color image into regions with perceptually uniform
Histogram thresholding is one of the widely used tech- colors by means of Munsell three-color attributes (H, <
niques for monochrome image segmentation [19]. It and C). A recursive thresholding method similar to the
assumes that images are composed of regions with di!er- one in Ref. [62] is executed to segment a color image into
ent gray level ranges, the histogram of an image can be some meaningful regions. A criterion for locating the best
separated into a number of peaks (modes), each cor- peak is de"ned as
responding to one region, and there exists a threshold
value corresponding to valley between the two adjacent f"(S ;100)/(¹ ;F )
N ? F
peaks. As for color images, the situation is di!erent from
monochrome image because of multi-features. Multiple where S represents a peak area between two successive
N
histogram-based thresholding divided color space by valleys, ¹ is the overall area of the histogram, and
?
thresholding each component histogram. There is some F denotes the full width at half-maximum of the peak. In
F
limitation when dividing multiple dimensions because order to obtain the maximum information between two
thresholding is a technique for gray scale images. For sources, mode (regions with high densities) and valley
example, the shape of the cluster is rectangle. Ref. [57] (regions with low densities), Ref. [64] adopted entropy
used a tool for clustering color space based on the least based thresholding method. Mode seeking is decided by
sum-of-squares criterion and gave an approximate solu- the multimodel probability density function (pdf) estima-
tion. Since the color information is represented by tris- tion, and the mode can be found by thresholding the pdf.
timulus R, G and B or their linear/nonlinear trans- In the above approaches, thresholding is performed on
formations, representing the histogram of a color image only one color component at a time. Thus the regions
in a 3-dimensional array and selecting threshold in this extracted are not based on the information available
histogram is not a trivial job [5], and detecting the from all three components simultaneously because the
clusters of points within this space will be computation- correlation among the three components is neglected.
ally expensive. One way to solve this problem is to This problem can be solved if we can "nd such a line that
develop e$cient methods for storing and processing the when the points in the 3-D space are projected onto it,
information of the image in the 3-D color space. Due to and the projected points can be well separated. Ref. [65]
the capability of the RAM is increasing rapidly, this utilized the Fisher Linear Discriminant to "nd such a
problem alleviates. Ref. [58] used a binary tree to store line for 1-D thresholding. The method operates in the
the 3-D histogram of a color image, where each node of CIE (¸HaHbH) color space. The cluster distribution in the
the tree includes RGB values as the key and the number 3-D space is "rst estimated using only 1-D histograms.
of points whose RGB values are within a range, with this Then, for 1-D thresholding, the clusters are projected onto
key being the center of the range. Ref. [59] also utilized the line determined by the Fisher discriminant method
the same data structure and similar method to detect which can minimize the clustering error rate. This permits
clusters in the 3-D normalized color space (X, >, I). An- the simultaneous utilization of all color information.
other way is to project the 3-D space onto a lower- Ref. [21] employed the segmentation approach in
dimensional space, such as 2-D or even 1-D. Ref. [60] Ref. [62] to extract a set of e!ective color features
used projections of 3-D normalized color space (X, >, I) through experiments. Instead of using redundant features
onto the 2-D planes (X!>, X!I, and >!I) to inter- for thresholding, it applied Karhunen}Loeve trans-
actively detect insect infestations in citrus orchards from formation to RGB color space to extract features with
aerial color infrared photographs. Ref. [61] provided large discriminant power to isolate the clusters in a given
segmentation approaches using 2-D projections of a region. Given a region S, let  be the covariance matrix of
color space. Ref. [62] suggested a multidimensional the distributions of R, G, and B in S, and  ,  and  be
  
histogram thresholding scheme using threshold values the eigenvalues of  and  * * . Let < "(v ,
   G 0G
2268 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

v , v ) be the eigenvectors of  corresponding to  , i.e., plied to one color component a time (RGB, >IQ, and
%G G G
I I I ), then scale-space "lter is used to locate the thre-
 ) < " ) < ,   
G G G sholds in each color component's histogram. These thre-
sholds are then used to partition the color space into
where i"1, 2, 3. Then, the color features F , F , and
  several hexahedra. The hexahedra that contain a number
F are de"ned as:
 of pixels exceeding a prede"ned threshold are declared as
F "v ;R#v ;G#v ;B valid clusters. All peaks in the 1-D histograms could be
G 0G %G G the cluster centers. The coarse segmentation attempts to
where < "1, and i"1, 2, 3. It can be proved that automatically "nd the number of classes and the center of
G
F , F , and F are uncorrelated, and F has the largest each class. At the "ne stage, FCM is used to assign the
   
variance equal to  , thus has the largest discriminant unclassi"ed pixels to the closest class using the cluster

power. F has the largest discriminant power among the centers detected at the coarse stage. This technique tried

vectors orthogonal to F . At each step of segmentation, to solve the problem of cluster validity by using

three new color features F , F , and F are adaptively a coarse}"ne concept. But it is still based on the assump-
  
calculated and used to compute the histograms for thre- tion that the histograms are not unimodal. It has the
sholding. However, performing K}L transformation at same disadvantage as most thresholding techniques
each segmentation step involves too much computa- have. Ref. [26] proposed an iterative multispectral image
tional time for real application. Ref. [21] just used it as segmentation approach using FCM, and performed ex-
a tool to "nd a set of color features as e!ective as those periments in RGB and Ohta color spaces. The Ohta
extracted by K}L transformation (see Section 3.2.3). feature space is a linear transformation of RGB space.
Ref. [66] also used the same idea to extract the princi- The three features are (R#G#B)/3, (R!B) and
pal components of a color distribution for detecting (2*R!G!B)/2. They are signi"cant in this order, and
clusters. A detail algorithm is given in Ref. [63] for in many cases good segmentation can be achieved by
determining the peaks in a histogram. `Clustering of using only the "rst two.
characteristic features applied to image segmentation is Ref. [68] proposed a method based on K-nearest
the multidimensional extension of the concept of thre- neighbor (K-NN) technique for detecting fruit and leaves
sholdinga [2]. Generally, two or more characteristic fea- in a color scene, which is used to build a vision system for
tures form a feature space and each class of regions is a robotic citric harvesting device. The feature vectors are
assumed to form a separate cluster in the space. The based on the >;< color space. In order to incorporate
reason to use multiple characteristic features to perform information about shape and surface of the fruit and
image segmentation is that, sometimes, there are prob- leaves, not only the color components of the pixel itself
lems not solvable with one feature but solvable with but also the components of its four neighbors are in-
multiple features. The characteristic features may be any cluded in the vector X(i, j)
features that could be used for the segmentation problem,
such as the gray level value of multi-spectral images, gray X(i, j)"[;(i, j), <(i, j), ;(i#h, j), <(i#h, j), ;(i!h, j),
level histogram, mean, deviation, texture, etc. For color
<(i!h, j), ;(i, j#h), <(i, j#h), ;(i, j!h),
images, a color space is a natural feature space, and
applying the clustering approach to color image segmen- <(i, j!h)]
tation is a straightforward idea, because the colors tend
to form clusters in the color space. Clustering has been where ;(i, j) and <(i, j) are the ; and < components of
used as an important pattern recognition technique for >;< color coordinates and h represents the neighbor.
many years. The biggest problem that it su!ers from is How to choose reference sample becomes important be-
how to determine the number of clusters in an unsuper- cause only "nite reference can be used in practice. Sec-
vised clustering scheme, which is known as cluster valid- ondly, mislabeled reference and/or `outlyera can reduce
ity. As for a color image, the selection of color space is the classi"cation accuracy. Thirdly, the more the number
quite critical to this approach. For example, if R, G, B is of references, the more noise or erroneous data. Finally,
selected for color clustering, because of the high correla- large reference sets can bring a problem of computational
tion among R, G and B, the object with a uniform color speed. The combined technique of multiediting and con-
but di!erent intensities could be segmented into di!erent densing is applied to reduce the size of the reference set.
objects. In other words, color image with shadows or By doing so, still with very high accuracy, the computa-
shading cannot be segmented properly. tional time can be reduced dramatically.
Ref. [67] presented a two-stage segmentation algo-
rithm for color images based on 1-D histogram thre- 4.2. Region based approaches
sholding and the fuzzy c-means (FCM) techniques which
are viewed as coarse and "ne segmentations, respectively. Region based approaches, including region growing,
At the coarse stage, 1-D histogram thresholding is ap- region splitting, region merging and their combination,
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2269

attempt to group pixels into homogeneous regions. In the In order to recognize the small object or local variance
region growing approach, a seed region is "rst selected, of color image, Ref. [70] proposed a hierarchical segmen-
then expanded to include all homogeneous neighbors, tation which identi"es the uniform region via a thre-
and this process is repeated until all pixels in the image sholding operation on a homogeneity histogram. The
are classi"ed. One problem with region growing is its homogeneity is de"ned as a composition of two com-
inherent dependence on the selection of seed region and ponents: variance and discontinuity of the intensity,
the order in which pixels and regions are examined. In (R#G#B)/3, the local information as well as global
the region splitting approach, the initial seed region is information is taken into account, therefore, the quality
simply the whole image. If the seed region is not homo- of the segmentation result is much improved. The region
geneous, it is usually divided into four squared sub- based approach is widely used in color image segmenta-
regions, which become new seed regions. This process is tion because it considers the color information and spa-
repeated until all sub-regions are homogeneous. The tial details at the same time.
major disadvantage of region splitting is that the result-
ing image tends to mimic the data structure used to 4.3. Edge detection
represent the image and comes out too square. The
region merging approach is often combined with region Edge detection is extensively utilized for gray level
growing or region splitting to merge the similar regions image segmentation, which is based on the detection of
for making a homogeneous region as large as possible. discontinuity in gray level, trying to locate points with
These techniques work best on images with an obvious abrupt changes in gray level. Edge detection techniques
homogeneity criterion and tend to be less sensitive to are usually classi"ed into two categories: sequential and
noise because homogeneity is typically determined stat- parallel [1,2]. A parallel edge detection technique means
istically. They are better than feature space thresholding that the decision of whether or not a set of points are on
or clustering techniques by taking into account both an edge is not dependent on whether other sets of points
feature space and the spatial relation between pixels lie on an edge or not. In principle, the edge detection
simultaneously. However, all region based approaches operator can be applied simultaneously all over the im-
are by nature sequential, and another problem with these age. One technique is high-emphasis spatial frequency
techniques is their inherent dependence on the selection "ltering. Since high spatial frequencies are associated
of seed region and the order in which pixels and regions with sharp changes in intensity, one can enhance or
are examined. Refs. [21,62,63,65] used region splitting extract edges by performing high-pass "ltering using the
approach to segment color images. The homogeneous Fourier operator. The problem here is how to design a
criteria utilized by them are based on 1-D histogram relevant "lter. There are many types of parallel di!eren-
thresholding on the features of color components or tial operators such as Roberts, Sobel, and Prewitt oper-
extracted from color spaces. ators, which are called the "rst-di!erence operators, and
Ref. [69] proposed a color segmentation approach the Laplacian operator, which is called the second-di!er-
which combines region growing and region merging ence operator. The main di!erences between these oper-
techniques. It starts with the region growing process ators are the weights assigned to each element of the
using the criteria based on both color similarity and mask. These operators require that there is a distinct
spatial proximity. Euclidean distance over R, G, B color change in gray level between two adjacent points, and
space is used to de"ne the color similarity, which de"nes only very abrupt edges between two regions could be
three criteria of color homogeneity: the local homogen- detected. They cannot detect ill-de"ned edges that are
eity criterion (LHC) corresponding to a local comparison formed by a gradual change in gray level across the edge.
between adjacent pixels; the "rst average homogeneity Since the computation is based on a small window, the
criterion (AHC1) corresponding to a local and regional result is quite susceptible to noise. Sequential edge detec-
comparison between a pixel and its neighborhood, con- tion means that the result at a point is dependent on the
sidering only the region under study; and the second result of the previously examined points. There are a
average homogeneity criterion (AHC2) corresponding to number of sequential techniques utilizing heuristic search
a global and regional comparison between a pixel and and dynamic programming. The performance of a se-
the studied region. The regions generated by region quential edge detection algorithm will depend on the
growing process are then merged on the basis of a global choice of a good initial point, and it is not easy to de"ne
homogeneity criterion based on color similarity to gener- a termination criterion.
ate a nonpartitioned segmentation consisting of spatially In a monochrome image, edge is de"ned as a discon-
disconnected but similar regions. The problem with this tinuity in the gray level, and can be detected only when
method is that the selection of the thresholds for these there is a di!erence of the brightness between two re-
criteria is rather subjective and the thresholds are image gions. However, in color images, the information about
dependent. Another problem is that it is not applicable to edge is much richer than that in monochrome case. For
images with shadows or shading. example, edges between two objects with the same
2270 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

brightness but di!erent hue can be detected in color the color information results in only a graceful degrada-
images [71]. Accordingly, in a color image, an edge tion in performance because the brightness edges and
should be de"ned by a discontinuity in a three-dimen- chromatic edges tend to be highly correlated. This is true
sional color space. Ref. [23] gave three alternatives for only for the two images employed in that paper, because
the de"nition of a color edge: (i) De"ne a metric distance there are hardly any shadows or shading in these two
in some color space and use discontinuities in the dis- images. For the images with considerable amount of
tance to determine edges. This makes color edge detec- shadows and shading, this conclusion will not be appro-
tion still be performed in 1-D space. Hence the result priate any more. The hue and saturation information
cannot be expected to be better than that achieved by turns out to be very important, because in such a case the
edge detection in an equivalent monochrome image. (ii) brightness information cannot, but the hue information
Regard a color image as composed of three monochrome can, be used to detect proper edges at the boundaries of
images formed by the three color components, respec- shadows or shading. Therefore, if there exists a large
tively, and perform gray level edge detection on these amount of shadows or shading, the degradation of per-
three images separately. Then the edges detected in the formance resulted from ignoring chromatic information
three images might be merged by some speci"ed proced- will no longer be graceful.
ures. This is still essentially a gray-level edge detection Ref. [20] discussed the choice of the color spaces for
technique and may be unsatisfactory in some cases, for edge detection. It considered RGB, YIQ, CIE (¸HaHbH),
example, when gradient edge detectors are employed, the GHGHGH and P P P , where GHGHGH is obtained using
        
three gradients for one pixel may have the same strength the model of the human visual system, and P , P and
 
but in opposite directions [72,73]. (iii) Impose some P are the three principal components obtained using the

uniformity constraints on the edges in the three color normalized covariance matrix of the red, green and blue
components to utilize all of the three color components components of a color image. The normalized covariance
simultaneously, but allow the edges in the three color matrices of the three components in the color spaces of
components to be largely independent. Actually, these a girl picture are given. For each color space, edge detec-
constraints directly a!ect the computation of the three tion is performed using the compass gradient edge detec-
color components, which makes de"nition (iii) essentially tion method. The color edge is determined by the
di!erent from de"nition (ii). maximum value of the 24 gradient values in three compo-
Ref. [23] used de"nition (iii) to de"ne a color edge, nents and eight directions at a pixel. By analyzing the
which is based on the edge operator provided by [11,12]. energy content and edge activity index of the color com-
This de"nition allows the edges in the three components ponents, it was concluded that the inherent cross-correla-
to be independent, except that the spatial angles of the tion of the three components in a color space should be
edges have to be the same. >, ¹ , and ¹ are used to considered for color edge detection, and the most e!ec-
 
compute color edges using an extension of the edge tive components for color edge detection are G, P ,

operator in Refs. [11,12], and the edges in hue and >, ¸H and GH which are related to the brightness compo-

saturation components are derived from the computa- nents in the "ve color spaces. However, Ref. [20] did not
tions performed on the >, ¹ , ¹ components. The take into account the e!ect of shadows or shading.
 
results of the experiments on two images show that most Ref. [74] argued that the edges detected in hue corre-
of the edges in the chromatic components correspond to late more directly with material boundaries than those
the boundaries of desired objects, and the number of detected in brightness, RGB, or normalized space, and the
edges detected in chromatic components (¹ , ¹ , hue color segmentation in the HSI space could make the
 
and saturation) is smaller than that of the edges detected segmentation results enhanced in images with high satu-
in the brightness component (>) because the color in an ration, even in the presence of confounding cues due to
image is relatively invariant over the object surface while shading, shadow, transparency, and highlights. It com-
the brightness may vary due to nonuniform illumination pared several color spaces such as RGB, Normalized
and re#ections, and for a large percentage of the cases, RGB, HSI and (¸HaHbH), discussed the properties of hue,
the brightness edge also exists where a chromatic edge and demonstrated that due to the additive/shift invari-
exists. However situations exist in images with low con- ance and multiplicative/scale invariance properties, hue
trast or poor illumination where the brightness edges are is invariant to transparencies and certain types of high-
absent but chromatic edges are present. The explanation lights, shadows and shading. The method for edge detec-
is that in natural scenes, it is unlikely that objects of tion in the paper is: "rst, use Canny edge operator [77] to
di!erent hue will accidentally have the same brightness generate an intensity edge map; then gradually eliminate
component. Based on the above observation, Ref. [23] the edges if the hue changes are small. The major prob-
suggested that the use of color is more likely to aid in lem of the HSI space is its singularity at the axis of the
obtaining reliable initial data in a multilevel segmenta- color cylinder where R"G"B or saturation"0. Hue
tion scheme because the chromatic edges seem to contain is unstable at low saturations, thus, using hue to segment
fewer spurious edges, and also concluded that ignoring regions with low saturations is not reliable. Ref. [74]
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2271

utilized a "rst-order membrane type stabilizer based on subsets rather than ordinary subsets [1]. In fuzzy subsets,
Markov random "elds to smooth the unstable hue re- each pixel in an image has a degree to which it belongs to
gions of low saturations or low intensities and obtained a region or a boundary, characterized by a membership
much improvement in color segmentation. The experi- value. By doing so, we can avoid making a crisp decision
ments show that starting with an intensity edge map and earlier and keep the information through the higher
applying the "rst-order smoothness operator to the hue processing levels as much as possible.
map generated by a modulo edge detector can result in Recently, there has been an increasing use of fuzzy
a hue edge map that discounts confounding cues due to logic theory for color image segmentation [20,26,63,
shading, shadow, transparency, and highlights. Based on 108}120]. As mentioned above, for color images, the
this work, Ref. [74] designed and fabricated an analog colors tend to form clusters in color space which can be
CMOS VLSI circuit with on-board phototransistor in- regarded as a natural feature space. One problem with
put that computes normalized color and hue. Ref. [75] traditional clustering techniques is that there are only
adopted an improved watershed algorithm which uses two values, either 1 or 0, to indicate to what degree a data
gradient of gray level to segment image into homogene- point belongs to a cluster. This requires well-de"ned
ous regions and oversegmentation is also considered. boundaries between clusters, which is not the usual case
Ref. [76] incorporated the zero-, "rst-, and second-order for real images. This problem can be solved by using
derivatives. Some advantages of the algorithm are: 1. fuzzy set methods. FCM is a method that can allow
Precise region boundaries are detected. 2. Overcome ambiguous boundaries between clusters, and has re-
undesirable undersegmentation caused by the over- ceived much attention [26]. It is an iterative optimization
smoothing and oversegmentation of ramp-like region. 3. method. It calculates the memberships of a data point in
Small segments caused by noise variations in statistically each of the clusters based on the distances between the
coherent regions are removed. point and the cluster centers. The cluster centers are then
We want to emphasize here that edge detection cannot updated based on the resulting clusters. In the iteration,
segment an image by itself. It can only provide useful an objective criterion function is used to minimize the
information about the region boundaries for the higher- distance between the data point in a cluster and the
level systems, or it can be combined with other cluster center, and to maximize the distance between
approaches, e.g., region based approaches [78}83], to cluster centers.
complete the segmentation tasks. In a two-stage (coarse and "ne) segmentation algo-
rithm, Ref. [67] used FCM to assign the unclassi"ed
4.4. Fuzzy techniques pixels at the coarse stage to the closest class using the
detected cluster centers. This technique tried to solve the
The segmentation approaches mentioned above take problem of cluster validity by using a coarse}"ne con-
crisp decisions about regions. Nevertheless, the regions in cept. But its applicability is still limited by the histogram
an image are not always crisply de"ned, and uncertainty thresholding technique. Ref. [26] proposed an iterative
can arise within each level of image analysis and pattern multispectral image segmentation approach based on
recognition. It can occur at the low level in the raw sensor FCM and performed experiments on a color image in
output, and extend all the way through intermediate and RGB and Ohta color spaces. There are two major prob-
higher levels. Since decisions at any level are based on the lems with the FCM method: (1) How to determine the
results of previous levels, any decision made at a previous number of clusters remains unsolved. (2) The computa-
level will have an impact on all higher-level activities. tional cost is quite high for large data sets.
A recognition or computer vision system must have su$- Ref. [108] de"ned a segment as `a collection of touch-
cient #exibility for processing of uncertainty in any of ing pixels having almost the same color while the change
these levels so that the system could retain as much in color is graduala and utilized a fuzzy approach to
information as possible at each level. In this way, the "nal attack this fuzzy concept. It de"ned a contrast fuzzy
output of the system may not be biased too much by membership function on a RGB space as follows:


lower level decisions, unlike the classical approaches. As
0 if Contrast)a
the "rst essential step of a recognition or vision system, 
image segmentation particularly should have such a pro-
" 1 if Contrast'a
A 
vision for representing and manipulating the uncertain- (Contrast!a )/(a !a ) otherwise
ties.   
Fuzzy set theory provides a mechanism to represent where Contrast"((R !R )#(G !G )#(B !B )
T U T U T U
and manipulate uncertainty and ambiguity. Fuzzy oper- is the contrast between two pixels v and w, a and a are
 
ators, properties, mathematics, and inference rules two predetermined thresholds. The concept of contrast is
(IF}THEN rules) have found considerable applications used to determine the homogeneity criterion for the re-
in image segmentation [1,84}107]. Prewitt "rst suggested gion-growing approach, which is de"ned as: the absolute
that the output of image segmentation should be fuzzy contrast (contrast between pixel and region) is low and
2272 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

the local or relative contrast (contrast between pixel and approach is more e!ective in "nding homogeneous re-
its neighbors in the growing direction) is also low. This gions than histogram based approach.
criterion can keep the segment growing even when there
is a gradual change of color on the surface of an object 4.5. Physics based approaches
and also prevents two regions with similar color but
separated by an edge from merging. Using fuzzy mem- Physics based segmentation approaches aim at solving
bership function, Ref. [108] also de"ned degree of farness this problem by employing physical models to locate the

, which is determined by the distance in spatial domain objects' boundaries while eliminating the spurious edges
D
and the contrast in color space, i.e.,
"
;
where of shadow or highlights in a color image. Among the
 B A
mu is the membership function of distance and
is the physics models, `dichromatic re#ection modela [36] and
B A
membership function of contrast. The farness is then used `approximate color-re#ectance model (ACRM)a [37] are
to assign the unclassi"ed pixels to the closest clusters. the most common ones.
The problem is that R, G and B are highly correlated and Re#ection is highly related to the nature of the mater-
subject to the illumination, which makes the contrast ials. Ref. [37] divided materials into di!erent classes:
de"ned by RGB not reliable. optically homogeneous materials like metals, glass and
Ref. [109] proposed a color edge detection approach crystals, and optically inhomogeneous materials such as
based on fuzzy IF}THEN rules in the HSI color space. plastics, paper, textiles and paints. Usually it is very
Linear fuzzy membership functions are used to describe helpful to identify or classify the material in the scene of
the absolute di!erence in the three components of HSI an image before the algorithm is applied. For example,
between two pixels, and, particularly, to describe the we should distinguish metals from dielectrics since they
relevance of hue due to the unstability of hue at low interact with lights in di!erent ways and require di!erent
saturation. Several 3;3 masks are used to describe the algorithms for image understanding.
potential edge structures, and for each mask, a fuzzy Using the color feature of human face, Ref. [51] built
IF}THEN rule is developed. These rules are then com- a skin color model to capture the chromatic character-
bined using fuzzy `ORa operator to infer fuzzy subsets istics. Human skin color fall in a small region in color
representing the potential edges. The fusion of the edges space and can be approximated by a Gaussian distribu-
detected in hue, saturation and intensity is quite natural tion. According to color and shape features of human
using fuzzy logic representation. The "nal fuzzy set faces, Ref. [51] concluded: regardless of the size, orienta-
PEP}HSI (Potential Edge Pixel in HSI space) is obtained tion and viewpoint, human face in color image can be
by a weighted combination of three sets representing identi"ed. In order to segment images containing multi-
three types of edge pixels (S : edge pixels in three compo- colored object and multiple materials, Ref. [52] proposed

nents, S : edge pixels in two components, and S : edge a model consisting of three elements: surface, illumina-
 
pixels in only one component). The membership function tion, and the light transfer function. The parameter space
for PEP}HIS is de"ned as for every element is divided into a set of subspaces, which
allow reasoning about the relationships of adjacent hy-

} "1/2;
#1/3;
#1/6;
. pothesis region. Based on physical parameters, Ref. [52]
NCN FGQ Q Q Q
gave all possible combinations of the subsets and some
Ref. [110] proposed a color image segmentation algo- rules of merging. This approach is easily expendable and
rithm based on fuzzy homogeneity. The fuzzy set theory allows greater complexity in images.
and maximum fuzzy entropy principle are used to map Based on `dichromatic re#ection modela, Ref. [36]
the color image from space domain to fuzzy domain, created a method to determine the amount of interface
which will keep the maximum information. Both the re#ection and body re#ection in a color image pixel by
global and local information is taken into account while pixel, and presented an algorithm for analyzing color
calculating fuzzy homogeneity histogram. The scale- values, which is very useful in color-image understand-
space "lter (SSF) is utilized to analyze the homogeneity ing. `ACRMa [37] demonstrated the independence of
histogram to "nd the appropriate segments. The "nal the spectral composition and geometrical scaling of the
result is transformed from fuzzy domain to space domain light re#ected. This model is consistent with dichromatic
using the inverse S-function. The experimental results re#ection model when the materials are inhomogeneous
demonstrate the e!ectiveness of the proposed approach. dielectrics.
Ref. [111] employed the concept of homogram to extract Physics based segmentation approaches use the same
homogeneous regions in a color image. Utilizing the segmentation techniques discussed before. For example,
fuzzy homogeneity approach to "nd thresholds for each [42] used Canny's edge detector [77] to segment an
color component. Then the segmentation results for the image of a valve based on the ACRM model, and [121]
three-color components are combined. In order to solve applied clustering method to color image segmentation
the problem of over-segmentation, a region merging based on the dichromatic re#ection model. The charac-
process is performed based on color similarity. This teristic of these approaches lies in that they use di!erent
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2273

re#ection models for color images. The existing physics plasm region which are useful for diagnosis. Ref. [125]
based models are e$cient only in image processing for attempted to segment sputum color images in order to
the materials whose re#ection properties are known and build an automatic diagnosis system for lung cancer.
easy to model. There are too many rigid assumptions of After masking nonsputum cell, HNN can make a crisp
these physics models regarding the material type, the classi"cation of cells by labelling pixels as background,
light source and illumination. These conditions may not cytoplasm and nucleus. The technique has yielded cor-
be satis"ed in the real world. Therefore, these models can rect segmentations of complex scene, however, more
be used only in a very limited scope of applications. work needs to be done to solve the overlap of the cells.
In Ref. [126], every boundary pixel is assigned to an
4.6. Neural networks approaches element of the HNN and a local minimum can be found
by HNN. In order to reduce the computing time, the
Arti"cial Neural Networks (ANN) are widely applied pyramid images are used to perform fast segmentation
for pattern recognition. Their extended parallel process- and obtain a global optimal solution. The active region
ing capability and nonlinear characteristics are used for segmentation method is based on a regularization ap-
classi"cation and clustering. ANN explore many compet- proach in which region growing is incorporated into edge
ing hypotheses simultaneously through parallel nets detection.
instead of performing a program of instructions
sequentially, hence ANN can be feasible for parallel pro- 4.6.2. The self-organizing map (SOM)
cessing. Neural networks are composed of many com- The self-organizing map (SOM) projects input space
putational elements connected by links with variable on prototypes of low-dimensional regular grids that can
weights. The complete network, therefore, represents be e!ectively utilized to visualize and explore properties
a very complex set of interdependencies which may in- of the data [127]. Self-organization of Kohonen Feature
corporate any degree of nonlinearity, allowing very gen- Map (SOFM) network is a powerful tool for data cluster-
eral function to be modeled. Training time are usually ing. Ref. [128] employed the watershed segmentation to
very long, but after training, the classi"cation using ANN the luminance component of color image. In order to
is rapid. solve the problem of oversegmentation, the Kohonen
self-organizing map (SOM) network is used. The area of
4.6.1. Hopxeld Neural Networks (HNN) the watershed segments, the chrominance components
Ref. [122] presented the segmentation problem for and the luminance component are input into the network
gray-level image as minimizing a suitable energy function for obtaining the information about how to merge the
for Hop"eld networks. It derived the network architec- image. In Ref. [129], "rst, SOFM is used for quantization
ture from the energy function. Based on the idea in Refs. of the input color image in order to reduce computa-
[122,123] described two algorithms to segment the color tional time, and get an indexed image. Second, local
images using HNN. The "rst algorithm locates the signif- histogram is calculated by using a moving window and
icant peaks by applying histogram analysis and designs index-count vectors are obtained. Third, the index-count
three di!erent networks (one for each color feature). The vectors are used as the training data for SOFM. Finally,
segmentation results of three color components are com- each cluster is mapped from the index-count space to the
bined to get the "nal image. RGB color features, the I , I original image. Ref. [129] stated that SOMF was a fast
 
and I color features, and the Karhunen}Loeve trans- training method and parallel hardware structure was

formation of the RGB color features (K¸}RGB) have also given.
been used in the experiments. Ref. [123] gave another
algorithm built a single Hop"eld network with M;N;S 4.6.3. Other neural networks
neurons to segment color images. M, N are the image Backpropagation (BP) algorithm can be used to seg-
size, and S is the number of selected clusters obtained by ment color images as well. Color features are successively
analyzing the histogram. For both algorithms, histogram input into BP, one pixel at a time, and three input nodes
analysis is very important since it produces the coarse are necessary. Ref. [130] presents the BP architecture
segmentations and determines both network structures and how to train BP network. C-mean and Learning
and their initializations. In the algorithms, the spatial Vector Quantization (LVQ) algorithms are compared
information is also considered in order to produce con- with the back-propagation neural approach. Ref. [131]
sistent color pixel labeling. studied local linear map network (LLM), which is related
An unsupervised algorithm [124] used HNN to seg- to self-organizing maps. LLM network, representing the
ment the color image of liver tissues prepared and stained to-be-learned mappings as a collection of locally valid
by standard method. The results show that the RGB linear mappings that are learned by separate units, is
color space representation of the color images is more a feedforward neural network. LLM is trained in a super-
suitable than HS< and H¸S color spaces. This algorithm vised-learning scheme to yield a probability value for
can automatically extract the nuclei region and cyto- each image pixel belonging to the face region. In order to
2274 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

Table 1
Monochrome image segmentation techniques

Segmentation Method description Advantages Disadvantages

technique

Histogram Requires that the histogram of an It does not need a prior (1) Does not work well for
thresholding image has a number of peaks, each information of the image. an image without any
(mode method) corresponds to a region For a wide class of images obvious peaks or with broad
satisfying the requirement, this and #at valleys
method works very well with low (2) Does not consider the
computation complexity spatial details, so cannot
guarantee that the
segmented regions are
contiguous
Feature space Assumes that each region in the Straightforward for classi"cation (1) How to determine the
clustering image forms a separate cluster in and easy for implementation number of clusters (known
the feature space. Can be generally as cluster validity)
broken into two steps: (1) (2) Features are often image
categorize the points in the feature dependent and how to select
space into clusters; (2) map the features so as to obtain
clusters back to the spatial domain satisfactory segmentation
to form separate regions results remains unclear
(3) Does not utilize spatial
information
Region-based Group pixels into homogeneous Work best when the region (1) Are by nature sequential
approaches regions. Including region growing, homogeneity criterion is easy to and quite expensive both in
region splitting, region merging or de"ne. They are also more noise computational time and
their combination immune than edge detection memory
approach (2) Region growing has
inherent dependence on the
selection of seed region and
the order in which pixels
and regions are examined
(3) The resulting segments
by region splitting appear
too square due to the
splitting scheme
Edge detection Based on the detection of Edge detecting technique is the way (1) Does not work well with
approaches discontinuity, normally tries to in which human perceives objects images in which the edges
locate points with more or less and works well for images having are ill-de"ned or there are
abrupt changes in gray level. good contrast between regions too many edges
Usually classi"ed into two (2) It is not a trivial job to
categories: sequential and parallel produce a closed curve or
boundary
(3) Less immune to noise
than other techniques, e.g.,
thresholding and clustering
Fuzzy approaches Apply fuzzy operators, properties, Fuzzy membership function can be (1) The determination of
mathematics, and inference rules used to represent the degree of fuzzy membership is not
(IF} THEN rules), provide a way to some properties or linguistic a trivial job
handle the uncertainty inherent in phrase, and fuzzy IF}THEN rules (2) The computation
a variety of problems due to can be used to perform involved in fuzzy
ambiguity rather than randomness approximate inference approaches could be
intensive
Neural network Using neural networks to perform No need to write complicated (1) Training time is long
approaches classi"cation or clustering programs. Can fully utilize the (2) Initialization may a!ect
parallel nature of neural networks the results
(3) Overtraining should be
avoided
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2275

Table 2
Characteristics of color spaces

Color space Advantages Disadvantages

RGB Convenient for display Not good for color image processing
due to the high correlation
>IQ Can be used to e$ciently encode color information in the Correlation still exists due to the linear
TV signal of American system; transformation, though not as high as
Partly gets rid of the correlation of RGB; RGB
Involves less computation time;
> is good for edge detection
>;< Can be used to e$ciently encode color information in the Correlation still exists due to the linear
TV signal of European system; transformation, though not as high as
Partly gets rid of the correlation of RGB; RGB
Involves less computation time

I I I Partly gets rid of the correlation of RGB; Correlation still exists due to the linear
  
Involves less computation time; transformation, though not as high as
Can be useful for color image processing RGB

HSI Based on human color perception; Nonremovable singularity and

Useful in some cases where the illumination level varies, numerically unstable at low saturation
because hue is invariant to certain types of highlights, due to nonlinear transformation
shading, and shadows;
Hue can be useful for separating objects with di!erent
colors
Nrgb The individual color components are independent on the Very noisy at low intensities due to
(Normalized rgb) brightness of the image; nonlinear transformation.
Convenient to represent the color plane;
Robust to the change of the illumination
CIE spaces Can control color and intensity information Have the same singularity problem as
(¸HuHvH or ¸HaHbH) independently; other nonlinear transformations do
Direct color comparison can be
performed based on geometric
separation within CIE space, and
e$cient in measuring small color
di!erence

get better segmentation results, a training database in- Constrain Satisfaction Neural Network (CSNN) to per-
cluding di!erent faces and views is used. An oscillatory form MAP segmentation which utilizes the advantages of
cellular neural network (OCNN) is employed in GMRF (Gauss}Markov Random Field). Real world and
Ref. [132] to segment color images. Its architecture synthetic images are tested based on RGB color space.
consists of an array of simple neural oscillators with
inter-connections limited to the nearest neighborhood. 4.7. Discussions on color image segmentation
The advantage of OCNN is that it solves a bottleneck
created by the immense number of interconnections There are two critical issues for color image segmenta-
between a global separator and the oscillators. Connec- tion: (1) what segmentation method should be utilized;
tedness between neighboring color pixels is de"ned as and (2) what color space should be adopted. At present,
color connectedness matrix (CCM) to group the cor- color image segmentation methods are generally ex-
related segments. Simulation results demonstrate the tended from monochrome segmentation approaches.
validity and performance of OCNN. Ref. [133] used a Several approaches applied to color image are discussed
2276 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

in this section, including histogram thresholding, region can model the cognitive activity of the human beings
based approaches, edge detection and fuzzy techniques. [114]. Fuzzy operators, properties, mathematics, and
A combination of these approaches is often utilized for inference rules (IF}THEN rules) have found more and
color image segmentation [21,33,62,63,65,78}83,134]. more applications in image segmentation. Despite the
Table 1 is a summary of these approaches. Other ap- computational cost, fuzzy approaches perform as well as
proaches, such as segmentation using Markov Random or better than their crisp counterparts. The more impor-
Field [135}137] and segmentation based on texture tant advantage of a fuzzy methodology lies in that the
[28,138,139] can also be found from the literature. fuzzy membership function provides a natural means to
The selection of a color space for image processing is model the uncertainty in an image. Subsequently, fuzzy
image/application dependent. There is no any color segmentation results can be utilized in feature extraction
space which is better than others and suitable to all kinds and object recognition phases of image processing and
of images yet. Table 2 lists the characteristics of di!erent computer vision. Fuzzy approach provides a promising
color spaces for color image segmentation. means for color image segmentation.

5. Summary
References
There is no universal theory on color image segmenta-
tion yet. All of the existing color image segmentation [1] S.K. Pal et al., A review on image segmentation tech-
niques, Pattern Recognition 29 (1993) 1277}1294.
approaches are, by nature, ad hoc. They are strongly
[2] K.S. Fu, J.K. Mui, A survey on image segmentation,
application dependent, in other words, there are no gen-
Pattern Recognition 13 (1981) 3}16.
eral algorithms and color space that are good for all [3] E.M. Riseman, M.A. Arbib, Computational techniques in
color images. An image segmentation problem is basi- the visual segmentation of static scenes, Comput. Vision
cally one of psychophysical perception, and it is essential Graphics Image Process. 6 (1977) 221}276.
to supplement any mathematical solutions by a priori [4] W. Skarbek, A. Koschan, Colour image segmentation
knowledge about the picture knowledge. Most gray level } a survey, Technical Report, Tech. Univ. of Berlin,
image segmentation techniques could be extended to October 1994.
color image, such as histogram thresholding, clustering, [5] R.M. Haralick, L.G. Shapiro, Image segmentation tech-
region growing, edge detection and fuzzy based ap- niques, Comput. Vision Graphics Image Process. 29
proaches. They can be directly applied to each compon- (1985) 100}132.
[6] P.K. Sahoo et al., A survey of thresholding techniques,
ent of a color space, then the results can be combined in
Comput. Vision Graphics Image Process. 41 (1988)
some way to obtain the "nal segmentation result. How- 233}260.
ever, one of the problems is how to employ the color [7] L. Spirkovska, A summary of image segmentation tech-
information as a whole for each pixel. When color is niques, NASA Technical Memorandum 104022, June
projected onto three components, the color information 1993.
is so scattered that the color image becomes simply [8] C.H. Chen, On the statistical image segmentation tech-
a multispectral image and the color information that niques, Proceedings of IEEE Conference on Pattern Rec-
human can perceive is lost. Another problem is how to ognition and Image Processing, 1981, pp. 262}266.
choose the color representation for segmentation, since [9] A. Rosenfeld, M. Thurston, Edge and curve detection for
each color representation has its advantages and disad- visual scene analysis, IEEE Trans. Comput. C-20 (1971)
562}569.
vantages.
[10] A. Rosenfeld, M. Thurston, Y. Lee, Edge and curve detec-
In most of the existing color image segmentation ap- tion further experiments, IEEE Trans. Comput. C-21
proaches, the de"nition of a region is based on similar (1972) 677}715.
color. This assumption often makes it di$cult for many [11] M. Hueckel, An operator which locates edges in digitized
algorithms to separate the objects with highlights, pictures, J. Assoc. Comput. Mach. 18 (1) (1971) 113}125.
shadows, shading or texture which cause inhomogeneous [12] M. Hueckel, A local visual operator which recognizes
colors of the objects' surface. Using HSI can solve this edges and lines, J. Assoc. Comput. Mach. 20 (4) (1973)
problem to some extent except that hue is unstable at low 634}647.
saturation. Some physics based models have been pro- [13] J. Gauch, Chi-Wan Hsia, A comparison of three color
image segmentation algorithm in four color spaces, SPIE
posed to "nd the objects' boundaries based on the type of
Vol. 1818 Visual Communications and Image Processing
materials, but these models have too many restrictions
'92, 1992, pp. 1168}1181.
which limit them to be extensively employed. [14] D.E.P. Hoy, On the use of color imaging in experimental
The fuzzy set theory has attracted more and more applications, Exp. Tech. 21 (4) (1997) 17}19.
attention in the area of image processing. Fuzzy set [15] M. Chapron, A new chromatic edge detector used for
theory provides us with a suitable tool which can repres- color image segmentation, IEEE International Confer-
ent the uncertainties arising in image segmentation and ence on Pattern Recognition, A, 1992, pp. 311}314.
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2277

[16] M.T. Orchard, C.A. Bouman, Color quantization of [34] S. Tominaga, Expansion of color images using three
images, IEEE Trans. Signal Process. 39 (12) (1991) perceptual attributes, Pattern Recognition Lett. 6 (1987)
2677}2690. 77}85.
[17] D. Comaniciu, P. Meer, Robust analysis of feature [35] N. Vandenbroucke, L. Macaire, J.G. Posaire, Color
spaces: color image segmentation, IEEE Conference pixels classi"cation in an hybrid color space, IEEE Inter-
on Computer Vision and Pattern Recognition, 1997, national Conference on Image Processing, 1998, pp.
pp. 750}755. 176}180.
[18] M. Pietikainen et al., Accurate color discrimination with [36] S.A. Shafer, Using color to separate re#ection compo-
classi"cation based on feature distributions International nents, Color Res. Appl. 10 (4) (1985) 210}218.
Conference on Pattern Recognition, C, 1996, pp. [37] G. Healey, Using color for geometry-insensitive segmen-
833}838. tation, Opt. Soc. Am. 22 (1) (1989) 920}937.
[19] E. Littmann, H. Ritter, Adaptive color segmentation [38] J. Kender, Saturation, hue, and normalized color: calcu-
} a comparison of neural and statistical methods, IEEE lation, digitization e!ects, and use, Computer Science
Trans. Neural Network 8 (1) (1997) 175}185. Technical Report, Carnegie Mellon University, 1976.
[20] G.S. Robinson, Color edge detection, Opt. Eng. 16 (5) [39] C.K. Yang, W.H. Tsai, Reduction of color space dimen-
(1977) 479}484. sionality by moment-preserving thresholding and its ap-
[21] Y. Ohta, T. Kanade, T. Sakai, Color information for plication for edge detection in color images, Pattern
region segmentation, Comput. Graphics Image Process. Recognition Lett. 17 (1996) 481}490.
13 (1980) 222}241. [40] G.J. Klinker, S.A. Shafer, T. Kanade, A physical ap-
[22] P. Golland, A.M. Bruckstein, Why R.G.B.? Or How proach to color image understanding, Int. J. Comput.
to design color displays for martians, Graphical Models Vision 4 (1990) 7}38.
Image Process. 58 (5) (1996) 405}412. [41] M.H. Brill, Image segmentation by object color:
[23] Nevatia, A color edge detector and its use in scene seg- a unifying framework and connection to color constancy,
mentation, IEEE Trans. System Man Cybernet. SMC-7 Opt. Soc. Am. 7 (10) (1990) 2041}2047.
(11) (1977) 820}826. [42] G. Healey, Segmenting images using normalized color,
[24] I. Andreadis, M.A. Browne, J.A. Swift, Image pixel classi- IEEE Trans. System Man Cybernet. 22 (1) (1992) 64}73.
"cation by chromaticity analysis, Pattern Recognition [43] A.P. Petrov, L.L. Kontsevich, Properties of color images
Lett. 11 (1990) 51}58. of surfaces under multiple illuminants, Opt. Soc. Am. 11
[25] J.C. Terrillon, M. David, S. Akamatsu, Detection of hu- (10) (1994) 2745}2749.
man faces in complex scene images by use of a skin color [44] B.A. Maxwell, S.A. Shafer, Physics-based segmentation:
model and of invariant Fourier-Mellin moments, IEEE moving beyond color, IEEE International Conference on
International Conference on Pattern Recognition, 1998, Computer Vision and Pattern Recognition, 1996, pp.
pp. 1350}1355. 742}749.
[26] T.L. Huntsberger, C.L. Jacobs, R.L. Cannon, Iterative [45] G.J. Klinker, S.A. Shafer, Takeo Kanade, Image segmen-
fuzzy image segmentation, Pattern Recognition 18 (2) tation and re#ection analysis through color, Proceedings
(1985) 131}138. of the Image Understanding Workshop, Morgan Kauf-
[27] T. Carron, P. Lambert, Color edge detector using jointly mann, San Mateo, CA, 1988, pp. 835}838.
hue, saturation and intensity, IEEE International Confer- [46] S.L. Guth, Model for color vision and light adaptation,
ence on Image Processing, Austin, USA, 1994, pp. Opt. Soc. Am. 8 (6) (1991) 976}993.
977}1081. [47] S.A. Shafer, T. Kanade, Using shadows in "nding surface
[28] Y. Rui, A.C. She, T.S. Huang, Automated region segmen- orientations, Comput. Vision Graphics Image Process.
tation using attraction-based grouping in spatial- 22 (1983) 145}176.
color-texture space, International Conference on Image [48] G.J. Klinker, S.A. Shafer, T. Kanade, The measurement of
Processing, A, 1996, pp. 53}56. highlights in color images, Int. J. Comput. Vision 2 (1)
[29] W.S. Kim, R.H. Park, Color image palette construction (1988) 7}32.
based on the HSI color system for minimizing the recon- [49] S. Shafer, T. Kanade, G. Klinker, C. Novak, Physics-
struction error, IEEE International Conference on Image based models for early vision by machine SPIE, Perceiv-
Processing, C, 1996, pp. 1041}1044. ing, Measuring, and Using Color, Vol. 1250, Santa Clara,
[30] P.W.M. Tsang, W.H. Tsang, Edge detection on object February 1990, pp. 222}235.
color, IEEE International Conference on Image Process- [50] R. Bajcsy, S. Wooklee, A. Leonardis, Detection of di!use
ing, C, 1996, pp. 1049}1052. and specular interface re#ections and inter-re#ections by
[31] E. Tepichin, J.G. Suarez-Romero, G. Ramirez, Hue, color image segmentation, Int. J. Comput. Vision 17 (3)
brightness, and saturation manipulation of di!ractive (1996) 241}272.
colors, Opt. Eng. 34 (10) (1995) 2886}2890. [51] M.H. Yang, N. Ahuja, Detecting human faces in color
[32] K.M. Kim, C.S. Lee, Y.H. Ha, Color image quantization images, IEEE International Conference on Image Pro-
using weighted distortion measure of HVS color activity, cessing, 1998, pp. 127}130.
IEEE International Conference on Pattern Recognition, [52] B.A. Maxwell, S.A. Shafer, Physics-based segmentation:
1996, pp. 1035}1039. looking beyond color, IEEE Computer Vision and Pat-
[33] D.C. Tseng, C.H. Chang, Color segmentation using per- tern Recognition, June 16}20, 1996, pp. 867}878.
ceptual attributes, IEEE International Conference on [53] W. Power, R. Clist, Comparison of supervised learning
Pattern Recognition, A, 1992, pp. 228}231. techniques applied to color segmentation of fruit image,
2278 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

Proceedings of SPIE, Intelligent Roberts and Computer [72] P.E. Trahanias, A.N. Venetsanopoulos, Color edge detec-
Vision XV: Algorithms, Techniques, Active Vision, and tion using vector order statistics, IEEE Trans Image
Materials Handling, Boston, Massachusetts, November Process. 2 (2) (1993) 259}265.
19}21, 1996, pp. 370}381. [73] P.E. Trahanias, A.N. Venetsanopoulos, Vector order
[54] G.A. Hance, S.E. Umbaugh, R.H. Moss, W.V. Stoecker, statistics operators as color edge detectors, IEEE Trans.
Unsupervised color image segmentation with application Systems Man Cybernet.-Part B: Cybernetics 26 (1) (1996)
to skin tumor borders, IEEE Eng. Med. Biol. 15 (1) (1996) 135}143.
104}111. [74] F. Perez, C. Koch, Toward color image segmentation in
[55] J. Wu, H. Yan, A.N. Chalmers, Color image segmentation analog VLSI: algorithm and hardware, Int. J. Comput.
using fuzzy clustering and supervised learning, J. Elec- Vision 12 (1) (1994) 17}42.
tron. Image 3 (4) (1994) 397}403. [75] S. Ji, H.W. Park, Image segmentation of color image
[56] K.B. Eom, Segmentation of monochrome and color tex- based on region coherency, IEEE International Confer-
ture using moving average modeling approach, Image ence on Image Processing, 1998, pp. 80}83.
Vision Comput. 17 (1999) 233}244. [76] J. Luo, R.T. Gray, H.C. Lee, Incorporation of derivative
[57] T. Uchiyama, M.A. Arbib, Color image segmentation priors in adaptive bayesian color image segmentation,
using competitive learning, IEEE Trans. Pattern Anal. IEEE International Conference on Image Processing,
Mach. Intell. 16 (12) (1994) 1197}1206. 1998, pp. 780}784.
[58] B. Schacter, L. Davis, A. Rosenfeld, Scene segmentation [77] J. Canny, A computational approach to edge detection,
by cluster detection in color space, Department of Com- IEEE Trans. Pattern Anal. Mach. Intell. 8 (6) (1986)
puter Science, University of Maryland, College Park, 679}698.
MD, November 1975. [78] N. Ito et al., The combination of edge detection and
[59] A. Sarabi, J.K. Aggarwal, Segmentation of chromatic region extraction in non-parametric color image segmen-
images, Pattern Recognition 13 (6) (1981) 417}427. tation, Inform. Sci. 92 (1996) 277}294.
[60] S.A. Underwood, J.K. Aggarwal, Interactive computer [79] Y. Xiaohan, J. Yla, Image segmentation combining re-
analysis of aerial color infrared photographs, Comput. gion growing with edge detection, 11th International
Graphics Image Process. 6 (1977) 1}24. Conference on Pattern Recognition, The Netherland,
[61] J.M. Tenenbaum, T.D. Garvey, S. Weyl, H.C. Wolf, An August 30}September 3 1992, pp. 481}484.
interactive facility for scene analysis research, Technical [80] T. Pavlidis, Y.T. Liow, Integrating region growing and
Note 87, Arti"cial Intelligence Center, Stanford Research edge detection, IEEE Trans. Pattern Anal. Mach. Intell.
Institute, Menlo Park, CA, 1974, 230}239. 12 (3) (1990) 225}233.
[62] R. Ohlander, K. Price, D.R. Reddy, Picture segmentation [81] Q. Huang et al., Foreground/background segmentation
using a recursive region splitting method, Comput. of color images by integration of multiple cues, Interna-
Graphics Image Process. 8 (1978) 313}333. tional Conference on Image Processing, A, 1995, pp.
[63] S. Tominaga, Color image segmentation using three per- 246}249.
ceptual attributes, IEEE Proceedings of the Conference [82] Q. Huang, B. Dom, N. Megiddo, W. Niblack, Segmenting
on Computer Vision and Pattern Recognition, Los and representing background in color images, Interna-
Alamos, CA, 1986, pp. 628}630. tional Conference on Pattern Recognition, C, 1996, pp.
[64] G.D. Guo, S. Yu, S.D. Ma, Unsupervised segmentation of 13}17.
color images, IEEE International Conference on Image [83] R.I. Taylor, P.H. Lewis, Color image segmentation using
Processing, 1998, pp. 299}302. boundary relaxation, IEEE International Conference on
[65] M. Celenk, A color clustering technique for image seg- Pattern Recognition, C, 1992, pp. 721}724.
mentation, Graphical Models Image Process. 52 (3) [84] J.M. Keller, M.R. Gray, J.A. Givens, A fuzzy K-nearest
(1990) 145}170. neighbor algorithm, IEEE Trans. Systems Sci. Cybernet.
[66] S. Tominaga, Color classi"cation of natural color images, SMC-15 (1985) 580}585.
Color Res. Appl. 17 (4) (1992). [85] J.M. Keller, C.L. Carpenter, Image segmentation in the
[67] Y.W. Lim, S.U. Lee, On the color image segmentation presence of uncertainty, Int. J. Intell. Systems SMC-15
algorithm based on the thresholding and the fuzzy c-means (1990) 193}208.
techniques, Pattern Recognition 23 (9) (1990) 935}952. [86] L.K. Huang, M.J. Wang, Image thresholding by minimiz-
[68] F. Ferri, E. Vidal, Color image segmentation and labeling ing the measures of fuzziness, Pattern Recognition 28 (1)
through multi-edit condensing, Pattern Recognition Lett. (1995) 41}51.
13 (1992) 561}568. [87] A. De Luca, S. Termini, A de"nition of a nonprobabilistic
[69] A. Tremeau, N. Borel, A region growing and merging entropy in the setting of fuzzy set theory, Inform. Control
algorithm to color segmentation, Pattern Recognition 30 20 (1972) 301}312.
(7) (1997) 1191}1203. [88] N.R. Pal, S.K. Pal, Object-background segmentation us-
[70] H.D. Cheng, Y. Sun, A hierarchical approach to color ing new de"nition of entropy, IEE Proc. Part E 136 (4)
image segmentation using homogeneity, IEEE Trans. Im- (1989) 284}295.
age Process. (2001), in press. [89] M.P. Windham, Geometrical fuzzy clustering algorithms,
[71] L. Macaire, V. Ultre, J.-G. Postaire, Determination of Fuzzy Sets and Systems 10 (1983) 271}279.
compatibility coe$cients for color edge detection by re- [90] S.K. Pal, R.A. King, On edge detection of X-ray images
laxation, International Conference on Image Processing, using fuzzy sets, IEEE Trans. Pattern Anal. Mach. Intell.
C, 1996, pp. 1045}1048. PAMI-5 (1) (1983) 69}77.
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2279

[91] D.N. Chun, H.S. Yang, Robust image segmentation using [111] H.D. Cheng, X.H. Jiang, Homogram thresholding ap-
genetic algorithm with a fuzzy measure, Pattern Recogni- proach to color image segmentation, International Con-
tion 29 (7) (1996) 1195}1211. ference on Computer Vision, Pattern Recognition and
[92] S.K. Pal, A. Rosenfeld, Image enhancement and thre- Image Processing, Atlantic City, February 27}March 3,
sholding by optimization of fuzzy compactness, Pattern 2000.
Recognition Lett. 7 (1988) 77}86. [112] T. Carron, P. Lambert, Symbolic fusion of hue-chroma-
[93] S. Tamura, S. Higuchi, K. Tanaka, Pattern classi"cation intensity features for region segmentation, International
based on fuzzy relations, IEEE Trans. System Man Conference on Image Processing, B, 1996, pp. 971}974.
Cybernet SMC-1 (1) (1971) 61}66. [113] A.E. Pienkowski, T.J. Dennis, Applications of fuzzy logic
[94] R.L. Cannon, J.V. Dave, J.C. Bezdek, E$cient implemen- to arti"cial color vision, SPIE Comput. Vision Robots
tation of the fuzzy c-means clustering algorithms, IEEE 595 (1985) 50}55.
Trans. Pattern Anal. Mach. Intell. PAMI-8 (2) (1986) [114] W. Pedrycz, Fuzzy sets in pattern recognition: methodo-
249}255. logy and methods, Pattern Recognition 23 (1/2) (1990)
[95] J.C. Bezdek, P.F. Castelaz, Prototype classi"cation and 121}146.
feature selection with fuzzy sets, Pattern Recognition [115] C.W. Tao, W.E. Thompson, A fuzzy if}then approach to
Lett. 14 (1993) 483}488. edge detection. FUZZY-IEEE 93, San-Francisco, USA,
[96] Gath, A.B. Geva, Unsupervised optimal fuzzy clustering, 1993, pp. 1356}1360.
IEEE Trans. Pattern Anal. Mach. Intell. PAMI-11 (7) [116] A. Moghaddamzadeh, N. Bourbakis, A fuzzy technique
(1989) 773}781. for image segmentation of color images, IEEE Word
[97] T.L. Huntsberger, C. Rangarajan, S.N. Jayaramamurphy, Congress on Computational Intelligence: FUZZY-IEEE,
Representation of the uncertainty in computer vision Orlando, Florida, June 1994.
using fuzzy sets, IEEE Trans. Comput. C-35 (2) (1993) [117] Y.S. Chen, H.Y. Hwang, B.T. Chen, Color image analysis
145}156. using fuzzy set theory, International Conference on Im-
[98] M.P. Windham, Geometrical fuzzy clustering algorithms, age Processing, A, 1995, pp. 242}245.
Fuzzy Sets and Systems 10 (1983) 271}279. [118] A. Moghaddamzadeh, N.G. Bourbakis, Segmentation of
[99] S.K. Pal, R.A. King, Prototype classi"cation and feature color images with highlights and shadows using fuzzy
selection with fuzzy sets, Electron. Lett. 16 (10) (1993) reasoning, IS & T/SPIEs Symposium, Electronic Imag-
376}378. ing: Science and Technology, San Jose, CA, February
[100] X.L. Xie, G. Beni, A validity measure for fuzzy clustering, 1995, pp. 5}10.
IEEE Trans. Pattern Anal. Mach. Intell. 13 (8) (1991) [119] A. Moghaddamzadeh, N. Bourbakis, A fuzzy approach
841}847. for smoothing and edge detection in color images, IS
[101] J.K. Udupa, S. Samarasekera, Fuzzy connectedness and & T/SPIE Symposium, Electronic Imaging: Science and
object de"nition: theory, algorithms and applications in Technology, San Jose, CA, February 1995.
image segmentation, Graphical Models Image Process. [120] M. Mari, S. Dellepiane, A segmentation method based on
58 (3) (1996) 246}261. fuzzy topology and clustering, IEEE International Con-
[102] I. Bloch, Fuzzy connectivity and mathematical morpho- ference on Pattern Recognition, B, 1996, pp. 565}569.
logy, Pattern Recognition Lett. 14 (1993) 483}488. [121] C.K. Ong, T. Matsuyama, Robust color segmentation
[103] A. Rosenfeld, Fuzzy digital topology, Inform. Control 40 using the dichromatic re#ection model, IEEE Interna-
(1) (1979) 76}87. tional Conference on Pattern Recognition, 1998, pp.
[104] A. Rosenfeld, The fuzzy geometry of image subsets, Pat- 780}784.
tern Recognition Lett. 2 (1984) 311}317. [122] C.-L Huang, Pattern image segmentation using modi"ed
[105] K. Tsuda, M. Minoh, K. Ikeda, Extracting straight lines Hop"eld model, Pattern Recognition Lett. 13 (1999)
by sequential fuzzy clustering, Pattern Recognition Lett. 345}353.
17 (1996) 643}649. [123] P. Campadelli, D. Medici, R. Schettini, Color image seg-
[106] S.Z. Selim, M.A. Ismail, On the local optimality of the mentation using Hop"eld networks, Image Vision Com-
fuzzy isodata clustering algorithm, IEEE Trans. Pattern put. 15 (1997) 161}166.
Anal. Mach. Intell. PAMI-8 (2) (1986) 284}288. [124] M. Sammouda, R. Sammouda, N. Niki, K. Mukai, Seg-
[107] S.K. Pal, Image segmentation using fuzzy correlation, mentation and analysis of liver cancer pathological color
Inform. Sci. 62 (1992) 223}250. image based on arti"cial neural networks, IEEE 1999
[108] A. Moghaddamzadeh, N. Bourbakis, A fuzzy region International Conference on Image Processing, October
growing approach for segmentation of color images, Pat- 24}28, 1999, Kobe, Japan, pp. 392}396.
tern Recognition 30 (6) (1997) 867}881. [125] R. Sammouda, N. Niki, H. Nishitani, Segmentation of
[109] T. Carron, P. Lambert, Fuzzy color edge extraction by sputum color image for lung cancer diagnosis based on
inference rules quantitative study and evaluation of per- neural networks, IEICE Trans. Inform. Systems E81-D
formances, International Conference on Image Process- (8) (1998) 862}871.
ing, B, 1995, pp. 181}184. [126] H. Iwata, H. Nagahashi, Active region segmentation of
[110] H.D. Cheng, J. Li, Fuzzy homogeneity and scale space color images using neural networks, Systems Comput. J.
approach to color image segmentation, International 29 (4) (1998) 1}10.
Conference on Computer Vision, Pattern Recognition [127] J. Vesanto, E. Alhoniemi, Clustering of the self-organiz-
and Image Processing, Atlantic City, February 27}March ing map, IEEE Trans. Neural Networks 11 (3) (2000)
3, 2000. 586}600.
2280 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281

[128] S. Ji, H.W. Park, Image segmentation of color image IEEE 1999 International Conference on Image
based on region coherency, 1998 International Confer- Processing, KOBE, Japan, October 24}28 1999,
ence on Image Processing, Chicago, Illinois, USA, Octo- pp. 236}239.
ber 4}7 1998, pp. 80}83. [134] R. Schettini, A segmentation algorithm for color images,
[129] Y.S. Lo, S.C. Pei, Color image segmentation using local Pattern Recognition Lett. 14 (1993) 499}506.
histogram and self-organization of Kohonen feature map, [135] C.L. Huang, T.Y. Cheng, C.C. Chen, Color
1999 International Conference on Image Processing, images segmentation using scale space "lter and markov
KOBE, Japan, October 24}28 1999, pp. 232}239. random "eld, Pattern Recognition 25 (10) (1992)
[130] P. Lescure, V. Meas-Yedid, H. Dupoisot, G. Stamon, 1217}1229.
Color segmentation on biological microscope images, [136] Y.N. Sun, C.S. Wu, X.Z. Lin, N.H. Chou, Color image
Proceeding of SPIE, Application of Arti"cial Neural analysis for liver tissue classi"cation, Opt. Eng. 32 (7)
Networks in Image Processing IV, San Jose, California, (1993) 1609}1614.
January 28}29, 1999, pp. 182}193. [137] J.Q. Liu, Y.H. Yang, Multiresolution color image seg-
[131] R. Rae, H.J. Ritter, Recognition of human head orienta- mentation, IEEE Trans. Pattern Anal. Mach. Intell. 16 (7)
tion based on arti"cial neural networks, IEEE Trans. (1994) 689}700.
Neural Network 9 (2) (1998) 257}265. [138] T. Caelli, D. Reye, On the classi"cation of image regions
[132] C.Y. Ho, H. Kurokawa, A learning algorithm for oscilla- by color, texture and shape, Pattern Recognition 26 (4)
tory cellular neural networks, Neural Networks 12 (1999) (1993) 461}470.
825}836. [139] K. Valkealahti, E. Oja, Reduced multidimensional histo-
[133] F. Kurugollu, B. Sankur, MAP segmentation of color grams in color texture description, International Confer-
images using constraint satisfaction neural network, ence on Pattern Recognition, 1998, pp. 1057}1061.

About the Author*HENG-DA CHENG received Ph.D. degree in Electrical Engineering from Purdue University, West Lafayette, IN,
in 1985 (supervisor: K.S. Fu). Now he is Full Professor, Department of Computer Science, and an Adjunct Full Professor, Department of
Electrical Engineering, Utah State University, Logan, UT. Dr. Cheng also holds a Visiting Professor position of Northern Jiaotong
University, China.
Dr. Cheng has published more than 180 technical papers, is the co-editor of the book, Pattern Recognition: Algorithms, Architectures
and Applications (World Scienti"c, 1991), and the editor for a couple conference proceedings. His research interests include image
processing, pattern recognition, computer vision, arti"cial intelligence, medical information processing, fuzzy logic, genetic algorithms,
neural networks, parallel processing, parallel algorithms, and VLSI architectures.
Dr. Cheng was the general chair and program chair of the Third International Conference on Computer Vision, Pattern Recognition
and Image Precessing (CVPRIP'2000), 2000; was the general chair and program chair of the First International Workshop on Computer
Vision, Pattern Recognition and Image Processing (CVPRIP'98), 1998, and was the Program Co-Chairman of Vision Interface '90,
1990. He served as a program committee member and session chair for many conferences, and as a reviewer for many scienti"cal
journals and conferences.
Dr. Cheng has been listed in Who's Who in the World, Who's Who in America, Who's Who in Communications and Media, Who's Who in
Finance and Industry, Who's Who in Science and Engineering, Men of Achievement, 2000 Notable American Men, International Leaders in
Achievement, Five Hundred Leaders of Inyuence, International Dictionary of Distinguished Leadership, etc. He is appointed as a Member of
the Advisory Council, the International Biographical Center, England, and a Member of the Board of Advisors, the American Biographical
Institute, USA.
Dr. Cheng is a Senior Member of the IEEE Society, and a Member of the Association of Computing Machinery. Dr. Cheng is also an
Associate Editor of Pattern Recognition and an Associate Editor of Information Sciences.

About the Author*XIHUA JIANG received the Bachelor of Science degree from the Department of Computer and System Science,
Nankai University, Tianjin, China, in 1988, the Master of Science degree from the Department of Computer Science and Technology,
Peking University, Beijing, China, in 1997, and the Master of Science degree from the Department of Computer Science, Utah State
University, Logan, UT, in 1999.
From 1988 to 1994, he was a research assistant and software engineer in the Department of Computer and System Science, Nankai
University, Tianjin, China, and participated in the design, development, and implementation of several robot control systems for the
national hi-tech plan in China. Since 1999, he has been with the Software Engineering Department at Citrix Systems, Inc. as a software
engineer. His research interests are fuzzy logic, image processing and server-client computing.

About the Author*YING SUN received the Bachelor of Science degree in computer software in 1989 from the Department of Computer
and System Science, Nankai University, Tianjin, China, and the Master of Science degree in computer science from Utah State
University, Logan, UT, in 1999.
From 1991 to 1995, she was a software engineer in the Computing Center of Tianjin Foreign Trade and Economic Relations
Committee, Tianjin, China, and participated in the design, development, and implementation of several "nancial management
information systems for the state-run foreign trade companies in Tianjin, China. From 1996 to 1997, she joined Motorola (China)
Electronics Ltd. and worked in Management Information System Department, Cellular Subscriber Sector, as an application analyst.
She conducted software project design, plan, program, and trouble-shooting. Since 1999, she has been with the information Technology
 H.D. Cheng et al. / Pattern Recognition 34 (2001) 2259}2281 2281

Group, Personal Communication Systems, Motorola Inc. in Libertyville, ILL, as a software engineer. Her research interests include
computer algorithms and image processing methodology.

About the Author*JINGLI WANG received the bachelor of science degree in computer science in 1990 from the Computer Science
Department, Zhengzhou University, China. From 1990 to 1999, she was a software engineer in Chinese Academy of Sciences,
participated in remote sensing instruments design, development, and implementation, worked on remote sensing information process-
ing, such as real-time control and data processing, simulation, and calibration for the satellite and airborne altimeter and conducted
software project designing, planning and programming. Since 1999, she has been a graduate student in Utah State University with
a major in Computer Science. Her research interests include software engineering and image processing methodology.