Digital Image Processing

Basic Methods for Image

Segmentation

Christophoros Nikou
[email protected]

Images taken from:

R. Gonzalez and R. Woods. Digital Image Processing, Prentice Hall, 2008.
Computer Vision course by Svetlana Lazebnik, University of North Carolina at Chapel Hill
(http://www.cs.unc.edu/~lazebnik/spring10/).

University of Ioannina - Department of Computer Science and Engineering

2 Image Segmentation
• Obtain a compact representation of the image to
be used for further processing.
• Group together similar pixels
• Image intensity is not sufficient to perform
semantic segmentation
– Object recognition
• Decompose objects to simple tokens (line segments, spots,
corners)
– Finding buildings in images
• Fit polygons and determine surface orientations.
– Video summarization
• Shot detection
C. Nikou – Digital Image Processing
3 Image Segmentation (cont.)
Goal: separate an image into “coherent”
regions.
• Basic methods
– point, line, edge detection
– thresholding
– region growing
– morphological watersheds
• Advanced methods
– clustering
– model fitting.
– probabilistic methods.
–…

C. Nikou – Digital Image Processing

4 Fundamentals
• Edge information is in general not sufficient.

Constant intensity
(edge-based
segmentation)

Textured region
(region-based
segmantation)

C. Nikou – Digital Image Processing

5 Point, line and edge detection
• First order derivatives produce
thick edges at ramps.
• Second order derivatives are non
zero at the onset and at the end
of a ramp or step edge (sign
change).
• Second order derivatives respond
stronger at isolated points and
thin lines than first order
derivatives.

C. Nikou – Digital Image Processing

6 Detection of isolated points
• Simple operation using the Laplacian.

1 if  2
f ( x, y)  T
g ( x, y )  

0 otherwise

C. Nikou – Digital Image Processing

7 Line detection
• The Laplacian is
also used here.
• It has a double
response
– Positive and
negative values at
the beginning and
end of the edges.
• Lines should be thin
with respect to the
size of the detector Absolute value Positive value

C. Nikou – Digital Image Processing

8 Line detection (cont.)
• The Laplacian is isotropic.
• Direction dependent filters localize 1 pixel thick
lines at other orientations (0, 45, 90).

C. Nikou – Digital Image Processing

9 Line detection (cont.)
• The Laplacian is isotropic.
• Direction dependent filters
localize 1 pixel thick lines at
other orientations (0, 45, 90).

C. Nikou – Digital Image Processing

10 Edge detection
• Edge models
– Ideally, edges should be 1 pixel thin.
– In practice, they are blurred and noisy.

Step edge Ramp edge Roof edge

C. Nikou – Digital Image Processing

11 Edge detection (cont.)

C. Nikou – Digital Image Processing

12 Edge detection (cont.)
Edge point detection
• Magnitude of the first
derivative.
• Sign change of the second
derivative.

Observations:
• Second derivative produces
two values for an edge
(undesirable).
• Its zero crossings may be
used to locate the centres of
thick edges.
C. Nikou – Digital Image Processing
13 Edge model and noise

C. Nikou – Digital Image Processing

14 Edge model and noise (cont.)

C. Nikou – Digital Image Processing

15
Fundamental steps in edge
detection
• Image smoothing for noise reduction.
– Derivatives are very sensitive to noise.
• Detection of candidate edge points.
• Edge localization.
– Selection of the points that are true members
of the set of points comprising the edge.

C. Nikou – Digital Image Processing

16 Image gradient
• The gradient of an image:

The gradient points in the direction of most rapid increase

in intensity.
The gradient direction is given by
The edge strength is given by the gradient magnitude

Source: Steve Seitz

C. Nikou – Digital Image Processing
17 Gradient operators
f ( x, y )
 f ( x  1, y)  f ( x, y )
x

f ( x, y )
 f ( x, y  1)  f ( x, y )
y

Roberts operators

C. Nikou – Digital Image Processing

18 Gradient operators (cont.)

Integrates image
smoothing

C. Nikou – Digital Image Processing

19 Gradient operators (cont.)

Diagonal edges

C. Nikou – Digital Image Processing

20 Gradient operators (cont.)
Image Sobel |gy|

Sobel |gx| Sobel |gx|+|gx|

C. Nikou – Digital Image Processing

21 Gradient operators (cont.)
Image Sobel |gy|

Image smoothed
prior to edge
detection.
The wall bricks are
smoothed out.

Sobel |gx| Sobel |gx|+|gx|

C. Nikou – Digital Image Processing

22 Gradient operators (cont.)
Diagonal Sobel filters

C. Nikou – Digital Image Processing

23 Gradient operators (cont.)
Thresholded Sobel gradient amplitydes at 33% of max value

Thresholded gradient Image smoothing prior to

gradient thresholding

C. Nikou – Digital Image Processing

24 The LoG operator

• A good place to look for edges is the maxima of the first

derivative or the zeros of the second derivative.
• The 2D extension approximates the second derivative by
the Laplacian operator (which is rotationally invariant):

 2
f  2
f
 f ( x, y )  2  2
2

x y

• Marr and Hildreth [1980] argued that a satisfactory

operator that could be tuned in scale to detect edges is the
Laplacian of the Gaussian (LoG).

C. Nikou – Digital Image Processing

25 The LoG operator (cont.)

  G( x, y)* f ( x, y)    G( x, y)* f ( x, y)
2 2

• The LoG operator is given by:

  2
 
2
 G ( x, y )   2  2  G ( x, y ) 
2

 x y 
x2  y 2 x2  y 2
    
2 2
 x  y  2  
2 2 2

 2  2 e 
2 2
2 2
e
 x y    4


C. Nikou – Digital Image Processing

26 The LoG operator (cont.)
x2  y 2
 G  G  x  y  2 
2 2 2 2 2 

The zero crossings are at:  2G ( x, y )    e

2 2
x 2 y 2  4 

x  y  2
2 2 2

C. Nikou – Digital Image Processing

27 The LoG operator (cont.)

• Fundamental ideas
– The Gaussian blurs the image. Iτ reduces the
intensity of structures at scales much smaller than σ.
– The Laplacian is isotropic and no other directional
mask is needed.
• The zero crossings of the operator indicate edge
pixels. They may be computed by using a 3x3
window around a pixel and detect if two of its
opposite neighbors have different signs (and
their difference is significant compared to a
threshold).
C. Nikou – Digital Image Processing
28 The LoG operator (cont.)

Image LoG

Zero crossings
Zero with a threshold
crossings of 4% of the
image max

C. Nikou – Digital Image Processing

29 The LoG operator (cont.)

• Filter the image at various scales and keep the zero

crossings that are common to all responses.
• Marr and Hildreth [1980] showed that LoG may be
approximated by a difference of Gaussinas (DOG):
x2  y 2 x2  y 2
 
1 212 1 2 22
DoG( x, y)  e  e , 1   2
2 2
1 2 2
2

• Certain channels of the human visual system are

selective with respect to orientation and frequency
and can be modeled by a DoG with a ratio of
standard deviations of 1.75.
C. Nikou – Digital Image Processing
30 The LoG operator (cont.)

• Meaningful comparison between LoG and

DoG may be obtained after selecting the
value of σ for LoG so that LoG has the same
zero crossings as DoG:

  2
 1 
2
2
  2
2
ln  2 
1 2
1   2   2 
2

• The two functions should also be scaled to

have the same value at the origin.
C. Nikou – Digital Image Processing
31 The LoG operator (cont.)

• LoG has fallen to some disfavour.

• It is not oriented and its response averages the
responses across the two directions.
– Poor response at corners.
– Difficulty in recording the topology of T-junctions
(trihedral vertices).
• Several studies showed that image components
along an edge contribute to the response of LoG to
noise but not to a true edge. Thus, zero crossings
may not lie exactly on an edge.

C. Nikou – Digital Image Processing

32 The LoG operator (cont.)
Poor corner and trihedral vertices detection

C. Nikou – Digital Image Processing

33 Designing an optimal edge detector
• Criteria for an “optimal” edge detector [Canny 1986]:
– Good detection: the optimal detector must minimize the probability of
false positives (detecting spurious edges caused by noise), as well as
that of false negatives (missing real edges).
– Good localization: the edges detected must be as close as possible to
the true edges
– Single response: the detector must return one point only for each true
edge point; that is, minimize the number of local maxima around the true
edge.

C. Nikou – Digital Image Processing

34 Canny edge detector
• Probably the most widely used edge detector.
• Theoretical model: step-edges corrupted by
additive Gaussian noise.
• Canny has shown that the first derivative of the
Gaussian closely approximates the operator that
optimizes the product of signal-to-noise ratio and
localization.
x2 x2
d  2 2 x  2 2
e  2 e
x 

C. Nikou – Digital Image Processing

35 Canny edge detector
• Generalization to 2D by applying the 1D operator in
the direction of the edge normal.
• However, the direction of the edge normal is
unknown beforehand and the 1D filter should be
applied in all possible directions.
• This task may be approximated by smoothing the
image with a circular 2D Gaussian, computing the
gradient magnitude and use the gradient angle to
estimate the edge strength.

C. Nikou – Digital Image Processing

36 Canny edge detector (cont.)
1. Filter image with derivative of Gaussian.
2. Find magnitude and orientation of
gradient.
3. Non-maximum suppression:
– Thin multi-pixel wide “ridges” down to single
pixel width.
4. Linking and thresholding (hysteresis):
– Define two thresholds: low and high
– Use the high threshold to start edge curves
and the low threshold to continue them.
C. Nikou – Digital Image Processing
37 Canny edge detector (cont.)

original image
C. Nikou – Digital Image Processing
38 Canny edge detector (cont.)

Gradient magnitude
C. Nikou – Digital Image Processing
39 Canny edge detector (cont.)

Non-maximum suppression and hysteresis thresholding

C. Nikou – Digital Image Processing
 40 Non-maximum suppression (cont.)
At pixel q, we have a
maximum if the value of
the gradient magnitude
is larger than the values
at both p and at r.

Interpolation provides
these values.

Source: D. Forsyth C. Nikou – Digital Image Processing

 41 Predict the next edge point
Assume the marked
point is an edge point.
Then we construct the
tangent to the edge
curve (which is normal
to the gradient at that
point) and use this to
predict the next points
(here either r or s).

Source: D. Forsyth C. Nikou – Digital Image Processing

42 Hysteresis Thresholding
Reduces false edge pixels. It uses a low (TL) and a high
threshold (TH) to create two additional images from the
gradient magnitude image g (x,y):
 g ( x, y ) g ( x, y )  TL  g ( x, y ) g ( x, y )  TH
g L ( x, y )   , g H ( x, y )  
 0 otherwise  0 otherwise

gL(x,y) has more non zero pixels than gH(x,y).

We eliminate from gL(x,y) all the common non zero pixels:
g L ( x, y)  g L ( x, y)  g H ( x, y)
gL(x,y) and gH(x,y) may be viewed as weak and strong edge
pixels.
Canny suggested a ratio of 2:1 to 3:1 between the thresholds.
C. Nikou – Digital Image Processing Source: S. Seitz
43 Hysteresis Thresholding (cont.)
• After the thresholdings, all strong pixels are
assumed to be valid edge pixels. Depending
on the value of TH, the edges in gH(x,y)
typically have gaps.
• All pixels in gL(x,y) are considered valid edge
pixels if they are 8-connected to a valid edge
pixel in gH(x,y).

C. Nikou – Digital Image Processing Source: S. Seitz

44 Hysteresis thresholding (cont.)

high threshold low threshold hysteresis threshold

(strong edges) (weak edges)
C. Nikou – Digital Image Processing Source: L. Fei-Fei
45 Canny vs LoG
Image Thresholded gradient

Both edges of the

concrete band lining the
bricks were preserved.

LoG Canny
C. Nikou – Digital Image Processing
46 Canny vs LoG
Image Thresholded gradient

The posterior boundary of

the brain was preserved.

LoG Canny
C. Nikou – Digital Image Processing
47 Edge Linking
• Even after hysteresis thresholding, the detected
pixels do not completely characterize edges
completely due to occlusions, non-uniform
illumination and noise. Edge linking may be:
– Local: requiring knowledge of edge points in
a small neighborhood.
– Regional: requiring knowledge of edge
points on the boundary of a region.
– Global: the Hough transform, involving the
entire edge image.

C. Nikou – Digital Image Processing

48 Edge Linking by Local Processing
A simple algorithm:
1. Compute the gradient magnitude and angle
arrays M(x,y) and a(x,y) of the input image f(x,y).
2. Let Sxy denote the neighborhood of pixel (x,y).
3. A pixel (s,t) in Sxy in is linked to (x,y) if:
M ( x, y)  M (s, t )  E and a( x, y)  a(s, t )  A

A record of linked points must be kept as the

center of Sxy moves.
Computationally expensive as all neighbors of
every pixel should be examined.
C. Nikou – Digital Image Processing
49
Edge Linking by Local Processing
(cont.)
A faster algorithm:
1. Compute the gradient magnitude and angle arrays
M(x,y) and a(x,y) of the input image f(x,y).
2. Form a binary image:

1 M ( x, y)  TM and a( x, y )  [ A  TA , A  TA ]
g ( x, y )  
0 otherwise

3. Scan the rows of g(x,y) (for Α=0) and fill (set to 1) all
gaps (zeros) that do not exceed a specified length K.
4. To detect gaps in any other direction Α=θ, rotate g(x,y)
by θ and apply the horizontal scanning.

C. Nikou – Digital Image Processing

50
Edge Linking by Local Processing
(cont.)
Image Gradient magnitude Horizontal linking

We may detect the

licence plate from the
ratio width/length (2:1 in
the USA)

Vertical linking Logical OR Morphological thinning

C. Nikou – Digital Image Processing

51
Edge Linking by Regional
Processing
• Often, the location of regions of interest is known
and pixel membership to regions is available.
• Approximation of the region boundary by fitting a
polygon. Polygons are attractive because:
– They capture the essential shape.
– They keep the representation simple.
• Requirements
– Two starting points must be specified (e.g.
rightmost and leftmost points).
– The points must be ordered (e.g. clockwise).

C. Nikou – Digital Image Processing

52
Edge Linking by Regional
Processing (cont.)
• Variations of the algorithm handle both open and
closed curves.
• If this is not provided, it may be determined by
distance criteria:
– Uniform separation between points indicate a closed
curve.
– A relatively large distance between consecutive
points with respect to the distances between other
points indicate an open curve.
• We present here the basic mechanism for
polygon fitting.

C. Nikou – Digital Image Processing

 53
Edge Linking by Regional
Processing (cont.)
– Given the end points A and
B, compute the straight line
AB.
– Compute the perpendicular
distance from all other
points to this line.
– If this distance exceeds a
threshold, the
corresponding point C
having the maximum
distance from AB is
declared a vertex.
– Compute lines AC and CB
and continue. C. Nikou – Digital Image Processing
54
Edge Linking by Regional
Processing (cont.)
Regional processing for edge linking is used in combination
with other methods in a chain of processing.

C. Nikou – Digital Image Processing

55 Hough transform
• An early type of voting scheme.
• Each line is defined by two points (xi,yi) and (xj,yj).
• Each point (xi,yi) has a line parameter space (a,b)
because it belongs to an infinite number of lines yi=axi+b.
• All the points (x,y) on a line y=a’x+b’ have lines in
parameter space that intersect at (a’x+b’).

Image space Parameter

space

C. Nikou – Digital Image Processing

56 Hough transform (cont.)
• Problems with the (a,b) parameter space:
– Unbounded parameter domain
– Vertical lines require infinite a.
• Polar (normal) representation of a line:
x cos  y sin   

Accumulator
array

C. Nikou – Digital Image Processing

57 Hough transform (cont.)

• A: intersection of curves corresponding to points 1, 3, 5.

• B: intersection of curves corresponding to points 2, 3, 4.
• Q, R and S show the reflective adjacency at the edges of
the parameter space. They do not correspond to points.

C. Nikou – Digital Image Processing

58 Hough transform (cont.)

• We only know the orientation of the runway (around 0 deg) and the
observer’s position relative to it (GPS, flight charts etc.).
• We look for the peak at the accumulator array around 0 deg and join
gaps below 20% of the image height.
• Applications in autonomous navigation.

C. Nikou – Digital Image Processing

59 Practical details
• Try to get rid of irrelevant features
– Take only edge points with significant gradient
magnitude.
• Choose a good grid / discretization
– Too coarse: large votes obtained when too many
different lines correspond to a single bucket.
– Too fine: miss lines because some points that are not
exactly collinear cast votes for different buckets.
• Increment also neighboring bins (smoothing in
accumulator array).

C. Nikou – Digital Image Processing

60 Thresholding
• Image partitioning into regions directly
from their intensity values.
0 if f ( x, y )  T1
1 if f ( x, y )  T 
g ( x, y )   g ( x, y )  1 if T1  f ( x, y )  T2
0 if f ( x, y )  T 2 if f ( x, y )  T
 2

C. Nikou – Digital Image Processing

61 Noise in Thresholding
Difficulty in determining the threshold due to noise

Noiseless Gaussian (μ=0, σ=10) Gaussian (μ=0, σ=10)

C. Nikou – Digital Image Processing
62 Illumination in Thresholding
Difficulty in determining the threshold due to non-uniform illumination

(a) Noisy image (b) Intensity ramp Multiplication

image
C. Nikou – Digital Image Processing
63 Basic Global Thresholding
• Algorithm
– Select initial threshold estimate T.
– Segment the image using T
• Region G1 (values > T) and region G2 (values < T).
– Compute the average intensities m1 and m2 of
regions G1 and G2 respectively.
– Set T=(m1+m2)/2
– Repeat until the change of T in successive
iterations is less than ΔΤ.

C. Nikou – Digital Image Processing

64 Basic Global Thresholding (cont.)

T=125

C. Nikou – Digital Image Processing

65
Optimum Global Thresholding
using Otsu’s Method
• The method is based on statistical decision theory.
• Minimization of the average error of assignment of pixels
to two (or more) classes.
• Bayes decision rule may have a nice closed form solution
to this problem provided
– The pdf of each class.
– The probability of class occurence.
• Pdf estimation is not trivial and assumptions are made
(Gaussian pdfs).
• Otsu (1979) proposed an attractive alternative maximizing
the between-class variance.
– Only the histogram of the image is used.

C. Nikou – Digital Image Processing

66 Otsu’s Method (cont.)
• Let {0, 1, 2,…, L-1} denote the intensities of a
MxN image and ni the number of pixels with
intensity i.
• The normalized histogram has components:
L 1
ni
pi 
MN
, p
i 0
i  1, pi  0

• Suppose we choose a threshold k to segment the

image into two classes:
• C1 with intensities in the range [0, k],
• C2 with intensities in the range [k+1, L-1].
C. Nikou – Digital Image Processing
67 Otsu’s Method (cont.)
• The probabilities of classes C1 and C2:
k L 1
P1 (k )   pi , P2 (k )  p i  1  P1 (k )
i 0 i  k 1

• The mean intensity of class C1 :

k k
P(C1 | i) P(i) 1 k
m1 (k )   iP(i | C1 )   i   ipi
i 0 i 0 P(C1 ) P1 (k ) i 0

Intensity i belongs to class C1 and P(C1 | i)  1

C. Nikou – Digital Image Processing

68 Otsu’s Method (cont.)
• Similarly, the mean intensity of class C2:
L 1 L 1
P(C2 | i) P(i) 1 L 1
m2 (k )   iP(i | C2 )   i   ipi
i  k 1 i  k 1 P(C2 ) P2 (k ) i k 1

• Mean image intensity:

L 1
mG   ipi  P1 (k )m1 (k )  P2 (k )m2 (k )
i 0

• Cumulative mean image intensity (up to k):

k
m(k )   ipi
i 0
C. Nikou – Digital Image Processing
69 Otsu’s Method (cont.)
• Between class variance:
 B (k )  P1 (k )[m1 (k )  mG ]  P2 (k )[m2 (k )  mG ]
2 2 2

• With some manipulation we get:

[ m P ( k )  m ( k )]2
 B (k )  G 1
2

P1 (k )[1  P1 (k )]
• The value of k is selected by sequential
search as the one maximazing:
k  max { (k )}
* 2
B
0 k  L 1

C. Nikou – Digital Image Processing

70 Otsu’s Method (cont.)
Example (no obvious valley in the histogram)

Image Histogram

Basic method Otsu’s method

C. Nikou – Digital Image Processing

71 Otsu’s Method (cont.)
Smoothing helps thresholding (may create histogram valley)

Noisy image Histogram Otsu no smoothing

Noisy image smoothed Histogram Otsu with smoothing

C. Nikou – Digital Image Processing
72 Otsu’s Method (cont.)
Otsu’s method, even with smoothing cannot extract small structures

Noisy image, σ=10 Histogram Otsu no smoothing

Noisy image smoothed Histogram Otsu with smoothing

C. Nikou – Digital Image Processing
73 Otsu’s Method (cont.)
• Dealing with small structures
– The histogram is unbalanced
• The background dominates
• No valleys indicating the small structure.
– Idea: consider only the pixels around edges
• Both structure of interest and background are present equally.
• More balanced histograms
– Use gradient magnitude or zero crossings of
the Laplacian.
• Threshold it at a percentage of the maximum value.
• Use it as a mask on the original image.
• Only pixels around edges will be employed.
C. Nikou – Digital Image Processing
74 Otsu’s Method (cont.)
Gradient magnitude
Noisy image, σ=10 Histogram mask (at 99.7% of max)

Multiplication Histogram Otsu

(Mask x Image)
C. Nikou – Digital Image Processing
75 Otsu’s Method (cont.)
Image Histogram Otsu on original histogram

We wish to extract
the bright spots
(nuclei) of the cells

Thresholded Histogram Otsu

absolute Laplacian
C. Nikou – Digital Image Processing
76 Otsu’s Method (cont.)
• The method may be extended to multiple thresholds
• In practice for more than 2 thresholds (3 segments)
more advanced methods are employed.
• For three classes, the between-class variance is:
 B2 (k1 , k2 )  P1 (k1 )[m1 (k1 )  mG ]  P2 (k1 , k2 )[m2 (k1, k2 )  mG ]  P3 (k2 )[m3 (k2 )  mG ]

• The thresholds are computed by searching all

pairs for values:

(k1* , k2* )  max { B2 (k1 , k2 )}

0 k1  k2  L 1

C. Nikou – Digital Image Processing

77 Otsu’s Method (cont.)
Image of iceberg segmented into three regions.

k1=80, k2=177

C. Nikou – Digital Image Processing

78 Variable Thresholding
Image partitioning.
The image is sub-divided and the method is applied to every sub-image.
Useful for illumination non-uniformity correction.
Shaded image Histogram Simple thresholding

Otsu Subdivision Otsu at each sub-image

C. Nikou – Digital Image Processing
79 Variable Thresholding(cont.)
Use of local image properties.
• Compute a threshold for every single pixel in the image
based on its neighborhood (mxy, σxy,…).

1 Q(local properties)is true

g ( x, y )  
0 Q(local properties)is false

 true f ( x, y)  a xy AND f ( x, y)  bmxy

Q( xy , mxy )  
false otherwise

C. Nikou – Digital Image Processing

 80 Variable Thresholding (cont.)

Image Otsu with

2 thresholds
Histogram

Image of local Local

standard deviations Thresholding

More accurate
nuclei extraction.

Subdivision
C. Nikou – Digital Image Processing
81 Variable Thresholding (cont.)
Moving averages.
• Generally used along lines, columns or in zigzag .
• Useful in document image processing.
• Let zk+1 be the intensity of a pixel in the scanning sequence
at step k+1. The moving average (mean intensity) at this
point is:
1 k 1 1
m(k  1)  
n i  k  2 n
zi  m(k )  ( zk 1  zk n )
n
n is the number of points used in the average
• Segmentation is then performed at each point comparing
the pixel value to a fraction of the moving average.

C. Nikou – Digital Image Processing

82 Variable Thresholding (cont.)

Shaded text images Otsu Moving averages

occur typically from
photographic flashes

C. Nikou – Digital Image Processing

83 Variable Thresholding (cont.)

Sinusoidal variations (the Otsu Moving averages

power supply of the scanner
not grounded properly)

The moving average works well when the structure of interest is

small with respect to image size (handwritten text).
C. Nikou – Digital Image Processing
84 Region Growing
• Start with seed points S(x,y) and grow to larger
regions satisfying a predicate.
• Needs a stopping rule.
• Algorithm
– Find all connected components in S(x,y) and erode
them to 1 pixel.
– Form image fq(x,y)=1 if f (x,y) satisfies the predicate.
– Form image g(x,y)=1 for all pixels in fq(x,y) that are 8-
connected with to any seed point in S(x,y).
– Label each connected component in g(x,y) with a
different label.

C. Nikou – Digital Image Processing

85 Region Growing (cont.)

X-Ray image of weld Histogram Seed image (99.9% of

with a crack we want to max value in the initial
segment. image).
Crack pixels are missing.

The weld is very bright. The predicate used for region growing is to
compare the absolute difference between a seed point and a pixel
to a threshold. If the difference is below it we accept the pixel as
crack. C. Nikou – Digital Image Processing
86 Region Growing (cont.)

Seed image eroded to 1 Difference between Histogram of the

pixel regions. the original image and difference image.
the initial seed image. Two valleys at 68
and 126 provided
The pixels are ready by Otsu.
to be compared to the
threshold.

C. Nikou – Digital Image Processing

87 Region Growing (cont.)

Otsu thresholding of the Thresholding of the Segmentation result

difference image to 3 difference image with by region growing.
regions (2 thresholds). the lowest of the dual
thresholds. The background is
not considered as
Notice that the crack.
background is also It is removed as it is
considered as crack. not 8-connected to
the seed pixels.
C. Nikou – Digital Image Processing
88 Region Splitting and Merging
• Based on quadtrees (quadimages).
• The root of the tree corresponds to the image.
• Split the image to sub-images that do not satisfy
a predicate Q.
• If only splitting was used, the final partition would
contain adjacent regions with identical
properties.
• A merging step follows merging regions
satisfying the predicate Q.

C. Nikou – Digital Image Processing

89
Region Splitting and Merging
(cont.)
• Algorithm
– Split into four disjoint quadrants any region Ri for
which Q(Ri)=FALSE.
– When no further splitting is possible, merge any
adjacent regions Ri and Rk for which Q(Ri Rk)=TRUE.
– Stop when no further merging is possible.
• A maximum quadregion size is specified beyond which no
further splitting is carried out.
• Many variations heve been proposed.
– Merge any adjacent regions if each one satisfies the
predicate individually (even if their union does not
satisfy it).

C. Nikou – Digital Image Processing

90
Region Splitting and Merging
(cont.)
Quadregions resulted from splitting.

Merging examples:
• R2 may be merged with R41.
• R41 may be merged with R42.
C. Nikou – Digital Image Processing
91
Region Splitting and Merging
(cont.)
• Image of the Cygnus Loop. We want to
segment the outer ring of less dense
matter.
• Characteristics of the region of interest:
• Standard deviation grater than the
background (which is near zero) and
the central region (which is
smoother).
• Mean value greater than the mean of
background and less than the mean
of the central region.
 true    AND 0  m  b
• Predicate: Q
false otherwise
C. Nikou – Digital Image Processing
92
Region Splitting and Merging
(cont.)
Varying the size of the smallest allowed quadregion.

Larger quadregions
lead to block-like
segmentation.

Smaller quadregions
lead to small black
regions. 32x32

16x16 seems to be
the best result.

16x16 8x8
C. Nikou – Digital Image Processing
93 Morphological Watersheds
• Visualize an image topographically in 3D
– The two spatial coordinates and the intensity (relief
representation).
• Three types of points
– Points belonging to a regional minimum.
– Points ta which a drop of water would fall certainly to
a regional minimum (catchment basin).
– Points at which the water would be equally likely to
fall to more than one regional minimum (crest lines
or watershed lines).
• Objective: find the watershed lines.

C. Nikou – Digital Image Processing

94 Morphological Watersheds (cont.)

Image • Topographic representation.

• The height is proportional to
the image intensity.
• Backsides of structures are
shaded for better visualization.
C. Nikou – Digital Image Processing
95 Morphological Watersheds (cont.)

• A hole is punched in each regional minimum and the topography is

flooded by water from below through the holes.
• When the rising water is about to merge in catchment basins, a dam is
built to prevent merging.
• There will be a stage where only the tops of the dams will be visible.
• These continuous and connected boundaries are the result of the
segmentation.
C. Nikou – Digital Image Processing
96 Morphological Watersheds (cont.)
Regional
minima

• Topographic representation of the image.

• A hole is punched in each regional minimum (dark
areas) and the topography is flooded by water (at
equal rate) from below through the holes.
C. Nikou – Digital Image Processing
97 Morphological Watersheds (cont.)

• Before flooding.
• To prevent water from spilling through the image
borders, we consider that the image is surrounded
by dams of height greater than the maximum image
intensity. C. Nikou – Digital Image Processing
98 Morphological Watersheds (cont.)

• First stage of flooding.

• The water covered areas corresponding to the dark
background.

C. Nikou – Digital Image Processing

99 Morphological Watersheds (cont.)

• Next stages of flooding.

• The water has risen into the other catchment basin.

C. Nikou – Digital Image Processing

100 Morphological Watersheds (cont.)

• Further flooding. The water has risen into the third

catchment basin.

C. Nikou – Digital Image Processing

101 Morphological Watersheds (cont.)

Short dam

• Further flooding.
• The water from the left basin overflowed into the
right basin.
• A short dam is constructed to prevent water from
merging. C. Nikou – Digital Image Processing
102 Morphological Watersheds (cont.)

Longer dam New dams

• Further flooding.
• The effect is more pronounced.
• The first dam is now longer.
• New dams are created.
C. Nikou – Digital Image Processing
103 Morphological Watersheds (cont.)
Final watershed lines
superimposed on the
image.

• The process continues until the maximum level of flooding is

reached.
• The final dams correspond to the watershed lines which is
the result of the segmentation.
• Important: continuous segment boundaries.
C. Nikou – Digital Image Processing
104 Morphological Watersheds (cont.)
• Dams are constructed by morphological dilation.
Cn-1(M1) Cn-1(M2)

Flooding step n-1.

Regional minima: M1 and M2.

Catchment basins associated: Cn-1(M1) and Cn-1(M2).
C[n  1]  Cn1 (M1 )  Cn1 (M 2 )
C[n-1] has two connected components.
C. Nikou – Digital Image Processing
105 Morphological Watersheds (cont.)

q
Cn-1(M1) Cn-1(M2)

Flooding step n-1. Flooding step n.

• If we continue flooding, then we will have one connected

component.
• This indicates that a dam must be constructed.
• Let q be the merged connected component if we perform
flooding a step n.
C. Nikou – Digital Image Processing
106 Morphological Watersheds (cont.)
q

• Each of the connected components is dilated by

the SE shown, subject to:
1. The center of the SE has to be contained in q.
2. The dilation cannot be performed on points that
would cause the sets being dilated to merge.
C. Nikou – Digital Image Processing
107 Morphological Watersheds (cont.)
Conditions
1. Center of SE in q.
2. No dilation if merging.

• In the first dilation, condition 1 was satisfied by every

point and condition 2 did not apply to any point.
• In the second dilation, several points failed condition 1
while meeting condition 2 (the points in the perimeter
which is broken).

C. Nikou – Digital Image Processing

108 Morphological Watersheds (cont.)
Conditions
1. Center of SE in q.
2. No dilation if merging.

• The only points in q that satisfied both conditions form

the 1-pixel thick path.
• This is the dam at step n of the flooding process.
• The points should satisfy both conditions.
C. Nikou – Digital Image Processing
109 Morphological Watersheds (cont.)
Image Gradient magnitude
• A common application
is the extraction of
nearly uniform, blob-
like objects from their
background.
• For this reason it is
generally applied to the
gradient of the image
and the catchment
basins correspond to
the blob like objects.

Watersheds Watersheds
on the image
C. Nikou – Digital Image Processing
110 Morphological Watersheds (cont.)

• Noise and local minima lead generally to oversegmentation.

• The result is not useful.
• Solution: limit the number of allowable regions by additional
knowledge.
C. Nikou – Digital Image Processing
111 Morphological Watersheds (cont.)
• Markers (connected components):
– internal, associated with the objects
– external, associated with the background.
• Here the problem is the large number of local
minima.
• Smoothing may eliminate them.
• Define an internal marker (after smoothing):
• Region surrounded by points of higher
altitude.
– They form connected components.
– All points in the connected component have the
same intensity.

C. Nikou – Digital Image Processing

112 Morphological Watersheds (cont.)

• After smoothing, the internal markers are shown in light gray.

• The watershed algorithm is applied and the internal markers
are the only allowable regional minima.
• The resulting watersheds are the external markers (shown in
white).
C. Nikou – Digital Image Processing
113 Morphological Watersheds (cont.)

• Each region defined by the external marker has a single internal marker
and part of the background.
• The problem is to segment each of these regions into two segments: a
single object and background.
• The algorithms we saw in this lecture may be used (including watersheds
applied to each individual region).
C. Nikou – Digital Image Processing
114 Morphological Watersheds (cont.)

Final segmentation.
C. Nikou – Digital Image Processing
115 Morphological Watersheds (cont.)

Image Watersheds Watersheds with markers

C. Nikou – Digital Image Processing

116 The use of motion in segmentation
• Powerful cue used by humans to extract
objects and regions.
• Motion arises from
– Objects moving in the scene.
– Relative displacement between the sensing
system and the scene (e.g. robotic
applications, autonomous navigation).
• We will consider motion
– Spatially.
– In the frequency domain.
C. Nikou – Digital Image Processing
117 Basic spatial motion segmentation
• Difference image and comparison with
respect to a threshold:

1 if f ( x, y, ti )  f ( x, y, t j )  T
dij ( x, y )  

0 otherwise

• The images should be registered.

• Illumination should be relatively constant
within the bounds defined by T.

C. Nikou – Digital Image Processing

118 Accumulative differences
• Comparison of a reference image with every
subsequent image in the sequence.
• A counter is incremented every time a pixel in
the current image is different from the reference
image.
• When the kth frame is being examined, the entry
in a given pixel of the accumulative difference
image (ADI) gives the number of times this pixel
differs from its counterpart in the reference
image.

C. Nikou – Digital Image Processing

119 Accumulative differences (cont.)
• Absolute ADI:
 Ak 1 ( x, y )  1 if R( x, y)  f ( x, y, tk )  T
Ak ( x, y )  
 Ak 1 ( x, y ) otherwise

• Positive ADI:
 Pk 1 ( x, y )  1 if R( x, y)  f ( x, y, tk )  T
Pk ( x, y )  
 Pk 1 ( x, y ) otherwise

• Negative ADI:
 N k 1 ( x, y )  1 if R( x, y)  f ( x, y, tk )  T
N k ( x, y )  
 N k 1 ( x, y ) otherwise

C. Nikou – Digital Image Processing

120 Accumulative differences (cont.)
• ADIs for a rectangular object moving to southeast.

Absolute Positive Negative

• The nonzero area of the positive ADI gives the size of the object.
• The location of the positive ADI gives the location of the object in the
reference frame.
• The direction and speed may be obtained fom the absolute and
negative ADIs.
• The absolute ADI contains both the positive and negative ADIs.
C. Nikou – Digital Image Processing
121 Accumulative differences (cont.)
• To establish a reference image in a non
stationary background.
– Consider the first image as the reference image.
– When a non stationary component has moved out of
its position in the reference frame, the corresponding
background in the current frame may be duplicated
in the reference frame. This is determined by the
positive ADI:
• When the moving object is displaced completely with
respect to the reference frame the positive ADI stops
increasing.

C. Nikou – Digital Image Processing

122 Accumulative differences (cont.)
• Subtraction of the car going from left to right to
establish a reference image.

• Repeating the task for all moving objects may

result in a static reference image.
• The method works well only in simple scenarios.

C. Nikou – Digital Image Processing

123 Frequency domain techniques
• Consider a sequence f (x,y,t), t=0,1,…,K-1 of size M x N.
• All frames have an homogeneous zero background
except of a single pixel object with intensity of 1 moving
with constant velocity.
• At time t=0, the object is at (x’, y’) and the image plane is
projected onto the vertical (x) axis. This results in a 1D
signal which is zero except at x’.
• If we multiply the 1D signal by exp[j2πα1xΔt], for
x=0,1,…,M-1 and sum the results we obtain the single
term exp[j2πα1x’Δt].
• In frame t=1, suppose that the object moved to (x’+1, y’),
that is, it has moved 1 pixel parallel to the x-axis. The
same procedure yields exp[j2πα1(x’+1)+Δt].
C. Nikou – Digital Image Processing
124
Frequency domain techniques
(cont.)
• Applying Euler’s formula, for t=0,1,…,K-1:
j 21 ( xt ) t
e  cos[21 ( x  t )t ]  j sin[21 ( x  t )t ]

• This is a complex exponential with frequency α1.

• If the object were moving V1 pixels (in the x-
direction) between frames the frequency would
have been V1α1.
• This causes the DFT of the 1D signal to have wo
peaks, one at V1α1 and one at K-V1α1.
• Division by a1 yields the velocity V1.
• These concepts may be generalized as follows.
C. Nikou – Digital Image Processing
125
Frequency domain techniques
(cont.)
• For a sequence of K images of size M x N, the
sum of the weighted projections onto the x-axis
at an integer instant of time is:
M 1 N 1
g x (t , 1 )    f ( x, y, t )e j 21xt , t  0,1,..., K  1
x 0 y 0

• Similarly, the sum of the weighted projections

onto the y-axis at an integer instant of time is:
M 1 N 1
g y (t ,  2 )    f ( x, y, t )e j 22 yt , t  0,1,..., K  1
x 0 y 0

C. Nikou – Digital Image Processing

126
Frequency domain techniques
(cont.)
• The 1D DFT of the above signals are:
K 1
Gx (u1 , 1 )   g x (t , 1 )e j 2 u1t / K , u1  0,1,..., K  1
t 0
K 1
Gy (u2 ,  2 )   g y (t ,  2 )e j 2 u2t / K
, u2  0,1,..., K  1
t 0

• The peaks in Gx(u1,α1) and Gy(u2,α2) provide the

velocities:
• u1=α1V1
• u2=α2V2

C. Nikou – Digital Image Processing

127
Frequency domain techniques
(cont.)
• The unit of velocity is in pixels per total frame
rate. For example, V1=10 is interpreted as a
motion of 10 pixels in K frames.
• For frames taken uniformly, the actual physical
speed depends on the frame rate and the
distance between pixels. Thus, for V1=10, K=30,
if the frame rate is two images/sec and the
distance between pixels is 0.5 m, the speed is:
– V1=(10 pixels)(0.5 m/pixel)(2 frames/sec) / (30 frames) =
1/3 m/sec.

C. Nikou – Digital Image Processing

128
Frequency domain techniques
(cont.)
• The sign of the x-component of the velocity is
obtained by using Fourier properties of sinusoids:

d 2 Re  g x (t , 1 ) d 2 Im  g x (t , 1 )
S1x  2
S2 x 
dt t n
dt 2 t n

• The above quantities will have the same sign at

an arbitrary time point t=n, if V1>0.
• Conversely, they will have opposite signs if V1<0.
• In practice, if either is zero, we consider the next
closest point.
C. Nikou – Digital Image Processing