Image Understanding II

● Today we look at methods that search for image

features with certain characteristics, e.g.
– a closed, smooth contour
– a hand
– a face
● That is, they segment an object from the image
● These methods all have in common
– an initial guess (sometimes any random initialization)
– an iterative process that modifies the initial guess
– a final, stable shape that the iterative process converges to
– this stable point is a (local) minimum of an energy functional
(implicit or explicitly defined energy functional)
 Image Understanding II
● Active Contour Models (= snakes, Deformable Contours)
– in 3D: Active/Deformable Surface Models
– adapts to boundaries in the image
● Active Shape Models (= Point Distribution Models)
– creates a model using various examples
– describes the shape and its variability
● Active Appearance Models
– ASM including grey values within the shape
● Level Set Method
– curve evolution applied to contour detection
– active contours that can split apart and merge together
● Book: sections 7.2, 7.3, 10.3 and 10.4
 Active Contour Models
● Kass, Witkin & Terzopoulos (1988)
● In 2D usually called snakes
● A flexible line or closed contour that evolves over time
to some minimal energy configuration
– hopefully matching object boundaries in the image!
● Used for:
– automatic segmentation of objects
– interactive delineation of objects
– tracking objects over time in video
 Snakes
● A flexible line:
T
v  s =  x  s , y s   s ∈[0,1]
● A closed contour:
v 0 = 
 v 1
 Snakes
● Modify the snake to find a local minimum in the energy
functional 1

E = ∫ E int  v sE ext  v sds

0

1
E int  v s  = { s∣v I s∣2  s∣v II s ∣2 }
2
“membrane” term (α and β often
“thin plate” term constant)
Gaussian smoothed input image

{
G⊗ I
2
v s  = −∣∇ I ∣
E ext    ∇ I =∇ G⊗ I 
∂
derivative of gradient direction
∂ n perpendicular to gradient
 Snakes
● Minimizing this:
1
1
E = ∫2 { ∣v
 I
 s∣2
∣v
 II
s∣2
}E ext  v sds
0

● through Euler-Lagrange equation yields this:

 v II  s− v IIII s−∇ E ext  
v s = 0

 int
F  ext
F
● We solve using gradient descent:
∂ v s , t  II IIII
=  v s , t − v s , t −∇ E ext  
v s ,t 
∂t
 Snakes, Discrete
● Discretizing the gradient gradient descent equation
∂ v s , t  II IIII
=  v s , t − v s ,t −∇ E ext  
  v s ,t 
∂t
Xt is a vector with x-values
∂ Xt
= A X t f x  X t ,Y t  of coordinates of points along
∂t the curve; Yt is the y-values

X t −X t −1 ∂
fx = E ext
= A X t f x  X t ,Y t  ∂x

step size A is a pentadiagonal banded
matrix (cyclic if curve is closed)
−1
X t =  I− A  { X t −1 f x  X t ,Y t  } (invert with Cholesky decomposition)

( we assume f x  X t ,Y t ≈f x X t −1 ,Y t −1  )
 Things to Consider
● Important parameters:
– α, β and γ
– Eext
– initial snake {X0,Y0}
– number of iterations
● The curve needs to be sampled densely to be able to
compute v  s accurately
IIII

● When the snake evolves, it is important to resample the
curve regularly
 Snakes in Action

input image - gradient magnitude evolving snake

Eext
 Snakes in Action

input image - gradient magnitude evolving snake

Eext
 Issues with Snakes
● Snakes cannot flow into elongated structures
● Snakes cannot move towards edges that are far away
– when increasing the size of the Gaussian for the gradient
magnitude, the edges of the object will move
(remember scale-space lecture)
● Snakes do not work well when the initial curve
intersects the edges you’re looking for

● Solution:
don’t use an external energy, but an external force!
When using E ext , the gradient descent equation had
an external force F ext=∇ E ext
 Improving the Snake
● Add a dynamic force:
(depends on the curve)
– the balloon force

● Change the static force:

(independent of the curve,
usually depends on the input image)
– using the distance transform
– Gradient Vector Flow
– Vector Field Convolution

– or anything else you can come up with...

 The Balloon Force
● Cohen (1989)
● Also known as pressure force
● Adds an outward, expanding force to the curve
(or an inward, contracting force)
● Requires the curve to be initialised inside the object
(or outside)
● Used in addition to the traditional static force
● Adds a new parameter to the method
II IIII
 v s− v s −∇ E ext  v s 
  n  s = 0

the unit normal vector

 The Balloon Force

input image - gradient magnitude evolving snake

Eext
The Balloon Force

high gradient

low gradient

high gradient

high intensity

high gradient

(play demo)
 Static Forces

 ext =−∇ E ext

F color = direction
saturation = strength
 The Distance Force
● Cohen & Cohen (1993)
● Force based on the distance transform of the edges
– the force fields pulls the snake towards the closest edge,
no matter how far away the snake starts

E = Canny I  D = DT E   ext = −∇ D
F
 The Distance Force

input image gradient of evolving snake

distance transform
 Gradient Vector Flow
● Xu & Prince (1998)
● More “refined” way of creating a force field across the
whole image
● Propagate gradient information
– stronger edges carry more weight
– force can pull snake into narrow structures
– normalize vectors
●  gvf =U gvf ,V gvf T that minimizes
Computed by finding F

∫  ∣∇ U gvf∣ ∣∇ V gvf∣ 

2 2 2
E =  ∣∇ E ext∣ ∣F
 gvf ∇ E ext∣ d x

smoothness term close to edges:

 gvf = −∇ E ext
F
Gradient Vector Flow
 Gradient Vector Flow

input image GVF force evolving snake

this is how far the forces were propagated,
keep iterating to fill the image!
 Vector Field Convolution
● Li & Acton (2007)
● Very similar to GVF, but much easier & faster to
compute:
– create a vector field kernel 
k
– convolve edge map with kernel F vfc = ∣∇ I∣⊗ 
k
– normalize vectors
−x −y
 −
k =  r   
n n = 
 , 
r r

γ ∈ [1,3] r =  x 2 y 2
γ = 2 ⇒ gravitation!

ε is some small
value to avoid
division by 0
 Vector Field Convolution

input image VFC force evolving snake

this is numerical inaccuracy
increase kernel size to avoid this!
 The External Forces

gradient distance GVF VFC

What do the improved external forces have in common?

 What About Noise?

gradient distance GVF VFC

 What About Noise?

gradient distance GVF VFC

 What About Initialization?

gradient distance GVF balloon

 Snakes in 3D
● Snakes are explicitly 2D constructs, but 3D
generalization is not impossible
● First “try” involved 2D snakes on successive slices of
the volume, initializing each with the final result of the
previous slice
● The better approach is with active surfaces:
– surface represented as a mesh
– internal forces (derivatives) calculated
using mesh neighbours
– all external force formulations
generalize easily to 3D
Snakes in 3D
 Active Shape Models
● A snake uses application-specific knowledge by:
– the initialization (the initial shape to be deformed)
– the choice of parameters (α, β, γ, κ, …)
– the choice of external force
● Once the snake starts to deform no further knowledge is
used (it just looks for a stable point)
● Sometimes we know in advance the shape of the object
we are looking for
● How can we incorporate this information into the snake
deformation algorithm?
Adding shape information allows detection of partially
occluded shapes, for example.
 Active Shape Models
● Cootes, Taylor, Cooper & Graham (1994)
● Like a snake, detects an object in an image
● Control the possible ways that the snake deforms
● Learn the possible shapes from examples
For example:
when looking for a hand, we know
that the fingers can have many
different positions with respect to the
hand, but we also know that there
are exactly 5 of them, that they all
have a fixed length, and that they
cannot bend at e.g. 90˚
(All the examples I’m showing here
are from the Cootes et al. paper)
 Active Shape Models
● The Active Shape Model (ASM) consists of a set of
points describing a boundary, and for each point, a
distance along the x and y axes it is allowed to move
● The ASM is trained by (manually) selecting points along
the object’s contour in a set of training images
– in each image, each point must match the same location
along the contour of the object
● It is required that the training images contain all
possible shape variations – no shape changes will be
allowed unless they are represented in the test set!
● The ASM is fitted to a new image much like a snake, but
instead of internal forces there is a constraint on the
relative position of the points representing the boundary
Example: Resistors
Example: Resistors
body position “shoulder” shape

wire curvature

These are the 3 most

important modes of
variation captured by
the model
 ASM Training
● The training set can be seen as M (2N)-dimensional
vectors: T M images
x = x 1 , y 1 , x 2 , y 2 , ... , x N , y N  2 dimensions
N points
● Aligning training set:
– rotate, scale & translate each shape to align with 1st shape
– calculate mean shape M
– 1
normalize orientation, scale and origin of mean x = ∑ x
i
– align all shapes to the mean M i=1

– recalculate mean & repeat until convergence

● After alignment, the vectors occupy a subset of ℝ2N
● Simplify the training set using PCA
Note: this divides up the differences between images into a
rigid transformation and a non-rigid deformation of the shape
 Principal Component Analysis
● Joakim will discuss PCA in detail
(here just the general idea)
● PCA extracts orthogonal vectors describing the most
important axes in the data
● Assumes normal (Gaussian) distribution of points!
λ1p1
d x i = x i − x
M
λ2p2 1 T
S = ∑
M i=1
d x
 i d x
 i

S pk =  k pk

eigenvalue
unit-length vector
S is 2Nx2N covariance matrix
 The Active Shape Model
● Any allowed shape is given by:
x = x P 
b P= p1 , p2 ,... , p K  K≤2 N
(matrix formed by first K eigenvectors)
weights to be fitted
−3  i ≤b i ≤3  i

● Any rotation, scaling and translation of an allowed

shape is also allowed
● Because of the assumption of normally distributed
points, certain shape variations cannot be modeled!
When the Model Fails
 Fitting an ASM to an Image
● Choose an initial placement
– origin (xo, yo), scale s & rotation θ
– use the mean shape, that is, all bi = 0

x = Ms ,   x P 
b T  x o , y o 
● Repeat until convergence:
– look for a better position for each of the points on the shape
– split the movement into rigid and non-rigid components
– transform non-rigid components into a shape change
– constrain shape change to allowed shapes
● “Convergence” can be defined in many ways...
 Fitting an ASM to an Image
● Step 1: Look, for each point, along normal of curve, for
a strong edge
– this yields an adjustment for each point d x j =  m n
∣∇ I∣

n m
x j

d x composed of all d x j so
that x d x is the new shape
 Fitting an ASM to an Image
● Step 2: Find a ds, dθ, dxo and dyo that best aligns
x to x d x
● Step 3: Find residual adjustments d u in local
coordinate frame:

x = Ms ,   x P 
b T  x o , y o 

x d x = M sd s , d   x P    T  x od x o , y o d y o 

bd u

du
 = Msd s −1 −1
, d  {M  x P b d x −T d x
s , o }
,d y o  −  x P 
b
 Fitting an ASM to an Image
● Step 4a: Map d 
u onto subdomain P:
d b = P T d 
u (note that P T =P−1 )
(this is a least-squares approximation!)

● Step 4b: Update shape parameters & limit 

b to allowed
range:

{
s  s s d s
   d −3   i ≤b i ≤3   i
xo  x o x d x o
o

yo  y o y d y o the weights ω make the

o movement slower – this avoids
bi  b i  b d b i
i
overshooting the target
 ASM Example

t=80

heart ventricle model

with 96 points, 12
degrees of freedom

t=0 t=200
 Active Appearance Models
● Just like ASM, but also includes information on grey
values inside of shape:
– 2N parameters describing boundary of shape
– PCA yields n shape vectors
– M parameters describing grey values
(i.e. the pixels after scaling, rotating and shifting the patch to the
mean shape)
– PCA yields m intensity vectors
– we now have n+m vectors, which we combine but weigh
differently, depending on relative importance
– perform PCA again on matrix of vectors, further simplifying
the model
● Joakim’s lecture on PCA will demonstrate this further
 Level Sets
● Developed for physics simulations, to model solid/liquid
interfaces that move at curvature-dependent speeds
Osher & Sethian (1988)
● Applied to images as a substitute for snakes
Caselles, Catté, Coll & Dibos (1993)
Malladi, Sethian & Vemuri (1995)
● Addresses problems with snakes:
– sampling of snake is problematic
– snake cannot split or merge
(if there’s two objects in the image, start with two snakes)
– snakes in higher dimensions are complex
● Simple to understand, a little harder to implement
 Level Sets
● Instead of defining a curve through a set of sample
points, we embed the curve in a higher-dimensional
space
– for example: instead of a 1D curve in 2D, we have a 2D
surface in 3D
● The curve is the set of points for which the surface
crosses the 0 level
z ψ
 x 
v s

 = { =0 } y
y γ

x
x
 Level Sets

Note:
● nothing special is required for the curve to split into two

● the function ψ is always well-behaved

● this is trivial to generalise to any number of dimensions

● the shape of the function away from the zero level set is not important

● the function is always positive inside the object, negative outside

 Level Sets
● The surface is modified according to a speed function

t  = { x , t=0 } ∂ F ∣∇ ∣ = 0

∂t
● The speed function contains the equivalent of the
internal and external forces of the snake

F = k  F A F G ∇
F G = ∇⋅
∣∇ ∣
geometry term,
curvature dependent
(= internal forces) 1
advection term, k =
1∣∇ G ⊗ I∣
constant speed
(= balloon force) (small at edges)
 Improving Level Sets
● If the image gradient is weak, the curve can pass it
● Once passed this point, it cannot go back
● Solution: add a term that pulls curve towards edges
– seems logical, considering what we learned with snakes!

∂t 
A
∣∇ ∣ 
∂   k F ∇⋅ ∇  ∣∇ ∣  ∇ k ∇  = 0

pulls towards edge

old speed function
 Implementing Level Sets
● The initial function ψ is generated from the initial closed
contour using a distance function,  x ,t=0 = ±d
where d is the distance from x to the contour
● The closed contour can always be recovered by looking
for the zero crossings of ψ
● The function k is an appropriate speed only on the
contour
– level sets other than the zero level set will move at different
speeds, which can create very large gradients in ψ
● Two solutions:
– 1: extend the function values of k for ψ=0
– 2: regularly reinitialise the function ψ from its zero crossings
 Narrow Band Implementation
● Level sets usually implemented in a narrow band
around ψ=0
– this saves a lot of computation
● When the curve comes too close to the band edge:
– reinitialise the function ψ
– this defines a new band

6 px
Example

(from Malladi et al., 1995)

 Summary
● Snake – Active Contour Model
– simple, versatile
– lots of parameters to tweak
– 3D extension not trivial but doable
– one object, one snake
● ASM – Active Surface Model
– a snake with knowledge
– trained with a set of examples
– robust against partial occlusions
● Level Set
– “different way of implementing a snake”
– nD extension trivial
– adapts to any number of contours