In this class

• Flexible shape models for image segmentation

• Examples:
H06W8a: Medical image analysis – Path finding
– Active contour models (‘snakes’)
Class 4: Deformable shape models – Level-set implementation
– Active shape models (‘ASM’)
– Active appearance models (‘AAM’)
Prof. Frederik Maes
[email protected]

Motivation Deformable models

• Medical image analysis involves complex objects and complex data • Model = parameterized computer representation of the appearance of the object
• Object segmentation is a pre-requisite for quantification of interest in the image based on prior knowledge
• Available prior knowledge is application dependent:
• Low-level segmentation strategies are too simplistic to deal with image ambiguity
– Geometry: position, shape, motion, deformation
• Incorporate prior knowledge about the object’s appearance in the image
– Photometry: intensity, contrast, texture
– Context: relations to other objects in the scene
• Modeling strategies:
– Heuristic: model imposes generic assumptions, e.g. boundary is smooth and
coincides with image edges
– Statistical: model is object specific and learned from the data itself
– Biomechanical: model mimics physical properties of the object tissue
• Models in medical image analysis should be able to cope with natural variability of
anatomical objects, hence need to be parameterized and adjustable (‘deformable’)
to the image at hand

Thresholding Edge detection Watershed

Deformable model fitting strategy Rigid model fitting

• General model fitting strategy: • Object has a fixed shape but unknown pose in the image (i.e. unknown position
• Define the model M(p) with parameters p (e.g. a 2D curve or 3D surface with points p) and orientation)
• Design an objective function C (‘cost function’) to measure “goodness of fit” • Model parameters p = pose parameters (in 2D: 2 translations + rotation)
between the image I and the model instance M(p) for any parameters p
• Find the optimal model M(p*) such that p* = arg min C(M(p),I) • General shapes: “template matching”
• For simple parametric shapes (lines, circles, …): “Hough transform”
• C typically will consist of 2 terms: C = ‘image data term’ + w . ‘model term’ • Applications of rigid models are limited in medical image analysis, due to
• First term: attracts the model to ‘deform’ towards suitable image features (that anatomical variability
are assigned low cost by the image data term)
• Second term: imposes suitable constraints on the model, by penalizing undesired
model ‘deformations’ (that are assigned high cost by the model term)
• w = weight parameter that balances both terms, to be set heuristically

• Requires suitable optimization method

• Typically an iterative procedure, converging to a local optimum
• Proper initialization needed to converge to the correct local optimum
 Template matching Hough transform

• Depict the shape as a template: • For simple parametric shapes (lines, circles, …)
è e.g. a graph, binary image (drawing), gray value image • Example: detecting lines in an image
• Cost function: • Line through (x,y): x cos q + y sin q = r
e.g. integrated gradient magnitude along shape boundary • ‘Cost’ function H(q,r) = accumulator matrix of contributions of all image
e.g. correlation of template intensity with image intensity points (x,y), after discretization of the parameter space (q,r):
• Find globally optimal pose p* by exhaustive search "xi,yi: add w(xi,yi) to H(qj,rj), "qj,rj = xi cos qj + yi sin qj
èafter suitable quantization of the search space – Lines in binary images: w(xj,yj) = 1 if I(xj,yj) = 1
èsearch space delimited based on prior knowledge – Linear edges in gray scale images: w(xj,yj) = |ÑI(xj,yj)|
• Local search strategies may be applicable as well, if properly initialized • Hence: all points (x,y) that are on the same line (qj,rj) contribute to the
è Especially for larger templates (larger attraction basin around local corresponding bin in H
optimum expected) • Lines (q,r) are detected as maxima in H(q,r)
è Cfr rigid-body / affine image registration

Hough transform: example Hough transform: example

Catheters in a 2D X-ray radiograph Gradient magnitude Edge map

Flexible shape representations Curve parametrizations

polygon spline Fourier Overlay
• Explicit geometry:
– Point set (xi,yi,zi), curve (x(t),y(t),z(t)), surface (x(u,v),y(u,v),z(u,v))
– Mesh: 1D: contour, 2D: triangulated surface, 3D: volumetric mesh
• Implicit geometry:
– Path of connected image pixels
– Level set: f(x,y,z) = 0
– Image ‘template’, ‘atlas’ = iconic representation of shape
polygon spline Fourier
• Discrete representations, e.g. point set, stack of curves
• Piece-wise continuous, e.g. polygon, triangulated mesh
• Continuous (& differentiable):
– Low-dimensional analytical forms , e.g. ‘superquadrics’
– Analytical forms using basis functions, e.g. splines, Fourier series, radial basis
functions (= interpolation of discrete point set)

• Local vs global parameterization LOCAL LOCAL GLOBAL

 Surface parameterizations Finite difference approximation

• For a continuous curve, tangent and normal direction and curvature can be
computed analytically and exactly: C (t ) = [x(t ) y(t )], t Î [0,1]
Tangent : [x' (t ) / L(t ) y ' (t ) / L(t )] , L(t ) = ( x' ) 2 + ( y ' ) 2
Normal : [ y ' (t ) / L(t ) - x' (t ) / L(t )]
x' (t ).y ' ' (t ) - x' ' (t ).y ' (t )
Curvature : k =
(( x' ) 2
+ ( y' )2 )
3/ 2

• For a discrete curve, numerical approximations have to be used to estimate these

quantities locally at each point of the curve
Tangent direction: e.g. (xi+1-xi-1,yi+1-yi-1)
Curvature: e.g. || (xi+1+xi-1,yi+1+yi-1)/2 - (xi,yi) ||

3D point set 2 (xi-1,yi-1)

Large curvature
triangulated mesh 1 (xi+1,yi+1)
(e.g. created by RBF interpolation ‘superquadrics’ (xi,yi)
and the marching cubes algorithm) (u,v) è (X(u,v),Y(u,v),Z(u,v)) Small curvature
(6 parameters)

Deformable model fitting strategy Graph search

• Consider a parametric curve/surface C(p) with parameters p • Object boundary is a connected path of pixels (= implicit geometry)
è for a discrete curve: p = the coordinates of the curve points
• Start pixel (S) and end pixel (E) on the boundary are given
• Define an objective function E (= ‘energy function’) that can be evaluated • Find optimal path P* between S and E, i.e. the path from S to E that ‘best’
for each curve C(p), containing 2 terms: fits the object boundary
• No global shape constraints imposed, only connectivity constraint
E(C(p))= Eext(C(p),I) + g . Eint(C(p))

Eext = external energy: drives C towards relevant features in image I, e.g. edges
Eint = internal energy: imposes shape constraints on C, e.g. smoothness S S
g = weight, tuning the relative importance of both terms
E E P*
• Find the curve C(p*) such that E(C(p*)) is minimal (= ‘energy minimization’)

• Specific implementation depends on the representation chosen for C(p)

Graph search Principle of ‘dynamic programming’

Cost image
• Construct a ‘cost image’ C, such that pixels with ‘low’ cost are more likely S
located on the object boundary than pixels with ‘high’ cost
• Typically: C = max(|ÑI|) – |ÑI| with |ÑI|= gradient magnitude (C ³ 0)
• Find boundary segment P* between S and E as the path with minimal total
cost over all paths connecting S and E
E

S
• Example: C(P(S,E)) = 5+6+5+9+4+2+2+3+1+1 = 38
• Problem: how to find the path P*(S,E) with minimal cost C*(S,E)?
E P* • Idea: "q Î P* : P*(S,E) = P*(S,q) È P*(q,E), C*(S,E) = C*(S,q) + C*(q,E)
• Hence: can ignore all ‘non-optimal’ paths P(S,q)
• Build solution incrementally by computing C*(S,q), "q in a region of interest
around pixel S, including pixel E
è ‘Accumulated cost’ matrix
Original image Gradient magnitude Cost image
 F* algorithm F* algorithm: initialization

Cost image (C) Accumulated cost (AC) Cost image (C) Accumulated cost (AC)
S

•Accumulated cost image AC = minimal cost from each pixel q to pixel S

•A is constructed using the principle of ‘dynamic programming’ • Initialization: all accumulated costs set to a ‘very high’ value
è F* algorithm (‘Dijkstra’s method’)

F* algorithm: forward pass F* algorithm: backward pass

Cost image (C) Accumulated cost (AC) Cost image (C) Accumulated cost (AC)

• Update AC in each pixel from the AC • Update AC in each pixel from the AC
values of its 8-connected neighbors and values of its 8-connected neighbors and
the cost C of this pixel, scanning the the cost C of this pixel, scanning the
image in forward direction image in backward direction

• Example: • Example:
27 = min(Inf, 27+2, 32+2, 37+2, 25+2) 30 = min(31, 34+2, 41+2, 37+2, 28+2)

F* algorithm: iterations F* algorithm: convergence

AC after run 2 AC after run 3
Accumulated cost (AC) Accumulated cost (AC)
after 1 F* iteration after 5 F* iterations

AC after run 4 AC after run 5

•AC has converged when no values change in the next update

•Convergence is guaranteed as C≥0: AC(0) = Inf, 0 ≤ AC(k+1) ≤ AC(k) , " iterations k

• Repeat updating AC (forward and backward passes) until convergence

 F* algorithm: minimal cost path Path finding: example

Cost image (C) Accumulated cost (AC) Cost image Accumulated cost image
S (computed in region of interest only)

• Minimal cost path from S to any pixel E can be extracted from AC as the path
of largest cost decrease (‘steepest descent’) starting from E E

• Example: minimal cost path from E to S has cost 31 S

Path finding: example Path finding: example

Interactive indication of boundary points, real-time path tracking Interactive indication of boundary points, real-time path tracking

Path finding: example Path finding: discussion

• Uses a cost function that indicates whether or not a pixel is likely to be on
the object boundary
• Minimal cost path between specified points is found using dynamic
programming (F*-algorithm)
• Yields connected curve segments (>< edge detection)
• No smoothness constraint, no shape model
èMay result in ‘irregular’, highly curved paths (see example)
• Several implementations: ‘Live wire’, ‘Intelligent scissors’
è Combination of cost functions, trained on the fly
 ‘Live wire’, ‘Intelligent scissors’ Graph search with shape constraints?

• Convert 2D search problem into a 1D problem by image resampling

• Allows to incorporate simple shape constraints (parallelism, local continuity)
through additional cost terms between consecutive points (xi-1,xi)
xi-1 xi xi+1 xi+2

Candidates
Resampled cost image
Mortensen & Barrett, Centerline Result
GMIP, 1998
Nuyts et al.
IEEE TMI, 1991

Active contour models (‘snakes’) Energy formulation

• Model P = curve X(x(s),y(s))

– explicit geometry, continuous parameterization assumed)
• Energy function:
• Internal energy S, depends only on the curve X itself:

èSmoothness constraint
èResistance against stretching (1st derivative) and bending (2nd derivative)
Mathews Jacob, EPFL èWeights a(s), b(s), often chosen as constants a, b

• External energy P, depends on both the curve X and the image data I

External energy Energy minimization

• Line integral along the curve of a potential function P: • Minimization of the functional E(X) is a variational problem
• Solution X should satisfy the Euler-Lagrange equation (see calculus of
variations):

• P(X) is designed to be small at desired images features:

edges: • Interpretation: balancing internal and external forces

lines:

s = width of Gaussian smoothing kernel Gs (= scale parameter)

(= force field defined

in each image point)
 Dynamic behavior (‘snake’) Gradient descent optimization

• Treat X(s) as a function of time X(s,t) • Alternatively: energy function E(C(p)) minimized directly by gradient descent
• Let X evolve (‘deform’) according to the forces acting on it: • For each curve parameter pi, compute DEi = E(C(p+dpi))-E(C(p)), i.e. the
change in the energy function when increasing curve parameter pi by some
small amount dpi>0 (= numerical approximation of the derivative)
(or: derive analytical expressions for the gradient)
• Update each pi iteratively by setting pi(k+1) = pi(k) – e.DEi(k)
(g = time step parameter)
e>0 = step size parameter
è ‘snake like’ behavior k = iteration number dp
• At equilibrium: if DEi > 0, decrease pi to decrease E
X(s,t) stabilizes è left hand side = 0 è X is solution of Euler-Lagrange eqn. if DEi < 0, increase pi to decrease E
• Provide initial X(s,0), e.g. by rough manual delineation • Well suited for parametric curves with limited
number of parameters (e.g. splines) E(C(p+dpi))
• E(C(p)) can also be optimized using non-gradient
optimization methods (è likely slower…) E(C(p))

Snakes & dynamic programming Generalization: force formulation

• Gradient descent converges to a local optimum • In the original ‘snakes’ approach (Kass et al., 1997), the curve fitting is
formulated as an energy minimization problem and the external forces
• Re-formulate the curve fitting as graph search problem and use dynamic
acting on the curve are derived as the derivative of the energy (=potential
programming to guarantee finding the globally optimal solution forces)
• Requires discretisation of the solution domain, i.e. each curve point vi is a • Generalization:
pixel, each point can move to pixel locations only and at each stage only a
limited number (m) of candidate positions is considered

• Add additional (heuristic) force terms, e.g. to increase attraction range

• Energy minimization concept is lost (Fext is no longer a potential force)
Emin = min Sn-1(vn,vn-1)
è no underlying objective function that is being optimized by the snake
= mxm cost matrix
Find (vn,vn-1) that minimize Sn-1
Backtrace over vi-1
Amini et al., IEEE TPAMI 1990

Alternative forces GVF (‘Gradient vector flow’) snake

Fp • Traditional ‘snakes’ have difficulties to ‘move’ into a boundary concavity

Pressure force (‘balloon’):
• Solution: alter the external force field such that the ‘snake’ is pushed in
Diffusion equation
f = original edge map
v(x,y,t) = GVF map
Ñ2 = Laplacian
g, h: to be specified
Interactive force (‘spring’): Initialization: v(x,y,0) = f

Original
snake
Fs

GVF
Xu and Prince,
1998
 GVF snake: example Numerical implementation

• Curve discretization:

(h = space step, Dt = time step, i = point index, m points)

• Finite difference approximation:

• Involves interpolation of the external force field at non-grid locations

Numerical implementation Snakes example 1: initialization

• Matrix notation:

(t = Dt/g, Xn = m x 2 matrix, A = m x m banded matrix)

• Matrix inversion:

Snakes example 1: converged 1 Snakes example 1: converged 2

Higher weight for

smoothness
constraint
Þ smoother curve
 Snakes example 2: initialization Snakes example 2: converged 1

Snakes example 2: converged 2 Snakes: discussion

Higher weight for image
term
Þ more sensitive to • No object-specific shape imposed, only generic internal constraints
irrelevant edges
(‘smoothness’)
• Heuristic external energy terms
• Individual contribution of each term to be tuned appropriately
èDifferent images may need different parameters…
èNo automated procedure for parameter selection…
• Converges towards local optimum
è Suitable initialization needed (‘close enough’ to desired solution)
Lower weight for
smoothness • Re-parameterization may be required during optimization (numerical
constraint issues, especially in 3D)
Þ less smooth
• Topologically is fixed (e.g. single closed curve), can not adapt easily to
different topologies (e.g. splitting or merging curves)

Deformable models with implicit geometry Curve evolution using level sets

• Original energy-minimizing ‘snakes’ formulation uses an explicit geometric • Snake evolution equation:
parameterization of the shape (curve or surface)
• Alternatively: embed the curve / surface as a level set in a higher • Only the force component normal to the shape is in fact relevant for curve
dimensional surface, and deform the curve as desired by manipulating this evolution (the tangential component only affects the parameterization):
surface directly (‘Eulerian’ instead of ‘Lagrangian’ formulation)
• Advantage: V = speed function
N = inward normal of the curve
– numerically more stable (no explicit parameterization needed) k = curvature
– can handle changes in curve topology
• Examples:
– V(k) = a.k: curvature deformation, akin to snakes’ internal energy
– V(k) = V0: constant deformation, akin to snakes’ balloon force

• Idea: let speed function V depend on image features such that curve
evolution stops at object boundaries
è implemented using the implicit level set representation
 Level set framework Example

• Level set function defined on 2D or 3D spatial domain, evolving Orginal (t=0)

over time Initial curve C specified
f(x,y,0) initialized to signed
• The contour X of interest is the zero-level set of f:
distance function of C
• Evolution of the zero-level set over time as f evolves (= front propagation):

(chain rule of differentiation)

(Ñf = spatial gradient of f)

• How to evolve f in order to impose a certain behavior on its zero-level set?

(f is negative inside and positive

outside its zero-level set)

G. Peyré
t

Mean curvature evolution of f and its zero-level set f(x,y,t) = 0

Speed functions: Geodesic snakes Speed functions: Chan-Vese

• ‘Geodesic snakes’ (Massadi 1995, Caselles 1993) • ‘Active contours without edges’ (Chan-Vese, 2001)
è region-based instead of edge based: f evolves such
as to make the regions inside and outside its
zero-level set ‘homogeneous’
è Mumford-Shah functional:

(length penalty)
(area penalty)
(intensity inside should be c1)
è Curve evolution is slowed down at image edges, but not pull-backed
è May leak trough boundary gaps (intensity outside should be c2)

(c1 and c2 can be pre-specified or estimated during curve evolution itself)

Numerical implementation Level sets: discussion

• Finite difference approximation of the level set evolution equation • Topologically adaptable: f remains well behaved, while its zero-level set
• Image-based speed function is defined (in a meaningful way) only for the can change topology, breka, merge, form sharp corners, …
zero-level set (i.e. for f(x,y,t) = 0) • f is evaluated on a fixed grid è can use finite difference approximations
è need to extent this to the whole image domain (i.e. all other level sets) for spatial and temporal derivatives è numerically stable solutions
in order to compute the evolution of f over the entire domain • Intrinsic geometric properties of the embedded curve can be derived
è alternatively: narrow-band approximation: update f only in a narrow directly from the level set function f (e.g. curvature)
band around its current zero-level set (using the original speed function) • Easily extendable from 2D to 3D, as no explicit parameterization is
required
• Requires suitable speed function (many variants, see some examples
above), sometimes derived by energy minimization, but not necessarily so
• Application dependent parameter tuning…

Malladi et al., IEEE TPAMI, 1995

 Graph cuts: principle Graph cuts: energy formulation

• Formulate segmentation problem on a graph G with

vertices V (= pixels p) and edges E (= between Regional cost (t-links):
neighbouring pixels p and q)
• Each pixel should get assigned a label A (‘obj’, ‘bkg’)
• Add nodes S for object (‘obj’, source) and T for
background (‘bkg’, sink)
• Two types of connections: Boundary cost (n-links):
– n-links: between neighbouring pixels p and q
– t-links: from each pixel p to each terminal
• Define a cost for each link
– n-links: boundary cost B(p,q): 0 if p and q have the
Hard constraints:
same label, >0 otherwise
– t-links: regional cost R(p|’obj’) and R(p|’bkg’)
• Define a suitable cost-function E(A) that contains local
cost terms (= link costs) only (= Markov Random Field)
• Find labeling A that minimizes E(A) = Markov Random field: terms depend on local neighboorhood only
• INSIGHT: This is equivalent to finding the minimal cut C
• This can be done efficiently using min-cut/max-flow Optimization of E(A) is equivalent to finding the “minimal cut” = ‘cutting’
algorithms.
the set of connections with cost w such that S and T are no longer
connected and the total cost over all cut edges is minimal.
Boykov, 2001 (= analogous to a maximal flow problem between source S and sink T)

Summary Next class

• Statistical shape models

Deformable models based on energy minimization

Representation Local optimization Global optimization

Explicit Snakes Path finding (2D)
Implicit Level set Graph cuts