Feature extraction: Corners and blobs

Review: Linear filtering and edge detection

• Name two different kinds of image noise
• Name a non-linear smoothing filter
• What advantages does median filtering have
over Gaussian smoothing?
• What is aliasing?
• How do we find edges?
• Why do we need to smooth before computing
image derivatives?
• What are some characteristics of an “optimal”
edge detector?
• What is nonmaximum suppression?
• What is hysteresis thresholding?
Why extract features?
• Motivation: panorama stitching
• We have two images – how do we combine them?
Why extract features?
• Motivation: panorama stitching
• We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
Why extract features?
• Motivation: panorama stitching
• We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
Step 3: align images
Characteristics of good features

• Repeatability
• The same feature can be found in several images despite geometric
and photometric transformations
• Saliency
• Each feature has a distinctive description
• Compactness and efficiency
• Many fewer features than image pixels
• Locality
• A feature occupies a relatively small area of the image; robust to
clutter and occlusion
Applications
Feature points are used for:
• Motion tracking
• Image alignment
• 3D reconstruction
• Object recognition
• Indexing and database retrieval
• Robot navigation
 Finding Corners

• Key property: in the region around a corner,

image gradient has two or more dominant
directions
• Corners are repeatable and distinctive

C.Harris and M.Stephens. "A Combined Corner and Edge Detector.“

Proceedings of the 4th Alvey Vision Conference: pages 147--151.
 The Basic Idea
• We should easily recognize the point by
looking through a small window
• Shifting a window in any direction should
give a large change in intensity

“flat” “edge”: “corner”:

region: no change significant
no change along the change in all
in all
Source: A. Efros
edge direction directions
 Harris Detector: Mathematics

Change of intensity for the shift [u,v]:

E (u , v)  w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2

x, y

Window Shifted Intensity

function intensity

Window function w(x,y) = or

1 in window, 0 outside Gaussian

Source: R. Szeliski
 Harris Detector: Mathematics

Change of intensity for the shift [u,v]:

E (u , v)  w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2

x, y

Second-order Taylor expansion of E(u,v) about (0,0)

(bilinear approximation for small shifts):
 Eu (0,0) 1  Euu (0,0) Euv (0,0)  u 
E (u , v) E (0,0)  [u v]   [u v]   v
E
 v ( 0, 0 )  2 E
 uv ( 0, 0 ) E vv ( 0,0 )  
 Harris Detector: Mathematics
The bilinear approximation simplifies to

 u
E (u , v)  [u v] M  
 v

where M is a 22 matrix computed from image derivatives:

 I x2 IxI y 
M  w( x, y )  2 
x, y  I x I y I y 

M
Interpreting the second moment matrix

First, consider an axis-aligned corner:

 Interpreting the second moment matrix

First, consider an axis-aligned corner:

  I x2 I I x y
  1 0 
M    
  I x I y I   0 2 
2
y

This means dominant gradient directions align with

x or y axis
If either λ is close to 0, then this is not a corner, so
look for locations where both are large.

Slide credit: David Jacobs

General Case
 1 0  1
Since M is symmetric, we have M R   R
 0 2 
We can visualize M as an ellipse with axis
lengths determined by the eigenvalues and
orientation determined by R
direction of the
fastest change
Ellipse equation:
direction of the
slowest change
 u
[u v] M   const (max)-1/2
 v (min)-1/2
Visualization of second moment matrices
Visualization of second moment matrices
Interpreting the eigenvalues
Classification of image points using eigenvalues
of M:
2 “Edge”
2 >> 1 “Corner”
1 and 2 are large,
 1 ~  2;
E increases in all
directions

1 and 2 are small;

E is almost constant “Flat” “Edge”
in all directions region 1 >> 2

1
Corner response function
R det( M )   trace( M ) 2 12   (1  2 ) 2
α: constant (0.04 to 0.06)
“Edge”
R<0 “Corner”
R>0

|R| small
“Flat” “Edge”
region R<0
Harris Detector: Steps
Harris Detector: Steps
Compute corner response R
Harris Detector: Steps
Find points with large corner response: R>threshold
Harris Detector: Steps
Take only the points of local maxima of R
Harris Detector: Steps
 Harris detector: Summary of steps

1. Compute Gaussian derivatives at each pixel

2. Compute second moment matrix M in a
Gaussian window around each pixel
3. Compute corner response function R
4. Threshold R
5. Find local maxima of response function
(nonmaximum suppression)
Invariance
• We want features to be detected despite
geometric or photometric changes in the
image: if we have two transformed versions of
the same image, features should be detected
in corresponding locations
Models of Image Change

Geometric
• Rotation

• Scale

• Affine
valid for: orthographic camera, locally planar object

Photometric
• Affine intensity change (I  a I + b)
Harris Detector: Invariance Properties
Rotation

Ellipse rotates but its shape (i.e. eigenvalues)

remains the same

Corner response R is invariant to image rotation

 Harris Detector: Invariance Properties
Affine intensity change

 Only derivatives are used =>

invariance to intensity shift I  I + b
 Intensity scale: I  a I

R R
threshold

x (image coordinate) x (image coordinate)

Partially invariant to affine intensity change

Harris Detector: Invariance Properties
Scaling

Corner

All points will

be classified
as edges

Not invariant to scaling

 Scale-invariant feature detection
• Goal: independently detect corresponding
regions in scaled versions of the same image
• Need scale selection mechanism for finding
characteristic region size that is covariant with
the image transformation
Scale-invariant features: Blobs
 Recall: Edge detection

Edge
f

d Derivative
g of Gaussian
dx

d Edge = maximum
f g of derivative
dx

Source: S. Seitz
 Edge detection, Take 2

Edge
f

2 Second derivative
d of Gaussian
2
g
dx (Laplacian)

d2 Edge = zero crossing

f 2g of second derivative
dx

Source: S. Seitz
From edges to blobs
• Edge = ripple
• Blob = superposition of two ripples

maximum

Spatial selection: the magnitude of the Laplacian

response will achieve a maximum at the center of
the blob, provided the scale of the Laplacian is
“matched” to the scale of the blob
 Scale selection
• We want to find the characteristic scale of the
blob by convolving it with Laplacians at several
scales and looking for the maximum response
• However, Laplacian response decays as scale
increases:

original signal increasing σ

(radius=8)

Why does this happen?

Scale normalization
• The response of a derivative of Gaussian
filter to a perfect step edge decreases as σ
increases

1
 2
Scale normalization
• The response of a derivative of Gaussian
filter to a perfect step edge decreases as σ
increases
• To keep response the same (scale-invariant),
must multiply Gaussian derivative by σ
• Laplacian is the second Gaussian derivative,
so it must be multiplied by σ2
 Effect of scale normalization
Original signal Unnormalized Laplacian response

Scale-normalized Laplacian response

maximum
Blob detection in 2D
Laplacian of Gaussian: Circularly symmetric
operator for blob detection in 2D

2 2
2 g  g
g 2  2
x y
 Blob detection in 2D
Laplacian of Gaussian: Circularly symmetric
operator for blob detection in 2D

 g  g
2 2
Scale-normalized: 2
norm g   2  2 
2

 x y 
Scale selection
• The 2D Laplacian is given by
 ( x 2  y 2 ) / 2 2
( x  y  2 ) e
2 2 2
(up to scale)

• Therefore, for a binary circle of radius r, the

Laplacian achieves a maximum at  r / 2

Laplacian response

r/ 2 scale (σ)
image
 Characteristic scale
• We define the characteristic scale as the
scale that produces peak of Laplacian
response

characteristic scale
T. Lindeberg (1998). "Feature detection with automatic scale selection."
International Journal of Computer Vision 30 (2): pp 77--116.
Scale-space blob detector
1. Convolve image with scale-normalized
Laplacian at several scales
2. Find maxima of squared Laplacian response
in scale-space
Scale-space blob detector: Example
Scale-space blob detector: Example
Scale-space blob detector: Example
 Efficient implementation
Approximating the Laplacian with a difference of
Gaussians:

L  2 Gxx ( x, y,  )  G yy ( x, y,  ) 
(Laplacian)

DoG G ( x, y, k )  G ( x, y,  )
(Difference of Gaussians)
 Efficient implementation

David G. Lowe. "Distinctive image features from scale-invariant keypoints.”

IJCV 60 (2), pp. 91-110, 2004.
From scale invariance to affine invariance
Affine adaptation
 I x2 IxI y   0
1 1
Recall: M  w( x, y )  2 
R   R
x, y  I x I y I y   0 2 
We can visualize M as an ellipse with axis
lengths determined by the eigenvalues and
orientation determined by R
direction of the
fastest change
Ellipse equation: direction of the
slowest change
 u (max)-1/2
[u v] M   const
 v (min)-1/2
Affine adaptation
• The second moment ellipse can be viewed as
the “characteristic shape” of a region
• We can normalize the region by transforming
the ellipse into a unit circle
Orientation ambiguity
• There is no unique transformation from an
ellipse to a unit circle
• We can rotate or flip a unit circle, and it still stays a unit circle
Orientation ambiguity
• There is no unique transformation from an
ellipse to a unit circle
• We can rotate or flip a unit circle, and it still stays a unit circle
• So, to assign a unique orientation to keypoints:
• Create histogram of local gradient directions in the patch
• Assign canonical orientation at peak of smoothed histogram

0 2
Affine adaptation
• Problem: the second moment “window”
determined by weights w(x,y) must match the
characteristic shape of the region
• Solution: iterative approach
• Use a circular window to compute second moment matrix
• Perform affine adaptation to find an ellipse-shaped window
• Recompute second moment matrix using new window and
iterate
 Iterative affine adaptation

K. Mikolajczyk and C. Schmid,

Scale and Affine invariant interest point detectors, IJCV 60(1):63-
86, 2004.
http://www.robots.ox.ac.uk/~vgg/research/affine/
Affine adaptation example

Scale-invariant regions (blobs)

Affine adaptation example

Affine-adapted blobs
 Summary: Feature extraction

Eliminate rotational Compute appearance

Extract affine regions Normalize regions ambiguity descriptors

SIFT (Lowe ’04)

Invariance vs. covariance
Invariance:
• features(transform(image)) = features(image)

Covariance:
• features(transform(image)) = transform(features(image))

Covariant detection => invariant description

Assignment 1 due February 14
Implement the Laplacian blob detector:

http://www.cs.unc.edu/~lazebnik/spring08/assignment1.html
Next time: Fitting