Keypoints and

Descriptors

Srikumar
Ramalingam

Problem
Statement
Keypoints and Descriptors
Scale Space
and Image
Kernels

Corner
Detection
Srikumar Ramalingam
SIFT
School of Computing
University of Utah
 Presentation Outline

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
1 Problem Statement
Statement

Scale Space
and Image
Kernels 2 Scale Space and Image Kernels
Corner
Detection

SIFT
3 Corner Detection

4 SIFT
 Main paper to be discussed

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels David G. Lowe, Distinctive Image Features from
Corner
Detection
Scale-Invariant Keypoints, IJCV, 2004.
SIFT
 How to find useful keypoints?

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Kristen Grauman
 How do you handle large scale changes?

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Kristen Grauman
 Problem Statement

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
Extract distinctive invariant features from images.
and Image
Kernels Perform reliable matching between different views of an
Corner object or scene.
Detection

SIFT
Provide invariance w.r.t scale changes, rotations, affine
distortions, viewpoints, noise, illumination.
 Uses of SIFT

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem Fundamental matrix estimation

Statement

Scale Space Pose estimation

and Image
Kernels 3D reconstruction
Corner
Detection Object recognition
SIFT Image retrieval
Visual odometry and robot navigation
..
 How to perform object recognition?

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement
The invariant features can be matched against a large
Scale Space
and Image
database of features of known objects from many images.
Kernels
The matching can be done using a fast nearest-neighbor
Corner
Detection algorithm.
SIFT
Pose estimation can be used for verification purposes.
Can handle clutter and occlusion.
 Searching for specific objects from movies

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Kristen Grauman, Sivic, Zisserman
 Searching for specific objects from movies

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Kristen Grauman, Joseph Sivic and Andrew Zisserman
 Repeatability

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

What if we don’t find the same features to match?

0
Source: Kristen Grauman
 Stages of generating SIFT features

Keypoints and
Descriptors

Srikumar Scale-space extrema detection:

Ramalingam
Searches over all scales and image locations.
Problem Efficient implementation using difference-of-Gaussian
Statement function to detect points that are invariant to scale and
Scale Space
and Image
orientation.
Kernels
Keypoint localization: A model is fit to determine
Corner
Detection location and scale. Keypoints are selected based on
SIFT measures of their stability.
Orientation assignment: Compute best orientation(s)
for each keypoint region.
Keypoint descriptor: Use local image gradients at
selected scale and rotation to describe each keypoint
region.
 Presentation Outline

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
1 Problem Statement
Statement

Scale Space
and Image
Kernels 2 Scale Space and Image Kernels
Corner
Detection

SIFT
3 Corner Detection

4 SIFT
 Detection of scale-space extrema

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem Search for stable features across all possible scales, using a
Statement
continuous function of scale known as scale space (Witkin,
Scale Space
and Image 1983).
Kernels

Corner
An image is represented as a one-parameter family of
Detection
smoothed images, the scale-space representation,
SIFT
parameterized by the size of the smoothing kernel used for
suppressing fine-scale structures.
Does it just correspond to the size of the image?
 Why do we use the Gaussian function as the
scale-space kernel?
Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement
We sometimes want to look at a leaf and other times an
Scale Space
entire tree. If it’s a tree, we get rid of small details such as
and Image
Kernels
leaves and twigs. While getting rid of these small details,
Corner we must ensure that we do not introduce new false details.
Detection
Koenderink (1984) and Lindeberg (1994) showed that the
SIFT
only way to ensure this under reasonable assumptions is by
using Gaussian function as the only scale-space kernel.

0
Source: aishack.in
 What is an image kernel?

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement
A kernel, convolution matrix, or mask is a small matrix
Scale Space
and Image that is useful for blurring, sharpening, embossing, edge
Kernels

Corner
detection, and more.
Detection
This is accomplished by means of convolution between a
SIFT
kernel and an image.

0
Source: wikipedia.org
 Identity kernel

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: wikipedia.org
 Edge kernel

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: wikipedia.org
 Box blur and Gaussian kernels

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: wikipedia.org
 Scale Space using Gaussian kernel- Witkin 1983

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
An edge position may shift with increasing scale
Kernels
Two edges may merge with increasing scale
Corner
Detection
An edge may not split into two with increasing scale
SIFT

0
Source: Noah Snavely
 Scale Space

Keypoints and
Descriptors

Srikumar
Ramalingam

The scale space of an image is a function L(x, y , σ).

Problem
Statement
We obtain L(x, y , σ) by convolution of a variable-scale
Scale Space
and Image Gaussian G (x, y , σ) with the input image I (x, y ):
Kernels
L(x, y , σ) = G (x, y , σ) ∗ I (x, y ), where ∗ is the
Corner
Detection convolution operator in x and y .
SIFT The Gaussian G (x, y , σ) is given below:

1 −(x 2 +y 2 )
G (x, y , σ) = e 2σ 2
2πσ 2
 Image kernel - Gaussian function

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: wikipedia.org
 Image kernel - Gaussian function

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

To create a scale space, progressively blur out images

using Gaussian kernel

0
Source: aishack.in
 SIFT - Scale Space and downsampling

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

Scale space images are obtained for different octaves. This

is done by progressively blurring out images using Gaussian
kernels, and then repeating the same after downsampling
the original image by half to create the next octave.

0
Source: aishack.in
 Difference of Gaussians

Keypoints and
Descriptors

Srikumar
Ramalingam
Stable keypoint locations are detected using scale-space
Problem
Statement
extrema in the difference-of-Gaussian function convolved
Scale Space
with the image D(x, y , σ).
and Image
Kernels The diference of Gaussian is computed at two nearby
Corner scales separated by a constant multiplicative factor k.
Detection

SIFT

D(x, y , σ) = (G (x, y , kσ) − G (x, y , σ)) ∗ I (x, y )

= L(x, y , kσ) − L(x, y , σ)

0
Source: aishack.in
 Difference of Gaussians

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT
For each octave of scale space, the initial image is
repeatedly convolved with Gaussians to produce the set of
scale space images shown on the left.
Adjacent Gaussian images are subtracted to produce the
difference-of-Gaussian images on the right.
After each octave, the Gaussian image is down-sampled by
a factor of 2, and the process repeated.
 Laplacian of a Gaussian

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement We take an image and blur it a little using Gaussian
Scale Space function.
and Image
Kernels Calculate the second order derivatives or the Laplacian -
Corner
Detection
locates edges and corners that are good for detecting
SIFT
keypoints.
Computation of the second order derivative is also
extremely sensitive to noise, and the blurring helps.
 Laplacian of a Gaussian

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
The laplacian of a Gaussian is represented as ∇2 G .
and Image
Kernels The scale-invariant laplacian of a Gaussian would be
Corner σ 2 ∇2 G .
Detection

SIFT
The difference of Gaussian images is approximately
equivalent to the laplacian of the Gaussian.
 Locate maxima-minima in DoG images

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

Maxima and minima of the DoG images are detected by

comparing a pixel (marked with X) to its 26 neighbors.
The marked points are the approximate maxima and
minima because the maxima/minima almost never lies
exactly on a pixel.
 Scale-space extrema detection

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT 32 images were synthetically transformed and noise added.

Sampling in scale for efficiency
How many scales should be used per octave? S=?
More scales evaluated, more keypoints found
S < 3, stable keypoints increased too
S > 3, stable keypoints decreased
S = 3, maximum stable keypoints found
0
Source: Kristen Grauman
 Accurate keypoint localization

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection
Fit a 3D quadratic function to the local sample points to
SIFT
determine the interpolated location of the maximum.
The idea is to use the Taylor expansion of the scale-space
function:
δD T 1 δ2D
D(x) = D + x + xT 2 x
δx 2 δx
Here D and its derivates are evaluated at the sample point
and x = (x, y , δ)T is the offset from this point.
 Removing low contrast features

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement
If the intensity value (i.e., without sign) at the current
Scale Space
and Image pixel in the DoG image (that is being checked for
Kernels

Corner
minima/maxima) is less than a certain value, it is rejected.
Detection
The subpixel intensity value is computed using Taylor
SIFT
expansion and if it is less than a threshold, it is rejected.

0
Source: aishack.in
 Presentation Outline

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
1 Problem Statement
Statement

Scale Space
and Image
Kernels 2 Scale Space and Image Kernels
Corner
Detection

SIFT
3 Corner Detection

4 SIFT
 Corners as keypoints

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

A flat region: Both the gradients are small.

An edge: One gradient will be big (perpendicular to the
edge) and the other will be small (along the edge).
A corner: Both gradients are large. If both the gradients
are large, we allow it to be treated as keypoints.
0
Source: Kristen Grauman
 Harris corner detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

Let us consider the intensity change by moving a window

by a small displacement:
X
E (x, y ) = w (x, y )[I (x + u, y + v ) − I (x, y )]2
x,y

0
Source:aishack.in
 Harris corner detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT X
E (x, y ) = w (x, y )[I (x + u, y + v ) − I (x, y )]2
x,y

E (x, y ) is the difference between the original and the

moved window.
(u, v ) is the window’s displacement.
I (x, y ) is the image intensity at (x, y ).
0
Source:aishack.in
 Harris corner detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection
Let us consider the intensity change by moving a window
SIFT
by a small displacement:
X
E (x, y ) = w (x, y )[I (x + u, y + v ) − I (x, y )]2
x,y

We are looking for windows that produce large E value.

Maximize E :
X
E (u, v ) ≈ w (x, y )[I (x, y ) + uIx + vIy − I (x, y )]2
x,y
 Harris corner detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
X
Statement E (u, v ) ≈ w (x, y )[uIx + vIy ]2
Scale Space x,y
and Image
Kernels
X
E (u, v ) ≈ w (x, y )[u 2 Ix2 + v 2 Iy2 + 2uvIx Iy ]
Corner
Detection x,y
SIFT
Ix2
  
X   Ix Iy u
E (u, v ) ≈ w (x, y ) u v
Ix Iy Iy2 v
x,y

0
Source:aishack.in
 Harris corner detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Ix2
  
Problem
X   Ix Iy u
E (u, v ) ≈ w (x, y ) u v
Statement
Ix Iy Iy2 v
Scale Space x,y
and Image  
Kernels   u
Corner E (u, v ) ≈ u v M
Detection v
SIFT
where
Ix2
 
X Ix Iy
M= w (x, y )
Ix Iy Iy2
x,y

0
Source:aishack.in
 Harris Corner Detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT
Let α and β be the Eigenvalues of the matrix M. The
Eigenvalues determine the stability of a matrix.

R = det(M) − k(trace(M))2
det(M) = α × β
trace(M) = α + β
0
Source: Harris and Stevens, 1988
 Harris Corner Detector

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT
R = det(M) − k(trace(M))2
det(M) = α × β
trace(M) = α + β
If both eigenvalues are small, then the pixel is ”flat” (the
white region).
If one eigenvalue is large, and the other is small, then the
 Images and 2D Derivatives

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 2D Derivatives as points

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Fit ellipses to the 2D derivatives

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Classification based on Eigen Values

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Corner Response Measure

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Robert Collins
 Presentation Outline

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
1 Problem Statement
Statement

Scale Space
and Image
Kernels 2 Scale Space and Image Kernels
Corner
Detection

SIFT
3 Corner Detection

4 SIFT
 Orientation assignment

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT Create histogram of local gradient directions at selected

scale.
Assign canonical orientation at peak of smoothed
histogram.
Each key specifies stable 2D coordinates (x,y,scale,
orientations)
If there are two major orientations, then use both.
0
Source: Kristen Grauman
 Keypoint localization with orientation

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

0
Source: Kristen Grauman
 Keypoint Descriptors

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
At this point, each keypoint has
Statement location
Scale Space scale
and Image
Kernels orientation
Corner Next is to compute a descriptor for the local image region
Detection
about each keypoint that is
SIFT
highly distinctive
invariant as possible to variations such as changes in
viewpoint and illumination

0
Source: Kristen Grauman
 Normalization

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image Rotate the window to standard orientation.
Kernels

Corner
Scale the window size based on the scale at which the
Detection point was found.
SIFT

0
Source: Kristen Grauman
 Lowe’s Keypoint Descriptor

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT

Shown with 2x2 descriptors over 8x8.

In experiments, 4x4 arrays of 8 bin histograms is used, a
total of 128 features for one keypoint.

0
Source: Kristen Grauman
 Lowe’s Keypoint Descriptor

Keypoints and
Descriptors

Srikumar
Ramalingam
Use the normalized region about the keypoint.
Problem
Statement Compute gradient magnitude and orientation at each
Scale Space point in the region.
and Image
Kernels Weight them by a Gaussian window overlaid on the circle.
Corner
Detection Create an orientation histogram over the 4x4 subregions of
SIFT the window.
4x4 descriptors over 16x16 sample arrays were used in
practice. 4x4 times 8 directions gives a vector of 128
values.

0
Source: Kristen Grauman
 SIFT for matching objects

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT
 SIFT for matching objects

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Kernels

Corner
Detection

SIFT
 Acknowledgments

Keypoints and
Descriptors

Srikumar
Ramalingam

Problem
Statement

Scale Space
and Image
Some presentation slides are adapted from David Lowe’s
Kernels landmark paper, Kristen Grauman, Andrew Zisserman, Joseph
Corner
Detection
Sivic, wikipedia.org, Robert Collins, and Utkarsh Sinha
SIFT (aishack.in)