Photogrammetry & Robotics Lab

Visual Features: Keypoints

(Harris, Shi-Tomasi, Förstner, DoG)

Cyrill Stachniss

Most slides have been created by Cyrill Stachniss but for several slides
courtesy by Gil Levi, A. Efros, J. Hayes, D. Lowe and S. Savarese
1
Motivation

2
Motivation

3
Visual Features:
Keypoints and Descriptors
§ Keypoint is a (locally) distinct
location in an image
§ The feature descriptor summarizes
the local structure around the keypoint

4
Keypoint and Descriptor

keypoint

descriptor at
the keypoint

5
Today’s Topics
§ Keypoints: Finding distinct points
§ Harris corners
§ Shi-Tomasi corner detector
§ Förstner operator
§ Difference of Gaussians

§ Features: Describing a keypoint

§ SIFT – Scale Invariant Feature Transform
§ BRIEF – Binary Robust Independent
Elementary Features
§ ORB – Oriented FAST Rotated BRIEF
6
 Keypoints
“Finding locally distinct points”

Part 1: Corners

7
Corners
§ Corners are often highly distinct points

8
Corners & Edges
§ Corners are often highly distinct points
§ Corners are invariant to translation,
rotation, and illumination
§ Corner = two edges in roughly
orthogonal directions
§ Edge = a sudden brightness change

9
Finding Corners
§ To find corners we need to search for
intensity changes in two directions
§ Compute the SSD of neighbor pixels
around

local patch sum of squared differences

around (x,y) of image intensity values of
pixels under a given shift
(du, dv)
10
Finding Corners
§ To find corners we need to search for
intensity changes in two directions
§ Compute the SSD of neighbor pixels
around

§ Using Taylor expansion, we obtain

Jacobian 11
Finding Corners
§ The Taylor approximation leads to

§ Written in matrix form as

12
Finding Corners
§ Given

§ Move the sums inside the matrix

structure matrix
13
Structure Matrix
§ The structure matrix is key to finding
edges and corners
§ It encodes the changes in image
intensities in a local area

§ Built from the image gradients

14
Computing the Structure Matrix
§ Matrix build from the image gradients

§ Jacobians computed via a convolution with a

gradient kernel such as Scharr or Sobel:

15
Computing the Structure Matrix
§ Matrix build from the image gradients

§ Jacobians via Scharr or Sobel Op:

16
Structure Matrix
§ Summarizes the dominant directions
of the gradient around a point

17
Structure Matrix Examples

18
Structure Matrix Examples

YES!

NO!

NO!
19
Corners from Structure Matrix

YES!

Key idea:
Considers points as corners if their
structure matrix has two large
Eigenvalues

20
Harris, Shi-Tomasi & Förstner
§ Three similar approaches
§ Proposed in
§ 1987 (Förstner)
§ 1988 (Harris)
§ 1994 (Shi-Tomasi)
§ All rely on the structure matrix
§ Use different criterion for deciding of a
point is a corner or not
§ Förstner offers subpixel estimation
21
Harris Corner Criterion
§ Criterion

§ with
: flat region

: edge
: corner
22
Harris Criterion Illustrated

“edge”

“corner”

“edge”

“flat” 23
Shi-Tomasi Corner Detector
§ Criterion: Threshold smallest
Eigenvalue

: corner

24
Shi-Tomasi Criterion Illustrated

“edge” “corner”

“flat” “edge”

25
Förstner Operator Criterion
§ Very similar to Harris corner detector
§ Defined on the inverse of the M
(covariance matrix of possible shifts)
§ Similar criterion on size and roundness
of the error ellipse of covariance
matrix
§ Extension for sub-pixel estimation

26
Non-Maxima Supression
§ Within a local region, looks for the
position with the maximum value
( ) and select this point

§ Example for the Förstner operator

27
Implementation Remarks
§ RGB to gray-scale conversion first
§ Real images are affected by noise,
smoothing of the input is suggested

28
 Summary Corner Detection

convolutions
(smoothing
& derivatives)

29
 Summary Corner Detection

convolutions
(smoothing multiplications
& derivatives)

30
 Summary Corner Detection

convolutions
(smoothing multiplications
& derivatives)
convolutions
(box-summing)

31
 Summary Corner Detection

convolutions
(smoothing multiplications
& derivatives)
convolutions
(box-summing) multiplications,
sums, sqrt
thresholding
non-max suppression
32
Example

Image courtesy: Förstner 33

Harris Corners Example

34
Corner Detectors Comparison
§ All three detectors perform similarly
§ Förstner was the first one and
additionally described subpixel estim.
§ Harris became the most famous corner
detector in the past
§ Shi-Tomasi seems to slightly
outperform Harris corners
§ Most libraries use Shi-Tomasi as the
default corner detector (e.g., openCV)
35
 Keypoints
“Finding locally distinct points”

Part 2: Difference of Gaussians

36
Difference of Gaussians
Keypoints
§ A variant of corner detection
§ Provides responses at corners, edges,
and blobs
§ Blob = mainly constant region but
different to its surroundings

37
Keypoints: Difference of Gaussians
Over Scale-Space Pyramid

Procedure
Over different image pyramid levels
§ Step 1: Gaussian smoothing
§ Step 2: Difference-of-Gaussians: find
extrema (over smoothing scales)
§ Step 3: maxima suppression at edges

38
Illustration

differently
blurred images
Image courtesy: Lowe39
Illustration

differently
blurred images
Image courtesy: Lowe40
Illustration

differently
sized images

differently
blurred images
Image courtesy: Lowe41
Difference of Gaussians
§ Subtract differently blurred images
from each other

§ Increases visibility of corners, edges,

and other detail present in the image
42
Scale Space Representation

t=0, 1, 4, 16, 64, 265 43

Difference of Gaussians
§ Blurring filters out high-frequencies
(noise)
§ Subtracting differently blurred images
from each other only keeps the
frequencies that lie between the blur
level of both images
§ DoG acts as a band-pass filter

44
Difference of Gaussians

Keypoints are extrema in the DoG

over different (smoothing) scales
45
Illustration

Image courtesy: Lowe 46

Extrema Suppression
§ The DoG finds blob-like and corner-
like image structures but also leads to
strong responses along edges
§ Edges are bad for matching
§ Eliminate edges via Eigenvalue test
(similar to Harris corners)

47
Keypoints
§ Two groups of approaches for finding
locally distinct points:
§ 1. Corners via structure matrix
§ Harris, Shi-Tomasi, Förstner
§ 2. Difference of Gaussians
§ Iterates over scales and blur
§ Finds corners and blobs
§ These approaches are key ingredients
of most hand-designed features
48
Summary
§ Keypoints and descriptor together
define common visual features
§ Keypoint defines the location
§ Most keypoints use image gradients
§ Corners and blobs are good keypoints

Outlook: Part 2 – Feature Descriptors

49
Slide Information
§ These slides have been created by Cyrill Stachniss as part of
the Photogrammetry courses taught in 2014 and 2019
§ The slides heavily reply on material by Gil Levi, Alexai Efros,
James Hayes, David Lowe, and Silvio Savarese
§ I tried to acknowledge all people from whom I used images or
videos. In case I made a mistake or missed someone, please
let me know.
§ If you are a university lecturer, feel free to use the course
material. If you adapt the course material, please make sure
that you keep the acknowledgements to others and please
acknowledge me as well. To satisfy my own curiosity, please
send me email notice if you use my slides.

Cyrill Stachniss, 2014

[email protected]

50