Interest Point Detection

Lecture-4
Contents
 Harris Corner Detector
 Sum of Squares Differences (SSD)
 Corrleation
 Taylor Series
 Eigen Vectors and Eigen Values
 Invariance and co-variance
What is an interest point
 Expressive texture
 The point at which the direction of the
boundary of object changes abruptly
 Intersection point between two or more
edge segments
What is an interest point
 Expressive texture
 The point at which the direction of the
boundary of object changes abruptly
 Intersection point between two or more
edge segments
Synthetic & Real
Interest Points

Corners are indicated in red

Properties of Interest Point
Detectors
 Detect all (or most) true interest points
 No false interest points
 Well localized.
 Robust with respect to noise.
 Efficient detection
Possible Approaches to
Corner Detection
 Based on brightness of images
 Usually image derivatives
 Based on boundary extraction
 First step edge detection
 Curvature analysis of edges
Where can we use it?
 Automate object tracking
 Point matching for computing disparity
 Stereo calibration
 Estimation of fundamental matrix
 Motion based segmentation
 Recognition
 3D object reconstruction
 Robot navigation
 Image retrieval and indexing
Correspondence across views

• Correspondence: matching points, patches,

edges, or regions across images

Slide Credit: James Hays

Example: estimating “fundamental matrix”
that corresponds two views

Slide from Silvio Savarese

Example: structure from motion

Slide Credit: James Hays

This class: interest points
original
• Suppose you have to
click on some point,
go away and come
back after I deform the
image, and click on the
same points again.
– Which points would
you choose?

deformed
Slide Credit: James Hays
Overview of Keypoint Matching
1. Find a set of
distinctive key-
points
A1
2. Define a region
around each
A2 A3 keypoint

3. Extract and
normalize the
region content
fA fB
4. Compute a local
descriptor from the
normalized region
d ( f A, fB )  T
5. Match local
descriptors
K. Grauman, B. Leibe
Goals for Keypoints

Detect points that are repeatable and distinctive

Key trade‐offs
A1

A2 A3

Detection of interest points

More Repeatable More Points

Robust detection Robust to occlusion
Precise localization Works with less texture

Description of patches

More Distinctive More Flexible

Minimize wrong matches Robust to expected variations
Slide Credit: James Hays Maximize correct matches
 Invariant Local Features

Image content is transformed into local feature coordinates that are

invariant to translation, rotation, scale, and other imaging parameters

Features Descriptors
Choosing interest points

Where would you

tell your friend to
meet you?

Slide Credit: James Hays

Feature extraction: Corners

9300 Harris Corners Pkwy, Charlotte, NC

Slides from Rick Szeliski, Svetlana Lazebnik, and Kristin Grauman

 Local features: main components
1) Detection: Identify the
interest points

2) Description :Extract feature

x1  [ x1(1) , , xd(1) ]
vector descriptor surrounding
each interest point.

3) Matching: Determine x 2  [ x1( 2) ,, xd( 2) ]

correspondence between
descriptors in two views

Kristen Grauman
 Goal: interest operator repeatability
• We want to detect (at least some of) the
same points in both images.

No chance to find true matches!

• Yet we have to be able to run the detection

procedure independently per image.
Kristen Grauman
 Goal: descriptor distinctiveness
• We want to be able to reliably determine
which point goes with which.

?
• Must provide some invariance to geometric
and photometric differences between the two
views.
Kristen Grauman
 Some patches can be localized
or matched with higher accuracy than
others.
 Some patches can be localized
or matched with higher accuracy than
others.
Harris Interest Point Detector

Cited by 8636
 Harris Corner Detector
 Corner point can be recognized in a window
 Shifting a window in any direction should give
a large change in intensity

C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

 Basic Idea

“flat” region: “edge”: “corner”:

no change in no change along significant change
all directions the edge direction in all directions
Aperture Problem
Aperture Problem
 Correlation
f  h   f k , l h k , l 
k l

f  Image
h  Kernel

f
h
f1 f2 f3 h1 h2 h3 f * h  f1h1  f 2 h2  f 3 h3
f4 f5 f6  h4 h5 h6  f 4 h4  f 5 h5  f 6 h6
f7 f8 f9 h7 h8 h9  f 7 h7  f 8 h8  f 9 h9
Correlation
f  h   f k , l h k , l  Cross correlation
k l

f  f   f k , l  f k , l  Auto correlation
k l
 Correlation Vs SSD
SSD    f k , l   h k , l 
2
Sum of Squares Difference
minimize k l


SSD   f k , l   2h k , l  f k , l   h k , l 
2 2

k l
SSD    2h k , l  f k , l 
minimize These terms do not depend
on correlation
k l
SSD   2h k , l  f k , l 
maximize k l
Correlation   h k , l  f k , l 
maximize k l

f  f   f k , l  f k , l 
k l
 Mathematics of Harris
Detector
 Change of intensity for the shift (u,v)
E (u, v )   w( x, y ) [ I ( x  u, y  v )  I ( x, y )] 2

x, y
   
window function shifted intensity intensity
Auto-correlation
Auto-Correlation
 Brook Taylor (1685-1731)
His marriage in 1721 with Miss Brydges of
Wallington, Surrey, led to an estrangement
from his father, which ended in 1723 after
her death in giving birth to a son, who also
died.
1725 he married—this time with his
father's approval—Sabetta Sawbridge of
Olantigh, Kent, who also died in childbirth
in 1730 ; in this case, however, his
daughter, Elizabeth, survived.

Taylor was elected a fellow of the Royal

Society early in 1712, and in the same year
sat on the committee for adjudicating the
claims of Sir Isaac Newton and Gottfried
Leibniz about Calculus.
 Taylor Series

ƒ (x ) Can be represented at point a in terms of its derivatives

( x  a) 2 ( x  a)3
f ( x)  f (a)  ( x  a) f x  f xx f xxx 
2! 3!
 Taylor Series
Express I ( x  u, y  v ) at ( x, y ) :

I ( x  u , y  v )  I ( x, y )  I x ( x  u  x )  I y ( y  v  y )

I ( x  u , y  v )  I ( x, y )  I x u  I y v
 Taylor Series

f ( x, y , t )  f ( x  dx , y  dy , t  dt )

Taylor Series of right side

f f f
f ( x, y , t )  f ( x, y , t )  ( x  dx  x )  ( y  dy  y )  (t  dt  t )
x y t
0  f x dx  f y dy  f t dt
dx dy dt
0  fx  fy  ft
dt dt dt
0  f xu  f y v  ft f xu  f y v  ft  0

Optical Flow Constrain Equation

 Mathematics of Harris
Detector
E (u, v )   [ I ( x  u, y  v )  I ( x, y )]2
x, y
  
shifted intensity intensity

E (u, v )   [ I ( x, y )  uI x  vI y  I ( x, y )]2 Taylor Series

x, y
   
shifted intensity intensity

E (u, v )   [uI x  vI ] y
2

x, y
2
  I x 
E (u, v )   u v   IxIx IxIy 
M  
I y I y 
x, y 
  I y 
 Ix  u x, y  I x I y
E (u, v )   u v  I x I y  
x, y Iy  v
  Ix   u 
E (u, v )  u v   I x I y   u 
 x , y  I y   v  E( u ,v )  u v M  
v
 Mathematics of Harris
Detector
u  IxIx IxIy 
E( u ,v )  u v M   M  
v x, y  I x I y I y I y 
 E(u,v) is an equation of an ellipse.

 Let 1 and 2 be eigenvalues of M

(2)
(1)
Eigen Vectors and Eigen Values
The eigen vector, x, of a matrix A is a special vector, with
the following property

Where  is called eigen value

To find eigen values of a matrix A first find the roots of:

Then solve the following linear system for each eigen

value to find corresponding eigen vector
Example

Eigen Values

Eigen Vectors
Eigen Values
Eigen Vectors
 MATLAB Fuction

[vectorC,valueC]=eig(C);
 Mathematics of Harris
Detector
Classification of 2 “Edge”
image points using 2 >> 1 “Corner”
eigenvalues of M: 1 and 2 are large,
1 ~ 2;
E increases in all
directions

1 and 2 are small;

E is almost constant “Flat” “Edge”
in all directions region 1 >> 2
1
 Mathematics of Harris
Detector
 Measure of cornerness in terms of 1, 2

1 1 0 
M  SDS D 
 0 2

R  det D  k trace D  R  12  k 1  2 

2 2
 Mathematics of Harris
Detector
2 “Edge” “Corner”
• R depends only on R<0
eigenvalues of M
• R is large for a corner R>0
• R is negative with large
magnitude for an edge
• |R| is small for a flat
region “Flat” “Edge”
|R| small R<0
1
Compute corner response
Find points with large corner response: R> threshold
Find points with large corner response: R> threshold
Take only the points of local maxima of R

If pixel value is
greater than
its neighbors
then it is a local
maxima.
Other Version of Harris
Detectors

R  1  2 Triggs

det( D ) 12
R  Szeliski (Harmonic mean)
trace( D ) 1  2

R  1 Shi-Tomasi
2

1
 Mathematics of Harris
Detector
 Change of intensity for the shift (u,v)
E (u, v )   w( x, y ) [ I ( x  u, y  v )  I ( x, y )] 2

x, y
   
window function shifted intensity intensity
Auto-correlation

Window functions 

UNIFORM GAUSSIAN
 Mathematics of Harris
Detector
 Change of intensity for the shift (u,v)
E (u, v )   w( x, y ) [ I ( x  u, y  v )  I ( x, y )] 2

x, y
   
window function shifted intensity intensity
Auto-correlation

Window functions 

UNIFORM GAUSSIAN
 Mathematics of Harris
Detector
E (u, v )   w(x, y) [ I ( x  u, y  v )  I ( x, y )]2
x, y
   
window function
w
x
,
y
shifted intensity intensity

E ( u, v )   ︵ 
︶ [ I ( x, y )  uI x  vI y  I ( x, y )]2

 
Taylor Series
  
w
i
n
d
o
w
f
u
n
c
t
i
o
n
x, y intensity
shifted intensity
w
x
,
y

E ( u, v )   ︵ ︶[uI x  vI y ]2
2
w
x
,
y

x, y
 I 
E ( u, v )   ︵ ︶ u v  x 
I 
  y  IxIx IxIy 
M  
x, y

I y I y 
w
x
,
y

︶u v  x I x u

I y  
I
E (u, v )   ︵ x, y  I x I y
I 
x, y  y v
w
x
,
y

  u 
︶ x I x I y  
I
E (u, v )  u v  ︵ u 
 x , y
I 
 y  v  E( u ,v )  u v M  
v
Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:

E (u, v)   w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2

x, y

I(x, y)
E(u, v)

E(3,2)

w(x, y)
Slide Credit: James Hays
Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:

E (u, v)   w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2

x, y

I(x, y)
E(u, v)

E(0,0)

w(x, y)
Slide Credit: James Hays
 Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:

E (u, v)   w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2

x, y

Window Shifted Intensity

function intensity

Window function w(x,y) = or

1 in window, 0 outside Gaussian

Source: R. Szeliski
 Corner Detection: Mathematics
Change in appearance of window w(x,y)
for the shift [u,v]:

E (u, v)   w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2

x, y

We want to find out how this function behaves for

small shifts
E(u, v)

Slide Credit: James Hays

 Corner Detection: Mathematics
The quadratic approximation simplifies to

u 
E (u , v)  [u v] M  
v 

where M is a second moment matrix computed from image

derivatives:

 I x2 IxI y 
M   w( x, y )  2 
x, y  I x I y I y 

M
Slide Credit: James Hays
Corners as distinctive interest points

I x I x IxI y 
M   w( x, y )  
I
 y x I IyIy 
2 x 2 matrix of image derivatives (averaged in
neighborhood of a point).

I I I I
Notation: Ix  Iy  IxI y 
x y x y
Slide Credit: James Hays
 Interpreting the second moment matrix
The surface E(u,v) is locally approximated by a
quadratic form. Let’s try to understand its shape.

u 
E (u , v)  [u v] M  
v 

 I x2 IxI y 
M   w( x, y )  2 
x, y  I x I y I y 

Slide Credit: James Hays

 Interpreting the second moment matrix

First, consider the axis-aligned case

(gradients are either horizontal or vertical)

 I x2 I x I y  1 0 
M   w( x, y )  2 
 
x, y  I x I y I y   0 2 

If either λ is close to 0, then this is not a corner, so

look for locations where both are large.

Slide Credit: James Hays

 Interpreting the second moment matrix
u 
Consider a horizontal “slice” of E(u, v): [u v] M    const
v 
This is the equation of an ellipse.

Slide Credit: James Hays

 Interpreting the second moment matrix
u 
Consider a horizontal “slice” of E(u, v): [u v] M    const
v 
This is the equation of an ellipse.
 0
1  1
Diagonalization of M: M R   R
 0 2 
The axis lengths of the ellipse are determined by the
eigenvalues and the orientation is determined by R
direction of the
fastest change
direction of the
slowest change

(max)-1/2
(min)-1/2
Visualization of second moment matrices

Slide Credit: James Hays

Algorithm

Harris Detector [Harris88]
• Second moment matrix
 I x2 ( D ) I x I y ( D )
 ( I ,  D )  g ( I )    1. Image Ix Iy
 I x I y ( D ) I y ( D ) 
2
derivatives
(optionally, blur first)

Ix2 Iy2 IxIy

2. Square of
derivatives
det M  12
trace M  1  2
3. Gaussian g(Ix2) g(Iy2) g(IxIy)
filter g(I)

4. Cornerness function – both eigenvalues are strong

har  det[ ( I , D)]   [trace( ( I , D)) 2 ] 
g ( I x2 ) g ( I y2 )  [ g ( I x I y )]2   [ g ( I x2 )  g ( I y2 )]2

5. Non-maxima suppression 70
har
Slide Credit: James Hays
 Invariance and covariance
• We want corner locations to be invariant to photometric
transformations and covariant to geometric transformations
• Invariance: image is transformed and corner locations do not change
• Covariance: if we have two transformed versions of the same image,
features should be detected in corresponding locations

Slide Credit: James Hays

 Affine intensity change

IaI+b

• Only derivatives are used =>

invariance to intensity shift I  I + b
• Intensity scaling: I  a I

R R
threshold

x (image coordinate) x (image coordinate)

Partially invariant to affine intensity change

Slide Credit: James Hays
Image translation

• Derivatives and window function are shift-invariant

Corner location is covariant w.r.t. translation

Slide Credit: James Hays

Image rotation

Second moment ellipse rotates but its shape

(i.e. eigenvalues) remains the same

Corner location is covariant w.r.t. rotation

Slide Credit: James Hays

Scaling

Corner

All points will

be classified
as edges

Corner location is not covariant to scaling!

Slide Credit: James Hays
Algorithm

Reading Material
 Section 4.1.1 Feature Detectors
 Richard Szeliski, "Computer Vision: Algorithms and Application“,
Springer.