3D Vision

Torsten Sattler

Spring 2018
 Schedule
Feb 19 Introduction
Feb 26 Geometry, Camera Model, Calibration
Mar 5 Features, Tracking / Matching
Mar 12 Project Proposals by Students

Mar 19 Structure from Motion (SfM) + papers

Mar 26 Dense Correspondence (stereo / optical flow) + papers
Apr 2 Easter Break
Apr 9 Bundle Adjustment & SLAM + papers
Apr 16 Student Midterm Presentations
Apr 23 Multi-View Stereo & Volumetric Modeling + papers
Apr 30 3D Modeling with Depth Sensors + papers
May 7 3D Scene Understanding + papers
May 14 4D Video & Dynamic Scenes + papers
May 21 Whitesuntide
May 28 Student Project Demo Day = Final Presentations
 3D Vision – Class 4

Structure-from-Motion

Chapter 7 in Szeliski’s Book

Chapter 9 in Hartley & Zisserman (online)
Tutorial chapters 3.2 and 4
 Last Lecture: Local Features

Features are key component of many 3D Vision algorithms

 Today:
Structure-from-Motion (SfM)

Johannes L. Schönberger and Jan-Michael Frahm. Structure-from-Motion revisited, CVPR 2016

 Topics Today

• Estimate motion between two images

• Epipolar geometry
• Two view Structure-from-Motion
• Estimate structure from motion
• Triangulation
• Estimate camera pose from structure
• Absolute camera pose solvers
Sequential / Incremental SfM

1. Initialize Motion

2. Initialize Structure

3. Extend Motion 4. Extend Structure

Sequential / Incremental SfM
• Two view reconstruction
• Epipolar geometry
1. Initialize Motion • Fundamental matrix F
• Essential matrix E
• Computing F and E

2. Initialize Structure

3. Extend Motion 4. Extend Structure

Epipolar Geometry

R
 The Fundamental Matrix F

• Algebraic representation of epipolar geometry:

• Projective mapping of points to lines:
x  l' l' = Fx
• F has rank 2 since projection
• Correspondence condition:
• Points x, x’ form correspondence x↔x’ if
x' l' = x' Fx  0
T T
 The Fundamental Matrix F
• Geometric derivation:
 Properties of F
F is the unique 3x3 rank 2 matrix
that satisfies x’TFx=0 for all x↔x’

• Transpose: if F is fundamental matrix for (I,I’), then

FT is fundamental matrix for (I’,I)
• Epipolar lines: l’=Fx & l=FTx’
• Epipoles: on all epipolar lines, thus e’TFx=0, x
e’TF=0, similarly Fe=0
• F has 7 degree of freedom (DOF),
i.e. 3x3 -1(homogeneous) -1(rank 2)
• F is a correlation, projective mapping from a point x to
a line l’=Fx (not a proper correlation, i.e. not
invertible)
 The Essential Matrix E
• Calibrated case: P1=K1[I|0], P2=K2[R|t]
X

x1 x2

R,t
 Properties of E
E is an essential matrix iff two of its singular
values are equal, third is 0

• Relationship to F?

• Inherits F’s properties (see previous slide)

• Degree of freedom?
Degree of Freedom of E
 Computation of F & E

• Linear (8-point) (F & E)

• Minimal (7-point) (F & E)
• Calibrated (5-point) (only E)
 Linear Solution (8-point)
• Basic epipolar equation: x'T
Fx  0
• Expand:
x' xf11  x' yf12  x' f13  y' xf21  y' yf22  y' f 23  xf31  yf32  f 33  0
• Separate known and unknown variables:
x' x, x' y, x' , y' x, y' y, y' , x, y,1 f11, f12 , f13 , f 21, f 22 , f 23 , f31, f32 , f33 T  0
(data) (unknowns)

• Write as linear equation:

 x'1 x1 x'1 y1 x'1 y'1 x1 y'1 y1 y'1 x1 y1 1
         f  0

 x'n xn x ' n yn x' n y'n xn y ' n yn y 'n xn yn 1

• 8 unknowns (up to scale): Use 8 points

 Normalized 8-point Algorithm
 f11 
f 
 12 
 f13 
 x1 x1´ y1 x1´ x1´ x1 y1´ y1 y1´ y1´ x1 y1 1  
x x ´ y x ´ x ´ x y ´ y y ´ y ´ x f
  21 
 2 2 2 2 2 2 2 2 2 2 2 y 2 1 f   0
22
           
   f 23 
 xn xn ´ y n x n ´ x n ´ x n y n ´ y n y n ´ y n ´ xn y n 1  f 
 31 
~10000 ~10000 ~100 ~10000 ~10000 ~100 ~100 ~100 1  f 
32
 
! Orders of magnitude difference f
 33 
between column of data matrix
 least-squares yields poor results
• Normalize point coordinates prior to computing F
• Same as for the normalized DLT algorithm for
homography estimation (see lecture 2)
The Singularity Constraint
 The Singularity Constraint
e'T F  0 Fe  0 detF  0 rank F  2

• SVD from linearly computed F matrix (rank 3):

 1 
F  U 2  VT  U  VT  U  VT  U  VT
 1 1 1 2 2 2 3 3 3

  3 
• Compute closest rank-2 approximation: min F - F' F
 1 
F'  U  2  VT  U11V1T  U2 2V2T
 0
The Singularity Constraint
 Minimal Case:
7 Point Correspondences
• Setup linear system from 7 correspondences:
 x'1 x1 x'1 y1 x'1 y'1 x1 y'1 y1 y'1 x1 y1 1
         f  0

 x'7 x7 x'7 y7 x'7 y'7 x7 y'7 y7 y'7 x7 y7 1

• Resulting solution has 2D solution space

A  U7x7diag 1 ,..., 7 ,0,0V9x9  A[V8V9 ]  09x2
T

• F is linear combination of V8 and V9:

x i (F1  F2 )xi  0, i  1...7
T

• … but F1+λF2 not automatically rank 2

 Minimal Case: 7 Point Correspondences
3

F7pts F
F1 F2

• Enforce rank-2 constraint from determinant:

det(F1  λF2 )  a3 λ 3  a2 λ 2  a1λ  a0  0
• Cubic equation in λ
• Either 1 or 3 solutions
Calibrated Case: 5-point Solver
D. Nister, An Efficient Solution to the Five-Point Relative Pose Problem, CVPR 2003

• Linear equations from 5 points  E11 

E 
 12 
 x1x1 x1 y1 x11 y1x1 y1 y1 y1 x1 y1 1  E13 
 x x x y x y x y y y x y 1  E 
 2 2 2 2 21 2 2 2 2 2 2 2   21 
 x3 x3 x3 y3 x31 y3 x3 y3 y3 y3 x3 y3 1  E22   0
  
 x 
4 x 4 x 
4 y 4 x 
4 1
y 
4 x 4 y 
4 y 4 y 
4 x 4 y 4 1  E23 
 x5 x5 x5 y5 x51 y5 x5 y5 y5 y5 x5 y5 1  E31 
 
 E32 
• 4D linear solution space: E 
 33 
E = xX + yY + zZ + wW scale does not matter, choose w = 1
• Insert into non-linear constraints
detE = 0
10 cubic polynomials

(assumes normalized coordinates)

Calibrated Case: 5-point Solver
D. Nister, An Efficient Solution to the Five-Point Relative Pose Problem, CVPR 2003

• Perform Gauss-Jordan elimination on

polynomials [n] represents polynomial of degree n in z

-z
-z
-z
 Automatic Computation of F

Step 1. Extract features

Step 2. Compute a set of potential matches
Step 3. Robust estimation of F via RANSAC
Step 4. Compute F based on all inliers
Step 5. Look for additional matches
Step 6. Refine F based on all correct matches
RANdom SAmple Consensus (RANSAC)
M. A.. Fischler, R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with
Applications to Image Analysis and Automated Cartography, CACM 1981

• Example: Robust line fitting

RANdom SAmple Consensus (RANSAC)
M. A.. Fischler, R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with
Applications to Image Analysis and Automated Cartography, CACM 1981

• Problem: Estimate F in presence of wrong matches

• RANSAC algorithm:
• Repeat:
• Randomly select minimal sample (5 or 7 points)
• Compute hypothesis for F from minimal sample
• Verify hypothesis: Count inliers
• Update best solution found so far
• Until probability of finding better solution < η

% inliers 90% 80% 70% 60% 50% 20%

#samples (5) 5 12 25 57 145 14k
η=0.01%
#samples (7) 7 20 54 162 587 359k
 Finding More Matches

• Restricted search around epipolar line

(e.g. 1.5 pixels)
 Summary: Epipolar Geometry

[Link]
Sequential / Incremental SfM
• Initialize motion from F or E
• Triangulate structure from motion
1. Initialize Motion

2. Initialize Structure

3. Extend Motion 4. Extend Structure

Initial Motion and Structure Estimation
(Calibrated Case)
• Recap Essential matrix: E=[t]xR
• Motion for two cameras: [I|0], [R|t]
• Essential Matrix decomposition: E=UΣVT

• Recover E and t as
• t=u3 or t=-u3
• R=UWVT or R=UWTVT
• Four solutions, but only one meaningful

(see Hartley and Zisserman, Sec.9.6)

Using the Cheirality Constraint

(see Hartley and Zisserman, Sec. 9.6)

 Triangulation
• Given: Motion, correspondence
• Estimate 3D point via triangulation

L2
C1 x1 X
L1

x2

C2
 Triangulation
• Backprojection

• Triangulation

• Maximum Likelihood Triangulation (geometric error)

 Optimal 3D Point in Epipolar Plane

• Given an epipolar plane, find best 3D point for (m1,m2)

m1 l1
l1 m2´ l2 m1
m1´ m2

m2
l2

• Select closest points (m1´,m2´) on epipolar lines

• Obtain 3D point through exact triangulation
• Guarantees minimal reprojection error (given this epipolar
plane)
Optimal Two-View Triangulation
• Non-iterative method: (Hartley and Sturm, CVIU´97)
• Determine optimal epipolar plane for
reconstruction
l1 m1 l2
m2
1DOF

Dm1,l1    Dm2 ,l 2   (polynomial of degree 6)

2 2

• Reconstruct optimal point from selected epipolar

plane
• Note: Only works for two views
Sequential / Incremental SfM
• Find camera with matches to
previous images
1. Initialize Motion • Matches define 2D-3D
correspondences
• Estimate camera pose wrt. 3D
structure
2. Initialize Structure

3. Extend Motion 4. Extend Structure

Pose Estimation from 2D-3D Matches

2D-3D 2D-3D

mi+1
mi
2D-2D new view

x i  Pi X(x1 ,..., x i 1 )
• Compute P using robust approach (6-point RANSAC)
i+1

• Extend and refine reconstruction

 Compute P with 6-point RANSAC

• Generate hypothesis using 6 points

(two equations per point)

• Planar scenes are degenerate!

(similar DLT algorithm as see in 2nd lecture for

homographies)
 3-Point-Perspective Pose – P3P
(Calibrated Case)

All techniques yield 4th order

polynomial 1841 1981

R. Haralick, D. Lee, K. Ottenburg, M. Nolle. Review and analysis of solutions of the three
point perspective pose estimation problem, CVPR 1991
 Incremental SfM
• Initialize:
• Compute pairwise epipolar geometry
• Find pair to initialize structure and motion
• Repeat:
• For each additional view
• Determine pose from structure
• Extend structure
• Refine structure and motion (bundle adjustment,
see lecture 7)
 Global SfM
• Initialize:
• Compute pairwise epipolar geometry
• Compute:
• Estimate all orientations
• Estimate all positions
• Triangulate structure
• Refine structure and motion (bundle
adjustment)
• Pros: More efficient, more accurate
• Con: Less robust
 SfM Software
• Colmap (Johannes Schönberger)
• Incremental SfM, very efficient, nice GUI,
open source
• VisualSFM (Changchang Wu)
• Incremental SfM, very efficient, GUI, binaries
• Bundler (Noah Snavely)
• Incremental SfM, open source
• OpenMVG (Pierre Moulon)
• Incremental and Global SfM, open source
• Theia (Chris Sweeney)
• Incremental and Global SfM, very efficient,
open source
 Summary

• Estimate motion between two images

• Epipolar geometry
• Estimate structure from motion
• Triangulation
• Estimate camera pose from structure
• Absolute camera pose solvers
• DLT 6-point solver (P6P)
• 3-point-perspective pose solver (P3P)
 Next week:
Dense Correspondence / Stereo