Computer Vision

Lecture 1 – Introduction

Prof. Dr.-Ing. Andreas Geiger

Autonomous Vision Group
University of Tübingen / MPI-IS
Martin-Brualla, Radwan, Sajjadi, Barron, Dosovitskiy and Duckworth: NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. CVPR, 2021. 2
Agenda

1.1 Organization

1.2 Introduction

1.3 History of Computer Vision

3
1.1
Organization
Team

Prof. Dr.-Ing. Carolin Katja

Andreas Geiger Schmitt Schwarz

Stefan Kashyap Niklas Michael

Baur Chitta Hanselmann Niemeyer
5
Contents

Goal: Students gain an understanding of the theoretical and practical concepts of

computer vision using deep neural networks and graphical models. A strong focus is
put on 3D vision. After this course, students should be able to develop and train
computer vision models, reproduce research results and conduct original research.

I History of computer vision I Learning in Graphical Models

I Image formation I Shape-from-X
I Structure-from-Motion I Coordinate-based Networks
I Stereo Reconstruction I Recognition
I Probabilistic Graphical Models I Compositional Models
I Applications of Graphical Models I Self-Supervision and Ethics
6
Organization
I SWS: 2V + 2Ü, 6 ECTS, Total Workload: 180h
I Lectures held via YouTube (provided ∼1 week before Lecture Q&A)
I Plenary lecture and exercise Q&A via Zoom: Thursdays 16:00-18:00
I Individual exercise Q&A sessions in small groups every other week
I Register now on ILIAS, max. 20 students per group, first come first serve
I Exam
I Written (date and time will be announced on the course website)
I To qualify for the exam, a student must have registered to the course on ILIAS
I A 0.3 bonus can be obtained upon submission of Latex notes for 1 lecture
I Course Website with YouTube, Zoom and ILIAS links:
https://uni-tuebingen.de/fakultaeten/mathematisch-naturwissenschaftliche-fakultaet/
fachbereiche/informatik/lehrstuehle/autonomous-vision/lectures/computer-vision/

7
Exercises

I Exercises are offered every 2 weeks with 6 assignments in total

I Exercises deepen the understanding with pen & paper and coding tasks
I Handed out on Thursday via the website and introduced via Zoom
I Can be conducted in groups
I Find a group partner via ILIAS forum “Group Partner for Exercises”
I Individual Q&A sessions offered every other Thursday via Zoom
I The exercises will not be graded
I Instead, solutions will be handed out and discussed at the final Q&A
I All lectures and exercises are relevant for the exam

8
Lecture Notes

I We will collaboratively create lecture notes for this new lecture

I Every student may write a summary for one lecture (yielding a 0.3 bonus)
I Summaries shall be lightweight, comprehensive and mathematically concise
I Summaries shall be using the same notation/symbols as used in the slides
I Only the most important illustrations/graphics shall be included
I The compiled pdf should not exceed ∼5 MB per lecture
I TAs consolidate materials and integrate into one large document
I Link to Latex / Overleaf template provided on course website
I Assignments of students to lectures announced via ILIAS on 30.4.
I Deadline for submission: 7 days after the corresponding lecture
I See assignments on ILIAS for your personal deadline
9
Course Materials
Books:
I Szeliski: Computer Vision: Algorithms and Applications
https://szeliski.org/Book/

I Hartley and Zisserman: Multiple View Geometry in Computer Vision

https://www.robots.ox.ac.uk/~vgg/hzbook/

I Nowozin and Lampert: Structured Learning and Prediction in Computer Vision

https://pub.ist.ac.at/~chl/papers/nowozin-fnt2011.pdf

I Goodfellow, Bengio, Courville: Deep Learning

http://www.deeplearningbook.org

I Deisenroth, Faisal, Ong: Mathematics for Machine Learning

https://mml-book.github.io

I Petersen, Pedersen: The Matrix Cookbook

http://cs.toronto.edu/~bonner/courses/2018s/csc338/matrix_cookbook.pdf
10
Course Materials
Tutorials:
I The Python Tutorial
https://docs.python.org/3/tutorial/
I NumPy Quickstart
https://numpy.org/devdocs/user/quickstart.html
I PyTorch Tutorial
https://pytorch.org/tutorials/
I Latex / Overleaf Tutorial
https://www.overleaf.com/learn

Frameworks / IDEs:
I Visual Studio Code
https://code.visualstudio.com/
I Google Colab
https://colab.research.google.com 11
Course Materials
Courses:
I Gkioulekas (CMU): Computer Vision
http://www.cs.cmu.edu/~16385/

I Owens (University of Michigan): Foundations of Computer Vision

https://web.eecs.umich.edu/~ahowens/eecs504/w20/

I Lazebnik (UIUC): Computer Vision

https://slazebni.cs.illinois.edu/spring19/

I Freeman and Isola (MIT): Advances in Computer Vision

http://6.869.csail.mit.edu/sp21/

I Seitz (University of Washington): Computer Vision

https://courses.cs.washington.edu/courses/cse576/20sp/

I Slide Decks covering Szeliski Book

http://szeliski.org/Book/
12
Prerequisites
I Basic math skills
I Linear algebra, probability and information theory. If unsure, have a look at:
Goodfellow et al.: Deep Learning (Book), Chapters 1-4
Luxburg: Mathematics for Machine Learning (Lecture)
Deisenroth et al.: Mathematics for Machine Learning (Book)

I Basic computer science skills

I Variables, functions, loops, classes, algorithms

I Basic Python and PyTorch coding skills

I https://docs.python.org/3/tutorial/ https://pytorch.org/tutorials/

I Experience with deep learning. If unsure, have a look at:

I Geiger: Deep Learning (Lecture)
13
Prerequisites

Linear Algebra:
I Vectors: x, y ∈ Rn
I Matrices: A, B ∈ Rm×n
I Operations: AT , A−1 , Tr(A), det(A), A + B, AB, Ax, x> y
I Norms: kxk1 , kxk2 , kxk∞ , kAkF
I SVD: A = UDV>

14
Prerequisites

Probability and Information Theory:

I Probability distributions: P (X = x)
R
I Marginal/conditional: p(x) = p(x, y)dy , p(x, y) = p(x|y)p(y)
I Bayes rule: p(x|y) = p(y|x)p(x)/p(y)
I Conditional independence: x ⊥ ⊥ y | z ⇔ p(x, y|z) = p(x|z)p(y|z)
R
I Expectation: Ex∼p [f (x)] = x p(x)f (x)dx
I Variance: Var(f (x)) = E (f (x) − E[f (x)])2
 

I Distributions: Bernoulli, Categorical, Gaussian, Laplace

I Entropy: H(x) , KL Divergence: DKL (pkq)

15
Prerequisites

Deep Learning:
I Machine learning basics, linear and logistic regression
I Computation graphs, backpropagation algorithm
I Activation and loss functions, initialization
I Regularization and optimization of deep neural networks
I Convolutional neural networks
I Recurrent neural networks
I Graph neural networks
I Autoencoders and generative adversarial networks

16
Time Management

Activity Time Total

Watching video lectures 2h / week 24h
Self-study of lecture materials 2h / week 24h
Participation in exercise Q&A sessions 2h / week 24h
Solving the assignments 10h / exercise 60h
Preparation for final exam 48h 48h
Total workload 180h

17
1.2
Introduction
Artificial Intelligence

“An attempt will be made to find how to make machines use language, form
abstractions and concepts, solve kinds of problems now reserved for humans, and
improve themselves.” [John McCarthy]

I Machine Learning
I Computer Vision
I Computer Graphics
I Natural Language Processing
I Robotics & Control
I Art, Industry 4.0, Education ...

19
Computer Vision

Goal of Computer Vision is to convert light into meaning (geometric, semantic, . . . )

Image Credits: Antonio Torralba 20

Computer Vision

I What does it mean, to see? The plain man’s

answer (and Aristotle’s, too) would be, to
know what is where by looking.

I To discover from images what is present in

the world, where things are, what actions
are taking place, to predict and anticipate
events in the world.

Slide Credit: Torralba, Freeman, Isola 21

Computer Vision Applications

Slide Credit: Torralba, Freeman, Isola 22

Computer Vision vs. Biological Vision

Over 50% of the processing in the human brain is dedicated to visual information.
23
Computer Vision vs. Computer Graphics

2D Image 3D Scene

Graphics

Vision
Pixel Matrix Objects Material
217 191 252 255 239
102
159
179
80
94
106
200
91
136
146
121
85
138
138
41
Shape/Geometry Motion
Semantics 3D Pose
115 129 83 112 67
94 114 105 111 89

24
Computer Vision vs. Computer Graphics

2D Image 3D Scene

Graphics

Vision
Computer Vision is an ill-posed inverse problem:
I Many 3D scenes yield the same 2D image
I Additional constraints (knowledge about world) required
24
Computer Vision vs. Image Processing

Slide Credits: Rick Szeliski 25

Computer Vision vs. Machine Learning

Deng, Dong, Socher, Li, Li and Li: ImageNet: A large-scale hierarchical image database. CVPR, 2009. 26
The Deep Learning Revolution

Slide Credit: Matthias Niessner 27

CVPR Submitted and Accepted Papers

Slide Credit: CVPR 2019 Welcome Slides 28

CVPR Attendance

Slide Credit: CVPR 2019 Welcome Slides 29

CVPR 2019 Sponsors

Slide Credit: CVPR 2019 Welcome Slides 30

Why is Visual Perception hard?

Papert: The Summer Vision Project. MIT AI Memos, 1966. 31

Why is Visual Perception hard?

32
Why is Visual Perception hard?

What we see What the computer sees

Slide Credits: Li Fei-Fei 33
Challenges: Images are 2D Projections of the 3D World

2D Image 3D Scene

Graphics

Vision

34
Challenges: Images are 2D Projections of the 3D World

Adelson and Pentland’s workshop metaphor:

To explain an image (a) in terms of reflectance, lighting and shape, (b) a painter, (c) a
light designer and (d) a sculptor will design three different, but plausible, solutions.

Adelson and Pentland: The perception of shading and reflectance. Perception as Bayesian inference, 1996. 35
Ames Room Illusion

36
Perspective Illusion

37
Challenges: Viewpoint Variation

Michelangelo (1475-1564) 38
Challenges: Deformation

Xu Beihong (1943) 39
Challenges: Occlusion

René Magritte (1957) 40

Challenges: Illumination

Slide Credits: Antonio Torralba 41

Challenges: Illumination

Slide Credits: Antonio Torralba 41

Challenges: Illumination

Slide Credits: Antonio Torralba 41

Challenges: Illumination

Image Credits: Jan Koenderink 42

Challenges: Motion

43
Challenges: Perception vs. Measurement

http://persci.mit.edu/gallery/checkershadow
Image Credits: Edward H. Adelson 44
Challenges: Perception vs. Measurement

Gaetano Kanizsa (1976) 45

Challenges: Perception vs. Measurement

46
Challenges: Local Ambiguities

Image Credits: Antonio Torralba 47

Challenges: Local Ambiguities

Image Credits: Antonio Torralba 48

Challenges: Local Ambiguities

Image Credits: Antonio Torralba 48

Challenges: Intra Class Variation

http://www.homeworkshop.com/
Image Credits: Antonio Torralba 49
Challenges: Number of Object Categories

Image Credits: Antonio Torralba 50

1.3
History of Computer Vision
Credits

Svetlana Lazebnik (UIUC): Computer Vision: Looking Back to Look Forward

I https://slazebni.cs.illinois.edu/spring20/

Steven Seitz (Univ. of Washington): 3D Computer Vision: Past, Present, and Future
I http://www.youtube.com/watch?v=kyIzMr917Rc
I http://www.cs.washington.edu/homes/seitz/talks/3Dhistory.pdf

52
Pre-History

Perspective Photometry Least Squares Stereopsis

Leonardo da Vinci Johann Heinrich Lambert Carl Friedrich Gauss Charles Wheatstone
(1452–1519) (1728–1777) (1777–1855) (1802–1875)

53
1510: Perspectograph

“Perspective is nothing else than the see-

ing of an object behind a sheet of glass,
smooth and quite transparent, on the
surface of which all the things may be
marked that are behind this glass. All
things transmit their images to the eye
by pyramidal lines, and these pyramids
are cut by the said glass. The nearer to
the eye these are intersected, the smaller
the image of their cause will appear.”
– Leonardo da Vinci

54
1839: Daguerreotype

I First publicly available photographic

process invented by Louis Daguerre
I Widely used during the 1840s and 1850s
I Polish a sheet of silver-plated copper and
treat with fumes to make light sensitive
I Make resulting latent image visible by
fuming it with mercury vapor and remove
its sensitivity to light by chemical treatment
I Rinse, dry and seal behind glass

55
1802-1871: Great Trigonometrical Survey

I Multi-decade project to measure

the entire Indian subcontinent with
scientific precision
I Under the leadership of George
Everest, the project was made
responsible of the Survey of India
I Manual bundle adjustment proves
Mt. Everest highest mountain on
earth mountain on earth

56
Overview

Waves of development:
I 1960-1970: Blocks Worlds, Edges and Model Fitting
I 1970-1981: Low-level vision: stereo, flow, shape-from-shading
I 1985-1988: Neural networks, backpropagation, self-driving
I 1990-2000: Dense stereo and multi-view stereo, MRFs
I 2000-2010: Features, descriptors, large-scale structure-from-motion
I 2010-now: Deep learning, large datasets, quick growth, commercialization

ng
ni
s
et

ar
ry
l
ve

s
N

Le
et

re
ks

-le

om
ra

tu

ep
oc

eu

a
Ge

De
Lo

Fe
Bl

1950 1960 1970 1980 N 1990 2000 2010 2020

57
A Brief History of Computer Vision

1957: Stereo
I Gilbert Hobrough demonstrated an
analog implementation of stereo
image correlation
I This led to the creation of the
Raytheon-Wild B8 Stereomat
I Used to create Elevation Maps
(Photogrammetry, since 1840)
eo
er
St

1950 1960 1970 1980 1990 2000 2010 2020

Roberts: Machine perception of 3-d solids. PhD Thesis, 1965. 58
A Brief History of Computer Vision

1958-1962: Rosenblatt’s Perceptron

I First algorithm and implementation
to train single linear threshold neuron
I Optimization of perceptron criterion:
X
L(w) = − w T xn yn
n∈M

I Novikoff proved convergence

n
tro
ep
rc
Pe

1950 1960 1970 1980 1990 2000 2010 2020

Rosenblatt: The perceptron - a probabilistic model for information storage and organization in the brain. Psychological Review, 1958. 59
A Brief History of Computer Vision

1958-1962: Rosenblatt’s Perceptron

I First algorithm and implementation
to train single linear threshold neuron
I Overhyped: Rosenblatt claimed that
the perceptron will lead to computers
that walk, talk, see, write, reproduce
and are conscious of their existence
n
tro
ep
rc
Pe

1950 1960 1970 1980 1990 2000 2010 2020

Rosenblatt: The perceptron - a probabilistic model for information storage and organization in the brain. Psychological Review, 1958. 59
A Brief History of Computer Vision

1963: Larry Robert’s Blocks World

I Scene understanding for robotics
I Extracts edges as primitives
I Infers 3D structure of an object from
topological structure of the 2D lines
I Interpret images as projections of 3D
scenes, not 2D pattern recognition or
ld
W
ks
oc
Bl

1950 1960 1970 1980 1990 2000 2010 2020

Roberts: Machine Perception of Three-Dimensional Solids. PhD Thesis, 1965. 60
A Brief History of Computer Vision

1966: MIT Summer Vision Project

I Underestimated the challenge of
computer vision, committed to
“blocks world”

t
ec
oj
Pr
er
m
m
Su

1950 1960 1970 1980 1990 2000 2010 2020

Papert: The Summer Vision Project. MIT AI Memos, 1966. 61
A Brief History of Computer Vision

1969: Minsky and Papert publish book

I Several discouraging results
I Showed that single-layer perceptrons
cannot solve some very simple
problems (XOR problem, counting)
I Symbolic AI research dominates 70s

t
per
Pa
y/
sk
in
M

1950 1960 1970 1980 1990 2000 2010 2020

Minsky and Papert: Perceptrons: An introduction to computational geometry. MIT Press, 1969. 62
A Brief History of Computer Vision

1970: MIT Copy Demo

I Vision system recovers structure of a
blocks scene, robot plans and builds
copy from another set of blocks
I Vision, planning and manipulation
I But low-level edge finding not robust
enough for task, led to attention on
low-level vision
o
m
De
py
Co

1950 1960 1970 1980 1990 2000 2010 2020

Papert: The Summer Vision Project. MIT AI Memos, 1966. 63
A Brief History of Computer Vision

1970: Shape from Shading

I Recover 3D from single 2D image
I Assumes Lambertian surface
and constant albedo
I Applies smoothness reglarization to
constrain the ill-posed problem

S
Sf

1950 1960 1970 1980 1990 2000 2010 2020

Horn: Shape From Shading: A Method for Obtaining the Shape of a Smooth Opaque Object From One View. MIT TR, 1970. 64
A Brief History of Computer Vision

1978: Intrinsic Images

I Decomposing an image into its
different intrinsic 2D layers, such as
reflectance, shading, shape and
motion components
I Useful for downstream tasks, e.g.,
object detection independent from

es
shadows and lighting

ag
Im
c
si
n
tri
In

1950 1960 1970 1980 1990 2000 2010 2020

Barrow and Tenenbaum: Recovering intrinsic scene characteristics from images. Computer Vision Systems, 1978. 65
A Brief History of Computer Vision

1980: Photometric Stereo

I Recover 3D from multiple 2D images,
taken from the same viewpoint with
different lighting conditions
I Requires at least 3 images
I Unprecedented detail and accuracy
I Lambertian assumption has been

o
ere
St
relaxed subsequently

ric
et
om
ot
Ph

1950 1960 1970 1980 1990 2000 2010 2020

Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineerings, 1980. 66
A Brief History of Computer Vision

1981: Essential Matrix

I Defines two-view geometry as matrix
mapping points to epipolar lines
I Reduces correspondence search to a
1D problem
I Can be estimated from a set of 2D
correspondences

rix
I Key ideas known for 100 years

at
lM
ia
nt
se
Es
1950 1960 1970 1980 1990 2000 2010 2020
Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature, 1981. 67
A Brief History of Computer Vision

1981: Binocular Scanline Stereo

I Correlate points along epipolar lines
I Use dynamic programming to
introduce constraints along
scanlines (image rows)
I Allows for overcoming abiguities, but
streaking artifacts between rows

eo
er
St
1950 1960 1970 1980 1990 2000 2010 2020
Baker and Binford: Depth from Edge and Intensity Based Stereo. IJCAI, 1981. 68
A Brief History of Computer Vision

1981: Dense Optical Flow

I Pattern of apparent motion of
objects, surfaces, and edges
in a visual scene
I Measured by (densely) tracking
pixels between two frames
I Investigated by Gibson to describe
the visual stimulus of animals

ow
Fl
I Horn-Schunck algorithm

al
tic
Op
1950 1960 1970 1980 1990 2000 2010 2020
Horn and Schunck: Determining Optical Flow. Artificial Intelligence, 1981. 69
A Brief History of Computer Vision

1984: Markov Random Fields

I MRFs for encoding prior knowledge
(e.g., about smoothness)
I Resolves ambiguities in many
ill-posed vision problems
(e.g., stereo, flow, denoising)
I Global optimization (e.g., variational
inference, sampling, belief
propagation, graph cuts)

s
RF
1950 1960 1970 1980 M 1990 2000 2010 2020
Geman and Geman: Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. TPAMI, 1984. 70
A Brief History of Computer Vision

1980s: Part-based Models

I 1973: Pictorial Structures
I 1976: Generalized Cylinders
(solids of revolution, swept curves)
I 1986: Superquadrics
(generalization of quadric surfaces)
I Express complex relationships
I Compact representation

rts
Pa

1950 1960 1970 1980 1990 2000 2010 2020

Pentland: Parts: Structured descriptions of shape. AAAI, 1986. 71
A Brief History of Computer Vision

1986: Backpropagation Algorithm

I Efficient calculation of gradients in a
deep network wrt. network weights
I Enables application of gradient
based learning to deep networks
I Known since 1961, but
first empirical success in 1986

n
io
at
I Remains main workhorse today

pag
pro
ck
1950 1960 1970 1980 Ba 1990 2000 2010 2020
Rumelhart, Hinton and Williams: Learning representations by back-propagating errors. Nature, 1986. 72
A Brief History of Computer Vision

1986: Self-Driving Car VaMoRs

I Developed by Ernst Dickmanns in
context of EUREKA-Prometheus
I Demonstration to Daimler-Benz
Research 1986 in Stuttgart
I Longitudinal & lateral guidance with
lateral acceleration feedback
I Speed: 0 to 36 km/h

s
oR
M
Va
1950 1960 1970 1980 1990 2000 2010 2020
73
A Brief History of Computer Vision

1988: Self-Driving Car ALVINN

I Forward-looking, vision based driving
I Fully connected neural network maps
road images to vehicle turn radius
I Trained on simulated road images
I Tested on unlined paths, lined city
streets and interstate highways
I 90 consecutive miles at up to 70 mph

N
N
VI
AL
1950 1960 1970 1980 1990 2000 2010 2020
Pomerleau: ALVINN: An Autonomous Land Vehicle in a Neural Network. NIPS, 1988. 74
A Brief History of Computer Vision

1992: Structure-from-Motion
I Estimating 3D structures from 2D
image sequences of static scenes
I Requires only a single camera
I Tomasi-Kanade factorization
provides closed-form (SVD-based)
solution for orthographic case
I Today: non-linear least squares

M
Sf
1950 1960 1970 1980 1990 2000 2010 2020
Tomasi and Kanade: Shape and motion from image streams under orthography: a factorization method. IJCV, 1992. 75
A Brief History of Computer Vision

1992: Iterative Closest Points

I Registering two point clouds by
iteratively optimizing a (rigid or
non-rigid) transformation
I Used to aggregate partial 2D or 3D
surfaces from different scans, to
estimate relative camera poses from
point clouds or to localize wrt. a map

P
IC
1950 1960 1970 1980 1990 2000 2010 2020
Besl and McKay: A Method for Registration of 3-D Shapes. PAMI, 1992. 76
A Brief History of Computer Vision

1996: Volumetric Fusion

I Aggregation of multiple implicitly
represented surfaces by averaging
signed distance values
I Mesh-extraction as post-processing

on
si
Fu
ric
et
m
lu
Vo
1950 1960 1970 1980 1990 2000 2010 2020
Curless and Levoy: A Volumetric Method for Building Complex Models from Range Images. SIGGRAPH, 1996. 77
A Brief History of Computer Vision

1998: Multi-View Stereo

I 3D reconstruction from multiple
input images using level-set methods
I Reconstruction vs. image matching
I Proper model of visibility
I Flexible topology
I Provable convergence
I Other approaches (dead-ends):
Voxel-coloring, space carving

VS
M
1950 1960 1970 1980 1990 2000 2010 2020
Faugeras and Keriven: Complete Dense Stereovision Using Level Set Methods. ECCV, 1998. 78
A Brief History of Computer Vision

1998: Stereo with Graph Cuts

I Popular discrete MAP inference
algorithm for Markov Random Fields
I First versions included unary
and pairwise terms
I Later versions also included specific
forms of higher-order potentials
I Global reasoning compared to

ts
Cu
scanline stereo

h
ap
Gr
1950 1960 1970 1980 1990 2000 2010 2020
Boykov, Veksler and Zabih: Markov Random Fields with Efficient Approximations. CVPR, 1998. 79
A Brief History of Computer Vision

1998: Convolutional Neural Networks

I Similar to Neocognitron, but trained
end-to-end using backpropagation
I Implements spatial invariance via
convolutions and max-pooling
I Weight sharing reduces parameters
I Tanh/Softmax activations
I Only good results on MNIST so far

et
nvN
Co
1950 1960 1970 1980 1990 2000 2010 2020
LeCun, Bottou, Bengio and Haffner: Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. 80
A Brief History of Computer Vision

1999: Morphable Models

I Single-view 3D face reconstruction
I Linear combination of
200 laser scans of faces
I Stunning results at the time

ls
e
od
M
le
ab
ph
or
M
1950 1960 1970 1980 1990 2000 2010 2020
Blanz and Vetter: A Morphable Model for the Synthesis of 3D Faces. SIGGRAPH, 1999. 81
A Brief History of Computer Vision

1999: SIFT
I Scale Invariant Feature Transform
I Detection and description of salient
local features in an image
I Enabled many applications
(e.g., image stitching, reconstruction,
motion estimation, . . . )

FT
SI
1950 1960 1970 1980 1990 2000 2010 2020
Lowe: Object Recognition from Local Scale-Invariant Features. ICCV, 1999. 82
A Brief History of Computer Vision

2006: Photo Tourism

I Large-scale 3D reconstruction
from internet photos
I Key ingredients: SIFT feature
matching, bundle adjustment
I Microsoft Photosynth (discont.)

m
ris
ou
ot
ot
Ph
1950 1960 1970 1980 1990 2000 2010 2020
Snavely, Seitz and Szeliski: Photo tourism: exploring photo collections in 3D. SIGGRAPH, 2006. 83
A Brief History of Computer Vision

2007: PMVS
I Patch-based Multi View Stereo
I Robust reconstruction of various
small and large objects
I Performance of 3D reconstruction
techniques continues to increase

VS
PM
1950 1960 1970 1980 1990 2000 2010 2020
Furukawa and Ponce: Accurate, Dense, and Robust Multi-View Stereopsis. CVPR 2007. 84
A Brief History of Computer Vision

2009: Building Rome in a Day

I 3D reconstruction of landmarks and
cities from unstructured Internet
photo-collections
I Follow-up: Rome on a Cloudless Day

y
Da
a
in
e
m
Ro
1950 1960 1970 1980 1990 2000 2010 2020
Agarwal, Snavely, Simon, Seitz and Szeliski: Building Rome in a day. ICCV, 2009. 85
A Brief History of Computer Vision

2011: Kinect
I Active light 3D sensing
I ML for 3D pose estimation
I Multiple hardware generations
I Early versions failed to
commercialize but heavily used for
robotics and vision research

c t
ne
Ki
1950 1960 1970 1980 1990 2000 2010 2020
Shotton et al.: Real-time human pose recognition in parts from single depth images. CVPR, 2011. 86
A Brief History of Computer Vision

2009-2012: ImageNet and AlexNet

ImageNet
I Recognition benchmark (ILSVRC)
I 10 million annotated images
I 1000 categories
AlexNet
I First neural network to win ILSVRC

et
N
via GPU training, deep models, data

ex
Al
e/
ag
Im
1950 1960 1970 1980 1990 2000 2010 2020
Krizhevsky, Sutskever and Hinton: ImageNet classification with deep convolutional neural networks. NIPS, 2012. 87
A Brief History of Computer Vision

2009-2012: ImageNet and AlexNet

ImageNet
I Recognition benchmark (ILSVRC)
I 10 million annotated images
I 1000 categories
AlexNet
I First neural network to win ILSVRC

et
N
via GPU training, deep models, data

ex
Al
e/
I Sparked deep learning revolution

ag
Im
1950 1960 1970 1980 1990 2000 2010 2020
Krizhevsky, Sutskever and Hinton: ImageNet classification with deep convolutional neural networks. NIPS, 2012. 87
A Brief History of Computer Vision

2002-now: Golden Age of Datasets

I Middlebury Stereo and Flow
I KITTI, Cityscapes: Self-driving
I PASCAL, MS COCO: Recognition
I ShapeNet, ScanNet: 3D DL
I Visual Genome: Vision/Language
I MITOS: Breast cancer

ts
se
ta
Da
1950 1960 1970 1980 1990 2000 2010 2020
Geiger, Lenz and Urtasun: Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. CVPR, 2012. 88
A Brief History of Computer Vision

2012-now: Synthetic Data

I Annotating real data is expensive
I Led to surge of synthetic datasets
I Creating 3D assets is also costly

tic
he
nt
Sy
1950 1960 1970 1980 1990 2000 2010 2020
Dosovitskiy et al.: FlowNet: Learning Optical Flow with Convolutional Networks. ICCV, 2015. 89
A Brief History of Computer Vision

2012-now: Synthetic Data

I Annotating real data is expensive
I Led to surge of synthetic datasets
I Creating 3D assets is also costly
I But even very simple 3D datasets
proved tremendously useful for
pre-training (e.g., in optical flow)

tic
he
nt
Sy
1950 1960 1970 1980 1990 2000 2010 2020
Dosovitskiy et al.: FlowNet: Learning Optical Flow with Convolutional Networks. ICCV, 2015. 89
A Brief History of Computer Vision

2014: Visualization
I Goal: provide insights into what the
network (black box) has learned
I Visualized image regions that most
strongly activate various neurons at
different layers of the network
I Found that higher levels capture
more abstract semantic information

n
io
at
a liz
su
Vi
1950 1960 1970 1980 1990 2000 2010 2020
Zeiler and Fergus: CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. CVPR Workshops, 2014. 90
A Brief History of Computer Vision

2014: Adversarial Examples

I Accurate image classifiers can be
fooled by imperceptible changes
(here magnified for visibility)
I All images in the right column are
classified as “ostrich”

es
pl
m
xa
E
v.
Ad
1950 1960 1970 1980 1990 2000 2010 2020
Szegedy et al.: Intriguing properties of neural networks. ICLR, 2014. 91
A Brief History of Computer Vision

2014: Generative Adversarial Networks

I Deep generative models (VAEs,
GANs) produce compelling images
I StyleGAN2 is state-of-the-art
I Results on faces hard to
distinguish from real images
I Active research on image translation,
domain adaptation, content and
scene generation and 3D GANs

s
N
GA
1950 1960 1970 1980 1990 2000 2010 2020
Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, Bengio: Generative Adversarial Networks. NIPS, 2014. 92
A Brief History of Computer Vision

2014: DeepFace
I Combination of model-based
alignment with deep learning
for face recognition
I First model to reach human-level
face recognition performance

ce
Fa
ep
De
1950 1960 1970 1980 1990 2000 2010 2020
Taigman, Yang, Ranzato and Wolf: DeepFace: Closing the Gap to Human-Level Performance in Face Verification. CVPR, 2014. 93
A Brief History of Computer Vision

2014: 3D Scene Understanding

I Parsing RGB and RGB-D images into
holistic 3D scene representations
I Methods for indoors and outdoors

es
en
Sc
3D
1950 1960 1970 1980 1990 2000 2010 2020
Geiger, Lauer, Wojek, Stiller and Urtasun: 3D Traffic Scene Understanding From Movable Platforms. PAMI, 2014. 94
A Brief History of Computer Vision

2014: 3D Scanning
I 3D scanning techniques allow for
creating accurate replicas
I Debevec’s team scans Obama
I Exhibition in Smithsonian
I https://dpo.si.edu/blog/

ng
ni
an
Sc
3D
1950 1960 1970 1980 1990 2000 2010 2020
Metallo et al.: Scanning and printing a 3D portrait of president Barack Obama. SIGGRAPH Studio, 2015. 95
A Brief History of Computer Vision

2015: Deep Reinforcement Learning

I Learning a policy (state→action)
through random exploration and
reward signals (e.g., game score)
I No other supervision
I Success on many Atari games
I But some games remain hard

RL
ep
De
1950 1960 1970 1980 1990 2000 2010 2020
Mnih et al.: Human-level control through deep reinforcement learning. Nature, 2015. 96
A Brief History of Computer Vision

2016: Style Transfer

I Manipulate photograph to adopt
style of a another image (painting)
I Uses deep network pre-trained on
ImageNet for disentangling
content from style
I It is fun! Try yourself:
https://deepart.io/

er
sf
an
Tr
yle
St
1950 1960 1970 1980 1990 2000 2010 2020
Gatys, Ecker and Bethge: Image Style Transfer Using Convolutional Neural Networks. CVPR, 2016. 97
A Brief History of Computer Vision

2015-2017: Semantic Segmentation

I Assign semantic class to every pixel
I Semantic segmentation starts to
work on challenging real-world
datasets (e.g., CityScapes)
I 2015: FCN, SegNet
I 2016: DeepLab, FSO
I 2017: DeepLabv3

s
tic
an
m
Se
1950 1960 1970 1980 1990 2000 2010 2020
Kundu, Vineet and Koltun: Feature Space Optimization for Semantic Video Segmentation. CVPR, 2016. 98
A Brief History of Computer Vision

2017: Mask R-CNN

I Deep neural network for joint object
detection and instance segmentation
I Outputs “structured object”, not only
a single number (class label)
I State-of-the-art on MS-COCO

N
CN
R-
k
as
M
1950 1960 1970 1980 1990 2000 2010 2020
He, Gkioxari, Dollár and Ross Girshick: Mask R-CNN. ICCV, 2017. 99
A Brief History of Computer Vision

2017: Image Captioning

I Growing interest in combining
vision with language
I Several new tasks emerged
including image captioning
and visual question answering
I However, models still lack
understanding / commonsense

ng
ni
io
pt
Ca
1950 1960 1970 1980 1990 2000 2010 2020
Karpathy and Fei-Fei: Deep Visual-Semantic Alignments for Generating Image Descriptions. PAMI, 2017. 100
A Brief History of Computer Vision

2018: Human Shape and Pose

I Human pose/shape models mature
I Rich parametric models
(e.g., SMPL, STAR)
I Regression from RGB images only
I Models of pose-dependent
deformation and clothing

s
an
um
H
1950 1960 1970 1980 1990 2000 2010 2020
Kanazawa, Black, Jacobs and Malik: End-to-End Recovery of Human Shape and Pose. CVPR, 2018. 101
A Brief History of Computer Vision

2016-2020: 3D Deep Learning

I First deep models to output
3D representations
I Voxels, point clouds, meshes,
implicit representations
I Prediction of 3D models
even from a single image
I Geometry, materials, light, motion

DL
3D
1950 1960 1970 1980 1990 2000 2010 2020
Niemeyer, Mescheder, Oechsle, Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. CVPR, 2020. 102
A Brief History of Computer Vision

Applications and Commercial Products

Google Portrait Mode Skydio 2 Drone Self-Driving Cars Microsoft Hololens Iris Recognition

1950 1960 1970 1980 1990 2000 2010 2020

103
A Brief History of Computer Vision

Current Challenges
I Un-/Self-Supervised Learning
I Interactive learning
I Accuracy (e.g., self-driving)
I Robustness and generalization
I Inductive biases
I Understanding and mathematics
I Memory and compute
I Ethics and legal questions

104