1

Case study:
Visual object detection
Course: RO47002
Lecturer: Julian Kooij
 2

Case study: Visual Object Detection

• Input: a camera image
• Task: detect all instances of a
particular “object”,
e.g. “Pedestrian”.
• Output: bounding boxes around
all detected instances
• Measures of success:
– Don’t miss any pedestrians
– Don’t report any pedestrians where there are none
– The bounding box should tightly fit around the pedestrian
 3

Case study: Visual Object Detection

Q: Should we address this
with machine learning?

A: Yes, uncontrolled outdoor conditions, large variance in object

appearance, size, shape, colour, background
 4

Case study: Visual Object Detection

Q: What category of machine
learning should we use?
• Supervised?
• Unsupervised?
• Reinforcement learning?

A: We want to capture a particular type of concept, “Pedestrian”. Best done

by supervised learning, i.e. giving labelled examples.
Not unsupervised (no clustering). Not RL (not learning in a control loop).
 5

Case study: Visual Object Detection

Q: What type of machine
learning task is this?
• Classification?
• Regression?

• Measures of success:
– Don’t miss any pedestrians
Classification
– Don’t report any pedestrians where there are none
– The bounding box should tightly fit around the pedestrian Regression
 6

Case study: Visual Object Detection

• We will here only focus on
classification problem
• Fixed size bounding boxes

• How to formulate this task

concretely as a function,
with fixed size domain, and
fixed size output?
 7

Visual Object Detection through Classification

Classify region proposals:

Is this a pedestrian?
Yes / No
 8

Visual Object Detection through Classification

Output the region proposals that
were classified positively

Is this a pedestrian?
Yes / No
 9

Visual Object Detection through Classification

☺
True positive:
Pedestrian found


False positive:
Incorrectly classified
as pedestrian


False negative:
Incorrectly classified
as not pedestrian
 10
x1 93

Images as High-Dimensional Vectors x2

x3
65
64
x4 100
x5 135
• I is a gray-scale image patch of 160 x 80 pixels x6 176
x7 •
we consider I is a vector in 12800-D space !
•
x8
•
x9
92
• Putting this in perspective I = x10 = 63
160 x11 61
256 (12800) ≈ 1×10 30720 possible images x12
pixels 108
“number of atoms in the universe” ≈ 4×1080 x13 148
x14 178

• •
• But, almost all of the possible images look like •
•
‘noise’, only relatively small amount can occur • •

in our data! 184

x12800
80 pixels
 11

Visualization – „Real“ Pedestrian Dataset

• „Real“ Pedestrian Dataset
collected at Daimler from a
moving vehicle

• Visualization by projecting
images to a 3D dimensional
space using Principal
Component Analysis (PCA)

• Note: natural images are

concentrated in regions of the
input space, not uniformly
scattered
 12

Image Features x1
Function x = f(I) extracts of features on image I x2
and represents its content by feature vector x. …
xM
x1 x1
x2 x2 Feature
vector
…

captures the

…
image content

xM xM
dense features sparse features
 13

Classifying Objects
1. Feature extraction: turn image region into a D-dimensional feature vector x
2. Classification: apply classifier h, test if h(x) is above a threshold 𝜏

ℎ x ≥𝜏
classify as
“Pedestrian”
x2

ℎ x <𝜏
classify as
“Not Pedestrian”

x1
 14

Classifying Objects
How to obtain useful classifier h(x) ? Training data
• Use representative training data to optimize its decision boundary
positive class:
“Pedestrian”

? negative class:
“Not pedestrian”
x2

Test data
sample to be
classified
decision boundary

x1
 15

Evaluating a Linear classifier

• Classify x : is the label y = -1 or y = +1 ?
+1 • On what side of decision boundry is x ?
x
w -1
• 𝒘⊤ x is positive if α < 90°
α
𝐚⊤ ⋅ 𝐛 = 𝐚 𝐛 cos 𝛼
 16

Evaluating a Linear classifier

• Classify x : is the label y = -1 or y = +1 ?
+1 • On what side of decision boundry is x ?

w -1
• 𝒘⊤ x is positive if α < 90°
α x • 𝒘⊤ x is negative if α > 90°

𝐚⊤ ⋅ 𝐛 = 𝐚 𝐛 cos 𝛼

• Classification rule: 𝑦ො = sign(𝒘⊤ x)

 17

Evaluating a Linear classifier

• Classify x : is the label y = -1 or y = +1 ?
+1 • On what side of decision boundry is x ?

-1
• 𝒘⊤ x is positive if α < 90°
w
x • 𝒘⊤ x is negative if α > 90°

𝐚⊤ ⋅ 𝐛 = 𝐚 𝐛 cos 𝛼
b'
• Classification rule: 𝑦ො = sign(𝒘⊤ x + b)
• Easy to compute, only dot product!
• Parameter wi weights contribution of xi
 18

Problem 1: dimensionality
• Amount of training data needed grows exponentially with dimensionality
• 1D case, 2 samples: few decision boundries possible

2D case, 2 samples: many decision boundries possible

Even worse in spaces with thousands of dimensions!

 19

Problem 2: classes not seperable

?
x2

x1
 20

Solution 1: Different classifiers

More complex classification boundary

Problems
• More complicated to train
• More parameters to optimize
• More expensive to evaluate
x2

? Remember:
many proposals to evaluate!

Latency could become a problem

x1
 21

Solution 2: Use a different representation

Extract better features

x’ = f’(I)
x22

Apply transformation f
x‘

x’ = f(x)
?

x11
x'
 22

What are good features?

What are good features to use?

Intuition
• Captures relevant object information: shape, color distribution, …
• Not affected by irrelevant photometric and geometric changes, noise
• Encodes image content locally (robust to partial occlusion)
• Is efficient: feature vector dimensionality << number of image pixels

What is “relevant” depends on application 22

 23

What are good features?

Example

When distinguishing 60 and 80 speed

limit signs, it makes sense to consider
grayscale and intensity-normalized

This is because color and intensity do

not facilitate differentiation, but
expand the feature space significantly.
 24

Summary
• We have seen how ML can be applied to a task in
Computer Vision
– Classify image patches, not images
– 1 Image does not equal 1 Classification problem
• Not all CV tasks require ML necessarily, but often
required for challenging real-world conditions
• Example of linear classification
• Role of image feature extraction