Deep Learning

Object Detection and Segmentation

Huỳnh Văn Thống
FPT Univ.
Semantic Segmentation
• Label each pixel in the image with
a category label.
• Don’t differentiate instances, only
care about pixels.

2/24/2025 2
 Segmentation: Dataset
• Pascal VOC: 16k training natural images divided into 20 classes.
• Cityscapes: 25K urban-street images divided into 30 classes.
• ADE20K: 25K (20 stands for 20K training) scene-parsing images
divided into 150 classes.
• MS COCO: 328K images with 80 things categories and 91 stuff
categories.

Models are often pre-trained in the

large MS-COCO dataset, before
finetuned to the specific dataset.

2/24/2025 3
Semantic Segmentation: FCN
• FCN = Fully Convolutional Network.
• Design a network as a bunch of convolutional layers to make
predictions for pixels all at once.

2/24/2025 4
Semantic Segmentation: FCN
• Design a network as a bunch of convolutional layers to make
predictions for pixels all at once.

Problem #1: Effective receptive field size Problem #2: Convolution on high res
is linear in number of conv layers: With L images is expensive! Recall ResNet stem
3x3 conv layers, receptive field is 1+2L aggressively downsamples.
2/24/2025 5
Semantic Segmentation: FCN
• Design network as a bunch of convolutional layers, with
downsampling and upsampling inside the network!

2/24/2025 6
Semantic Segmentation: FCN
• Design network as a bunch of convolutional layers, with
downsampling and upsampling inside the network!

Downsampling: Upsampling : ?
Pooling, strided convolution

2/24/2025 7
In-Network Upsampling: “Unpooling”

2/24/2025 8
In-Network Upsampling: Bilinear Interpolation

Use two closest neighbors in 𝑥 and 𝑦

to construct linear approximations

2/24/2025 9
In-Network Upsampling: Bicubic Interpolation

Use three closest neighbors in 𝑥 and 𝑦 to

construct cubic approximations.
(This is how we normally resize images)

2/24/2025 10
In-Network Upsampling: “Max Unpooling”
Max Pooling: Remember Max Unpooling: Place into
which position had the max remembered positions

Pair each downsampling layer with

an upsampling layer

2/24/2025 11
Learnable Upsampling: Transposed Convolution

Recall: Normal 3 x 3 convolution, stride 1, pad 1

2/24/2025 12
Learnable Upsampling: Transposed Convolution

Recall: Normal 3 x 3 convolution, stride 2, pad 1

2/24/2025 13
Learnable Upsampling: Transposed Convolution

Recall: Normal 3 x 3 convolution, stride 2, pad 1

Convolution with stride > 1 is “Learnable Downsampling”

Can we use stride < 1 for “Learnable Upsampling”?

2/24/2025 14
Learnable Upsampling: Transposed Convolution

3 x 3 transposed convolution, stride 2

2/24/2025 15
Learnable Upsampling: Transposed Convolution

3 x 3 transposed convolution, stride 2

2/24/2025 16
Learnable Upsampling: Transposed Convolution

3 x 3 transposed convolution, stride 2 Sum where outputs

are overlap

2/24/2025 17
Learnable Upsampling: Transposed Convolution

3 x 3 transposed convolution, stride 2 Sum where outputs

are overlap

2/24/2025 18
Transposed Convolution: 1D example

Output has copies of filter

weighted by input.

Stride 2: Move 2 pixels output

for each pixel in input.

Sum at overlaps.

2/24/2025 19
Transposed Convolution: 1D example
Many name:
• Deconvolution (bad).
• Upconvolution.
• Fractionally strided
convolution.
• Backward strided
convolution.
• Transposed Convolution
(best).
2/24/2025 20
Semantic Segmentation: FCN
• Design network as a bunch of convolutional layers, with
downsampling and upsampling inside the network!

Downsampling: Upsampling :
Pooling, strided convolution Iinterpolation,
transposed conv
2/24/2025 21
Semantic Segmentation: FCN
• Combine predictions with different resolutions

Fully Convolutional Networks for Semantic Segmentation. Long et al., CVPR, 2015
2/24/2025 22
Semantic Segmentation: U-Net
• Incorporating the low-
level information.

U-Net: Convolutional Networks for Biomedical Image

Segmentation, Ronneberger et al., MICCAI 2015
2/24/2025 23
Semantic Segmentation: DeepLabV3+
• Encode multi-scale contextual
information by applying atrous
convolution at multiple scales

Encoder-Decoder with Atrous Separable Convolution

for Semantic Image Segmentation, Chen et al., ECCV
2/24/2025 2018 24
Atrous Convolution

Sparse feature extraction with

standard convolution on a
low-resolution input feature
map.

Dense feature extraction with

atrous convolution with rate r=2,
applied on a high-resolution input
feature map.

2/24/2025 25
Semantic Segmentation: DeepLabV3+
• Encode multi-scale contextual
information by applying atrous
convolution at multiple scales.

• Refine the segmentation

results along object
boundaries.

Encoder-Decoder with Atrous Separable Convolution

for Semantic Image Segmentation, Chen et al., ECCV
2/24/2025 2018 26
Computer Vision Tasks
Object Detection: Detects individual Semantic Segmentation: Gives per
object instances, but only gives box. pixel labels, but merges instances

2/24/2025 27
Things and Stuff
Things: Object categories that
can be separated into object
instances (e.g. cats, cars,
person).

Stuff: Object categories that

cannot be separated into
instances (e.g. sky, grass,
water, trees)

2/24/2025 28
Computer Vision Tasks
Object Detection: Detects individual Semantic Segmentation: Gives per
object instances, but only gives box. pixel labels, but merges instances.
(Only things) (Both things and stuff)

2/24/2025 29
Computer Vision Tasks
Instance Segmentation: Detect all objects Semantic Segmentation: Gives per
in the image and identify the pixels that pixel labels, but merges instances.
belong to each object. (Only things!) (Both things and stuff)

2/24/2025 30
Computer Vision Tasks: Instance Segmentation
Instance Segmentation: Detect all
objects in the image, and identify the
pixels that belong to each object.
(Only things!)

Approach: Perform object detection,

then predict a segmentation mask
for each object!

2/24/2025 31
Beyond Instance Segmentation: Panoptic Segmentation

• Label all pixels in the image

(both things and stuff).

• For “thing” categories also

separate into instances.

2/24/2025 32
Beyond Instance Segmentation: Panoptic Segmentation

2/24/2025 33
Panoptic quality (PQ) measure
• Computed per-category and results are averaged
across categories.
• The ground truth and predicted segments are
matched with an IoU threshold 0.5
• TP (matched pairs), FP (unmatched predicted
segments), and FN (unmatched ground truth
segments).

SQ: how close the predicted segments are to the

ground truth segment (does not consider bad RQ: just like for detection, we want to know if we are missing
predictions!) any instances (FN) or predicting more instances (FP)
2/24/2025 34
Next
• Visualization and Understanding
• Attention and Transformer
• Foundation Models and Promptable Segmentation.
• ….

2/24/2025 35
 Questions?

2/24/2025 36