Scribd
Upload
2 views

Lecture-21-Semantic-Segmentation

The document discusses semantic segmentation, which involves assigning labels to each pixel in an image for pixel-level analysis. It highlights common datasets like PASCAL VOC and MSCOCO, and applications including autonomous navigation, medical diagnosis, and image editing. The lecture also covers techniques before deep learning, the use of Fully Convolutional Networks (FCN), and various architectures such as SegNet and U-Net for semantic segmentation tasks.

Uploaded by

muneebke
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
2 views

Lecture-21-Semantic-Segmentation

The document discusses semantic segmentation, which involves assigning labels to each pixel in an image for pixel-level analysis. It highlights common datasets like PASCAL VOC and MSCOCO, and applications including autonomous navigation, medical diagnosis, and image editing. The lecture also covers techniques before deep learning, the use of Fully Convolutional Networks (FCN), and various architectures such as SegNet and U-Net for semantic segmentation tasks.

Uploaded by

muneebke
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 24

Semantic Segmentation

Lecture 17
21

11/30/2021 CAP5415 - Lecture 17 2


Semantic Segmentation
Assign a label to each pixel in an image:
• Pixel-level image annotation/analysis (vs. object-level analysis)

Common datasets: PASCAL VOC (2012) and MSCOCO


11/30/2021 CAP5415 - Lecture 17 3
Semantic Segmentation
▪ A key part of Scene Understanding
▪ Applications
▪ Autonomous navigation
▪ Assisting the partially sighted
▪ Medical diagnosis
▪ Image editing

(Clockwise11/30/2021 CAP5415 - Lecture 17


from top) [1] Cityscapes Dataset. [2] ISBI Challenge 2015, dental x-ray images. [3] Royal National Institute of Blind People 4
Semantic Segmentation
▪ Applications
▪ Autonomous navigation

(Clockwise11/30/2021 CAP5415 - Lecture 17


from top) [1] Cityscapes Dataset. [2] ISBI Challenge 2015, dental x-ray images. [3] Royal National Institute of Blind People 5
Semantic Segmentation
▪ Applications
▪ Assisting the partially sighted

(Clockwise11/30/2021 CAP5415 - Lecture 17


from top) [1] Cityscapes Dataset. [2] ISBI Challenge 2015, dental x-ray images. [3] Royal National Institute of Blind People 6
Semantic Segmentation
▪ Applications
▪ Medical diagnosis

(Clockwise11/30/2021 CAP5415 - Lecture 17


from top) [1] Cityscapes Dataset. [2] ISBI Challenge 2015, dental x-ray images. [3] Royal National Institute of Blind People 7
Semantic Segmentation
▪ Applications
▪ Image editing

(Clockwise11/30/2021 CAP5415 - Lecture 17


from top) [1] Cityscapes Dataset. [2] ISBI Challenge 2015, dental x-ray images. [3] Royal National Institute of Blind People 8
Segmentation tasks
A sample image from the PASCAL VOC2011 dataset

Original (input) Image Instance (object) Segmentation Semantic (class) Segmentation

Question: How about the frame, the fruit (tomato?) and the papers on the table?
11/30/2021 CAP5415 - Lecture 17 9
Image Source: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/segexamples/index.html
Before deep learning
Many techniques… Such as:
• TextonBoost
• J. Shotton, J. Winn, C. Rother, and A. Criminisi,
TextonBoost: Joint Appearance, Shape And Context Modeling For Multi-class Object
Recognition And Segmentation, ECCV 2006.

• TextonForest
• “Semantic Texton Forests for Image Categorization and Segmentation”, 2008

• Conditional Random Forest based approaches:


• SuperParsing (J. Tighe and S. Lazebnik, SuperParsing: Scalable Nonparametric Image
Parsing with Superpixels, ECCV 2010)

11/30/2021 CAP5415 - Lecture 17 10


Fully Convolutional networks (FCN)

J. Long, E. Shelhamer, and T. Darrell, Fully Convolutional Networks for Semantic Segmentation, CVPR 2015

11/30/2021 CAP5415 - Lecture 17 11


Fully “CONVOLUTIONAL” Networks (FCN)

• Use pre-trained networks for classification for segmentation! (VGG,


AlexNet, etc.)

• Re-interpret the fully-connected layers as fully convolutional networks.

• Utilize skip-layer concept to improve the segmentation accuracy.

11/30/2021 CAP5415 - Lecture 17 12


Fully Convolutional networks (FCN)

Interpret the FC layers as conv layers.

11/30/2021 CAP5415 - Lecture 17 13


J. Long, E. Shelhamer, and T. Darrell, Fully Convolutional Networks for Semantic Segmentation, CVPR 2015
FCN

11/30/2021 CAP5415 - Lecture 17 14


(21 is the number of classes here!)
FCN

No skip connection 1-skip connection 2-skip connections

11/30/2021 CAP5415 - Lecture 17 15


J. Long, E. Shelhamer, and T. Darrell, Fully Convolutional Networks for Semantic Segmentation, CVPR 2015
Upsampling

11/30/2021 CAP5415 - Lecture 17 16


Upsampling
Bi-linear interpolation

11/30/2021 CAP5415 - Lecture 17 17


Upsampling

11/30/2021 CAP5415 - Lecture 17 18


Deconvolution

11/30/2021 CAP5415 - Lecture 17 19


Deconvolution Network for Semantic Segmentation

11/30/2021 CAP5415 - Lecture 17 20


H. Noh, S. Hong, and B. Han, Learning Deconvolution Network for Semantic Segmentation, ICCV 2015
Input image 14 × 14 deconvolutional layer 28 × 28 unpooling layer 28 × 28 deconvolutional layer 56 × 56 unpooling layer

56 × 56 deconvolutional layer 112 × 112 unpooling layer 112 × 112 deconvolutional layer 224 × 224 unpooling layer 224 × 224 deconvolutional layer
11/30/2021 CAP5415 - Lecture 17 21
Image source: H. Noh, S. Hong, and B. Han, Learning Deconvolution Network for Semantic Segmentation, ICCV 2015
Learned upsampling architectures

11/30/2021 CAP5415 - Lecture 17 22


Figure source
SegNet

Uses VGG architecture!


Image source: http://mi.eng.cam.ac.uk/projects/segnet/
No FC layer!
11/30/2021 CAP5415 - Lecture 17 23
V Badrinarayanan, et al., A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling, 2015
U-Net

11/30/2021 CAP5415 - Lecture 17 24


Source: Olaf Ronneberger, Philipp Fischer, Thomas Brox “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI, 2015
Questions?

Sources for this lecture include materials from works by Sedat Ozer, Ulas
Bagci, and Svetlana Lazebnik

11/30/2021 CAP5415 - Lecture 17 25

You might also like