ELB1502

Study Guide

ELECTRICAL ENGINEERING ROBOTICS

Unit 5: Introduction to Robot Vision

Diploma in Electrical Engineering

In the Department of Electrical Engineering

School of Engineering

College of Science, Engineering & Technology (CSET)

University of South Africa

Compiled by: Dr. E.M. Migabo (PhD Computer Science & DEng Electrical Engineering)

Instructors: Dr. M.E. Migabo & Mr. A.M. Dlamini

May, 2023
 ELB1502
Study Guide

I. Learning objectives

The learning objectives for this study unit are:

a. To define what is computer vision.

b. To understand human vision system and machine
c. To understand images as matrices
d. To understand the camera model
e. To understand robotic applications of machine vision
II. Unit summary
The following set of slides summarize very well the content of the study unit in terms of learning
objectives a. to d.:

2
 STUDY UNIT 4

Introduction to Computer
Vision for Robotics
Compiled by: Dr. E.M. Migabo (PhD)
Introduction to Electrical Robotics
ELB1502

Introduction to Computer Vision 1

8QLW Outline
● Introduction
○ What is CV?
○ Overview of the field
○ A look at history
○ Hard Problem?
● Human Vision System & the Machine
○ The human vision system
○ Fooling humans
○ The computer vision system
● Images as matrices.
○ How cameras work to produce these matrices
○ Meaning of Intensity, Color etc
○ Shoutout to Image Processing

Introduction to Computer Vision 2

8QLW Outline
● Camera Model
○ Pinhole Camera Model
○ Intrinsic Camera Matrix
○ Camera Calibration

Introduction to Computer Vision 3

Introduction

Introduction to Computer Vision 4

 What is Computer Vision?
Universe

Computer
Image Vision Information
System
Image
Image
Processing

Introduction to Computer Vision 5

What is Computer Vision?

Image Credits: CS131, Fall ‘18, Stanford

Introduction to Computer Vision 6

What is Computer Vision?
● Computer Vision is deals with extracting information regarding the 3D world
we live in using a single or a bunch of images.
● Computer Vision like most other fields today, is at the junction of numerous
disciplines from Biology to Computer Science and has applications only
limited by our imagination.

Introduction to Computer Vision 7

Overview of the field

Image Credits: XKCD, 1425, 2014

Introduction to Computer Vision 8

Overview of the field

Image Credits: https://tinyurl.com/y53by9pr Image Credits: XKCD, 1425, 2014

Image Credits: https://tinyurl.com/y53by9pr

Introduction to Computer Vision 9

 Overview of the field
What kind of Information?
Universe

(Image
Processing + )
Image
Computer Vision
System

Introduction to Computer Vision 10

Overview of the field
What kind of Information?

Image Credits: Karpathy, CVPR’15

Image Credits: https://tinyurl.com/lxuex6o

Introduction to Computer Vision 11

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection

Recognition: Cat?
Image: https://tinyurl.com/yanp2o5e

Introduction to Computer Vision 12

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection

Recognition: Cat?
Localization: Where is the cat?
Image: https://tinyurl.com/yanp2o5e

Introduction to Computer Vision 13

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection

Object Detection: Which Objects

are here and where?
Image: https://tinyurl.com/y4ly96rd

Introduction to Computer Vision 14

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation

Segmentation: Which pixels

belong to which object?
Credits: Own Work

Introduction to Computer Vision 15

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements

Image Colorization: From

Grayscale to Colored Images
Credits: Richard Zhang, CVPR 2016

Introduction to Computer Vision 16

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements

Image Enhancement: Real Time

Image Enhancement
Credits: Michael Gharbi, ACM Graphics 2017

Introduction to Computer Vision 17

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements

Super Resolution: Upsampling

Images while preserving quality
Credits: https://github.com/tensorlayer/srgan

Introduction to Computer Vision 18

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements
4. Image to Text

Image Description: Automatic

semantic description for images
Credits: Karpathy, CVPR 2015

Introduction to Computer Vision 19

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements
4. Image to Text
5. Image Generation
Image Generation: A style based
generator architecture for GANs
Credits: Tero Karras, arXiv 2018

Introduction to Computer Vision 20

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements
4. Image to Text
5. Image Generation
Optical Flow: Lucas Kanade
6. Motion Estimation method for motion estimation
Credits: https://tinyurl.com/y5rloh3g

Introduction to Computer Vision 21

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements
4. Image to Text
5. Image Generation
3D Reconstruction: REMODE,
6. Motion Estimation Real Time Reconstruction
Credits: Matia Pizzoli, ICRA 2014
7. 3D reconstruction from Images

Introduction to Computer Vision 22

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements
4. Image to Text
5. Image Generation
3D Reconstruction: REMODE,
6. Motion Estimation Real Time Reconstruction
Credits: Matia Pizzoli, ICRA 2014
7. 3D reconstruction from Images
8. Visual SLAM

Introduction to Computer Vision 23

Overview of the field
Primary themes in Computer Vision are:

1. Object Detection
2. Segmentation
3. Image Modifications/Enhancements
4. Image to Text
5. Image Generation
Biometrics : Fingerprint
6. Motion Estimation Detection, Apple Face ID
7. 3D reconstruction from Images Credits:https://tinyurl.com/y2a7wybz,
TheVerge Youtube
8. Visual SLAM
9. Biometrics and more ...

Introduction to Computer Vision 24

A look at history
● Robert Nathan started writing
computer programs for enhancing
Credits: EE604, nasa.gov
images from NASA’s spacecraft’s at
Jet Propulsion Lab, NASA.
● The Summer Vision Project: Project at
MIT to solve a significant part of
visual system. Primary Objective was
to divide the image into object,
background and chaos regions, over
the course of a summer.
Credits: https://tinyurl.com/y6bpo4nk

Introduction to Computer Vision 25

A look at history

Credits: Prof. Tanaya Guha, EE698K

Introduction to Computer Vision 26

A look at history

Credits: Prof. Tanaya Guha, EE698K

Introduction to Computer Vision 27

Hard Problem?
● Why are we still working on roughly the same problem as the “summer vision
project”?
● Why is it that creating 3D models of chairs is easier than identifying them?

Introduction to Computer Vision 28

Hard Problem?
● Why are we still working on roughly the same problem as the “summer vision
project”?
● Why is it that creating 3D models of chairs is easier than identifying them?

➔ There is a large between some ~1920x1080x3 numbers and the high-level

abstract meaning we associate with them.
➔ Images are 2D representation of information from 3D world.

Introduction to Computer Vision 29

Human Vision System & Computer Vision System

Introduction to Computer Vision 30

The human vision system

Credits: https://tinyurl.com/y6bkhnqa

Introduction to Computer Vision 31

The human vision system

Introduction to Computer Vision 32

The human vision system

Credits: Ulas Bagci, UCF

Introduction to Computer Vision 33 Harsh Sinha

Fooling humans

Credits: https://tinyurl.com/y49rp7sd
Credits: Wikipedia, Spinning Dancer
Credits: Oleg Shuplyak, Pinterest
Introduction to Computer Vision 34
The (human) (computer) vision system

Credits: CS131, Stanford

Introduction to Computer Vision 35

Fooling Computers

Credits: https://tinyurl.com/l5pwp6t Credits: Wikipedia, Barber Pole Illusion

Introduction to Computer Vision 36

Images as Matrices

Introduction to Computer Vision 37

Camera Models

Introduction to Computer Vision 38

Camera Models

Credits: https://tinyurl.com/y6qen2vb

Introduction to Computer Vision 39

Camera Models
Not this one but models as
in modelling a phenomena

Credits: https://tinyurl.com/y6qen2vb

Introduction to Computer Vision 40

Camera Models
● Like so many things in engineering, we create a simple “model” of a camera
to which is easy to understand and can approximate the actual functioning of
a camera to a good degree.
● There are different models:
■ Pinhole camera model
■ Lens model
■ ...

Introduction to Computer Vision 41

Pinhole camera model

aperture

Credits: Wikipedia, Pinhole Camera Model

Introduction to Computer Vision 42

Pinhole camera model

aperture

Credits: Wikipedia, Pinhole Camera Model

Introduction to Computer Vision 43

Pinhole camera model

aperture

Credits: Wikipedia, Pinhole Camera Model

Introduction to Computer Vision 44

Pinhole camera model
where x’i = yi and z = x3

where c is an offset in pixels

Can we make this into a matrix multiplication of the

form p’ = Mp?

Introduction to Computer Vision 45

Intrinsic camera matrix

Credits: Edwin Olson, University of Michigan

Introduction to Computer Vision 46

Intrinsic camera matrix

Credits: Edwin Olson, University of Michigan

Introduction to Computer Vision 47

Intrinsic camera matrix

Credits: Edwin Olson, University of Michigan

Introduction to Computer Vision 48

Intrinsic camera matrix

Credits: Edwin Olson, University of Michigan

Introduction to Computer Vision 49

Intrinsic camera matrix

Credits: Edwin Olson, University of Michigan

Introduction to Computer Vision 50

Camera calibration

Pi
Calibration Rig Image Pci

Credits: Gaurav Pandey, Ford

Introduction to Computer Vision 51

 ELB1502
Study Guide

Summary notes and Computer Vision Applications to Robotics

Machine vision is concerned with the sensing of vision data and its interpretation by a computer.
The typical vision system consists of the camera and digitizing hardware, a digital computer and
hardware and software necessary to interface them. This interface hardware and software is often
referred to as a pre-processor. The operation of the vision system consists of three functions:

1. Sensing and digitizing image data

The sensing and digitizing functions involve the input of vision data by means of a camera
focused on the scene of interest. Special lighting techniques are frequently used to obtain an
image of sufficient contrast for later processing. The image viewed by the camera is typically
digitized and stored in computer memory.

The digital image is called frame of vision data and is frequently captured by a hardware device
called as frame grabber. These devices are capable of digitizing images at the rate of 30 frames
per second.

2. Image processing and analysis

The digitized image matrix for each frame is stored and then subjected to image processing and
analysis functions for data reduction and interpretation of the image. These steps are required to
permit the real-time application of vision analysis required in robot applications.

Typically an image frame will be threshold to produce a binary image and then various feature
measurements will further reduce the data representation of the image. This data reduction can
change the representation of a frame from several hundred thousand bytes of raw image data to
several hundred bytes of feature value data. The resultant feature data can be analysed in the
available time for action by the robot system.

3. Application

The third function of a machine vision system is the applications function. The current
applications of machine vision in robotics include inspection part identification, location and
orientation.

The relationship between three function is shown in figure 1

3
 ELB1502
Study Guide

Figure 1: Relationship between the three functions

4. Sensing and digitizing function in machine vision

Image sensing requires some type of image formation device such as a camera and a digitizer
which stores a video frame in the computer memory. We divide the sensing and digitizing
functions into several steps.

The initial step involves capturing the image of the scene with the vision camera. The image
consists of relative light intensities corresponding to the various portions of the scene. These light
intensities are continuous analogue values which must be sampled and converted into a digital
form.

The second step, digitizing, is achieved by an analogue to digital converter. The A/D converter is
either part of a digital video camera or the front end of a frame grabber. The choice is dependent
on the type of hardware in the system.

The frame grabber representing the third step, is an image storage and computational device which
stores a given pixel array. The frame grabber can vary in capability from one which simply stores
an image to significant computation capability.

In the more powerful frame grabbers, thresholding, windowing, and histogram modification
calculations can be carried out under computer control. The stored image is then subsequently
processed and analysed by the combination of the frame grabber and the vision controller.

4
 ELB1502
Study Guide

4.1.Robotic applications

Robotic application of machine vision falls into three broad categories listed below:

· Inspection

The first category is one in which the primary function is the inspection process. This is carried
out by the machine vision system, and the robot us used in a secondary to support the application.

The objectives of machine vision inspection include checking for gross surface defects, discovery
of flaws in labelling, verification of the presence of components in assembly and checking for the
presence of holes and other features in a part.

When these kinds of inspection operations are performed manually, there is a tendency for human
error. Also, the time required in most manual inspection operations require that the procedures are
carried out automatically using 100 percent inspection and usually in much less time.

· Identification

This is concerned with applications in which the purpose of the machine vision system is to
recognise and classify an object rather than to inspect it. Inspection implies that the part must be
either accepted or rejected. Identification involves a recognition process in which the “part itself,
or its position and/or orientation, is determined.

This is usually followed by subsequent decision and action taken by the robot. Identification
applications of machine vision include part sorting, palletizing and depalletizing and picking parts
that are randomly oriented from a conveyer or bin.

· Visual serving and navigation

In the third category, visual serving and navigation control, the purpose of the vision system is to
direct the actions of the robot based on its visual input.

The generic example of robot visual serving is where the machine vision system is used to control
the trajectory of the robot’s end effector toward an object in the workspace. Industrial examples
of this application include part positioning, retrieving parts moving along a conveyor, retrieving,
and reorienting parts moving along a conveyor, assembly etc.

5
 ELB1502
Study Guide

III. Tutorials
1. Q: What is computer vision? A: Computer vision is the field of study that focuses on
enabling computers to interpret and understand visual information from digital images
or videos.
2. Q: How does human vision differ from machine vision? A: Human vision is a complex
process involving the eyes, brain, and perception, while machine vision refers to the
use of computer algorithms and techniques to extract information from images or
videos.
3. Q: How are images represented in computer vision? A: In computer vision, images are
represented as matrices or grids of pixels, where each pixel stores numerical values
representing the color or intensity of the corresponding image location.
4. Q: What are the components of the camera model in computer vision? A: The camera
model includes intrinsic parameters (focal length, principal point) and extrinsic
parameters (position and orientation) that describe the relationship between the 3D
world and 2D image coordinates.
5. Q: What are some robotic applications of machine vision? A: Robotic applications of
machine vision include object recognition and localization, robot navigation, industrial
automation, surveillance, autonomous vehicles, and augmented reality.
6. Q: Define computer vision. A: Computer vision is an interdisciplinary field that focuses
on developing algorithms and techniques for machines to extract, analyze, and interpret
information from digital images or videos.
7. Q: How does the human vision system work? A: The human vision system involves
the eyes capturing light, which is then processed by the brain to form visual perception,
including recognition, depth perception, and object tracking.
8. Q: Explain images as matrices in computer vision. A: In computer vision, images are
represented as matrices, where each element in the matrix represents a pixel value that
encodes color or intensity information.
9. Q: What is the camera model in computer vision? A: The camera model describes the
mathematical relationship between 3D points in the world and their projection onto a
2D image plane. It includes intrinsic and extrinsic parameters.
10. Q: Provide examples of robotic applications that utilize machine vision. A: Examples
include industrial robots for quality control, autonomous vehicles for road scene
understanding, surgical robots for precise image-guided procedures, and drones for
object tracking.
11. Q: How would you define computer vision in the context of robotics? A: In robotics,
computer vision refers to the application of image processing and analysis techniques
to enable robots to perceive and interpret visual information from the environment.
12. Q: What are the primary stages of human vision processing? A: Human vision
processing involves image formation on the retina, feature extraction in the visual
cortex, and higher-level interpretation in the brain for object recognition and
understanding.
13. Q: How can images as matrices be manipulated in computer vision? A: Matrices
representing images can be processed using various techniques, such as filtering, edge
detection, morphological operations, and transformations like rotation or scaling.
14. Q: Explain the concept of intrinsic parameters in the camera model. A: Intrinsic
parameters describe the internal characteristics of the camera, such as focal length,
principal point, and lens distortion, which affect the mapping of 3D points to the image
plane.

6
 ELB1502
Study Guide

15. Q: What are some examples of robotic applications that utilize machine vision for
object recognition? A: Examples include industrial robots identifying and sorting
objects on an assembly line, autonomous drones detecting and avoiding obstacles, and
robots in healthcare assisting in surgical procedures.
16. Q: How does the machine vision process contribute to robot navigation? A: Machine
vision allows robots to perceive and understand the environment by analyzing visual
information, which aids in tasks such as obstacle detection, mapping, and localization.
17. Q: Explain the concept of extrinsic parameters in the camera model. A: Extrinsic
parameters define the position and orientation of the camera in the 3D world coordinate
system, enabling the transformation from 3D world points to the 2D image plane.
18. Q: What are some challenges in robotic applications of machine vision? A: Challenges
include handling variations in lighting conditions, occlusions, complex scenes, real-
time processing requirements, and robustness to noise and uncertainties.
19. Q: How does machine vision contribute to industrial automation? A: Machine vision
systems are used in industrial automation for tasks such as quality control, defect
detection, object sorting, robotic assembly, and visual inspection.
20. Q: What is the role of computer vision in robot navigation? A: Computer vision enables
robots to perceive and interpret the environment, allowing them to understand
obstacles, landmarks, and spatial relationships. This information is crucial for tasks
such as mapping, localization, path planning, and obstacle avoidance. By analyzing
visual data from cameras or other sensors, robots can make informed decisions to
navigate their surroundings safely and efficiently. Computer vision provides valuable
input for autonomous navigation systems, enabling robots to adapt to dynamic
environments and handle complex scenarios.

IV. Exercises and problems:

1. Q: Convert a color image with dimensions 640x480 pixels into a grayscale image.
Calculate the resulting image size.
A: The resulting image size will be 640x480 pixels since a grayscale image has only
one channel.
2. Q: Given an image represented as a 3x3 matrix, perform element-wise multiplication
by a scalar value of 2.
A: If the original image matrix is
1 2 3
[4 5 6]
7 8 9
, the resulting image matrix will be
2 4 6
[ 8 10 12]
14 16 18

3. Q: Calculate the total number of pixels in a grayscale image with dimensions 800x600
pixels.
A: The total number of pixels will be 800 * 600 = 480,000 pixels.
4. Q: Given a camera with a focal length of 50 mm and an object distance of 2 meters,
1 1 1
calculate the image distance using the camera model equation = + )
𝑓 𝑑0 𝑑𝑖
A: Using the camera model equation, the image distance (𝑑𝑖 ) will be 0.040 meters or
40 millimeters.

7
 ELB1502
Study Guide

5. Q: Determine the intrinsic matrix K given the camera's focal length of 500 pixels and
principal point coordinates (320, 240).
A: The intrinsic matrix K will be:
500 0 320
[ 0 500 240]
0 0 1

6. Q: Calculate the aspect ratio of an image with dimensions 1024x768 pixels.

A: The aspect ratio is calculated by dividing the width by the height, resulting in
1024/768 ≈ 1.3333.
7. Q: Given an RGB image with dimensions 640x480 pixels, calculate the total number
of color channels.
A: RGB images have three color channels (Red, Green, and Blue). So, the total number
of color channels will be 3.
8. Q: Determine the field of view (FOV) of a camera with a focal length of 35 mm and
an image sensor size of 22.3 mm x 14.9 mm.
A: The horizontal FOV can be calculated using the formula:
𝑠𝑒𝑛𝑠𝑜𝑟𝑤𝑖𝑑𝑡ℎ
𝐹𝑂𝑉 = 2 × tan−1 ( )
2 × 𝑓𝑜𝑐𝑎𝑙𝑙𝑒𝑛𝑔𝑡ℎ
Substituting the values, the horizontal FOV will be approximately 37.1 degrees.
9. Q: Given a camera with a pixel size of 5 μm and a resolution of 2048x1536 pixels,
calculate the physical size of the image sensor.
A: The physical size of the image sensor can be calculated by multiplying the pixel
size by the resolution. In this case, it will be 5 μm * 2048 x 5 μm * 1536 ≈ 10.24 mm
x 7.68 mm.
10. Q: Calculate the Euclidean distance between two points A(3, 4) and B(7, 1) in an
image.
A: The Euclidean distance can be calculated using the formula:
√(𝑥2 − 𝑥1 )2 + (𝑦2 − 𝑦1 )2
Substituting the values, the Euclidean distance between points A and B will be
√(7 − 3)2 + (1 − 4)2 = √(16 + 9) = √25 = 5 𝑢𝑛𝑖𝑡𝑠
11. Problem: A robot is equipped with a camera that captures images at a resolution of
800x600 pixels. Each pixel represents a 0.1 cm x 0.1 cm area in the real world. The
robot needs to determine the size of an object in the image. If the object occupies 200
pixels in width, what is its size in centimeters?
Solution: The size of the object in centimeters can be calculated by multiplying the
number of pixels by the pixel size. In this case, the object size is 200 pixels * 0.1
cm/pixel = 20 cm.
12. Problem: A robot is using machine vision to detect defects on manufactured parts. The
camera captures images at a rate of 30 frames per second. If each image processing
operation takes 20 milliseconds to complete, what is the maximum number of parts the
robot can inspect per minute?
Solution: The time taken to process one image is 20 milliseconds. Therefore, the robot
can process 1000 milliseconds / 20 milliseconds = 50 images per second. Multiplying
this by 60 seconds gives a maximum inspection rate of 50 images/second * 60 seconds
= 3000 parts per minute.
13. Problem: A robot is performing object recognition using machine vision. The camera
captures images at a resolution of 1280x960 pixels. The robot's algorithm requires the
images to be converted to grayscale. If each grayscale conversion operation takes 5

8
 ELB1502
Study Guide

milliseconds, what is the total processing time for a video sequence of 100 frames?
Solution: The time taken to convert one frame to grayscale is 5 milliseconds.
Therefore, the total processing time for 100 frames will be 100 frames * 5 milliseconds
= 500 milliseconds or 0.5 seconds.
14. Problem: A robot is equipped with a depth-sensing camera that measures the distance
of objects in a scene. The camera has a depth resolution of 1 millimeter. If the robot
detects an object at a distance of 5 meters, what is the depth measurement accuracy in
centimeters?
Solution: The depth measurement accuracy is equal to the depth resolution, which is 1
millimeter. Converting this to centimeters gives an accuracy of 0.1 centimeters.
15. Problem: A robot is using machine vision to navigate through a maze. The camera
captures images at a resolution of 640x480 pixels. The robot's algorithm requires the
images to be resized to a resolution of 320x240 pixels. If each image resizing operation
takes 10 milliseconds, what is the processing time for a video sequence of 50 frames?
Solution: The time taken to resize one frame is 10 milliseconds. Therefore, the total
processing time for 50 frames will be 50 frames * 10 milliseconds = 500 milliseconds
or 0.5 seconds.

V. References

[1] J. J. Craig, Introduction to Robotics, Global Edition, 3rd ed. Harlow, England: Pearson
Education Limited, 2014