IMAGE PROCESSING

AND
COMPUTER VISION
Introduction

Dr. Sk. Md. Masudul Ahsan

Professor, Dept of CSE, KUET
What is an image?
 We can think of an image as a function, f, from R2 to R:
 f( x, y ) gives the intensity at position ( x, y )
 Realistically, we expect the image only to be defined over a rectangle, with a finite range:
 f: [a,b]x[c,d]  [0,1]

 A color image is just three functions pasted together. We can write this as a
“vector-valued” function:

 r ( x, y ) 
f ( x, y )   g ( x, y ) 
 

 b ( x, y ) 

Digital Image
 A digitized image is one in which:
 Spatial and grayscale values have been made discrete.
 Intensities measured across a regularly spaced grid in x and y directions are
sampled to
8 bits (256 values) per point for black and white,
 3x8 bits per point for color images.

 Stored as a 2D arrays of gray-level values. The array elements are called pixels and
identified by their x, y coordinates.
What is a Digital Image? (cont…)
 Common image formats include:
 1 sample per point (B&W or Grayscale)
 3 samples per point (Red, Green, and Blue)
 4 samples per point (Red, Green, Blue, and “Alpha”, a.k.a. Opacity)

 For most of this course we will focus on grey-scale images

Visualizing Image
Images as Surfaces
What is Digital Image Processing?
 Digital image processing focuses on two major tasks
 Improvement of pictorial information for human interpretation
 Processing of image data for storage, transmission and representation for autonomous
machine perception
 Some argument about where image processing ends and fields such as
image analysis and computer vision start
Image Processing
Hubble telescope –
image restoration
example:
 A defective mirror
made many of Hubble’s
images useless.

 Image restoration
techniques were
used to improve image
quality before fixing the
problem.
Image Processing
 Image Compression
Computer Vision
 Make computers understand images and video.
 Computing properties of the 3D world from visual data (measurement)
 Algorithms and representations to allow a machine to recognize objects, people, scenes, and
activities. (perception and interpretation)

What kind of scene?

Where are the cars?

How far is the building?

…
What is Computer Vision?
 Computer vision is the science and technology of machines that see.

 Concerned with the theory for building artificial systems that obtain information from images.

 The image data can take many forms, such as a video sequence, depth images, views from
multiple cameras, or multi-dimensional data from a medical scanner
DIP to CV
 The continuum from image processing to computer vision can be broken up
into low-, mid- and high-level processes

Low Level Process Mid Level Process High Level Process

Input: Image Input: Image Input: Attributes
Output: Image Output: Attributes Output: Understanding
Examples: Noise Examples: Object Examples: Scene
removal, image recognition, understanding,
sharpening segmentation autonomous navigation

In this course we will

stop here
 Why Computer Vision is Hard?

19
Vision is really hard
Vision is Challenging

 Inverse problems
 Apriori-knowledge is required
 Complexity is extensive
 Non-local operations

21
 Vision is really hard

24
Related disciplines

Artificial
intelligence
Machine
Graphics learning
Computer
Image vision Cognitive
processing science
Algorithms
Vision and graphics

graphics

vision

Inverse problems: analysis and synthesis.

What is computer vision? (2D->3D)

3D geometry

Estimation
physics
Why vision matters?
 Images and
video are
everywhere!

Personal photo albums Movies, news, sports

Surveillance and security Medical and scientific images

Slide credit; L. Lazebnik
Image Processing and
Computer Vision Applications
 Visual inspection/quality control
 Surveillance and security
 Autonomous vehicles
 Space applications
 Medical imaging
 Digital photography and 3D modeling
 Games and much more …...
Image Processing and
Computer Vision Applications

Safety Health Security

Comfort Fun Access

Industry and Applications
 Automobile driver assistance
 Lane departure warning
 Adaptive cruise control
 Obstacle warning
 Digital Photography
 Image Enhancement
 Compression
 Color manipulation MobilEye system
 Image editing
 Digital cameras
 Sports analysis
 sports refereeing and commentary
 3D visualization and tracking sports actions

46
Industry and Applications
 Film and Video
 Editing
 Special effects
 Image Database
 Content based image retrieval
 visual search of products
 Face recognition
 Industrial Automation and Inspection
 vision-guided robotics
 Inspection systems
 Medical and Biomedical
 Surgical assistance
 Sensor fusion
 Vision based diagnosis
 Astronomy
 Astronomical Image Enhancement

47 Chemical/Spectral Analysis
Industry and Applications
 Arial Photography
 Image Enhancement
 Missile Guidance
 Geological Mapping
 Robotics
 Autonomous Vehicles
 Security and Safety
 Biometry verification (face, iris)
 Surveillance (fences, swimming pools)
 Military
 Tracking and localizing
 Detection
 Missile guidance
 Traffic and Road Monitoring
 Traffic monitoring Cruise Missiles
 Adaptive traffic lights
48
Key Processes in Image Analysis

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Image Acquisition

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Image Enhancement

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Image Restoration

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Processing

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Segmentation

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Representation & Description

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Object Recognition

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
Image Compression

Image
Processing
Restoration

Image Segmentation
Enhancement

Image Feature
Acquisition Extraction

Object
Recognition
Problem Domain
Image Compression
VISION CHALLENGES
viewpoint variation

Michelangelo 1475-1564
Illumination
Illumination
Scale
Deformation
Occlusion
Background Clutter
Background Clutter
Object intra-class variation
Local ambiguity
Challenges or opportunities?
 Images are confusing, but they also reveal the structure of the world
through numerous cues
 Our job is to interpret the cues!

Image source: J. Koenderink

Bottom line
 Perception is an inherently ambiguous problem
 Many different 3D scenes could have given rise to a particular 2D picture

 Possible solutions
 Bring in more constraints ( or more images)
 Use prior knowledge about the structure of the world
 Need both exact measurements and statistical inference!
Some more Applications of IP and CV
Image Enhancement

Contrast stretching Deblurring

74
Image Enhancement

Denoising Inpainting
Texture Synthesis

80
Industry and Applications
 Image Demosaicing  Face detection
 Almost all digital cameras now detect
faces

82
Optical character recognition (OCR)
Technology to convert scanned docs to text
• If you have a scanner, it probably came with OCR software

Digit recognition, AT&T labs License plate readers

http://www.research.att.com/~yann/ http://en.wikipedia.org/wiki/Automatic_number_plate_recognition
Machine vision

Automated visual inspection

Face detection
 Many new digital cameras now detect faces
 Canon, Sony, Fuji, …

Age recognition

Smile recognition
Smile detection?

Sony Cyber-shot® T70 Digital Still Camera

Face makeovers
Industry and Applications
 Face recognition

“How the Afghan Girl was Identified by Her Iris Patterns” Read the story

Who is she?
Facial Expression Recognition

http://www.youtube.com/watch?v=M1WgnisIyPQ&feature=related
Earth Viewers (3D Modeling)

Image from Microsoft’s Virtual Earth

(see also: Google Earth)
Login without a password…

Face recognition systems now

Fingerprint scanners on
beginning to appear more widely
many new laptops, http://www.sensiblevision.com/
other devices
Authentication Using Biometrics
Industry and Applications
 Object recognition -> This is becoming real:
 Microsoft Research
 Point & Find, Nokia
Object Recognition

Toshiba Tech IS-910T 2013

DataLogic LaneHawk LH4000 2012

Special effects: shape capture

The Matrix movies, ESC Entertainment, XYZRGB, NRC

Special effects: motion capture

Pirates of the Carribean, Industrial Light and Magic

Click here for interactive demo
Automatic Panorama Stitching
3D Reconstruction from
internet photo collections

“Statue of Liberty” “Half Dome, Yosemite” “Colosseum, Rome”

Flickr photos

3D model

see “Building Rome in

a day” project at U. Washington http://grail.cs.washington.edu/rome/
3D from thousands of images

Building Rome in a Day: Agarwal et al. 2009

Sports

Sportvision first down line

Nice explanation on www.howstuffworks.com
Smart cars Slide content courtesy of Amnon Shashua

 Mobileye
 Vision systems currently in high-end BMW, GM, Volvo models
 By 2010: 70% of car manufacturers.
 Video demo
 BMW 5 series

BMW night vision

Games and Assistive Technologies

Kinect

Assistive technologies
Nintendo Wii has camera-based IR
tracking built in. See Lee’s work atCMU on clever tricks on using it to
create a multi-touch display!
Virtual Fitting

2015
Interactive Games: Kinect
 Object Recognition: http://www.youtube.com/watch?feature=iv&v=fQ59dXOo63o
 Mario: http://www.youtube.com/watch?v=8CTJL5lUjHg
 3D: http://www.youtube.com/watch?v=7QrnwoO1-8A
 Robot: http://www.youtube.com/watch?v=w8BmgtMKFbY
Vision in space

NASA'S Mars Exploration Rover Spirit captured this westward view from atop
a low plateau where Spirit spent the closing months of 2007.

Vision systems (JPL) used for several tasks

• Panorama stitching
• 3D terrain modeling
• Obstacle detection, position tracking
• For more, read “Computer Vision on Mars” by Matthies et al.
Vision in Space

NASA'S Mars Exploration Rover Spirit

• Vision systems used for several tasks

– Obstacle detection
– Position tracking
– 3D terrain modeling

For more info, read “Computer Vision on Mars” by Matthies et al.

International Journal of Computer Vision, 2007.
Crater Detection

Verification

Multi-scale
Edge Detection
Convex Grouping Hypotheses

Line fitting

Ebrahim Emami, Touqeer Ahmad, George Bebis, Ara Nefian, and Terry Fong, "Crater
Detection Using Unsupervised Algorithms and Convolutional Neural Networks",
IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 8, 2019.
Robotics

NASA’s Mars Spirit Rover http://www.robocup.org/

http://en.wikipedia.org/wiki/Spirit_rover
Medical imaging

Image guided surgery

3D imaging
Grimson et al., MIT
MRI, CT
Image Operations
 Geometric Operations
 Point Operations
 Spatial Operations
 Global Operations (Freq. domain)
 Multi-Resolution Operations

125
Geometric Operations

126
Point Operations

127
Geometric and Point Operations

128
Spatial Operations

129
Global Operations

130
Global Operations

Image domain

Freq. domain

131
Multi-Resolution

Low resolution

High resolution

133
What skills you need ?
 Strong programming skills (i.e., C, C++, Python, Matlab)
 Good knowledge of Data Structures and Algorithms
 Good skills in analyzing algorithm performance (i.e., time and memory requirements).
 Strong background in mathematics, especially in:
 Linear Algebra
 Probabilities and Statistics
 Numerical Analysis
 Geometry
 Calculus
Textbook
 Digital Image Processing
Rafael C. Gonzalez & Richard E. Woods,

 Digital Image Processing

S Esakkirajan T Veerakumar , S Jayaraman
https://books.google.co.in/books?id=JeDGn6Wmf1kC&printsec=frontcover&source=gbs_ge_summa
ry_r&cad=0#v=onepage&q&f=false

 Digital Image Processing An Algorithmic Introduction Using

Java
Wilhelm Burger, Mark J. Burge
Secondary Text
• Concise Computer Vision
Reinhard Klette
• Fundamentals of Digital Image Processing
Chris Solomon, Toby Breckon

http://szeliski.org/Book/