Fr

This book makes use of OpenCV 3 to tackle increasingly

challenging computer vision problems. You begin by
taking a critical look at inputs, in order to decide which
types of light, lenses, and image formats are best suited
for a given purpose. You then consider the finer aspects
of computational photography as you build an automated
camera to assist nature photographers. You will gain a deep
understanding of some of the most applicable techniques
in object detection, feature selection, and even biometric
recognition. You will also build Android projects in which you
explore the complexities of camera motion.
By the end of this book, you will have a much richer
understanding of imaging, motion, machine learning, and
the architecture of computer vision libraries and applications!

What you will learn from this book

Select and configure camera systems to see
invisible light, fast motion, and distant objects
Build a "camera trap", as used by nature
photographers, and process photos to
create beautiful effects
Develop a facial expression recognition
system with various feature extraction
techniques and machine learning methods

Steven Puttemans

P U B L I S H I N G

Joseph Howse

Fuse data from videos and gyroscopes

to stabilize videos shot from your mobile
phone and create hyperlapse videos

Utkarsh Sinha

Create a person identification and registration

system based on the biometric properties
of that person, such as their fingerprint, iris,
and face

$ 44.99 US
28.99 UK

community experience distilled

pl

C o m m u n i t y
Quan Hua

This book is ideal for you if you aspire to build computer

vision systems that are smarter, faster, and more practical
than the competition. This is an advanced book, intended
for those who already have some experience in setting
up an OpenCV development environment and building
applications with OpenCV.

Sa
m

Build an Android panorama application using

the OpenCV stitching module in C++, with
NDK support
Optimize your object detection model, make
it rotation invariant, and apply scene-specific
constraints to make it faster and more robust

Who this book is written for

OpenCV 3 Blueprints

OpenCV 3 Blueprints

ee

E x p e r i e n c e

OpenCV 3 Blueprints
Expand your knowledge of computer vision by building amazing
projects with OpenCV 3

Prices do not include

local sales tax or VAT
where applicable

Visit www.PacktPub.com for books, eBooks,

code, downloads, and PacktLib.

D i s t i l l e d

Joseph Howse

Steven Puttemans

Quan Hua

Utkarsh Sinha

In this package, you will find:

The authors biography

A preview chapter from the book, Chapter 1 'Getting the Most out of
Your Camera System'
A synopsis of the books content
More information on OpenCV 3 Blueprints

About the Authors

Joseph Howse lives in Canada. During the cold winters, he grows a beard

and his four cats grow thick coats of fur. He combs the cats every day. Sometimes,
the cats pull his beard.
Joseph has been writing for Packt Publishing since 2012. His books include
OpenCV for Secret Agents, OpenCV 3 Blueprints, Android Application Programming
with OpenCV 3, Learning OpenCV 3 Computer Vision with Python, and Python Game
Programming by Example.
When he is not writing books or grooming cats, Joseph provides consulting,
training, and software development services through his company, Nummist
Media (http://nummist.com).

Steven Puttemans is a PhD research candidate at the KU Leuven, Department

of Industrial Engineering Sciences. At this university, he is working for the
EAVISE research group, which focuses on combining computer vision and
artificial intelligence. He obtained a master of science degree in Electronics-ICT
and further expanded his interest in computer vision by obtaining a master of
science in artificial intelligence.
As an enthusiastic researcher, his goal is to combine state-of-the-art computer
vision algorithms with real-life industrial problems to provide robust and complete
object detection solutions for the industry. His previous projects include TOBCAT,
an open source object detection framework for industrial object detection problems,
and a variety of smaller computer vision-based industrial projects. During these
projects, Steven worked closely with the industry.
Steven is also an active participant in the OpenCV community. He is a moderator
of the OpenCV Q&A forum, and has submitted or reviewed many bug fixes and
improvements for OpenCV 3. He also focuses on putting as much of his research
as possible back into the framework to further support the open source spirit.
More info about Steven's research, projects, and interests can be found at
https://stevenputtemans.github.io.

Quan Hua is a software engineer at Autonomous, a start-up company in robotics,

where he focuses on developing computer vision and machine learning applications
for personal robots. He earned a bachelor of science degree from the University of
Science, Vietnam, specializing in computer vision, and published a research paper
in CISIM 2014. As the owner of Quan404.com, he also blogs about various computer
vision techniques to share his experience with the community.

Utkarsh Sinha lives in Pittsburgh, Pennsylvania, where he is pursuing a master's

in computer vision at Carnegie Mellon University. He intends to learn the state of
the art of computer vision at the university and work on real-life industrial scale
computer vision challenges.

Before joining CMU, he worked as a Technical Director at Dreamworks Animation

on movies such as Home, How to Train your Dragon 2, and Madagascar 3. His
work spans multiple movies and multiple generations of graphics technology.
He earned his bachelor of engineering degree in computer science and his master
of science degree in mathematics from BITS-Pilani, Goa. He has been working in the
field of computer vision for about 6 years as a consultant and as a software engineer
at start-ups.
He blogs at http://utkarshsinha.com/ about various topics in technologymost
of which revolve around computer vision. He also publishes computer vision tutorials
on the Internet through his website, AI Shack (http://aishack.in/). His articles help
people understand concepts in computer vision every day.

Preface
Open source computer vision projects, such as OpenCV 3, enable all kinds of users
to harness the forces of machine vision, machine learning, and artificial intelligence.
By mastering these powerful libraries of code and knowledge, professionals and
hobbyists can create smarter, better applications wherever they are needed.
This is exactly where this book is focused, guiding you through a set of hands-on
projects and templates, which will teach you to combine fantastic techniques in
order to solve your specific problem.
As we study computer vision, let's take inspiration from these words:

"I saw that wisdom is better than folly, just as light is better than darkness."
Ecclesiastes, 2:13
Let's build applications that see clearly, and create knowledge.

What this book covers

Chapter 1, Getting the Most out of Your Camera System, discusses how to select and
configure camera systems to see invisible light, fast motion, and distant objects.
Chapter 2, Photographing Nature and Wildlife with an Automated Camera, shows how
to build a "camera trap", as used by nature photographers, and process photos to
create beautiful effects.
Chapter 3, Recognizing Facial Expressions with Machine Learning, explores ways to
build a facial expression recognition system with various feature extraction
techniques and machine learning methods.

Preface

Chapter 4, Panoramic Image Stitching Application Using Android Studio and NDK,
focuses on the project of building a panoramic camera app for Android with
the help of OpenCV 3's stitching module. We will use C++ with Android NDK.
Chapter 5, Generic Object Detection for Industrial Applications, investigates ways
to optimize your object detection model, make it rotation invariant, and apply
scene-specific constraints to make it faster and more robust.
Chapter 6, Efficient Person Identification Using Biometric Properties, is about building a
person identification and registration system based on biometric properties of that
person, such as their fingerprint, iris, and face.
Chapter 7, Gyroscopic Video Stabilization, demonstrates techniques for fusing data
from videos and gyroscopes, how to stabilize videos shot on your mobile phone,
and how to create hyperlapse videos.

Getting the Most out of Your

Camera System
Claude Monet, one of the founders of French Impressionist painting, taught his
students to paint only what they saw, not what they knew. He even went as far as
to say:
"I wish I had been born blind and then suddenly gained my sight so that I could
begin to paint without knowing what the objects were that I could see before me."
Monet rejected traditional artistic subjects, which tended to be mystical,
heroic, militaristic, or revolutionary. Instead, he relied on his own observations
of middle-class life: of social excursions; of sunny gardens, lily ponds, rivers, and
the seaside; of foggy boulevards and train stations; and of private loss. With deep
sadness, he told his friend, Georges Clemenceau (the future President of France):
"I one day found myself looking at my beloved wife's dead face and just
systematically noting the colors according to an automatic reflex!"
Monet painted everything according to his personal impressions. Late in life, he
even painted the symptoms of his own deteriorating eyesight. He adopted a reddish
palette while he suffered from cataracts and a brilliant bluish palette after cataract
surgery left his eyes more sensitive, possibly to the near ultraviolet range.

[1]

Getting the Most out of Your Camera System

Like Monet's students, we as scholars of computer vision must confront a distinction

between seeing and knowing and likewise between input and processing. Light, a lens, a
camera, and a digital imaging pipeline can grant a computer a sense of sight. From the
resulting image data, machine-learning (ML) algorithms can extract knowledge or at
least a set of meta-senses such as detection, recognition, and reconstruction (scanning).
Without proper senses or data, a system's learning potential is limited, perhaps
even nil. Thus, when designing any computer vision system, we must consider the
foundational requirements in terms of lighting conditions, lenses, cameras, and
imaging pipelines.
What do we require in order to clearly see a given subject? This is the central
question of our first chapter. Along the way, we will address five subquestions:

What do we require to see fast motion or fast changes in light?

What do we require to see distant objects?

What do we require to see with depth perception?

What do we require to see in the dark?

How do we obtain good value-for-money when purchasing lenses

and cameras?
For many practical applications of computer vision, the environment
is not a well-lit, white room, and the subject is not a human face at a
distance of 0.6m (2')!

The choice of hardware is crucial to these problems. Different cameras and lenses
are optimized for different imaging scenarios. However, software can also make or
break a solution. On the software side, we will focus on the efficient use of OpenCV.
Fortunately, OpenCV's videoio module supports many classes of camera systems,
including the following:

Webcams in Windows, Mac, and Linux via the following frameworks,

which come standard with most versions of the operating system:

Windows: Microsoft Media Foundation (MSMF), DirectShow,

or Video for Windows (VfW)

Mac: QuickTime

Linux: Video4Linux (V4L), Video4Linux2 (V4L2), or libv4l

Built-in cameras in iOS and Android devices

OpenNI-compliant depth cameras via OpenNI or OpenNI2, which are

open-source under the Apache license
[2]

Chapter 1

Other depth cameras via the proprietary Intel Perceptual Computing SDK

Photo cameras via libgphoto2, which is open source under the GPL license.
For a list of libgphoto2's supported cameras, see http://gphoto.org/proj/
libgphoto2/support.php.
Note that the GPL license is not appropriate for use in closed
source software.

IIDC/DCAM-compliant industrial cameras via libdc1394, which is

open-source under the LGPLv2 license

For Linux, unicap can be used as an alternative interface for

IIDC/DCAM-compliant cameras, but unicap is GPL-licensed
and thus not appropriate for use in closed-source software.

Other industrial cameras via the following proprietary frameworks:

Allied Vision Technologies (AVT) PvAPI for GigE Vision cameras

Smartek Vision Giganetix SDK for GigE Vision cameras

XIMEA API
The videoio module is new in OpenCV 3. Previously, in OpenCV
2, video capture and recording were part of the highgui module,
but in OpenCV 3, the highgui module is only responsible for GUI
functionality. For a complete index of OpenCV's modules, see the
official documentation at http://docs.opencv.org/3.0.0/.

However, we are not limited to the features of the videoio module; we can use
other APIs to configure cameras and capture images. If an API can capture an array
of image data, OpenCV can readily use the data, often without any copy operation
or conversion. As an example, we will capture and use images from depth cameras
via OpenNI2 (without the videoio module) and from industrial cameras via the
FlyCapture SDK by Point Grey Research (PGR).

[3]

Getting the Most out of Your Camera System

An industrial camera or machine vision camera typically has

interchangeable lenses, a high-speed hardware interface (such as
FireWire, Gigabit Ethernet, USB 3.0, or Camera Link), and a complete
programming interface for all camera settings.
Most industrial cameras have SDKs for Windows and Linux. PGR's
FlyCapture SDK supports single-camera and multi-camera setups on
Windows as well as single-camera setups on Linux. Some of PGR's
competitors, such as Allied Vision Technologies (AVT), offer better
support for multi-camera setups on Linux.

We will learn about the differences among categories of cameras, and we will test the
capabilities of several specific lenses, cameras, and configurations. By the end of the
chapter, you will be better qualified to design either consumer-grade or industrialgrade vision systems for yourself, your lab, your company, or your clients. I hope to
surprise you with the results that are possible at each price point!

Coloring the light

The human eye is sensitive to certain wavelengths of electromagnetic radiation. We
call these wavelengths "visible light", "colors", or sometimes just "light". However,
our definition of "visible light" is anthropocentric as different animals see different
wavelengths. For example, bees are blind to red light, but can see ultraviolet light
(which is invisible to humans). Moreover, machines can assemble human-viewable
images based on almost any stimulus, such as light, radiation, sound, or magnetism.
To broaden our horizons, let's consider eight kinds of electromagnetic radiation and
their common sources. Here is the list, in order of decreasing wavelength:

Radio waves radiate from certain astronomical objects and from lightning.
They are also generated by wireless electronics (radio, Wi-Fi, Bluetooth,
and so on).

Microwaves radiated from the Big Bang and are present throughout
the Universe as background radiation. They are also generated by
microwave ovens.

Far infrared (FIR) light is an invisible glow from warm or hot things such
as warm-blooded animals and hot-water pipes.

Near infrared (NIR) light radiates brightly from our sun, from flames, and
from metal that is red-hot or nearly red-hot. However, it is a relatively weak
component in commonplace electric lighting. Leaves and other vegetation
brightly reflect NIR light. Skin and certain fabrics are slightly transparent
to NIR.
[4]

Chapter 1

Visible light radiates brightly from our sun and from commonplace
electric light sources. Visible light includes the colors red, orange, yellow,
green, blue, and violet (in order of decreasing wavelength).

Ultraviolet (UV) light, too, is abundant in sunlight. On a sunny day, UV

light can burn our skin and can become slightly visible to us as a blue-gray
haze in the distance. Commonplace, silicate glass is nearly opaque to UV
light, so we do not suffer sunburn when we are behind windows (indoors
or in a car). For the same reason, UV camera systems rely on lenses made
of non-silicate materials such as quartz. Many flowers have UV markings
that are visible to insects. Certain bodily fluids such as blood and urine are
more opaque to UV than to visible light.

X-rays radiate from certain astronomical objects such as black holes. On

Earth, radon gas, and certain other radioactive elements are natural X-ray
sources.

Gamma rays radiate from nuclear explosions, including supernovae. To

lesser extents the sources of gamma rays also include radioactive decay
and lightning strikes.

NASA provides the following visualization of the wavelength and temperature

associated with each kind of light or radiation:

[5]

Getting the Most out of Your Camera System

Passive imaging systems rely on ambient (commonplace) light or radiation sources

as described in the preceding list. Active imaging systems include sources of their
own so that the light or radiation is structured in more predictable ways. For
example, an active night vision scope might use a NIR camera plus a NIR light.
For astronomy, passive imaging is feasible across the entire electromagnetic
spectrum; the vast expanse of the Universe is flooded with all kinds of light
and radiation from sources old and new. However, for terrestrial (Earth-bound)
purposes, passive imaging is mostly limited to the FIR, NIR, visible, and UV ranges.
Active imaging is feasible across the entire spectrum, but the practicalities of power
consumption, safety, and interference (between our use case and others) limit the
extent to which we can flood an environment with excess light and radiation.
Whether active or passive, an imaging system typically uses a lens to bring light or
radiation into focus on the surface of the camera's sensor. The lens and its coatings
transmit some wavelengths while blocking others. Additional filters may be placed
in front of the lens or sensor to block more wavelengths. Finally, the sensor itself
exhibits a varying spectral response, meaning that for some wavelengths, the sensor
registers a strong (bright) signal, but for other wavelengths, it registers a weak (dim)
signal or no signal. Typically, a mass-produced digital sensor responds most strongly
to green, followed by red, blue, and NIR. Depending on the use case, such a sensor
might be deployed with a filter to block a range of light (whether NIR or visible)
and/or a filter to superimpose a pattern of varying colors. The latter filter allows
for the capture of multichannel images, such as RGB images, whereas the unfiltered
sensor would capture monochrome (gray) images.
The sensor's surface consists of many sensitive points or photosites. These are
analogous to pixels in the captured digital image. However, photosites and pixels
do not necessarily correspond one-to-one. Depending on the camera system's design
and configuration, the signals from several photosites might be blended together to
create a neighborhood of multichannel pixels, a brighter pixel, or a less noisy pixel.
Consider the following pair of images. They show a sensor with a Bayer filter, which
is a common type of color filter with two green photosites per red or blue photosite.
To compute a single RGB pixel, multiple photosite values are blended. The left-hand
image is a photo of the filtered sensor under a microscope, while the right-hand
image is a cut-away diagram showing the filter and underlying photosites:

[6]

Chapter 1

The preceding images come from Wikimedia. They are contributed by

the users Natural Philo, under the Creative Commons Attribution-Share
Alike 3.0 Unported license (left), and Cburnett, under the GNU Free
Documentation License (right).

As we see in this example, a simplistic model (an RGB pixel) might hide
important details about the way data are captured and stored. To build efficient
image pipelines, we need to think about not just pixels, but also channels and
macropixelsneighborhoods of pixels that share some channels of data and are
captured, stored, and processed in one block. Let's consider three categories of
image formats:

A raw image is a literal representation of the photosites' signals, scaled

to some range such as 8, 12, or 16 bits. For photosites in a given row of the
sensor, the data are contiguous but for photosites in a given column, they
are not.

A packed image stores each pixel or macropixel contiguously in memory.

That is to say, data are ordered according to their neighborhood. This is
an efficient format if most of our processing pertains to multiple color
components at a time. For a typical color camera, a raw image is not packed
because each neighborhood's data are split across multiple rows. Packed
color images usually use RGB channels, but alternatively, they may use
YUV channels, where Y is brightness (grayscale), U is blueness (versus
greenness), and V is redness (also versus greenness).

[7]

Getting the Most out of Your Camera System

A planar image stores each channel contiguously in memory. That is to

say, data are ordered according to the color component they represent.
This is an efficient format if most of our processing pertains to a single
color component at a time. Packed color images usually use YUV channels.
Having a Y channel in a planar format is efficient for computer vision
because many algorithms are designed to work on grayscale data alone.

An image from a monochrome camera can be efficiently stored and processed in its
raw format or (if it must integrate seamlessly into a color imaging pipeline) as the Y
plane in a planar YUV format. Later in this chapter, in the sections Supercharging the
PlayStation Eye and Supercharging the GS3-U3-23S6M-C and other Point Grey Research
cameras, we will discuss code samples that demonstrate efficient handling of various
image formats.
Until now, we have covered a brief taxonomy of light, radiation, and colortheir
sources, their interaction with optics and sensors, and their representation as
channels and neighborhoods. Now, let's explore some more dimensions of image
capture: time and space.

Capturing the subject in the moment

Robert Capa, a photojournalist who covered five wars and shot images of the first
wave of D-Day landings at Omaha Beach, gave this advice:
"If your pictures aren't good enough, you're not close enough."
Like a computer vision program, a photographer is the intelligence behind the lens.
(Some would say the photographer is the soul behind the lens.) A good photographer
continuously performs detection and tracking tasksscanning the environment,
choosing the subject, predicting actions and expressions that will create the right
moment for the photo, and choosing the lens, settings, and viewpoint that will most
effectively frame the subject.
By getting "close enough" to the subject and the action, the photographer can observe
details quickly with the naked eye and can move to other viewpoints quickly because
the distances are short and because the equipment is typically light (compared to a
long lens on a tripod for a distant shot). Moreover, a close-up, wide-angle shot pulls
the photographer, and viewer, into a first-person perspective of events, as if we
become the subject or the subject's comrade for a single moment.

[8]

Chapter 1

Photographic aesthetics concern us further in Chapter 2, Photographing Nature and

Wildlife with an Automated Camera. For now, let's just establish two cardinal rules:
don't miss the subject and don't miss the moment! Poor visibility and unfortunate
timing are the worst excuses a photographer or a practitioner of computer vision
can give. To hold ourselves to account, let us define some measurements that are
relevant to these cardinal rules.
Resolution is the finest level of detail that the lens and camera can see. For many
computer vision applications, recognizable details are the subject of the work, and
if the system's resolution is poor, we might miss this subject completely. Resolution
is often expressed in terms of the sensor's photosite counts or the captured image's
pixel counts, but at best these measurements only tell us about one limiting factor. A
better, empirical measurement, which reflects all characteristics of the lens, sensor, and
setup, is called line pairs per millimeter (lp/mm). This means the maximum density
of black-on-white lines that the lens and camera can resolve, in a given setup. At any
higher density than this, the lines in the captured image blur together. Note that lp/
mm varies with the subject's distance and the lens's settings, including the focal length
(optical zoom) of a zoom lens. When you approach the subject, zoom in, or swap out a
short lens for a long lens, the system should of course capture more detail! However,
lp/mm does not vary when you crop (digitally zoom) a captured image.
Lighting conditions and the camera's ISO speed setting also have an effect on lp/
mm. High ISO speeds are used in low light and they boost both the signal (which
is weak in low light) and the noise (which is as strong as ever). Thus, at high ISO
speeds, some details are blotted out by the boosted noise.
To achieve anything near its potential resolution, the lens must be properly focused.
Dante Stella, a contemporary photographer, describes a problem with modern
camera technology:
"For starters, it lacks thought-controlled predictive autofocus."
That is to say, autofocus can fail miserably when its algorithm is mismatched to a
particular, intelligent use or a particular pattern of evolving conditions in the scene.
Long lenses are especially unforgiving with respect to improper focus. The depth
of field (the distance between the nearest and farthest points in focus) is shallower
in longer lenses. For some computer vision setupsfor example, a camera hanging
over an assembly linethe distance to the subject is highly predictable and in such
cases manual focus is an acceptable solution.

[9]

Getting the Most out of Your Camera System

Field of view (FOV) is the extent of the lens's vision. Typically, FOV is measured as
an angle, but it can be measured as the distance between two peripherally observable
points at a given depth from the lens. For example, a FOV of 90 degrees may also be
expressed as a FOV of 2m at a depth of 1m or a FOV of 4m at a depth of 2m. Where
not otherwise specified, FOV usually means diagonal FOV (the diagonal of the lens's
vision), as opposed to horizontal FOV or vertical FOV. A longer lens has a narrower
FOV. Typically, a longer lens also has higher resolution and less distortion. If our
subject falls outside the FOV, we miss the subject completely! Toward the edges of
the FOV, resolution tends to decrease and distortion tends to increase, so preferably
the FOV should be wide enough to leave a margin around the subject.
The camera's throughput is the rate at which it captures image data. For many
computer vision applications, a visual event might start and end in a fleeting
moment and if the throughput is low, we might miss the moment completely or
our image of it might suffer from motion blur. Typically, throughput is measured
in frames per second (FPS), though measuring it as a bitrate can be useful, too.
Throughput is limited by the following factors:

Shutter speed (exposure time): For a well-exposed image, the shutter speed
is limited by lighting conditions, the lens's aperture setting, and the camera's
ISO speed setting. (Conversely, a slower shutter speed allows for a narrower
aperture setting or slower ISO speed.) We will discuss aperture settings after
this list.

The type of shutter: A global shutter synchronizes the capture across all
photosites. A rolling shutter does not; rather, the capture is sequential such
that photosites at the bottom of the sensor register their signals later than
photosites at the top. A rolling shutter is inferior because it can make an
object appear skewed when the object or the camera moves rapidly. (This is
sometimes called the "Jell-O effect" because of the video's resemblance to a
wobbling mound of gelatin.) Also, under rapidly flickering lighting, a rolling
shutter creates light and dark bands in the image. If the start of the capture is
synchronized but the end is not, the shutter is referred to as a rolling shutter
with global reset.

The camera's onboard image processing routines, such as conversion of

raw photosite signals to a given number of pixels in a given format. As the
number of pixels and bytes per pixel increase, the throughput decreases.

The interface between the camera and host computer: Common camera
interfaces, in order of decreasing bit rates, include CoaXPress full, Camera
Link full, USB 3.0, CoaXPress base, Camera Link base, Gigabit Ethernet,
IEEE 1394b (FireWire full), USB 2.0, and IEEE 1394 (FireWire base).

[ 10 ]

Get more information OpenCV 3 Blueprints

Where to buy this book

You can buy OpenCV 3 Blueprints from the Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.

www.PacktPub.com

Stay Connected: