XCELLENCE IN EMBEDDED VISION

by Fernando Martinez Vallina

HLS Design Methodology Engineer
Xilinx, Inc.
[email protected]
Jos Roberto Alvarez
Engineering Director
for Video Technology
Xilinx, Inc.
[email protected]

Pairing Vivado HLS with the OpenCV

libraries enables rapid prototyping and
development of Smarter Vision systems
targeting the Zynq All Programmable SoC.
omputer vision has been a
well-established discipline in
academic circles for several
years; many vision algorithms today
hail from decades of research. But
only recently have we seen the proliferation of computer vision in many
aspects of our lives. We now have selfdriving cars, game consoles that react
to our every move, autonomous vacuum cleaners and mobile phones that
respond to our gestures, among other
vision products.
The challenge today is how to
implement these and future vision systems efficiently while meeting strict
power and time-to-market constraints.
The Zynq All Programmable SoC
can be the foundation for such products, in tandem with the widely used
computer vision library OpenCV and
the high-level synthesis (HLS) tools
that accelerate critical functions in
hardware. Together, this combination
makes a powerful platform for
designing and implementing Smarter
Vision systems.
Embedded systems are ubiquitous
in the market today. However, limitations in computing capabilities, especially when dealing with large picture
sizes and high frame rates, have
restricted their use in practical implementations for computer/machine
vision. Advances in image sensor
technologies have been essential in
opening the eyes of embedded
devices to the world so they can interact with their environment using computer vision algorithms. The combination of embedded systems and computer/machine vision constitutes

Second Quarter 2013

embedded vision, a discipline that is

fast becoming the basis for designing
machines that see and understand
their environments.
DEVELOPMENT OF EMBEDDED
VISION SYSTEMS
Embedded vision involves running
intelligent computer vision algorithms
in a computing platform. For many
users, a standard desktop-computing
processing platform provides a conveniently accessible target. However, a
general computing platform may not
meet the requirements for producing
highly embedded products that are
compact, efficient and low in power
when processing large image-data
sets, such as multiple streams of realtime HD video at 60 frames/second.
Figure 1 illustrates the flow that
designers commonly employ to create
embedded vision applications. The
algorithm design is the most important step in this process, since the
algorithm will determine whether we
meet the processing and quality criteria for any particular computer vision
task. At first, designers explore algorithm choices in a numerical-computing environment like MATLAB in
order to work out high-level processing options. Once they have determined the proper algorithm, they typically model it in a high-level language,
generally C/C++, for fast execution
and adherence to the final bit-accurate implementation.
System partitioning is an important
step in the development process. Here,
through algorithm performance analysis, designers can determine what porXcell Journal

25

XCELLENCE IN EMBEDDED VISION

Algorithm
Design

Modeling

System
Partitioning

Prototyping

Implementation

Release

Figure 1 Embedded vision system

development process

tions of the algorithm they will need

to accelerate in hardware given the
real-time requirements for processing
representative input data sets. It is
also important to prototype the entire
system in the target platform, so as to
realistically measure performance
expectations. Once the prototyping
process indicates that a design has
met all performance and quality targets, designers can then start implementing the final system in the actual
targeted device. Finally, the last step
is to test the design running on the
chip in all use-case scenarios. When
everything checks out, the team can
release the final product.
ZYNQ SOC: SMARTEST CHOICE
FOR EMBEDDED VISION
In the development of machine
vision applications, it is very important for design teams to choose a
highly flexible device. They need a
computing platform that includes
powerful general-purpose process26

Xcell Journal

ing capabilities supporting a wide

software ecosystem, along with
robust digital signal-processing capabilities for implementing computationally demanding and memory-efficient computer vision algorithms.
Tight integration at the silicon level is
essential for implementing efficient
and complete systems.
Xilinx All Programmable SoCs are
processor-centric devices that offer
software, hardware and I/O programmability in a single chip. The Zynq
SoC features an ARM dual-core
Cortex-A9 MPCore processing
system coupled with FPGA logic and
key peripherals on a single device. As
such, the device enables designers to
implement extremely efficient embedded vision systems.
This level of integration between
the processing subsystem, FPGA logic
and peripherals in the Zynq SoC
ensures faster data transfer speeds and
a much lower power requirement and
BOM cost than a system designed with
individual components. It is feasible to
implement systems in the Zynq SoC
that require real-time processing for
1080p60 video sequences (1,920 x 1,080

RGB pictures at 60 frames/s) with processing capabilities in the hundreds of

giga-operations per second.
To take full advantage of the many
features of the Zynq SoC, Xilinx provides the Vivado Design Suite, an IPand system-centric design environment that increases designer development productivity with the fast integration and implementation cycles
needed to dynamically develop
smarter embedded products. A component of that suite, Vivado HLS,
allows you to take algorithms youve
developed in C/C++ and compile them
into RTL to run in the FPGA logic.
The Vivado HLS tool is particularly
well-suited to embedded vision design.
In this flow, you create your algorithms
in C/C++; compile the algorithm or
parts of the algorithm into RTL using
Vivado HLS; and determine which functions are better suited to run in FPGA
logic and which are better suited to run
on the ARM processor. In this way, your
design team can home in on the optimal
performance for their vision systems
running in the Zynq SoC.
To further help embedded vision
developers to create Smarter Vision

Accelerate Algorithmic C-to-IP Integration

C, C++ or SystemC

Algorithmic Specification

Vivado HLS

Microarchitecture Exploration

RTL Implementation

VHDL or Verilog

System IP Integration

+
_

+
_

Comprehensive Integration with

the Xilinx Design Environment

Figure 2 High-level synthesis design flow

Second Quarter 2013

XCELLENCE IN EMBEDDED VISION

systems, Xilinx has added to Vivado

support for the OpenCV libraries of
computer vision algorithms. Xilinx
has also introduced the new IP
Integrator tools and SmartCORE IP
to support these kinds of designs (see
cover story, page 8).
OPENCV MAKES COMPUTER
VISION ACCESSIBLE
OpenCV provides a path to the development of intelligent computer vision
algorithms that are predicated on realtime performance. The libraries provide designers with an environment
for experimentation and fast prototyping of algorithms.
The design framework of OpenCV
is supported in multiple platform. But
in many cases, to make the libraries
efficient for embedded products
requires implementation in an embedded platform that is capable of accelerating demanding routines for realtime performance.
While OpenCV was designed with
computational efficiency in mind, it
originated in traditional computing
environments with support for multicore processing. These computing
platforms may not be optimal for an
embedded application where efficiency, cost and power consumption are
paramount.
FEATURES OF OPENCV
OpenCV is an open-source computer
vision library released under a BSD
license. This means that it is free to
use in academic and commercial
applications. It was originally
designed to be computationally efficient on general-purpose multiprocessing systems, with a strong focus
on real-time applications. In addition, OpenCV provides access to
multiple programming interfaces
like C/C++ and Python.
The advantage of an open-source
project is that the user community is
constantly improving algorithms and
extending them for use in a wide
variety of application domains. There
Second Quarter 2013

Current Frame

Output Frame

OpenCV
Motion-Detection
Algorithm
Previous Frame

Detected New Car

Figure 3 Motion-detection example from the OpenCV library of algorithms

are now more than 2,500 functions

implemented in OpenCV; here are
some examples:
Matrix math
Utilities and data structures
General image-processing
functions
Image transforms
Image pyramids
Geometric descriptor functions
Feature recognition, extraction
and tracking
Image segmentation and fitting
Camera calibration, stereo and
3D processing
Machine learning: detection,
recognition
For a more detailed description of
OpenCV, please go to opencv.org and
opencv.willowgarage.com.
ACCELERATING OPENCV
FUNCTIONS USING HLS
Once you have partitioned the
architecture of an embedded vision
system to single out computationally demanding portions, HLS tools

can help you accelerate these functions while still written in C++.
Vivado HLS makes use of C, C++ or
SystemC code to produce an efficient RTL implementation.
Furthermore, the Vivado IP-centric
design environment provides a wide
range of processing IP SmartCOREs
that simplify connections to imaging
sensors, networks and other necessary I/O interfaces, easing the process
of implementing those functions in the
OpenCV libraries. This is a distinct
advantage from other implementation
alternatives, where there is a need to
accelerate even the most fundamental
OpenCV I/O functionality.
WHY HIGH-LEVEL SYNTHESIS?
The Vivado HLS compiler from Xilinx
is a software compiler designed to
transform algorithms implemented in
C, C++ or SystemC into optimized RTL
for a user-defined clock frequency and
device in the Xilinx product portfolio.
It has the same core technology underpinnings as a compiler for an x86
processor in terms of interpretation,
analysis and optimization of C/C++
programs. This similarity enables a
rapid migration from a desktop development environment into an FPGA
Xcell Journal

27

XCELLENCE IN EMBEDDED VISION

Figure 4 Motion detection on the Zynq SoC using the ARM processor

implementation. The default behavior

of Vivado HLS will generate an RTL
implementation without the need for
user input after you have selected the
target clock frequency and device. In
addition, Vivado HLS, like any other
compiler, has optimization levels.
Since the final execution target of the
algorithm is a tailor-made microarchitecture, the level of optimizations possible in Vivado HLS is finer-grained
than in a traditional compiler. The
concept of O1 O3 optimizations typical in software design for a processor
is replaced with architecture-exploration directives. These directives
draw on the expertise of the user to
guide Vivado HLS in creating the best
possible implementation for a particular algorithm in terms of power, area
and performance.
The user design flow for the HLS
compiler is shown in Figure 2. At a
28

Xcell Journal

conceptual level, the user provides a

C/C++/SystemC algorithmic description and the compiler generates an
RTL implementation. The transformation of a program code into RTL is
divided into four major regions: algorithm specification, microarchitecture exploration, RTL implementation and IP packaging.
The algorithmic-specification stage
refers to the development of the software application that will be targeted
to the FPGA fabric. This specification
can be developed in a standard desktop software-development environment and can make complete use of
Xilinx-provided software libraries
such as OpenCV. In addition to
enabling a software-centric development flow, Vivado HLS elevates the
verification abstraction from RTL to
C/C++. The user can carry out complete functional verification of the

algorithm using the original software.

After RTL generation through Vivado
HLS, the generated design is analogous to processor assembly code generated by a traditional software compiler. The user can debug at this
assembly code level, but is not
required to do so.
While Vivado HLS can handle
almost all C/C++ code targeted at
other software compilers, there is
one restriction placed on the code.
For compilation into an FPGA using
Vivado HLS, the user code cannot
include any run-time dynamic memory allocations. Unlike a processor,
where the algorithm is bounded by a
single memory architecture, an FPGA
implementation has an algorithmspecific memory architecture. By
analyzing the usage patterns of
arrays and variables, Vivado HLS can
determine the physical-memory laySecond Quarter 2013

XCELLENCE IN EMBEDDED VISION

out and memory types that will best

fit the storage and bandwidth
requirements in an algorithm. The
only requirement for this analysis to
work is that you explicitly describe
all memory that an algorithm consumes in the C/C++ code in the form
of arrays.
The second step in the transformation from a C/C++ implementation into an optimized FPGA implementation is the microarchitecture
exploration. At this stage, you apply
Vivado HLS compiler optimizations
to test out different designs for the
right mix of area and performance.
You can implement the same C/C++
code at different performance points
without having to modify source
code. The Vivado HLS compiler optimizations or directives are how the
performance of different portions of
an algorithm are stated.

The final stage in the Vivado HLS

compiler flow is RTL implementation
and IP packaging. These are automatic stages inside the Vivado HLS compiler that do not require RTL knowledge from users. The details of optimized RTL creation for different
devices in the Xilinx product portfolio are built into the compiler. At this
stage, the tool is, for all intents and
purposes, a pushbutton utility that
has been thoroughly tested and verified to produce timing-driven and
FPGA fabric-driven RTL. The output
from the Vivado HLS compiler is
automatically packaged in formats
accepted by other Xilinx tools such as
IP-XACT. Therefore, there are no
additional steps in using an HLS-generated IP core in Vivado.
The OpenCV libraries from Xilinx
provide a shortcut in the process of
design optimization using Vivado

HLS. These libraries have been

precharacterized to yield functions
capable of pixel-rate processing at
1080p resolutions. The optimization
knowledge required to guide the
Vivado HLS compiler is embedded
into these libraries. Thus, you are free
to quickly iterate between an OpenCV
concept application in a desktop environment to a running OpenCV application on the Zynq SoC, which enables
operations on the ARM processor and
the FPGA fabric.
Figure 3 shows an overview of a
motion-detection application developed in OpenCV. The goal of this
design is to detect moving objects in
a video stream by comparing the current image frame with the previous
one. The first stage in the algorithm
is to detect the edges in both frames.
This data-reduction operation makes
it easier to analyze the relative

Figure 5 Motion detection on the Zynq SoC using the programmable fabric

Second Quarter 2013

Xcell Journal

29

XCELLENCE IN EMBEDDED VISION

FPGA S OLUTIONS
from

Mars ZX3 SoC Module

? Xilinx Zynq-7000 All Programmable SoC

(Dual Cortex-A9 + Xilinx Artix-7 FPGA)
? DDR3 SDRAM + NAND Flash
? Gigabit Ethernet + USB 2.0 OTG
? SO-DIMM form factor (68 x 30 mm)

Mercury KX1 FPGA Module

?
?
?
?

Xilinx Kintex-7 FPGA

High-performance DDR3 SDRAM
USB 3.0, PCIe 2.0 + 2 Gigabit Ethernet ports
Smaller than a credit card

Mars AX3 Artix-7 FPGA Module

? 100K Logic Cells + 240 DSP Slices

? DDR3 SDRAM + Gigabit Ethernet
? SO-DIMM form factor (68 x 30 mm)

FPGA Manager Solution

Simple, fast host-to-FPGA data transfer, for PCI

Express, USB 3.0 and Gigabit Ethernet. Supports
user applications written in C, C++, C#, VB.net,
MATLAB, Simulink and LabVIEW.

We speak FPGA.
www.enclustra.com
30

Xcell Journal

change between consecutive frames.

Once the edge information has been
extracted, edges are compared to
detect edges that appear in the current image but not in the previous
image. These new detected edges
create a movement-detection mask
image. Before the results of the new
edge detection can be highlighted on
the current image, you must take
into account the effects of image
sensor noise. This noise, which can
vary from frame to frame, can lead
to random false edges in the motiondetection mask image. Therefore,
you must filter this image to reduce
the impact of noise on the quality of
the algorithm.
Noise reduction for this application
is accomplished through the use of a
7x7 median filter on the movementdetection mask image. The central
idea of a median filter is to take the
middle value in a 7x7 window of
neighboring pixels. The filter then
reports back the median value as the
final value for the center pixel of the
window. After noise reduction, the
movement-detection mask image is
combined with the live input image to
highlight moving edges in red.
You can fully implement the application to run on the ARM processing
subsystem with the source code to
Zynq SoC mapping shown in Figure
4. The only hardware elements in this
implementation are for the cvgetframe and showimage functions.
These video I/O functions are implemented using Xilinx video I/O subsystems in the FPGA fabric. At the
time of the cvgetframe function call,
the input side of the video I/O subsystem handles all the details, grabbing and decoding a video stream
from an HDMI interface and placing
the pixel data into DDR memory. At
the time of showimage, this subsystem handles the transfer of pixel data
from DDR memory into a video display controller in order to drive a TV
or other HDMI-compliant video display device.

Vivado HLS-optimized OpenCV

libraries for hardware acceleration
enable the porting of the code in
Figure 4 into a real-time 60-frame/s
pixel-processing pipeline on the
FPGA fabric. These OpenCV libraries
provide foundational functions for
the elements in OpenCV that require
hardware acceleration. Without hardware accelerationthat is, if you
were to run all the code in the ARM
processors onlythis algorithm has
a throughput of merely 1 frame every
13 seconds (0.076 frames/s). Figure 5
shows the new mapping of the application after Vivado HLS compilation.
Note that the video I/O mapping of
the original system is reused. The
computational core of the algorithm,
which was previously executing on
the ARM processor, is compiled into
multiple Vivado HLS-generated IP
blocks. These blocks, which are connected to the video I/O subsystem in
Vivado IP Integrator, are optimized
for 1080p resolution video processing at 60 frames/s.
The All Programmable environment provided by the Zynq SoC and
Vivado Design Suite is well-suited
for the design, prototyping and testing of embedded vision systems at
the high data-processing rates
required by the latest high-definition
video technologies. Using the opensource set of libraries included in
OpenCV is the best way to implement demanding computer vision
applications in short development
times. Since OpenCV libraries are
written in C++, we use Vivado HLS
to create source code that can be
efficiently translated to hardware
RTL in the Zynq SoC FPGA fabric,
and used as convenient processing
accelerators without sacrificing the
flexibility of the design environment
originally envisioned by OpenCV.
For more information on creating
Smarter Vision designs with Vivado
Design Suite, visit http://www.xilinx.com/products/design-tools/vivado/index.htm.
Second Quarter 2013