Computer Vision Winter Workshop 2006, Ondrej Chum, Vojtech Franc (eds.

)
Telc, Czech Republic, February 68
Czech Pattern Recognition Society

First steps in enhancing 3D vision technique using 2D/3D sensors

T.D.Arun Prasad, Klaus Hartmann, Wolfgang Weihs, Seyed Eghbal Ghobadi, and Arnd Sluiter
Center for Sensor Systems
University of Siegen
Germany
(Prasad,Hartmann,Weihs,Ghobadi,[email protected])
Abstract In this paper, a novel imaging technique for
enhancing 3-D vision is proposed. The enhancement is
achieved with the help of 3-D time-of-flight camera, this
3D camera delivers the intensity and depth information of
the scene in real time. Although this kind of 3-D cameras
provide accurate depth information, its low 2-D image resolution tends to be a hindering factor for many image processing applications. This limitation can be inundated with
the proposed setup; a 2-D sensor (CCD/CMOS) with higher
resolution is used to improve the image resolution. The 2-D
and 3-D cameras are placed in a special housing, so that
the field-of-view (FOV) is nearly same for both the cameras.
The 2D/3D data fusion becomes facile for mapping and/or
image registration. Hence the new system provides better
image quality with range data. Within this paper, we discuss
the initial results and findings of the proposed system.

Introduction

conditions. Thus, 3-D optical sensors deliver in real scale

the dimensions of an object, which are rotation- translationand illumination-invariant. Most of the problems of industrial inspection, reverse engineering, and virtual reality
require data about the geometrical shape of objects in 3Dimensional space [2].
At last, a novel idea of measuring the depth of the scene,
with time-of-flight principle using incoherent optical modulation came in the form of solid state device. This sensor overcomes several drawbacks faced by the commercial 2-D sensors. The time-of-flight camera has an edge
over the other 3-D measuring devices, which contains moving mechanical parts. The aging and wearing could create inaccuracies in the system, since it uses moving parts.
The commonly known time-of-flight systems [PMD,Swiss
Ranger,Canesta] [12] produce the 3-D information using
slightly modified techniques, does not have any moving or
mechanical parts.

Related Work

The anthropomorphous thinking often convinces human being to consider the camera as a technical eye. The conventional image sensors (CCD/CMOS) measure the intensity of
the scene, but it lacks the depth information of the scene.
As the nature introduces severe physical limits to acquire
remote 3-D information, contact less extraction of depth information is still one of the key task in image processing.
The insensible depth information is generally solved by
two means. By using mathematics, the computer vision
community in the past decades have strived to develop a
suitable method for 3-D vision. This was often done with
the help of 2-D images, using techniques drawn mainly from
linear algebra and matrix theory. Other means of finding the
unfound depth are based on the laws of physics using methods such as - interferometry, triangulation or time-of-flight
(TOF) methods [5]. The innovation of laser scanners and
range sensors gave a big helping hand on this issue.

The basic component of the 3-D vision a time-of-flight sensor system which has been presented in this paper is a well
known and evolving 3-D vision technique. To the knowledge of the authors, no system exists with this optical insitu measurement (with common lens) for 2D/3D vision
systems. However there is a combination of 2D-3D sensor system [9]which consists of movable-mechanical components. The main goal of this paper is to show the procedure to increase the image resolution of the 3-D sensor,
which is currently available about 19,000 pixels. In the following section, we will discuss about a TOF camera and its
measuring (sensing) techniques. In section 3, we introduce
2D/3D imaging technique. Then in section 4,5 about characterisation and experimental setup. In section 6, we show
some initial results from the 2D/3D setup as well as its future perspectives.

1.1

Need for 3 Dimensional Sensors

Nowadays, 3-D image acquisition and processing appear to

be on the verge of a comparably stormy and far-reaching
development. Three-dimensional optical form measurements deliver the absolute 3-D geometry of objects that are
largely independent of the objects surface reflectivity, the
distance of the objects from the sensor, and illumination

2D/3D Camera Concept

The 2-D sensors (CCD/CMOS) come up with very high image resolution (millions of pixels), one can use intelligent
image processing algorithms to calculate the depth information of the scene, recover the shape or reveal the structure.
These are extracted at high computational cost. Further, it is
obvious for many real world problems it cannot achieve the

First steps in enhancing 3D vision technique using 2D/3D sensors

required robustness with intelligent image processing algorithms.
Even if it were possible the same algorithm often cannot
solve other problems due to illumination, reflectivity of the
surface and other ambient problems.
Rather in the quasi-straightforward approach such as a
3-D TOF sensors, which have a maximum of about 19,000
pixels till date. This resolution is quite insufficient for some
applications like navigation or autonomous transport system. So a combination of 2-D and 3-D sensor can address
the issue of intelligence. Therefore our approach addresses
this issue, such that an intelligent image processing algorithm can be supported by intelligent camera systems - such
as 2D/3D system.
To overcome the drawbacks and to increase the resolution
(primarily image, later also depth) of the sensing system, a
special optical setup is designed. This imaging setup comprises of some key components such as a special optics called as image multiplier, a CCD camera and a PMD camera system.
3.1

Image Multiplier

The image multiplier is an optical device which makes a

continuous multi-shutter operation possible and affordable.
Through a mutual recording and data transferring of both
cameras (PMD and CCD in our case) the exact over lap is
achieved. This image multiplier is made up of prisms and
mirrors which deliver the same information to both the eyepots. The optics helps in providing nearly the same field of
view for both sensors. The figure 1 shows the image multiplier. This makes the image registration or mapping more
efficient. Therefore the pixel correspondence also becomes
trivial.

[]
Mixer Device) pixels mix the modulated light directly in
optical active area in the sensing process, which leads to a
smart operation. The distance information is weighted on
each pixel independently. A 3-D information is obtained
by evaluating at least two frames. The distance information
can be calculated inside the camera system, only the 3-D
data is transferred to the PC. All these operations are be performed in the PMD system as it has an in built 32 bit (AMD
Elan SC520) microprocessor running under ELinOS operating system. The timing requirements are met by the FPGA
(Field Programmable Gate Array) circuitry inside the system. The PMD camera system has a frame rate up to 50 fps
(frames per second).
3.2.1 Lighting Module The PMD camera system has its
own lighting module. The array of infrared LEDs are used
for scene illumination. These LEDs are modulated with 20
MHz. The system emits an RF modulated optical radiation
field (typically 20 MHz or higher) in the infra-red spectrum.
The diffused backscattered signal from the scene is detected
by the camera. Each pixel has the capability to demodulate
the signal and detect its phase, which is proportional to the
distance of the reflecting object. The signal frequency of
20MHz defines the unambiguous distance range of 7.5 m.
The PMD sensor with 1024 pixels is used in this experiment,since it has a matured technology and a key advantage of rejecting back ground illumination. This is done by
means of an active Suppression of Background Illumination
(SBI) circuitry in each pixels of the chip, which makes it
efficient, to operate indoor (even in the dark) and outdoor
(with ambient light) with its own active illumination and not
affected by ambient light including sunlight.
3.3

CCD Camera

The CCD camera used in this case, is a black and white

camera with SVGA resolution (780 (H) x 582 (V)) (AVT
MARLIN F-046B) is used, which is equipped with a SONY
1/2 type progressive CCD array. The full frame rate is up to
53 fps. This camera allows to select the ROI (Region Of Interest) easily, this is quite an advantage for the experimental
setup which is currently used. It also makes data transmission and image processing efficient in terms of time and data
size.

Figure 1: The Image Multiplier used in the 2D/3D setup- the 2D

and 3D cameras are mounted on the eye-pots which have two Cmounts

This device needs to be calibrated for better results. The calibration can be classified into three steps for such a system;
Calibration using Optics (Image Multiplier), Calibration using Intensity Information and Calibration using Range Information. More about the special calibration device and
technique used for this 2D/3D system is discussed in [1].
4.1

3.2

Time-of-Flight Camera System

As the 3-D TOF cameras measure depth information using time-of-flight principle, the RF-modulated light signal
with variable phase shift is sent to the 3-D-object and reflected back to the camera system. The PMD (Photonic

Characterisation of the 2D/3D Sensor

Setup

Calibration using Optics

Initially the optics is calibrated, so that the image multiplier

provides the exact overlap between the two sensors. As the
CCD and the PMD sensor have different chip and pixel size.
The image multiplier is calibrated with a grid plate to calibrate the cameras in order to have minimum mismatch.

T.D.Arun Prasad, Klaus Hartmann, Wolfgang Weihs, Seyed Eghbal Ghobadi, and Arnd Sluiter

[]

4.2 Calibration using Intensity Information

The intensity information of the 3-D time-of-flight camera
has relatively less number of pixel and the large pixel size.
This makes them to become less reliable for the gray scale
image. The intensity data is calibrated with the planar surface such as checker board. And the pixel error of 2 and 3.5
are obtained for horizontal and vertical axis.
4.3 Calibration using Range Information
The sensor performs a time-of-flight measurement by detecting the phase shift in the modulated emitted signal,
which can in turn be translated into a distance. In practice,
due to propagation delay in the driving circuits of the camera (PMD), a distance offset has to be included. Due to the
optics the range information has some inaccuracies. To over
come this, the 3-D TOF camera system is placed before the
planar surface (such as card board or wall). By knowing the
actual distance and comparing the measured value a linearity existed. This linear value is added to the measured data
for compensation. The range information had resolution of
about 1 cm.
As the system has a special optical setup a new calibration procedure is yet to be accomplished and it is still under
investigation [1] for better accuracies. The improved calibration technique will be derived from the techniques used
in planar surface (checker board) and/or using a laser pointer
in the case of multi-camera calibrations [8].

Figure 2: 2D/3D camera setup with PMD, RF modulated optical

source, CCD and the Image Multiplier

of the 3-D data set is smaller than that of the 2-D data interpolation becomes essential. The Figure 3 shows the steps
involved in the enhanced 3-D vision.

Experimental Setup

The pre-processed data was acquired from the RPC (Remote

Procedure Call) server in the PMD system via the Ethernet
interface to the PC, where the registration is performed. The
pre-processed 3-D data are acquired into the MATLAB, using Java RPC client. This automated process was achieved
in near time because of the stable RPC technology. The
PMD camera system also has IEEE1394 (FireWire), which
was not used in this experiment. The 2-D data from the
CCD camera was acquired by IEEE1394 - DCAM standards. Then the 3-D image enhancement was done in the
MATLAB.
Figure 2 shows the components used in the experimental
setup. This setup was used to register the image for a static
scene.
Table 1: Pixel information of the 2-D and 3-D sensors

Sensor
Number of Pixels
Pixel Size (m)

PMD(H)x(V)
64 x 16
155.3 x 210.8

CCD(H)x(V)
780 x 582
8.3 x 8.3

The above table describes the PMD to CCD pixel

mapping. Approximately 475 CCD pixels fits into one
PMD pixel. This shows that lots of information such as
edges and corners could be missed by the PMD system.

Steps involved in Enhanced 3-D Vision

The enhanced 3-D vision is attained by interpolating the

range data and registering it with the 2-D data. As the size

Figure 3: Steps involved in enhanced 3-D vision technique

The 2-D image received from the CCD sensor is shown in

First steps in enhancing 3D vision technique using 2D/3D sensors

figure 4. The 2-D gray scale image obtained from the PMD
camera has low resolution, this is shown in the figure 5 with
the grid. This low resolution image from the PMD reduces
its application in the field of machine vision or other image
based applications. More about the problem is discussed in
[1]. The 3-D range information obtained from the PMD is
plotted in figure 6. The captured 2-D image has relatively
high resolution, so the 3-D data set is interpolated to the size
of 2-D data set. The result of interpolation is shown in figure
7. The registered 2D/3D image is shown in figure 8. Thus
a 3-D image with higher resolution is obtained in near time.
Figure 9 shows the tilted view of the registered 2D/3D image
in order to partially see the depth map.
The nearly exact overlap of the 2-D and 3-D data makes
the 2D/3D image registration relatively faster, resulting up
to 10 fps (frames per second). The speed of the image registration could play more important role for the dynamic scene
or moving objects.

[]

PMD Distance proj

70
0

60
50
40
30
5

20
10

10

15
5
10
15
20
25
30

Figure 6: 3-D depth information from the TOF camera

70

PMD Distance proj

60
50

40
30
20
10

10

15
0

20

40

50

100

150

200

250

300

350

400

450

Figure 4: 2-D intensity image from CCD camera

Figure 7: Interpolated range data

Figure 8: Registered 2D/3D image

Figure 5: 2-D intensity image from PMD camera

The registered image has improved intensity. However

the accuracy of the range resolution has to be increased,

since it is about 1 cm. The range resolution can be more accurate by improving the optics (image multiplier) and also
from the Shape from - Shading, Surface approximation and
Motion [11] techniques.

T.D.Arun Prasad, Klaus Hartmann, Wolfgang Weihs, Seyed Eghbal Ghobadi, and Arnd Sluiter

[4]

[5]

[6]

[7]

Figure 9: Tilted view of the registered 2D/3D image

Conclusion

A new state-of-art 2D/3D sensor setup with common optics

is presented with the preliminary results and findings. The
key advantage of the 2D/3D system is its ability to generate
real 3-D range information and high intensity data in near
time. Also this can be achieved at less computational cost
and higher data rate. The results are promising. So the future
work will be focused on this system to improve its range
resolution (accuracy) using the improved optics and a device
specific calibration technique. A common hardware for 2D and 3-D cameras to meet the timing,digitising and data
acquisition could increase the performance of the system.

[8]

[9]

[10]

[11]

[12]

[]

-Aktuelle Entwicklungen und industrieller Einsatz der

Bildverarbeitung, MIT GmbH, pp. 273-278, Aachen
1996.
Todt, S., Rezk-Salama, C., Kolb, A.;, Real Time
Fusion of Depth and Light Field Images, poster;
ACM SIGGRAPH 2005.
Besl, P.;, Active Optical Range Imaging Sensors,
Machine Vision and Applications, Vol. 1, No. 2, pp.
127-152, 1988.
Justen, D.;, Untersuchung eines neuartigen
2D-gesttzten 3D-PMD Bildverarbeitungssysteme
Dissertation Universitt Siegen, 2001.
Justen, D., Hartmann, K.;, Three-dimensional image
processing through grayscale images and distance
data related to individual pixels; SPIE; Visual
Information Processing VIII Vol.3716-17;1999.
Svoboda,T., Martinec,D.,Pajdla,T.;, A convenient
multi-camera self-calibration for virtual
environments. PRESENCE: Teleoperators and
Virtual Environments, 14(4), August 2005.
El-Hakim, S.F., Brenner, C., Roth, G.;, A
Multi-Sensor Approach to Creating Accurate Virtual
Environments, ISPRS Journal of Phorogrammetry
and Remote Sensing (Elsevier). 53(6): 379-391. 1998.
Schwarte, R., Loffeld, O., Heinol, H., Xu, Z.;,
Integrierte 2D/3D-Geometrieprfung und Szenenanalyse,DFGSchwerpunktprogramm:Automatische
Sichtprfung technischer Objekte, Siegen, 1995-1998.
Jiang, X., Bunke, H.;, Dreidimensionales
Computersehen / Gewinnung und Analyse von
Tiefenbildern,Springer-Verlag, 1997.
3 D Range Sensors - www.pmdtec.com;
www.swissranger.ch; www.canesta.com.

Acknowledgement
We are highly indebted to all the co-workers in ZESS, especially Wolf Twelsiek, Rolf Wurmbach for their dedicated
cooperation. We would also like to gratefully acknowledge
the support of DAADs IPP program and the funding of
the Ministry of Science of Northrhine Westphalia. Our sincere thanks to the Deutsche Forschungsgemeinschaft (German Research Foundation) for funding this 2D/3D camera
project(LO 455/10-1).

References
[1] Hartmann,K., Loffeld,O., Ghobadi,S.E., Peters,V.,
Prasad,T.D.A., Sluiter,A., Weihs,W., Lerch,T.,
Lottner,O.;, Klassifizierungsaspekte bei der
3D-Szenenexploration mit einerneuen
2D/3D-Multichip-Kamera,
pp.74-80;SpringerVerlag;ASM 2005.
[2] Schwarte, R., Heinol, H., Buxbaum, B., Ringbeck, T.,
Xu,Z., Hartmann.K.;, Principles of Three
Dimensional Imaging techniques, Handbook of
Computer Vision and Applications, Volume 1,
Sensors and Imaging, Jahne, B., Hauecker, H.,
Geiler, P. (Hrsg.), The Academic Press,1999.
[3] Schwarte,R.;, Eine neuartige 3D-Kamera auf Basis
eines 2D-Gegentaktkorrelator-Arrays, Symposium