2017 International Conference on Nascent Technologies in the Engineering Field (ICNTE-2017)

Real Time 3D Depth Estimation and

Measurement of Un-calibrated Stereo and
Thermal Images
Dr.J L Mazher Iqbal Shaik Shakir Basha
Professor, ECE Department ECE Department
Madanapalle Institute of Technology and Science Madanapalle Institute of Technology and Science
Madnapalle, India Madnapalle, India
[email protected] [email protected]

Abstract - The research topics in the areas of recognition and collision prevention depend on
photogrammetry and computer vision to Model 3D accurate scene analysis. Currently existing approaches
objects from image sequences are significant for many for stereo vision are classified into two categories such
years. This paper proposes rectification of un-calibrated as monocular sequences and binocular or trinocular
stereo images and real time depth estimation and sequences or stereo sequences according to the
measurement of objects using Intel compute stick. The dimension of data available. One image at a time is
proposed system uses a pair of hand held USB camera classified as monocular. Two or three images at each
with unknown focal length to visualize un-calibrated instant are classified as bi- or trinocular sequences, or
images and thermal camera (Testo 870-1). The proposed
stereo sequences [1]. The stereo vision consists of
system uses sum of absolute difference (SAD) algorithm
to match the strongest points feature in the stereo images.
active and passive system. The active stereo system
The correctly matching points between the montage such as ultra sound and laser gives depth directly. In
images must satisfy the epipolar constraints. The passive stereo system algorithms such as sum of
algorithm computes the epipolar fundamental matrix absolute difference (SAD), random sample consensus
and obtains inliers point in order to get a 3D rectified (RANSAC) is used for binocular or trinocular images
stereo image. Additionally the propose algorithm stereo system to estimate depth. 3-Dimension objects
perform depth estimation to measure real time distance are in high demand for the development of computer
of objects dynamically. The proposed system uses Intel technology. The hardware requirement for active
computer stick with windows 8.1, video graphics array stereo requires high cost equipment such as 3D laser
(VGA) to high-definition multimedia interface (HDMI) scanner. The stereo vision analysis with calibration
converter, webcam, thermal camera, MATLAB 14a method seems to be complex process which does not
software. The proposed system finds many applications satisfies time and space limit [2]. This limitation is
such as fusion of real virtual scenes, robotic vision, traffic overcome by using uncelebrated stereo image
scene analysis, motion tracking and object tracking.
sequence. In realistic 3D models from uncalibrated
Thus the proposed system estimate accurate real time
image sequences is a hot research aspect in computer
distance of object and obtain 3D rectified stereo images.
graphics and computer vision fields because of less
Keywords-un-calibrated stereo images, depth inexpensive and simpler hardware. The oriented
estimation, epipolar constraint, Intel compute stick, cameras capture uncaliberated images. The
webcam, thermal camera. corresponding points in between the two images are
obtained from any one of the three methods such as
1. INTRODUCTION 2D-2D relative orientation (also called recovery of
Movement is an inherently continuous process. motion), 2D-3D image orientation (or space resection
The relative motions between objects in a scene and or camera calibration) and 3D-3D absolute orientation
cameras, between different moving objects in the (3D similarity transformation). 2D-3D image
scene produce the apparent motions of objects orientation are used in the proposed system [3]. On the
between frames. The objective of motion analysis is to critical configurations of points in 3D object space,
infer the real motion of objects in the scene from the i.e.Center of perspective and points on the object
changes between adjacent frames. To determine an surface. The relationship between the world
object‘s distance relative to an observer (sensor) is a coordinates and the pixel coordinates is linear
common field of research in Computer Vision and projective. Epipolar geometry concept is used to
Image Analysis. A growing number of modern reduce geometry in the captured image which cut
applications such as position determination, object down the complexity of the stereo correspondence

978-1-5090-2794-1/17/$31.00 ©2017 IEEE

2017 International Conference on Nascent Technologies in the Engineering Field (ICNTE-2017)

process. The 3-D objects are reconstructed after

matching between stereo correspondence points. The Z
depth is estimated from the difference between the
images of left and right eyes. This makes use of two
P
cameras in different positions: binocular stereovision.
However, for a given point in one image, there exist
probably many homologous points in the other image
using the geometric constraint such as epipolar X1 X2
constraint. Stereo viewing can be very beneficial and
enhance performance on a wide variety of tasks that
are visually demanding and requiring accurate depth f
perception.
L R X
II. STEREO VISION
Stereoscopic vision is a binocular vision technique Fig.1. Geometry of stereoscopic vision
in which a scene is represented by two distinct images.
In living organisms, there are many benefits of having In Fig.1 the 3D location P of any visible point in the
binocular vision. For example, the presence of two scene is the projection of the two point of the camera
eyes provides sight even if one eye is damaged. laying on the straight line passes through the optical
Additionally, stereoscopic vision allows for center. Triangulation method is used in many
stereopsis, which is the ability to discriminate depth applications such as surveying, navigation, metrology,
from the parallax images originating from separate astrometry, binocular vision and gun direction of
locations Stereopsis. This concept motivates to use weapons. Complex triangulation problems involving
two cameras in the proposed system. Two eyes or real-world observations with errors require the
cameras looking at the same scene from different
solution of large system.
perspectives provide a mean for determining three-
dimensional shape and position. Scientifically this is
B. Epipolar Constraints
called stereo vision, stereopsis or machine vision. It
has a rich history in psychology, biology and more Fig.2 shows the epipolar geometry of a normal
recently, in the computational model of perception. image pair. The points P1 (x1,y1) and P2 (x2, y2) are
corresponding point in 2D image I1 and I2
A. Traingulation respectively. The epipolar geometry of simple two
Once stereo images have been brought into point- image planes identical and parallel to the base line. As
to-point correspondence, the depth estimation by the point P1 and P2 lie on the same axis, the 2D image
triangulation is straightforward. The simplest is reduced 1D epipolar geometry. The stereo matching
stereoscopic vision is the positioning of cameras along is simplified as relative orientation of camera is known
one axis or translation axis and same angle as shown in 2D search space. In general the cameras are not
in Fig.1. Thus, there is no rotational relationship always placed in normal and simple orientation
between the two cameras, and so the images differ because this position may not lead to big enough
only by a shift system. In the coordinate system, the X disparities to calculate accurate depth information.
and Y world coordinates are those which reside in the The general stereo setup has arbitrary positions and
plane parallel to the image plane. The Z coordinate is orientations of the cameras each of which view a
the range from the camera to the object of interest. The significant sub volume of the object. Fig.3 shows
coordinates are determined using physical parameters epipolar geometry of general image pair overcomes
of the system. the limitation of normal image pair to estimate depth.
The plane in Fig.3 contains 3D point P and two optical
In particular, the baseline separation, b, between the centers c1, c2 and two image points p1,p2 to which p
two cameras, and also the focal length, f, of the projects is epipolar plane. Two lines e1and e2 results
cameras is vital in uncalibrated stereo image. X and Y intersection of epipolar plane with two images planes
world coordinate are determined from equation (1) and I1, I2 called epipolar lines constraints. The constraint
(2). In the given equation (1) and (2), the uppercase states a pair of points in the scene and their
letters represent world coordinates, and lowercase corresponding projections of two image, ordering
letters represent image coordinates. constraints states are identified. If these points lie on a
continuous surface in scene, they are ordered in same
X=x (1) way along the epipolar lines in each of the images
because of exact matching points in the images on
same 3D surface is unknown. The above method helps
Y= y1 (2) in finding potential matching.
2017 International Conference on Nascent Technologies in the Engineering Field (ICNTE-2017)

The depth estimation is obtained from rectified image.

The accuracy and depth estimation is clearly analyses
during night using thermal camera.

Read stereo image pair

Detect speeded up robust

features(SURF) Features In gray
Fig. 2. Epipolar geometry of a normal image pair scale image

Extract interest points

Find matching image features

Estimate Fundamental matrix from

corresponding points in stereo images

A rectified image appears

Fig.3. Epipolar geometry of general image pair

III. PROPOSED ALGORITHM Obtain real time distance

The proposed algorithm for depth estimation is measurement and depth estimation
shown in Fig. 4. The interesting static real scene is
captured by pair of web cameras. The baseline Fig.4. Flowchart of proposed algorithm
between the two cameras remains on the same axis.
The captured images I1 and I2 are converted from
color to gray scale features. The matching process in
the proposed algorithm required gray scale. The A. Hardware component in the proposed system
captured images are projected onto a common image Fig.11 shows the hardware setup of the proposed
plane. This process is called rectification. The system. Each component of the proposed system are
rectification is to aligning the capture images such as as follows
corresponding points appear in the same rows in both
1) Intel Compute Stick
images. The rectification process requires a set of
point correspondences between the two images. To The Intel computer stick is a new
generate these correspondences, point of interest in generation of computing that transforms any HDMI
both the images are collected and then potential (High Definition Multimedia Interface) display into a
matches between them are selected. The proposed fully functional computer. Intel compute stick is ready
algorithm further detect and extract SURF (speeded up to go out of the box and offers the performance, quality
robust features) features. The extracted interest points and value expected from intel. It is preinstalled with
in both images are matched. The matched points are windows 8.1 or ubuntu get a complete experience on a
corrected by epipolar constraint using fundamental ultra-small power efficient device that is just ‘4’
matrix. The fundamental matrix is computed using inches long. The power and reliability of a quad core
inlier points to meet epipolar constraint. The
Intel atomTM processor with built in wireless
corresponding points appear on the same row and
rectified image appears. In order to visualize rectified connectivity on board storage and a micro SD card slot
image red-cyan stereo glasses is used. The 3D glasses for additional storage. The computer stick is
are used to get 3D effect and minimize the overlap essentially a complete PC but in the compact form
region which appears in a corrected image. The depth factor of a slightly large pen drive. It connects a TV
of rectified image for real time distance measurement via HDMI. It has to be powered with micro USB port.
is estimated by the process of disparity map.
Additionally the same images are captured using 2) HDMI to VGA
thermal camera, strongest point’s selection using HDMI to VGA convertor is shown in Fig.6 (a). It
SURF, SAD and inlinear point to match the features. is used to connect an HDMI source laptop, PC, ultra-
2017 International Conference on Nascent Technologies in the Engineering Field (ICNTE-2017)

book, to a VGA display like projector, LCD monitor Step 4: A Matlab compiler popups, add .m file as
etc., convert the HDMI signal to VGA output. It main file and select the run time download from web,
doesn’t require any external power source. HDMI then apply package to create EXE file
female to female coupler is shown in Fig.6 (b). It is 2) Run EXE File In Intel Compute Stick
used to interface with HDMI to VGA cable to transfer
In this model hardware (Intel compute stick)
data from an HDMI-compliant source device to a
connected to monitor through HDMI to VGA and then
compatible computer monitor device. HDMI female to female convertor is connected
computing stick port. In monitor select standalone
package, then select testing folder in which run EXE
application file. Real time images capture in webcams
and rectified images can be seen in monitor as shown
in Fig.12, Fig.13 and Fig.14.
3) Real Time Measurement And Depth Estimation
Fig.5. Intel compute stick
The rectification of image depth estimation is
analyzed in section 3, disparity map is used to find the
3) Webcam and thermal camera
depth of the particular object in the scene, nearer and
Webcam and thermal camera is used to capture the far in the rectified image. The depth is estimate by
real time images subtracting the background of left and right image
which give a binary image with threshold value added
in them. The clarity is obtained by adjusting the
threshold value. For distance calculation webcams
separated with 15 cm distance is considered as
baseline to get the clear image. Table 1 shows the real
distance measurement of true values at room
condition. Now known distance of the object at
different positions (10 …300 cm) is calculated and the
corresponding values are noted. Apply values in spline
(a) (b) function which compares the actual value and known
Fig.6. Converters (a) Hdmi to vga converter (b) Hdmi female to values which gives the real time distance of object in
female coupler
the specified area.

(a) (b)

Fig.7.Camera (a) USB webcam (b) Thermal Camera

Fig.8.Deploy tool
IV. EXPERIMENTAL RESULTS
In this model there are mainly ‘4’ modules as given
below:
1) Generating EXE file from MATLAB
Sequence to generating .m to EXE file:
Step 1: complete the simulation of .m file
Step 2: apply deploy tool command in the command
window-Matlab
Step 3: Select application compiler package
MATLAB program for deployment as standalone Fig.9. Upload .m file
applications.
2017 International Conference on Nascent Technologies in the Engineering Field (ICNTE-2017)

Fig.10.Package generates as output file

Fig.14.Stereo rectified image

Fig.15.Depth map of rectified image

Fig.11. Proposed system-hardware setup intel computer stick
display output on monitor TABLE1. REAL DISTANCE MEASURED TRUE VALUES AT
ROOM CONDITION

Serial Reality distance (cm) Distance obtain Intel

number compute stick (cm)
1 5 5.6
2 7 7.8
3 9 9.7
4 15 16
5 20 21.1
6 25 26.3
7 30 31.5

Fig.12. Hardware setup for thermal camera 4) Real Time Measurement And Depth
Estimation using thermal camera

Depth Estimation is further clearly and accurately

developed with advance thermal camera Testo ( 870-
1) with it experimental results shown below.

Fig.13. Stereo montage of left and right image

Fig.16.Real time night scenecapture in thermal camera

2017 International Conference on Nascent Technologies in the Engineering Field (ICNTE-2017)

V. CONCLUSION
The proposed system performs the rectification of
un-calibrated stereo images, real time depth estimation
and real time distance measurement using stereo
images, Intel compute stick, webcam and thermal
camera. The algorithm uses SAD, triangulation
method, epipolar constraints and disparity map for 3D
rectification, depth estimation and distance
Fig.17.Real time person capture in thermal camera measurement of object. The real time applications of
the proposed system are fusion of real and virtual
scenes, robotic vision, traffic scene analysis, motion
tracking and object tracking. The proposed system is
effective in finding the distance of moving objects
based on color and feature matching. It also estimate
accurate real time distance of the object and obtain 3D
rectified stereo images using thermal camera.
ACKNOWLEDGEMENT
This work is funded by UGC, Major Research
Project, MRP-MAJOR-ELEC-2013-19701.
Fig.18. Stereo montage of left and right image
REFERENCES
[1] Z.zhang and O.Faugers, “3D Dynamics Scene Analysis”,
Springer Series in Information Sciences, Springer-Verlag,
Berlin Heidelberg GmbH, 1992.
[2] M.Bennamoun and G.j.Mamic. “Object Recognition
Fundamentals & Case Studies” , Springer Series in Advances
in Pattern Recognition ,Springer-Verlag London Ltd.2001.
[3] A.Gruen and T.s.Huang(Eds).” Calibration and Orientation of
Cameras in Computer vision”. Springer Series in Information
Sciences, Springer-Verlag, Berlin Heidelberg GmbH,2001.
[4] Y. S. Kang, and Y. S. Ho, "An efficient image rectification
Fig.19.Stereo rectified image method for parallel multi-camera arrangement,", in IEEE
transaction on Consumer Electronics, vol. 57, no. 3, pp.1041-
1048, Aug.2011.
[5] Ravindraiah, Dr. J. L. Mazher Iqbal (2014), Hard Exudates
Detection in Proliferative Diabetic Retinopathy using
Gradient Controlled Fuzzy C MEANS Clustering Algorithm,
GESJ: Computer Science and Telecommunications,
VolNo.4(44), PP 27-31.
[6] R. Ravindraiah, Dr. J. L. Mazher Iqbal (2014), Segmentation of
Diabetic Retinopathy Images through a Subjective
Approach, GESJ: Computer Science and
Telecommunications, VolNo.4 (44), PP 32-34, 2014.
[7] T.Hareesh, J. L. Mazher Iqbal, “ARM 7 based Intelligent
System for Surveillance and Street Light Management”,
International Journal of Computer Applications, volume 107 –
Fig.20. Real time window scene capture in thermal camera No 9, PP 12-16, December 2014.
[8] J.L.Mazher Iqbal, J.Lavanya and S.Arun, “Abnormal Human
Activity Recognition using Scale Invariant Feature
Transform” International Journal of Current Engineering and
Technology, Vol.5, No.6, PP 3748-3751, December 2015.
[9] J.L. Mazher Iqbal, M. Suriya ParveenS. Arun, “Image stitching
and 2D to 3D Image Reconstruction for Abnormal Activity
Detection”, International Journal of Computer Applications,
Volume 133 – No.17, PP 1-7 January 2016.

(a) (b)

Fig.21. Stereo images (a) Stereo montage of left and right image (b)
Stereo rectified image