Autonomous Generation, Segmentation, and Categorization of Point

Clouds
David Skiff, Computer Science in the College of Engineering, 2012, [email protected]

Abstract— Microsoft Kinect dynamically provides texture solution is by far the most popular. However, with the added
mapped 3-dimensional point clouds. Combining this with new information provided by the Kinect, we can locate landmarks
Simultaneous Localization and Mapping techniques, fast gen- in a 3-dimensional space accurately, and then use RANSAC
eration of detailed, fully textured, 3-dimensional point clouds
of an environment, and the Kinect’s location in it, can be to estimate transformations. This allows us to skip the use
found on the fly. One application of the this new ability is of EKF, since mapping can be effectively performed without
to the segmentation and categorization of an environment, and knowing the location or transformation ahead of time. This
the generation of a simple floor plan. The projects goal is to is used by RGBD-SLAM [2], and has provided impressive
generate a simple, reasonably fast set of tools that segments results with the Kinect. However, the method is in its early
simple Point Clouds into seperate objects, assigns them basic
categories (movable and unmovable), and generates a simple stages, and no paper is available on it, yet. SLAM methods
blueprint for the environment. The results can be used for often are used to generate Occupancy Maps, gradient maps
navigation, or tasks such as moving around and sorting objects indicating the perceived cost of traveling over a region. [9]
in a room. Another possible use is the detection of doorways. In
the future, more work on the segmentation algorithms would III. A PPROACH
be desired.
The project is entirely implemented in the Robot Operating
I. INTRODUCTION System (ROS), which can be found at www.ros.org. The
The recent introduction of Microsoft’s Kinect has cre- project is broken up into seperate steps, and the results from
ated a significant amount of research into non-uniform 3- each are piped between the them (see Fig. 2).
dimensional point cloud manipulation and analysis. This
project applies several known robotics methods for both
2-dimensional and 3-dimensional problems to the task of
Simultaneous Localization and Mapping (SLAM), Mesh
Segmentation, and 3-Dimensional Object Classification.
In robotics, its often valuable to be able to dynamically
discover an environment, produce a map of the environment,
and locate objects with-in it. Navigation and Control are two Fig. 1. Data Path for Project
of the most important aspects of robotics. Having a map
of the environment, the robots location in the environment,
A. RGBD-SLAM
and the location and class of the objects with-in it, provide
the robot with invaluable information for both of the tasks. SLAM is usually difficult do to the classic chick or the egg
Indeed, the first localization and mapping are the focus of problem, where you need to the position and transformation
SLAM, an important and heavily researched field of robotics. to map, and you need the map in order to locate yourself.
As a result, this project seeks to generate a 2-dimensional However, by using the method from [2], the mapping is
blueprint of a room dynamically. Ideal it would contain performed by estimating the transformation using RANdom
immovable objects, as well as walls, and doorways. This SAmple Consensus (RANSAC) [4]. This removes the mutual
differs Occupancy grid method used by most SLAM projects. requirement, and significantly simplifies the problem. The
The project seeks to map walls, and specific points where it solution from [2] involves the use of Speeded Up Rubust
cannot walk, but allows for moving of objects. Features (SURF) [3] to generate landmarks which are then
projected into 3-dimensional space. For each frame received
II. R ELATED W ORK from the kinect, the transformation from the previous states
There’s a lot of prior work in SLAM. The process of is estimated using RANSAC. This allows the creation of a
performing SLAM is usually broken up into five separate graph connecting camera viewpoints between frames. Lo-
problems: State Estimation, State Update, Landmark Extrac- cation is easy to derive, since we have depth information
tion, Landmark Update, and Data Association. There are from the camera. [2] Each point cloud is combined into a
several solutions already available, most notably ones based cumulative Point Cloud.
on extended Kalman Filters [1] (or EKF-SLAM) and based
on particle filters [1] (or fastSLAM). Both methods provide B. Segmentation
a means for estimating the location of the robot based on In order to achieve all of the goals (identify movable ob-
transformations in the landmarks after movement. The EKF jects, identify doors, and create a blueprint) we segment the
object into walls and seperate objects. There’s a significant
amount of noise and excessive clustering of pixels, so a
Gaussian Filter is used to remove noise, and a Voxel Grid
is used to down sample the data. While most research for
segmentation has been done for 2-dimensions, there is some
prior work for 3-dimensional, or mesh, segmentation. How-
ever, most of it requires uniform, non-noisy, Point Clouds. As
a result, we use Planar and k-Means Clustering Segmentation
[6]. We assume the floor is a large, flat plan, and that the
walls are large planes perpendicular to the floor. Thus, we
fit planes to the points at the specific orientations using
PCL’s provided parallel plane consensus segmentation which
uses RANSAC [4]. Once the floors and walls have been
removed, k-Means Clustering is used to segment the objects
with Euclidean Distance is used as the distance function.
The result is a set of expected wall, floor, and object Point
Fig. 2. From [8]. (a) A range image of an example scene with an
Clouds. armchair in the front. The black cross marks the position of an interest
point. (b): Visualization how the descriptor is calculated. The top shows
C. Categorization a range value patch of the top right corner of the armchair. The actual
descriptor is visualized on the bottom. Each of the 20 cells of the descriptor
The walls and floors are already categorized for us, so corresponds to one of the beams (green) visualized in the patch, with two
what remains is to classify the objects as either movable or of the correspondences marked with arrows. The additional (red) arrow
pointing to the top right shows the extracted dominant orientation. (c): The
immovable. While a number of methods could potentially descriptor distances to every other point in the scene (the brighter the higher
be used for this, this project decided to use a Bag of Key- the value). Note that mainly top right rectangular corners get low values.
points approach [7]. First, a suitable feature extractor must
be chosen. We need something that’s scale, position, and
rotation invariant, such that we can merely compare features IV. R ESULTS
any point in an image and calculate distance. While there
RGBD-SLAM is produces incredible results. However,
are quite a few methods available for 2-dimensional image,
it’s incredibly buggy, and combined with rviz’s crash sus-
3-dimensional feature extractors are more scarse, NARF was
ceptability, long runs were somewhat restricted with on the
used by this project. Normal Aligned Radial Feature NARF
fly viewing (or even without). Regardless, full point clouds
(NARF) [8] is a rotation invariant feature extractor. NARF
of several rooms were generated successfully, and basic
first extracts interest points, and then generates a feature
blueprints were successfully generated. However, object clas-
descriptor at that point. It does so by:
sification failed to perform anywhere near accurately. This
• calculating a normal aligned range value patch in the
is likely due to too small of training set being generated.
point, which is a small range image with the observer Excluding the objects, the blueprints themselves were qual-
looking at the point along the normal, itatively good.
• overlay a star pattern onto this patch, where each beam
corresponds to a value in the nal descriptor, that captures V. CONCLUSIONS AND FUTURE WORKS
how much the pixels under the beam change,
• extract a unique orientation from the descriptor, A. Conclusions
• and shift the descriptor according to this value to make RGBD-SLAM provides an excellent, albeit noisy and non-
it invariant to the rotation. uniform, Point Cloud, which can be suitably filtered and seg-
(from [8]). Now that we have a feature extractor, we simply mented to generate basic blueprints of a room. While object
create test sets and extract features from them. Identify the classification isn’t working correctly, yet. The generation of
most a larger training set would likely elleviate this issue.

D. Map B. Future Works

We know have a collection of wall point clouds, a col- Generate a larger training set for the object classification.
lection of object point clouds, and a floor and roof point At the time of writing this, the object classification fails to
cloud. In order to generate the blueprint, we project all walls perform well, and is likely due to limited training data.
and immovable objects down to 2-dimensions, and then use The current method of object segmentation, k-Means,
linear regression to fit lines to the clouds. Then, collision allows for simple non-contiguous segmentation. However, if
points between the lines are taken as corners. The resulting objects touch, or overlap, then the k-Means identifies the the
map is output as a collection of endpoints for the walls, and seperate elements as a single object. There are alternative
the convex hulls of the projection of each individual object Mesh Segmentation algorithms available, like that suggested
cloud. by [5], but are significantly more arduous to implement.
 It was an initial goal of this project to identify doorways
from the resulting blueprint. However, time was too much
of a constraint. However, since we’d only be looking at the
wall blueprint, We can search around the endpoints of lines,
identify gaps, and match them against the size restrictions
for a door.
R EFERENCES
[1] Durrant-Whyte, H., and T. Bailey. ”Simultaneous Localization and
Mapping: Part I.” IEEE Robotics & Automation Magazine 13.2 (2006):
99-110. Print.
[2] Ruchti, Philipp, Felix Endres, Juergen Hess, Nikolas
Engelhard, Juergen Sturm, Wolfram Burgard, and Daniel
Kuhner. ”RGBD-6D-SLAM.” ROS Wiki. Web. 03 Apr. 2011.
¡http://www.ros.org/wiki/openni/ Contests/ROS 3D/RGBD-6D-
SLAM¿
[3] Bay, H., A. Ess, T. Tuytelaars, and L. Vangool. ”Speeded-Up Robust
Features (SURF).” Computer Vision and Image Understanding 110.3
(2008): 346-59. Print.
[4] Bolles, Robert C., and Martin A. Fischler. ”Random Sample Con-
sensus: a Paradigm for Model Fitting with Applications to Image
Analysis and Automated Cartography.” Communications of the ACM
24.6 (1981). Print.
[5] Jagannathan, Anupama. ”Segmentation and Recognition of 3D Point
Clouds within Graph-theoretic and Thermodynamic Frameworks.”
Thesis. Northeastern University, 2005. Print.
[6] Lloyd, S. ”Least Squares Quantization in PCM.” IEEE Transactions
on Information Theory 28.2 (1982): 129-37. Print.
[7] Csurka, Gabriella, Christopher Dance, Lixin Fan, Jutta Willamowski,
and Cdric Bray. Visual Categorization with Bags of Keypoints. Xerox
Research Centre Europe. Print.
[8] Bastion Steder,Radu Bogdan Rusu, Kurt Konolige, Wolfram Burgard
”NARF: 3D Range Image Features for Object Recognition” (2010)
[9] Mark Whitty, Stephen Cossell, Kim Son Dang, Jose Guivant and
Jayantha Katupitiya ”Autonomous Navigation using a Real-Time 3D
Point Cloud”