Point Transformer

Hengshuang Zhao1,2 Li Jiang3 Jiaya Jia3 Philip Torr1 Vladlen Koltun4

1
University of Oxford 2 The University of Hong Kong
3
The Chinese University of Hong Kong 4 Intel Labs
arXiv:2012.09164v2 [cs.CV] 26 Sep 2021

Abstract

n
ti o
Self-attention networks have revolutionized natural lan- airplane lamp

ca
bed

fi
ssi
guage processing and are making impressive strides in im-

cla
age analysis tasks such as image classification and object part
detection. Inspired by this success, we investigate the ap- segmentation

plication of self-attention networks to 3D point cloud pro-

semgme
se
an ntat
cessing. We design self-attention layers for point clouds and

tic ion
use these to construct self-attention networks for tasks such
as semantic scene segmentation, object part segmentation, Point Transformer
and object classification. Our Point Transformer design im-
proves upon prior work across domains and tasks. For ex-
ample, on the challenging S3DIS dataset for large-scale se-
Figure 1. The Point Transformer can serve as the backbone for var-
mantic scene segmentation, the Point Transformer attains
ious 3D point cloud understanding tasks such as object classifica-
an mIoU of 70.4% on Area 5, outperforming the strongest tion, object part segmentation, and semantic scene segmentation.
prior model by 3.3 absolute percentage points and crossing
the 70% mIoU threshold for the first time.
in natural language processing [39, 45, 5, 4, 51] and image
analysis [10, 28, 54]. The transformer family of models is
1. Introduction particularly appropriate for point cloud processing because
3D data arises in many application areas such as au- the self-attention operator, which is at the core of trans-
tonomous driving, augmented reality, and robotics. Unlike former networks, is in essence a set operator: it is invariant
images, which are arranged on regular pixel grids, 3D point to permutation and cardinality of the input elements. The
clouds are sets embedded in continuous space. This makes application of self-attention to 3D point clouds is therefore
3D point clouds structurally different from images and pre- quite natural, since point clouds are essentially sets embed-
cludes immediate application of deep network designs that ded in 3D space.
have become standard in computer vision, such as networks We flesh out this intuition and develop a self-attention
based on the discrete convolution operator. layer for 3D point cloud processing. Based on this layer,
A variety of approaches to deep learning on 3D point we construct Point Transformer networks for a variety of
clouds have arisen in response to this challenge. Some vox- 3D understanding tasks. We investigate the form of the self-
elize the 3D space to enable the application of 3D discrete attention operator, the application of self-attention to local
convolutions [23, 32]. This induces massive computational neighborhoods around each point, and the encoding of po-
and memory costs and underutilizes the sparsity of point sitional information in the network. The resulting networks
sets in 3D. Sparse convolutional networks relieve these limi- are based purely on self-attention and pointwise operations.
tations by operating only on voxels that are not empty [9, 3]. We show that Point Transformers are remarkably effec-
Other designs operate directly on points and propagate in- tive in 3D deep learning tasks, both at the level of detailed
formation via pooling operators [25, 27] or continuous con- object analysis and large-scale parsing of massive scenes.
volutions [42, 37]. Another family of approaches connect In particular, Point Transformers set the new state of the art
the point set into a graph for message passing [44, 19]. on large-scale semantic segmentation on the S3DIS dataset
In this work, we develop an approach to deep learning on (70.4% mIoU on Area 5), shape classification on Model-
point clouds that is inspired by the success of transformers Net40 (93.7% overall accuracy), and object part segmenta-
tion on ShapeNetPart (86.6% instance mIoU). Our full im- putation and memory costs due to the cubic growth in the
plementation and trained models will be released upon ac- number of voxels as a function of resolution. The solution
ceptance. In summary, our main contributions include the is to take advantage of sparsity, as most voxels are usually
following. unoccupied. For example, OctNet [29] uses unbalanced
octrees with hierarchical partitions. Approaches based on
• We design a highly expressive Point Transformer layer
sparse convolutions, where the convolution kernel is only
for point cloud processing. The layer is invariant
evaluated at occupied voxels, can further reduce computa-
to permutation and cardinality and is thus inherently
tion and memory requirements [9, 3]. These methods have
suited to point cloud processing.
demonstrated good accuracy but may still lose geometric
• Based on the Point Transformer layer, we construct detail due to quantization onto the voxel grid.
high-performing Point Transformer networks for clas- Point-based networks. Rather than projecting or quantiz-
sification and dense prediction on point clouds. These ing irregular point clouds onto regular grids in 2D or 3D, re-
networks can serve as general backbones for 3D scene searchers have designed deep network structures that ingest
understanding. point clouds directly, as sets embedded in continuous space.
• We report extensive experiments over multiple do- PointNet [25] utilizes permutation-invariant operators such
mains and datasets. We conduct controlled studies to as pointwise MLPs and pooling layers to aggregate features
examine specific choices in the Point Transformer de- across a set. PointNet++ [27] applies these ideas within a
sign and set the new state of the art on multiple highly hierarchical spatial structure to increase sensitivity to local
competitive benchmarks, outperforming long lines of geometric layout. Such models can benefit from efficient
prior work. sampling of the point set, and a variety of sampling strate-
gies have been developed [27, 7, 46, 50, 11].
2. Related Work A number of approaches connect the point set into
a graph and conduct message passing on this graph.
For 2D image understanding, pixels are placed in regu- DGCNN [44] performs graph convolutions on kNN graphs.
lar grids and can be processed with classical convolution. PointWeb [55] densely connects local neightborhoods.
In contrast, 3D point clouds are unordered and scattered ECC [31] uses dynamic edge-conditioned filters where con-
in 3D space: they are essentially sets. Learning-based ap- volution kernels are generated based on edges inside point
proaches to processing 3D point clouds can be classified clouds. SPG [15] operates on a superpoint graph that rep-
into the following types: projection-based, voxel-based, and resents contextual relationships. KCNet [30] utilizes kernel
point-based networks. correlation and graph pooling. Wang et al. [40] investigate
Projection-based networks. For processing irregular in- the local spectral graph convolution. GACNet [41] employs
puts like point clouds, an intuitive way is to transform ir- graph attention convolution and HPEIN [13] builds a hierar-
regular representations to regular ones. Considering the chical point-edge interaction architecture. DeepGCNs [19]
success of 2D CNNs, some approaches [34, 18, 2, 14, 16] explore the advantages of depth in graph convolutional net-
adopt multi-view projection, where 3D point clouds are pro- works for 3D scene understanding.
jected into various image planes. Then 2D CNNs are used A number of methods are based on continuous convolu-
to extract feature representations in these image planes, fol- tions that apply directly to the 3D point set, with no quan-
lowed by multi-view feature fusion to form the final output tization. PCCN [42] represents convolutional kernels as
representations. In a related approach, TangentConv [35] MLPs. SpiderCNN [49] defines kernel weights as a fam-
projects local surface geometry onto a tangent plane at ev- ily of polynomial functions. Spherical CNN [8] designs
ery point, forming tangent images that can be processed by spherical convolution to address the problem of 3D rota-
2D convolution. However, this approach heavily relies on tion equivariance. PointConv [46] and KPConv [37] con-
tangent estimation. In projection-based frameworks, the ge- struct convolution weights based on the input coordinates.
ometric information inside point clouds is collapsed during InterpCNN [22] utilizes coordinates to interpolate point-
the projection stage. These approaches may also underuti- wise kernel weights. PointCNN [20] proposes to reorder the
lize the sparsity of point clouds when forming dense pixel input unordered point clouds with special operators. Um-
grids on projection planes. The choice of projection planes menhofer et al. [38] apply continuous convolutions to learn
may heavily influence recognition performance and occlu- particle-based fluid dynamics.
sion in 3D may impede accuracy. Transformer and self-attention. Transformer and self-
Voxel-based networks. An alternative approach to trans- attention models have revolutionized machine translation
forming irregular point clouds to regular representations is and natural language processing [39, 45, 5, 4, 51]. This
3D voxelization [23, 32], followed by convolutions in 3D. has inspired the development of self-attention networks for
When applied naively, this strategy can incur massive com- 2D image recognition [10, 28, 54, 6]. Hu et al. [10] and
Ramachandran et al. [28] apply scalar dot-product self- where yi is the output feature. φ, ψ, and α are pointwise
attention within local image patches. Zhao et al. [54] de- feature transformations, such as linear projections or MLPs.
velop a family of vector self-attention operators. Dosovit- δ is a position encoding function and ρ is a normalization
skiy et al. [6] treat images as sequences of patches. function such as softmax. The scalar attention layer com-
Our work is inspired by the findings that transform- putes the scalar product between features transformed by φ
ers and self-attention networks can match or even outper- and ψ and uses the output as an attention weight for aggre-
form convolutional networks on sequences and 2D images. gating features transformed by α.
Self-attention is of particular interest in our setting because In vector attention, the computation of attention weights
it is intrinsically a set operator: positional information is is different. In particular, attention weights are vectors that
provided as attributes of elements that are processed as a can modulate individual feature channels:
set [39, 54]. Since 3D point clouds are essentially sets of
points with positional attributes, the self-attention mecha- \label {eq:vector} \ve {y}_{i} = \sum _{\xx _j \in \xX } \rho \big (\gamma (\beta (\varphi (\ve {x}_i), \psi (\ve {x}_j))+\delta )\big ) \odot \alpha (\ve {x}_{j}), (2)
nism seems particularly suitable to this type of data. We
thus develop a Point Transformer layer that applies self- where β is a relation function (e.g., subtraction) and γ is
attention to 3D point clouds. a mapping function (e.g., an MLP) that produces attention
There are a number of previous works [48, 21, 50, 17] vectors for feature aggregation.
that utilize attention for point cloud analysis. They apply Both scalar and vector self-attention are set operators.
global attention on the whole point cloud, which introduces The set can be a collection of feature vectors that represent
heavy computation and renders these approaches inapplica- the entire signal (e.g., sentence or image) [39, 6] or a collec-
ble to large-scale 3D scene understanding. They also utilize tion of feature vectors from a local patch within the signal
scalar dot-product attention, where different channels share (e.g., an image patch) [10, 28, 54].
the same aggregation weights. In contrast, we apply self-
attention locally, which enables scalability to large scenes 3.2. Point Transformer Layer
with millions of points, and we utilize vector attention, Self-attention is a natural fit for point clouds because
which we show to be important for achieving high accuracy. point clouds are essentially sets embedded irregularly in a
We also demonstrate the importance of appropriate position metric space. Our point transformer layer is based on vec-
encoding in large-scale point cloud understanding, in con- tor self-attention. We use the subtraction relation and add
trast to prior approaches that omitted position information. a position encoding δ to both the attention vector γ and the
Overall, we show that appropriately designed self-attention transformed features α:
networks can scale to large and complex 3D scenes, and
substantially advance the state of the art in large-scale point \label {eq:pointtransformer} \ve {y}_{i} = \sum _{\xx _j \in \xX (i)} \rho \big (\gamma (\varphi (\ve {x}_i) - \psi (\ve {x}_j) + \delta )\big ) \odot \big (\alpha (\ve {x}_{j}) + \delta \big ) (3)
cloud understanding.

3. Point Transformer Here the subset X (i) ⊆ X is a set of points in a local neigh-
borhood (specifically, k nearest neighbors) of xi . Thus we
We begin by briefly revisiting the general formulation of adopt the practice of recent self-attention networks for im-
transformers and self-attention operators. Then we present age analysis in applying self-attention locally, within a lo-
the point transformer layer for 3D point cloud processing. cal neighborhood around each datapoint [10, 28, 54]. The
Lastly, we present our network architecture for 3D scene mapping function γ is an MLP with two linear layers and
understanding. one ReLU nonlinearity. The point transformer layer is il-
lustrated in Figure 2.
3.1. Background
Transformers and self-attention networks have revolu- input: (x, p)
tionized natural language processing [39, 45, 5, 4, 51] and
have demonstrated impressive results in 2D image analy- 𝜑, 𝜓: linear 𝛿: mlp 𝛼: linear
sis [10, 28, 54, 6]. Self-attention operators can be classi-
fied into two types: scalar attention [39] and vector atten-
𝛾: m
tion [54]. lp
Let X = {xi }i be a set of feature vectors. The stan-
aggregation
dard scalar dot-product attention layer can be represented
as follows: output: (y, p)

\label {eq:scalar} \ve {y}_{i} = \sum _{\xx _j \in \xX } \rho \big (\varphi (\ve {x}_i)^\top \psi (\ve {x}_j) + \delta \big ) \alpha (\ve {x}_{j}), (1) Figure 2. Point transformer layer.
 (N, 32) (N/4, 64) (N/16, 128) (N/64, 256) (N/256, 512) (N/256, 512) (N/64, 256) (N/16, 128) (N/4, 64) (N, 32) (N, Dout)

Point Transformer

Label: chair
MLP
Transition Down
Global AvgPooling
Transition Up
(N, 32) (N/4, 64) (N/16, 128) (N/64, 256) (N/256, 512) (1, 512) (1, Dout)

Figure 3. Point transformer networks for semantic segmentation (top) and classification (bottom).

input: (x, p) input: (x, p1) input1: (x1, p1) input2: (x2, p2)

linear farthest point sampl. linear linear

point transformer kNN, mlp interpolation

linear local max pooling summation

output: (y, p) output: (y, p2) output: (y, p2)

(a) point transformer block (b) transition down (c) transition up

Figure 4. Detailed structure design for each module.

3.3. Position Encoding 3.4. Point Transformer Block

Position encoding plays an important role in self- We construct a residual point transformer block with the
attention, allowing the operator to adapt to local structure point transformer layer at its core, as shown in Figure 4(a).
in the data [39]. Standard position encoding schemes for The transformer block integrates the self-attention layer,
sequences and image grids are crafted manually, for exam- linear projections that can reduce dimensionality and ac-
ple based on sine and cosine functions or normalized range celerate processing, and a residual connection. The input
values [39, 54]. In 3D point cloud processing, the 3D point is a set of feature vectors x with associated 3D coordinates
coordinates themselves are a natural candidate for position p. The point transformer block facilitates information ex-
encoding. We go beyond this by introducing trainable, pa- change between these localized feature vectors, producing
rameterized position encoding. Our position encoding func- new feature vectors for all data points as its output. The
tion δ is defined as follows: information aggregation adapts both to the content of the
feature vectors and their layout in 3D.
\label {eq:positionencoding} \delta = \theta (\ve {p}_i - \ve {p}_j). (4)
3.5. Network Architecture
Here pi and pj are the 3D point coordinates for points i We construct complete 3D point cloud understanding
and j. The encoding function θ is an MLP with two linear networks based on the point transformer block. Note that
layers and one ReLU nonlinearity. Notably, we found that the point transformer is the primary feature aggregation op-
position encoding is important for both the attention gener- erator throughout the network. We do not use convolu-
ation branch and the feature transformation branch. Thus tions for preprocessing or auxiliary branches: the network is
Eq. 3 adds the trainable position encoding in both branches. based entirely on point transformer layers, pointwise trans-
The position encoding θ is trained end-to-end with the other formations, and pooling. The network architectures are vi-
subnetworks. sualized in Figure 3.
Backbone structure. The feature encoder in point trans- 4. Experiments
former networks for semantic segmentation and classifica-
tion has five stages that operate on progressively downsam- We evaluate the effectiveness of the presented Point
pled point sets. The downsampling rates for the stages are Transformer design on a number of domains and tasks. For
[1, 4, 4, 4, 4], thus the cardinality of the point set produced 3D semantic segmentation, we use the challenging Stanford
by each stage is [N, N/4, N/16, N/64, N/256], where N is the Large-Scale 3D Indoor Spaces (S3DIS) dataset [1]. For
number of input points. Note that the number of stages and 3D shape classification, we use the widely adopted Mod-
the downsampling rates can be varied depending on the ap- elNet40 dataset [47]. And for object part segmentation, we
plication, for example to construct light-weight backbones use ShapeNetPart [52].
for fast processing. Consecutive stages are connected by Implementation details. We implement the Point Trans-
transition modules: transition down for feature encoding former in PyTorch [24]. We use the SGD optimizer with
and transition up for feature decoding. momentum and weight decay set to 0.9 and 0.0001, respec-
tively. For semantic segmentation on S3DIS, we train for
Transition down. A key function of the transition down 40K iterations with initial learning rate 0.5, dropped by 10x
module is to reduce the cardinality of the point set as re- at steps 24K and 32K. For 3D shape classification on Mod-
quired, for example from N to N/4 in the transition from elNet40 and 3D object part segmentation on ShapeNetPart,
the first to the second stage. Denote the point set provided we train for 200 epochs. The initial learning rate is set to
as input to the transition down module as P1 and denote 0.05 and is dropped by 10x at epochs 120 and 160.
the output point set as P2 . We perform farthest point sam- 4.1. Semantic Segmentation
pling [27] in P1 to identify a well-spread subset P2 ⊂ P1
with the requisite cardinality. To pool feature vectors from Data and metric. The S3DIS [1] dataset for semantic scene
P1 onto P2 , we use a kNN graph on P1 . (This is the same parsing consists of 271 rooms in six areas from three differ-
k as in Section 3.2. We use k = 16 throughout and report ent buildings. Each point in the scan is assigned a semantic
a controlled study of this hyperparameter in Section 4.4.) label from 13 categories (ceiling, floor, table, etc.). Follow-
Each input feature goes through a linear transformation, fol- ing a common protocol [36, 27], we evaluate the presented
lowed by batch normalization and ReLU, followed by max approach in two modes: (a) Area 5 is withheld during train-
pooling onto each point in P2 from its k neighbors in P1 . ing and is used for testing, and (b) 6-fold cross-validation.
The transition down module is schematically illustrated in For evaluation metrics, we use mean classwise intersection
Figure 4(b). over union (mIoU), mean of classwise accuracy (mAcc),
and overall pointwise accuracy (OA).
Transition up. For dense prediction tasks such as seman- Performance comparison. The results are presented in Ta-
tic segmentation, we adopt a U-net design in which the bles 1 and 2. The Point Transformer outperforms all prior
encoder described above is coupled with a symmetric de- models according to all metrics in both evaluation modes.
coder [27, 3]. Consecutive stages in the decoder are con- On Area 5, the Point Transformer attains mIoU/mAcc/OA
nected by transition up modules. Their primary function is of 70.4%/76.5%/90.8%, outperforming all prior work by
to map features from the downsampled input point set P2 multiple percentage points in each metric. The Point Trans-
onto its superset P1 ⊃ P2 . To this end, each input point former is the first model to pass the 70% mIoU bar, outper-
feature is processed by a linear layer, followed by batch forming the prior state of the art by 3.3 absolute percent-
normalization and ReLU, and then the features are mapped age points in mIoU. The Point Transformer outperforms
onto the higher-resolution point set P1 via trilinear inter- MLPs-based frameworks such as PointNet [25], voxel-
polation. These interpolated features from the preceding based architectures such as SegCloud [36], graph-based
decoder stage are summarized with the features from the methods such as SPGraph [15], attention-based methods
corresponding encoder stage, provided via a skip connec- such as PAT [50], sparse convolutional networks such as
tion. The structure of the transition up module is illustrated MinkowskiNet [3], and continuous convolutional networks
in Figure 4(c). such as KPConv [37]. Point Transformer also substantially
outperforms all prior models under 6-fold cross-validation.
Output head. For semantic segmentation, the final decoder The mIoU in this mode is 73.5%, outperforming the prior
stage produces a feature vector for each point in the input state of the art (KPConv) by 2.9 absolute percentage points.
point set. We apply an MLP to map this feature to the final The number of parameters in Point Transformer (4.9M) is
logits. For classification, we perform global average pool- much smaller than in current high-performing architectures
ing over the pointwise features to get a global feature vec- such as KPConv (14.9M) and SparseConv (30.1M).
tor for the whole point set. This global feature is passed Visualization. Figure 5 shows the Point Transformer’s pre-
through an MLP to get the global classification logits. dictions. We can see that the predictions are very close to
 Method OA mAcc mIoU ceiling floor wall beam column window door table chair sofa bookcase board clutter
PointNet [25] – 49.0 41.1 88.8 97.3 69.8 0.1 3.9 46.3 10.8 59.0 52.6 5.9 40.3 26.4 33.2
SegCloud [36] – 57.4 48.9 90.1 96.1 69.9 0.0 18.4 38.4 23.1 70.4 75.9 40.9 58.4 13.0 41.6
TangentConv [35] – 62.2 52.6 90.5 97.7 74.0 0.0 20.7 39.0 31.3 77.5 69.4 57.3 38.5 48.8 39.8
PointCNN [20] 85.9 63.9 57.3 92.3 98.2 79.4 0.0 17.6 22.8 62.1 74.4 80.6 31.7 66.7 62.1 56.7
SPGraph [15] 86.4 66.5 58.0 89.4 96.9 78.1 0.0 42.8 48.9 61.6 84.7 75.4 69.8 52.6 2.1 52.2
PCCN [42] – 67.0 58.3 92.3 96.2 75.9 0.3 6.0 69.5 63.5 66.9 65.6 47.3 68.9 59.1 46.2
PAT [50] – 70.8 60.1 93.0 98.5 72.3 1.0 41.5 85.1 38.2 57.7 83.6 48.1 67.0 61.3 33.6
PointWeb [55] 87.0 66.6 60.3 92.0 98.5 79.4 0.0 21.1 59.7 34.8 76.3 88.3 46.9 69.3 64.9 52.5
HPEIN [13] 87.2 68.3 61.9 91.5 98.2 81.4 0.0 23.3 65.3 40.0 75.5 87.7 58.5 67.8 65.6 49.4
MinkowskiNet [37] – 71.7 65.4 91.8 98.7 86.2 0.0 34.1 48.9 62.4 81.6 89.8 47.2 74.9 74.4 58.6
KPConv [37] – 72.8 67.1 92.8 97.3 82.4 0.0 23.9 58.0 69.0 81.5 91.0 75.4 75.3 66.7 58.9
PointTransformer 90.8 76.5 70.4 94.0 98.5 86.3 0.0 38.0 63.4 74.3 89.1 82.4 74.3 80.2 76.0 59.3
Table 1. Semantic segmentation results on the S3DIS dataset, evaluated on Area 5.

Method OA mAcc mIoU 4.2. Shape Classification

PointNet [25] 78.5 66.2 47.6 Data and metric. The ModelNet40 [47] dataset contains
RSNet [12] – 66.5 56.5 12,311 CAD models with 40 object categories. They are
SPGraph [15] 85.5 73.0 62.1 split into 9,843 models for training and 2,468 for testing.
PAT [50] – 76.5 64.3 We follow the data preparation procedure of Qi et al. [27]
PointCNN [20] 88.1 75.6 65.4 and uniformly sample the points from each CAD model
PointWeb [55] 87.3 76.2 66.7 together with the normal vectors from the object meshes.
ShellNet [53] 87.1 – 66.8 For evaluation metrics, we use the mean accuracy within
RandLA-Net [37] 88.0 82.0 70.0 each category (mAcc) and the overall accuracy (OA) over
KPConv [37] – 79.1 70.6 all classes.
PointTransformer 90.2 81.9 73.5
Performance comparison. The results are presented in Ta-
Table 2. Semantic segmentation results on the S3DIS dataset, eval-
uated with 6-fold cross-validation.
ble 3. The Point Transformer sets the new state of the art in
both metrics. The overall accuracy of Point Transformer on
ModelNet40 is 93.7%. It outperforms strong graph-based
the ground truth. Point Transformer captures detailed se- models such as DGCNN [44], attention-based models such
mantic structure in complex 3D scenes, such as the legs of as A-SCN [48] and Point2Sequence [21], and strong point-
chairs, the outlines of poster boards, and the trim around based models such as KPConv [37].
doorways. Visualization. To probe the representation learned by the
Point Transformer, we conduct shape retrieval by retrieving
Method input mAcc OA nearest neighbors in the space of the output features pro-
3DShapeNets [47] voxel 77.3 84.7 duced by the Point Transformer. Some results are shown in
VoxNet [23] voxel 83.0 85.9 Figure 6. The retrieved shapes are very similar to the query,
Subvolume [26] voxel 86.0 89.2 and when they differ, they differ along aspects that we per-
MVCNN [34] image – 90.1 ceive as less semantically salient, such as legs of desks.
PointNet [25] point 86.2 89.2
A-SCN [48] point 87.6 90.0 4.3. Object Part Segmentation
Set Transformer [17] point – 90.4
Data and metric. The ShapeNetPart dataset [52] is anno-
PAT [50] point – 91.7
tated for 3D object part segmentation. It consists of 16,880
PointNet++ [27] point – 91.9
models from 16 shape categories, with 14,006 3D models
SpecGCN [40] point – 92.1
for training and 2,874 for testing. The number of parts for
PointCNN [20] point 88.1 92.2
each category is between 2 and 6, with 50 different parts
DGCNN [44] point 90.2 92.2
in total. We use the sampled point sets produced by Qi et
PointWeb [55] point 89.4 92.3
al. [27] for a fair comparison with prior work. For evalua-
SpiderCNN [49] point – 92.4
tion metrics, we report category mIoU and instance mIoU.
PointConv [46] point – 92.5
Point2Sequence [21] point 90.4 92.6 Performance comparison. The results are presented in Ta-
KPConv [37] point – 92.9 ble 4. The Point Transformer outperforms all prior mod-
InterpCNN [22] point – 93.0 els as measured by instance mIoU. (Note that we did not
PointTransformer point 90.6 93.7 use loss-balancing during training, which can boost cate-
Table 3. Shape classification results on the ModelNet40 dataset. gory mIoU.)
 Method cat. mIoU ins. mIoU Number of neighbors. We first investigate the setting of
PointNet [25] 80.4 83.7 the number of neighbors k, which is used in determining
A-SCN [48] – 84.6 the local neighborhood around each point. The results are
PCNN [42] 81.8 85.1 shown in Table 5. The best performance is achieved when
PointNet++ [27] 81.9 85.1 k is set to 16. When the neighborhood is smaller (k = 4
DGCNN [44] 82.3 85.1 or k = 8), the model may not have sufficient context for its
Point2Sequence [21] – 85.2 predictions. When the neighborhood is larger (k = 32 or
SpiderCNN [49] 81.7 85.3 k = 64), each self-attention layer is provided with a large
SPLATNet [33] 83.7 85.4 number of datapoints, many of which may be farther and
PointConv [46] 82.8 85.7 less relevant. This may introduce excessive noise into the
SGPN [43] 82.8 85.8 processing, lowering the model’s accuracy.
PointCNN [20] 84.6 86.1 Softmax regularization. We conduct an ablation study
InterpCNN [22] 84.0 86.3 on the normalization function ρ in Eq. 3. The perfor-
KPConv [37] 85.1 86.4 mance without softmax regularization on S3DIS Area5 is
PointTransformer 83.7 86.6 66.5%/72.8%/89.3%, in terms of mIoU/mAcc/OA. It is
Table 4. Object part segmentation results on the ShapeNetPart much lower than the performance with softmax regulariza-
dataset. tion (70.4%/76.5%90.8%). This suggests that the normal-
k mIoU mAcc OA ization is essential in this setting.
4 59.6 66.0 86.0 Position encoding. We now study the choice of the posi-
8 67.7 73.8 89.9 tion encoding δ. The results are shown in Table 6. We can
16 70.4 76.5 90.8 see that without position encoding, the performance drops
32 68.3 75.0 89.8 significantly. With absolute position encoding, the perfor-
64 67.7 74.1 89.9 mance is higher than without. Relative position encoding
Table 5. Ablation study: number of neighbors k in the definition yields the highest performance. When relative position en-
of local neighborhoods. coding is added only to the attention generation branch (first
term in Eq. 3) or only to the feature transformation branch
Pos. encoding mIoU mAcc OA (second term in Eq. 3), the performance drops again, in-
none 64.6 71.9 88.2 dicating that adding the relative position encoding to both
absolute 66.5 73.2 88.9 branches is important.
relative 70.4 76.5 90.8
Attention type. Finally, we investigate the type of self-
relative for attention 67.0 73.0 89.3
attention used in the point transformer layer. The results are
relative for feature 68.7 74.4 90.4
shown in Table 7. We examine four conditions. ‘MLP’ is
Table 6. Ablation study: position encoding. a no-attention baseline that replaces the point transformer
layer in the point transformer block with a pointwise MLP.
Operator mIoU mAcc OA ‘MLP+pooling’ is a more advanced no-attention baseline
MLP 61.7 68.6 87.1 that replaces the point transformer layer with a pointwise
MLP+pooling 63.7 71.0 87.8 MLP followed by max pooling within each kNN neighbor-
scalar attention 64.6 71.9 88.4 hood: this performs feature transformation at each point
vector attention 70.4 76.5 90.8 and enables each point to exchange information with its lo-
Table 7. Ablation study: form of self-attention operator. cal neighborhood, but does not leverage attention mecha-
nisms. ‘scalar attention’ replaces the vector attention used
in Eq. 3 by scalar attention, as in Eq. 1 and in the original
Visualization. Object part segmentation results on a num- transformer design [39]. ‘vector attention’ is the formula-
ber of models are shown in Figure 7. The Point Trans- tion we use, presented in Eq. 3. We can see that scalar at-
former’s part segmentation predictions are clean and close tention is more expressive than the no-attention baselines,
to the ground truth. but is in turn outperformed by vector attention. The per-
formance gap between vector and scalar attention is signif-
4.4. Ablation Study
icant: 70.4% vs. 64.6%, an improvement of 5.8 absolute
We now conduct a number of controlled experiments that percentage points. Vector attention is more expressive since
examine specific decisions in the Point Transformer design. it supports adaptive modulation of individual feature chan-
These studies are performed on the semantic segmentation nels, not just whole feature vectors. This expressivity ap-
task on the S3DIS dataset, tested on Area 5. pears to be very beneficial in 3D data processing.
 Input Ground Truth Point Transformer Input Ground Truth Point Transformer

ceiling floor wall beam column window door table chair sofa bookcase board clutter

Figure 5. Visualization of semantic segmentation results on the S3DIS dataset.

Figure 6. Visualization of shape retrieval results on the ModelNet40 dataset. The leftmost column shows the input query and the other
columns show the retrieved models.

Figure 7. Visualization of object part segmentation results on the ShapeNetPart dataset. The ground truth is in the top row, Point Trans-
former predictions on the bottom.

5. Conclusion former networks is fundamentally a set operator. We have

shown that beyond this conceptual compatibility, transform-
Transformers have revolutionized natural language pro- ers are remarkably effective in point cloud processing, out-
cessing and are making impressive gains in 2D image anal- performing state-of-the-art designs from a variety of fam-
ysis. Inspired by this progress, we have developed a trans- ilies: graph-based models, sparse convolutional networks,
former architecture for 3D point clouds. Transformers are continuous convolutional networks, and others. We hope
perhaps an even more natural fit for point cloud process- that our work will inspire further investigation of the proper-
ing than they are for language or image processing, be- ties of point transformers, the development of new operators
cause point clouds are essentially sets embedded in a metric and network designs, and the application of transformers to
space, and the self-attention operator at the core of trans- other tasks, such as 3D object detection.
A. Appendix vain Gelly, et al. An image is worth 16x16 words: Trans-
formers for image recognition at scale. ICLR, 2021. 2, 3
More detailed results. In Table A.1, we present the de- [7] Oren Dovrat, Itai Lang, and Shai Avidan. Learning to sam-
tailed comparison of the semantic segmentation results on ple. In CVPR, 2019. 2
the S3DIS dataset [1], under the 6-fold cross-validation [8] Carlos Esteves, Christine Allen-Blanchette, Ameesh Maka-
setting. We get the highest mIoU as 73.5%, outperform- dia, and Kostas Daniilidis. Learning so (3) equivariant rep-
ing previous approaches (e.g., RandLA-Net [11] and KP- resentations with spherical cnns. In ECCV, 2018. 2
Conv [37]) by a large margin. For most of the categories [9] Benjamin Graham, Martin Engelcke, and Laurens Van
(like wall, column, table, etc.), our method gets the best ac- Der Maaten. 3d semantic segmentation with submanifold
curacy. We will release all the implementation details and sparse convolutional networks. In CVPR, 2018. 1, 2
trained models to the community soon. [10] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local
relation networks for image recognition. In ICCV, 2019. 1,
Inference time and memory. We test the infer- 2, 3
ence time and memory consumption of Point Trans- [11] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan
former on one Quadro RTX 6000, with different Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.
size of input point clouds. The inference time and Randla-net: Efficient semantic segmentation of large-scale
memory consumption are 44ms/86ms/222ms/719ms and point clouds. In CVPR, 2020. 2, 9
1702M/2064M/2800M/4266M for 10k/20k/40k/80k input [12] Qiangui Huang, Weiyue Wang, and Ulrich Neumann. Recur-
points respectively and they can be further reduced with op- rent slice networks for 3d segmentation of point clouds. In
timized implementation. CVPR, 2018. 6, 10
[13] Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-
kNN efficiency. For kNN, when constructing local point
Wing Fu, and Jiaya Jia. Hierarchical point-edge interaction
cloud regions, previous methods like KPConv [37] and
network for point cloud semantic segmentation. In ICCV,
RandLA-Net [11] use precomputed kNN indices, which 2019. 2, 6
limits the flexibility of the overall framework. In our archi- [14] Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi
tecture, we implement a high-efficiency solution for kNN Nishida. Rotationnet: Joint object categorization and pose
using the heap sort algorithm. We test the running time of estimation using multiviews from unsupervised viewpoints.
our efficient implementation on one Quadro RTX 6000; the In CVPR, 2018. 2
results are listed in Table A.2. We also test some naive im- [15] Loic Landrieu and Martin Simonovsky. Large-scale point
plementations, the running time is 56ms/228ms when given cloud semantic segmentation with superpoint graphs. In
10k/20k points, which is much slower than ours. More- CVPR, 2018. 2, 5, 6, 10
over, naive implementations run out of memory when given [16] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou,
larger point clouds. Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders
for object detection from point clouds. In CVPR, 2019. 2
References [17] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Se-
ungjin Choi, and Yee Whye Teh. Set transformer: A frame-
[1] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioan- work for attention-based permutation-invariant neural net-
nis Brilakis, Martin Fischer, and Silvio Savarese. 3D seman- works. In ICML, 2019. 3, 6
tic parsing of large-scale indoor spaces. In CVPR, 2016. 5, [18] Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from
9 3d lidar using fully convolutional network. In RSS, 2016. 2
[2] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. [19] Guohao Li, Matthias Muller, Ali Thabet, and Bernard
Multi-view 3d object detection network for autonomous Ghanem. Deepgcns: Can gcns go as deep as cnns? In ICCV,
driving. In CVPR, 2017. 2 2019. 1, 2
[3] Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d [20] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan
spatio-temporal convnets: Minkowski convolutional neural Di, and Baoquan Chen. Pointcnn: Convolution on X -
networks. In CVPR, 2019. 1, 2, 5 transformed points. In NIPS, 2018. 2, 6, 7, 10
[4] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, [21] Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias
Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Zwicker. Point2sequence: Learning the shape representa-
Attentive language models beyond a fixed-length context. In tion of 3d point clouds with an attention-based sequence to
ACL, 2019. 1, 2, 3 sequence network. In AAAI, 2019. 3, 6, 7
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina [22] Jiageng Mao, Xiaogang Wang, and Hongsheng Li. Interpo-
Toutanova. BERT: Pre-training of deep bidirectional trans- lated convolutional networks for 3d point cloud understand-
formers for language understanding. In NAACL-HLT, 2019. ing. In ICCV, 2019. 2, 6, 7
1, 2, 3 [23] Daniel Maturana and Sebastian Scherer. Voxnet: A 3d con-
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, volutional neural network for real-time object recognition. In
Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, IROS, 2015. 1, 2, 6
Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl-
 Method OA mAcc mIoU ceiling floor wall beam column window door table chair sofa bookcase board clutter
PointNet [25] 78.5 66.2 47.6 88.0 88.7 69.3 42.4 23.1 47.5 51.6 54.1 42.0 9.6 38.2 29.4 35.2
RSNet [12] – 66.5 56.5 92.5 92.8 78.6 32.8 34.4 51.6 68.1 60.1 59.7 50.2 16.4 44.9 52.0
SPGraph [15] 85.5 73.0 62.1 89.9 95.1 76.4 62.8 47.1 55.3 68.4 73.5 69.2 63.2 45.9 8.7 52.9
PAT [50] – 76.5 64.3 93.0 98.4 73.5 58.5 38.9 77.4 67.7 62.7 67.3 30.6 59.6 66.6 41.4
PointCNN [20] 88.1 75.6 65.4 94.8 97.3 75.8 63.3 51.7 58.4 57.2 71.6 69.1 39.1 61.2 52.2 58.6
PointWeb [55] 87.3 76.2 66.7 93.5 94.2 80.8 52.4 41.3 64.9 68.1 71.4 67.1 50.3 62.7 62.2 58.5
ShellNet [53] 87.1 – 66.8 90.2 93.6 79.9 60.4 44.1 64.9 52.9 71.6 84.7 53.8 64.6 48.6 59.4
RandLA-Net [37] 88.0 82.0 70.0 93.1 96.1 80.6 62.4 48.0 64.4 69.4 69.4 76.4 60.0 64.2 65.9 60.1
KPConv [37] – 79.1 70.6 93.6 92.4 83.1 63.9 54.3 66.1 76.6 64.0 57.8 74.9 69.3 61.3 60.3
PointTransformer 90.2 81.9 73.5 94.3 97.5 84.7 55.6 58.1 66.1 78.2 77.6 74.1 67.3 71.2 65.7 64.8
Table A.1. Semantic segmentation results on S3DIS, evaluated with 6-fold cross-validation.

#pts k=8 k=16 k=32 k=64 k=128 k=256 [33] Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji,
10k 2 2 5 10 17 21 Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz.
Splatnet: Sparse lattice networks for point cloud processing.
20k 3 5 8 23 43 49
In CVPR, 2018. 7
40k 8 12 26 71 127 144
[34] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and
80k 23 37 82 198 356 399
Erik G. Learned-Miller. Multi-view convolutional neural
100k 32 46 99 248 445 494 networks for 3d shape recognition. In ICCV, 2015. 2, 6
200k 104 125 225 545 992 1091 [35] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-
500k 639 695 867 1589 2865 3143 Yi Zhou. Tangent convolutions for dense prediction in 3d. In
1m 2496 2648 2949 4087 6362 6878 CVPR, 2018. 2, 6
Table A.2. High-efficiency kNN implementation with heap sort [36] Lyne P. Tchapmi, Christopher B. Choy, Iro Armeni, JunY-
algorithm. The leftmost column stands for the number of points oung Gwak, and Silvio Savarese. Segcloud: Semantic seg-
and the topmost row specifies the number of nearest neighbors. mentation of 3d point clouds. In 3DV, 2017. 5, 6
The reported running time is in milliseconds. [37] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud,
Beatriz Marcotegui, François Goulette, and Leonidas J
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, Guibas. Kpconv: Flexible and deformable convolution for
James Bradbury, Gregory Chanan, Trevor Killeen, Zeming point clouds. In ICCV, 2019. 1, 2, 5, 6, 7, 9, 10
Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An [38] Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and
imperative style, high-performance deep learning library. In Vladlen Koltun. Lagrangian fluid simulation with continu-
NIPS, 2019. 5 ous convolutions. In ICLR, 2020. 2
[25] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
Leonidas J. Guibas. Pointnet: Deep learning on point sets reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
for 3d classification and segmentation. In CVPR, 2017. 1, 2, Polosukhin. Attention is all you need. In NIPS, 2017. 1, 2,
5, 6, 7, 10 3, 4, 7
[26] Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela [40] Chu Wang, Babak Samari, and Kaleem Siddiqi. Local spec-
Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and tral graph convolution for point set feature learning. In
multi-view cnns for object classification on 3d data. In ECCV, 2018. 2, 6
CVPR, 2016. 6 [41] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and
[27] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Jie Shan. Graph attention convolution for point cloud seman-
Guibas. Pointnet++: Deep hierarchical feature learning on tic segmentation. In CVPR, 2019. 2
point sets in a metric space. In NIPS, 2017. 1, 2, 5, 6, 7 [42] Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei
[28] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Pokrovsky, and Raquel Urtasun. Deep parametric continu-
Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone ous convolutional neural networks. In CVPR, 2018. 1, 2, 6,
self-attention in vision models. In NeurIPS, 2019. 1, 2, 3 7
[29] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. [43] Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neu-
Octnet: Learning deep 3d representations at high resolutions. mann. Sgpn: Similarity group proposal network for 3d point
In CVPR, 2017. 2 cloud instance segmentation. In CVPR, 2018. 7
[30] Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. Min- [44] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma,
ing point cloud local structures by kernel correlation and Michael M. Bronstein, and Justin M. Solomon. Dynamic
graph pooling. In CVPR, 2018. 2 graph cnn for learning on point clouds. TOG, 2019. 1, 2, 6,
[31] Martin Simonovsky and Nikos Komodakis. Dynamic edge- 7
conditioned filters in convolutional neural networks on [45] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin,
graphs. In CVPR, 2017. 2 and Michael Auli. Pay less attention with lightweight and
[32] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Mano-
dynamic convolutions. In ICLR, 2019. 1, 2, 3
lis Savva, and Thomas Funkhouser. Semantic scene comple-
tion from a single depth image. In CVPR, 2017. 1, 2
[46] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep [51] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell,
convolutional networks on 3d point clouds. In CVPR, 2019. Ruslan Salakhutdinov, and Quoc V. Le. XLNet: General-
2, 6, 7 ized autoregressive pretraining for language understanding.
[47] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Lin- In NeurIPS, 2019. 1, 2, 3
guang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d [52] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen,
shapenets: A deep representation for volumetric shapes. In Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Shef-
CVPR, 2015. 5, 6 fer, and Leonidas Guibas. A scalable active framework for
[48] Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. At- region annotation in 3d shape collections. TOG, 2016. 5, 6
tentional shapecontextnet for point cloud recognition. In [53] Zhiyuan Zhang, Binh-Son Hua, and Sai-Kit Yeung. Shell-
CVPR, 2018. 3, 6, 7 net: Efficient point cloud convolutional neural networks us-
[49] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. ing concentric shells statistics. In ICCV, 2019. 6, 10
Spidercnn: Deep learning on point sets with parameterized [54] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring
convolutional filters. In ECCV, 2018. 2, 6, 7 self-attention for image recognition. In CVPR, 2020. 1, 2, 3,
[50] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, 4
Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point [55] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia.
clouds with self-attention and gumbel subset sampling. In PointWeb: Enhancing local neighborhood features for point
CVPR, 2019. 2, 3, 5, 6, 10 cloud processing. In CVPR, 2019. 2, 6, 10