PointNet: Deep Learning on Point Sets

for 3D Classification and

Segmentation- Part 1
Big Data+DeepRepresentation
Learning
Robot Perception Augmented Reality Shape
Design

source: Scott J source: Google source:

Grunewald Tango solidsolutions

Emerging 3D Applications
Big Data+DeepRepresentation
Learning
Robot Perception Augmented Reality Shape
Design

source: Scott J source: Google source:

Grunewald Tango solidsolutions

Need for 3D Deep Learning!

3D Representation: Point
Cloud
Point cloud is close to raw sensor data

Point cloud is canonical

Mes
h

LiDA
Volumetr
R
ic

Depth
Point Cloud
Sensor
Depth
Map
Previous
Works
Most existing point cloud features are
handcrafted
towards specific tasks

Source: https://github.com/PointCloudLibrary/pcl/wiki/Overview-and-Comparison-of-Features
 Question:

Can we achieve effective feature

learning directly on point
clouds?
Our Work:
PointNet
End-to-end learning for scattered, unordered point
data

PointNet
Our Work:
PointNet
End-to-end learning for scattered, unordered point
data
Unified framework for various tasks

Object Classification

PointNet Object Part Segmentation

Semantic Scene Parsing
...
Our Work:
PointNet
End-to-end learning for scattered, unordered point
data
Unified framework for various tasks
Challenge
s

Unordered point set as input

Model needs to be invariant to N! permutations.

Invariance under geometric transformations

Point cloud rotations should not alter classification
results.
Challenge
s

Unordered point set as input

Model needs to be invariant to N! permutations.

Invariance under geometric transformations

Point cloud rotations should not alter classification
results.
Unordered
Input
Point cloud: N orderless points, each represented
by a D dim vector
D

N
Unordered
Input
Point cloud: N orderless points, each represented
by a D dim vector
D D

N represents the same set N

as
Unordered
Input
Point cloud: N orderless points, each represented
by a D dim vector
D D

N represents the same set N

as

Model needs to be invariant to N! permutations

Permutation Invariance: Symmetric
Function
f (x1 , x2 ,, xn )  , x ,, x ), xi D

f (x ℝ
1 2 n
Permutation Invariance: Symmetric
Function
f (x1 , x2 ,, xn )  , x ,, x ), xi D

f (x ℝ
1 2 n

Examples:
f (x1 , x2 ,, xn )  max{x1 , x2 ,,
xn }
f (x1 , x2 ,, xn )  x1  x2  xn
…
Permutation Invariance: Symmetric
Function
f (x1 , x2 ,, xn )  , x ,, x ), xi D

f (x ℝ
1 2 n

Examples:
f (x1 , x2 ,, xn )  max{x1 , x2 ,, xn }
f (x1 , x2 ,, xn )  x1  x2  xn
…
How can we construct a family of symmetric
functions by neural networks?
 Permutation Invariance: Symmetric
Function
Observe:

f (x1, x2 ,, xn )   ∘ g(h(x1 ),, h(xn )) is symmetric if g is

symmetric
 Permutation Invariance: Symmetric
Function
Observe:

f (x1, x2 ,, xn )   ∘ g(h(x1 ),, h(xn )) is symmetric if g is

symmetric

h
(1,2,3)
(1,1,1)
…

(2,3,2)
 Permutation Invariance: Symmetric
Function
Observe:

f (x1, x2 ,, xn )   ∘ g(h(x1 ),, h(xn )) is symmetric if g is

symmetric
(1,2,3 simple symmetric
) h function
(1,1,1 g
)
(2,3,2
)
…

(2,3,4
 Permutation Invariance: Symmetric
Function
Observe:

f (x1, x2 ,, xn )   ∘ g(h(x1 ),, h(xn )) is symmetric if g is

symmetric
(1,2,3 simple symmetric
) h function
(1,1,1 g 
)
(2,3,2
)
…

(2,3,4 PointNet (vanilla)

)
Permutation Invariance: Symmetric
Function
What symmetric functions can be constructed by
PointNet?

Symmetric functions

PointNet

(vanilla)
Universal Set Function
Approximator
Theorem:
A Hausdorff continuous symmetric function f : 2 X 
ℝ can be arbitrarily approximated by PointNet.

S  ℝd PointNet (vanilla)
 Basic PointNet
Architecture
Empirically, we use multi-layer perceptron (MLP) and max
pooling:
h
(1,2,3 MLP
) g
(1,1,1 MLP 
) ma MLP
(2,3,2 MLP
x
)
…

(2,3,4 MLP PointNet (vanilla)

)
Challenge
s

Unordered point set as input

Model needs to be invariant to N! permutations.

Invariance under geometric transformations

Point cloud rotations should not alter classification
results.
Input Alignment by Transformer
Network
Idea: Data dependent transformation for automatic
alignment
3 T- transfor 3
Net m
params
N Transfor N
m

Transform
Dat
ed
a
Data
Input Alignment by Transformer
Network
Idea: Data dependent transformation for automatic
alignment
3 T- transfor 3
Net m
params
N Transfor N
m

Transform
Dat
ed
a
Data
Input Alignment by Transformer
Network
Idea: Data dependent transformation for automatic
alignment
3 T- transfor 3
Net m
params
N Transfor N
m

Transform
Dat
ed
a
Data
Input Alignment by Transformer
Network
The transformation is just matrix
multiplication!
3 T- transfor 3
Net params:
m
3x3
Matrix
N
Mult.

Transform
Dat
ed
a
Data
Embedding Space
Alignment

transform
T- params:
Net 64x64
Matri
x
Mult
.
Input Transform
embedding ed
s: Nx64 embedding
s: Nx64
Embedding Space
Alignment

transform
T- params: Regularization:
Net 64x64
Matri Transform matrix A
x 64x64 close to
Mult orthogonal:
.
Input Transform
embedding ed
s: Nx64 embedding
s: Nx64
PointNet Classification
Network
PointNet Classification
Network
PointNet Classification
Network
PointNet Classification
Network
PointNet Classification
Network
PointNet Classification
Network
PointNet Classification
Network
Extension to
PointNet Segmentation
Network

local embedding global feature

Extension to
PointNet Segmentation
Network

local embedding global feature

Result
s
Results on Semantic Scene
Parsing
Input
Output

dataset: Stanford 2D-3D-S (Matterport scans)

Visualizing Global Point
Cloud Features
3 MLP 102
4 maxpo
ol
n
share global feature
d

Which input points are contributing to the global

feature?
(critical points)
Visualizing Global Point
Cloud Features
Original
Shape:

Critical Point
Sets:
Visualizing Global Point
Cloud Features
3 MLP 102
4 maxpo
ol
n
share global feature
d

Which points won’t affect the global

feature?
Conclusio
n
• PointNet is a novel deep neural network that
directly consumes point cloud.
• A unified approach to various 3D recognition
tasks.
• Rich theoretical analysis and experimental
results.
Code & Data Available!
http://stanford.edu/~rqi/poi
ntnet

See you at Poster 9!

Permutation Invariance:
How about Sorting?
“Sort” the points before feeding them into a
network.
Unfortunately, there is no canonical order in high
dim space.
lexsorte
(1,2,3 d
) (1,1,1)
(1,1,1 (2,3,2
(1,2,3) MLP
) )
(2,3,2 (2,3,4
) )
(2,3,4
)
Permutation Invariance:
How about Sorting?
“Sort” the points before feeding them into a
network.
Unfortunately, there is no canonical order in high
dim space. Multi-Layer Perceptron
lexsorte (ModelNet shape
d classification)
(1,2,3 Accuracy
) (1,1,1)
(1,1,1 (2,3,2
(1,2,3) MLP Unordered 12%
) ) Input
(2,3,2 (2,3,4 Lexsorted Input 40%
) ) PointNet 87%
(2,3,4 (vanilla)
)
Permutation Invariance:
How about RNNs?
Train RNN with permutation
augmentation. However, RNN
forgets and order matters.
LSTM Network
… (ModelNet shape
LSTM LSTM LSTM LSTM
classification)
Accuracy
MLP MLP MLP … MLP LSTM 75%

PointNet 87%
(1,2,3 (1,1,1 (2,3,2 (2,3,4 (vanilla)
) ) ) )
PointNet Classification
Network

ModelNet40 Accuracy
PointNet (vanilla) 87.1%
+ input 3x3 87.9%
+ feature 64x64 86.9%
+ feature 64x64 + 87.4%
reg
+ both 89.2%
VisualizingPoint
Functions
1x 1x102
Compact 3 FCs 4
View:
1x 1x102
Expanded 3 FC FC FC FC FC 4
View: 6 6 6 12
4 4 4 8

Which input point will activate neuron X?

Find the top-K points in a dense volumetric grid that activates
neuron X.