Computer Vision Group

TSDF Volume Reconstruction

Martin Herrmann
Simon Trendel
Neeraj Sujan

Technische Universität München

Department of Informatics
Computer Vision Group

October 5, 2015

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 1 / 19

 Computer Vision Group

Outline

1 Introduction

2 Approach

3 Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 2 / 19

 Computer Vision Group

Outline

1 Introduction

2 Approach

3 Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 3 / 19

 Computer Vision Group

Introduction

Reconstruct 3D voxel grid from multiple input frames

Frames consist of color (RGB) and depth images
Must be fast enough to use in real-time with a Kinect (or a
similar sensor)

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 4 / 19

 Computer Vision Group

Introduction

Reconstruct 3D voxel grid from multiple input frames

Frames consist of color (RGB) and depth images
Must be fast enough to use in real-time with a Kinect (or a
similar sensor)

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 4 / 19

 Computer Vision Group

Introduction

Reconstruct 3D voxel grid from multiple input frames

Frames consist of color (RGB) and depth images
Must be fast enough to use in real-time with a Kinect (or a
similar sensor)

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 4 / 19

 Computer Vision Group

Outline

1 Introduction

2 Approach

3 Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 5 / 19

 Computer Vision Group

The Pipeline

t +1
Bilateral Calculate ICP
Filter Normals

TSDF Raycasting
Integration

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 6 / 19

 Computer Vision Group

Transformations between the different coordinate systems

!
x
y
z

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 7 / 19

 Computer Vision Group

Transformations between the different coordinate systems

y
z

x
xg
! !
x
y yg
z zg

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 7 / 19

 Computer Vision Group

Transformations between the different coordinate systems

y
z

x
xg xc
! ! !
x
y yg yc
z zg zc

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 7 / 19

 Computer Vision Group

Transformations between the different coordinate systems

y
z

x
xg xc
! ! !
x 
xp

y yg yc yp
z zg zc

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 7 / 19

 Computer Vision Group

Transformations between the different coordinate systems

y
z

x
xg xc
! ! !
x 
xp

y yg yc yp
z zg zc
 g  c
x x
! !
x gridPosx
y + gridPosy yg yc
camPose −1 · z g  (K · z c )/z p
z gridPosz
1 1

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 7 / 19

 Computer Vision Group

Transformations between the different coordinate systems

 c  p p
x x ·z
xg yc yp · zp
! !
gridPosx −1
yg − gridPosy camPose · z c  K ·  zp 
zg gridPosz 1 1

xg xc
! ! !
x 
xp

y yg yc yp
z zg zc

y
z

x
xg xc
! ! !
x 
xp

y yg yc yp
z zg zc
 g  c
x x
! !
x gridPosx
y + gridPosy yg yc
camPose −1 · z g  (K · z c )/z p
z gridPosz
1 1

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 7 / 19

 Computer Vision Group

Truncated Signed Distance Function (TSDF)

Get distance of the corresponding pixel of each voxel within

the voxel grid
Subtract it from the distance of the voxel itself and divide by
the truncation threshold
Update TSDF and color values in global memory

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 8 / 19

 Computer Vision Group

Truncated Signed Distance Function (TSDF)

Get distance of the corresponding pixel of each voxel within

the voxel grid
Subtract it from the distance of the voxel itself and divide by
the truncation threshold
Update TSDF and color values in global memory

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 8 / 19

 Computer Vision Group

Truncated Signed Distance Function (TSDF)

Get distance of the corresponding pixel of each voxel within

the voxel grid
Subtract it from the distance of the voxel itself and divide by
the truncation threshold
Update TSDF and color values in global memory

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 8 / 19

 Computer Vision Group

Weighting of Color and Depth Values

Bad samples exist and must be weighted accordingly

Idea: use angle of incidence – lower angles usually correspond
to better samples
Implementation: multiply by z-coordinate of normal

Camera

θ Camera
θ
n~ n~

Surface Surface

(a) Good sample (b) Bad sample

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 9 / 19

 Computer Vision Group

Weighting of Color and Depth Values

Bad samples exist and must be weighted accordingly

Idea: use angle of incidence – lower angles usually correspond
to better samples
Implementation: multiply by z-coordinate of normal

Camera

θ Camera
θ
n~ n~

Surface Surface

(a) Good sample (b) Bad sample

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 9 / 19

 Computer Vision Group

Weighting of Color and Depth Values

Bad samples exist and must be weighted accordingly

Idea: use angle of incidence – lower angles usually correspond
to better samples
Implementation: multiply by z-coordinate of normal

Camera

θ Camera
θ
n~ n~

Surface Surface

(a) Good sample (b) Bad sample

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 9 / 19

 Computer Vision Group

Weighting of Color and Depth Values

Bad samples exist and must be weighted accordingly

Idea: use angle of incidence – lower angles usually correspond
to better samples
Implementation: multiply by z-coordinate of normal

Camera

θ Camera
θ
n~ n~

Surface Surface

(a) Good sample (b) Bad sample

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 9 / 19

 Computer Vision Group

Adaptive Raycasting

Cast a ray for each pixel of the picture being rendered

Take a step in z-direction and transform coordinates to voxel
grid
Check TSDF value (using trilineal interpolation); if
zero-crossing (= edge) was detected, use increasingly smaller
step size until we are as close to zero as possible
Write color value (using trilinear interpolation) to picture
Algorithm does not include lighting or shadows

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 10 / 19

 Computer Vision Group

Adaptive Raycasting

Cast a ray for each pixel of the picture being rendered

Take a step in z-direction and transform coordinates to voxel
grid
Check TSDF value (using trilineal interpolation); if
zero-crossing (= edge) was detected, use increasingly smaller
step size until we are as close to zero as possible
Write color value (using trilinear interpolation) to picture
Algorithm does not include lighting or shadows

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 10 / 19

 Computer Vision Group

Adaptive Raycasting

Cast a ray for each pixel of the picture being rendered

Take a step in z-direction and transform coordinates to voxel
grid
Check TSDF value (using trilineal interpolation); if
zero-crossing (= edge) was detected, use increasingly smaller
step size until we are as close to zero as possible
Write color value (using trilinear interpolation) to picture
Algorithm does not include lighting or shadows

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 10 / 19

 Computer Vision Group

Adaptive Raycasting

Cast a ray for each pixel of the picture being rendered

Take a step in z-direction and transform coordinates to voxel
grid
Check TSDF value (using trilineal interpolation); if
zero-crossing (= edge) was detected, use increasingly smaller
step size until we are as close to zero as possible
Write color value (using trilinear interpolation) to picture
Algorithm does not include lighting or shadows

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 10 / 19

 Computer Vision Group

Adaptive Raycasting

Cast a ray for each pixel of the picture being rendered

Take a step in z-direction and transform coordinates to voxel
grid
Check TSDF value (using trilineal interpolation); if
zero-crossing (= edge) was detected, use increasingly smaller
step size until we are as close to zero as possible
Write color value (using trilinear interpolation) to picture
Algorithm does not include lighting or shadows

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 10 / 19

 Computer Vision Group

Normal Calculation using PCA

Ideal for a more robust approach to calculate normals
Combination of k-d tree and PCA
Not used in final version due to large performance hit (CPU
implementation) and only negligible improvements

(a) k-d tree (b) PCA

Figure: Robust Normal Estimation
Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 11 / 19
 Computer Vision Group

Outline

1 Introduction

2 Approach

3 Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 12 / 19

 Computer Vision Group

Results
Grid Size vs Time
120

100

80
Time

60

40

20

0
0 20 40 60 80 100
Martin Herrmann Simon Trendel Grid
Neeraj Sujan: TSDF Size
Volume Reconstruction 13 / 19
 Computer Vision Group

Results – different color weighting methods

(a) Exponential falloff (b) Linear falloff

(c) No falloff
Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 14 / 19
 Computer Vision Group

Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 15 / 19

 Computer Vision Group

Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 16 / 19

 Computer Vision Group

Results

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 17 / 19

 Computer Vision Group

Thank you
for your attention!

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 18 / 19

 Computer Vision Group

References
[Curless and Levoy, 1996] Curless, B. and Levoy, M. (1996).
A volumetric method for building complex models from range images.
In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages
303–312. ACM.
[Holz et al., 2012] Holz, D., Holzer, S., Rusu, R. B., and Behnke, S. (2012).
Real-time plane segmentation using rgb-d cameras.
In RoboCup 2011: Robot Soccer World Cup XV, pages 306–317. Springer.
[Izadi et al., 2011] Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges,
S., Freeman, D., Davison, A., et al. (2011).
Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera.
In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559–568.
ACM.
[Sturm et al., 2013] Sturm, J., Bylow, E., Kahl, F., and Cremers, D. (2013).
Copyme3d: Scanning and printing persons in 3d.
In Pattern Recognition, pages 405–414. Springer.

Martin Herrmann Simon Trendel Neeraj Sujan: TSDF Volume Reconstruction 19 / 19