Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose

and Shape Estimation

Mohamed Omran1 Christoph Lassner2∗ Gerard Pons-Moll1 Peter V. Gehler2∗

1
Bernt Schiele
1
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany
2
Amazon, Tübingen, Germany
arXiv:1808.05942v1 [cs.CV] 17 Aug 2018

{mohomran, gpons, schiele}@mpi-inf.mpg.de, {classner, pgehler}@amazon.com∗

Abstract try. Third, the 3D model output is one step closer to a faith-
ful 3D reconstruction of people in images.
Direct prediction of 3D body pose and shape remains Traditional model-based approaches typically optimize
a challenge even for highly parameterized deep learning an objective function that measures how well the model fits
models. Mapping from the 2D image space to the predic- the image observations – for example, 2D keypoints [6, 24].
tion space is difficult: perspective ambiguities make the These methods do not require paired 3D training data (im-
loss function noisy and training data is scarce. In this ages with 3D pose), but only work well when initialized
paper, we propose a novel approach (Neural Body Fitting close to the solution. By contrast, initialization is not re-
(NBF)). It integrates a statistical body model within a CNN, quired in forward prediction models, such as CNNs that di-
leveraging reliable bottom-up semantic body part segmen- rectly predict 3D keypoints. However many images with 3D
tation and robust top-down body model constraints. NBF pose annotations are required, which are difficult to obtain,
is fully differentiable and can be trained using 2D and 3D unlike images with 2D pose annotations.
annotations. In detailed experiments, we analyze how the
components of our model affect performance, especially the Therefore, like us, a few recent works have proposed hy-
use of part segmentations as an explicit intermediate repre- brid CNN architectures that are trained using model-based
sentation, and present a robust, efficiently trainable frame- loss functions [56, 62, 22, 38]. Specifically, from an im-
work for 3D human pose estimation from 2D images with age, a CNN predicts the parameters of the SMPL 3D body
competitive results on standard benchmarks. Code will be model [28], and the model is re-projected onto the image
made available at http://github.com/mohomran/ to evaluate the loss function in 2D space. Consequently,
neural_body_fitting 2D pose annotations can be used to train such architectures.
While these hybrid approaches share similarities, they all
differ in essential design choices, such as the amount of 3D
1. Introduction vs 2D annotations for supervision and the input representa-
Much research effort has been successfully directed to- tion used to lift to 3D.
wards predicting 3D keypoints and stick-figure representa- To analyze the importance of such components, we intro-
tions from images of people. Here, we consider the more duce Neural Body Fitting (NBF), a framework designed to
challenging problem of estimating the parameters of a de- provide fine-grained control over all parts of the body fitting
tailed statistical human body model from a single image. process. NBF is a hybrid architecture that integrates a sta-
We tackle this problem by incorporating a model of the tistical body model within a CNN. From an RGB image or a
human body into a deep learning architecture, which has semantic segmentation of the image, NBF directly predicts
several advantages. First, the model incorporates limb ori- the parameters of the model; those parameters are passed
entations and shape, which are required for many applica- to SMPL to produce a 3D mesh; the joints of the 3D mesh
tions such as character animation, biomechanics and virtual are then projected to the image closing the loop. Hence,
reality. Second, anthropomorphic constraints are automati- NBF admits both full 3D supervision (in the model or 3D
cally satisfied – for example limb proportions and symme- Euclidean space) and weak 2D supervision (if images with
∗ This work was done while Christoph Lassner and Peter V. Gehler were only 2D annotations are available). NBF combines the ad-
with the MPI for Intelligent Systems and the University of Tübingen, and vantages of direct bottom-up methods and top-down meth-
additionally while Peter was with the University of Würzburg. ods. It requires neither initialization nor large amounts of

1
Figure 1: Given a single 2D image of a person, we predict a semantic body part segmentation. This part segmentation is
represented as a color-coded map and used to predict the parameters of a 3D body model.

3D training data. components to achieve good performance in hybrid archi-

One key question we address with our study is whether to tectures and provide insights for its real-world applicability
use an intermediate representation rather than directly lift- that we believe may hold for many related methods as well.
ing to 3D from the raw RGB image. Images of humans can
vary due to factors such as illumination, clothing, and back- 2. Related Work
ground clutter. Those effects do not necessarily correlate
Human pose estimation is a well-researched field and we
with pose and shape, thus we investigate whether a simpli-
focus on 3D methods; for a recent extensive survey on the
fication of the RGB image into a semantic segmentation of
field we refer to [48].
body parts improves 3D inference. We also consider the
Model-based Methods. To estimate human pose and shape
granularity of the body part segmentation as well as seg-
from images, model-based [42] works use a parametric
mentation quality, and find that: (i) a color-coded 12-body-
body model or template. Early models were based on ge-
part segmentation contains sufficient information for pre-
ometric primitives [34, 11, 39, 50, 54], while more recent
dicting shape and pose, (ii) the use of such an intermediate
ones are estimated from 1000s of scans of real people,
representation results in competitive performance and eas-
and are typically parameterized by separate body pose and
ier, more data-efficient training compared to similar meth-
shape components [5, 13, 28, 71, 41], with few exceptions,
ods that predict pose and shape parameters from raw RGB
e.g., [21]. Most model-based approaches fit a model to im-
images, (iii) segmentation quality is a strong predictor of fit
age evidence through complex non-linear optimization, re-
quality.
quiring careful initialization to avoid poor local minima.
We also demonstrate that only a small fraction of the To reduce the complexity of the fitting procedure, the
training data needs to be paired with 3D annotations. We output of 2D keypoint detection has been used as addi-
make use of the recent UP-3D dataset [24] that contains tional guidance. The progress in 2D pose estimation using
8000 training images in the wild along with 3D pose an- CNNs [65, 7, 15] has contributed significantly to the task of
notations. Larger 2D datasets exist, but UP-3D allows us to 3D pose estimation even in challenging in-the-wild scenar-
perform a controlled study. ios. For example, the 3D parameters of SMPL [28] can be
In summary, our contribution is twofold: first we in- obtained with reasonable accuracy by fitting it to 2D key-
troduce NBF, which unites deep learning-based with tra- points [6, 24]. However, lifting to 3D from 2D information
ditional model-based methods taking advantage of both. alone is an ambiguous problem. Adversarial learning can
Second, we provide an in-depth analysis of the necessary potentially identify plausible poses [18, 22]. By contrast,
the seminal works of [57, 53] address lifting by reasoning tant limitation: it does not require an initialization for op-
about kinematic depth ambiguities. Recently, using monoc- timization because it incorporates a CNN based bottom-up
ular video and geometric reasoning, accurate and detailed component. Furthermore, at test time, NBF predictions are
3D shape, including clothing is obtained [3, 2]. fast and do not require optimization. By integrating SMPL
Learning-Based Models. Recent methods in this category directly into the CNN, we do not require multiple network
typically predict 3D keypoints or stick figures from a single heads to backpropagate 2D and 3D losses. Lastly, we use
image using a CNN. High capacity models are trained on a semantic segmentation as proxy representation, which (1)
standard 3D datasets [17, 49], which are limited in terms abstracts away irrelevant image information for 3D pose,
of appearance variation, pose, backgrounds and occlusions. (2) is a richer semantic representation than keypoints or sil-
Consequently, it is not clear – despite excellent performance houettes, and (3) allows us to analyze the importance of part
on standard benchmarks – how methods [59, 25, 37, 26] granularity and placement for 3D prediction.
generalize to in-the-wild images. To add variation, some
methods resort to generating synthetic images [46, 64, 23] 3. Method
but it is complex to approximate fully realistic images with
sufficient variance. Similar to model-based methods, learn- Our goal is to fit a 3D mesh of a human to a single
ing approaches have benefited from the advent of robust 2D static image (see Figure 2). This task involves multiple
pose methods – by matching 2D detections to a 3D pose steps and we want to apply 3D but also 2D losses due
database [8, 66], by regressing pose from 2D joint distance to the strongly varying difficulty to obtain ground truth
matrices [35], by exploiting pose and geometric priors for data. Nevertheless, we aim to build a simple processing
lifting [69, 1, 51, 19, 32, 70, 47]; or simply by training a pipeline with parts that can be optimized in isolation and
feed forward network to directly predict 3D pose from 2D avoiding multiple network heads. This reduces the number
joints [30]. Another way to exploit images with only 2D of hyperparameters and interactions, e.g., loss weights, and
data is by re-using the first layers of a 2D pose CNN for the allows us to consecutively train the model parts. There
task of 3D pose estimation [60, 33, 31]. Pavlakos et al. [36] are two main stages in the proposed architecture: in a
take another approach by relying on weak 3D supervision first stage, a body part segmentation is predicted from the
in form of a relative 3D ordering of joints, similar to the RGB image. The second stage takes this segmentation to
previously proposed PoseBits [40]. predict a low-dimensional parameterization of a body mesh.
Closer to ours are approaches that train using separate
2D and 3D losses [44, 67, 55]. However, since they do 3.1. Body Model
not integrate a statistical body model in the network, limbs For our experiments we use the SMPL body model due
and body proportions might be unnatural. Most importantly, to its good trade-off between high anatomic flexibility and
they only predict 3D stick figures as opposed to a full mesh. realism. SMPL parameterizes a triangulated mesh with
Some works regress correspondences to a body model N = 6890 vertices with pose parameters θ ∈ R72 and shape
which are then used to fit the model to depth data [43, 58]. parameters β ∈ R10 – optionally the translation parameters
Recently, correspondences to the SMPL body surface are γ ∈ R3 can be taken into account as well. Shape Bs (β)
regressed from images directly [12] by leveraging dense and pose dependent deformations Bp (θ) are first applied
keypoint annotations; however, the approach can not re- to a base template Tµ ; then the mesh is posed by rotating
cover 3D human pose and shape. [63] fits SMPL to CNN each body part around skeleton joints J(β) using a skinning
volumetric outputs as a post-process step. 3D model fit- function W :
ting within a CNN have been proposed for faces [61]; faces
however, are not articulated like bodies which simplifies the M (β, θ) = W (T (β, θ), J(β), θ, W), (1)
regression problem.
A few recent works (concurrent to ours) integrate the T (β, θ) = Tµ + Bs (β) + Bp (θ), (2)
SMPL [28] model within a network [62, 22, 38]. The ap-
proaches differ primarily in the proxy representation used to where M (β, θ) is the SMPL function, and T (β, θ) outputs
lift to 3D: RGB images [22], images and 2D keypoints [62] an intermediate mesh in a T-pose after pose and shape de-
and 2D keypoints and silhouettes [38], and the kind of su- formations are applied. SMPL produces realistic results us-
pervision (3D vs 2D) used for training. In contrast to pre- ing relatively simple mathematical operations – most im-
vious work, we analyze the importance of such components portantly for us SMPL is fully differentiable with respect
for good performance. Kanazawa et al. [22] also integrate to pose θ and shape β. All these operations, including the
a learned prior on the space of poses. We draw inspiration ones to determine projected points of a posed and parame-
from model-based and learning approaches in several as- terized 3D body, are available in Tensorflow. We use them
pects of our model design. Firstly, NBF addresses an impor- to make the 3D body a part of our deep learning model.
 End-to-end supervision

Input Model Output 3D keypoints 2D keypoints

pose ✓
Proxy CNN SMPL P (·)
CNN w M(✓, )
shape

Training data with matching samples hard to obtain.

Figure 2: Summary of our proposed pipeline. We process the image with a standard semantic segmentation CNN into 12
semantic parts (see Sec. 4.2). An encoding CNN processes the semantic part probability maps to predict SMPL body model
parameters (see Sec. 3.2). We then use our SMPL implementation in Tensorflow to obtain a projection of the pose-defining
points to 2D. With these points, a loss on 2D vertex positions can be back propagated through the entire model (see Sec. 3.3).

3.2. Neural Body Fitting Parameterization 3.3. Loss Functions

NBF predicts the parameters of the body model from a We experiment with the following loss functions:
colour-coded part segmentation map I ∈ R224×224×3 using 3D latent parameter loss: This is an L1 loss on the model
a CNN-based predictor parameterized by weights w. The parameters θ and β. Given a paired dataset {Ii , θ i , β i }N
i ,
estimators for pose and shape are thus given by θ(w, I) and the loss is given by:
β(w, I) respectively.
N
We integrate the SMPL model and a simple 2D projec- X
tion layer into our CNN estimator, as described in Sec. 3.1. Llat (w) = |r(θ(w, Ii ))−r(θ i )|+|β(w, Ii )−β i |, (6)
i
This allows us to output a 3D mesh, 3D skeleton joint loca-
tions or 2D joints, depending on the kind of supervision we where r are the vectorized rotation matrices of the 24 parts
want to apply for training while keeping the CNN mono- of the body. Similar to [24, 38], we observed better perfor-
lithic. mance by imposing the loss on the rotation matrix represen-
Mathematically, the function N3D (w, I) that maps from tation of θ rather than on its ‘native’ axis angle encoding as
semantic images to meshes is given by defined in SMPL. This requires us to project the predicted
matrices to the manifold of rotation matrices. We perform
N3D (w, I) = M (θ(w, I), β(w, I)) (3) this step using SVD to maintain differentiability.
= W (T (β(w, I), θ(w, I), 3D joint loss: Given a paired dataset with skeleton annota-
J(β(w, I)), θ(w, I), W)), (4) tions {Ii , θ i , J}N
i we compute the loss in terms of 3D joint
position differences as:
which is the SMPL Equation (1) parameterized by net-
N
work parameters w. NBF can also predict the 3D joints X
NJ (w, I) = J(β(w, I), because they are a function of the L3D (w) = kNJ (w, Ii ) − Ji k2 (7)
i
model parameters. Furthermore, using a projection opera-
tion π(·) we can project the 3D joints onto the image plane 2D joint loss: If the dataset {Ii , J2D }N
i provides solely 2D
joint position ground truth, we define a similar loss in terms
N2D (w, I) = π(J(w, I)), (5)
of 2D distance and rely on error backpropagation through
where N2D (w, I) is the NBF function that outputs 2D joint the projection:
locations. All of these operations are differentiable and al- N
low us to use gradient-based optimization to update model
X
L2D (w) = kN2D (w, Ii ) − J2D,i k2 (8)
parameters with a suitable loss function. i
Joint 2D and 3D loss: To maximize the amounts of usable Scale ambiguity: The SMPL shape parameters encode
training data, ideally multiple data sources can be combined among other factors a person’s size. Additionally, both dis-
with a subset of the data D3D providing 3D annotations and tance to the camera and focal length determine how large
another subset D3D providing 2D annotations. We can triv- a person appears in an image. To eliminate this ambiguity
ially integrate all the data with different kinds of supervi- during training, we constrain scale information to the shape
sion by falling back to the relevant losses and setting them parameters by making the following assumptions: The cam-
to zero if not applicable. era is always at the SMPL coordinate origin, the optical
axis always points in the same direction, and a person is
L2D+3D (w, D) = L2D (w, D2D ) + L3D (w, D3D ) (9) always at a fixed distance from the camera. We render the
ground truth SMPL fits and scale the training images to fit
In our experiments, we analyze the performance of each the renderings (using the corresponding 2D joints). This
loss and their combinations. In particular, we evaluate how guarantees that the the only factor affecting person size in
much gain in 3D estimation accuracy can be obtained from the image are the SMPL shape parameters. At test-time, we
weak 2D annotations which are much cheaper to obtain than estimate person height and location in the image using 2D
accurate 3D annotations. DeeperCut keypoints [16], and center the person within a
512x512 crop such that they have a height of 440px, which
4. Results roughly corresponds to the setting seen during training.
Architecture: We use a two-stage approach: The first
4.1. Evaluation Settings stage receives the 512x512 input crop and produces a part
We used the following three datasets for evaluation: UP- segmentation. We use a RefineNet [27] model (based on
3D, [24], HumanEva-I [49] and Human3.6M[17]. We per- ResNet-101 [14]). This part segmentation is color-coded,
form a detailed analysis of our approach on UP-3D and Hu- resized to 224x224 and fed as an RGB image to the second
man3.6M, and compare against state-of-the-art methods on stage, itself composed of two parts: a regression network
HumanEVA-I and Human3.6M. (ResNet-50) that outputs the 226 SMPL parameters (shape
UP-3D, is a challenging, in-the-wild dataset that draws and pose), and a non-trainable set of layers that implement
from existing pose datasets: LSP [20], LSP-extended [20], the SMPL model and an image projection. Such layers can
MPII HumanPose [4], and FashionPose [10]. It augments produce a 3D mesh, 3D joints or 2D joints given the pre-
images from these datasets with rich 3D annotations in the dicted pose and shape. Training both stages requires a total
form of SMPL model parameters that fully capture shape of 18 (12+6) hours on a single Volta V100 GPU. More de-
and pose, allowing us to derive 2D and 3D joint as well as tails are provided in the supplementary material as well as
fine-grained segmentation annotations. The dataset consists in the code (to be released).
of training (5703 images), validation (1423 images) and test
(1389 images) sets. For our analysis, we use the training set 4.3. Analysis
and provide results on the validation set. Which Input Encoding? We investigate here what input
The HumanEVA-I dataset is recorded in a controlled representation is effective for pose and shape prediction.
environment with marker-based ground truth synchronized Full RGB images certainly contain more information than
with video. The dataset includes 3 subjects and 2 motion for example silhouettes, part segmentations or 2D joints.
sequences per subject. Human3.6M also consists of simi- However, some information may not be relevant for 3D in-
larly recorded data but covers more subjects, with 15 action ference, such as appearance, illumination or clothing, which
sequences per subject repeated over two trials. For our anal- might make the network overfit to nuisances factors
ysis on this dataset, we reserve subjects S1, S5, S6 and S7 To this end, we train a network on different image rep-
for training, holding out subject S8 for validation. We com- resentations and compare their performance on the UP-3D
pare to the state of art on the test sequences S9 and S11. and Human3.6M validation sets. We compare RGB im-
ages, color-coded part segmentations of varying granular-
4.2. Implementation Details
ities, and color-coded joint heatmaps (see supplementary
Data preparation: To train our model, we require im- material for examples). We generate both using the ground
ages paired with 3D body model fits (i.e. SMPL parameters) truth SMPL annotations to establish an upper bound on per-
as well as pixelwise part labels. The UP-3D dataset pro- formance, and later consider the case where we do not have
vides such annotations, while Human3.6M does not. How- access to such information at test time.
ever, by applying MoSH [29] to the 3D MoCap marker data The results are reported in Table 1. We observe that
provided by the latter we obtain the corresponding SMPL explicit part representations (part segmentations or joint
parameters, which in turn allows us to generate part labels heatmaps) are more useful for 3D shape/pose estimation
by rendering an appropriately annotated SMPL mesh [24]. compared to RGB images and plain silhouettes. The dif-
 type of input UP H36M
RGB 98.5 48.9 0.9

Segmentation (1 part) 95.5 43.0 0.8

Segmentation (3 parts) 36.5 37.5 0.7
Segmentation (6 parts) 29.4 36.2

F1-score
Segmentation (12 parts) 27.8 33.5 0.6
Segmentation (24 parts) 28.8 31.8 0.5
Joints (14) 28.8 33.4 0.4
Joints (24) 27.7 33.4
0.3
Table 1: Input Type vs. 3D error in millimeters
0.2
1 2 5 10 20 40 80
joint error (in mm)
ference is especially pronounced on the UP-3D dataset, Figure 3: Segmentation quality (F1-score) vs. fit quality
which contains more visual variety than the images of Hu- (3D joint error). The darkness indicates the difficulty of the
man3.6M, with an error drop from 98.5 mm to 27.8 mm pose, i.e. the distance from the upright pose with arms by
when using a 12 part segmentation. This demonstrates that the sides.
a 2D segmentation of the person into sufficient parts carries
a lot of information about 3D pose/shape, while also pro-
viding full spatial coverage of the person (compared to joint Train
Val
VGG ResNet RefineNet GT
heatmaps). Is it then worth learning separate mappings first
from image to part segmentation, and then from part seg- VGG 107.2 119.9 135.5 140.7
mentation to 3D shape/pose? To answer this question we
ResNet 97.1 96.3 112.2 115.6
first need to examine how 3D accuracy is affected by the
RefineNet 89.6 89.9 82.0 83.3
quality of real predicted part segmentations.
GT 62.3 60.5 35.7 27.8
Which Input Quality? To determine the effect of seg-
mentation quality on the results, we train three different part Table 2: Effect of segmentation quality on the quality of the
segmentation networks. Besides RefineNet, we also train 3D fit prediction modules (errjoints3D )
two variants of DeepLab [9], based on VGG-16 [52] and
ResNet-101 [14]. These networks result in IoU scores of
67.1, 57.0, and 53.2 respectively on the UP validation set. the higher the F-1 score, the lower the 3D joint error.
Given these results, we then train four 3D prediction net- Which Types of Supervision? We now examine differ-
works - one for each of the part segmentation networks, and ent combinations of loss terms. The losses we consider are
an additional one using the ground truth segmentations. We Llat (on the latent parameters), L3D (on 3D joint/vertex lo-
report 3D accuracy on the validation set of UP3D for each cations), L2D (on the projected joint/vertex locations). We
of the four 3D networks, diagonal numbers of Table 2. As compare performance using three different error measures:
one would expect, the better the segmentation, the better (i) errjoints3D , the Euclidean distance between ground
the 3D prediction accuracy. As can also be seen in Table 2, truth and predicted SMPL joints (in mm). (ii) P CKh [4],
better segmenters at test time always lead to improved 3D the percentage of correct keypoints with the error thresh-
accuracy, even when the 3D prediction networks are trained old being 50% of head size, which we measure on a per-
with poorer segmenters. This is perhaps surprising, and it example basis. (iii) errquat , quaternion distance error of
indicates that mimicking the statistics of a particular seg- the predicted joint rotations (in radians).
mentation method at training time plays only a minor role Given sufficient data - the full 3D-annotated UTP train-
(for example a network trained with GT segmentations and ing set with mirrored examples (11406) - only applying a
tested using RefineNet segmentations performs comparably loss on the model parameters yields reasonable results, and
to a network that is trained using RefineNet segmentations in this setting, additional loss terms don’t provide benefits.
(83.3mm vs 82mm)). To further analyze the correlation be- When only training with L3D , we obtain similar results in
tween segmentation quality and 3D accuracy, in Figure 3 we terms of errjoints3D , however, interestingly errquat is sig-
plot the relationship between F-1 score and 3D reconstruc- nificantly higher. This indicates that predictions produce
tion error. Each dot represents one image, and the color its accurate 3D joints positions in space, but the limb orienta-
respective difficulty – we use the distance to mean pose as tions are incorrect. This further demonstrates that methods
a proxy measure for difficulty. The plot clearly shows that trained to produce only 3D keypoints do not capture orien-
 Loss errjoints3D PCKh errquat Method Mean Median
Ramakrishna et al. [45] 168.4 145.9
Llat 83.7 93.1 0.278
Zhou et al. [68] 110.0 98.9
Llat + L3D 82.3 93.4 0.280
Llat + L2D 83.1 93.5 0.278 SMPLify [6] 79.9 61.9
Llat + L3D + L2D 82.0 93.5 0.279 Random Forests [24] 93.5 77.6
L3D 83.7 93.5 1.962 SMPLify (Dense) [24] 74.5 59.6
L2D 198.0 94.0 1.971 Ours 64.0 49.4

Table 5: HumanEva-I results. 3D joint errors in mm.

Table 3: Loss ablation study. Results in 2D and 3D error
metrics (joints3D: Euclidean 3D distance, mesh: average Method Mean Median
vertex to vertex distance, quat: average body part rotation Akhter & Black [1] 181.1 158.1
error in radians). Ramakrishna et al. [45] 157.3 136.8
Zhou et al. [68] 106.7 90.0
Ann.perc. SMPLify [6] 82.3 69.3
Error 100 50 20 10 5 2 1 0
SMPLify (dense) [24] 80.7 70.0
errjoints3D 83.1 82.8 82.8 83.6 84.5 88.1 93.9 198 SelfSup [62] 98.4 -
errquat 0.28 0.28 0.27 0.28 0.29 0.30 0.33 1.97 Pavlakos et al. [38] 75.9 -
HMR (H36M-trained)[22] 77.6 72.1
Table 4: Effect of 3D labeled data. We show the 3D as well HMR [22] 56.8 -
as the estimated body part rotation error for varying ratios Ours 59.9 52.3
of data with 3D labels. For all of the data, we assume that
2D pose labels are available. Both errors saturate at 20% of Table 6: Human 3.6M. 3D joint errors in mm.
3D labeled training examples.

the UP-3D dataset can be found in Figure 4. We show ex-

tation, which is needed for many applications. amples from the four different error quartiles. Fits from the
We also observe that only training with the 2D repro- first three quartiles still reproduce the body pose somewhat
jection loss (perhaps unsurprisingly) results in poor perfor- faithfully, and only in the last row and percentile, problems
mance in terms of 3D error, showing that some amount of become clearly visible. We show more failure modes in the
3D annotations are necessary to overcome the ambiguity in- supplementary material.
herent to 2D keypoints as a source of supervision for 3D.
Due to the SMPL layers, we can supervise learning with
4.4. Comparison to State-of-the-Art
any number of joints/mesh vertices. We thus experimented Here we compare to the state of the art on HumanEva-I
with the 91 landmarks used by [24] for their fitting method (Table 5) and Human3.6M (Table 6). We perform a per-
but find that the 24 SMPL joints are sufficient in this setting. frame rigid alignment of the 3D estimates to the ground
truth using Procrustes Analysis and report results in terms
How Much 3D Supervision Do We Need? The use of of reconstruction error, i.e. the mean per joint position error
these additional loss terms also allows us to leverage data after alignment (given in mm). The model we use here is
for which no 3D annotations are available. With the follow- trained on Human3.6M data.
ing set of experiments, we attempt to answer two questions: We compare favourably to similar methods, but these are
(i) Given a small amount of 3D-annotated data, does ex- not strictly comparable since they train on different datasets.
tra 2D-annotated data help?, (ii) What amount of 3D data Pavlakos et al. [38] do not use any data from Human3.6M,
is necessary? To this end we train multiple networks, each whereas HMR [22] does, along with several other datasets.
time progressively disabling the 3D latent loss and replacing We retrained the latter with the original code only using Hu-
it with the 2D loss for more training examples. The results man3.6M data for a more direct comparison to ours (HMR
are depicted in Table 4. We find that performance barely (H36M-trained) in Table 6). Given Table 1, we hypothesize
degrades as long as we have a small amount of 3D anno- that their approach requires more training data for good per-
tations. In contrast, using small amounts of 3D data and formance because it uses RGB images as input.
no extra data with 2D annotations yields poor performance.
This is an important finding since obtaining 3D annotations
5. Conclusion
is difficult compared to simple 2D keypoint annotations.
In this paper, we make several principled steps towards
Qualitative Results A selection of qualitative results from a full integration of parametric 3D human pose models into
Figure 4: Qualitative results by error quartile in terms of errjoints3D . The rows show representative examples from different
error quartiles, top to bottom: 0-25%, 25-50%, 50-75%, 75-100%

deep CNN architectures. We analyze (1) how the 3D model enables end-to-end training. We found a loss that combines
can be integrated into a deep neural network, (2) how loss 2D as well as 3D information to work best. The flexible im-
functions can be combined and (3) how a training can be set plementation allowed us to experiment with the 3D losses
up that works efficiently with scarce 3D data. only for parts of the data, moving towards a weakly su-
In contrast to existing methods we use a region-based pervised training scenario that avoids expensive 3D labeled
2D representation, namely a 12-body-part segmentation, as data. With 3D information for only 20% of our training
an intermediate step prior to the mapping to 3D shape and data, we could reach similar performance as with full 3D
pose. This segmentation provides full spatial coverage of annotations.
a person as opposed to the commonly used sparse set of We believe that this encouraging result is an important
keypoints, while also retaining enough information about finding for the design of future datasets and the develop-
the arrangement of parts to allow for effective lifting to 3D. ment of 3D prediction methods that do not require expen-
We used a stack of CNN layers on top of a segmentation sive 3D annotations for training. Future work will involve
model to predict an encoding in the space of 3D model pa- extending this to more challenging settings involving mul-
rameters, followed by a Tensorflow implementation of the tiple, possibly occluded, people.
3D model and a projection to the image plane. This full Acknowledgements We would like to thank Dingfan
integration allows us to finely tune the loss functions and Chen for help with training the HMR [22] model.
References [13] N. Hasler, C. Stoll, M. Sunkel, B. Rosenhahn, and H.-
P. Seidel. A statistical model of human pose and body
[1] I. Akhter and M. J. Black. Pose-conditioned joint an- shape. In Computer Graphics Forum, volume 28, pages
gle limits for 3d human pose reconstruction. In IEEE 337–346, 2009. 2
Conference on Computer Vision and Pattern Recogni-
tion (CVPR), pages 1446–1455, 2015. 3, 7 [14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual
learning for image recognition. In IEEE Conference
[2] T. Alldieck, M. Magnor, W. Xu, C. Theobalt, and on Computer Vision and Pattern Recognition (CVPR),
G. Pons-Moll. Detailed human avatars from monocu- 2016. 5, 6
lar video. In 3D Vision (3DV), 2018 Sixth International
Conference on, 2018. 3 [15] E. Insafutdinov, M. Andriluka, L. Pishchulin, S. Tang,
E. Levinkov, B. Andres, and B. Schiele. Arttrack: Ar-
[3] T. Alldieck, M. Magnor, W. Xu, C. Theobalt, and ticulated multi-person tracking in the wild. In 30th
G. Pons-Moll. Video based reconstruction of 3d peo- IEEE Conference on Computer Vision and Pattern
ple models. In IEEE Conference on Computer Vision Recognition (CVPR 2017), Honolulu, HI, USA, 2017.
and Pattern Recognition, June 2018. 3 IEEE. 2
[4] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. [16] E. Insafutdinov, L. Pishchulin, B. Andres, M. An-
2D Human Pose Estimation: New Benchmark and State driluka, and B. Schiele. Deepercut: A deeper, stronger,
of the Art Analysis. In IEEE Conference on Computer and faster multi-person pose estimation model. In Eu-
Vision and Pattern Recognition (CVPR), June 2014. 5, ropean Conference on Computer Vision (ECCV), 2016.
6 5
[5] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, [17] C. Ionescu, D. Papava, V. Olaru, and C. Sminchis-
J. Rodgers, and J. Davis. Scape: shape completion and escu. Human3.6m: Large scale datasets and predictive
animation of people. In ACM Transactions on Graphics methods for 3d human sensing in natural environments.
(TOG), volume 24, pages 408–416. ACM, 2005. 2 IEEE Transactions on Pattern Analysis and Machine
[6] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, Intelligence (PAMI), 36(7):1325–1339, 2014. 3, 5
J. Romero, and M. J. Black. Keep it SMPL: Automatic [18] D. Jack, F. Maire, A. Eriksson, and S. Shirazi. Adver-
estimation of 3D human pose and shape from a single sarially parameterized optimization for 3d human pose
image. In European Conference on Computer Vision estimation. 2017. 2
(ECCV), 2016. 1, 2, 7
[19] E. Jahangiri and A. L. Yuille. Generating multiple
[7] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Real- diverse hypotheses for human 3d pose consistent with
time multi-person 2d pose estimation using part affinity 2d joint detections. In IEEE International Conference
fields. In Proc. of CVPR, 2016. 2 on Computer Vision (ICCV) Workshops (PeopleCap),
[8] C.-H. Chen and D. Ramanan. 3d human pose esti- 2017. 3
mation = 2d pose estimation + matching. In CVPR [20] S. Johnson and M. Everingham. Clustered pose and
2017-IEEE Conference on Computer Vision & Pattern nonlinear appearance models for human pose estima-
Recognition, 2017. 3 tion. In British Machine Vision Conference (BMVC),
[9] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, 2010. doi:10.5244/C.24.12. 5
and A. L. Yuille. Deeplab: Semantic image seg- [21] H. Joo, T. Simon, and Y. Sheikh. Total capture: A 3d
mentation with deep convolutional nets, atrous con- deformation model for tracking faces, hands, and bod-
volution, and fully connected crfs. arXiv preprint ies. arXiv preprint arXiv:1801.01615, 2018. 2
arXiv:1606.00915, 2016. 6 [22] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik.
[10] M. Dantone, J. Gall, C. Leistner, and L. V. Gool. Body End-to-end recovery of human shape and pose. In Pro-
parts dependent joint regressors for human pose esti- ceedings of the IEEE Conference on Computer Vision
mation in still images. IEEE Transactions on Pattern and Pattern Recognition, 2018. 1, 2, 3, 7, 8
Analysis and Machine Intelligence, 36(11):2131–2143, [23] C. Lassner, G. Pons-Moll, and P. V. Gehler. A gen-
Nov 2014. 5 erative model of people in clothing. In International
[11] D. M. Gavrila and L. S. Davis. 3-d model-based track- Conference on Computer Vision (ICCV), 2017. 3
ing of humans in action: a multi-view approach. In [24] C. Lassner, J. Romero, M. Kiefel, F. Bogo, M. J.
Proc. CVPR, pages 73–80. IEEE, 1996. 2 Black, and P. V. Gehler. Unite the people: Closing
[12] R. A. Güler, N. Neverova, and I. Kokkinos. Dense- the loop between 3d and 2d human representations. In
pose: Dense human pose estimation in the wild. arXiv IEEE Conf. on Computer Vision and Pattern Recogni-
preprint arXiv:1802.00434, 2018. 3 tion (CVPR), 2017. 1, 2, 4, 5, 7
[25] S. Li and A. B. Chan. 3d human pose estimation [37] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Dani-
from monocular images with deep convolutional neu- ilidis. Coarse-to-fine volumetric prediction for single-
ral network. In Asian Conference on Computer Vision image 3D human pose. In CVPR 2017-IEEE Confer-
(ACCV), pages 332–347, 2014. 3 ence on Computer Vision & Pattern Recognition, 2017.
[26] S. Li, W. Zhang, and A. B. Chan. Maximum-margin 3
structured learning with deep networks for 3d human [38] G. Pavlakos, L. Zhu, X. Zhou, and K. Daniilidis.
pose estimation. In IEEE International Conference on Learning to estimate 3D human pose and shape from
Computer Vision (ICCV), pages 2848–2856, 2015. 3 a single color image. In Proceedings of the IEEE Con-
[27] G. Lin and I. R. Anton Milan, Chunhua Shen. Re- ference on Computer Vision and Pattern Recognition,
finenet: Multi-path refinement networks for high- 2018. 1, 3, 4, 7
resolution semantic segmentation. In Computer Vision [39] R. Plankers and P. Fua. Articulated soft objects for
and Pattern Recognition (CVPR), 2017. 5 video-based body modeling. In International Confer-
[28] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, ence on Computer Vision, Vancouver, Canada, number
and M. J. Black. SMPL: A skinned multi-person lin- CVLAB-CONF-2001-005, pages 394–401, 2001. 2
ear model. ACM Trans. Graphics (Proc. SIGGRAPH [40] G. Pons-Moll, D. J. Fleet, and B. Rosenhahn. Posebits
Asia), 34(6):248:1–248:16, Oct. 2015. 1, 2, 3 for monocular human pose estimation. In IEEE Con-
[29] M. M. Loper, N. Mahmood, and M. J. Black. MoSh: ference on Computer Vision and Pattern Recognition
Motion and shape capture from sparse markers. ACM (CVPR), pages 2337–2344, 2014. 3
Transactions on Graphics, (Proc. SIGGRAPH Asia), [41] G. Pons-Moll, J. Romero, N. Mahmood, and M. J.
33(6):220:1–220:13, Nov. 2014. 5 Black. Dyna: a model of dynamic human shape in mo-
tion. ACM Transactions on Graphics (TOG), 34:120,
[30] J. Martinez, R. Hossain, J. Romero, and J. J. Little. A
2015. 2
simple yet effective baseline for 3d human pose estima-
tion. In IEEE International Conference on Computer [42] G. Pons-Moll and B. Rosenhahn. Model-Based Pose
Vision (ICCV), 2017. 3 Estimation, chapter 9, pages 139–170. Springer, 2011.
2
[31] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotny-
chenko, W. Xu, and C. Theobalt. Monocular 3d human [43] G. Pons-Moll, J. Taylor, J. Shotton, A. Hertzmann,
pose estimation in the wild using improved cnn super- and A. Fitzgibbon. Metric regression forests for cor-
vision. In 3D Vision (3DV), 2017 Fifth International respondence estimation. International Journal of Com-
Conference on. IEEE, 2017. 3 puter Vision, 113(3):163–175, 2015. 3
[32] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, S. Srid- [44] A.-I. Popa, M. Zanfir, and C. Sminchisescu. Deep
har, G. Pons-Moll, and C. Theobalt. Single-shot multi- multitask architecture for integrated 2d and 3d human
person 3d pose estimation from monocular rgb. In 3D sensing. IEEE Conference on Computer Vision and Pat-
Vision (3DV), 2018 Sixth International Conference on, tern Recognition (CVPR), 2017. 3
2018. 3 [45] V. Ramakrishna, T. Kanade, and Y. Sheikh. Recon-
[33] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, structing 3d human pose from 2d image landmarks. In
M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, and European Conference on Computer Vision, pages 573–
C. Theobalt. Vnect: Real-time 3d human pose esti- 586. Springer, 2012. 7
mation with a single rgb camera. ACM Transactions on [46] G. Rogez and C. Schmid. Mocap-guided data aug-
Graphics, 36(4), July 2017. 3 mentation for 3d pose estimation in the wild. In
[34] D. Metaxas and D. Terzopoulos. Shape and non- Advances in Neural Information Processing Systems,
rigid motion estimation through physics-based synthe- pages 3108–3116, 2016. 3
sis. IEEE Trans. PAMI, 15(6):580–591, 1993. 2 [47] G. Rogez, P. Weinzaepfel, and C. Schmid. Lcr-net++:
[35] F. Moreno-Noguer. 3d human pose estimation from a Multi-person 2d and 3d pose detection in natural im-
single image via distance matrix regression. In CVPR ages. arXiv preprint arXiv:1803.00455, 2018. 3
2017-IEEE Conference on Computer Vision & Pattern [48] N. Sarafianos, B. Boteanu, B. Ionescu, and I. A. Kaka-
Recognition, 2017. 3 diaris. 3d human pose estimation: A review of the liter-
[36] G. Pavlakos, X. Zhou, and K. Daniilidis. Ordinal ature and analysis of covariates. Computer Vision and
depth supervision for 3d human pose estimation. In Image Understanding, 152:1–20, 2016. 2
Proceedings of the IEEE Conference on Computer Vi- [49] L. Sigal, A. O. Balan, and M. J. Black. Humaneva:
sion and Pattern Recognition, 2018. 3 Synchronized video and motion capture dataset and
 baseline algorithm for evaluation of articulated human [61] A. Tewari, M. Zollhöfer, H. Kim, P. Garrido,
motion. International Journal of Computer Vision F. Bernard, P. Perez, and C. Theobalt. Mofa: Model-
(IJCV), 87(1-2):4–27, 2010. 3, 5 based deep convolutional face autoencoder for unsu-
pervised monocular reconstruction. In The IEEE Inter-
[50] L. Sigal, S. Bhatia, S. Roth, M. J. Black, and M. Is-
national Conference on Computer Vision (ICCV), vol-
ard. Tracking loose-limbed people. In Computer Vision
ume 2, 2017. 3
and Pattern Recognition, 2004. CVPR 2004. Proceed-
ings of the 2004 IEEE Computer Society Conference [62] H.-Y. Tung, H.-W. Tung, E. Yumer, and K. Fragki-
on, volume 1, pages I–421. IEEE, 2004. 2 adaki. Self-supervised learning of motion capture. In
[51] E. Simo-Serra, A. Quattoni, C. Torras, and F. Moreno- Advances in Neural Information Processing Systems,
Noguer. A joint model for 2d and 3d pose estima- pages 5242–5252, 2017. 1, 3, 7
tion from a single image. In Conference on Computer [63] G. Varol, D. Ceylan, B. Russell, J. Yang, E. Yumer,
Vision and Pattern Recognition (CVPR), pages 3634– I. Laptev, and C. Schmid. Bodynet: Volumetric in-
3641, 2013. 3 ference of 3d human body shapes. arXiv preprint
[52] K. Simonyan and A. Zisserman. Very deep con- arXiv:1804.04875, 2018. 3
volutional networks for large-scale image recognition. [64] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J.
arXiv preprint arXiv:1409.1556, 2014. 6 Black, I. Laptev, and C. Schmid. Learning from Syn-
[53] C. Sminchisescu and B. Triggs. Kinematic jump pro- thetic Humans. In CVPR, 2017. 3
cesses for monocular 3d human tracking. In Com- [65] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh.
puter Vision and Pattern Recognition, 2003. Proceed- Convolutional pose machines. In 2015 IEEE Con-
ings. 2003 IEEE Computer Society Conference on, vol- ference on Computer Vision and Pattern Recognition
ume 1, pages I–I. IEEE, 2003. 3 (CVPR), 2016. 2
[54] C. Stoll, N. Hasler, J. Gall, H.-P. Seidel, and [66] H. Yasin, U. Iqbal, B. Krüger, A. Weber, and J. Gall.
C. Theobalt. Fast articulated motion tracking using a A Dual-Source Approach for 3D Pose Estimation from
sums of gaussians body model. In Proc. ICCV, pages a Single Image. In Conference on Computer Vision and
951–958. IEEE, 2011. 2 Pattern Recognition (CVPR), 2016. 3
[55] X. Sun, J. Shang, S. Liang, and Y. Wei. Com- [67] X. Zhou, Q. Huang, X. Sun, X. Xue, and Y. Wei. To-
positional human pose regression. arXiv preprint wards 3d human pose estimation in the wild: A weakly-
arXiv:1704.00159, 2017. 3 supervised approach. In Proceedings of the IEEE Con-
[56] J. K. V. Tan, I. Budvytis, and R. Cipolla. Indirect deep ference on Computer Vision and Pattern Recognition,
structured learning for 3d human body shape and pose pages 398–407, 2017. 3
prediction. In BMVC, volume 3, page 6, 2017. 1 [68] X. Zhou, S. Leonardos, X. Hu, and K. Daniilidis. 3D
[57] C. J. Taylor. Reconstruction of articulated objects shape estimation from 2D landmarks: A convex re-
from point correspondences in a single uncalibrated im- laxation approach. In IEEE Conference on Computer
age. In IEEE Conference on Computer Vision and Pat- Vision and Pattern Recognition (CVPR), pages 4447–
tern Recognition (CVPR), volume 1, pages 677–684, 4455, 2015. 7
2000. 3 [69] X. Zhou, M. Zhu, S. Leonardos, K. Derpanis, and
[58] J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon. The K. Daniilidis. Sparseness Meets Deepness: 3D Hu-
vitruvian manifold: Inferring dense correspondences man Pose Estimation from Monocular Video. In IEEE
for one-shot human pose estimation. In Computer Vi- Conference on Computer Vision and Pattern Recogni-
sion and Pattern Recognition (CVPR), 2012 IEEE Con- tion (CVPR), 2015. 3
ference on, pages 103–110. IEEE, 2012. 3 [70] X. Zhou, M. Zhu, G. Pavlakos, S. Leonardos, K. G.
[59] B. Tekin, I. Katircioglu, M. Salzmann, V. Lepetit, and Derpanis, and K. Daniilidis. Monocap: Monocular hu-
P. Fua. Structured Prediction of 3D Human Pose with man motion capture using a cnn coupled with a geo-
Deep Neural Networks. In British Machine Vision Con- metric prior. IEEE Transactions on Pattern Analysis
ference (BMVC), 2016. 3 and Machine Intelligence, 2018. 3
[60] B. Tekin, P. Márquez-Neila, M. Salzmann, and P. Fua. [71] S. Zuffi and M. J. Black. The stitched puppet: A
Learning to fuse 2d and 3d image cues for monocular graphical model of 3d human shape and pose. In
body pose estimation. In IEEE International Confer- 2015 IEEE Conference on Computer Vision and Pattern
ence on Computer Vision (ICCV), 2017. 3 Recognition (CVPR), pages 3537–3546. IEEE, 2015. 2
A. Further Qualitative Results critical to train both the segmentation network and the fit-
ting network with strong data augmentation, especially by
One of our findings is the high correlation between in- introducing random jitter and scaling. For the fitting net-
put segmentation quality and output fit quality. We pro- work, such augmentation has to take place prior to training
vide some additional qualitative examples that illustrate this since it affects the SMPL parameters. We also mirror the
correlation. In Fig. 5, we present the four worst examples data, but this requires careful mirroring of both the part la-
from the validation set in terms of 3D joint reconstruction bels as well as the SMPL parameters. This involves remap-
error when we use our trained part segmentation network; in ping the parts, as well as inverting the part rotations.
Fig. 6, we present the worst examples when the network is
trained to predict body model parameters given the ground References
truth segmentations. This does not correct all estimated 3D
bodies, but the remaining errors are noticeably less severe. [1] V. Belagiannis, C. Rupprecht, G. Carneiro, and
N. Navab. Robust optimization for deep regression.
B. Training Details 2015.
[2] S. Geman and D. McClure. Statistical methods for
We present examples of paired training examples and
tomographic image reconstruction. Bulletin of the In-
ground truth in Fig 7.
ternational Statistical Institute, 52(4):5–21, 1987.
Segmentation Network We train our own TensorFlow [3] D. P. Kingma and J. Ba. Adam: A method for stochas-
implementation of a RefineNet [4] network (based on tic optimization. 2014.
ResNet-101) to predict the part segmentations. The images [4] G. Lin and I. R. Anton Milan, Chunhua Shen. Re-
are cropped to 512x512 pixels, and we train for 20 epochs finenet: Multi-path refinement networks for high-
with a batch size of 5 using the Adam [3] optimizer. Learn- resolution semantic segmentation. In Computer Vision
ing rate and weight decay are set to 0.00002 and 0.0001 and Pattern Recognition (CVPR), 2017.
respectively, with a polynomial learning rate decay. Data
augmentation improved performance a lot, in particular hor-
izontal reflection (which requires re-mapping the labels for
left and right limbs), scale augmentation (0.9 - 1.1 of the
original size) as well as rotations (up to 45 degrees). For
training the segmentation network on UP-3D we used the
5703 training images. For Human3.6M we subsampled the
videos, only using every 10th frame from each video, which
results in about 32000 frames. Depending on the amount of
data, training the segmentation networks takes about 6-12
hours on a Volta V100 machine.

Fitting Network For the fitting network we repurpose a

ResNet-50 network pretrained on ImageNet to regress the
SMPL model parameters. We replace the final pooling layer
with a single fully-connected layer that outputs the 10 shape
and 216 pose parameters. We train this network for 75
epochs with a batch size of 5 using the Adam optimizer.
The learning rate is set to 0.00004 with polynomial decay
and we use a weight decay setting of 0.0001. We found that
an L1 loss on the SMPL parameters was a little better than
an L2 loss. We also experimented with robust losses (e.g.
Geman-McClure [2] and Tukey’s biweight loss [1]) but did
not observe benefits. Training this network takes about 1.5
hours for the UP-3D dataset and six hours for Human3.6M.

Data Augmentation At test-time we cannot guarantee

that the person will be perfectly centered in the input crop,
which can lead to degraded performance. We found it thus
Figure 5: Worst examples from the validation set in terms of 3D error given imperfect segmentations.

Figure 6: Worst examples from the validation set in terms of 3D error given perfect segmentations.

Figure 7: Example training images annotations illustrating different types and granularities.