Learning Spatiotemporal Features with 3D Convolutional Networks

Du Tran1,2 , Lubomir Bourdev1 , Rob Fergus1 , Lorenzo Torresani2 , Manohar Paluri1

1
Facebook AI Research, 2 Dartmouth College
{dutran,lorenzo}@cs.dartmouth.edu {lubomir,robfergus,mano}@fb.com

Abstract ing, and retrieving tasks much more scalable; (iii) it needs to
arXiv:1412.0767v4 [cs.CV] 7 Oct 2015

be efficient to compute, as thousands of videos are expected

We propose a simple, yet effective approach for spa- to be processed every minute in real world systems; and
tiotemporal feature learning using deep 3-dimensional con- (iv) it must be simple to implement. Instead of using com-
volutional networks (3D ConvNets) trained on a large scale plicated feature encoding methods and classifiers, a good
supervised video dataset. Our findings are three-fold: 1) descriptor should work well even with a simple model (e.g.
3D ConvNets are more suitable for spatiotemporal feature linear classifier).
learning compared to 2D ConvNets; 2) A homogeneous ar- Inspired by the deep learning breakthroughs in the image
chitecture with small 3 × 3 × 3 convolution kernels in all domain [24] where rapid progress has been made in the past
layers is among the best performing architectures for 3D few years in feature learning, various pre-trained convolu-
ConvNets; and 3) Our learned features, namely C3D (Con- tional network (ConvNet) models [16] are made available
volutional 3D), with a simple linear classifier outperform for extracting image features. These features are the activa-
state-of-the-art methods on 4 different benchmarks and are tions of the network’s last few fully-connected layers which
comparable with current best methods on the other 2 bench- perform well on transfer learning tasks [47, 48]. However,
marks. In addition, the features are compact: achieving such image based deep features are not directly suitable for
52.8% accuracy on UCF101 dataset with only 10 dimen- videos due to lack of motion modeling (as shown in our
sions and also very efficient to compute due to the fast in- experiments in sections 4,5,6). In this paper we propose
ference of ConvNets. Finally, they are conceptually very to learn spatio-temporal features using deep 3D ConvNet.
simple and easy to train and use. We empirically show that these learned features with a sim-
ple linear classifier can yield good performance on various
1. Introduction video analysis tasks. Although 3D ConvNets were proposed
before [15, 18], to our knowledge this work exploits 3D
Multimedia on the Internet is growing rapidly result- ConvNets in the context of large-scale supervised training
ing in an increasing number of videos being shared every datasets and modern deep architectures to achieve the best
minute. To combat the information explosion it is essen- performance on different types of video analysis tasks. The
tial to understand and analyze these videos for various pur- features from these 3D ConvNets encapsulate information
poses like search, recommendation, ranking etc. The com- related to objects, scenes and actions in a video, making
puter vision community has been working on video analysis them useful for various tasks without requiring to finetune
for decades and tackled different problems such as action the model for each task. C3D has the properties that a good
recognition [26], abnormal event detection [2], and activity descriptor should have: it is generic, compact, simple and
understanding [23]. Considerable progress has been made efficient. To summarize, our contributions in this paper are:
in these individual problems by employing different spe-
cific solutions. However, there is still a growing need for • We experimentally show 3D convolutional deep net-
a generic video descriptor that helps in solving large-scale works are good feature learning machines that model
video tasks in a homogeneous way. appearance and motion simultaneously.
There are four properties for an effective video descrip- • We empirically find that 3 × 3 × 3 convolution kernel
tor: (i) it needs to be generic, so that it can represent differ- for all layers to work best among the limited set of
ent types of videos well while being discriminative. For ex- explored architectures.
ample, Internet videos can be of landscapes, natural scenes, • The proposed features with a simple linear model out-
sports, TV shows, movies, pets, food and so on; (ii) the de- perform or approach the current best methods on 4 dif-
scriptor needs to be compact: as we are working with mil- ferent tasks and 6 different benchmarks (see Table 1).
lions of videos, a compact descriptor helps processing, stor- They are also compact and efficient to compute.

1
 Dataset Sport1M UCF101 ASLAN YUPENN UMD Object
Task action recognition action recognition action similarity labeling scene classification scene classification object recognition
Method [29] [39]([25]) [31] [9] [9] [32]
Result 90.8 75.8 (89.1) 68.7 96.2 77.7 12.0
C3D 85.2 85.2 (90.4) 78.3 98.1 87.7 22.3
Table 1. C3D compared to best published results. C3D outperforms all previous best reported methods on a range of benchmarks except
for Sports-1M and UCF101. On UCF101, we report accuracy for two groups of methods. The first set of methods use only RGB frame
inputs while the second set of methods (in parentheses) use all possible features (e.g. optical flow, improved Dense Trajectory).

2. Related Work video classification. Simonyan and Zisserman [36] used

two stream networks to achieve best results on action recog-
Videos have been studied by the computer vision com- nition.
munity for decades. Over the years various problems like Among these approaches, the 3D ConvNets approach
action recognition [26], anomaly detection [2], video re- in [15] is most closely related to us. This method used a hu-
trieval [1], event and action detection [30, 17], and many man detector and head tracking to segment human subjects
more have been proposed. Considerable portion of these in videos. The segmented video volumes are used as inputs
works are about video representations. Laptev and Linde- for a 3-convolution-layer 3D ConvNet to classify actions. In
berg [26] proposed spatio-temporal interest points (STIPs) contrast, our method takes full video frames as inputs and
by extending Harris corner detectors to 3D. SIFT and HOG does not rely on any preprocessing, thus easily scaling to
are also extended into SIFT-3D [34] and HOG3D [19] for large datasets. We also share some similarities with Karpa-
action recognition. Dollar et al. proposed Cuboids features thy et al. [18] and Simonyan and Zisserman [36] in terms
for behavior recognition [5]. Sadanand and Corso built Ac- of using full frames for training the ConvNet. However,
tionBank for action recognition [33]. Recently, Wang et al. these methods are built on using only 2D convolution and
proposed improved Dense Trajectories (iDT) [44] which is 2D pooling operations (except for the Slow Fusion model
currently the state-of-the-art hand-crafted feature. The iDT in [18]) whereas our model performs 3D convolutions and
descriptor is an interesting example showing that temporal 3D pooling propagating temporal information across all the
signals could be handled differently from that of spatial sig- layers in the network (further detailed in section 3). We also
nal. Instead of extending Harris corner detector into 3D, it show that gradually pooling space and time information and
starts with densely-sampled feature points in video frames building deeper networks achieves best results and we dis-
and uses optical flows to track them. For each tracker cor- cuss more about the architecture search in section 3.2.
ner different hand-crafted features are extracted along the
trajectory. Despite its good performance, this method is 3. Learning Features with 3D ConvNets
computationally intensive and becomes intractable on large-
scale datasets. In this section we explain in detail the basic operations of
With recent availability of powerful parallel machines 3D ConvNets, analyze different architectures for 3D Con-
(GPUs, CPU clusters), together with large amounts of train- vNets empirically, and elaborate how to train them on large-
ing data, convolutional neural networks (ConvNets) [28] scale datasets for feature learning.
have made a come back providing breakthroughs on visual
3.1. 3D convolution and pooling
recognition [10, 24]. ConvNets have also been applied to
the problem of human pose estimation in both images [12] We believe that 3D ConvNet is well-suited for spatiotem-
and videos [13]. More interestingly these deep networks poral feature learning. Compared to 2D ConvNet, 3D Con-
are used for image feature learning [7]. Similarly, Zhou et vNet has the ability to model temporal information better
al. and perform well on transferred learning tasks. Deep owing to 3D convolution and 3D pooling operations. In
learning has also been applied to video feature learning in 3D ConvNets, convolution and pooling operations are per-
an unsupervised setting [27]. In Le et al. [27], the au- formed spatio-temporally while in 2D ConvNets they are
thors use stacked ISA to learn spatio-temporal features for done only spatially. Figure 1 illustrates the difference, 2D
videos. Although this method showed good results on ac- convolution applied on an image will output an image, 2D
tion recognition, it is still computationally intensive at train- convolution applied on multiple images (treating them as
ing and hard to scale up for testing on large datasets. 3D different channels [36]) also results in an image. Hence,
ConvNets were proposed for human action recognition [15] 2D ConvNets lose temporal information of the input sig-
and for medical image segmentation [14, 42]. 3D convo- nal right after every convolution operation. Only 3D con-
lution was also used with Restricted Boltzmann Machines volution preserves the temporal information of the input
to learn spatiotemporal features [40]. Recently, Karpathy et signals resulting in an output volume. The same phenom-
al. [18] trained deep networks on a large video dataset for ena is applicable for 2D and 3D polling. In [36], although
 k
H k k k
H L H d<L
output k k
output
L L output
W
W W
(a) 2D convolution (b) 2D convolution on multiple frames (c) 3D convolution

Figure 1. 2D and 3D convolution operations. a) Applying 2D convolution on an image results in an image. b) Applying 2D convolution
on a video volume (multiple frames as multiple channels) also results in an image. c) Applying 3D convolution on a video volume results
in another volume, preserving temporal information of the input signal.

the temporal stream network takes multiple frames as input, size of d where d is the kernel temporal depth (we will later
because of the 2D convolutions, after the first convolution vary the value d of these layers to search for a good 3D ar-
layer, temporal information is collapsed completely. Simi- chitecture). All of these convolution layers are applied with
larly, fusion models in [18] used 2D convolutions, most of appropriate padding (both spatial and temporal) and stride
the networks lose their input’s temporal signal after the first 1, thus there is no change in term of size from the input
convolution layer. Only the Slow Fusion model in [18] uses to the output of these convolution layers. All pooling lay-
3D convolutions and averaging pooling in its first 3 convo- ers are max pooling with kernel size 2 × 2 × 2 (except for
lution layers. We believe this is the key reason why it per- the first layer) with stride 1 which means the size of output
forms best among all networks studied in [18]. However, it signal is reduced by a factor of 8 compared with the input
still loses all temporal information after the third convolu- signal. The first pooling layer has kernel size 1 × 2 × 2
tion layer. with the intention of not to merge the temporal signal too
In this section, we empirically try to identify a good ar- early and also to satisfy the clip length of 16 frames (e.g.
chitecture for 3D ConvNets. Because training deep net- we can temporally pool with factor 2 at most 4 times be-
works on large-scale video datasets is very time-consuming, fore completely collapsing the temporal signal). The two
we first experiment with UCF101, a medium-scale dataset, fully connected layers have 2048 outputs. We train the net-
to search for the best architecture. We verify the findings on works from scratch using mini-batches of 30 clips, with ini-
a large scale dataset with a smaller number of network ex- tial learning rate of 0.003. The learning rate is divided by
periments. According to the findings in 2D ConvNet [37], 10 after every 4 epochs. The training is stopped after 16
small receptive fields of 3 × 3 convolution kernels with epochs.
deeper architectures yield best results. Hence, for our ar- Varying network architectures: For the purposes of
chitecture search study we fix the spatial receptive field to this study we are mainly interested in how to aggregate tem-
3 × 3 and vary only the temporal depth of the 3D convolu- poral information through the deep networks. To search
tion kernels. for a good 3D ConvNet architecture, we only vary kernel
Notations: For simplicity, from now on we refer video temporal depth di of the convolution layers while keeping
clips with a size of c × l × h × w where c is the number of all other common settings fixed as stated above. We ex-
channels, l is length in number of frames, h and w are the periment with two types of architectures: 1) homogeneous
height and width of the frame, respectively. We also refer temporal depth: all convolution layers have the same ker-
3D convolution and pooling kernel size by d × k × k, where nel temporal depth; and 2) varying temporal depth: kernel
d is kernel temporal depth and k is kernel spatial size. temporal depth is changing across the layers. For homoge-
Common network settings: In this section we describe neous setting, we experiment with 4 networks having ker-
the network settings that are common to all the networks we nel temporal depth of d equal to 1, 3, 5, and 7. We name
trained. The networks are set up to take video clips as inputs these networks as depth-d, where d is their homogeneous
and predict the class labels which belong to 101 different temporal depth. Note that depth-1 net is equivalent to ap-
actions. All video frames are resized into 128 × 171. This plying 2D convolutions on separate frames. For the varying
is roughly half resolution of the UCF101 frames. Videos temporal depth setting, we experiment two networks with
are split into non-overlapped 16-frame clips which are then temporal depth increasing: 3-3-5-5-7 and decreasing: 7-
used as input to the networks. The input dimensions are 5-5-3-3 from the first to the fifth convolution layer respec-
3 × 16 × 128 × 171. We also use jittering by using random tively. We note that all of these networks have the same size
crops with a size of 3 × 16 × 112 × 112 of the input clips of the output signal at the last pooling layer, thus they have
during training. The networks have 5 convolution layers the same number of parameters for fully connected layers.
and 5 pooling layers (each convolution layer is immediately Their number of parameters is only different at convolution
followed by a pooling layer), 2 fully-connected layers and layers due to different kernel temporal depth. These differ-
a softmax loss layer to predict action labels. The number ences are quite minute compared to millions of parameters
of filters for 5 convolution layers from 1 to 5 are 64, 128, in the fully connected layers. For example, any two of the
256, 256, 256, respectively. All convolution kernels have a above nets with temporal depth difference of 2, only has
 0.5 0.46

0.45
0.44 we call this net C3D from now on. All of 3D convolution
0.4
0.42
filters are 3 × 3 × 3 with stride 1 × 1 × 1. All 3D pooling

clip accuracy
clip accuracy

0.4

0.35 0.38
layers are 2 × 2 × 2 with stride 2 × 2 × 2 except for pool1
0.3 0.36 which has kernel size of 1 × 2 × 2 and stride 1 × 2 × 2
0.34
0.25
depth−1 0.32
with the intention of preserving the temporal information in
depth−3 depth−3
0.2
depth−5
depth−7
0.3 increase
descrease
the early phase. Each fully connected layer has 4096 output
0 2 4 6 8
# epoch
10 12 14 16 0 2 4 6 8
# epoch
10 12 14 16 units.
Figure 2. 3D convolution kernel temporal depth search. Action Dataset. To learn spatiotemproal features, we train
recognition clip accuracy on UCF101 test split-1 of different ker- our C3D on Sports-1M dataset [18] which is currently the
nel temporal depth settings. 2D ConvNet performs worst and 3D largest video classification benchmark. The dataset consists
ConvNet with 3 × 3 × 3 kernels performs best among the experi- of 1.1 million sports videos. Each video belongs to one
mented nets. of 487 sports categories. Compared with UCF101, Sports-
17K parameters fewer or more from each other. The biggest 1M has 5 times the number of categories and 100 times the
difference in number of parameters is between depth-1 net number of videos.
and depth-7 net where depth-7 net has 51K more parame- Training: Training is done on the Sports-1M train split.
ters which is less than 0.3% of the total of 17.5 millions pa- As Sports-1M has many long videos, we randomly extract
rameters of each network. This indicates that the learning five 2-second long clips from every training video. Clips are
capacity of the networks are comparable and the differences resized to have a frame size of 128 × 171. On training, we
in number of parameters should not affect the results of our randomly crop input clips into 16×112×112 crops for spa-
architecture search. tial and temporal jittering. We also horizontally flip them
with 50% probability. Training is done by SGD with mini-
3.2. Exploring kernel temporal depth batch size of 30 examples. Initial learning rate is 0.003,
We train these networks on the train split 1 of UCF101. and is divided by 2 every 150K iterations. The optimization
Figure 2 presents clip accuracy of different architectures on is stopped at 1.9M iterations (about 13 epochs). Beside the
UCF101 test split 1. The left plot shows results of nets with C3D net trained from scratch, we also experiment with C3D
homogeneous temporal depth and the right plot presents re- net fine-tuned from the model pre-trained on I380K.
sults of nets that changing kernel temporal depth. Depth- Sports-1M classification results: Table 2 presents
3 performs best among the homogeneous nets. Note that the results of our C3D networks compared with Deep-
depth-1 is significantly worse than the other nets which we Video [18] and Convolution pooling [29]. We use only a
believe is due to lack of motion modeling. Compared to the single center crop per clip, and pass it through the network
varying temporal depth nets, depth-3 is the best performer, to make the clip prediction. For video predictions, we av-
but the gap is smaller. We also experiment with bigger spa- erage clip predictions of 10 clips which are randomly ex-
tial receptive field (e.g. 5 × 5) and/or full input resolution tracted from the video. It is worth noting some setting dif-
(240 × 320 frame inputs) and still observe similar behav- ferences between the comparing methods. DeepVideo and
ior. This suggests 3 × 3 × 3 is the best kernel choice for C3D use short clips while Convolution pooling [29] uses
3D ConvNets (according to our subset of experiments) and much longer clips. DeepVideo uses more crops: 4 crops per
3D ConvNets are consistently better than 2D ConvNets for clip and 80 crops per video compared with 1 and 10 used by
video classification. We also verify that 3D ConvNet con- C3D, respectively. The C3D network trained from scratch
sistently performs better than 2D ConvNet on a large-scale yields an accuracy of 84.4% and the one fine-tuned from
internal dataset, namely I380K. the I380K pre-trained model yields 85.5% at video top-
5 accuracy. Both C3D networks outperform DeepVideo’s
3.3. Spatiotemporal feature learning networks. C3D is still 5.6% below the method of [29].
Network architecture: Our findings in the previous sec- However, this method uses convolution pooling of deep
tion indicate that homogeneous setting with convolution image features on long clips of 120 frames, thus it is not
kernels of 3 × 3 × 3 is the best option for 3D ConvNets. directly comparable to C3D and DeepVideo which oper-
This finding is also consistent with a similar finding in 2D ate on much shorter clips. We note that the difference in
ConvNets [37]. With a large-scale dataset, one can train a top-1 accuracy for clips and videos of this method is small
3D ConvNet with 3×3×3 kernel as deep as possible subject (1.6%) as it already uses 120-frame clips as inputs. In prac-
to the machine memory limit and computation affordability. tice, convolution pooling or more sophisticated aggregation
With current GPU memory, we design our 3D ConvNet to schemes [29] can be applied on top of C3D features to im-
have 8 convolution layers, 5 pooling layers, followed by two prove video hit performance.
fully connected layers, and a softmax output layer. The net- C3D video descriptor: After training, C3D can be used
work architecture is presented in figure 3. For simplicity, as a feature extractor for other video analysis tasks. To
 Conv1a Conv2a Conv3a Conv3b Conv4a Conv4b Conv5a Conv5b fc6 fc7

softmax
Pool1

Pool2

Pool3

Pool4

Pool5
64 128 256 256 512 512 512 512 4096 4096

Figure 3. C3D architecture. C3D net has 8 convolution, 5 max-pooling, and 2 fully connected layers, followed by a softmax output layer.
All 3D convolution kernels are 3 × 3 × 3 with stride 1 in both spatial and temporal dimensions. Number of filters are denoted in each box.
The 3D pooling layers are denoted from pool1 to pool5. All pooling kernels are 2 × 2 × 2, except for pool1 is 1 × 2 × 2. Each fully
connected layer has 4096 output units.

Method Number of Nets Clip hit@1 Video hit@1 Video hit@5

DeepVideo’s Single-Frame + Multires [18] 3 nets 42.4 60.0 78.5
DeepVideo’s Slow Fusion [18] 1 net 41.9 60.9 80.2
Convolution pooling on 120-frame clips [29] 3 net 70.8* 72.4 90.8
C3D (trained from scratch) 1 net 44.9 60.0 84.4
C3D (fine-tuned from I380K pre-trained model) 1 net 46.1 61.1 85.2
Table 2. Sports-1M classification result. C3D outperforms [18] by 5% on top-5 video-level accuracy. (*)We note that the method of [29]
uses long clips, thus its clip-level accuracy is not directly comparable to that of C3D and DeepVideo.

extract C3D feature, a video is split into 16 frame long tiple nets setting, we concatenate the L2-normalized C3D
clips with a 8-frame overlap between two consecutive clips. descriptors of these nets.
These clips are passed to the C3D network to extract fc6 Baselines: We compare C3D feature with a few base-
activations. These clip fc6 activations are averaged to lines: the current best hand-crafted features, namely im-
form a 4096-dim video descriptor which is then followed proved dense trajectories (iDT) [44] and the popular-used
by an L2-normalization. We refer to this representation as deep image features, namely Imagenet [16], using Caffe’s
C3D video descriptor/feature in all experiments, unless we Imagenet pre-train model. For iDT, we use the bag-of-word
clearly specify the difference. representation with a codebook size of 5000 for each feature
What does C3D learn? We use the deconvolution channel of iDT which are trajectories, HOG, HOF, MBHx,
method explained in [46] to understand what C3D is learn- and MBHy. We normalize histogram of each channel sepa-
ing internally. We observe that C3D starts by focusing on rately using L1-norm and concatenate these normalized his-
appearance in the first few frames and tracks the salient mo- tograms to form a 25K feature vector for a video. For Im-
tion in the subsequent frames. Figure 4 visualizes deconvo- agenet baseline, similar to C3D, we extract Imagenet fc6
lution of two C3D conv5b feature maps with highest acti- feature for each frame, average these frame features to make
vations projected back to the image space. In the first exam- video descriptor. A multi-class linear SVM is also used for
ple, the feature focuses on the whole person and then tracks these two baselines for a fair comparison.
the motion of the pole vault performance over the rest of the
Results: Table 3 presents action recognition accuracy
frames. Similarly in the second example it first focuses on
of C3D compared with the two baselines and current best
the eyes and then tracks the motion happening around the
methods. The upper part shows results of the two base-
eyes while applying the makeup. Thus C3D differs from
lines. The middle part presents methods that use only RGB
standard 2D ConvNets in that it selectively attends to both
frames as inputs. And the lower part reports all current best
motion and appearance. We provide more visualizations in
methods using all possible feature combinations (e.g. opti-
the supplementary material to give a better insight about the
cal flows, iDT).
learned feature.
C3D fine-tuned net performs best among three C3D nets
4. Action recognition described previously. The performance gap between these
three nets, however, is small (1%). From now on, we refer
Dataset: We evaluate C3D features on UCF101 to the fine-tuned net as C3D, unless otherwise stated. C3D
dataset [38]. The dataset consists of 13, 320 videos of 101 using one net which has only 4, 096 dimensions obtains an
human action categories. We use the three split setting pro- accuracy of 82.3%. C3D with 3 nets boosts the accuracy
vided with this dataset. to 85.2% with the dimension is increased to 12, 288. C3D
Classification model: We extract C3D features and in- when combined with iDT further improves the accuracy to
put them to a multi-class linear SVM for training models. 90.4%, while when it is combined with Imagenet, we ob-
We experiment with C3D descriptor using 3 different nets: serve only 0.6% improvement. This indicates C3D can well
C3D trained on I380K, C3D trained on Sports-1M, and C3D capture both appearance and motion information, thus there
trained on I380K and fine-tuned on Sports-1M. In the mul- is no benefit to combining with Imagenet which is an ap-
Figure 4. Visualization of C3D model, using the method from [46]. Interestingly, C3D captures appearance for the first few frames but
thereafter only attends to salient motion. Best viewed on a color screen.

Method Accuracy (%) 80

Imagenet + linear SVM 68.8 75

iDT w/ BoW + linear SVM 76.2 70

Deep networks [18] 65.4 65

60
Spatial stream network [36] 72.6

Accuracy
55
LRCN [6] 71.1
50
LSTM composite model [39] 75.8 45
C3D (1 net) + linear SVM 82.3 40
Imagenet
C3D (3 nets) + linear SVM 85.2 35 iDT
C3D
iDT w/ Fisher vector [31] 87.9 30
0 50 100 150 200 250 300 350 400 450 500

Temporal stream network [36] 83.7 Number of dimensions

Two-stream networks [36] 88.0 Figure 5. C3D compared with Imagenet and iDT in low dimen-
LRCN [6] 82.9 sions. C3D, Imagenet, and iDT accuracy on UCF101 using PCA
LSTM composite model [39] 84.3 dimensionality reduction and a linear SVM. C3D outperforms Im-
Conv. pooling on long clips [29] 88.2 agenet and iDT by 10-20% in low dimensions.
LSTM on long clips [29] 88.6
Multi-skip feature stacking [25] 89.1
C3D (3 nets) + iDT + linear SVM 90.4
Table 3. Action recognition results on UCF101. C3D compared
with baselines and current state-of-the-art methods. Top: sim-
ple features with linear SVM; Middle: methods taking only RGB
frames as inputs; Bottom: methods using multiple feature combi-
nations.

pearance based deep feature. On the other hand, it is bene- Imagenet C3D
ficial to combine C3D with iDT as they are highly comple- Figure 6. Feature embedding. Feature embedding visualizations
mentary to each other. In fact, iDT are hand-crafted features of Imagenet and C3D on UCF101 dataset using t-SNE [43]. C3D
based on optical flow tracking and histograms of low-level features are semantically separable compared to Imagenet suggest-
gradients while C3D captures high level abstract/semantic ing that it is a better feature for videos. Each clip is visualized as a
information. point and clips belonging to the same action have the same color.
Best viewed in color.
C3D with 3 nets achieves 85.2% which is 9% and 16.4%
better than the iDT and Imagenet baselines, respectively. methods in term of network architecture and basic opera-
On the only RGB input setting, compared with CNN-based tions. In addition, C3D is trained on Sports-1M and used as
approaches, Our C3D outperforms deep networks [18] and is without any finetuning. Compared with Recurrent Neural
spatial stream network in [36] by 19.8% and 12.6%, respec- Networks (RNN) based methods, C3D outperforms Long-
tively. Both deep networks [18] and spatial stream network term Recurrent Convolutional Networks (LRCN) [6] and
in [36] use AlexNet architecture. While in [18], the net is LSTM composite model [39] by 14.1% and 9.4%, respec-
fine-tuned from their model pre-trained on Sports-1M, spa- tively. C3D with only RGB input still outperforms these
tial stream network in [36] is fine-tuned from Imagenet pre- two RNN-based methods when they used both optical flows
trained model. Our C3D is different from these CNN-base and RGB as well as the temporal stream network in [36].
However, C3D needs to be combined with iDT to outper- 1

form two-stream networks [36], the other iDT-based meth- 0.9

ods [31, 25], and the method that focuses on long-term mod- 0.8

true positive rate

eling [29]. Apart from the promising numbers, C3D also 0.7

has the advantage of simplicity compared to the other meth- 0.6

ods. 0.5
C3D is compact: In order to evaluate the compactness 0.4 C3D
of C3D features we use PCA to project the features into Human Performance
0.3 STIP [21]
OSSML [22]
lower dimensions and report the classification accuracy of MIP [20]
0.2
MIP+STIP+MBH [11]
the projected features on UCF101 [38] using a linear SVM. iDT+FV [45]
0.1 Imagenet
We apply the same process with iDT [44] as well as Ima- Random Chance
0
genet features [7] and compare the results in Figure 5. At 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

the extreme setting with only 10 dimensions, C3D accuracy false positive rate
is 52.8% which is more than 20% better than the accuracy Figure 7. Action similarity labeling result. ROC curve of C3D
of Imagenet and iDT which are about 32%. At 50 and 100 evaluated on ASLAN. C3D achieves 86.5% on AUC and outper-
dim, C3D obtains an accuracy of 72.6% and 75.6% which forms current state-of-the-art by 11.1%.
are about 10-12% better than Imagenet and iDT. Finally,
with 500 dimensions, C3D is able to achieve 79.4% accu- Method Features Model Acc. AUC
racy which is 6% better than iDT and 11% better than Im- [21] STIP linear 60.9 65.3
agenet. This indicates that our features are both compact [22] STIP metric 64.3 69.1
and discriminative. This is very helpful for large-scale re- [20] MIP metric 65.5 71.9
trieval applications where low storage cost and fast retrieval [11] MIP+STIP+MBH metric 66.1 73.2
are crucial. [45] iDT+FV metric 68.7 75.4
We qualitatively evaluate our learned C3D features to Baseline Imagenet linear 67.5 73.8
verify if it is a good generic feature for video by visual- Ours C3D linear 78.3 86.5
izing the learned feature embedding on another dataset. We Table 4. Action similarity labeling result on ASLAN. C3D sig-
randomly select 100K clips from UCF101, then extract fc6 nificantly outperforms state-of-the-art method [45] by 9.6% in ac-
features for those clips using for features from Imagenet and curacy and by 11.1% in area under ROC curve.
C3D. These features are then projected to 2-dimensional
distances provided in [21]. With 4 types of features, we ob-
space using t-SNE [43]. Figure 6 visualizes the feature
tain 48-dimensional (12 × 4 = 48) feature vector for each
embedding of the features from Imagenet and our C3D on
video pair. As these 48 distances are not comparable to each
UCF101. It is worth noting that we did not do any fine-
other, we normalize them independently such that each di-
tuning as we wanted to verify if the features show good
mension has zero mean and unit variance. Finally, a linear
generalization capability across datasets. We quantitatively
SVM is trained to classify video pairs into same or differ-
observe that C3D is better than Imagenet.
ent on these 48-dim feature vectors. Beside comparing with
current methods, we also compare C3D with a strong base-
5. Action Similarity Labeling line using deep image-based features. The baseline has the
Dataset: The ASLAN dataset consists of 3, 631 videos same setting as our C3D and we replace C3D features with
from 432 action classes. The task is to predict if a given Imagenet features.
pair of videos belong to the same or different action. We Results: We report the result of C3D and compare with
use the prescribed 10-fold cross validation with the splits state-of-the-art methods in table 4. While most current
provided with the dataset. This problem is different from methods use multiple hand-crafted features, strong encod-
action recognition, as the task focuses on predicting action ing methods (VLAD, Fisher Vector), and complex learning
similarity not the actual action label. The task is quite chal- models, our method uses a simple averaging of C3D fea-
lenging because the test set contains videos of “never-seen- tures over the video and a linear SVM. C3D significantly
before” actions. outperforms state-of-the-art method [45] by 9.6% on accu-
Features: We split videos into 16-frame clips with an racy and 11.1% on area under ROC curve (AUC). Imagenet
overlap of 8 frames. We extract C3D features: prob, fc7, baseline performs reasonably well which is just 1.2% below
fc6, pool5 for each clip. The features for videos are com- state-of-the-art method [45], but 10.8% worse than C3D due
puted by averaging the clip features separately for each type to lack of motion modeling. Figure 7 plots the ROC curves
of feature, followed by an L2 normalization. of C3D compared with current methods and human perfor-
Classification model: We follow the same setup used mance. C3D has clearly made a significant improvement
in [21]. Given a pair of videos, we compute the 12 different which is a halfway from current state-of-the-art method to
 Dataset [4] [41] [8] [9] Imagenet C3D Method iDT Brox’s Brox’s C3D
Maryland 43.1 74.6 67.7 77.7 87.7 87.7 Usage CPU CPU GPU GPU
YUPENN 80.7 85.0 86.0 96.2 96.7 98.1 RT (hours) 202.2 2513.9 607.8 2.2
Table 5. Scene recognition accuracy. C3D using a simple linear FPS 3.5 0.3 1.2 313.9
SVM outperforms current methods on Maryland and YUPENN. x Slower 91.4 1135.9 274.6 1
Table 6. Runtime analysis on UCF101. C3D is 91x faster than
human performance (98.9%). improved dense trajectories [44] and 274x faster than Brox’s GPU
implementation in OpenCV.
6. Scene and Object Recognition
to work that well on this task. The result is very surprising
Datasets: For dynamic scene recognition, we evaluate and shows how generic C3D is on capturing appearance and
C3D on two benchmarks: YUPENN [4] and Maryland [35]. motion information in videos.
YUPENN consists of 420 videos of 14 scene categories and
Maryland has 130 videos of 13 scene categories. For object
recognition, we test C3D on egocentric dataset [32] which
7. Runtime Analysis
consists 42 types of everyday objects. A point to note, this We compare the runtime of C3D and with iDT [44] and
dataset is egocentric and all videos are recorded in a first the Temporal stream network [36]. For iDT, we use the code
person view which have quite different appearance and mo- kindly provided by the authors [44]. For [36], there is no
tion characteristics than any of the videos we have in the public model available to evaluate. However, this method
training dataset. uses Brox’s optical flows [3] as inputs. We manage to eval-
Classification model: For both datasets, we use the uate runtime of Brox’s method using two different versions:
same setup of feature extraction and linear SVM for classifi- CPU implementation provided by the authors [3] and the
cation and follow the same leave-one-out evaluation proto- GPU implementation provided in OpenCV.
col as described by the authors of these datasets. For object We report runtime of the three above-mentioned methods
dataset, the standard evaluation is based on frames. How- to extract features (including I/O) for the whole UCF101
ever, C3D takes a video clip of length 16 frames to extract dataset in table 6 using using a single CPU or a single K40
the feature. We slide a window of 16 frames over all videos Tesla GPU. [36] reported a computation time (without I/O)
to extract C3D features. We choose the ground truth label of 0.06s for a pair of images. In our experiment, Brox’s
for each clip to be the most frequently occurring label of the GPU implementation takes 0.85-0.9s per image pair includ-
clip. If the most frequent label in a clip occurs fewer than ing I/O. Note that this is not a fair comparison for iDT as it
8 frames, we consider it as negative clip with no object and uses only CPU. We cannot find any GPU implementation
discard it in both training and testing. We train and test C3D of this method and it is not trivial to implement a parallel
features using linear SVM and report the object recognition version of this algorithm on GPU. Note that C3D is much
accuracy. We follow the same split provided in [32]. We faster than real-time, processing at 313 fps while the other
also compare C3D with a baseline using Imagenet feature two methods have a processing speed of less than 4 fps.
on these 3 benchmarks.
Results: Table 5 reports our C3D results and compares it 8. Conclusions
with the current best methods. On scene classification, C3D
outperforms state-of-the-art method [9] by 10% and 1.9% In this work we try to address the problem of learn-
on Maryland and YUPENN respectively. It is worth noth- ing spatiotemporal features for videos using 3D ConvNets
ing that C3D uses only a linear SVM with simple averaging trained on large-scale video datasets. We conducted a sys-
of clip features while the second best method [9] uses dif- tematic study to find the best temporal kernel length for
ferent complex feature encodings (FV, LLC, and dynamic 3D ConvNets. We showed that C3D can model appear-
pooling). The Imagenet baseline achieves similar perfor- ance and motion information simultaneously and outper-
mance with C3D on Maryland and 1.4% lower than C3D forms the 2D ConvNet features on various video analysis
on YUPENN. On object recognition, C3D obtains 22.3% tasks. We demonstrated that C3D features with a linear
accuracy and outperforms [32] by 10.3% with only linear classifier can outperform or approach current best methods
SVM where the comparing method used RBF-kernel on on different video analysis benchmarks. Last but not least,
strong SIFT-RANSAC feature matching. Compared with the proposed C3D features are efficient, compact, and ex-
Imagenet baseline, C3D is still 3.4% worse. This can be tremely simple to use.
explained by the fact that C3D uses smaller input resolution C3D source code and pre-trained model are available
(128 × 128) compared to full-size resolution (256 × 256) at http://vlg.cs.dartmouth.edu/c3d.
using by Imagenet. Since C3D is trained only on Sports- Acknowledgment: we would like to thank Yann Lecun
1M videos without any fine-tuning while Imagenet is fully for his valuable feedback, Nikhil Johri and Engineering at
trained on 1000 object categories, we did not expect C3D Facebook AI Research for data and infrastructure support.
 0.44
lution.
Decovolutions of C3D: We randomly select 20K clips
0.42
from UCF101. We group clips that fire strongly for the
0.4
same feature map at a pre-selected convolution layer. We
clip accuracy

0.38 use deconvolution [46] to project the top activations of

0.36 these clips back into image space. We visualize the gra-
0.34
dients causing the activiation together with the correspond-
ing cropped image sequences. Note that we did not do any
0.32
fine-tuning of C3D model on UCF101.
0.3
net−128 Figure 9 and 10 visualize deconvolutions of C3D learned
0.28 net−64 feature maps at the layers conv2a and conv3b. Visual-
net−256 izations of the same feature map are grouped together. For
0.26
0 2 4 6 8 10 12 14 16
# epoch figures 11, 12, 13, and 14, each figure presents the deconvo-
lutions of one learned feature map of the conv5b layer. Fi-
Figure 8. 3D ConvNets with different input resolutions. Action nally, figure 15 compares the deconvolutions of several C3D
recognition clip accuracy on UCF101 test split-1 of 3D ConvNets conv5b feature maps with optical flows. As showed in
with different input resolutions. the visualizations, at early convolution layer conv2a, C3D
learns low-level motion patterns such as moving edges,
blobs, short changes, edge orientation changes, or color
Appendix A: Effects of Input Resolution changes. At a higher layer of conv3b, C3D learns bigger
As part of the architecture study, we examine the ef- moving patterns of corners, textures, body parts, and trajec-
fects of input resolution on 3D ConvNets. We use the same tories. Finally, at the deepest convolution layer, conv5b,
common network setting described in section 3. We fix all C3D learns more complicated motion patterns such as mov-
convolution kernels to 3 × 3 × 3 and vary the input reso- ing circular objects, biking-like motions.
lutions to study the effects. We experiment with 3 differ-
ent nets with input resolutions of 64 × 64, 128 × 128, and References
256 × 256, namely net-64, net-128, and net-256, respec- [1] M. Bendersky, L. Garcia-Pueyo, J. Harmsen, V. Josifovski, and
tively. Note that net-128 is equivalent to the depth-3 net in D. Lepikhin. Up next: retrieval methods for large scale related video
section 3.2. Because of the difference in input resolutions, suggestion. In ACM SIGKDD, pages 1769–1778, 2014. 2
[2] O. Boiman and M. Irani. Detecting irregularities in images and in
these nets have different output size at the last pooling layer, video. IJCV, 2007. 1, 2
thus leading to a significant difference in terms of number [3] T. Brox and J. Malik. Large displacement optical flow: Descriptor
of parameters. Table 7 reports the numbers of parameters matching in variational motion estimation. IEEE TPAMI, 33(3):500–
and the training time of these nets. Figure 8 presents the 513, 2011. 8
[4] K. Derpanis, M. Lecce, K. Daniilidis, and R. Wildes. Dynamic scene
clip accuracy of these nets on UCF101 test split-1. Net-128 understanding: The role of orientation features in space and time in
outperforms net-64 by 3.1% and attains a comparable ac- scene classification. In CVPR, 2012. 8
curacy with net-256. This indicates that net-128 provides a [5] P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie. Behavior recog-
good trade-off between training time, accuracy, and mem- nition via sparse spatio-temporal features. In Proc. ICCV VS-PETS,
2005. 2
ory consumption. We note that with the current GPU mem- [6] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venu-
ory limit, one has to use model parallelism to train C3D gopalan, K. Saenko, and T. Darrell. Long-term recurrent convo-
with 256 × 256 input resolution. lutional networks for visual recognition and description. CoRR,
abs/1411.4389, 2014. 6
[7] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and
Net net-64 net-128 net-256
T. Darrell. Decaf: A deep convolutional activation feature for generic
# of params (M) 11.1 17.5 34.8 visual recognition. In ICML, 2013. 2, 7
Train time (mins/epoch) 92 270 1186 [8] C. Feichtenhofer, A. Pinz, and R. P. Wildes. Spacetime forests with
complementary features for dynamic scene recognition. In BMVC,
Table 7. Number of parameters and training time comparison of 2013. 8
3D ConvNets with different input resolutions. Note that net-128 is [9] C. Feichtenhofer, A. Pinz, and R. P. Wildes. Bags of spacetime ener-
gies for dynamic scene recognition. In CVPR, 2014. 2, 8
equivalent to the depth-3 net in the paper.
[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierar-
chies for accurate object detection and semantic segmentation. arXiv
preprint arXiv:1311.2524, 2013. 2
Appendix B: Visualization of C3D Learned Fea- [11] Y. Hanani, N. Levy, and L. Wolf. Evaluating new variants of motion
tures interchange patterns. In CVPR workshop, 2013. 7
[12] A. Jain, J. Tompson, M. Andriluka, G. W. Taylor, and C. Bregler.
For a better understanding of what C3D learned inter- Learning human pose estimation features with convolutional net-
nally, we provide additional visualizations using deconvo- works. In ICLR, 2014. 2
Figure 9. Deconvolutions of C3D conv2a feature maps. Each group is a C3D conv2a learned feature map. First two rows: the learned
filters detect moving edges and blobs. The last row: the learned filters detect shot changes, edge orientation changes, and color changes.
Best viewed in a color screen.

[13] A. Jain, J. Tompson, Y. LeCun, and C. Bregler. Modeep: A deep networks. In CVPR, 2014. 1, 2, 3, 4, 5, 6
learning framework using motion features for human pose estima- [19] A. Kläser, M. Marszałek, and C. Schmid. A spatio-temporal descrip-
tion. In ACCV, 2014. 2 tor based on 3d-gradients. In BMVC, 2008. 2
[14] V. Jain, B. Bollmann, M. Richardson, D. Berger, M. Helmstaedter, [20] O. Kliper-Gross, Y. Gurovich, T. Hassner, and L. Wolf. Motion in-
K. Briggman, W. Denk, J. Bowden, J. Mendenhall, W. Abraham, terchange patterns for action recognition in unconstrained videos. In
K. Harris, N. Kasthuri, K. Hayworth, R. Schalek, J. Tapia, J. Licht- ECCV, 2012. 7
man, and H. Seung. Boundary learning by optimization with topo-
[21] O. Kliper-Gross, T. Hassner, and L. Wolf. The action similarity la-
logical constraints. In CVPR, 2010. 2
beling challenge. TPAMI, 2012. 7
[15] S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neural networks
for human action recognition. IEEE TPAMI, 35(1):221–231, 2013. [22] O. Kliper-Grossa, T. Hassner, and L. Wolf. The one shot similarity
1, 2 metric learning for action recognition. In Workshop on SIMBAD,
2011. 7
[16] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for [23] D. B. Kris M. Kitani, Brian D. Ziebart and M. Hebert. Activity fore-
fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 1, 5 casting. In ECCV, 2012. 1
[17] Y. Jiang, J. Liu, A. Zamir, G. Toderici, I. Laptev, M. Shah, and [24] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification
R. Sukthankar. THUMOS challenge: Action recognition with a large with deep convolutional neural networks. In NIPS, 2012. 1, 2
number of classes, 2014. 2 [25] Z. Lan, M. Lin, X. Li, A. G. Hauptmann, and B. Raj. Beyond
[18] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and gaussian pyramid: Multi-skip feature stacking for action recognition.
L. Fei-Fei. Large-scale video classification with convolutional neural CoRR, abs/1411.6660, 2014. 2, 6, 7
Figure 10. Deconvolutions of C3D conv3b feature maps. Each group is a C3D conv3b learned feature map. Upper: feature maps detect
moving corners and moving textures. Middle: feature maps detect moving body parts. Lower: feature maps detect object trajectories and
circular objects. Best viewed in a color screen.

[26] I. Laptev and T. Lindeberg. Space-time interest points. In ICCV, [28] Y. LeCun and Y. Bengio. Convolutional networks for images, speech,
2003. 1, 2 and time-series. Brain Theory and Neural Networks, 1995. 2
[27] Q. V. Le, W. Y. Zou, S. Y. Yeung, and A. Y. Ng. Learning hierar- [29] J. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga,
chical invariant spatio-temporal features for action recognition with and G. Toderici. Beyond short snippets: Deep networks for video
independent subspace analysis. In CVPR, 2011. 2 classification. In CVPR, 2015. 2, 4, 5, 6, 7
Figure 11. Deconvolutions of a C3D conv5b learned feature map which detects moving motions of circular objects. In the second last
clip, it detects a moving head while in the last clip, it detects the moving hair-curler. Best viewed in a color screen.
Figure 12. Deconvolutions of a C3D conv5b learned feature map which detects biking-like motions. Note that the last two clips have no
biking but their motion patterns are similar to biking motions. Best viewed in a color screen.
Figure 13. Deconvolutions of a C3D conv5b learned feature map which detects face-related motions: applying eye-makeup, applying
lipstick, and brushing tooth. Best viewed in a color screen.
Figure 14. Deconvolutions of a C3D conv5b learned feature map which detects balance-beam-like motions. In the last clip, it detects
hammering which shares similar motion patterns with balance beam. Best viewed in a color screen.
Figure 15. Deconvlotuions of C3D conv5b learned feature maps compared with optical flows. Optical flows fire at all of moving pixels
while C3D just pays attention to only salient motions. Best viewed in a color screen.

[30] P. Over, G. Awad, M. Michel, J. Fiscus, G. Sanders, W. Kraaij, [41] C. Theriault, N. Thome, and M. Cord. Dynamic scene classification:
A. Smeaton, and G. Quenot. Trecvid’14–an overview of the goals, Learning motion descriptors with slow features analysis. In CVPR,
tasks, data, evaluation and metrics. In TRECVID, 2014. 2 2013. 8
[31] X. Peng, L. Wang, X. Wang, and Y. Qiao. Bag of visual words and fu- [42] S. Turaga, J. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman,
sion methods for action recognition: Comprehensive study and good W. Denk, and S. Seung. Convolutional networks can learn to generate
practice. CoRR, abs/1405.4506, 2014. 2, 6, 7 affinity graphs for image segmentation. Neural Comp., 2010. 2
[32] X. Ren and M. Philipose. Egocentric recognition of handled objects: [43] L. van der Maaten and G. Hinton. Visualizing data using t-sne.
Benchmark and analysis. In Egocentric Vision workshop, 2009. 2, 8 JMLR, 9(2579-2605):85, 2008. 6, 7
[44] H. Wang and C. Schmid. Action recognition with improved trajecto-
[33] S. Sadanand and J. Corso. Action bank: A high-level representation
ries. In ICCV, 2013. 2, 5, 7, 8
of activity in video. In CVPR, 2012. 2
[45] Q. P. X. Peng, Y. Qiao and Q. Wang. Large margin dimensional-
[34] P. Scovanner, S. Ali, and M. Shah. A 3-dimensional sift descriptor ity reduction for action similarity labeling. IEEE Signal Processing
and its application to action recognition. In ACM MM, 2007. 2 Letter, 2014. 7
[35] N. Shroff, P. K. Turaga, and R. Chellappa. Moving vistas: Exploiting [46] M. Zeiler and R. Fergus. Visualizing and understanding convolu-
motion for describing scenes. In CVPR, 2010. 8 tional networks. In ECCV, 2014. 5, 6, 9
[36] K. Simonyan and A. Zisserman. Two-stream convolutional networks [47] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev. Panda:
for action recognition in videos. In NIPS, 2014. 2, 6, 8 Pose aligned networks for deep attribute modeling. In CVPR, 2014.
[37] K. Simonyan and A. Zisserman. Very deep convolutional networks 1
for large-scale image recognition. In ICLR, 2015. 3, 4 [48] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning
deep features for scene recognition using places database. In NIPS,
[38] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 2014. 1
human action classes from videos in the wild. In CRCV-TR-12-01,
2012. 5, 7
[39] N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised
learning of video representations using LSTMs. In ICML, 2015. 2, 6
[40] G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. Convolutional
learning of spatio-temporal features. In ECCV, pages 140–153.
Springer, 2010. 2