CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly

Congested Scenes
Yuhong Li1,2 , Xiaofan Zhang1 , Deming Chen1
1
University of Illinois at Urbana-Champaign
2
Beijing University of Posts and Telecommunications
{leeyh,xiaofan3,dchen}@illinois.edu
arXiv:1802.10062v4 [cs.CV] 11 Apr 2018

Abstract
We propose a network for Congested Scene Recognition
called CSRNet to provide a data-driven and deep learning
method that can understand highly congested scenes and
perform accurate count estimation as well as present high-
quality density maps. The proposed CSRNet is composed
of two major components: a convolutional neural network
(CNN) as the front-end for 2D feature extraction and a di-
lated CNN for the back-end, which uses dilated kernels to
deliver larger reception fields and to replace pooling opera-
tions. CSRNet is an easy-trained model because of its pure Figure 1. Pictures in first row show three images all containing 95
convolutional structure. We demonstrate CSRNet on four people in ShanghaiTech Part B dataset [18], while having totally
datasets (ShanghaiTech dataset, the UCF CC 50 dataset, different spatial distributions. Pictures in second row show their
the WorldEXPO’10 dataset, and the UCSD dataset) and density maps.
we deliver the state-of-the-art performance. In the Shang-
haiTech Part B dataset, CSRNet achieves 47.3% lower comes from the prediction manner: since the generated den-
Mean Absolute Error (MAE) than the previous state-of-the- sity values follow the pixel-by-pixel prediction, output den-
art method. We extend the targeted applications for count- sity maps must include spatial coherence so that they can
ing other objects, such as the vehicle in TRANCOS dataset. present the smooth transition between nearest pixels. Also,
Results show that CSRNet significantly improves the output the diversified scenes, e.g., irregular crowd clusters and dif-
quality with 15.4% lower MAE than the previous state-of- ferent camera perspectives, would make the task difficult,
the-art approach. especially for using traditional methods without deep neu-
ral networks (DNNs). The recent development of congested
scene analysis relays on DNN-based methods because of
1. Introduction
the high accuracy they have achieved in semantic segmen-
Growing number of network models have been devel- tation tasks [7, 8, 9, 10, 11] and the significant progress they
oped [1, 2, 3, 4, 5] to deliver promising solutions for crowd have made in visual saliency [12]. The additional bonus of
flows monitoring, assembly controlling, and other security using DNNs comes from the enthusiastic hardware com-
services. Current methods for congested scenes analysis are munity where DNNs are rapidly investigated and imple-
developed from simple crowd counting (which outputs the mented on GPUs [13], FPGAs [14, 15, 16], and ASICs [17].
number of people in the targeted image) to density map pre- Among them, the low-power, small-size schemes are es-
senting (which displays characteristics of crowd distribu- pecially suitable for deploying congested scene analysis in
tion) [6]. This development follows the demand of real- surveillance devices.
life applications since the same number of people could Previous works for congested scene analysis are mostly
have completely different crowd distributions (as shown in based on multi-scale architectures [4, 5, 18, 19, 20]. They
Fig. 1), so that just counting the number of crowds is not have achieved high performance in this field but the de-
enough. The distribution map helps us for getting more ac- signs they used also introduce two significant disadvantages
curate and comprehensive information, which could be crit- when networks go deeper: large amount of training time
ical for making correct decisions in high-risk environments, and non-effective branch structure (e.g., multi-column CNN
such as stampede and riot. However, it is challenging to (MCNN) in [18]). We design an experiment to demon-
generate accurate distribution patterns. One major difficulty strate that the MCNN does not perform better compared to a
deeper, regular network in Table 1. The main reason of us-
ing MCNN in [18] is the flexible receptive fields provided
by convolutional filters with different sizes across the col-
umn. Intuitively, each column of MCNN is dedicated to a
certain level of congested scene. However, the effectiveness
of using MCNN may not be prominent. We present Fig. 2
to illustrate the features learned by three separated columns
(representing large, medium, and small receptive fields) in
MCNN and evaluate them with ShanghaiTech Part A [18]
dataset. The three curves in this figure share very similar
patterns (estimated error rate) for 50 test cases with different
congest densities meaning that each column in such branch
structure learn nearly identical features. It performs against
the original intention of the MCNN design for learning dif- Figure 2. The estimated error of 50 samples from the testing set
ferent features for each column. in ShanghaiTech Part A [18] generated by the three pre-trained
In this paper, we design a deeper network called CSR- columns of MCNN. Small, Medium, Large respectively stand for
Net for counting crowd and generating high-quality den- the columns with small, medium or large kernels in the MCNN.
sity maps. Unlike the latest works such as [4, 5] which
use the deep CNN for ancillary, we focus on designing a Method Parameters MAE MSE
CNN-based density map generator. Our model uses pure Col. 1 of MCNN 57.75k 141.2 206.8
convolutional layers as the backbone to support input im- Col. 2 of MCNN 45.99k 160.5 239.0
ages with flexible resolutions. To limit the network com- Col. 3 of MCNN 25.14k 153.7 230.2
plexity, we use the small size of convolution filters (like MCNN Total 127.68k 110.2 185.9
3 × 3) in all layers. We deploy the first 10 layers from A deeper CNN 83.84k 93.0 142.2
VGG-16 [21] as the front-end and dilated convolution lay-
ers as the back-end to enlarge receptive fields and extract Table 1. To demonstrate that MCNN [18] may not be the best
deeper features without losing resolutions (since pooling choice, we design a deeper, single-column network with fewer
layers are not used). By taking advantage of such innovative parameters compared to MCNN. The architecture of the pro-
structure, we outperform the state-of-the-art crowd count- posed small network is: CR(32, 3) − M − CR(64, 3) − M −
ing solutions (a MCNN based solution called CP-CNN [5]) CR(64, 3)−M −CR(32, 3)−CR(32, 3)−CR(1, 1). CR(m, n)
represents the convolutional layer with m filters whose size is n×n
with 7%, 47.3%, 10.0%, and 2.9% lower Mean Abso-
followed by the ReLu layer. M is the max-pooling layer. Results
lute Error (MAE) in ShanghaiTech [18] Part A, Part B,
show that the single-column version achieves higher performance
UCF CC 50 [22], and WorldExpo’10 [3] datasets respec- on ShanghaiTech Part A dataset [18] with the lowest MAE and
tively. Also, we achieve high performance on the UCSD Mean Squared Error (MSE)
dataset [23] with 1.16 MAE. After extending this work to
vehicle counting on TRANCOS dataset [20], we achieve 2.1. Detection-based approaches
15.4% lower MAE than the current best approach, called Most of the early researches focus on detection-based
FCN-HA [24]. approaches using a moving-window-like detector to detect
The rest of the paper is structured as follows. Sec. 2 people and count their number [26]. These methods re-
presents the previous works for crowd counting and den- quire well-trained classifiers to extract low-level features
sity map generation. Sec. 3 introduces the architecture and from the whole human body (like Haar wavelets [27] and
configuration of our model while Sec. 4 presents the exper- HOG (histogram oriented gradients) [28]). However, they
imental results on several datasets. In Sec. 5, we conclude perform poorly on highly congested scenes since most of
the paper. the targeted objects are obscured. To tackle this problem,
researchers detect particular body parts instead of the whole
2. Related work body to complete crowd scenes analysis [29].
Following the idea proposed by Loy et al. [25], the po-
2.2. Regression-based approaches
tential solutions for crowd scenes analysis can be classified
into three categories: detection-based methods, regression- Since detection-based approaches can not be adapted
based methods, and density estimation-based methods. By to highly congested scenes, researchers try to deploy
combining the deep learning, the CNN-based solutions regression-based approaches to learn the relations among
show even stronger ability in this task and outperform the extracted features from cropped image patches, and then
traditional methods. calculate the number of particular objects. More features,
such as foreground and texture features, have been used for work [18]. Such bloated network structure requires more
generating low-level information [30]. Following similar time to train. (2) Multi-column CNNs introduce redundant
approaches, Idrees et al. [22] propose a model to extract structure as we mentioned in Sec. 1. Different columns
features by employing Fourier analysis and SIFT (Scale- seem to perform similarly without obvious differences. (3)
invariant feature transform) [31] interest-point based count- Both solutions require density level classifier before send-
ing. ing pictures in the MCNN. However, the granularity of den-
sity level is hard to define in real-time congested scene anal-
2.3. Density estimation-based approaches ysis since the number of objects keeps changing with a large
When executing the regression-based solution, one crit- range. Also, using a fine-grained classifier means more
ical feature, called saliency, is overlooked which causes in- columns need to be implemented which makes the design
accurate results in local regions. Lempitsky et al. [32] pro- more complicated and causes more redundancy. (4) These
pose a method to solve this problem by learning a linear works spend a large portion of parameters for density level
mapping between features in the local region and its object classification to label the input regions instead of allocating
density maps. It integrates the information of saliency dur- parameters to the final density map generation. Since the
ing the learning process. Since the ideal linear mapping is branch structure in MCNN is not efficient, the lack of pa-
hard to obtain, Pham et al. [33] use random forest regression rameters for generating density map lowers the final accu-
to learn a non-linear mapping instead of the linear one. racy. Taking all above disadvantages into consideration, we
propose a novel approach to concentrate on encoding the
2.4. CNN-based approaches deeper features in congested scenes and generating high-
Literature also focuses on the CNN-based approaches quality density map.
to predict the density map because of its success in clas-
sification and recognition [34, 21, 35]. In the work pre-
3. Proposed Solution
sented by Walach and Wolf [36], a method is demonstrated The fundamental idea of the proposed design is to de-
with selective sampling and layered boosting. Instead of ploy a deeper CNN for capturing high-level features with
using patch-based training, Shang et al. [37] try an end-to- larger receptive fields and generating high-quality density
end regression method using CNNs which takes the entire maps without brutally expanding network complexity. In
image as input and directly outputs the final crowd count. this section, we first introduce the architecture we proposed,
Boominathan et al. [19] present the first work purely us- and then we present the corresponding training methods.
ing convolutional networks and dual-column architecture
for generating density map. Marsden et al. [38] explore 3.1. CSRNet architecture
single-column fully convolutional networks while Sindagi Following the similar idea in [19, 4, 5], we choose VGG-
et al. [39] propose a CNN which uses the high-level prior in- 16 [21] as the front-end of CSRNet because of its strong
formation to boost the density prediction performance. An transfer learning ability and its flexible architecture for eas-
improved structure is proposed by Zhang et al. [18] who ily concatenating the back-end for density map generation.
introduce a multi-column based architecture (MCNN) for In CrowdNet [19], the authors directly carve the first 13 lay-
crowd counting. Similar idea is shown in Onoro and Sas- ers from VGG-16 and add a 1 × 1 convolutional layer as
tre [20] where a scale-aware, multi-column counting model output layer. The absence of modifications results in very
called Hydra CNN is presented for object density estima- weak performance. Other architectures, such as [4], uses
tion. It is clear that the CNN-based solutions outperform VGG-16 as the density level classifier for labeling input im-
the previous works mentioned in Sec. 2.1 to 2.3. ages before sending them to the most suitable column of
the MCNN, while the CP-CNN [5] incorporates the result
2.5. Limitations of the state-of-the-art approaches
of classification with the features from density map gen-
Most recently, Sam et al. [4] propose the Switch-CNN erator. In these cases, the VGG-16 performs as an ancil-
using a density level classifier to choose different regres- lary without significantly boosting the final accuracy. In
sors for particular input patches. Sindagi et al. [5] present this paper, we first remove the classification part of VGG-
a Contextual Pyramid CNN, which uses CNN networks to 16 (fully-connected layers) and build the proposed CSRNet
estimate context at various levels for achieving lower count with convolutional layers in VGG-16. The output size of
error and better quality density maps. These two solutions this front-end network is 1/8 of the original input size. If
achieve the state-of-the-art performance, and both of them we continue to stack more convolutional layers and pooling
used multi-column based architecture (MCNN) and den- layers (basic components in VGG-16), output size would
sity level classifier. However, we observe several disad- be further shrunken, and it is hard to generate high-quality
vantages in these approaches: (1) Multi-column CNNs are density maps. Inspired by the works [10, 11, 40], we try
hard to train according to the training method described in to deploy dilated convolutional layers as the back-end for
extracting deeper information of saliency as well as main-
taining the output resolution.

3.1.1 Dilated convolution

One of the critical components of our design is the di-
lated convolutional layer. A 2-D dilated convolution can be
defined as follow:
Figure 3. 3 × 3 convolution kernels with different dilation rate as
M X
X N 1, 2, and 3.
y(m, n) = x(m + r × i, n + r × j)w(i, j) (1)
i=1 j=1

y(m, n) is the output of dilated convolution from input

x(m, n) and a filter w(i, j) with the length and the width
of M and N respectively. The parameter r is the dilation
rate. If r = 1, a dilated convolution turns into a normal
convolution.
Dilated convolutional layers have been demonstrated in
segmentation tasks with significant improvement of accu-
racy [10, 11, 40] and it is a good alternative of pooling
layer. Although pooling layers (e.g., max and average pool-
ing) are widely used for maintaining invariance and con- Figure 4. Comparison between dilated convolution and max-
trolling overfitting, they also dramatically reduce the spatial pooling, convolution, upsampling. The 3 × 3 Sobel kernel is used
resolution meaning the spatial information of feature map in both operations while the dilation rate is 2.
is lost. Deconvolutional layers [41, 42] can alleviate the
loss of information, but the additional complexity and ex- factor = 2 stride. The output is shared the same dimension
ecution latency may not be suitable for all cases. Dilated as the input (meaning pooling and deconvolutional layers
convolution is a better choice, which uses sparse kernels (as are not required). Most importantly, the output from dilated
shown in Fig. 3) to alternate the pooling and convolutional convolution contains more detailed information (referring
layer. This character enlarges the receptive field without to the portions we zoom in).
increasing the number of parameters or the amount of com-
putation (e.g., adding more convolutional layers can make 3.1.2 Network Configuration
larger receptive fields but introduce more operations). In
dilated convolution, a small-size kernel with k × k filter is We propose four network configurations of CSRNet in
enlarged to k + (k − 1)(r − 1) with dilated stride r. Thus Table 3 which have the same front-end structure but differ-
it allows flexible aggregation of the multi-scale contextual ent dilation rate in the back-end. Regarding the front-end,
information while keeping the same resolution. Examples we adapt a VGG-16 network [21] (except fully-connected
can be found in Fig. 3 where normal convolution gets 3 × 3 layers) and only use 3 × 3 kernels. According to [21], us-
receptive field and two dilated convolutions deliver 5 × 5 ing more convolutional layers with small kernels is more
and 7 × 7 receptive fields respectively. efficient than using fewer layers with larger kernels when
For maintaining the resolution of feature map, the di- targeting the same size of receptive field .
lated convolution shows distinct advantages compared to By removing the fully-connected layers, we try to deter-
the scheme of using convolution + pooling + deconvolu- mine the number of layers we need to use from VGG-16.
tion. We pick one example for illustration in Fig. 4. The The most critical part relays on the tradeoff between ac-
input is an image of crowds, and it is processed by two curacy and the resource overhead (including training time,
approaches separately for generating output with the same memory consumption, and the number of parameters). Ex-
size. In the first approach, input is downsampled by a max periment shows a best tradeoff can be achieved when keep-
pooling layer with factor 2, and then it is passed to a convo- ing the first ten layers of VGG-16 [21] with only three pool-
lutional layer with a 3 × 3 Sobel kernel. Since the generated ing layers instead of five to suppress the detrimental effects
feature map is only 1/2 of the original input, it needs to be on output accuracy caused by the pooling operation. Since
upsampled by the deconvolutional layer (bilinear interpola- the output (density maps) of CSRNet is smaller (1/8 of in-
tion). In the other approach, we try dilated convolution and put size), we choose bilinear interpolation with the factor
adapt the same 3 × 3 Sobel kernel to a dilated kernel with a of 8 for scaling and make sure the output shares the same
 Dataset Generating method Configurations of CSRNet
ShanghaiTech Part A [18] A B C D
Geometry-adaptive kernels
UCF CC 50 [22] input(unfixed-resolution color image)
ShanghaiTech Part B [18] Fixed kernel: σ = 15 front-end
TRANCOS [44] Fixed kernel: σ = 10 (fine-tuned from VGG-16)
The WorldExpo’10 [3] conv3-64-1
Fixed kernel: σ = 3
The UCSD [23] conv3-64-1
Table 2. The ground truth generating methods for different datasets max-pooling
conv3-128-1
resolution as the input image. With the same size, CSR- conv3-128-1
Net generated results are comparable with the ground truth max-pooling
results using the PSNR (Peak Signal-to-Noise Ratio) and conv3-256-1
conv3-256-1
SSIM (Structural Similarity in Image [43]).
conv3-256-1
max-pooling
3.2. Training method conv3-512-1
In this section, we provide specific details of CSRNet conv3-512-1
training. By taking advantage of the regular CNN network conv3-512-1
(without branch structures), CSRNet is easy to implement back-end (four different configurations)
conv3-512-1 conv3-512-2 conv3-512-2 conv3-512-4
and fast to deploy.
conv3-512-1 conv3-512-2 conv3-512-2 conv3-512-4
conv3-512-1 conv3-512-2 conv3-512-2 conv3-512-4
3.2.1 Ground truth generation conv3-256-1 conv3-256-2 conv3-256-4 conv3-256-4
conv3-128-1 conv3-128-2 conv3-128-4 conv3-128-4
Following the method of generating density maps in conv3-64-1 conv3-64-2 conv3-64-4 conv3-64-4
[18], we use the geometry-adaptive kernels to tackle the conv1-1-1
highly congested scenes. By blurring each head annotation
Table 3. Configuration of CSRNet. All convolutional layers use
using a Gaussian kernel (which is normalized to 1), we gen- padding to maintain the previous size. The convolutional layers’
erate the ground truth considering the spatial distribution of parameters are denoted as “conv-(kernel size)-(number of filters)-
all images from each dataset. The geometry-adaptive kernel (dilation rate)”, max-pooling layers are conducted over a 2 × 2
is defined as: pixel window with stride 2.

N
X 3.2.3 Training details
F (x) = δ(x − xi ) × Gσi (x), with σi = βdi (2)
i=1 We use a straightforward way to train the CSRNet as
an end-to-end structure. The first 10 convolutional layers
For each targeted object xi in the ground truth δ, we use are fine-tuned from a well-trained VGG-16 [21]. For the
di to indicate the average distance of k nearest neighbors. other layers, the initial values come from a Gaussian ini-
To generate the density map, we convolve δ(x − xi ) with tialization with 0.01 standard deviation. Stochastic gradient
a Gaussian kernel with parameter σi (standard deviation), descent (SGD) is applied with fixed learning rate at 1e-6
where x is the position of pixel in the image. In experiment, during training. Also, we choose the Euclidean distance to
we follow the configuration in [18] where β = 0.3 and measure the difference between the ground truth and the es-
k = 3. For input with sparse crowd, we adapt the Gaussian timated density map we generated which is similar to other
kernel to the average head size to blur all the annotations. works [19, 18, 4]. The loss function is given as follow:
The setups for different datasets are shown in Table 2.

N
3.2.2 Data augmentation 1 X 2
L(Θ) = k Z(Xi ; Θ) − ZiGT k2 (3)
2N i=1
We crop 9 patches from each image at different loca-
tions with 1/4 size of the original image. The first four
patches contain four quarters of the image without overlap- where N is the size of training batch and Z(Xi ; Θ) is
ping while the other five patches are randomly cropped from the output generated by CSRNet with parameters shown as
the input image. After that, we mirror the patches so that we Θ. Xi represents the input image while ZiGT is the ground
double the training set. truth result of the input image Xi .
4. Experiments Architecture MAE MSE
CSRNet A 69.7 116.0
We demonstrate our approach in five different public
CSRNet B 68.2 115.0
datasets [18, 3, 22, 23, 44]. Compared to the previous state-
CSRNet C 71.91 120.58
of-the-art methods [4, 5], our model is smaller, more accu-
rate, and easier to train and deploy. In this section, the eval- CSRNet D 75.81 120.82
uation metrics are introduced, and then an ablation study Table 4. Comparison of architectures on ShanghaiTech Part A
of ShanghaiTech Part A dataset is conducted to analyze the dataset
configuration of our model (shown in Table 3). Along with
the ablation study, we evaluate and compare our proposed their back-end respectively while CSRNet C combines the
method to the previous state-of-the-art methods in all these dilated rate of 2 and 4. The number of parameters of these
five datasets. The implementation of our model is based on four models are the same as 16.26M. We intend to compare
the Caffe framework [13]. the results by using different dilation rates. After training
on Shanghai Part A dataset using the method mentioned
4.1. Evaluation metrics in Sec. 3.2, we perform the evaluation metrics defined in
Sec. 4.1. We try dropout [45] for preventing the potential
The MAE and the MSE are used for evaluation which
overfitting problem but there is no significant improvement.
are defined as:
So we do not include dropout in our model. The detailed
N evaluation results are shown in Table 4, where CSRNet B
1 X
M AE = | Ci − CiGT | (4) achieves the lowest error (the highest accuracy). Therefore,
N i=1 we use CSRNet B as the proposed CSRNet for the follow-
v ing experiments.
u
u1 X N
M SE = t | Ci − CiGT |2 (5) 4.3. Evaluation and comparison
N i=1
4.3.1 ShanghaiTech dataset
where N is the number of images in one test sequence
and CiGT is the ground truth of counting. Ci represents the ShanghaiTech crowd counting dataset contains 1198 an-
estimated count which is defined as follows: notated images with a total amount of 330,165 persons [3].
This dataset consists of two parts as Part A containing 482
L X
W
X images with highly congested scenes randomly downloaded
Ci = zl,w (6)
from the Internet while Part B includes 716 images with
l=1 w=1
relatively sparse crowd scenes taken from streets in Shang-
L and W show the length and width of the density map hai. Our method is evaluated and compared to other six
respectively while zl,w is the pixel at (l, w) of the generated recent works and results are shown in Table 5. It indicates
density map. Ci means the estimated counting number for that our method achieves the lowest MAE (the highest ac-
image Xi . curacy) in Part A compared to other methods and we get
We also use the PSNR and SSIM to evaluate the quality 7% lower MAE than the state-of-the-art solution called CP-
of the output density map on ShanghaiTech Part A dataset. CNN. CSRNet also delivers 47.3% lower MAE in Part B
To calculate the PSNR and SSIM, we follow the preprocess compared to the CP-CNN. To evaluate the quality of gen-
given by [5], which includes the density map resizing (same erated density map, we compare our method to the MCNN
size with the original input) with interpolation and normal- and the CP-CNN using Part A dataset and we follow the
ization for both ground truth and predicted density map. evaluation metrics in Sec. 3.2. Samples of the test cases can
be found in Fig 5. Results are shown in Table 6 which in-
4.2. Ablations on ShanghaiTech Part A dicates CSRNet achieves the highest SSIM and PSNR. We
In this subsection, we perform an ablation study to an- also report the quality result of ShanghaiTech dataset in Ta-
alyze the four configurations of the CSRNet on Shang- ble 11.
haiTech Part A dataset [18] which is a new large-scale
crowd counting dataset including 482 images for congested 4.3.2 UCF CC 50 dataset
scenes with 241,667 annotated persons. It is challenging
to count from these images because of the extremely con- UCF CC 50 dataset includes 50 images with different
gested scenes, the varied perspective, and the unfixed res- perspective and resolutions [22]. The number of annotated
olution. These four configurations are shown in Table 3. persons per image ranges from 94 to 4543 with an average
CSRNet A is the network with all the dilation rate of 1. number of 1280. 5-fold cross-validation is performed fol-
CSRNet B and D maintain the dilation rate of 2 and 4 in lowing the standard setting in [22]. Result comparisons of
 Method MAE MSE
Idrees et al. [22] 419.5 541.6
Zhang et al. [3] 467.0 498.5
MCNN [18] 377.6 509.1
Onoro et al. [20] Hydra-2s 333.7 425.2
Onoro et al. [20] Hydra-3s 465.7 371.8
Walach et al. [36] 364.4 341.4
Marsden et al. [38] 338.6 424.5
Cascaded-MTL [39] 322.8 397.9
Switching-CNN [4] 318.1 439.2
CP-CNN [5] 295.8 320.9
CSRNet (ours) 266.1 397.5

Table 7. Estimation errors on UCF CC 50 dataset

are shown in Table 8. The proposed CSRNet delivers the

best accuracy in 4 out of 5 scenes and it achieves the best
accuracy on average.
Figure 5. The first row shows the samples of the testing set in
ShanghaiTech Part A dataset. The second row shows the ground Method Sce.1 Sce.2 Sce.3 Sce.4 Sce.5 Avg.
truth for each sample while the third row presents the generated Chen et al. [46] 2.1 55.9 9.6 11.3 3.4 16.5
density map by CSRNet. Zhang et al. [3] 9.8 14.1 14.3 22.2 3.7 12.9
MCNN [18] 3.4 20.6 12.9 13.0 8.1 11.6
Shang et al. [37] 7.8 15.4 14.9 11.8 5.8 11.7
Part A Part B Switching-CNN [4] 4.4 15.7 10.0 11.0 5.9 9.4
Method MAE MSE MAE MSE CP-CNN [5] 2.9 14.7 10.5 10.4 5.8 8.86
CSRNet (ours) 2.9 11.5 8.6 16.6 3.4 8.6
Zhang et al. [3] 181.8 277.7 32.0 49.8
Marsden et al. [38] 126.5 173.5 23.8 33.1 Table 8. Estimated errors on the WorldExpo’10 dataset
MCNN [18] 110.2 173.2 26.4 41.3
Cascaded-MTL [39] 101.3 152.4 20.0 31.1
Switching-CNN [4] 90.4 135.0 21.6 33.4 4.3.4 The UCSD dataset
CP-CNN [5] 73.6 106.4 20.1 30.1
The UCSD dataset [23] has 2000 frames captured by
CSRNet (ours) 68.2 115.0 10.6 16.0
surveillance cameras. These scenes contain sparse crowd
Table 5. Estimation errors on ShanghaiTech dataset varying from 11 to 46 persons per image. The region of
interest (ROI) is also provided. Because the resolution of
Method PSNR SSIM each frame is fixed and small (238 × 158), it is difficult to
MCNN [18] 21.4 0.52 generate a high-quality density map after frequent pooling
CP-CNN [5] 21.72 0.72 operations. So we preprocess the frames by using bilinear
CSRNet (ours) 23.79 0.76 interpolation to resize them into 952 × 632. Among the
2000 frames, we use frames 601 through 1400 as training
Table 6. Quality of density map on ShanghaiTech Part A dataset
set and the rest of them as testing set according to [23].
Before blurring the annotation as we mentioned in Sec. 3.2,
MAE and MSE are listed in Table 7 while the quality of all the frames and the corresponding dot maps are masked
generated density map can be found in Table 11. with ROI. The accuracy of running UCSD dataset is shown
in Table 9 and we outperform most of the previous meth-
ods except MCNN in the MAE category. Results indicate
4.3.3 The WorldExpo’10 dataset
that our method can perform not only counting tasks for
The WorldExpo’10 dataset [3] consists of 3980 anno- extremely dense crowds but also tasks for relative sparse
tated frames from 1132 video sequences captured by 108 scenes. Also, we provides the quality of generated density
different surveillance cameras. This dataset is divided into map in Table 11.
a training set (3380 frames) and a testing set (600 frames)
from five different scenes. The region of interest (ROI) is 4.3.5 TRANCOS dataset
provided for the whole dataset. Each frame and its dot maps
are masked with ROI during preprocessing, and we train our Beyond the crowd counting, we setup an experiment
model following the instructions given in Sec. 3.2. Results on the TRANCOS dataset [44] for vehicle counting to
 Method MAE MSE
Zhang et al. [3] 1.60 3.31
CCNN [20] CCNN 1.51 -
Switching-CNN [4] 1.62 2.10
FCN-rLSTM [24] 1.54 3.02
CSRNet (ours) 1.16 1.47
MCNN [18] 1.07 1.35

Table 9. Estimation errors on the UCSD dataset

demonstrate the robustness and generalization of our ap-

proach. TRANCOS is a public traffic dataset containing
1244 images of different congested traffic scenes captured
by surveillance cameras with 46796 annotated vehicles.
Also, the region of interest (ROI) is provided for the eval-
uation. The perspectives of images are not fixed and the
images are collected from very different scenarios. The
Grid Average Mean Absolute Error (GAME) [44] is used
for evaluation in this test. The GAME is defined as follow:
Figure 6. The first row shows samples of the testing set in TRAN-
COS [44] dataset with ROI. The second row shows the ground
4 L
N truth for each sample. The third row shows the generated density
1 X X  l 
GAM E(L) = ( DIn − DIl ngt ) (7) map by CSRNet.

N n=1
l=1
Dataset PSNR SSIM
ShanghaiTech Part A [18] 23.79 0.76
where N is the number of images in testing set, and DIl n
ShanghaiTech Part B [18] 27.02 0.89
is the estimated result of the input image n within region l.
UCF CC 50 [22] 18.76 0.52
DIl gt is the corresponding ground truth result. For a specific
n The WorldExpo’10 [3] 26.94 0.92
level L, the GAME(L) subdivides the image using a grid The UCSD [23] 20.02 0.86
of 4L non-overlapping regions which cover the full image, TRANCOS [44] 27.10 0.93
and the error is computed as the sum of the MAE in each of
these regions. When L = 0, the GAME is equivalent to the Table 11. The quality of density maps generated by CSRNet in 5
MAE metric. datasets
We compare our approach with the previous state-of-
the-art methods [47, 32, 20, 24]. The method in [20] uses 5. Conclusion
the MCNN-like network to generate the density map while
the model in [24] deploys a combination of fully convolu- In this paper, we proposed a novel architecture called
tional neural networks (FCN) and a long short-term mem- CSRNet for crowd counting and high-quality density map
ory network (LSTM). Results are shown in Table 10 with generation with an easy-trained end-to-end approach. We
three examples shown in Fig. 6. Our model achieves a used the dilated convolutional layers to aggregate the multi-
significant improvement on four different GAME metrics. scale contextual information in the congested scenes. By
Compared to the result from [20], CSRNet delivers 67.7% taking advantage of the dilated convolutional layers, CSR-
lower GAME(0), 60.1% lower GAME(1), 48.7% lower Net can expand the receptive field without losing resolution.
GAME(2), and 22.2% lower GAME(3), which is the best We demonstrated our model in four crowd counting datasets
solution. We also present the quality of generated density with the state-of-the-art performance. We also extended our
map in Table 11. model to vehicle counting task and our model achieved the
best accuracy as well.
Method GAME 0 GAME 1 GAME 2 GAME 3
Fiaschi et al. [47] 17.77 20.14 23.65 25.99
Lempitsky et al. [32] 13.76 16.72 20.72 24.36
Hydra-3s [20] 10.99 13.75 16.69 19.32 6. Acknowledgement
FCN-HA [24] 4.21 - - -
CSRNet (Ours) 3.56 5.49 8.57 15.04
This work was supported by the IBM-Illinois Center for
Table 10. GAME on the TRANCOS dataset Cognitive Computing System Research (C3SR) - a research
collaboration as part of the IBM AI Horizons Network.
References and Trevor Darrell. Caffe: Convolutional architecture for fast
feature embedding. In Proceedings of the 22nd ACM inter-
[1] Beibei Zhan, Dorothy N Monekosso, Paolo Remagnino, Ser- national conference on Multimedia, pages 675–678. ACM,
gio A Velastin, and Li-Qun Xu. Crowd analysis: a survey. 2014.
Machine Vision and Applications, 19(5-6):345–357, 2008.
[14] Jiantao Qiu, Jie Wang, Song Yao, Kaiyuan Guo, Boxun Li,
[2] Teng Li, Huan Chang, Meng Wang, Bingbing Ni, Richang Erjin Zhou, Jincheng Yu, Tianqi Tang, Ningyi Xu, Sen Song,
Hong, and Shuicheng Yan. Crowded scene analysis: A sur- Yu Wang, and Huazhong Yang. Going deeper with embed-
vey. IEEE transactions on circuits and systems for video ded FPGA platform for convolutional neural network. In
technology, 25(3):367–386, 2015. Proceedings of the 2016 ACM/SIGDA International Sympo-
[3] Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang sium on Field-Programmable Gate Arrays, FPGA ’16, pages
Yang. Cross-scene crowd counting via deep convolutional 26–35, New York, NY, USA, 2016. ACM.
neural networks. In Proceedings of the IEEE Conference on
[15] Xiaofan Zhang, Xinheng Liu, Anand Ramachandran,
Computer Vision and Pattern Recognition, pages 833–841,
Chuanhao Zhuge, Shibin Tang, Peng Ouyang, Zuofu Cheng,
2015.
Kyle Rupnow, and Deming Chen. High-performance video
[4] Deepak Babu Sam, Shiv Surya, and R Venkatesh Babu. content recognition with long-term recurrent convolutional
Switching convolutional neural network for crowd counting. network for FPGA. In Field Programmable Logic and Appli-
In Proceedings of the IEEE Conference on Computer Vision cations (FPL), 2017 27th International Conference on, pages
and Pattern Recognition, volume 1, page 6, 2017. 1–4. IEEE, 2017.
[5] Vishwanath A Sindagi and Vishal M Patel. Generating high- [16] Xiaofan Zhang, Anand Ramachandran, Chuanhao Zhuge,
quality crowd density maps using contextual pyramid CNNs. Di He, Wei Zuo, Zuofu Cheng, Kyle Rupnow, and Deming
In Proceedings of the IEEE Conference on Computer Vision Chen. Machine learning on FPGAs to face the IoT revolu-
and Pattern Recognition, pages 1861–1870, 2017. tion. In Computer-Aided Design (ICCAD), 2017 IEEE/ACM
[6] Cong Zhang, Kai Kang, Hongsheng Li, Xiaogang Wang, International Conference on, pages 819–826. IEEE, 2017.
Rong Xie, and Xiaokang Yang. Data-driven crowd under- [17] Renzo Andri, Lukas Cavigelli, Davide Rossi, and Luca
standing: a baseline for a large-scale crowd dataset. IEEE Benini. Yodann: An ultra-low power convolutional neu-
Transactions on Multimedia, 18(6):1048–1061, 2016. ral network accelerator based on binary weights. In VLSI
[7] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully (ISVLSI), 2016 IEEE Computer Society Annual Symposium
convolutional networks for semantic segmentation. In Pro- on, pages 236–241. IEEE, 2016.
ceedings of the IEEE Conference on Computer Vision and [18] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao,
Pattern Recognition, pages 3431–3440, 2015. and Yi Ma. Single-image crowd counting via multi-column
[8] Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, convolutional neural network. In Proceedings of the IEEE
Ming-Ming Cheng, Jiashi Feng, Yao Zhao, and Shuicheng Conference on Computer Vision and Pattern Recognition,
Yan. Stc: A simple to complex framework for weakly- pages 589–597, 2016.
supervised semantic segmentation. IEEE transactions on [19] Lokesh Boominathan, Srinivas SS Kruthiventi, and
pattern analysis and machine intelligence, 39(11):2314– R Venkatesh Babu. Crowdnet: a deep convolutional network
2320, 2017. for dense crowd counting. In Proceedings of the 2016 ACM
[9] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming on Multimedia Conference, pages 640–644. ACM, 2016.
Cheng, Yao Zhao, and Shuicheng Yan. Object region mining [20] Daniel Onoro-Rubio and Roberto J López-Sastre. Towards
with adversarial erasing: A simple classification to semantic perspective-free object counting with deep learning. In Eu-
segmentation approach. In IEEE CVPR, 2017. ropean Conference on Computer Vision, pages 615–629.
[10] Fisher Yu and Vladlen Koltun. Multi-scale context aggrega- Springer, 2016.
tion by dilated convolutions. In ICLR, 2016. [21] Karen Simonyan and Andrew Zisserman. Very deep convo-
[11] L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and lutional networks for large-scale image recognition. arXiv
A. L. Yuille. Deeplab: Semantic image segmentation with preprint arXiv:1409.1556, 2014.
deep convolutional nets, atrous convolution, and fully con- [22] Haroon Idrees, Imran Saleemi, Cody Seibert, and Mubarak
nected crfs. IEEE Transactions on Pattern Analysis and Ma- Shah. Multi-source multi-scale counting in extremely dense
chine Intelligence, PP(99):1–1, 2017. crowd images. In Proceedings of the IEEE Conference
[12] Junting Pan, Elisa Sayrol, Xavier Giro-i Nieto, Kevin on Computer Vision and Pattern Recognition, pages 2547–
McGuinness, and Noel E O’Connor. Shallow and deep con- 2554, 2013.
volutional networks for saliency prediction. In Proceedings [23] A. B. Chan, Zhang-Sheng John Liang, and N. Vasconce-
of the IEEE Conference on Computer Vision and Pattern los. Privacy preserving crowd monitoring: Counting peo-
Recognition, pages 598–606, 2016. ple without people models or tracking. In 2008 IEEE Con-
[13] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey ference on Computer Vision and Pattern Recognition, pages
Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, 1–7, June 2008.
[24] Shanghang Zhang, Guanhang Wu, Joao P Costeira, and [38] Mark Marsden, Kevin McGuiness, Suzanne Little, and
Jose MF Moura. Fcn-rlstm: Deep spatio-temporal neural Noel E O’Connor. Fully convolutional crowd counting on
networks for vehicle counting in city cameras. In Proceed- highly congested scenes. arXiv preprint arXiv:1612.00220,
ings of the IEEE Conference on Computer Vision and Pattern 2016.
Recognition, pages 3667–3676, 2017. [39] Vishwanath A Sindagi and Vishal M Patel. Cnn-based cas-
[25] Chen Change Loy, Ke Chen, Shaogang Gong, and Tao Xi- caded multi-task learning of high-level prior and density es-
ang. Crowd counting and profiling: Methodology and eval- timation for crowd counting. In Advanced Video and Signal
uation. In Modeling, Simulation and Visual Analysis of Based Surveillance (AVSS), 2017 14th IEEE International
Crowds, pages 347–382. Springer, 2013. Conference on, pages 1–6. IEEE, 2017.
[26] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Per- [40] Liang-Chieh Chen, George Papandreou, Florian Schroff, and
ona. Pedestrian detection: An evaluation of the state of the Hartwig Adam. Rethinking atrous convolution for semantic
art. IEEE transactions on pattern analysis and machine in- image segmentation. CoRR, abs/1706.05587, 2017.
telligence, 34(4):743–761, 2012.
[41] Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and
[27] Paul Viola and Michael J Jones. Robust real-time face detec- Rob Fergus. Deconvolutional networks. In Computer Vision
tion. International journal of computer vision, 57(2):137– and Pattern Recognition (CVPR), 2010 IEEE Conference on,
154, 2004. pages 2528–2535. IEEE, 2010.
[28] Navneet Dalal and Bill Triggs. Histograms of oriented gra- [42] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han.
dients for human detection. In Computer Vision and Pat- Learning deconvolution network for semantic segmentation.
tern Recognition, 2005. CVPR 2005. IEEE Computer Society In Proceedings of the IEEE International Conference on
Conference on, volume 1, pages 886–893. IEEE, 2005. Computer Vision, pages 1520–1528, 2015.
[29] Pedro F Felzenszwalb, Ross B Girshick, David McAllester,
[43] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si-
and Deva Ramanan. Object detection with discriminatively
moncelli. Image quality assessment: from error visibility to
trained part-based models. IEEE transactions on pattern
structural similarity. IEEE transactions on image processing,
analysis and machine intelligence, 32(9):1627–1645, 2010.
13(4):600–612, 2004.
[30] Antoni B Chan and Nuno Vasconcelos. Bayesian poisson re-
gression for crowd counting. In Computer Vision, 2009 IEEE [44] Roberto Lpez-Sastre Saturnino Maldonado Bascn Ricardo
12th International Conference on, pages 545–551. IEEE, Guerrero-Gmez-Olmedo, Beatriz Torre-Jimnez and Daniel
2009. Ooro-Rubio. Extremely overlapping vehicle counting. In
Iberian Conference on Pattern Recognition and Image Anal-
[31] D. G. Lowe. Object recognition from local scale-invariant ysis (IbPRIA), 2015.
features. In Proceedings of the Seventh IEEE International
Conference on Computer Vision, volume 2, pages 1150– [45] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya
1157 vol.2, 1999. Sutskever, and Ruslan Salakhutdinov. Dropout: A simple
way to prevent neural networks from overfitting. Journal of
[32] Victor Lempitsky and Andrew Zisserman. Learning to count
Machine Learning Research, 15:1929–1958, 2014.
objects in images. In Advances in Neural Information Pro-
cessing Systems, pages 1324–1332, 2010. [46] Ke Chen, Shaogang Gong, Tao Xiang, and Chen
Change Loy. Cumulative attribute space for age and crowd
[33] Viet-Quoc Pham, Tatsuo Kozakaya, Osamu Yamaguchi, and
density estimation. In Proceedings of the IEEE conference
Ryuzo Okada. Count forest: Co-voting uncertain number of
on computer vision and pattern recognition, pages 2467–
targets using random forest for crowd density estimation. In
2474, 2013.
Computer Vision (ICCV), 2015 IEEE International Confer-
ence on, pages 3253–3261. IEEE, 2015. [47] L. Fiaschi, U. Koethe, R. Nair, and F. A. Hamprecht. Learn-
[34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ing to count with regression forest and structured labels. In
Imagenet classification with deep convolutional neural net- Proceedings of the 21st International Conference on Pattern
works. In Advances in neural information processing sys- Recognition (ICPR2012), pages 2685–2688, Nov 2012.
tems, pages 1097–1105, 2012.
[35] F. Chollet. Xception: Deep learning with depthwise sepa-
rable convolutions. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pages 1800–1807,
July 2017.
[36] Elad Walach and Lior Wolf. Learning to count with CNN
boosting. In European Conference on Computer Vision,
pages 660–676. Springer, 2016.
[37] Chong Shang, Haizhou Ai, and Bo Bai. End-to-end crowd
counting via joint learning local and global count. In Image
Processing (ICIP), 2016 IEEE International Conference on,
pages 1215–1219. IEEE, 2016.
7. Appendix: supplementary material
In this appendix, additional results generated by CSRNet from five datasets (ShanghaiTech [18], UCF CC 50 [22], World-
Expo’10 [3], UCSD [23], and TRANCOS [44]) are presented to demonstrate the validity of our design. Two criteria are used
as the PSNR (Peak Signal-to-Noise Ratio) and the SSIM (Structural Similarity in Image [43] to evaluate our design’s quality
of generated density maps. Samples from these 5 datasets are shown in Fig. 7 to Fig. 12, which represent a variety of density
levels.

Figure 7. Samples generated by CSRNet from ShanghaiTech Part A [18] dataset. The left column shows the original images; the medium
column displays the ground truth density maps while the right column indicates our generated density maps.
Figure 8. Samples generated by CSRNet from ShanghaiTech Part B [18] dataset. The left column shows the original images; the medium
column displays the ground truth density maps while the right column indicates our generated density maps.
Figure 9. Samples generated by CSRNet from UCF CC 50 [22] dataset. The left column shows the original images; the medium column
displays the ground truth density maps while the right column indicates our generated density maps.
Figure 10. Samples generated by CSRNet from WorldExpo’10 [3] dataset. The left column shows the images masked by the ROI (region
of interest); the medium column displays the ground truth density maps and the right column indicates our generated density maps.
Figure 11. Samples generated by CSRNet from UCSD [23] dataset. The left column shows the images masked by the ROI; the medium
column displays the ground truth density maps and the right column indicates our generated density maps.
Figure 12. Samples generated by CSRNet from TRANCOS [44] dataset. The left column shows the images masked by the ROI; the
medium column displays the ground truth density maps while the right column shows the generated density maps.