# Crowd Counting via Perspective-Guided Fractional-Dilation Convolution

Zhaoyi Yan, Ruimao Zhang, Hongzhi Zhang, Qingfu Zhang, and Wangmeng Zuo The
submitted manuscript is an extension of our ICCV 2019 paper “Perspective-
guided Convolution Networks for Crowd Counting”. In this version, we have
completely rewritten all the paper. Moreover, we present a novel perspective-
guided fractional-dilation convolution to allocate spatially variant receptive
fields. Additionally, all experimental results are updated and more
experiments are conducted for comprehensive evaluation. The conference paper
is also provided as the support materials along with the submission.Z. Yan is
with the School of Computer Science and Technology, Harbin Institute of
Technology, Harbin 150001, China (e-mail: yanzhaoyi@outlook.com).R. Zhang is
with the School of Data Science, The Chinese University of Hong Kong,
Shenzhen, and also with Shenzhen Research Institute of Big Data, Shenzhen
518172, China (e-mail: ruimao.zhang@ieee.org).H. Zhang is with the School of
Computer Science and Technology, Harbin Institute of Technology, Harbin
150001, China (e-mail: zhanghz@hit.edu.cn).Q. Zhang is with the Department of
Computer Science, City University of Hong Kong, Hong Kong, and also with the
Shenzhen Research Institute, City University of Hong Kong, Shenzhen 518057,
China. (e-mail: qingfu.zhang@cityu.edu.hk).W. Zuo is with the School of
Computer Science and Technology, Harbin Institute of Technology, Harbin
150001, China (e-mail: cswmzuo@hit.edu.cn).

###### Abstract

Crowd counting is critical for numerous video surveillance scenarios. One of
the main issues in this task is how to handle the dramatic scale variations of
pedestrians caused by the perspective effect. To address this issue, this
paper proposes a novel convolution neural network-based crowd counting method,
termed Perspective-guided Fractional-Dilation Network (PFDNet). By modeling
the continuous scale variations, the proposed PFDNet is able to select the
proper fractional-dilation kernels for adapting to different spatial
locations. It significantly improves the flexibility of the state-of-the-arts
that only consider the discrete representative scales. In addition, by
avoiding the multi-scale or multi-column architecture that used in other
methods, it is computationally more efficient. In practice, the proposed
PFDNet is constructed by stacking multiple Perspective-guided Fractional-
Dilation Convolutions (PFC) on a VGG16-BN backbone. By introducing a novel
generalized dilation convolution operation, the PFC can handle fractional
dilation ratios in the spatial domain under the guidance of perspective
annotations, achieving continuous scales modeling of pedestrians. To deal with
the problem of unavailable perspective information in some cases, we further
introduce an effective perspective estimation branch to the proposed PFDNet,
which can be trained in either supervised or weakly-supervised setting once
the branch has been pre-trained. Extensive experiments show that the proposed
PFDNet outperforms state-of-the-art methods on ShanghaiTech A, ShanghaiTech B,
WorldExpo’10, UCF-QNRF, UCF_CC_50 and TRANCOS dataset, achieving MAE $53.8$,
$6.5$, $6.8$, $84.3$, $205.8$, and $3.06$ respectively.

##  I Introduction

With the growth of global population and urbanization, the frequency of crowd
gathering consistently rises in recent years. In some scenarios, stampedes and
crushes can be life threatening and should always be prevented. In recent
years, with the widespread use of intelligent monitoring [1, 2, 3] in public
areas such as congested squares, stations and markets, it becomes possible to
adopt these techniques to manage, control and prevent crowd gathering in
cities. Among various congested scene content analysis techniques, crowd
counting [4, 5, 6, 7, 8], which aims to estimate the count of the pedestrians
in a scene, is one of the fundamental tasks and has drawn considerable
research attention.

Accurate counting in a single image remains a challenging topic due to the
complex distribution of people, non-uniform illumination, inter- and intra-
scene scale variations, cluttering and occlusions, etc. In the past decades,
many methods have been proposed, which can be roughly categorised into
detection-based [9, 10, 11, 12], regression-based [13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 17], and deep learning-based approaches [4, 5, 6, 7, 8, 6, 23, 24,
25, 26, 27, 28, 29]. With powerful representation ability of modern
convolutional neural networks (CNNs), deep learning-based methods become
dominant and have achieved superior performances in terms of accuracy and
robustness.

Although significant progress has been achieved in the past decade by using
deep models, the dramatic intra-scene scale variations of people due to the
perspective effect is still a big problem, limiting the improvement of
counting accuracy. To address this issue, existing methods [4, 5, 6, 7, 8]
usually adopt multi-scale or multi-column architectures to aggregate the
features from different scales. However, such schemes usually suffer from
several limitations. Firstly, the multi-column architecture [4] inevitably
introduces redundant parameters, huge computation burden and difficulty in
optimization. For example, as shown in [30], such methods cannot even compete
with a deeper CNN due to the high correlation of the features learned by
different columns. Secondly, they only consider discrete scales, which is
limited when addressing continuous scale variations in practical scenarios.

Recently, a single-column network architecture termed CSRNet [30] has been
proposed to deal with crowd counting problem. By applying dilated
convolutions, this method achieves state-of-the-art performance. However, it
still delivers fixed receptive field for different scales of people, thereby
remaining vulnerable to the highly variant intra-scene scales. As illustrates
in Fig. 1 (b), this method performs well at intermediate scales, but reveals
its incapability for small or large scales of pedestrian heads, indicating its
limitation in dealing with scale variations. Based on the above observation,
we argue it is difficult to implicitly model large scale continuous variations
by only using normal components in a multi- or single-column architectures.
Thus we take a step forward to explicitly tackle the scale variations.

Figure 1: Density maps predicted by CSRNet [30] and our PFDNet. For this
image, the MAE of PFDNet is 3.9, much lower than that of CSRNet (10.1). It is
observed that PFDNet delivers consistent superior performances at either
smaller or larger scales among the marked regions.

Since the perspective information encodes the localized resolution of a scene
in the image, it is natural to exploit such annotation as an indication to
estimate the scale of pedestrians in a scene. Thus, this work proposes a novel
Perspective-guided Fractional-Dilation Network (i.e., PFDNet), which aims to
allocate spatially variant receptive fields under the guidance of perspective
information. In our implementation, the proposed PFDNet is constructed by
stacking multiple Perspective-guided Fractional-Dilation Convolutions (PFC) on
a VGG16-BN backbone network. In each PFC, the perspective information is
firstly normalized by a customized normalization function and then transformed
into a dilation rate map via the linear mapping. Then we introduce a novel
generalized dilation convolution operation, which performs feature aggregation
of each position over the sliding window with spatially variant dilation rate,
achieving fractional dilation ratios in the spatial domain. Without bells and
whistles, our proposed PFDNet achieves significant performance gain (e.g.
$17.4\%$ and $33.7\%$ improvement on ShanghaiTech A / B dataset) by replacing
normal (dilated) convolutions in CSRNet [30] with PFC when using ground-truth
perspective maps.

In practice, to deal with the problem of unavailable perspective information,
we further introduce an effective perspective estimation branch to the
proposed PFDNet, which can be trained in either supervised or weakly-
supervised setting once the branch has been pre-trained. And a three-step
estimation strategy is proposed to make the perspective estimation more
stable.

In summary, the main contributions of this paper are three folds. (1) We
propose a Perspective-guided Fractional-Dilation Convolution (PFC) to mitigate
the scale variation in crowd counting. It is a plug and play module, which is
able to work as a spatial allocator of varying receptive field. (2) Based on
PFC, we build up PFDNet for crowd counting, with which a perspective
information estimation branch is introduced, making the whole net can be
trained with or without perspective annotations. (3) With the proposed PFDNet,
we achieve dramatic performance gain over the state-of-the-arts on several
crowd counting benchmarks.

This work is an extension of our previous conference paper [31]. Compared with
[31], this paper extends previous method in the following aspects:

  * •

We propose perspective-guided fractional-dilation convolution, generalizing
perspective-guided convolution in [31] . PFC is a generalized dilation
convolution that even works when the dilation rate is a fractional number.

  * •

PFDNet uses a lightweight single-column architecture compared with PGCNet
[31]. While PGC [31] concatenates the smoothed feature maps together with the
original feature maps, making the parameters increase dramatically.

  * •

We also conduct other counting experiments (i.e. vehicle counting) in this
version by using our proposed PFDNet, further validating its broad
applications.

##  II Related Work

We provide a brief review on the three categories of crowd counting methods,
i.e., detection-based, regression-based and CNN-based methods. Besides, the
use of perspective normalization in crowd counting, including PGCNet [31], is
also reviewed.

###  II-A Detection-based Methods

One straightforward way for single image crowd counting is to conduct human
detection and count the number of bounding boxes. Early researches usually
exploit hand-crafted low-level features (e.g., Haar wavelets [9] and histogram
of oriented gradient (HOG) [10]) which are then fed to the subsequent
classifiers. However, they tend to deliver poor performances when the scene is
highly-congested. To alleviate such issue, part-based detection is proposed
for pedestrian counting by detecting particular body parts [11, 12].

###  II-B Regression-based Methods

Regression-based methods directly learn a mapping from an input image to a
numeric crowd count, which gives rise to two other issues, i.e., how to
achieve effective feature extraction and how to perform reliable regression.
An early work [13] adopts holistic features to characterize the global
property of a scene, local feature descriptors, e.g., LBP [14], SIFT [15] and
even CNN features [16], have also been introduced to boost the effectiveness
of feature extraction [17, 18]. In terms of regression models, linear
regression and its improvements [19, 20, 21, 22] have been widely employed to
learn the regression mapping, and in [17] the multi-scale Markov random field
is introduced to aggregate the local counts.

###  II-C CNN-based Methods

Many CNN-based crowd counting methods have been proposed in recent years.
These methods usually focus on typical techniques, including multi-scale [4,
5, 6, 7, 8], context [6], multi-task [23, 24, 25], attention [32, 33, 34, 35,
36], GNN [37], loss functions [38, 39], classification [40], detection [26,
41], NAS [42], reinforcement learning [43] and others [44, 27, 28, 45, 46, 47,
48, 49]. Recently, some other methods have been proposed in handling the scale
variation issue. For instance, Zhang et al. [4] suggested a multi-column
architecture (MCNN) that combines features with different sizes of receptive
fields. In Switching-CNN [5], one of the three regressors is assigned for an
input image by referring to its specific crowd density. CP-CNN [6]
incorporates MCNN with local and global contexts. SANet [50] employs scale
aggregation modules for multi-scale representation. Instead of multi-column
architecture, CSRNet [30] enlarges receptive fields by stacking dilated
convolutions. Cheng et al. [51] aimed to replace Euclidean distance with
Maximum Excess over Pixels (MEP) loss and achieved the promising performance.
Ma et al. [38] proposed Bayesian loss which constructs a density contribution
probability model from point annotations. Liu et al. [52] proposed local
counting map and Wang et al. [39] reformulated crowd counting as a
distribution matching problem and propose DM-Count based on optimal transport.

###  II-D Perspective Normalization

Chan et al. [19] first proposed feature normalization for foreground objects
under the guidance of perspective information. Later, Lempitsky et al. [53]
proposed a novel MESA-distance to deal with perspective distortion. Among the
CNN-based methods, a lot of works [23, 4, 5, 24] prefer the perspective
information as a prior in generating better density maps, but few of them
attempt to encode such information directly into the network architecture. To
our best knowledge, PACNN [29] is the most relevant to our method. However,
PACNN is still based on the multi-column architecture, so is limited in only
discrete scale representations. It aims to combine two density maps (predicted
by two columns pool4_2 and pool4_1 on VGG-16 [54], respectively) via
corresponding weights generated by the perspective map. In comparison, we
argue that a better choice would be to allocate spatially variant receptive
fields under the explicit guidance of perspective information during the
training process.

###  II-E Perspective-guided convolution

In PGCNet [31], Yan et al. proposed perspective-guided convolution which
employs spatially variant Gaussian kernels to smooth the feature maps. A
normal convolution is followed afterwards to aggregate information on the
smoothed feature maps. Generally speaking, we point out that there are three
notable differences between PGC [31] and our proposed PFC.

1) The implementation is different. PGC [31] utilizes adaptive Gaussian
kernels to perform spatially variant smoothing and obtain the smoothed
features, and then aggregate the features via a conventional convolution,
while PFC directly samples features by bilinear interpolation on the spatially
variant windows.

2) Our PFC is much faster than PGC [31]. Taken feature map with size
$256\times 96\times 128$ as an example, PGC [31] has to generate Gaussian
kernel $G_{{\sigma}_{i,j}}$ for each position $(i,j)$ and then performs extra
matrix multiplications, resulting in the forward runtime $\sim 10ms$. In
contrast, PFC generalizes the dilated convolution, thus shares similar runtime
with conventional dilated convolution (i.e., it takes $\sim 3.3ms$), which
only takes $\sim 3.7ms$. Besides, when the receptive field of allocation
rises, due to the intrinsic characteristic of Gaussian smoothing in PGC [31],
the time complexity increases in a quadratic manner attributed to the
computation of Gaussian blurring. While, for our proposed PFC, it shares
similar runtime with conventional dilated convolution whose runtime nearly
keeps static regardless of the changes of dilation rate.

3) PGC [31] suffers from severe feature confusion where our PFC does not
suffer. Due to the spatially variant Gaussian smoothing, the sampling region
of Gaussian kernel largely overlaps with each other, which becomes more severe
when the size of Gaussian kernel is large. Such large overlaps inevitably
introduce severe mixed information for the smoothed features, which limits the
further performance improvement. While, PFC generalizes dilated convolution
where each sampling point in computed via bilinear interpolation of the
nearest four grid points, which obviously does not suffer from feature
confusion.

##  III Method

In this section, we first give the definition and implementation details of
perspective-guided fractional-dilation convolution (PFC) in Sec III-A, and
then introduce a perspective information estimation branch to the proposed
PFDNet in Sec III-B. At last, the learning objectives and the architecture of
PFDNet are presented in Sec III-C.

###  III-A Perspective-guided Fractional-dilation Convolution

Firstly, we give a brief introduction of perspective map. Perspective maps
encode the localized resolution of the scene in an image, and the perspective
value $\mathbf{s}^{gt}_{j}$ at each location $j$ represents the number of
imaging pixels for a one-meter object at the location in the real scene [23].
From [23, 29], we have

$\mathbf{s}^{gt}=\frac{h}{H}=\frac{1}{C-H}y_{h},$ (1)

where $C$ is the camera height from the ground, $H$ is the height of a person
on the ground, and the person’s head top are mapped on the image plane at
$y_{h}$. $h$ is the observed person height and $H$ is set to the mean height
of adults ($1.75m$). Due to $C$ is fixed for each image, $\mathbf{s}^{gt}$ is
a linear mapping of $y_{h}$ and remains the same in each row of $y_{h}$. In
[23], the heights $h_{i}$ of several pedestrians at different positions has
been manually labeled in each image, perspective value $p^{g}_{j}$ can be
obtained by $p^{g}_{j}=\frac{h_{j}}{1.75}$ and $C$ can be estimated. Finally,
the entire ground-truth perspective map can be generated by employing a linear
regression method to fit Eqn. (1).

A straightforward way to deal with intra-scene variations is to adopt the
larger receptive field for the people at larger scales, while to use the
smaller one for that at smaller scales. To this end, we propose a novel
perspective-guided fractional-dilation convolution to adaptively adjust the
receptive field of each convolution operation. Perspective-guided fractional-
dilation convolution has two appealing properties. (1) By removing the
limitation of discrete dilation rates, it extends regular dilation convolution
to a more flexible form. (2) It allocates spatially-variant receptive fields
for different regions under the guidance of the dilation rate map generated
from perspective maps. In the following, we firstly introduce how to generate
the dilation rate map from the perspective map, and then describe the
definition and implementation of fractional-dilation convolution.

\begin{overpic}[scale={0.15}]{./fig/fdc3.pdf}
\par\put(17.0,21.0){\scalebox{0.6}{{\color[rgb]{0,0,0}$\mathbf{p}$}}}
\put(51.0,21.0){\scalebox{0.6}{{\color[rgb]{0,0,0}$\mathbf{p}$}}}
\put(85.0,21.0){\scalebox{0.6}{{\color[rgb]{0,0,0}$\mathbf{p}$}}}
\par\put(3.0,27.0){\scalebox{0.6}{{\color[rgb]{0,0,0}$\mathbf{p}+\mathbf{p}_{k}\cdot
r_{1}$}}}
\put(35.0,32.0){\scalebox{0.6}{{\color[rgb]{0,0,0}$\mathbf{p}+\mathbf{p}_{k}\cdot
r_{2}$}}}
\put(70.0,30.5){\scalebox{0.6}{{\color[rgb]{0,0,0}$\mathbf{p}+\mathbf{p}_{k}\cdot
r_{3}$}}} \par\end{overpic} Figure 2: Illustration of normal convolution,
dilated convolution and factional-dilation convolution, as demonstrated in
(a), (b) and (c). The red points indicates the sampling positions, the dashed
lines show the grids and the intersections are grid points. Pixel/Feature
values only exist on the grid points. When dilation rate $r$ is an integer
(e.g., $r_{1}=1$ in (a) and $r_{2}=2$ in (b)), the sampling positions are on
the grid points, thus feature aggregation can be directly performed. However,
when $r$ is a fractional (e.g., $r_{3}$), we need to compute the values of
sampling positions via bilinear interpolation.

####  III-A1 Perspective-guided dilation rate map

Let $\mathbf{x}$ and $\mathbf{s}$ be the feature maps and perspective map of
an image, respectively. Without loss of generality, $\mathbf{s}$ is
downsampled to the same size $H\times W$ with $\mathbf{x}$. Following [29],
$\mathbf{s}$ is firstly normalized to $[0,1]$ via a custom sigmoid-like
function $\zeta(\cdot)$

$\widetilde{\mathbf{s}}=\zeta\left(\mathbf{s}\right)=\frac{1}{1+e^{-\alpha\left(\mathbf{s}-\beta\right)}}$
(2)

where $\alpha$ and $\beta$ are two trainable parameters. Then the dilation
rate map is defined as

$\mathbf{r}=\max(~{}\gamma*\widetilde{\mathbf{s}}+\theta,~{}0~{})$ (3)

where $\gamma$ and $\theta$ are another two parameters that should be learned
in the training phase. Here, we adopt $\mathbf{r}$ as the dilation rate map
for fractional-dilation convolution.

####  III-A2 Fractional-dilation Convolution

A convolution filter with the kernel size $K\times K$ mainly accumulates the
sampled values weighted by the filter parameters $\mathbf{w}$, where the
sampled values are usually taken from a regular grid $\mathcal{R}$ on the
input feature maps $\mathbf{x}$. $\mathcal{R}$ enumerates all the integer
points within the range $[-\lfloor K/2\rfloor,\lfloor K/2\rfloor]$. In this
way, for any position $\mathbf{p}$ in the output feature map $\mathbf{y}$, its
feature representation can be defined as

$\mathbf{y}(\mathbf{p})=\sum_{\mathbf{p}_{k}\in\mathcal{R}}\mathbf{w}(\mathbf{p}_{k})\cdot\mathbf{x}(\mathbf{p}+\mathbf{p}_{k}),$
(4)

where $\mathbf{p}$ and $\mathbf{p}_{k}$ indicate the original coordinate and
offset coordinate, respectively. Similarly, in a dilated convolution with
dilation rate $r$, we have the following definition,

$\mathbf{y}(\mathbf{p})=\sum_{\mathbf{p}_{k}\in\mathcal{R}}\mathbf{w}(\mathbf{p}_{k})\cdot\mathbf{x}(\mathbf{p}+\mathbf{p}_{k}\cdot
r)$ (5)

where $r$ is a positive integer to make sure that the sampling positions are
on the grid points. Different from this normal definition, we propose
fractional-dilation convolution of which the dilation rate can be a fractional
number. Fig. 2 shows the differences between original convolution, dilated
convolution and fractional-dilation convolution.

To realize Eqn. (5) by using a fractional rate, we adopt bilinear
interpolation to calculate the feature representation of the point with the
fraction coordinate. As illustrated in Fig. 3, suppose the fraction coordinate
of $\mathbf{\hat{p}}$ is $(\hat{i},\hat{j})$, we firstly search the nearest
four grid points $(i,j)$, $(i,j\\!+\\!1)$, $(i\\!+\\!1,j)$ and
$(i\\!+\\!1,j\\!+\\!1)$ respectively. Then the bilinear interpolation is
performed along the horizontal direction to calculate,

$\begin{split}\mathbf{x}(i,\hat{j})&=\mu\cdot\mathbf{x}(i,j\\!+\\!1)+(1-\mu)\cdot\mathbf{x}(i,j)\\\
\mathbf{x}(i\\!+\\!1,\hat{j})&=\mu\cdot\mathbf{x}(i\\!+\\!1,j\\!+\\!1)+(1-\mu)\cdot\mathbf{x}(i\\!+\\!1,j)\\\
\end{split}$ (6)

where $\mu$ is determined by the location of $(i,j)$ and $(i,j\\!+\\!1)$.
Then, we compute $\mathbf{x}(\hat{i},\hat{j})$ along vertical direction.

$\begin{split}\mathbf{x}(\hat{i},\hat{j})&=\lambda\cdot\mathbf{x}(i\\!+\\!1,\hat{j})+(1-\lambda)\cdot\mathbf{x}(i,\hat{j})\\\
&=\lambda\cdot\left[~{}\mu\cdot\mathbf{x}(i\\!+\\!1,j\\!+\\!1)+(1-\mu)\cdot\mathbf{x}(i\\!+\\!1,j)~{}\right]\\\
&+(1-\lambda)\cdot\left[~{}\mu\cdot\mathbf{x}(i,j\\!+\\!1)+(1-\mu)\cdot\mathbf{x}(i,j)~{}\right]\end{split}$
(7)

From Fig. 3(a), we have:

$\lambda=1-|(i+1)-\hat{i}|\vspace{-0.6em}$ (8) $\mu=1-|(j+1)-\hat{j}|$ (9)

And the coefficients of grid points in Eqn. (7) can be represented by:

$\lambda\mu=(1-|(i+1)-\hat{i}|)\cdot(1-|(j+1)-\hat{j}|)$ (10)
$\lambda(1-\mu)=(1-|(i+1)-\hat{i}|)\cdot(1-|j-\hat{j}|)$ (11)
$(1-\lambda)\mu=(1-|i-\hat{i}|)\cdot(1-|(j+1)-\hat{j}|)$ (12)
$(1-\lambda)(1-\mu)=(1-|i-\hat{i}|)\cdot(1-|j-\hat{j}|)$ (13)

Let $\Omega(\mathbf{\hat{p}})$ be the set of the nearest four grid points of
$\mathbf{\hat{p}}$. $\mathbf{q}=(q_{i},q_{j})$ belongs to
$\Omega(\mathbf{\hat{p}})$. According to Eqn. (7) and Eqn. (10), (11), (12),
(13), we have the following simplified form,

$\begin{split}\mathbf{x}(\mathbf{\hat{p}})=\sum_{\mathbf{q}\in\Omega(\mathbf{\hat{p}})}(1-|q_{i}-p_{i}|)\cdot(1-|q_{j}-p_{j}|)\cdot\mathbf{x}(\mathbf{q}).\end{split}$
(14)

Then, we can further rewrite Eqn. (14) as follows,

$\mathbf{x}(\mathbf{\hat{p}})=\sum_{\mathbf{q}\in\Omega(\mathbf{\hat{p}})}G(\mathbf{q},\mathbf{\hat{p}})\cdot\mathbf{x}(\mathbf{q}),$
(15)

where $\mathbf{\hat{p}}$ denotes an arbitrary location. According to Eqn. (5),
we have $\mathbf{\hat{p}}=\mathbf{p}+\mathbf{p}_{k}\cdot r$. And $\mathbf{q}$
enumerates integral spatial locations in the feature map $\mathbf{x}$,
$G(\cdot,\cdot)$ denotes the bilinear interpolation kernel, which can be
written as,

$G(\mathbf{q},\mathbf{\hat{p}})=g(q_{i},p_{i})\cdot g(q_{j},p_{j}),$ (16)

where $g(m,n)=\max(0,1\\!-\\!|m\\!-\\!n|)$. According to Eqn. (16), the output
of such bilinear interpolation kernel is non-zero for only a few neighbor
points of $\mathbf{\hat{p}}$. So we can obtain the meaningful feature
representation of any fraction point.

\begin{overpic}[scale={0.118}]{./fig/bilinear_v2.pdf}
\par\put(10.0,34.5){\scalebox{0.5}{$\mu$}}
\put(4.0,32.0){\scalebox{0.5}{$\lambda$}}
\par\put(24.0,34.5){\scalebox{0.5}{$1-\mu$}}
\put(4.0,20.0){\scalebox{0.5}{$1-\lambda$}}
\par\put(0.5,38.0){\scalebox{0.5}{$(i,j)$}}
\put(26.0,38.0){\scalebox{0.5}{$(i,j+1)$}}
\put(0.5,5.0){\scalebox{0.5}{$(i+1,j)$}}
\put(23.5,5.0){\scalebox{0.5}{$(i+1,j+1)$}}
\par\put(16.0,38.0){\scalebox{0.5}{$(i,\hat{j})$}}
\put(14.0,5.0){\scalebox{0.5}{$(i+1,\hat{j})$}}
\put(17.0,26.0){\scalebox{0.5}{$(\hat{i},\hat{j})$}}
\par\put(49.5,28.0){\scalebox{0.7}{{\color[rgb]{0.125,0.69140625,0.015625}$\mathbf{p}_{k}$}}}
\par\put(81.0,28.0){\scalebox{0.7}{{\color[rgb]{0.125,0.69140625,0.015625}$\mathbf{p}_{k}$}}}
\par\end{overpic} Figure 3: Illustration of bilinear interpolation and
gradient w.r.t. $r$ in fractional-dilation convolution, where (a) shows the
procedure of bilinear interpolation and subfigure (b) demonstrates the
gradient with respect to $r$. In subfigure (a), when $r$ is a fractional
number, the sampling location $(\hat{i},\hat{j})$ is not on the grid point. We
need to search the nearest four grid points and then perform linear
interpolation in the horizontal direction to obtain $\mathbf{x}(i,\hat{j})$
and $\mathbf{x}(i+1,\hat{j})$. After that, we compute
$\mathbf{x}(\hat{i},\hat{j})$ via linear interpolation in the vertical
direction. In subfigure (b), the blue dashed arrows indicate the derivatives
with respect to sampling locations. The green arrows show the direction
vectors, and the red arrows represent the projected vectors of blue vectors
onto the corresponding green vectors.

####  III-A3 Back-propagation of Fractional-dilation Convolution

In our fractional-dilation convolution, we optimize the filter weights and the
dilation rate together. The gradient with respect to the dilation rate $r$ is
calculated by,

$\begin{split}&\frac{\partial{\mathbf{y}}(\mathbf{p})}{\partial{r}}=\sum_{\mathbf{p}_{k}\in\mathcal{R}}\mathbf{w}(\mathbf{p}_{k})\cdot\frac{\partial{\mathbf{x}(\mathbf{p}+\mathbf{p}_{k}\cdot
r)}}{\partial{r}}\\\
&=\sum_{\mathbf{p}_{k}\in\mathcal{R}}\left[\mathbf{w}(\mathbf{p}_{k})\cdot\sum_{\mathbf{q}\in\Omega(\mathbf{\hat{p}})}\frac{\partial{G}(\mathbf{q},\mathbf{p}+\mathbf{p}_{k}\cdot
r)}{\partial{r}}\cdot\mathbf{x}(\mathbf{q})\right]\\\
&=\sum_{\mathbf{p}_{k}\in\mathcal{R}}\left[\mathbf{w}(\mathbf{p}_{k})\cdot\mathbf{p}_{k}\cdot\sum_{\mathbf{q}\in\Omega(\mathbf{\hat{p}})}\frac{\partial{G}(\mathbf{q},\mathbf{p}+\mathbf{p}_{k}\cdot
r)}{\partial{(\mathbf{p}+\mathbf{p}_{k}\cdot
r)}}\cdot\mathbf{x}(\mathbf{q})\right]\end{split}$ (17)

In fact,
$\sum_{\mathbf{q}\in\Omega(\hat{\mathbf{p}})}\frac{\partial{G}(\mathbf{q},\mathbf{p}+\mathbf{p}_{k}\cdot
r)}{\partial{(\mathbf{p}+\mathbf{p}_{k}\cdot r)}}\cdot\mathbf{x}(\mathbf{q})$
is 2D and denotes the derivative of bilinear interpolation with respect to the
sampling location $\mathbf{p}+\mathbf{p}_{k}\cdot r$ (i.e.,
$\hat{\mathbf{p}}$). We use the brief expression of the gradient with respect
to sampling location $\mathbf{p}+\mathbf{p}_{k}\cdot r$ to represent
$\sum_{\mathbf{q}\in\Omega(\hat{\mathbf{p}})}\frac{\partial{G}(\mathbf{q},\mathbf{p}+\mathbf{p}_{k}\cdot
r)}{\partial{(\mathbf{p}+\mathbf{p}_{k}\cdot r)}}\cdot\mathbf{x}(\mathbf{q})$.
And $\mathbf{p}_{k}$ can represent a direction vector that starts from point
$\mathbf{p}$ and ends at point $\mathbf{p}+\mathbf{p}_{k}$. Intuitively, the
formulation can be figuratively explained as Fig. 3(b). Here,
$\mathbf{p}_{k}\cdot\sum_{\mathbf{q}\in\Omega(\hat{\mathbf{p}})}\frac{\partial{G}(\mathbf{q},\mathbf{p}+\mathbf{p}_{k}\cdot
r)}{\partial{(\mathbf{p}+\mathbf{p}_{k}\cdot r)}}\cdot\mathbf{x}(\mathbf{q})$
can be regarded as the result when the derivatives with respect to sampling
locations are projected onto the corresponding direction vectors, shown as the
red arrows in Fig. 3. If the result is positive/negative, the red arrow is in
the same/opposite direction of the corresponding direction vector
$\mathbf{p}_{k}$ and it encourages an expanding/shrinking of the dilation rate
$r$. Afterwards, the projected results are then weighted by the corresponding
grid weight $\mathbf{w}(\mathbf{p}_{k})$ and finally summed up as the final
decision on scaling dilation rate $r$. After obtaining the gradient of the
dilation rate $r$ for each position, we could further calculate the parameters
in Eqn. (2) and Eqn. (3) according to gradient chain rule.

\begin{overpic}[scale={0.235}]{./fig/pfc_vs_pgc/pfc_vs_pgc.pdf}
\par\par\end{overpic} Figure 4: Comparison on sampling positions of our
proposed PFC and PGC [31]. The dashed black lines show the grids and the
intersections are grid points. Suppose we want to perform feature aggregation
on a square region marked in purple. The region can be exactly covered by a
fractional dilation convolution with dilation rate $r=1.5$. The input feature
map is termed as $\mathbf{x}$. (a) and (b) respectively show the procedure of
PFC and PGC. In (a), the sampling positions are marked with small balls and
most of them are not on the grid points. Taken the upper-left red sampling
position and the upper-right green sampling position for example, their values
are estimated by the bilinear interpolation with the values (from
$\mathbf{x}$) of two groups, four grid points per group. _Obviously the two
groups of points do not overlap with each other._ Finally, the feature
aggregation is performed on the values of the small balls. In (b), when
conducting PGC on the region, we need to perform feature aggregation with the
values (from the blurred feature $\mathbf{\widetilde{x}}$, instead of directly
from input feature $\mathbf{x}$) of the grid points (i.e., here is $9$ big
balls in the purple square). The value of red ball is obtained by performing
Gaussian smoothing (with kernel size $3\times 3$ and $\sigma=\frac{3}{4}$) on
the values (from $\mathbf{x}$) in the region of red square. Similarly, the
value of green ball can be computed. The procedure is named as spatially
variant Gaussian smoothing, and the blurred feature is represented as
$\mathbf{\widetilde{x}}$. One can see that _large overlap (colored in yellow)
exists between the sampling points when performing Gaussian smoothing._
Furthermore, larger overlaps exist when the Gaussian kernel size goes
bigger(e.g., it reaches $7\times 7$ in [31]), resulting in severe feature
confusion.  \begin{overpic}[scale={0.11}]{./fig/net_arch/PFC_v7.pdf}
\put(48.5,0.55){\scalebox{0.8}{{\color[rgb]{0,0,0}$\mathbf{r}$}}}
\put(47.0,12.1){\scalebox{0.8}{{\color[rgb]{0,0,0}$\mathbf{x}$}}}
\par\end{overpic} Figure 5: The architecture of the proposed PFDNet. PFNet
aims to predict reasonable dilation rate map for each input image and then PFC
takes the backbone features and estimated dilation rate map to perform
spatially variant receptive field allocation. Six PFCs are embedded in our
PFDNet. The perspective map estimated by PENet is normalized via a custom
sigmoid function to generate the dilation rate map. Afterwards, the sampled
locations $\mathbf{p}+\mathbf{p}_{k}\cdot r$ for each point $\mathbf{p}$ on
the feature map is obtained.

####  III-A4 Differences to PGC [31]

We firstly give a brief introduction to PGC. PGC [31] generates
$\widetilde{s}$ via Eqn. (2), and then the blur map $\boldsymbol{\sigma}$ is
obtained via linear mapping of $\widetilde{s}$, defined as

$\boldsymbol{\sigma}=\max(a(\widetilde{s}-s_{0}),0),$ (18)

where $a$ and $s_{0}$ are trainable parameters. With spatially variant
Gaussian smoothing, the smoothing result $\widetilde{x}_{i,j}$ is computed by

$\widetilde{x}_{i,j}=\sum_{k}\sum_{l}x_{k,l}G_{\sigma_{i,j}}(i,j,k,l),$ (19)

where $G_{\sigma_{i,j}}(i,j,k,l)$ is defined as below:

$\small
G_{\sigma_{i,j}}(i,j,k,l)\\!=\\!\frac{1}{\sqrt{2\pi}\sigma_{i,j}}\exp\left(-\frac{\left((k\\!-\\!i)^{2}\\!+\\!(l\\!-\\!j)^{2}\right)}{2\sigma_{i,j}^{2}}\right).$
(20)

And the perspective-guided convolution is formulated as,

$\mathbf{y}=\mathbf{W}^{T}\widetilde{\mathbf{x}},$ (21)

where $\mathbf{W}$ denotes the conventional convolution kernel.

From Fig. 4, it can be seen that PGC suffers from severe feature confusion due
to large overlaps of sampling positions. Intuitively, when the size of
Gaussian kernel goes larger (e.g., in PGC [31], the maximum Gaussian kernel
size is $7\times 7$), larger overlap exists between the regions of Gaussian
smoothing, which inevitably limits the performance of PGC. By contrast, each
sampling position of PFC is computed via bilinear interpolation of the nearest
four grid points, and the groups of grid points hardly overlap with each
other. Therefore, PFC is effective in avoiding feature confusion. Generally
speaking, from the discussion in Sec. II-E, our proposed PFC runs much faster,
more flexible on allocating larger receptive field, can avoid feature
confusion, and thus achieves better performance.

###  III-B Perspective Estimation

Although perspective maps are critical for calculating the dilation rate of
fractional-dilation convolution, they are not always available in real
applications. To address such an issue, we introduce a perspective estimation
branch to predict corresponding perspective information under the above
situation. Intuitively, one can pre-train a perspective estimation network on
available pairs of image and perspective map, and then adopt it as the
perspective estimation branch of the whole network. However, we find a plain
network optimized with the end-to-end manner usually achieves an unacceptable
prediction accuracy. Inspired by [55], we propose a three-phase training
strategy to train a robust auto-encoder for perspective estimation.

In the first phase, we use perspective maps from ShanghaiTech A [4] to train
an auto-encoder $D_{s}(E_{s}(\mathbf{s};\Theta^{E}_{s});\Theta^{D}_{s})$ as a
perspective re-constructor for each input $\mathbf{s}$. Here, $\Theta^{E}_{s}$
and $\Theta^{D}_{s}$ denotes the parameters of the encoder $E_{s}$ and decoder
$D_{s}$, respectively. We adopt the $\ell_{2}$ reconstruction loss as the
objective function,

${\cal
L}_{s2s}=\frac{1}{2N}\sum_{i=1}^{N}\|D_{s}(E_{s}(\mathbf{s}_{i};\Theta^{E}_{s});\Theta^{D}_{s})-\mathbf{s}_{i}\|_{2}^{2},$
(22)

where $N$ is the total number of perspective maps. Once the model is trained,
the latent code $E_{s}(\mathbf{s};\Theta^{E}_{s})$ could encode the internal
structure and contextual relationships in $\mathbf{s}$, while the decoder
$D_{s}$ could accurately recover a high quality perspective map from this
latent code.

In the second phase, we train another auto-encoder which takes an image
$\mathbf{I}$ as the input and predicts the corresponding perspective map
$D_{s}(E_{I}(\mathbf{I};\Theta^{E}_{I});\Theta^{D}_{s})$. Specially, we reuse
the decoder parameters $\Theta^{D}_{s}$ learned in the first phase, and only
optimize the encoder parameters $\Theta^{E}_{I}$ by minimizing the following
loss function,

${\cal
L}_{I2s}=\frac{1}{2N}\sum_{i=1}^{N}\|D_{s}(E_{I}({I}_{i};\Theta^{E}_{I});\Theta^{D}_{s})-\mathbf{s}_{i}\|_{2}^{2}.$
(23)

In the third phase, we transfer the optimized auto-encoder in the second phase
onto the target training set. The decoder is fixed during the training phase.
If the perspective maps are available on the target training set, we could
optimize both ${\cal L}_{I2s}$ and density estimation loss (i.e., the main
loss of the crowd counting task) together. If the perspective maps are
unavailable, we could train the perspective estimation branch together with
the main backbone network with an end-to-end manner. In practice, benefited
from the robustness of the decoder, even if the encoder is not well optimized,
the decoder can still recover a roughly reasonable perspective map. According
to the above three training phases, our perspective estimation branch can be
optimized in a weakly-supervised way even without using the corresponding
perspective annotations.

###  III-C Network Architecture

Fig. 5 illustrates the architecture of our PFDNet, which is comprised of the
main backbone network together with the perspective estimation network
(PENet). We adopt CSRNet [30] as our main backbone network, and replace all of
the regular dilated convolutions with our proposed PFCs. The dilation rate map
for the PFC are calculated based on the output of PENet according to the Eqn.
(2) and Eqn. (3). The detailed PENet architecture is illustrated in Table I.

TABLE I: The architecture of PENet. “LReLU” denotes leaky ReLU with the slope
of 0.2. “UpConv” represents transposed convolution. $H$ and $W$ are the height
and width of input respectively.

Part | Layer | Output Size
---|---|---
Encoder | Conv. (3, 3, 32), stride=2; LReLU | $32\times H/2\times W/2$
Conv. (3, 32, 64), stride=2; LReLU | $64\times H/4\times W/4$
Conv. (3, 64, 128), stride=2; LReLU | $128\times H/8\times W/8$
Conv. (3, 128, 256), stride=2; LReLU | $256\times H/16\times W/16$
Conv. (3, 256, 512), stride=2; LReLU | $512\times H/32\times W/32$
Conv. (3, 512, 1024), stride=2; LReLU | $1024\times H/64\times W/64$
| UpConv. (3, 1024, 512), stride=2; ReLU | $512\times H/32\times W/32$
Decoder | UpConv. (3, 512, 256), stride=2; ReLU | $256\times H/16\times W/16$
UpConv. (3, 256, 128), stride=2; ReLU | $128\times H/8\times W/8$
UpConv. (3, 128, 64), stride=2; ReLU | $64\times H/4\times W/4$
UpConv. (3, 64, 32), stride=2; ReLU | $32\times H/2\times W/2$
UpConv. (3, 3, 1), stride=2; ReLU | $1\times H\times W$

##  IV Experimental Results and Dataset

In this section, we will first give the description of the implementation
details, and then present the comparison between state-of-the-arts and our
PFDNet on five datasets, namely ShanghaiTech [4], WorldExpo’10 [23], UCF-QNRF
[56], UCF_CC_50 [17] and TRANCOS [57]. Extensive ablation study is then
conducted to clarify the contribution of each component in PFDNet. The pre-
trained models and source code are online available at
https://github.com/Zhaoyi-Yan/PFDNet.

###  IV-A Evaluation Metric and Dataset

We adopt Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) as the
evaluation metrics, the same as previous works [4, 5, 30]. Let $N$ be the
number of test samples, $C_{i}$ and $\hat{C}_{i}$ are respectively the ground-
truth count and prediction one. Then MAE and RMSE can be defined as follows:

$MAE=\frac{1}{N}\sum_{i=1}^{N}|C_{i}-\hat{C}_{i}|,$ (24)

$RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\|C_{i}-\hat{C}_{i}\|^{2}}.$ (25)

We evaluate the performance of the proposed PFDNet and other state-of-the-art
methods on five popular benchmarks. The detailed descriptions are as follows.

ShanghaiTech contains $1,198$ images with a total of $330,165$ annotated
people. ShanghaiTech A contains $482$ crowd images with crowd numbers varying
from $33$ to $3139$, and ShanghaiTech B contains $716$ high-resolution images
with crowd numbers from $9$ to $578$. For Part A, the images are collected
from the web, while the images in Part B are Shanghai street views.

WorldExpo’10 contains $3,980$ images from the $2010$ Shanghai WorldExpo. The
training set contains $3,380$ images, while the test set is divided into five
different scenes with $120$ images each. ROIs are provided to indicate the
target regions for training / testing. Following [30], each image and its
ground-truth density map are masked with the ROI in preprocessing.

UCF-QNRF is a large crowd counting dataset with $1,535$ high-resolution images
and $1.25$ million head annotations. There are $1,201$ training images and the
rest are used as test images. It contains extremely congested scenes where the
maximum count of an image $12,865$.

UCF_CC_50 contains $50$ images of diverse scenes, which is a challenging
dataset. The head count per image varies drastically (from $94$ to $4,543$).
Following [17], we split the dataset into five subsets and perform a $5$-fold
cross-validation.

TRANCOS is a public traffic dataset containing $1,244$ images of different
congested traffic scenes captured by surveillance cameras with $46,796$
annotated vehicles. The regions of interest (ROI) are provided for evaluation.
We evaluate PFDNet on TRANCOS dataset to validate the generalization of our
method and the robustness of PENet. The Grid Average Absolute Error (GAME) is
adopted as the evaluation metric, which can be referred in [57, 30]. GAME(L)
subdivides the image using a grid of $4^{L}$ non-overlapping regions, and the
error is computed as the sum of the mean absolute errors in each of these
regions. Obviously, GAME(0) is equivalent to the MAE metric.

###  IV-B Implementation Details

Let $\Theta$ denote the model parameters of PFDNet $\Phi(I;\Theta)$. Then the
main loss function of the crowd counting estimation task is as follows,

${\cal
L}\left(\Theta\right)=\frac{1}{2N}\sum_{i=1}^{N}\|\Phi(I;\Theta)-Y_{i}\|_{2}^{2},$
(26)

where $Y_{i}$ is the ground-truth density map of $I_{i}$. We generate ground-
truth density maps by adopting a normalized Gaussian kernel to blur each head
annotation, and make sure the summations equals to the crowd counts. In our
experiments, we use a fixed $15\\!\times\\!15$ Gaussian kernel with standard
deviation of $4$ to generate density maps.

We adopt Adam [58] as the optimizer. The learning rate is set as $1e^{-4}$ for
the parameters of backbone network. Original CSRNet perfers one whole image in
an iteration when training, consuming a lot of time for some datasets (e.g.,
ShanghaiTech A which contains images of various sizes). To make the training
procedure more efficient, we adopt VGG16-BN [54] model instead of VGG-16 as
the backbone, and add extra BatchNorm [59] layers after each dilated
convolution in CSRNet. BatchNorm layer stabilizes the model when training with
random cropping images and gets rid of performance degradation when
evaluating. Besides, we upsample the predicted density map by the factor 4,
making the final predicted density map $\frac{1}{2}$ of the original input
size. After all the changes, we call it CSRNet* and take it as the baseline of
our method. It should be noted that CSRNet* shows its superiority in faster
training (i.e., batch size can be larger than 1, and is 16 in our
experiments.) and higher performances than original CSRNet on the validation
datasets. The weights of PFDNet are initialized with Gaussian distribution of
zero mean and 0.01 standard deviation, except the weights of backbone are
initialized with the VGG16-BN pretrained on ImageNet. We implement our PFDNet
on Pytorch [60]. Random flipping and color jittering are adopted for data
augmentation.

For the initialization of $\alpha$, $\beta$, $\gamma$ and $\theta$, according
to CSRNet [30], the average dilated rate over the whole map is expected to be
around $2$. Empirically, we set $\alpha$, $\beta$, $\gamma$ and $\theta$
respectively with value $1$, $1$, $1.5$, $1$ for all the PFC layers. For
simplicity, we adopt the same initialization values for these four super-
parameters on all the datasets.

For ShanghaiTech and WorldExpo’10, since the corresponding ground-truth
perspective maps are available, they are used as the guidance of PFC directly,
and our model is trained for $300$ epochs without PENet. For the datasets
without perspective annotations, the PENet is pre-trained for the first two
phases of Sec. III-B, both using $500$ epochs. After that the main network and
PENet are jointly trained for another $300$ epochs, following the description
of the third phase of Sec. III-B. In practice, we build PFDNet by stacking six
PFC layers after a truncated VGG16-BN for all datasets. It takes about $2$
days to train the network on UCF-QNRF with an NVIDIA Tesla P100 GPU.

###  IV-C Evaluations and Results

Five datasets are adopted in our experiments, including ShanghaiTech [4],
WorldExpo’10 [23], UCF-QNRF [56], UCF_CC_50 [17] and TRANCOS [57]. It is noted
that the first two contain available perspective maps111Annotations of
ShanghaiTech are from [29] and WorldExpo’10 provides official annotations.,
thus we can directly use these annotations to train our PFDNet. For the later
three datasets that perspective map annotations are unavailable, we denote
_Ours A_ as directly adopting the estimated perspective maps (based on the
PENet trained on perspective annotations from ShanghaiTech A in the first two
phases) as _ground-truth_ and feed them to the training of PFC layers; while
_Ours B_ as the end-to-end training without perspective map annotations in the
third training phase described in Sec. III-B, where the main network and PENet
are jointly trained without any perspective map annotations. As the resolution
of perspective map required by PFCs is only $1/8$ size of the original image,
thus all input images are firstly downsampled to $1/8$ resolution of the
original size in the two training phases of PENet.

For _Ours A_ / _Ours B_ , by following Sec. III-B with ShanghaiTech A, we get
$30.6$ dB PSNR in the first phase and $25.1$ dB PSNR in the second phase.
Besides, Fig. 6 shows a test sample for visualization of the perspective
estimation. Fig. 6(b)(d) shows the quite similar estimation results, which
indicates the auto-encoder is well trained in the first phase. In the second
phase, the auto-encoder aims at predicting perspective estimations (e.g., Fig.
6(c)) from RGB images. It is observed that Fig. 6(c) is still visually-
satisfying, indicating that the PENet could actually recover good perspective
annotations from a RGB image.

Figure 6: Perspective estimations of PENet in the first two phases. (a) is the
input; (b)(c) are respectively the predictions in the first and second phase;
(d) is the ground-truth perspective annotations. TABLE II: Comparisons on
ShanghaiTech A, ShanghaiTech B and WorldExpo’10. It is noted that we adopt
ground-truth perspective annotations to feed PFCs. The results of top two
performance are highlighted in red and blue respectively.

Method |  Year |  Venue | ShanghaiTech A | ShanghaiTech B | WorldExpo’10
---|---|---|---|---|---
|  |  MAE |  RMSE |  MAE |  RMSE |  Sce.1 |  Sce.2 |  Sce.3 |  Sce.4 |  Sce.5 |  Avg
CSRNet [30] |  2018 |  CVPR |  68.2 |  115.0 |  10.6 |  16.0 |  2.9 |  11.5 |  8.6 |  16.6 |  3.4 |  8.6
SANet [50] |  2018 |  CVPR |  67.0 |  104.5 |  8.4 |  13.6 |  2.6 |  13.2 |  9.0 |  13.3 |  3.0 |  8.2
HA-CCN [33] |  2019 |  TIP |  62.9 |  94.9 |  8.1 |  13.4 |  - |  - |  - |  - |  - |  -
TEDnet [61] |  2019 |  CVPR |  64.2 |  109.1 |  8.2 |  12.8 |  2.3 |  10.1 |  11.3 |  13.8 |  2.6 |  8.0
ADCrowdNet [32] |  2019 |  CVPR |  63.2 |  98.9 |  8.2 |  15.7 |  1.6 |  13.2 |  8.7 |  10.6 |  2.6 |  7.3
SFCN [62] |  2019 |  CVPR |  65.2 |  109.4 |  7.2 |  12.2 |  - |  - |  - |  - |  - |  -
PACNN [29] |  2019 |  CVPR |  62.4 |  102.0 |  7.6 |  11.8 |  2.3 |  12.5 |  9.1 |  11.2 |  3.8 |  7.8
CAN [63] |  2019 |  CVPR |  62.3 |  100.0 |  7.2 |  11.1 |  2.9 |  12.0 |  10.0 |  7.9 |  4.3 |  7.4
PSDDN [48] |  2019 |  CVPR |  65.9 |  112.3 |  9.1 |  14.2 |  - |  - |  - |  - |  - |  -
SPN [64] |  2019 |  ICCV |  64.2 |  98.4 |  7.2 |  11.1 |  - |  - |  - |  - |  - |  -
BL [63] |  2019 |  ICCV |  62.3 |  100.0 |  7.2 |  11.1 |  - |  - |  - |  - |  - |  -
S-DCNet [40] |  2019 |  ICCV |  58.3 |  95.0 |  6.7 |  10.7 |  - |  - |  - |  - |  - |  -
PGCNet [31] |  2019 |  ICCV |  57.0 |  86.0 |  8.8 |  13.7 |  2.5 |  12.7 |  8.4 |  13.7 |  3.2 |  8.1
DensityCNN [65] |  2020 |  TMM |  63.1 |  106.3 |  9.1 |  16.3 |  1.8 |  11.2 |  10.1 |  7.9 |  3.4 |  6.9
DENet [66] |  2020 |  TMM |  65.5 |  101.2 |  9.6 |  15.4 |  2.8 |  10.7 |  8.6 |  15.2 |  3.5 |  8.2
EPA [67] |  2020 |  TIP |  60.9 |  91.6 |  7.9 |  11.6 |  2.1 |  9.3 |  9.4 |  11.0 |  3.7 |  7.4
KDMG [68] |  2020 |  TPAMI |  63.8 |  99.2 |  7.8 |  12.7 |  - |  - |  - |  - |  - |  -
LSC-CNN [41] |  2020 |  TPAMI |  66.4 |  117.0 |  8.1 |  12.7 |  2.9 |  11.3 |  9.4 |  12.3 |  4.3 |  8.0
LCM [42] |  2020 |  ECCV |  61.6 |  98.4 |  7.0 |  11.0 |  - |  - |  - |  - |  - |  -
AMSNet [52] |  2020 |  ECCV |  56.7 |  93.4 |  6.7 |  10.2 |  1.6 |  8.8 |  10.8 |  10.4 |  2.5 |  6.8
DM-Count [39] |  2020 |  NeurIPS |  61.9 |  99.6 |  7.4 |  11.8 |  - |  - |  - |  - |  - |  -
Baseline |  - |  - |  65.1 |  112.4 |  9.8 |  15.2 |  2.7 |  11.1 |  8.5 |  16.7 |  3.6 |  8.5
Ours |  - |  - |  53.8 |  89.2 |  6.5 |  10.7 |  2.5 |  9.7 |  8.1 |  10.1 |  3.5 |  6.8

TABLE III: Comparisons on UCF-QNRF, UCF_CC_50 and TRANCOS. Baseline is the
model CSRNet*, and Ours A means we directly use the estimated perspective maps
predicted by PENet. While Ours B indicates that PENet embeded as a branch and
the whole training is end-to-end. The results of top two performance are
highlighted in red and blue respectively.

Method |  Year |  Venue | UCF-QNRF | UCF_CC_50 | TRANCOS
---|---|---|---|---|---
|  |  MAE |  RMSE |  MAE |  RMSE |  GE(0) |  GE(1) |  GE(2) |  GE(3)
CSRNet [30] |  2018 |  CVPR |  - |  - |  266.1 |  397.5 |  3.56 |  5.49 |  8.57 |  15.04
SANet [50] |  2018 |  CVPR |  - |  - |  258.4 |  334.9 |  - |  - |  - |  -
HA-CCN [33] |  2019 |  TIP |  118.1 |  180.4 |  256.2 |  348.4 |  - |  - |  - |  -
TEDnet [61] |  2019 |  CVPR |  113 |  188 |  294.4 |  354.5 |  - |  - |  - |  -
ADCrowdNet [32] |  2019 |  CVPR |  - |  - |  257.1 |  363.5 |  - |  - |  - |  -
SFCN [62] |  2019 |  CVPR |  102.0 |  171.4 |  214.2 |  318.2 |  - |  - |  - |  -
PACNN [29] |  2019 |  CVPR |  - |  - |  241.7 |  320.7 |  - |  - |  - |  -
CAN [63] |  2019 |  CVPR |  107 |  183 |  212.2 |  243.7 |  - |  - |  - |  -
PSDDN [48] |  2019 |  CVPR |  - |  - |  359.4 |  514.8 |  4.79 |  5.43 |  6.68 |  8.40
SPN [64] |  2019 |  ICCV |  104.7 |  173.6 |  259.2 |  335.9 |  - |  - |  - |  -
BL [63] |  2019 |  ICCV |  88.7 |  154.8 |  229.3 |  308.2 |  - |  - |  - |  -
S-DCNet [40] |  2019 |  ICCV |  104.4 |  176.1 |  204.2 |  301.3 |  - |  - |  - |  -
PGCNet [31] |  2019 |  ICCV |  - |  - |  244.6 |  361.2 |  - |  - |  - |  -
DensityCNN [65] |  2020 |  TMM |  101.5 |  186.9 |  244.6 |  341.8 |  3.17 |  4.78 |  6.30 |  8.26
DENet [66] |  2020 |  TMM |  - |  - |  241.9 |  345.4 |  - |  - |  - |  -
LSC-CNN [41] |  2020 |  TPAMI |  120.5 |  218.2 |  225.6 |  302.7 |  4.6 |  5.4 |  6.9 |  8.3
EPA [67] |  2020 |  TIP |  - |  - |  250.1 |  342.1 |  3.21 |  - |  - |  -
KDMG [68] |  2020 |  TPAMI |  99.5 |  173.0 |  - |  - |  3.13 |  4.79 |  6.20 |  8.68
LCM [42] |  2020 |  ECCV |  86.6 |  152.2 |  184.0 |  265.8 |  - |  - |  - |  -
AMSNet [52] |  2020 |  ECCV |  101.8 |  163.2 |  208.4 |  297.3 |  - |  - |  - |  -
DM-Count [39] |  2020 |  NeurIPS |  85.6 |  148.3 |  211.0 |  291.5 |  - |  - |  - |  -
Baseline |  - |  - |  103.1 |  188.6 |  257.1 |  363.8 |  3.49 |  5.30 |  8.16 |  14.45
Ours A |  - |  - |  87.5 |  152.4 |  224.6 |  304.4 |  3.22 |  4.58 |  6.74 |  10.69
Ours B |  - |  - |  84.3 |  141.2 |  205.8 |  289.3 |  3.06 |  4.15 |  6.22 |  9.86

Table II shows the results on ShanghaiTech and WorldExpo’10, in which we feed
ground-truth perspective annotations to the PFCs. It is seen that our method
shows significant performance gain against the baseline CSRNet* with $17.4\%$
relative MAE decrease on ShanghaiTech A and $33.7\%$ relative MAE decrease on
ShanghaiTech B. For WorldExpo’10 dataset, our method achieves the best $6.8$
average MAE with $20.0\%$ performance improvement. For all these three
datasets, our method performs the best, indicating the superiority of
spatially variant receptive field allocation. Besides, some test cases can be
found in Fig. 7, clearly indicating that the superiority of PFDNet to CSRNet*
in estimating a better density map.

For the datasets UCF-QNRF, UCF_CC_50 and TRANCOS, they do not provide official
perspective annotations. Thus, we conduct the two comparison experiments _Ours
A_ and _Ours B_ described at the beginning of this section. Table III shows
the results of these three datasets. Compared with the baseline CSRNet*, _Ours
A_ achieves significant gain on both MAE and RMSE. Specifically, $15.7\%$,
$18.1\%$ performance gains on MAEs for datasets UCF-QNRF and UCF_CC_50,
achieving $87.5$ and $224.5$ MAE. Notably, _Ours B_ further reduces the MAEs,
ranks first on UCF-QNRF and achieves the second best performance on UCF_CC_50,
which shows the feasibility of our end-to-end training strategy described in
Sec. III-B.

Furthermore, we emphasize that PFDNet still delivers obvious performance gain
even in object counting over baseline CSRNet* and achieves $3.06$ MAE. It is
noted that perspective estimation aims to predict the depth information of a
scene, which is object-unrelated. We show a test sample in Fig. 8. For both
phases Ours A and Ours B, PENet generally creates fine perspective
estimations, which is the guarantee of fine performance. It is generally known
that the perspective map is roughly of horizontal-linear-growth and vertical-
value-consistent for a plain scene [29]. Fig. 8(c) outperforms Fig. 8(b) in
its higher accordance with the prior, and Fig. 8(f) outperforms Fig. 8(e) in
predicting more accurate density maps. These observations validate the
robustness and feasibility of our method.

###  IV-D Comparisons with state-of-the-arts

Among the existing deep crowd counting approaches, we select three state-of-
the-art methods (i.e., LCM [42], AMSNet [52], DM-Count [39]) which show the
best completing performance against our PFDNet.

(1) LCM [42] adopts NAS (Neural Architecture Search) to automatically develop
a multi-scale architecture to address the scale variation in crowd counting.
However, the operations in the searching space are conventional network
components, thus all these operations suffer from the intrinsic limitation in
modeling continuous scale variation.

(2) AMSNet [52] attempts to model continuous scales by stacking multiple
manually-designed complicated and computational-heavy modules (i.e., Scale-
aware Module, Mixture Regression Module and Adaptive Soft Interval Module),
largely increasing the inference time (i.e. $\sim$$1.72s$ for a
1024$\times$768 image). In contrast, our PFDNet only takes $\sim 70ms$.

(3) DM-Count [39] proposes optimal transport loss to replace MSE-based loss as
the density map prediction constraint, so as to tackle the scale variation to
a certain extent. Nonetheless, the network (i.e., VGG-19 [54]) adopted in DM-
Count is built with conventional operators. Therefore, continuous scales of
features are compressed into discrete scales of feature representations,
resulting in irreparable loss of information. Actually, a better approach
would be explicitly modeling the scale variation via our proposed PFD layer,
which can be validated from the performance comparisons.

Figure 7: Density maps estimated by CSRNet* and Ours. It is seen that our
PFDNet delivers superior performance over CSRNet* in both estimating better
density maps and more accurate counts. Figure 8: Visualization of a test
example from TRANCOS dataset. (a) is the input; (b)(c) / (e)(f) are estimated
perspective / density maps of Ours A and Ours B; and (d) is the ground truth.

###  IV-E Ablation Study

In this section, we will first demonstrate the influence of the number of PFCs
stacked in our network. Besides, we validate the feasibility of PFC as an
insertable component for existing network to improve performance. Moreover, we
exploit the necessity of spatially variant receptive fields. Furthermore, we
visualize the dilation rate maps of different PFC layers. In addition, we
compare our PFC with deformable convolution to have a deeper understanding of
continuous scale modeling. Apart from these, we also verify the importance of
the pre-training of PENet in our method. Finally, we confirm the reliability
of the prediction of PENet.

####  IV-E1 Influence of the Number of PFCs

Table IV shows the performance of our network when replacing dilated
convolutions in CSRNet* with our proposed PFCs, as Fig. 5 shows. We know that
CSRNet* is built upon VGG16-BN backbone with six dilated convolutions stacked
afterward, and finally ends with a normal convolution to output density map
prediction. We validate the effectiveness of our method by replacing dilated
convolution in CSRNet* with PFC layer by layer, in the order from top to
bottom. The performance increases with the number of applied PFCs, from 65.1
and 9.8 MAE as the baseline to peak values 53.8 and 6.5 MAE with six PFCs on
Part A and Part B, respectively. Fig. 9 demonstrates the predicted density
maps when applied several PFCs. It is seen that density maps are gradually
turning to be more accurate on either small scales or large ones.

Here, we need to make two clarifications.

i) Better performance may be achieved when extra PFCs are applied and fine
parameter-tuning is adopted, however, exploring additional PFCs is unfair in
comparison with the original CSRNet*. Therefore, we do not exploit more PFCs.

ii) Intuitively, our PFC is also able to substitute normal convolution layer
which can be regarded as a special type of dilated convolution with dilation
rate $1$. Whereas, when we replace all convolutions in CSRNet* with our PFCs,
we do not obtain continuous performance improvements over different datasets.
To be precise, performances of ShanghaiTech A / B turn out to be $55.4$ /
$7.1$ MAE. There exist two main reasons for the degradation. Firstly, we do
not conduct experiments on substituting normal convolutions layer by layer,
which requires massive computation. Thus, the optimal number of normal
convolutions to be replaced with PFCs are not fully searched. Secondly, we do
not train our PFDNet in an incremental scheme, as it is quite complicated,
though such training strategy usually leads to better performance. Instead,
the parameters in PFCs are initialized with the same setting and backbone is
initialized with VGG16-BN as described in Sec. IV-B. Based on these
observations, we construct our final network as the architecture that all six
dilated convolutions in CSRNet* are replaced with our proposed PFCs, resulting
in $70$ ms total time cost for a $1024\times 768$ image in testing.

TABLE IV: Influence of the number of stacked PFCs in our method on
ShanghaiTech dataset.

# of PFCs | Part A | Part B
---|---|---
MAE | RMSE | MAE | RMSE
0 | 65.1 | 112.4 | 9.8 | 15.2
1 | 63.3 | 106.6 | 8.6 | 13.9
2 | 60.4 | 100.3 | 7.9 | 13.3
3 | 57.8 | 96.8 | 7.5 | 12.4
4 | 56.2 | 92.1 | 7.1 | 11.5
5 | 54.4 | 90.5 | 6.7 | 11.1
6 | 53.8 | 89.2 | 6.5 | 10.7

Figure 9: Density maps predicted by replacing different numbers of dilated
convolutions with our PFCs. CSRNet* is the baseline and is equivalent to the
situation of ‘#PFC = 0’.

####  IV-E2 Extensibility to Another Backbone

In order to verify the extensibility of our PFC, we build a new baseline that
consists of a truncated ResNet-50 [69] with the first $22$ convolutional
layers, four dilated convolutions (with dilation rate $2$ and $512$, $256$,
$128$, $64$ channels respectively) and a normal convolution that aims to
reduce the channel dimension to $1$. Table V shows the comparison results on
ShanghaiTech Part A / B, where _ResNet-50 (backbone)_ is the backbone itself,
and _ResNet-50 (partly PFC)_ only replaces the four dilated convolutions with
our PFCs and _ResNet-50 (fully PFC)_ replaces all convolutions with PFCs. All
models are trained for $300$ epochs with the first $22$ convolutional layers
initialized by pre-trained weights on ImageNet [70]. For _ResNet-50
(backbone)_ , we get $89.5/18.6$ MAE on ShanghaiTech Part A / B. When we
replace all dilated convolutions with PFCs, a significant performance gain of
$13.1/5.9$ MAE on ShanghaiTech Part A / B. Besides, if all normal convolutions
are also be replaced by PFCs, we observe further gain of $2.3/1.5$ MAE. This
indicates that PFC is not only a superior alternative to dilation convolution
and even has the potential to substitute normal convolution. Such observations
validate the effectiveness and extensibility of the PFC on another backbone.

TABLE V: Comparisons on the ResNet-50 backbone.

Method | Part A | Part B
---|---|---
MAE | RMSE | MAE | RMSE
ResNet-50 (backbone) | 89.5 | 163.2 | 18.6 | 26.6
ResNet-50 (partly PFC) | 76.4 | 145.3 | 12.7 | 19.2
ResNet-50 (fully PFC) | 74.1 | 140.5 | 11.2 | 16.8

####  IV-E3 Necessity of Spatially Variant Receptive Fields

In this section, we conduct an experiment to validate the vital importance of
spatially variant dilation rate guided by perspective maps. Specifically, we
average the perspective map of each image, and use the mean perspective maps
to replace the original ones. Then, we train our PFDNet with the mean
perspective maps and name the trained model as Ours (mean perspective map). In
Table VI, Ours (mean perspective map) performs much worse than Ours (original
perspective map) does. It verifies that adaptive receptive allocation is
essential for estimating accurate pedestrian counts. Although the severe
performance degradation it suffers when we adopt mean perspective map for each
image, it still slightly outperforms CSRNet*. In fact, Ours (mean perspective
map) can be taken as a weaken version of PFDNet, in which dynamic receptive
fields allocation is only performed among images instead of among pixels.

TABLE VI: Comparisons on different dilation rate maps.

Method | Part A | Part B
---|---|---
MAE | RMSE | MAE | RMSE
Ours (original perspective map) | 53.8 | 89.2 | 6.5 | 10.7
Ours (mean perspective map) | 63.7 | 104.8 | 9.0 | 14.3
CSRNet* | 65.1 | 112.4 | 9.8 | 15.2

####  IV-E4 Visualization of Dilation Rate Maps

In order to have a deeper understanding of PFC layer, we further visualize the
dilation maps of different PFC layers in our PFDNet. Fig. 10(c) shows the
dilation rate maps of the first PFC (the $1$-st PFC layer, in a top to bottom
order) and Fig. 10(d) demonstrates the last (the $6$-th PFC layer, in the same
order) PFC layer in our PFDNet. Obviously, in different PFC layers, the
dilation maps are different due to the learnable parameters of $\alpha$,
$\beta$, $\gamma$ and $\theta$. Besides, for the same position, the value in
Fig. 10(c) tends to be smaller than that in Fig. 10(d). This accords with the
fact: For the same scale of a head, when the region can be covered by a
fractional dilation convolution with small dilation rate in the top layer
(i.e., here is the $1$-st PFC layer in a top to bottom order), then for the
lower layer (i.e., here is the $6$-th PFC layer in a top to bottom order), we
need a fractional dilation convolution with larger dilation rate to cover the
region. Finally, we show the receptive fields of two heads with small red
balls in (a), it can be seen that the receptive fields basically fit the
scales of heads, which indicates the rationale of PFC layer.

Figure 10: Visualization of dilation rate maps of different PFC layers in our
PFDNet. (a) is the input image, (b) is the perspective map. (c) and (d) are
dilation rate maps of the first PFC (the $1$-st PFC layer, in a top to bottom
order) and the last (the $6$-th PFC layer, in the same order) PFC,
respectively.

####  IV-E5 Comparison with Deformable Convolution

Intuitively, deformable convolution [71] learns to allocate adaptive receptive
field via enabling the offsets on the sampling positions. Thus, we conduct an
experiment by replacing normal dilated convolution in CSRNet* with deformable
convolution [71] layer by layer, in the order from top to bottom, All the
settings are kept the the same as those described in Sec. IV-E1. We term this
network CSRNet*(DeConv). Table VII shows the performance comparison on
ShanghaiTech dataset. It is seen that when all the six dilated convolutions
are replaced by deformable convolutions, CSRNet*(DeConv) delivers the best
performance with MAE $56.7/7.6$ on ShanghaiTech A / B, respectively, however,
still inferior to our PFDNet with the margin of $2.9/1.1$ MAE. This indicates
the superiority of our PFC against deformable convolution. Furthermore, in
Fig. 11, we visualize the sampling positions of PFC and deformable
convolution. Concretely, we show two typical cases. In the region surrounded
by white dashed lines, it is observed that the layout of sampling positions is
approximately in the shape of a square, resulting in a roughly Gaussian-like
blob in Fig. 11(e). In the area marked with orange dashed lines, two heads are
too close and the corresponding two groups of offsets mix with each other,
leading to inadequate feature aggregation and unsatisfying prediction in Fig.
11(e). While, for our PFC shown in Fig. 11(f), as guided by the prior of the
perspective map, it can be seen that the sampling positions fall uniformly in
an appropriate square in the head, leading to much more accurate and clear
estimation in both cases.

TABLE VII: Comparison of PFC with deformable convolution on ShanghaiTech
dataset.

# of PFCs / DeConvs | Part A | Part B
---|---|---
PFC | DeConv [71] | PFC | DeConv [71]
0 | 65.1 | 65.1 | 9.8 | 9.8
1 | 63.3 | 62.6 | 8.6 | 9.2
2 | 60.4 | 60.3 | 7.9 | 8.7
3 | 57.8 | 58.4 | 7.5 | 8.3
4 | 56.2 | 57.8 | 7.1 | 8.0
5 | 54.4 | 57.1 | 6.7 | 7.7
6 | 53.8 | 56.7 | 6.5 | 7.6

Figure 11: Visualization of sampling locations of deformable convolution and
PFDNet. (a) is the input image, (b) and (c) are input image with some sampling
positions. (d),(e),(f) are the density maps of ground-truth, CSRNet*(DeConv)
and PFDNet. It is seen that PFDNet surpasses CSRNet*(DeConv) in estimating
more accurate counts and more reasonable sampling postions.

####  IV-E6 Importance of PENet Pre-training

The pre-training of PENet makes the decoder part a robust perspective map
constructor. By doing so, Ours B can recover a roughly satisfying perspective
estimation as shown in Fig. 8. In this section, we validate the necessity the
pre-training of PENet. To this end, we conduct two experiments without pre-
training of PENet on UCF_CC_50 [17] and TRANCOS [57]. As shown in Table VIII,
the results with PENet pre-training significantly outperform those without
pre-training. Such performance margin is attributed to the confusing guidance
of PENet, since without pre-training, we observe that the output of PENet is
still messy even after considerable epochs of training.

TABLE VIII: The performances with / without pre-training of PENet on UCF_CC_50
[17] and TRANCOS [57].

Method | UCF_CC_50 | TRANCOS
---|---|---
MAE | RMSE | GAME 0
PENet(w/ pre-training) | 205.8 | 289.3 | 3.06
PENet(w/o pre-training) | 264.8 | 383.5 | 5.28

####  IV-E7 Reliability of PENet Prediction

PENet is designed as a compromise of the situation that perspective
annotations are unavailable, in which the reliability of PENet is crucial.
Therefore, we conduct an experiment to confirm the feasibility of PENet. Table
IX demonstrates the comparisons of adopting the ground-truth or the estimated
perspective map as the guidance of PFC on ShanghaiTech Part A/B and
WorldExpo’10. MAEs are respectively $55.0$, $7.5$ and $7.3$, with a small
decrease of $1.2$, $1.0$ and $0.5$, respectively. This indicates that PENet is
competent to a reasonable perspective map estimator.

TABLE IX: Different guidances of PFC on ShanghaiTech Part A/B and
WorldExpo’10.

Perspective Map | ShanghaiTech Part A/B | WorldExpo’10
---|---|---
Estimated | 55.0/7.5 | 7.3
Ground-truth |  53.8/6.5 | 6.8

##  V Conclusion

In this paper, we have presented a perspective-guided fractional-dilation
convolutional network (PFDNet) for crowd counting. The key idea of PFDNet is
the perspective-guided fractional-dilation convolution, which plays a role as
an insertable module that successfully handles the continuous intra-scene
scale variation issue under the guidance of perspective map annotations. We
have also proposed a perspective estimation branch as well as its learning
strategy, which is incorporated into our method to form an end-to-end
trainable network, even without perspective map annotations. Experiments on
five benchmark datasets have shown the superiority of our PFDNet against the
state-of-the-arts.

## Acknowledgment

This project is partially supported by the National Natural Scientific
Foundation of China (NSFC) under Grant No.s U19A2073 and 61872118.

## References

  * [1] X. Liu, W. Liu, T. Mei, and H. Ma, “Provid: Progressive and multimodal vehicle reidentification for large-scale urban surveillance,” _IEEE Transactions on Multimedia_ , vol. 20, no. 3, pp. 645–658, 2017.
  * [2] M. Wang, B. Cheng, and C. Yuen, “Joint coding-transmission optimization for a video surveillance system with multiple cameras,” _IEEE Transactions on Multimedia_ , vol. 20, no. 3, pp. 620–633, 2017.
  * [3] G. Wang, B. Li, Y. Zhang, and J. Yang, “Background modeling and referencing for moving cameras-captured surveillance video coding in hevc,” _IEEE Transactions on Multimedia_ , vol. 20, no. 11, pp. 2921–2934, 2018.
  * [4] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 589–597.
  * [5] D. B. Sam, S. Surya, and R. V. Babu, “Switching convolutional neural network for crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 1, no. 3, 2017, p. 6.
  * [6] V. A. Sindagi and V. M. Patel, “Generating high-quality crowd density maps using contextual pyramid cnns,” in _Proceedings of the IEEE International Conference on Computer Vision_. IEEE, 2017, pp. 1879–1888.
  * [7] L. Zhang, M. Shi, and Q. Chen, “Crowd counting via scale-adaptive convolutional neural network,” in _2018 IEEE Winter Conference on Applications of Computer Vision_. IEEE, 2018, pp. 1113–1121.
  * [8] Z. Shen, Y. Xu, B. Ni, M. Wang, J. Hu, and X. Yang, “Crowd counting via adversarial cross-scale consistency pursuit,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5245–5254.
  * [9] P. Viola and M. J. Jones, “Robust real-time face detection,” _International Journal of Computer Vision_ , vol. 57, no. 2, pp. 137–154, 2004.
  * [10] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _IEEE Computer sSociety Conference on Computer Vision and Pattern Recognition_ , vol. 1. IEEE, 2005, pp. 886–893.
  * [11] B. Wu and R. Nevatia, “Detection of multiple, partially occluded humans in a single image by bayesian combination of edgelet part detectors,” in _Proceedings of the IEEE International Conference on Computer Vision_ , vol. 1. IEEE, 2005, pp. 90–97.
  * [12] M. Wang and X. Wang, “Automatic adaptation of a generic pedestrian detector to a specific traffic scene,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2011, pp. 3401–3408.
  * [13] A. C. Davies, J. H. Yin, and S. A. Velastin, “Crowd monitoring using image processing,” _Electronics & Communication Engineering Journal_, vol. 7, no. 1, pp. 37–47, 1995.
  * [14] D.-C. He and L. Wang, “Texture unit, texture spectrum, and texture analysis,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 28, no. 4, pp. 509–512, 1990.
  * [15] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Proceedings of the IEEE International Conference on Computer Vision_ , vol. 2. Ieee, 1999, pp. 1150–1157.
  * [16] C. Wang, H. Zhang, L. Yang, S. Liu, and X. Cao, “Deep people counting in extremely dense crowds,” in _Proceedings of the ACM international conference on Multimedia_. ACM, 2015, pp. 1299–1302.
  * [17] H. Idrees, I. Saleemi, C. Seibert, and M. Shah, “Multi-source multi-scale counting in extremely dense crowd images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 2547–2554.
  * [18] D. Ryan, S. Denman, C. Fookes, and S. Sridharan, “Crowd counting using multiple local features,” in _Digital Image Computing: Techniques and Applications_. IEEE, 2009, pp. 81–88.
  * [19] A. B. Chan, Z.-S. J. Liang, and N. Vasconcelos, “Privacy preserving crowd monitoring: Counting people without people models or tracking,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008, pp. 1–7.
  * [20] A. B. Chan and N. Vasconcelos, “Bayesian poisson regression for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_. IEEE, 2009, pp. 545–551.
  * [21] K. Chen, C. C. Loy, S. Gong, and T. Xiang, “Feature mining for localised crowd counting,” in _British Machine Vision Conference_ , 2012.
  * [22] A. Marana, L. d. F. Costa, R. Lotufo, and S. Velastin, “On the efficacy of texture analysis for crowd monitoring,” in _International Symposium on Computer Graphics_. IEEE, 1998, pp. 354–361.
  * [23] C. Zhang, H. Li, X. Wang, and X. Yang, “Cross-scene crowd counting via deep convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 833–841.
  * [24] S. Huang, X. Li, Z. Zhang, F. Wu, S. Gao, R. Ji, and J. Han, “Body structure aware deep crowd counting,” _IEEE Transactions on Image Processing_ , vol. 27, no. 3, pp. 1049–1059, March 2018.
  * [25] X. Liu, J. Van De Weijer, and A. D. Bagdanov, “Leveraging unlabeled data for crowd counting by learning to rank,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7661–7669.
  * [26] J. Liu, C. Gao, D. Meng, and A. G. Hauptmann, “Decidenet: Counting varying density crowds through attention guided detection and density estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5197–5206.
  * [27] L. Liu, H. Wang, G. Li, W. Ouyang, and L. Lin, “Crowd counting using deep recurrent spatial-aware network,” _arXiv preprint arXiv:1807.00601_ , 2018\.
  * [28] V. Ranjan, H. Le, and M. Hoai, “Iterative crowd counting,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 270–285.
  * [29] M. Shi, Z. Yang, C. Xu, and Q. Chen, “Revisiting perspective information for efficient crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7279–7288.
  * [30] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1091–1100.
  * [31] Z. Yan, Y. Yuan, W. Zuo, X. Tan, Y. Wang, S. Wen, and E. Ding, “Perspective-guided convolution networks for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 952–961.
  * [32] N. Liu, Y. Long, C. Zou, Q. Niu, L. Pan, and H. Wu, “Adcrowdnet: An attention-injective deformable convolutional network for crowd understanding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3225–3234.
  * [33] V. A. Sindagi and V. M. Patel, “Ha-ccn: Hierarchical attention-based crowd counting network,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 323–335, 2019.
  * [34] A. Zhang, J. Shen, Z. Xiao, F. Zhu, X. Zhen, X. Cao, and L. Shao, “Relational attention network for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6788–6797.
  * [35] A. Zhang, L. Yue, J. Shen, F. Zhu, X. Zhen, X. Cao, and L. Shao, “Attentional neural fields for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5714–5723.
  * [36] X. Jiang, L. Zhang, M. Xu, T. Zhang, P. Lv, B. Zhou, X. Yang, and Y. Pang, “Attention scaling for crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4706–4715.
  * [37] A. Luo, F. Yang, X. Li, D. Nie, Z. Jiao, S. Zhou, and H. Cheng, “Hybrid graph neural networks for crowd counting,” _arXiv preprint arXiv:2002.00092_ , 2020\.
  * [38] Z. Ma, X. Wei, X. Hong, and Y. Gong, “Bayesian loss for crowd count estimation with point supervision,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * [39] B. Wang, H. Liu, D. Samaras, and M. Hoai, “Distribution matching for crowd counting,” in _Advances in Neural Information Processing Systems_ , 2020\.
  * [40] H. Xiong, H. Lu, C. Liu, L. Liang, Z. Cao, and C. Shen, “From open set to closed set: Counting objects by spatial divide-and-conquer,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019\.
  * [41] D. B. Sam, S. V. Peri, M. N. Sundararaman, A. Kamath, and V. B. Radhakrishnan, “Locate, size and count: Accurately resolving people in dense crowds via detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [42] Y. Hu, X. Jiang, X. Liu, B. Zhang, J. Han, X. Cao, and D. Doermann, “Nas-count: Counting-by-density with neural architecture search,” _arXiv preprint arXiv:2003.00217_ , 2020.
  * [43] L. Liu, H. Lu, H. Zou, H. Xiong, Z. Cao, and C. Shen, “Weighing counts: Sequential crowd counting by reinforcement learning,” in _European Conference on Computer Vision_. Springer, 2020, pp. 164–181.
  * [44] Z. Shi, L. Zhang, Y. Liu, X. Cao, Y. Ye, M.-M. Cheng, and G. Zheng, “Crowd counting with deep negative correlation learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5382–5390.
  * [45] J. Wan and A. Chan, “Adaptive density map generation for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1130–1139.
  * [46] V. A. Sindagi and V. M. Patel, “Multi-level bottom-top and top-bottom feature fusion for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1002–1012.
  * [47] Z. Shi, P. Mettes, and C. G. Snoek, “Counting with focus for free,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4200–4209.
  * [48] Y. Liu, M. Shi, Q. Zhao, and X. Wang, “Point in, box out: Beyond counting persons in crowds,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6469–6478.
  * [49] J. Wan and A. Chan, “Modeling noisy annotations for crowd counting,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [50] X. Cao, Z. Wang, Y. Zhao, and F. Su, “Scale aggregation network for accurate and efficient crowd counting,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 734–750.
  * [51] Z. Cheng, J. Li, Q. Dai, X. Wu, and A. Hauptmann, “Learning spatial awareness to improve crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * [52] X. Liu, J. Yang, and W. Ding, “Adaptive mixture regression network with local counting map for crowd counting,” _arXiv preprint arXiv:2005.05776_ , 2020\.
  * [53] V. Lempitsky and A. Zisserman, “Learning to count objects in images,” in _Advances in Neural Information Processing Systems_ , 2010, pp. 1324–1332.
  * [54] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [55] M. Mostajabi, M. Maire, and G. Shakhnarovich, “Regularizing deep networks by modeling and predicting label structure,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , June 2018.
  * [56] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. Al-Maadeed, N. Rajpoot, and M. Shah, “Composition loss for counting, density map estimation and localization in dense crowds,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 532–546.
  * [57] R. Guerrero-Gómez-Olmedo, B. Torre-Jiménez, R. López-Sastre, S. Maldonado-Bascón, and D. Onoro-Rubio, “Extremely overlapping vehicle counting,” in _Iberian Conference on Pattern Recognition and Image Analysis_. Springer, 2015, pp. 423–431.
  * [58] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [59] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International Conference on Machine Learning_ , 2015, pp. 448–456.
  * [60] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 8026–8037.
  * [61] X. Jiang, Z. Xiao, B. Zhang, X. Zhen, X. Cao, D. Doermann, and L. Shao, “Crowd counting and density estimation by trellis encoder-decoder networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6133–6142.
  * [62] Q. Wang, J. Gao, W. Lin, and Y. Yuan, “Learning from synthetic data for crowd counting in the wild,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8198–8207.
  * [63] W. Liu, M. Salzmann, and P. Fua, “Context-aware crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5099–5108.
  * [64] C. Xu, K. Qiu, J. Fu, S. Bai, Y. Xu, and X. Bai, “Learn to scale: Generating multipolar normalized density maps for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8382–8390.
  * [65] X. Jiang, L. Zhang, T. Zhang, P. Lv, B. Zhou, Y. Pang, M. Xu, and C. Xu, “Density-aware multi-task learning for crowd counting,” _IEEE Transactions on Multimedia_ , 2020.
  * [66] L. Liu, W. Jia, J. Jiang, S. Amirgholipour, Y. Wang, M. Zeibots, and X. He, “Denet: A universal network for counting crowd with varying densities and scales,” _IEEE Transactions on Multimedia_ , 2020.
  * [67] Y. Yang, G. Li, D. Du, Q. Huang, and N. Sebe, “Embedding perspective analysis into multi-column convolutional neural network for crowd counting,” _IEEE Transactions on Image Processing_ , 2020.
  * [68] J. Wan, Q. Wang, and A. B. Chan, “Kernel-based density map generation for dense object counting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [69] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [70] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [71] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 764–773.

|  Zhaoyi Yan is a Ph.D candidate of Computer Science, in Harbin Institute of
Technology, Harbin, China. His research interests include deep learning and
crowd counting.
---|---
|  Ruimao Zhang is currently a Research Assistant Professor in the school of
Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-SZ), China.
He received the B.E. and Ph.D. degrees from Sun Yat-sen University, Guangzhou,
China, in 2011 and 2016, respectively. He was a Postdoctoral Fellow in CUHK
from 2017 to 2019. Before that, he was a visiting Ph.D. student with Hong Kong
Polytechnic University (PolyU) from 2013 to 2014. His current research
interests include computer vision, deep learning, and related multimedia
applications.
---|---
|  Hongzhi Zhang received his Ph.D. degree in computer science and technology
from Harbin institute of Technology (HIT), China, in 2007. He is an associate
Professor at the School of Computer Science and Technology, HIT. His research
interests include theoretic approaches to problems in signal processing,
computer vision and machine learning. His research has been supported by
grants from the National Natural Science Foundation of China.
---|---
|  Qingfu Zhang (M’01-SM’06-F’17) received the BSc degree in mathematics from
Shanxi University, China in 1984, the MSc degree in applied mathematics and
the PhD degree in information engineering from Xidian University, China, in
1991 and 1994, respectively. He is a Chair Professor of Computational
Intelligence at the Department of Computer Science, City University of Hong
Kong. His main research interests include evolutionary computation,
optimization, neural networks, data analysis, and their applications. Dr.
Zhang is an Associate Editor of the IEEE Transactions on Evolutionary
Computation and the IEEE Transactions on Cybernetics. He is a Clarivate Highly
Cited Researcher in Computer Science for four consecutive years from 2016.
---|---
|  Wangmeng Zuo (M’09-SM’14) received the Ph.D. degree in computer application
technology from the Harbin Institute of Technology, Harbin, China, in 2007. He
is currently a Professor in the School of Computer Science and Technology,
Harbin Institute of Technology. His current research interests include image
enhancement and restoration, image and face editing, object detection, visual
tracking, and image classification. He has published over 100 papers in top
tier academic journals and conferences. He has served as a Tutorial Organizer
in ECCV 2016, an Associate Editor of the _IET Biometrics_ and _Journal of
Electronic Imaging_.
---|---