# Lessons from Computational Modelling of Reference Production in Mandarin and
English

Guanyi Chen  Kees van Deemter
Department of Information and Computing Sciences
Utrecht University
{g.chen<EMAIL_ADDRESS>
###### Abstract

Referring expression generation (REG) algorithms offer computational models of
the production of referring expressions. In earlier work, a corpus of
referring expressions (REs) in Mandarin was introduced. In the present paper,
we annotate this corpus, evaluate classic REG algorithms on it, and compare
the results with earlier results on the evaluation of REG for English
referring expressions. Next, we offer an in-depth analysis of the corpus,
focusing on issues that arise from the grammar of Mandarin. We discuss
shortcomings of previous REG evaluations that came to light during our
investigation and we highlight some surprising results. Perhaps most
strikingly, we found a much higher proportion of under-specified expressions
than previous studies had suggested, not just in Mandarin but in English as
well.

##  1 Introduction

Figure 1: Two scenes from the tuna experiment, in which (a) is a situation
from the furniture domain while (b) is from the people domain.

Referring expression generation (REG) originated as a sub-task of traditional
natural language generation systems (NLG, Reiter and Dale, 2000). The task is
to generate expressions that help hearers to identify the referent that a
speaker is thinking about. REG has important practical value in natural
language generation (Gatt and Krahmer, 2018), computer vision (Mao et al.,
2016), and robotics (Fang et al., 2015). Additionally, REG algorithms can be
seen as models of human language use (van Deemter, 2016).

In line with this second angle, and unlike REG studies which have started to
use black-box Neural Network based models (e.g., Mao et al. (2016); Ferreira
et al. (2018) and Cao and Cheung (2019)), we focus on two aspects (cf.,
Krahmer and van Deemter (2012)): 1) designing and conducting controlled
elicitation experiments, yielding corpora which are then used for analysing
and evaluating REG algorithms to gain insight into linguistic phenomena, e.g.,
gre3d3 (Dale and Viethen, 2009), tuna (Gatt et al., 2007; van Deemter et al.,
2012), coconut (Jordan and Walker, 2005), and MapTask (Gupta and Stent, 2005).
2) designing algorithms that mimic certain behaviours used by human beings,
for example the maximisation of discriminatory power (Dale, 1989) and/or the
preferential use of cognitively “attractive” attributes (Dale and Reiter,
1995); see Gatt et al. (2013) for discussion.

The focus of these studies was mostly on Indo-European languages, such as
English, Dutch (Koolen and Krahmer, 2010) and German (Howcroft et al., 2017).
Recently researchers have started to have a look at Mandarin Chinese (van
Deemter et al., 2017), collecting a corpus of Mandarin REs, namely mtuna. So
far, only a preliminary analysis has been performed on mtuna, and this
analysis has focused on issues of Linguistic Realisation (van Deemter et al.,
2017): the REs in the corpus have not yet been compared with those in other
languages, and the performance of REG algorithms on the corpus has not been
evaluated.

To fill this gap, we provide a more detailed analysis of the use of Mandarin
REs on the basis of the mtuna corpus. We annotated the mtuna corpus in line
with the annotation scheme of tuna (van der Sluis et al., 2006), after which
we used this annotation to evaluate the classical REG algorithms and compared
the results with those for the English etuna corpus. Since it has been claimed
that Mandarin favours brevity over clarity – the idea that Mandarin is
“cooler” than these other languages (Newnham, 1971; Huang, 1984) – relying
more on communicative context for disambiguation than western languages, we
concentrated on the use of over- and under-specification. After all, if
Mandarin favours brevity over clarity to a greater extent than English and
Dutch, then one would expect to see less over-specification and more over-
specification in Mandarin.

##  2 Background

The analysis reported in paper is based on the mtuna (for Mandarin) and etuna
(for English) corpus. We start by briefly introducing the tuna experiments in
general, and we highlight some special features of mtuna together with its
initial findings.

###  2.1 The tuna Experiments

tuna (Gatt et al., 2007; van der Sluis et al., 2007) is a series of controlled
elicitation experiments that were set up to aid computational linguist’s
understanding of human reference production. In particular, the corpora to
which these experiments gave rise were employed to evaluate REG algorithms, by
comparing their output with the REs in these corpora. The stimuli in the tuna
experiments were divided into two types of visual scenes: scenes that depict
furniture and scenes that depict people. Figure 1 shows an example for each of
these two types of scenes. In each trial, one or two objects in the scene were
chosen as the target referent(s), demarcated by red borders. The subjects were
asked to produce referring expressions that identify the target referents from
the other objects in the scene (their “distractors”). For example, for the
scene in Figure 1, one might say _the large chair_. The trials in the people
domain were intended to be more challenging than those in the furniture
domain.

The resulting corpus, which we will call etuna, was subsequently studied for
evaluating a set of “classic” REG algorithms (van Deemter et al., 2012).
Although RE has given rise to a good number of other corpora, with subtly
different qualities (e.g., Dale and Viethen (2009)), we focus here on the tuna
corpora for two reasons: firstly the etuna corpus was used in a series of
Shared Task Evaluation Campaign (Gatt and Belz, 2010), which caused it to be
relatively well known. Secondly and more importantly from the perspective of
the present paper, etuna inspired a number of similarly constructed corpora
for Dutch (dtuna, Koolen and Krahmer, 2010), German (gtuna, Howcroft et al.,
2017), and Mandarin (van Deemter et al., 2017).

###  2.2 The Mandarin tuna

The different tuna corpora were set up in highly similar fashion: for
instance, they all use a few dozen stimuli, which were offered in isolation
(i.e., participants were encouraged to disregard previous scenes and previous
utterances), and chosen from the same sets of furniture and people images;
furthermore, participants were asked to enter a type-written RE following a
question.

Yet there were subtle differences between these corpora as well, reflecting
specific research questions that the various sets of authors brought to the
task. The stimuli used by mtuna were inherited from the dtuna, where there are
totally 40 trials. Different from other tunas which always asked subjects
essentially the same question, namely _Which object/objects appears/appear in
a red window?_ , mtuna distinguished between referring expressions in subject
and object position.111This was done because the literature on Mandarin (e.g.,
Chao (1965)) suggests that Mandarin NPs in pre-verbal position may be
interpreted as definite unless there is information to the contrary. More
precisely, subjects were asked to use REs for filling in blanks in either of
the following patterns: .[(a)] ____ 在红色方块中 ‘Please complete the sentence: ____
is in the red frame(s)’ .̱[(b)] 红色方块中的是 ____ ‘What’s in the red frame is ____’

where (a) asked subjects to place the referring expression in subject position
while (b) asked to place it in object position. The initial analysis in van
Deemter et al. (2017) focused on how definiteness was expressed in Mandarin
REs. They found that 1) most definite REs are bare nouns. 2) indefinite REs
also appear quite often, especially in subject position.

##  3 Research Questions

Analogous to studies of earlier tuna corpora, our primary research question
(RQ1) is _how classic REG algorithms perform on mtuna and how this is
different from the performance on etuna?_ We were curious to see whether the
value of each evaluation metric for each algorithms will change very much, and
whether the rank order of the algorithms stays the same. If, as hypothesised,
Mandarin prefers brevity over clarity, then the Full Brevity algorithm (which
always yields REs with minimally number of properties), is expected to have
higher performance on mtuna than on etuna. The expected effect on other
classic algorithms is less clear.

It is thought that, since TYPE helps create a “conceptual gestalt” of the
target referent (which benefits the hearer (Levelt, 1993, Chapter 4)) speakers
tend to include a TYPE in their REs regardless of its discriminatory
power.222Note that 92.25% of the REs in etuna contain a superfluous TYPE (van
der Sluis et al., 2007)). For this reason, algorithms such as the Incremental
Algorithm (Dale and Reiter, 1995) always append a TYPE to the REs they
produce. However, Lv (1979) found that the head of a noun phrase in Mandarin
is often omitted if this noun is the only possibility given the context. This
suggests that, if all objects in a scene share the same type (e.g., all the
objects in the people domain of tuna are male scientists), then it is less
likely for Mandarin speakers to express a TYPE. Accordingly, our second
research question (RQ2) asks _to what extent the role of TYPE differs between
English and Mandarin._ Connected with this, we were curious to what extent
this issue affects the performance of the classic REG algorithms.

As discussed in section 1, the coolness hypothesis stated that Chinese relies
more on the communicative context for disambiguation than western languages,
such as English, based on which Chinese is also seen as a discourse-based
language while English is a sentence-based language. The existence of primary
evidence for this issue in REG was identified in van Deemter et al. (2017),
indicating that Mandarin speakers rarely explicitly express number, maximality
and giveness in REs, and in Chen et al. (2018), indicting that they sometimes
even drop REs. In this study, we were curious about (RQ3) _the use of over-
specification and under-specifications in mtuna versus etuna_, hypothesising
that Mandarin REs use fewer over-specifications and more under-specifications
than English.

We have seen that mtuna asked its participants to produce REs in different
syntactic positions. van Deemter et al. (2017) found more indefinite NPs in
the subject position, which is inconsistent with linguistic theories (James et
al., 2009) that suggests subjects and other pre-verbal positions favour
definiteness. Building on these findings, we investigated (RQ4) _how syntactic
position influences the use of over-/under-specification and the performance
of REG algorithms_.

##  4 Method

| Domain | Total | Mini. | Real | Nom. | Num. | Wrong | Other | Under
---|---|---|---|---|---|---|---|---|---
mtuna | furniture | 377 | 46 | 117 | 132 | 2 | 11 | 5 | 64
people | 371 | 16 | 216 | 68 | 13 | 4 | 6 | 48
mtuna-ol | furniture | 264 | 9 | 83 | 104 | 0 | 8 | 4 | 56
people | 222 | 14 | 144 | 36 | 2 | 1 | 3 | 22
etuna | furniture | 158 | 1 | 58 | 62 | 0 | 0 | 0 | 37
people | 132 | 3 | 75 | 37 | 0 | 0 | 0 | 7
Table 1: Frequencies of referring expressions that fall in each type specifications in mtuna, mtuna-ol and etuna respectively. Specifically, total is the total number of descriptions in each corpus. mini. is the minimal over-specification, real is the real over-specification, nom. is the nominal over-specification, num. is the numerical over-specification, wrong is the duplicate-attribute over-specification, other stands for the RE that cannot be classified into any of these categories, and under is the under-specification. Furniture | People
---|---
Model | DICE (SD) | PRP | Model | DICE (SD) | PRP
IA-COS | 0.875 (0.17) | 55.7 | IA-GBHOATSS | 0.637 (0.26) | 16.3
IA-CSO | 0.847 (0.21) | 55.1 | IA-BGHOATSS | 0.629 (0.25) | 15.5
IA-OCS | 0.797 (0.16) | 20.5 | IA-GHBOATSS | 0.617 (0.25) | 13.0
IA-SCO | 0.754 (0.18) | 15.0 | IA-BHGOATSS | 0.577 (0.24) | 7.5
IA-OSC | 0.740 (0.20) | 18.3 | IA-HGBOATSS | 0.589 (0.23) | 6.1
IA-SOC | 0.690 (0.21) | 14.7 | IA-HBGOATSS | 0.559 (0.24) | 6.1
- | - | - | IA-SSTAOHBG | 0.347 (0.23) | 1.9
FB+TYPE | 0.830 (0.18) | 39.9 | FB+TYPE | 0.669 (0.26) | 23.2
FB | 0.574 (0.25) | 3.0 | FB | 0.446 (0.32) | 9.9
GR | 0.802 (0.21) | 39.3 | GR | 0.613 (0.29) | 19.9
Table 2: Experiment results on mtuna, in which the string after each IA
algorithm represents the preference order it uses. For example, “COS” means
COLOUR $>$ ORIENTATION $>$ SIZE and “BGHOATSS” stands for hasGlasses $>$ BEARD
$>$ HAIR $>$ ORIENTATION $>$ AGE $>$ hasTie $>$ hasShirt $>$ hasSuit.

Before we address the four research questions in section 3, we explain how we
annotated the corpus. The annotated corpus is available at
github.com/a-quei/mtuna-annotated

###  4.1 Annotating the Corpus

1650 REs were semantically annotated (after omitting some unfinished REs from
the corpus) following the scheme of van der Sluis et al. (2006).333This
includes the trials that have one target referent and those that have two
targets, but, in this paper, we focus on the former one. The annotated corpus
is public available at: xxx. For simplicity, instead of XML we use the JSON
for the annotation. Because the scenes stay the same when different subjects
accomplished the experiment, we annotated the scene and the REs in mtuna
separately. For the attribute hairColour, both van der Sluis et al. (2006) and
Gatt et al. (2008) (and all the annotate scheme used by the previous tuna
corpora) annotated both hair colour and beard colour as hairColour. However,
this would cause us to overlook some key phenomena, because some participants
used the colour of a person’s beard for distinguishing the target. Therefore,
we decided to use hairColour and beardColour as separate attributes. As
pointed out in van Deemter et al. (2012), since the attribute hairColour is
depend on hasHair, the authors merged these two into a single attribute Hair
during the evaluation. We did the same thing and obtained two merged
attributes: Hair and Beard.

To avoid compromising the comparison between mtuna and etuna, we did not only
annotate mtuna but we also re-annotated the etuna corpus, using the same
annotators. Details about which properties were annotated and examples of
annotated REs can be found in Appendix A.

###  4.2 Annotating Over-/Under-specifications

To gain an insightful analysis of the speakers’ use of over- and under-
specification, and to ensure that our annotations are well defined, we will
offer some definitions. In addition, given our interest in the role of TYPE,
we will sub-categorise by distinguishing different types of over-
specifications. Concretely, we asked the annotators to consider the following
types of specifications:

Minimal Description. an RE that successfully singles out the target referent
and does this by using the minimum possible number of properties. These are
the REs that match Dale and Reiter’s Full Brevity;

Numerical Over-specification. an RE that uses more properties than the
corresponding minimal description uses, yet the removal of any of them results
in a referential confusion. For instance, for the scene in 1(a), the RE _the
green chair_ is a numerical over-specification as it uses more properties than
the minimal description _the large one_ ;

Nominal Over-specification. an RE from which only one of its properties is
removable, namely the TYPE of the target;

Real Over-specification. an RE from which at least one of its non-TYPE
properties is removable;

Under-specification. an RE all of whose properties are true of the referent
but that causes referential confusion (i.e., it is not a distinguishing
description in the sense of Dale (1992));

Wrong Description. an RE whose properties use one or more incorrect values for
a given attribute. In line with previous TUNA evaluations, we only consider a
value to be wrong if it could prevent a hearer from recognising the target.
For example, the RE _the pink chair_ is not called wrong if the referent is a
red chair.

We annotated each RE in both corpora 444When applying this annotation scheme
to REs that have multiple targets, adaptations need to be made. But since the
focus of this paper is on singular REs, we will not offer details., and we
annotated each scene in each corpus. Thus, for each RE, we annotate which of
the above specification types it falls in, and how many over-specified/under-
specified properties the RE contains. In Appendix B, Table 7 records, for each
scene, how many different minimal descriptions the scene permits (most often
just 1, but sometimes 2 or 3). The results per RE are depicted in Table 1555We
observed a large number of minimal descriptions in the furniture domain of
mtuna. This is a result of the fact that some trials in mtuna use TYPE in
their minimal descriptions.  and the results per scene are in Appendix B.

##  5 Analysis

Furniture | People
---|---
| etuna | mtuna-ol |  | etuna | mtuna-ol
Model | DICE (SD) | PRP | DICE (SD) | PRP | Model | DICE (SD) | PRP | DICE (SD) | PRP
IA-COS | 0.919 (0.12) | 62.8 | 0.915 (0.14) | 65.5 | IA-GBHOATSS | 0.862 (0.17) | 50.0 | 0.724 (0.22) | 22.8
IA-CSO | 0.919 (0.12) | 62.8 | 0.915 (0.14) | 65.5 | IA-BGHOATSS | 0.861 (0.17) | 50.8 | 0.719 (0.21) | 21.0
IA-OCS | 0.832 (0.14) | 26.3 | 0.823 (0.15) | 25.4 | IA-GHBOATSS | 0.774 (0.20) | 27.3 | 0.674 (0.25) | 19.6
IA-SCO | 0.817 (0.14) | 20.5 | 0.808 (0.15) | 19.4 | IA-BHGOATSS | 0.761 (0.19) | 25.0 | 0.621 (0.22) | 7.8
IA-OSC | 0.805 (0.16) | 23.7 | 0.798 (0.17) | 23.8 | IA-HGBOATSS | 0.705 (0.17) | 3.8 | 0.609 (0.22) | 4.1
IA-SOC | 0.782 (0.16) | 19.9 | 0.767 (0.17) | 19.4 | IA-HBGOATSS | 0.670 (0.19) | 4.5 | 0.570 (0.23) | 3.7
- | - | - | - | - | IA-SSTAOHBG | 0.339 (0.10) | 0.0 | 0.285 (0.17) | 0.0
FB+TYPE | 0.849 (0.17) | 41.7 | 0.849 (0.16) | 42.5 | FB+TYPE | 0.847 (0.17) | 44.7 | 0.734 (0.23) | 27.4
FB | 0.590 (0.23) | 0.6 | 0.602 (0.24) | 3.6 | FB | 0.556 (0.16) | 2.3 | 0.541 (0.26) | 11.0
GR | 0.849 (0.17) | 41.7 | 0.849 (0.16) | 42.5 | GR | 0.727 (0.25) | 33.3 | 0.650 (0.28) | 21.9
Table 3: Experiment results on the mtuna-ol and etuna. Algorithms are listed
from top to bottom in order of their performance on etuna.

Before reporting results and analysis, we explain what datasets and algorithms
were analysed, and how evaluation was performed.

Dataset. The sources of our dataset are the mtuna and etuna corpora. For
Mandarin, we used the whole mtuna dataset. For comparing between languages
fairly, we only used REs for scenes that were shared between mtuna and etuna;
we call this set of shared scenes mtuna-ol. The original mtuna has 20 trials,
with 10 trials for each domain. The mtuna-ol and etuna contains 13 trials, in
which there are 7 and 6 trials from furniture and people domain respectively.
(More details of which scene is used can be found in the Appendix.)

Algorithms. We tested the classic REG algorithms, including: 1) the Full
Brevity algorithm (FB Dale, 1989): an algorithm that finds the shortest RE; 2)
the Greedy algorithm (GR Dale, 1989): an algorithm that iteratively selects
properties that rule out a maximum number of distractors (i.e., a property
that has the highest “Discriminative Power”); and 3) the Incremental
Algorithm: an algorithm that makes use of a fixed “preference order” of
attributes (IA Dale and Reiter, 1995).

Evaluation Metrics. We used what are still the most commonly used metrics for
evaluating attribute choice in REG. One is the DICE metric (Dice, 1945), which
measures the overlap between two attributes sets:

$\mbox{DICE}(\mathcal{D}_{H},\mathcal{D}_{A})=\frac{2\times|\mathcal{D}_{H}\cap\mathcal{D}_{A}|}{|\mathcal{D}_{H}|+|\mathcal{D}_{A}|}$

where $\mathcal{D}_{H}$ is the set of attributes expressed in the description
produced by a human author and $\mathcal{D}_{A}$ is the set of attributes
expressed in the logical form generated by an algorithm. We also report the
“perfect recall percentage” (PRP), the proportion of times the algorithm
achieves a DICE score of 1, which is seen as a indicator of the recall of an
algorithm.

###  5.1 Performance of Algorithms on mtuna

We report the evaluation results on mtuna and mtuna-ol in the Table 2 and 3.
For the FB algorithm, we tested both the original version and the version that
always appends a TYPE (named FB+TYPE). Moreover, since we did not observe any
significant difference in the frequencies of use of each attribute between
mtuna and etuna corpora, we let the IA make use of the same set of preference
orders as van Deemter et al. (2012).

In line with the previous findings in other languages, in the furniture
domain, it is IA (with a good preference order) that perform the best in both
mtuna and mtuna-ol. Interestingly, the people domain yields very different
results: this time, FB+TYPE becomes the winner.

An ANOVA test comparing GR, FB+TYPE, and the best IA suggests a significant
effect of algorithms on both domains and on both mtuna and mtuna-ol
(Furniture: $F(2,1008)=49.20,p=.002$; People: $F(2,1065)=11.97,p<.001$) and
mtuna-ol (Furniture: $F(2,699)=14,p<.001$; People: $F(2,622)=4.22,p=.015$). As
for each algorithm, by Tukey’s Honestly Significant Differences (HSD), we
found that IA defeats other algorithms in the furniture domain in both corpora
($p<.001$) and that the victory of FB+TYPE for people domain is significant in
mtuna ($p=.001$) but not in mtuna-ol ($p=0.96$).

The scores for algorithms in the people domain are much lower than those in
the furniture domain, even lower than the scores for the people domain in
etuna. This may be because, based on the numbers in Table 1, a Chi Squared
Test suggests that, in mtuna, there are more real over-specifications
($\chi^{2}(1,747)=55.95,p<.001$) but fewer nominal over-specifications
($\chi^{2}(1,747)=26.57,p<.001$) in the people domain than in the furniture
domain666This highlights the importance of sub-categorising the different
kinds of over-specifications, as we have done in section 4. As for the former,
real over-specifications are notoriously hard to model accurately by
deterministic REG algorithms, which is one of the motivations behind
probabilistic modelling (van Gompel et al., 2019) or Bayesian Modelling (Degen
et al., 2020); such an approach might have additional benefits for the
modelling of reference in Mandarin. The relative lack of nominal over-
specifications in Mandarin descriptions of people could be addressed along
similar lines, adding TYPE probabilistically. Another evidence is that, in the
mtuna people domain, FB outperforms many IAs on PRP, which does not happen in
the furniture domain.

By comparing the results for mtuna and mtuna-ol, we found that the rank order
(by performance) of algorithms stays the same, but the absolute scores for the
latter corpus are much higher. If we look into the annotations for the trials
from mtuna that are not in mtuna-ol (Appendix B), most of these trials have
multiple possible minimal descriptions and numerical over-specifications.
Every RE in the corpus that results in a successful communication can be seen
as either a minimal description or a numerical over-specification, with 0 or
more attributes added to it. When computing the DICE similarity score between
a generated RE and human produced REs, if it is close to a minimal
description, it will differ from another minimal description. For example,
suppose we have a trial having two miminal descriptions: _the large one_ and
_the green one_. Our FB produce the second minimal description (as it can only
produce one RE at a time). When we computing DICE, we obtain $\frac{2}{3}$ for
the RE _the green chair_ while 0 for the RE _the large chair_ , but, in fact,
either of them has only one superfluous attributes. This implies that when a
corpus contains multiple minimal REs, this will artificially lower the DICE
scores.777An analogous problem has been identified in the task of evaluating
image capturing (Yi et al., 2020), where the collision of multiple references
for a single image was considered. For the same reason, the performance of FB
increases a lot from mtuna/People to mtuna-ol/People because all trials in
mtuna-ol have only one possible minimal description. Another reason lies in
the decrease in the number of under-specifications from mtuna/People to mtuna-
ol/People.

###  5.2 Cross-linguistic Comparison

Table 3 reports the results for both mtuna-ol and etuna, from which, except
for the fact that FB+TYPE becomes having the best performance, we see no
difference on the order of the their performance. An interesting observation
is that, after correcting a few errors in the annotation of etuna (cf. section
4.1), the difference between IA and FB+TYPE is no longer significant in the
people domain in terms of Tukey’s HSD (compare the conclusion in van Deemter
et al. (2012)). In other words, in both languages there is no significant
difference between the performance of these two algorithms on the people
domain. We also checked the influence of language on the performance of FB and
FB+TYPE: the influence of the former is significant ($F(1,349)=23.63,p<.001$)
while that of later is not ($F(1,349)=0.36,p=.548$). This suggest that, in
fact, it is English speakers who show more brevity, except in terms of use of
TYPE. This might also explain the differences in absolute scores for all
algorithms in both etuna and mtuna, especially in the people domain. Another
possible reason for these differences is the fact that the REs in mtuna-ol
show slightly higher diversity in the choice of content than etuna, as the
standard deviations for every model is higher.

###  5.3 RQ2: the Role of TYPE

Figure 2: Change of the performance with respected to different probabilities
of inserting superfluous TYPE for either the FB+TYPE and IA on either the
people domain of mtuna-ol and etuna.

On the use of TYPE, we first look at the number of REs that uses TYPE in
mtuna-ol and etuna. 98.4% and 95.93% of REs in the furniture and people
domains of etuna contain TYPE. For mtuna-ol, those numbers are 91.29% and
74.77%, suggesting that Mandarin speakers are less likely to use superfluous
TYPE. Second, for Lv’s hypothesis introduced in section 3, we observed a
smaller proportion of uses of TYPE in the people domain
($\chi^{2}(1,485)=24.16,p<.001$), where all the objects share the same value
of TYPE. Comparing the performance of REG algorithms on the furniture domain
of mtuna and mtuna-ol, the difference is not as huge as that in the people
domain. This implies that the complement of Lv’s hypothesis might also hold,
namely, if the value of TYPE is not the only possibility, then it will not be
omitted.

To further assess the role of TYPE and to find more evidence regarding Lv’s
hypothesis, we investigated how introducing uncertainties in whether or not to
include a TYPE affects the performance of REG algorithms for the people
domain. We tried out different probabilities, and for each probability for
inserting the TYPE we ran the algorithm 100 times; we report the average DICE
score, drawing the lines indicating the change of performance over different
probabilities in Figure 2.

We found that: 1) the decrease in performance on mtuna-ol is smaller than that
on etuna; 2）IA and FB+TYPE have similar performance for Mandarin while IA
performs better for English; 3) The difference between the performance of
these algorithms becomes smaller when the influence of TYPE is ignored (i.e.,
when the probability of inserting TYPE is close to zero), especially for the
Full Brevity algorithm. On top of these findings, we observe that although
Mandarin speakers are less likely to use superfluous TYPE, always adding TYPE
achieves the best performances for all the algorithms. Such a result maybe be
caused by the dependencies between the use of different properties. In other
words, introducing uncertainty to only the TYPE cannot sufficiently model the
uncertainties in REG: when to drop a TYPE might also depend on the use of
other properties.

###  5.4 RQ3: Over-/Under-specification

In light of Table 1, some obvious conclusions can be drawn. For example, more
“real” over-specifications are used for more complex domains (i.e., the people
domain) than for simple ones. Focusing on RQ3 in section 3, its two hypotheses
are both rejected: no significant difference has been found in the use of
over-specifications ($\chi^{2}(1,775)=0.82,p=0.052$) or in the use of under-
specifications ($\chi^{2}(1,775)=0.745,p=0.105$). Focusing on the people
domain, where FB+TYPE performed better in English than in Mandarin, we found
no significant difference ($\chi^{2}(1,354)=2.53,p=0.112$).

###  5.5 RQ4: Syntactic Position

Model | Furniture | People
---|---|---
IA (subj.) | 0.940 (0.11)† | 0.728 (0.23)
IA (obj.) | 0.890 (0.16) | 0.719 (0.21)
GR (subj.) | 0.884 (0.13)† | 0.629 (0.30)
GR (obj.) | 0.815 (0.18) | 0.669 (0.25)
FB+TYPE (subj.) | 0.884 (0.13)† | 0.736 (0.23)
FB+TYPE (obj.) | 0.815 (0.18) | 0.733 (0.22)
Table 4: The performance of REG algorithms for REs in different syntactic
positions, in which IA is the IA with highest performance in the previous
experiments, i.e., the IA-COS and IA-GBHOATSS. ${\dagger}$ indicates that
there is significant influence of the syntactic position on that algorithm in
that domain.

For RQ4, we counted the number of real over-specifications and under-
specification in subject and object position. In the mtuna-ol corpus, there
are 247 and 239 descriptions in the subject and object positions,
respectively. No significant difference on the use of over-specifications was
found ($\chi^{2}(1,485)=1.57,p=0.209$) but a significant difference regarding
the use of under-specifications did exist ($\chi^{2}(1,485)=19.27,p<.001$).
Considering the fact that there are more indefinite RE in subject position
(van Deemter et al., 2017), the present finding might suggest that those
indefinite REs are not suitable for identifying a target referent. It appears
that further research is required to understand these issues in more detail.

As for the computational modelling, generally speaking, all algorithms
performed better for REs in subject position than for REs is object position,
with one exception, namely the GR algorithm for the people domain; the
difference is significant in the Furniture domain, but is not in the people
domain, possibly because the furniture domain contains more under-
specifications.

##  6 Discussion

###  6.1 Lessons about RE use

Regarding the “coolness” hypothesis, which focuses on the trade-off between
brevity and clarity, we found that the brevity of Mandarin is only reflected
in the use of TYPE but not in the other attributes, and, interestingly, no
evidence was found that this leads to a loss of clarity; our findings are
consistent with the possibility that Mandarin speakers may have found a better
optimum than English.

Although Mandarin speakers are less likely to over-specify TYPE, following Lv
(1979), we conclude that TYPE is often omitted _if and only if_ it has only
one possible value given the domain. This appears to happen “unpredictably”
(i.e., in one and the same situation, TYPE is often expressed but often
omitted as well). However, we saw that introducing probability for the use of
TYPE alone does not work well. This suggests that, to do justice to the data,
a REG model may have to embrace non-determinism more wholeheartedly, as in the
probabilistic approaches of van Gompel et al. (2019) and Degen et al. (2020).

We found significant influence of the syntactic position of the RE on the use
of under-specification and on the performance of REG algorithms. This flies in
the face of earlier research on REG – which has tended to ignore syntactic
position – yet it is in line with the theory of Chao (1965). On the other
hand, it gives rise to various questions: why are more under-specifications
used in subject positions, and why do all REG models perform better for REs in
subject positions than for those in object position? These questions invite
further studies including, for example, reader experiments to find out how REs
in different positions are comprehended. It would also be interesting to
investigate what role syntactic position plays in other languages, where this
issue has not yet been investigated.

Perhaps our most surprising findings regard the use of under-specification:
firstly (deviating from what van Deemter et al. (2017) hypothesised), we did
not find significantly more under-specifications in mtuna than in etuna. We
found a very substantial proportion, of nearly 20%, under-specified REs in
both mtuna and etuna. This was surprising, because, at least in Western
languages, in situations where Common Ground is unproblematic (Horton and
Keysar, 1996), under-specification is widely regarded as a rarity in the
language use of adults, to such an extent that existing REG algorithms are
typically designed to prevent under-specification completely (see e.g.,
Krahmer and van Deemter (2012)). Proportions of under-specifications in
corpora are often left reported, but (Koolen et al., 2011) report that only 5%
of REs in dtuna were under-specifications.888The difference might be that
dtuna used participants who came into the lab separately, whereas mtuna
participants sat together in a classroom.

These findings give rise to the following questions: 1) Why did previous
investigators either find far fewer under-specified REs (e.g., Koolen et al.
(2011), see Footnote 8) or ignored under-specification? 2) How does the
presence of under-specification influence the performance of the classic REG
algorithms (which never produce any under-specified REs, except when no
distinguishing RE exists)? and 3) If a REG model aims – as most do – to
produce human-like output, then what is the most effective way for them to
model under-specification?

###  6.2 Lessons about REG Evaluation

Most REG evaluations so far have made use of the DICE score (Dice, 1945).
However, on top of the discussions of van Deemter and Gatt (2007) and of
section 5, we identify the following three issues for evaluating REG with
DICE. First, if a scene has multiple possible minimal descriptions or
numerical over-specifications, then this causes DICE scores to be artificially
lowered (section 2.2) and hence distorted. Second, there is no guarantee that
an RE with a high DICE score is a distinguishing description. Third, DICE
punishes under-specification more heavily than over-specification. Suppose we
have a reference RE $d$ which uses $n$ attributes, a over-specification
$d_{o}$ with one more superfluous comparing to $d$ (so it uses $n+1$
attributes), and a under-specification $d_{u}$ which can be repaired to $d$ by
adding one attribute (using $n-1$ attributes), the DICE score of $d_{o}$ is
$2n/(2n+1)$ while $d_{u}$’s DICE is $2n-2/(2n-1)$. In other words, $d_{o}$ has
a higher DICE than $d_{u}$. Whether this should be considered a shortcoming of
DICE or a feature is a matter for debate.

Finally, our analysis suggests that previous tuna experiments may have been
insufficiently controlled. For example, some trials in mtuna and dtuna use
TYPE for distinguishing the target, causing nominal over-specifications not to
be counted as over-specification. Different trials have different numbers of
minimal descriptions and different numbers of numerical over-specifications.
As shown in section 5, these issues impact evaluation results and this might
cause the conclusions from evaluating algorithms with tuna not to be
reproducable.

Comparisons between corpora need to be approached with caution, and the
present situation is no exception. For all the similarities between them, we
have seen that there are significant differences in the ways in which the tuna
corpora were set up.999Most tuna experiments involved type-written REGs, but
dtuna elicited spoken REs. In most tuna experiments the linguistic context was
uniform, but mtuna elicited REs in different syntactic positions, as we have
seen. Although these differences exist for a reason (i.e., for testing
linguistic hypotheses), we believe that it would be worthwhile to design new
multilingual datasets, where care is taken to ensure that utterances in the
different languages are elicited under circumstances that are truly as similar
as they can be.

## Acknowledgements

We thank the anonymous reviewers for their helpful comments. Guanyi Chen is
supported by China Scholarship Council (No.201907720022).

## References

  * Cao and Cheung (2019) Meng Cao and Jackie Chi Kit Cheung. 2019.  Referring expression generation using entity profiles.  _arXiv preprint arXiv:1909.01528_.
  * Chao (1965) Yuen Ren Chao. 1965.  _A grammar of spoken Chinese_.  Univ of California Press.
  * Chen et al. (2018) Guanyi Chen, Kees van Deemter, and Chenghua Lin. 2018.  Modelling pro-drop with the rational speech acts model.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 57–66. Association for Computational Linguistics (ACL).
  * Dale (1989) Robert Dale. 1989.  Cooking up referring expressions.  In _27th Annual Meeting of the association for Computational Linguistics_ , pages 68–75.
  * Dale (1992) Robert Dale. 1992.  _Generating referring expressions: Constructing descriptions in a domain of objects and processes._ The MIT Press.
  * Dale and Reiter (1995) Robert Dale and Ehud Reiter. 1995.  Computational interpretations of the gricean maxims in the generation of referring expressions.  _Cognitive science_ , 19(2):233–263.
  * Dale and Viethen (2009) Robert Dale and Jette Viethen. 2009.  Referring expression generation through attribute-based heuristics.  In _Proceedings of the 12th European workshop on natural language generation (ENLG 2009)_ , pages 58–65.
  * van Deemter (2016) Kees van Deemter. 2016.  _Computational models of referring: a study in cognitive science_.  MIT Press.
  * van Deemter and Gatt (2007) Kees van Deemter and Albert Gatt. 2007.  Content determination in GRE: Evaluating the evaluator.  In _Using Corpora for Natural Language Generation: Language Generation and Machine Translation_.
  * van Deemter et al. (2012) Kees van Deemter, Albert Gatt, Ielka van der Sluis, and Richard Power. 2012.  Generation of referring expressions: Assessing the incremental algorithm.  _Cognitive science_ , 36(5):799–836.
  * van Deemter et al. (2017) Kees van Deemter, Le Sun, Rint Sybesma, Xiao Li, Bo Chen, and Muyun Yang. 2017.  Investigating the content and form of referring expressions in Mandarin: introducing the mtuna corpus.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 213–217, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Degen et al. (2020) Judith Degen, Robert D Hawkins, Caroline Graf, Elisa Kreiss, and Noah D Goodman. 2020.  When redundancy is useful: A bayesian approach to “overinformative” referring expressions.  _Psychological Review_.
  * Dice (1945) Lee R Dice. 1945.  Measures of the amount of ecologic association between species.  _Ecology_ , 26(3):297–302.
  * Fang et al. (2015) Rui Fang, Malcolm Doering, and Joyce Y Chai. 2015.  Embodied collaborative referring expression generation in situated human-robot interaction.  In _Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction_ , pages 271–278.
  * Ferreira et al. (2018) Thiago Castro Ferreira, Diego Moussallem, Ákos Kádár, Sander Wubben, and Emiel Krahmer. 2018.  Neuralreg: An end-to-end approach to referring expression generation.  _arXiv preprint arXiv:1805.08093_.
  * Gatt and Belz (2010) Albert Gatt and Anja Belz. 2010.  Introducing shared tasks to nlg: The tuna shared task evaluation challenges.  In _Empirical methods in natural language generation_ , pages 264–293. Springer.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the state of the art in natural language generation: Core tasks, applications and evaluation.  _Journal of Artificial Intelligence Research_ , 61:65–170.
  * Gatt et al. (2013) Albert Gatt, Emiel Krahmer, Roger van Gompel, and Kees van Deemter. 2013.  Production of referring expressions: Preference trumps discrimination.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 35.
  * Gatt et al. (2007) Albert Gatt, Ielka van der Sluis, and Kees van Deemter. 2007.  Evaluating algorithms for the generation of referring expressions using a balanced corpus.  In _Proceedings of the 11th European Workshop on Natural Language Generation (ENLG’07)_ , pages 49–56, Schloss Dagstuhl, Germany. Association for Computational Linguistics.
  * Gatt et al. (2008) Albert Gatt, Ielka van der Sluis, and Kees van Deemter. 2008.  Xml format guidelines for the tuna corpus.  Technical report, Technical report, Dept of Computing Science, University of Aberdeen.
  * van Gompel et al. (2019) Roger van Gompel, Kees van Deemter, Albert Gatt, Rick Snoeren, and Emiel J Krahmer. 2019.  Conceptualization in reference production: Probabilistic modeling and experimental testing.  _Psychological review_ , 126(3):345.
  * Gupta and Stent (2005) Surabhi Gupta and Amanda Stent. 2005.  Automatic evaluation of referring expression generation using corpora.  In _Proceedings of the Workshop on Using Corpora for Natural Language Generation_ , pages 1–6. Citeseer.
  * Horton and Keysar (1996) William S Horton and Boaz Keysar. 1996.  When do speakers take into account common ground?  _Cognition_ , 59(1):91–117.
  * Howcroft et al. (2017) David M Howcroft, Jorrig Vogels, and Vera Demberg. 2017.  G-tuna: a corpus of referring expressions in german, including duration information.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 149–153.
  * Huang (1984) C-T James Huang. 1984.  On the distribution and reference of empty pronouns.  _Linguistic inquiry_ , pages 531–574.
  * James et al. (2009) Huang C-T James, Y-H Audrey Li, and Yafei Li. 2009.  The syntax of chinese.  _Cambridge, Cambridge). doi_ , 10.
  * Jordan and Walker (2005) Pamela W Jordan and Marilyn A Walker. 2005.  Learning content selection rules for generating object descriptions in dialogue.  _Journal of Artificial Intelligence Research_ , 24:157–194.
  * Koolen et al. (2011) Ruud Koolen, Albert Gatt, Martijn Goudbeek, and Emiel Krahmer. 2011.  Factors causing overspecification in definite descriptions.  _Journal of Pragmatics_ , 43(13):3231–3250.
  * Koolen and Krahmer (2010) Ruud Koolen and Emiel Krahmer. 2010.  The d-tuna corpus: A dutch dataset for the evaluation of referring expression generation algorithms.  In _LREC_.
  * Krahmer and van Deemter (2012) Emiel Krahmer and Kees van Deemter. 2012.  Computational generation of referring expressions: A survey.  _Computational Linguistics_ , 38(1):173–218.
  * Levelt (1993) Willem JM Levelt. 1993.  _Speaking: From intention to articulation_ , volume 1.  MIT press.
  * Lv (1979) Shuxiang Lv. 1979.  Problems in the analysis of chinese grammar.
  * Mao et al. (2016) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016.  Generation and comprehension of unambiguous object descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 11–20.
  * Newnham (1971) Richard Newnham. 1971.  _About Chinese_.  Penguin Books Ltd.
  * Reiter and Dale (2000) Ehud Reiter and Robert Dale. 2000.  _Building natural language generation systems_.  Cambridge university press.
  * van der Sluis et al. (2006) Ielka van der Sluis, Albert Gatt, and Kees van Deemter. 2006.  Manual for the tuna corpus: Referring expressions in two domains.  Technical Report AUCS/TR0705, Department of Computing Science, Univ. of Aberdeen.
  * van der Sluis et al. (2007) Ielka van der Sluis, Albert Gatt, and Kees van Deemter. 2007.  Evaluating algorithms for the generation of referring expressions: Going beyond toy domains.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP’07)_ , Borovets, Bulgaria. RANLP.
  * Yi et al. (2020) Yanzhi Yi, Hangyu Deng, and Jinglu Hu. 2020.  Improving image captioning evaluation by considering inter references variance.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 985–994, Online. Association for Computational Linguistics.

##  Appendix A Annotating mtuna

Following and adapting from van der Sluis et al. (2006), the attributes and
corresponding values we used in our annotation (annotating the mtuna and re-
annotating the etuna) are shown in Table 5 and 6 for furniture and people
domain respectively. An example of the annotated RE sample in mtuna is shown
in Figure 3.

Attribute | Possible Values | Freq.
---|---|---
TYPE | _chair, sofa, desk, fan_ | 347
COLOUR | _blue, red, green, grey_ | 326
ORIENTATION | _front, back, left, right_ | 185
SIZE | _large, small_ | 141
X-DIMENSION | 1, 2, 3, 4, 5 | 5
Y-DIMENSION | 1, 2, 3 | 5
OTHER | - | 10
Table 5: Attributes and their values for REs in furniture domain. Attribute | Possible Values | Freq.
---|---|---
TYPE | _person_ | 278
AGE | _young, old_ | 74
ORIENTATION | _front, left, right_ | 6
hasBread | 0, 1 | 169
breadColour | _dark, light_ | 95
hasHair | 0, 1 | 116
hairColour | _dark, light_ | 100
hasGlasses | 0, 1 | 165
hasShirt | 0, 1 | 14
hasTie | 0, 1 | 26
hasSuit | 0, 1 | 60
X-DIMENSION | 1, 2, 3, 4, 5 | 6
Y-DIMENSION | 1, 2, 3 | 6
OTHER | - | 151
Table 6: Attributes and their values for REs in people domain.

⬇

{ "sno": "Object",

"subject_id": "2",

"object": [

{ "attributes": [

{"name": "COLOUR",

"value": "dark"

},

{"name": "TYPE",

"value": "table"

}]

}],

"trial_id": "1",

"utt": "灰桌子"

}

Figure 3: An example annotated data sample from mtuna, for the RE _灰桌子_ (_grey
table_).

##  Appendix B Results for Annotating Over-/Under-specifications

Table 7 listed all trails we used for analysing the use of over- and under-
specifications in this study, each of which we annotated how many alternative
minimal descriptions and numerical over-specifications are there. The raw data
of the mtuna corpus (and trial ID) can be found at:
homepages.abdn.ac.uk/k.vdeemter-/pages/mtuna-webpage/, and those of etuna are
available at: https://www.abdn.ac.uk/ncs/departments/computing-
science/tuna-318.php.

mtuna | etuna | #mini. | #num.
---|---|---|---
1 | 1 | 1 | 0
2 | 2 | 1 | 0
3 | 3 | 1 | 0
4 | 4 | 1 | 0
5 | 5 | 1 | 0
6 | 6 | 1 | 0
7 | 7 | 1 | 0
8 | - | 1 | 1
9 | - | 2 | 0
10 | - | 3 | 0
21 | 21 | 1 | 1
22 | 22 | 1 | 0
23 | 23 | 1 | 0
24 | 24 | 1 | 0
25 | 25 | 2 | 1
26 | 26 | 1 | 2
27 | - | 1 | 2
28 | - | 1 | 2
29 | - | 2 | 1
30 | - | 1 | 3
Table 7: The number of possible minimal descriptions (mini. and numerical
over-specifications (num.) for each trial. The first two colomns are the trial
IDs in mtuna and etuna.


# Scale-Aware Network with Regional and Semantic Attentions for Crowd Counting
under Cluttered Background

Qiaosi Yi$\dagger$, Yunxing Liu$\dagger$, Aiwen Jiang*, Juncheng Li, Kangfu
Mei, Mingwen Wang $\dagger$: Equal contribution. *: The corresponding
author.Q. Yi, and J. Li are with the Shanghai Key Laboratory of
Multidimensional Information Processing, East China Normal University,
Shanghai, China, and also with the school of Computer Science and Technology,
East China Normal University, Shanghai, China. (E-mail:
<EMAIL_ADDRESS>cvjunchengli@gmail.com)Y. Liu, A. Jiang, and M. Wang
are with the School of Computer and Information Engineering, Jiangxi Normal
University, Nanchang, China. (E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>mwwang@jxnu.edu.cn)K. Mei is with the Department of
Mathematics, The Chinese University of Hong Kong (ShengZheng), ShengZheng,
China. (E-mail<EMAIL_ADDRESS>
###### Abstract

Crowd counting is an important task that shown great application value in
public safety-related fields, which has attracted increasing attention in
recent years. In the current research, the accuracy of counting numbers and
crowd density estimation are the main concerns. Although the emergence of deep
learning has greatly promoted the development of this field, crowd counting
under cluttered background is still a serious challenge. In order to solve
this problem, we propose a Scale-Aware Crowd Counting Network (SACCN) with
regional and semantic attentions. The proposed SACCN distinguishes crowd and
background by applying regional and semantic self-attention mechanisms on the
shallow layers and deep layers, respectively. Moreover, the asymmetric multi-
scale module (AMM) is proposed to deal with the problem of scale diversity,
and regional attention based dense connections and skip connections are
designed to alleviate the variations on crowd scales. Extensive experimental
results on multiple public benchmarks demonstrate that our proposed SACCN
achieves satisfied superior performances and outperform most state-of-the-art
methods. All codes and pre-trained models will be released soon.

###### Index Terms:

Crowd counting, scale-aware, cluttered background, attention mechanism.

##  I Introduction

High-density crowds are prone to public accidents and accelerate the spread of
diseases. In the past year, the COVID-19 has swept the world. In order to
prevent its spread again, it is of great significance to monitor and analyze
high-density crowds and provide timely and effective evacuation. As one of the
most important assistance for the analysis of high-density population, crowd
counting is conducive to real-time guidance and control. Therefore, it has a
wide range of applications in public safety-related fields and has attracted
widespread attention in the field of computer vision. The task of crowd
counting is aims to estimate the number and the distribution of people in a
crowd. In the past, the mainstream methods relied on object detection [1, 2,
3] and regression [4, 5]. Recently, with the success of deep learning, crowd
counting tasks have also made significant progress [6, 7, 8, 9]. However, it
is still a challenging task since crowd scale variation and cluttered
backgrounds is very common in the real world.

Figure 1: Two challenges on crowd counting task. a) Cluttered backgrounds. b)
Variation on crowd scale.

As shown in Fig. 1 (a), the background of the crowd scene is very complicated.
Due to the gathering and interference of objects other than people, certain
parts of the crowd are not easy to be distinguished from a cluttered
background. In this case, crowd counting methods are prone to fail accurately
counting the crowd and evaluating its distribution. As shown in Fig. 1 (b),
the sizes of persons in the crowd have large variations, either caused by
camera perspective or camera position. All these unexpected variations put
forward rigorous requirements on the robustness of crowd counting algorithms.
Therefore, in this paper, we aim to design effective solutions for these two
obstacles.

To alleviate the counting errors caused by cluttered backgrounds, current
mainstream solutions [10, 11, 12, 13] adopt visual attention mechanism. In the
latest works, such as SDANet [13] and HA-CCN [11], researchers found that
background and crowd were easier to be distinguished on shallow feature maps.
Therefore, they suggested to apply pixel-wise attention on low-level feature
maps. However, as we know, features in different channels contribute different
discriminative information on classifying cluttered backgrounds and crowds.
Limitedly applying pixel-attention is not enough. In this paper, we proposed
to design region attention module (RAM) on shallow layers, in which channel-
wise attention and pixel-attention collaborate to highlight crowd regions.

Moreover, we argue that distinguishing clutter backgrounds and crowd
foreground can be taken as a semantic classification task. According to [14],
features in deep layers have richer semantic information and less noise than
shallow-layer features for semantic categorization. Therefore, we further
propose a semantic attention module (SAM) on deep layers, in which spatial
self-attention and channel-wise self-attention collaborate to identify persons
in semantic space.

To alleviate the negative influences from crowd scale variations, current
mainstream works proposed to utilize strategies like multi-column CNN [7],
stacked multi-branch blocks [15, 16] and multi-scale feature fusion [17, 13].
However, the crowd scale variation is still far from being solved. As we know,
features of different resolutions at different levels redundantly contain
complementary information for crowd identification. Therefore, in this paper,
we proposed to construct our crowd counting network in a U-like encoder-
decoder structure for adequate feature reuse. Compared with traditional U-like
network, the big differences are that, in the encoder stage, regional
attention based dense connections are constructed to strength information
fusion specifically for crowd identification. Regional attention based skip-
connections between encoder and decoder stages and asymmetric multi-scale
residual blocks in the decoder stage are originally proposed to account for
information aggregation of different resolutions.

The main contributions can be summarized as followings:

  * •

We propose a Scale-Aware Crowd Counting Network (SACCN), which is concentrated
on overcoming adverse influences from cluttered background and scale
variations in crowd scenes. Comprehensive experiments demonstrate that our
SACCN achieves superior performance.

  * •

We design two attention modules for crowd identification under cluttered
backgrounds. One is the regional attention module (RAM), which calculates
spatial and channel-wise attention on shallow features. The other is the
semantic attention module (SAM) that imposes both spatial and channel self-
attention on deep feature maps. These two attention modules are
collaboratively employed in cascade and can effectively highlight crowd areas
from cluttered backgrounds.

  * •

We propose an asymmetric multi-scale module (AMM) to deal with the problem of
scale variations. The module employs asymmetric convolutions with different
scales to learn crowd density maps. Ablation experiments confirmed that it can
effectively improve the model performance and reduce the model parameters.

  * •

We design the regional attention based connections for complementary feature
reuse, including the dense connections among shallow layers of the encoder and
the skip connections between encoder and decoder. Ablation studies demonstrate
that both connections play important roles in performance improvement.

The remainders are organized as followings. Related works on crowd counting
are briefly reviewed in Section II. Implementation details of the proposed
methods are described in Section III. Experimental results and analysis are
discussed in Section IV. Conclusion is presented in Section V.

##  II Related Work

The methods of crowd counting can be roughly divided into three paradigms:
traditional detection-based methods, regression-based methods, and modern CNN-
based methods. In this section, we briefly review some typical related works
for completeness.

###  II-A Detection-based Methods

Traditional detection-based methods [18, 1] mainly focused on the
characteristics of people, like face, body, etc. Methods of this paradigm
usually employed sliding window-based detection schemes. Hand-crafted features
like Haar wavelets [2] and Histogram of Gradient (HoG) [19] were extracted
from images for object classification. Generally, these methods can provide
accurate count in scenes with sparse objects, but in scenes with complex
backgrounds and dense objects. These methods cannot work well when there exist
severe crowd occlusion and large variations on crowd scales.

###  II-B Regression-based Methods

In order to deal with complex scenes with dense crowds, some methods [20, 3]
proposed a regression method to avoid hard detection. These methods generally
learned regression functions from local features of image patches to counting
numbers. In these cases, spatial distribution information of people was often
ignored.

Some other researchers [21, 22] also proposed density-based regression methods
to take full advantage of annotation. However, variations in the crowd scale
and cluttered backgrounds are still the main obstacles.

Figure 2: The complete architecture of our proposed Scale-Aware Crowd Counting
Network (SACCN).

###  II-C CNN-based Methods

With the success of the convolution neural network (CNN) in the computer
vision field, the crowd counting community has also realized to make use of
the strong representation learning ability of CNN to estimate crowd density.
CNN-based methods [7, 8, 13, 11] have shown significant improvements over
previous methods.

Most of CNN-based methods address the problems of crowd scale variation and
cluttered backgrounds through strategies such as multi-column CNN [7], stacked
multi-branch blocks [15], multi-scale feature fusion [17, 13], selective
regression [23, 24] and visual attention [12].

Crowdnet [4] was the first work to generate a density map by using the
convolution network. It adopted a dual-column structure to solve crowd scale
variation problems. MCNN [7] adopted a multi-column structure for crowd
counting. CSRNet [8] adopted dilated convolutions that expanded the receptive
field without parameters increasing. SANet [25] stacked multi-branch blocks as
the feature extractor to overcome variations on crowd scales. MBTTBF-SCFB[17]
propose a multilevel bottom-top and top-bottom fusion method. SDANet [13]
found that using channel-wise attention in shallow layers can distinguish
background and crowd better than in deep layers. HA-CCN [11] uses spatial
attention in different levels of VGGNet [26] to distinguish the background and
the crowd.

The closest methods to ours are the latest SDANet [13] and the HA-CCN [11].
However, there exist many big differences. For example, they only apply single
attention, either limited channel-wise attention or spatial attention. In
contrast, our method combines channel-wise attention and spatial attention
together on feature maps at shallow layers to distinguish complicated
backgrounds and crowd. Moreover, we employ spatial and channel-wise self-
attention on deep layers at the same time to enhance discriminability of
semantic representations for crowd.

##  III Scale-Aware Crowd Counting Network

In this section, we describe the proposed Scale-Aware Crowd Counting
Network(SACCN) in detail. As shown in Fig. 2, our method is mainly composed of
three parts: encoder part, decoder part and regional attention based
connections. Specifically, in the encoder part, the pre-trained VGGNet16[26]
model is employed as the backbone network for feature extraction. Regional
attention based dense connections are constructed to specifically strengthen
information fusion for crowd identification. In the decoder part, asymmetric
multi-scale residual blocks are employed to account for information
aggregation of different resolutions. Regional attention based skip-
connections between encoder and decoder layers are for feature reuse and crowd
highlights. More details are explained in the following subsections.

###  III-A Attention Modules

Generally, crowd scene images not only contain people, but also backgrounds
such as buildings, trees, cars, etc. Attention has been proved to be a very
helpful mechanism for crowd counting [13, 11], especially when dealing with
clutter backgrounds. Therefore, we specifically propose both regional
attention and semantic attention modules for crowd representations
highlighting.

####  III-A1 Regional Attention Module

Figure 3: The structure of Regional Attention Module (RAM).

The structure of regional attention module (RAM) is shown in Fig. 3. It
cascade combines channel-wise attention and spatial attention in a holistic
block. The channel-wise attention learns to capture the importance of
different feature channels, helping suppress unnecessary noisy information.
Spatial attention is to focus on crowd relevant areas. Through explicitly
employing spatial attention on the feature maps processed by channel-wise
attention, the proposed network can be more easily encouraged to focus on
representations learning for the interested crowd areas.

Formally, given a feature map $X\in\mathbb{R}^{C\times H\times W}$, in
channel-wise attention, average-pooling and max-pooling are firstly performed
to produce pooled features $Y_{avg}\in\mathbb{R}^{C\times 1\times
1},Y_{max}\in\mathbb{R}^{C\times 1\times 1}$ respectively. Then, the $Y_{avg}$
and $Y_{max}$ pass through two fully-connected (FC) layers to produce
$F_{avg}$ and $F_{max}$, as shown Equation 1. Parameters of the two FC layers
are shared across the two paths.

$\displaystyle F_{avg}$
$\displaystyle=\text{FC}_{2}(\text{ReLU}(\text{FC}_{1}(Y_{avg}))),$ (1)
$\displaystyle F_{max}$
$\displaystyle=\text{FC}_{2}(\text{ReLU}(\text{FC}_{1}(Y_{max}))).$

Then, channel-wise attention vector $S_{a}$ is generated according to Equation
2.

$\small S_{a}=sigmoid(F_{avg}+F_{max}).$ (2)

Intermediate feature map $X_{c}$ is therefore obtained through element-wise
weighting on input $X$.

$\small X_{c}=X\otimes S_{a}.$ (3)

Similar to channel-wise attention, spatial attention in parallel employs max-
pooling and average-pooling on $X_{c}$ in channel direction to produce
$\hat{Y}_{max}\in\mathbb{R}^{1\times H\times W}$ and
$\hat{Y}_{avg}\in\mathbb{R}^{1\times H\times W}$ respectively. Then,
$\hat{Y}_{avg}$ and $\hat{Y}_{max}$ are concatenated in channel to get
$\hat{Y_{s}}$. We employ a convolution with kernel size $3\times 3$ on it to
produce a spatial attention map $S_{b}$.

$\small S_{b}=sigmoid(conv(\hat{Y_{s}})).$ (4)

Finally, the feature map $X_{s}=X_{c}\otimes S_{b}$ is taken as the output of
regional attention module, where $\otimes$ represents elementary
multiplication.

####  III-A2 Semantic Attention Module

Figure 4: The structure of the Spatial Self-Attention (SSA) and Channel-wise
Self-Attention (CSA). The $\odot$ represents the matrix multiplication.

Motivated by the success of self-attention [27, 28, 29] in many high-level
vision tasks, we explore to utilize it on crowd counting tasks.

The structure of the proposed semantic attention module (SAM) is shown in Fig.
2. It is mainly composed of spatial self-attention (SSA), channel-wise self-
attention (CSA), and some necessary convolutions. The structural details of
SSA and CSA are shown in Fig. 4.

In case of spatial self-attention (SSA), given a feature map
$X\in\mathbb{R}^{C\times H\times W}$, two channel-wise features are firstly
transformed to $C_{1}$-dimension feature space, both through convolution with
kernel size $1\times 1$. Then, the resulted feature map is reshaped to
$X_{1}\in\mathbb{R}^{(HW)\times C_{1}}$ and
$X_{2}\in\mathbb{R}^{C_{1}\times(HW)}$. A spatial attention map $W_{s}$ is
consequently generated through feature correlation as shown in Equation 5.

$\small W_{s}=softmax(X_{1}\odot X_{2}),$ (5)

where, $\odot$ represents matrix product.

In parallel, we perform feature transformation on input $X$ through a
convolution with kernel size $1\times 1$, and generate an intermediate feature
$X_{3}\in\mathbb{R}^{C\times(HW)}$.

The output $X_{ssa}$ of SSA block is then calculated in residual mode, as
shown in Equation 6.

$\small X_{ssa}=X+reshape(X_{3}\odot W_{s}),$ (6)

where, $\odot$ represents matrix product.

In case of channel-wise self-attention (CSA), given a feature map
$X\in\mathbb{R}^{C\times H\times W}$, we firstly reshape the feature map to
produce $X_{4}\in\mathbb{R}^{C\times(HW)}$ and $X_{5}\in\mathbb{R}^{(HW)\times
C}$. Then, we perform matrix product and softmax operations to generate a
channel-wise attention vector $W_{c}$, as shown in Equation 7.

$W_{c}=softmax(X_{4}\odot X_{5}).$ (7)

The $X_{4}\in\mathbb{R}^{(HW)\times C}$ is weighted by $W_{c}$. The output
$X_{csa}$ of CSA is therefore generated according to Equation 8.

$\small X_{csa}=X+reshape(X_{4}\odot W_{c}).$ (8)

###  III-B Asymmetric Multi-scale Module

Figure 5: The structure of the Asymmetric Multi-scale Module (AMM). $Convk_{1}\times k_{2}$ represents an asymmetric convolution with kernel size of $k_{1}\times k_{2}$, $p$ is the padding and $d$ stands of the dilated rate. TABLE I: Comparison results of different methods on the ShanghaiTech Part_A/Part_B, UCF_CC_50 and UCF_QNRF. Best results and Second results are highlighted. Method | Part_A | Part_B | UCF_CC_50 | UCF_QNRF
---|---|---|---|---
MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE
MCNN[7](CVPR’16) | 110.2 | 173.2 | 26.4 | 41.3 | 377.6 | 509.1 | 277.0 | 426.0
CP-CNN[24](ICCV’17) | 73.6 | 106.4 | 20.1 | 30.1 | 295.8 | 320.9 | - | -
Switch-CNN[23](CVPR’17) | 90.4 | 135.0 | 21.6 | 33.4 | 318.1 | 439.2 | 228.0 | 445.0
CSRNet[8](CVPR’18) | 68.2 | 115.0 | 10.6 | 16.0 | 266.1 | 397.5 | 135.4 | 207.4
SANet[25](ECCV’18) | 67.0 | 104.5 | 8.4 | 13.6 | 258.5 | 334.9 | - | -
ic-CNN[30](ECCV’18) | 69.8 | 117.3 | 10.7 | 16.0 | 260.9 | 365.5 | - | -
HA-CCN[11](TIP’19) | 62.9 | 94.9 | 8.1 | 13.4 | 256.2 | 348.4 | 118.1 | 180.4
MACC[31](TCSVT’19) | 61.8 | 100.0 | 8.6 | 13.3 | 245.4 | 349.3 | - | -
COBC[32](TCSVT’19) | 62.8 | 102.0 | 8.6 | 16.4 | 239.6 | 322.2 | 118.0 | 192.0
ADCrowdNet[12](CVPR’19) | 63.2 | 98.9 | 8.2 | 15.7 | 266.4 | 358.0 | - | -
CAN[33](CVPR’19) | 61.3 | 100.0 | 7.8 | 12.2 | 212.2 | 243.7 | 107.0 | 183.0
PSDDN[34](CVPR’19) | 85.4 | 159.2 | 16.1 | 27.9 | 359.4 | 514.8 | - | -
SPN+L2SM[35](ICCV’19) | 64.2 | 98.4 | 7.2 | 11.1 | 188.4 | 315.3 | - | -
MBTTBF-SCFB[17](ICCV’19) | 60.2 | 94.1 | 8.0 | 15.5 | 233.1 | 300.9 | 97.5 | 165.2
DSSINet[36](ICCV’19) | 60.6 | 96.1 | 6.9 | 10.4 | 216.9 | 302.4 | 99.1 | 159.2
SDANet[13](AAAI’20) | 63.6 | 101.8 | 7.8 | 10.2 | 227.6 | 316.4 | - | -
DUBNet[37](AAAI’20) | 64.4 | 106.8 | 7.7 | 12.5 | 243.8 | 329.3 | 105.6 | 180.5
HSRNet[38](ECAI’20) | 62.3 | 100.3 | 7.2 | 11.8 | - | - | - | -
KDMG [39](T-PAMI’20) | 63.8 | 99.2 | 7.8 | 12.7 | - | - | 99.5 | 173
SACCN (Ours) | 59.2 | 98.0 | 6.8 | 10.5 | 178.0 | 258.5 | 96.1 | 167.8

In crowd counting related works, multi-scale modules are widely used to solve
scale variations. Inspired by Inception-v3 [40] and ACNet [41], we propose to
use asymmetric convolutions for density estimation. As shown in Fig. 5, we
design an asymmetric multi-scale module (AMM) which approximates conventional
convolution with a kernel size of $k\times k$ by the summation of two
asymmetric convolutions with kernel size of $k\times 1$ and $1\times k$. The
$1\times 1$ convolution is employed for feature fusion after the concatenation
of multi-branch convolutions.

Moreover, in order to enlarge the receptive field without parameters increase,
all asymmetric convolutions are set to be dilated convolutions. In this paper,
the dilation rate is set to be 2. To ensure the size of the output feature
keeping the same as input, we set the padding for each convolution, such as
the padding (0,2) for $Conv1\times 3$ is and (0,4) for $Conv1\times 5$.

###  III-C Regional Attention based Connections

In previous work [17, 13], multi-scale fusion is verified to be helpful for
crowd counting. Therefore, in order to make full use of the redundant and
complementary information from different layers, we construct several kinds of
regional attention based connections in our crowd counting network. The
connections are shown in Fig. 2.

####  III-C1 Regional Attention Based Dense Connections

We employ the first 13 layers of VGG16 as the backbone network for feature
extraction. Besides, we add extra dense connections among layers $Conv2\\_2$,
$Conv3\\_3$, $Conv4\\_3$, and $Conv5\\_3$ for multi-resolution feature reuse.
In order to highlight crowd information while suppress background, we utilize
region attention mechanism in dense connection paths, as shown in Fig. 2. We
formulate the process in Equation 9.

$\displaystyle I_{3}$ $\displaystyle=Conv2\\_2\downarrow_{2},$ (9)
$\displaystyle I_{4}$
$\displaystyle=\texttt{RAM}(Conv2\\_2)\downarrow_{4}+Conv3\\_3\downarrow_{2},$
$\displaystyle I_{5}$
$\displaystyle=\texttt{RAM}(Conv2\\_2)\downarrow_{8}+\texttt{RAM}(Conv3\\_3)\downarrow_{4}+Conv4\\_3\downarrow_{2},$

where, $I_{k}$ is the input of the ”$Conv~{}\textbf{{k}}$” layer in encoder.
RAM represents the regional attention module, $\downarrow_{m}$ means down-
sampling by $m$ times .

####  III-C2 Regional Attention Based Skip-connections

As we know, when the encoder going deeper, the resolutions of the feature map
are becoming lower due to the down-sampling process. For adequate feature
reuse, skip-connections are adopted in a conventional U-like encoder-decoder
network. However, from the viewpoint of information reuse, convention skip
connections transfer information uniformly without specific focus. In our
method, we propose to introduce regional attention into skip-connections to
highlight crowd identification.

For the convenience of description, we denote the $k^{th}$ layer in the
decoder having the same size as its counterpart layer ”Conv $k$” in the
encoder. We formulate the skip connections process as Equation 10 and Equation
11.

$\small D_{k}=\left\\{\begin{aligned} &Conv_{1\times
1}([\texttt{AMM}(E_{k});D_{k+1}])\uparrow^{2},k=4,3,2\\\
&\texttt{SAM}(E_{5})\uparrow^{2},k=5\end{aligned}\right.$ (10)

$\small E_{k}=\left\\{\begin{aligned}
&\texttt{RAM}(Conv\texttt{k})+D_{k+1},k=4,3,2\\\
&\texttt{RAM}(Conv5),k=5\end{aligned}\right.$ (11)

where, $D_{k}$ stands the output of the $k^{th}$ counterpart layer in decoder,
$E_{k}$ represents the fused features after skip connections at the $k^{th}$
counterpart layer in decoder. RAM (or SAM) represents the Regional Attention
Module (or Semantic Attention Module). AMM represents Asymmetric Multi-scale
Module. $Conv_{1\times 1}$ represents convolution layer with kernel size of
$1\times 1$. ”$[*;*]$” represents concatenation in channel. $\uparrow^{k}$
represents $k$ times Upsampling.

##  IV Experiments

In this section, we will describe our experimental evaluations in details.
Firstly, experiment settings including training details, evaluation metrics
and datasets are introduced. Then, comparisons among the proposed SACCN with
some popular state-of-the-art approaches are performed on several public
benchmarks. Finally, in ablation studies, the effects of key components
proposed in SACCN is investigated and analyzed.

###  IV-A Experimental Setup

####  IV-A1 Training Details

The network is trained end-to-end using the Adam optimizer with a learning
rate of 0.0001 and a momentum of 0.9 on single NVIDIA 2080Ti GPU. The batch
size is set to be 4. Mean Squared Error (MSE) is taken as the loss function to
constrain the network output and ground truth. Following HA-CCN, we employ
similar data augmentation methods on training data, such as randomly cropping
$400\times 400$ image patches from the images, and randomly flipping patches
horizontally with a probability of 0.5.

####  IV-A2 Evaluation Metrics

Following most of crowd counting methods, we use mean absolute error (MAE) and
mean square error (MSE) as evaluation metrics. The smaller the value of MAE
and MSE, the better the performance.

$\small MAE=\frac{1}{N}\sum_{i=1}^{N}|C_{i}-\hat{C_{i}}|$ (12) $\small
MSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}|C_{i}-\hat{C_{i}}|^{2}}$ (13)

where, $C_{i}$ is the predicted counting number, and $\hat{C_{i}}$ stands for
the ground truth.

For vehicle counting, we use GAME metric[42] on TRANCOS, as defined in
Equation 14:

$\small\text{GAME}^{L}=\frac{1}{N}\sum_{i}\sum_{l=1}^{2^{L}}|C_{i}^{l}-\hat{C_{i}^{l}}|$
(14)

where $\text{GAME}^{L}$ means the case that images are divided into $2^{L}$
non-overlapping regions.

###  IV-B Datasets

We conduct experiments on several benchmark datsets, including ShanghaiTech
Part_A and Part_B [6], UCF_CC_50 [43], UCF-QNRF [44], WorldExpo’10 [6], Mall
[3]. and the TRANCOS [42] dataset.

####  IV-B1 ShanghaiTech Dataset

ShanghaiTech dataset [6] is composed of 1198 annotated images, including
330165 annotated people. The dataset consists of Part_A and Part_B. Part_A
contains 482 images of highly congested scenes randomly downloaded from the
Internet. Among them, 300 images are for training and 182 images are for the
test. Part_B contains 716 images of relatively sparse crowd scene taken from
streets in Shanghai. There are 400 images in the training set and 316 images
in the test set.

TABLE II: Comparison results of different methods on the WorldExpo’10 Dataset. Best results and Second results are highlighted. Method | S1 | S2 | S3 | S4 | S5 | Average
---|---|---|---|---|---|---
MCNN[7] | 3.4 | 20.6 | 12.9 | 13.0 | 8.1 | 11.6
MSCNN[45] | 7.8 | 15.4 | 14.9 | 11.8 | 5.8 | 11.7
CP-CNN[24] | 2.9 | 14.7 | 10.5 | 10.4 | 5.8 | 8.86
ic-CNN[30] | 17.0 | 12.3 | 9.2 | 8.1 | 4.7 | 10.3
SANet[25] | 2.6 | 13.2 | 9.0 | 13.3 | 3.0 | 8.2
CSRNet[8] | 2.9 | 11.5 | 8.6 | 16.6 | 3.4 | 8.6
ADCrowdNet[12] | 1.7 | 14.4 | 11.5 | 7.9 | 3.0 | 7.7
CAN[33] | 2.9 | 12.0 | 10.0 | 7.9 | 4.3 | 7.4
MACC[31] | 2.2 | 11.5 | 11.6 | 13.9 | 2.5 | 8.34
COBC[32] | 2.1 | 11.5 | 8.1 | 16.8 | 2.5 | 8.2
DSSINet[36] | 1.6 | 9.5 | 9.5 | 10.4 | 2.5 | 6.7
ZoomCount[46] | 2.1 | 15.3 | 9.0 | 10.3 | 4.5 | 8.3
SDANet[13] | 2.0 | 14.3 | 12.5 | 9.5 | 2.5 | 8.1
SACCN (Ours) | 1.5 | 10.6 | 8.1 | 8.9 | 2.7 | 6.4

####  IV-B2 UCF_CC_50 Dataset

UCF_CC_50 dataset [43] is an extremely challenging crowd dataset. All 50
images of different resolutions are randomly collected from the Internet. The
number of people in these images varies greatly in a wide range from 94 to
4543. There are a total of 63974 head annotations. The average number per
image is 1280. Besides, the dataset contains diverse scenes with varying
perspective distortions. For better verification of model accuracy, we follow
standard setting [43] and conduct 5-fold cross-validation for the test.

####  IV-B3 UCF_QNRF Dataset

UCF_QNRF dataset [44] is a recently proposed large-scale crowd dataset. It
contains 1,535 images in which there are about 1.25 million head annotations.
These images have many different scenes, backgrounds, and perspectives. The
training dataset contains 1201 training images. The test dataset containing
334 images. Many of them are in very high-resolution. Therefore, we transform
all images to a fixed size of $1024\times 768$ before any pre-processing.

####  IV-B4 WorldExpo’10 Dataset

WorldExpo’10 dataset [6] is a large scale data-driven cross-scene crowd
counting dataset which is collected from Shanghai 2010 WorldExpo. The dataset
consists of 1132 annotated video sequences captured by 108 different
surveillance cameras. Each camera with different view angles provides the
region of interest (ROI). The whole dataset contains a total of 3920 frames in
a fixed resolution of $576\times 720$. It is divided into a training set with
3380 frames and a test set with 600 frames.

####  IV-B5 Mall Dataset

Mall dataset [3] is a dataset with relatively sparse crowd scenes. The dataset
contains 2000 frames in a video, which are captured by using a surveillance
camera installed in a shopping mall. The scenes significantly vary in the
crowd’s scale and appearances due to perspective distortion. In addition,
there exist challenging occlusions caused by objects in the scene. We use the
first 800 frames as the training set, and the left 1200 frames as the test set
to evaluate our model.

Figure 6: The visualization of experiment samples on different datasets. The number below image indicates the number of crowd. Figure 7: Visualization of estimated density map and intermediate RAM attention map of SACCN. The number below image indicates the number of crowd. TABLE III: Comparison results of different methods on the TRANCOS Dataset. Best results and Second results are highlighted. Method | GAME 0 | GAME 1 | GAME 2 | GAME 3
---|---|---|---|---
Fiaschi et al.[47] | 17.77 | 20.14 | 23.65 | 25.99
Lempitsky et al.[21] | 13.76 | 16.72 | 20.72 | 24.367
Hydra-3s[48] | 10.99 | 13.75 | 16.69 | 19.32
CSRNet[8] | 3.67 | 5.82 | 8.53 | 14.40
PSDDN[34] | 4.79 | 5.43 | 6.68 | 8.40
COBC[32] | 3.15 | 5.45 | 8.34 | 15.02
HSRNet[38] | 3.03 | 4.57 | 6.46 | 9.68
KDMG[39] | 3.13 | 4.79 | 6.20 | 8.68
SACCN (Ours) | 2.20 | 3.11 | 4.18 | 5.77
TABLE IV: Comparison results of different methods on the Mall Dataset. Best results and Second results are highlighted. Method |  MAE |  MSE
---|---|---
CNN-Boosting[49] |  2.01 |  -
W-VLAD[50] |  2.67 |  11.72
ConvLSTM[51] |  2.24 |  8.5
DecideNet[10] |  1.52 |  1.9
E3D[52] |  1.64 |  2.13
SAAN[53] |  1.28 |  1.68
AT-CNN[54] |  2.28 |  2.9
Deem-CFCN[55] |  2.11 |  2.71
FMLF[56] |  1.85 |  2.34
HSRNet[38] |  1.8 |  2.28
SACCN (Ours) |  1.42 |  1.85

####  IV-B6 TRANCOS Dataset

TRANCOS dataset [42] is the first one for vehicle counting in traffic jam
scenes. It contains 1244 traffic images with vehicle numbers varying from 9 to
107. The dataset is often used to evaluate the generalization ability of
population counting methods.

###  IV-C Experiment Results

We compare our proposed SACCN with state-of-the-art methods on ShanghaiTech
Part_A/Part_B, UCF_CC_50, UCF_QNRF, WorldExpo’10, Mall, and TRANCOS.

From TABLE I, on the ShanghaiTech Part_A/Part_B, UCF_CC_50, and UCF_QNRF, it
is not difficult to observe that the proposed SACCN achieves the lowest MAE in
almost all datasets, showing more superiority over the other methods.

The comparison results on the WorldExpo’10 dataset are shown in TABLE II. From
TABLE II, we can see that the proposed SACCN delivers top-2 accuracy in 3 out
of 5 scenes and achieves the best accuracy on average.

For the Mall dataset and TRANCOS dataset, the comparison results are shown in
TABLE IV and TABLE III respectively. In the case of the Mall dataset, the
proposed SACCN achieves second lower MAE and MSE among all methods. In the
case of the TRANCOS dataset, the proposed method delivers the best results
according to the GAME metric.

All the quantitative comparison results on different datasets indicate that
our method is extremely robust to changes in scale, perspective and density.
More visualized results on different datasets are shown in Fig. 6. The
counting numbers are recorded below each GT and estimated density map.

From Fig. 6, it can be seen that the distribution of the generated density map
by SACCN is very similar to the distribution of GT. Even though with a large
span of counting numbers in Fig. 6, the proposed SACCN can still accurately
estimate the density map, which implies its robustness.

In addition, in Fig. 7, to display the effect of RAM, we visualize the RAM
attention map for some samples with different backgrounds, perspectives,
densities, and crowd scales. From Fig. 7, we can observe that the RAM
attention map accurately highlights the crowd areas and suppresses the non-
crowd areas. This shows that RAM components can effectively distinguish the
crowd, which helps to count more accurately.

###  IV-D Ablation Studies

In this section, we conduct several ablation studies to demonstrate the
effects of key components used in the proposed SACCN. Moreover, we further
verify the influences of the proposed components on mitigating cluttered
background and scale variation. All ablation experiments are conducted on
ShanghaiTech Part_B dataset.

TABLE V: The effects of the proposed components on ShanghaiTech Part_B. DC represents the regional attention based dense connections. SC is the regional attention based skip-connections. Method | MAE | MSE | RAM | SAM | AMM | DC | SC
---|---|---|---|---|---|---|---
SACCN-ram | 8.3 | 12.9 | - | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
SACCN-sam | 7.2 | 11.7 | $\checkmark$ | - | $\checkmark$ | $\checkmark$ | $\checkmark$
SACCN-amm | 6.9 | 10.8 | $\checkmark$ | $\checkmark$ | - | $\checkmark$ | $\checkmark$
SACCN-dc | 7.0 | 11.0 | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | $\checkmark$
SACCN-sc | 7.1 | 11.4 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | -
SACCN | 6.8 | 10.5 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$
TABLE VI: The effects of different attention on ShanghaiTech Part_B. CA is the channel attention and SP is the spatial attention. Method | MAE | MSE | CA | SP | RAM
---|---|---|---|---|---
$\text{SACCN}_{CA}$ | 7.8 | 12.3 | $\checkmark$ | - | -
$\text{SACCN}_{SP}$ | 7.6 | 11.5 | - | $\checkmark$ | -
SACCN | 6.8 | 10.5 | - | - | $\checkmark$
Figure 8: Visualization on the effects of the proposed key components. The
number below image indicates the GT or estimated number of crowd.

####  IV-D1 The effects of the proposed components

TABLE V shows the performance of all the proposed components used in the
proposed SACCN. The DC specifically represents the regional attention based
dense connections. The SC represents the regional attention based skip-
connections.

Firstly, without RAM, from the comparison between the first row ”SACCN-ram”
and the last row ”SACCN”, we can find that the performance has severely
deteriorated, indicating that region identification is very helpful for crowd
counting.

Secondly, from the comparison between the second row ”SACCN-sam” and the last
row ”SACCN”, we can infer that at deep layers, using semantic self-attentions
is beneficial. This is because the feature maps in deeper layers have stronger
semantics and can help to distinguish the crowd and the background in semantic
space.

Thirdly, to demonstrate the effect of asymmetric convolution, the ablation
method ”SACCN-amm” uses the convention convolution with $3\times 3$ instead of
the asymmetric convolution with $1\times 3$ and $3\times 1$, and use the
convention convolution with $5\times 5$ instead of the asymmetric convolution
with $1\times 5$ and $5\times 1$ in AMM. From the comparison result in TABLE
V, it is unsurprising to find the performance gap between convention
convolution and asymmetric convolution is very small. However, we should point
out that herein asymmetric convolutions have fewer parameters than the
convention convolution, which can greatly reduce the model size.

Finally, after comparing the cases that remove dense connections or skip-
connections from the proposed SACCN, we can conclude that combining features
of different scales can improve the accuracy of crowd counting. This method
can also prevent gradient vanishing caused by network deepening.

####  IV-D2 The effects of different attentions

In order to verify the effect of different attention, we respectively replace
RAM with single channel attention(CA) and spatial attention (SP).

In the TABLE VI, the first row shows the results that use CA instead of RAM.
The second row shows the results that use SP instead of RAM. Through the
performance comparison, we can see that the performance of only using CA or SP
is worse than using RAM. Therefore, we have reasons to infer that, when
distinguishing the crowd area from the background, both spatial attention and
channel attention are important and complementary.

TABLE VII: The effects of the proposed components on mitigating cluttered background and scale variation. Method | MAE | MSE | clutter background | scale variation
---|---|---|---|---
RAM | SAM | AMM | DC | SC
SACCN/all | 8.9 | 14.4 | - | - | - | - | -
SACCN/cb | 8.3 | 13.7 | - | - | $\checkmark$ | $\checkmark$ | $\checkmark$
SACCN/sv | 7.7 | 12.7 | $\checkmark$ | $\checkmark$ | - | - | -
SACCN | 6.8 | 10.5 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$

####  IV-D3 The effects of the proposed key components on mitigating adverse
influences of cluttered background and scale variation

As we have summarized, cluttered background and scale variation are the main
concentrated challenges in crowd counting. In this paper, we have proposed
SACCN with some key innovative components to mitigate them. Among these
components, RAM and SAM are attention modules mainly focusing on crowd
identification from cluttered backgrounds. AMM, DC and SC mainly focus on
scale variation. For a better understanding of the effects of these key
components, we conduct more experiments (TABLE VII) to demonstrate their
respective performance on mitigating the mentioned obstacles. The ablation
results are shown in TABLE VII.

From TABLE VII, the comparison results indicate that each group of the
proposed components can effectively improve the performance. Moreover, from
Fig. 8, we can observe that the density maps generated by the proposed SACCN
are more similar to the GT. The method without the proposed components cannot
accurately count the crowd in the image.

##  V Conclusion

In this paper, we proposed a Scale-Aware Crowd Counting Network (SACCN) that
can efficiently deal with the task of crowd counting under clutter background
and scale variations. In the proposed SACCN, regional attention and semantic
attention are employed for highlighting crowd area. Meanwhile, the asymmetric
multi-scale module is specifically designed to handle the scale variations
problem, and regional attention based connections are constructed for feature
reuse and representation enhancement. Extensive experiments demonstrated that
the proposed SACCN achieves superior performance against other state-of-the-
art methods.

## References

  * [1] P. Dollar, C. Wojek, B. Schiele, and P. Perona, “Pedestrian detection: An evaluation of the state of the art,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 34, no. 4, pp. 743–761, 2011.
  * [2] P. Viola and M. J. Jones, “Robust real-time face detection,” _International journal of computer vision_ , vol. 57, no. 2, pp. 137–154, 2004.
  * [3] K. Chen, C. C. Loy, S. Gong, and T. Xiang, “Feature mining for localised crowd counting.” in _BMVC_ , vol. 1, no. 2, 2012, p. 3.
  * [4] L. Boominathan, S. S. Kruthiventi, and R. V. Babu, “Crowdnet: A deep convolutional network for dense crowd counting,” in _Proceedings of the 24th ACM international conference on Multimedia_ , 2016, pp. 640–644.
  * [5] K. Chen, S. Gong, T. Xiang, and C. Change Loy, “Cumulative attribute space for age and crowd density estimation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 2467–2474.
  * [6] C. Zhang, H. Li, X. Wang, and X. Yang, “Cross-scene crowd counting via deep convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 833–841.
  * [7] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 589–597.
  * [8] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1091–1100.
  * [9] H. Zheng, Z. Lin, J. Cen, Z. Wu, and Y. Zhao, “Cross-line pedestrian counting based on spatially-consistent two-stage local crowd density estimation and accumulation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 3, pp. 787–799, 2018.
  * [10] J. Liu, C. Gao, D. Meng, and A. G. Hauptmann, “Decidenet: Counting varying density crowds through attention guided detection and density estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5197–5206.
  * [11] V. A. Sindagi and V. M. Patel, “Ha-ccn: Hierarchical attention-based crowd counting network,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 323–335, 2019.
  * [12] N. Liu, Y. Long, C. Zou, Q. Niu, L. Pan, and H. Wu, “Adcrowdnet: An attention-injective deformable convolutional network for crowd understanding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3225–3234.
  * [13] Y. Miao, Z. Lin, G. Ding, and J. Han, “Shallow feature based dense attention network for crowd counting.” in _AAAI_ , 2020, pp. 11 765–11 772.
  * [14] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [15] V. A. Sindagi and V. M. Patel, “Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting,” in _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 2017, pp. 1–6.
  * [16] M. Shi, Z. Yang, C. Xu, and Q. Chen, “Revisiting perspective information for efficient crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7279–7288.
  * [17] V. A. Sindagi and V. M. Patel, “Multi-level bottom-top and top-bottom feature fusion for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1002–1012.
  * [18] B. Leibe, E. Seemann, and B. Schiele, “Pedestrian detection in crowded scenes,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 878–885.
  * [19] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 886–893.
  * [20] A. B. Chan and N. Vasconcelos, “Bayesian poisson regression for crowd counting,” in _IEEE 12th international conference on computer vision_. IEEE, 2009, pp. 545–551.
  * [21] V. Lempitsky and A. Zisserman, “Learning to count objects in images,” _Advances in neural information processing systems_ , vol. 23, pp. 1324–1332, 2010.
  * [22] V.-Q. Pham, T. Kozakaya, O. Yamaguchi, and R. Okada, “Count forest: Co-voting uncertain number of targets using random forest for crowd density estimation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 3253–3261.
  * [23] D. B. Sam, S. Surya, and R. V. Babu, “Switching convolutional neural network for crowd counting,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2017, pp. 4031–4039.
  * [24] V. A. Sindagi and V. M. Patel, “Generating high-quality crowd density maps using contextual pyramid cnns,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1861–1870.
  * [25] X. Cao, Z. Wang, Y. Zhao, and F. Su, “Scale aggregation network for accurate and efficient crowd counting,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 734–750.
  * [26] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [27] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [28] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _International Conference on Machine Learning_ , 2019, pp. 7354–7363.
  * [29] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3146–3154.
  * [30] V. Ranjan, H. Le, and M. Hoai, “Iterative crowd counting,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 270–285.
  * [31] S. Jiang, X. Lu, Y. Lei, and L. Liu, “Mask-aware networks for crowd counting,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [32] L. Liu, H. Lu, H. Xiong, K. Xian, Z. Cao, and C. Shen, “Counting objects by blockwise classification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [33] W. Liu, M. Salzmann, and P. Fua, “Context-aware crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5099–5108.
  * [34] Y. Liu, M. Shi, Q. Zhao, and X. Wang, “Point in, box out: Beyond counting persons in crowds,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6469–6478.
  * [35] C. Xu, K. Qiu, J. Fu, S. Bai, Y. Xu, and X. Bai, “Learn to scale: Generating multipolar normalized density maps for crowd counting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8382–8390.
  * [36] L. Liu, Z. Qiu, G. Li, S. Liu, W. Ouyang, and L. Lin, “Crowd counting with deep structured scale integration network,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1774–1783.
  * [37] M.-h. Oh, P. A. Olsen, and K. N. Ramamurthy, “Crowd counting with decomposed uncertainty.” in _AAAI_ , 2020, pp. 11 799–11 806.
  * [38] Z. Zou, Y. Liu, S. Xu, W. Wei, S. Wen, and P. Zhou, “Crowd counting via hierarchical scale recalibration network,” _arXiv preprint arXiv:2003.03545_ , 2020.
  * [39] J. Wan, Q. Wang, and A. B. Chan, “Kernel-based density map generation for dense object counting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [40] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [41] X. Ding, Y. Guo, G. Ding, and J. Han, “Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1911–1920.
  * [42] R. Guerrero-Gómez-Olmedo, B. Torre-Jiménez, R. López-Sastre, S. Maldonado-Bascón, and D. Onoro-Rubio, “Extremely overlapping vehicle counting,” in _Iberian Conference on Pattern Recognition and Image Analysis_. Springer, 2015, pp. 423–431.
  * [43] H. Idrees, I. Saleemi, C. Seibert, and M. Shah, “Multi-source multi-scale counting in extremely dense crowd images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 2547–2554.
  * [44] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. Al-Maadeed, N. Rajpoot, and M. Shah, “Composition loss for counting, density map estimation and localization in dense crowds,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 532–546.
  * [45] L. Zeng, X. Xu, B. Cai, S. Qiu, and T. Zhang, “Multi-scale convolutional neural networks for crowd counting,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 465–469.
  * [46] U. Sajid, H. Sajid, H. Wang, and G. Wang, “Zoomcount: A zooming mechanism for crowd counting in static images,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2020.
  * [47] L. Fiaschi, U. Köthe, R. Nair, and F. A. Hamprecht, “Learning to count with regression forest and structured labels,” in _Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012)_. IEEE, 2012, pp. 2685–2688.
  * [48] D. Onoro-Rubio and R. J. López-Sastre, “Towards perspective-free object counting with deep learning,” in _European Conference on Computer Vision_. Springer, 2016, pp. 615–629.
  * [49] E. Walach and L. Wolf, “Learning to count with cnn boosting,” in _European conference on computer vision_. Springer, 2016, pp. 660–676.
  * [50] B. Sheng, C. Shen, G. Lin, J. Li, W. Yang, and C. Sun, “Crowd counting via weighted vlad on a dense attribute feature map,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 28, no. 8, pp. 1788–1797, 2016\.
  * [51] F. Xiong, X. Shi, and D.-Y. Yeung, “Spatiotemporal modeling for crowd counting in videos,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 5151–5159.
  * [52] Z. Zou, H. Shao, X. Qu, W. Wei, and P. Zhou, “Enhanced 3d convolutional networks for crowd counting,” _arXiv preprint arXiv:1908.04121_ , 2019.
  * [53] M. Hossain, M. Hosseinzadeh, O. Chanda, and Y. Wang, “Crowd counting using scale-aware attention networks,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1280–1288.
  * [54] M. Zhao, J. Zhang, C. Zhang, and W. Zhang, “Leveraging heterogeneous auxiliary tasks to assist crowd counting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 736–12 745.
  * [55] M. Zhao, C. Zhang, J. Zhang, F. Porikli, B. Ni, and W. Zhang, “Scale-aware crowd counting via depth-embedded convolutional neural networks,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [56] X. Ding, F. He, Z. Lin, Y. Wang, H. Guo, and Y. Huang, “Crowd density estimation using fusion of multi-layer features,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020.