# AdaMixer: A Fast-Converging Query-Based Object Detector

Ziteng Gao1 Limin Wang1 ✉ Bing Han2 Sheng Guo2
1State Key Laboratory for Novel Software Technology, Nanjing University, China
2MYbank, Ant Group, China

###### Abstract

Traditional object detectors employ the dense paradigm of scanning over
locations and scales in an image. The recent query-based object detectors
break this convention by decoding image features with a set of learnable
queries. However, this paradigm still suffers from slow convergence, limited
performance, and design complexity of extra networks between backbone and
decoder. In this paper, we find that the key to these issues is the
adaptability of decoders for casting queries to varying objects. Accordingly,
we propose a fast-converging query-based detector, named AdaMixer, by
improving the adaptability of query-based decoding processes in two aspects.
First, each query adaptively samples features over space and scales based on
estimated offsets, which allows AdaMixer to efficiently attend to the coherent
regions of objects. Then, we dynamically decode these sampled features with an
adaptive MLP-Mixer under the guidance of each query. Thanks to these two
critical designs, AdaMixer enjoys architectural simplicity without requiring
dense attentional encoders or explicit pyramid networks. On the challenging MS
COCO benchmark, AdaMixer with ResNet-50 as the backbone, with 12 training
epochs, reaches up to 45.0 AP on the validation set along with 27.9 APs in
detecting small objects. With the longer training scheme, AdaMixer with
ResNeXt-101-DCN and Swin-S reaches 49.5 and 51.3 AP. Our work sheds light on a
simple, accurate, and fast converging architecture for query-based object
detectors. The code is made available at https://github.com/MCG-NJU/AdaMixer.

†† ✉: Corresponding author (lmwang@nju.edu.cn).

##  1 Introduction

Figure 1: Convergence curves of our AdaMixer, DETR, Deformable DETR and Sparse
R-CNN with ResNet-50 as the backbone on MS COCO minival set.

Object detection has been a fundamental task in the computer vision area for
decades, as it aims to locate varying objects in a single image and categorize
them. For a long time, researchers have used spatial dense prior on grids in
an image to cover potential objects with great variations. This dense paradigm
dates back from sliding window methods [12, 43, 38] and still remains
prevalent in anchor-based or point-based detectors [16, 36, 25, 28, 34, 40,
53] at the age of convolutional neural networks. Although dense priors has
been a dominant role in object detection for its remarkable performance to
cover potential objects, they are criticized for several shortcomings in
various aspects, including anchor designs [48, 35, 44], training sample
selection [52, 51, 15, 14], and post-processing operators on potential
redundant detections [2, 19].

Though sorts of remedies on these issues are proposed every year, the
underlying detection scheme with dense grid-like prior had remained almost
untouched for a long time. Recently, query-based object detectors [4, 56, 39]
bring a new perspective on object detection, that is, to use learnable
embeddings, also termed queries, to directly represent potential objects by
using attention-like operators [42]. This scheme, on the other hand, requires
a strong representation power of the network to cast limited queries to
potential objects to cope with great variations of objects across images.
However, the adaptability of currently employed query decoders to the image
content is limited on both how to spatially sample features and how to process
sampled features. For instance, attention-based decoder in DETR-like detectors
[4, 56] are adaptive on which feature to sample but remain static on how to
decode it, whereas dynamic interaction head in Sparse R-CNN [39] goes vice
versa. The insufficiency in the adaptability to different images leaves the
current decoder in a dilemma between limited query representation power and
great variations of objects. Also, as compensation for this, query-based
object detectors usually bring extra attentional encoders or explicit pyramid
necks after the backbone and before the query decoder, in order to involve
more semantic or multi-scale modeling, such as TransformerEncoder [42],
MultiScaleDeformableTransformerEncoder [56] and FPN [24]. These extra
components result in the higher complexity of built detection pipelines in
both design and computational aspects. In addition, detectors with them are
hungry for more training time and rich data augmentation due to the introduced
modules.

In this paper, we present a fast-converging and accurate query-based object
detector with a simplified architecture, named AdaMixer, to mitigate issues
above. Specifically, to effectively use queries to represent objects, AdaMixer
introduces the adaptive 3D feature sampler and the adaptive mixing of channel
semantics and spatial structures holistically. First, by regarding multi-scale
feature maps from the backbone as a 3D feature space, our proposed decoder can
flexibly sample features over space and scales to adaptively handle both of
location and scale variations of objects based on queries. Then, the adaptive
mixing applies the channel and spatial mixing to the sampled features with
dynamic kernels under the guidance of queries. The adaptive location sampling
and holistic content decoding notably enhances the adaptability of queries to
varying images in detecting varying objects. As a result, AdaMixer is simply
made up of a backbone network and our proposed decoder without extra
attentional encoders or explicit pyramid networks.

Experimental results show that in a standard 12 epochs training ($1\times$
training scheme) with the random flipping as the only augmentation, our
AdaMixer with ResNet-50 [18] as the backbone achieves 42.7, 44.1, and 45.0 AP
on MS COCO validation set under the settings of 100, 300, and 500 queries,
with 24.7, 27.0, and 27.9 APs in small object detection. With longer $3\times$
training time and stronger data augmentation aligned to other query-based
detectors, our AdaMixer with ResNet-101, ResNeXt-101-DCN [47, 55], and Swin-S
[29] achieves 48.0, 49.5, and 51.3 AP with the single scale and single model
testing, significantly outperforming the previous state-of-the-art query-based
detectors. We hope AdaMixer, as a simply-designed, fast-converging, relatively
efficient, and more accurate object detector, will serve as a strong baseline
in the future research for the query-based object detection.

| adaptive to decode locations? | adaptive to decode content? |  extra networks before the query decoder1?
---|---|---|---
DETR [4] |  yes, multi-head attention aggregation |  no, linear projection | TransformerEncoder
Deformable DETR [56] |  yes, multi-scale multi-head adaptive sampling |  no, linear projection2 | Multi-scale DeformTransEncoder
Sparse R-CNN [39] |  restricted, RoIAlign [17] |  partially yes, adaptive point-wise conv. | FPN
AdaMixer (ours) |  yes, adaptive 3D sampling |  yes, adaptive channel and spatial mixing | linear projection to form 3D feature space

Table 1: Comparisons of the adaptability of decoders across different query-
based object detectors. 1We specify trainable networks introduced after pre-
trained backbones before the query decoder. 2We regard the softmax aggregation
in deformable attention as one step in decoding locations as the softmax
weights normalize to one.

##  2 Related Work

Dense object detectors. The dense paradigm of object detectors dates back to
sliding window-based approaches [43, 11, 12], which involve exhaustive
classification over space and scales due to the assumption of potential
objects emerging uniformly and densely w.r.t. spatial locations in an image.
This assumption about natural images remains effective in the deep learning
era for its power to cover potential objects [38]. Prevalent object detectors
in the past few years, _e.g_., one-stage detectors [50, 28, 34, 35, 40, 53],
multiple-stage detectors [36, 17, 3, 5, 32] or point-based methods [53, 21,
54, 10], are also rooted in this dense assumption in either region proposal
networks or entire object detector architectures. They apply dense priors,
such as anchors or anchor points, upon the feature map to exhaustively find
foreground objects or directly classify them.

Query-based object detectors. Recently transformer-based detector, DETR [4],
formulates object detection as a direct set prediction task and achieves
promising performance. DETR predicts a set of objects by attending queries to
the feature map with the transformer decoder. The original architecture of
DETR is simply based on the Transformer [42], which contains the multi-layer
attentional encoder and decoder. The training for set prediction in DETR is
based on the bipartite matching between the predictions and ground-truth
objects. While DETR outperforms competitive Faster R-CNN baselines, it still
suffers from limited spatial resolution, poor small object detection
performance, and slow training convergence. There have been several work to
tackle these issues. Deformable DETR [56] considers the shift-equivalence in
natural images and introduces the multi-scale deformable family of attention
operators in both encoders and decoders of DETR. SMCA [13], Conditional DETR
[31], and Anchor DETR [46] explicitly model the positional attention for
foreground objects to fasten the convergence. Efficient DETR [49] bridges the
dense prior to queries in DETR to improve the performance. Sparse R-CNN [39]
brings the query-based paradigm of DETR to Cascade R-CNN [3] and introduces
the dynamic instance interaction head and its query-adaptive point-wise
convolution, to effectively cast queries to potential objects.

Our AdaMixer generally follows this research line of using queries to attend
features for object detection. However, we improve the query-based object
detection paradigm from a new perspective: the adaptability of decoding
queries across images. Specifically, we focus on how to make decoding scheme
on queries more adaptive to the content of images from both semantic and
spatial aspects. We present adaptive 3D feature sampling and adaptive content
decoding to improve its flexibility to relate queries with each image. This
makes AdaMixer a fast-converging query-based object detector without the
introduction of extra feature encoders or explicit pyramid networks.

##  3 Approach

In this paper, we focus on the query decoder in query-based object detectors
since the decoder design is essential to casting learned queries to potential
objects in each image. We first revisit decoders in popular query-based object
detectors from the perspective of semantic and positional adaptability, and
then elaborate on our proposed adaptive query decoder.

###  3.1 Object Query Decoder Revisited

Plain attention decoders. DETR [4] applies plain multi-head cross attention
between queries and features to cast object queries to potential objects. As
depicted in Table 1, the cross attention decoder is adaptive to decode
sampling locations in the sense that it exploits the relation of object
queries and features to aggregate features. However, the linear transformation
of features after aggregation fails to adaptively decode them based on the
query.

Deformable multi-scale attention decoders. Deformable DETR [56] improves the
ability of decoding sampling locations in plain cross attention in terms of
shift equivalence and scale invariance by introducing explicit reference
points and multi-scale features. But like DETR, the content decoding of
sampled features still remains static by the linear transformation. Overall,
decoders in DETR and Deformable DETR lack the reasoning of aggregated features
conditionally on the query and thus limit the semantic adaptability of queries
to features. As a result, both of them require stacks of extra attentional
encoders to enrich feature semantics.

RoIAlign and dynamic interactive head as decoders. Sparse R-CNN [39], as the
intersection between region-based and query-based detectors, uses the RoIAlign
operator and dynamic interactive head as the query decoder. The dynamic
interactive head uses point-wise convolutions, whose kernel is adaptive based
on the query, to process RoI features. This enables the adaptability of
queries to RoI features but only partially, in the sense that the adaptive
point-wise convolution can not infer adaptive spatial structures from those
features to build queries. Moreover, the sampling locations by RoIAlign
operator [17] are restricted inside of the box indicated by a query and a
specific level in FPN [24], which limits positional adaptability and requires
explicit pyramid networks for multi-scale modeling.

Summary. Given a limited number of queries and varying potential objects
across images, an ideal decoder should consider both the semantic and
positional adaptability of such queries to the content of images, that is, how
to adaptively decode sampling locations and sampled content. This naturally
motivates our design of AdaMixer.

###  3.2 Our Object Query Definition

Starting from the object query definition, we associate two vectors with a
query following our semantic and positional view of decoders: one is the
content vector $\mathbf{q}$ and the other is the positional vector
$(x,y,z,r)$. This is also in line with [39, 46, 56] to disentangle the
location or the represented bounding box of a query from its content. The
content vector $\mathbf{q}$ is a vector in $\mathbb{R}^{d_{q}}$ and $d_{q}$ is
the channel dimension. The vector $(x,y,z,r)$ describes scaled geometric
properties of the bounding box indicated by a query, that is, the x- and
y-axis coordinates of its center point and the logarithm of its scale and
aspect ratio. The $x,y,z$ components also directly represent coordinates of a
query in the 3D feature space, which will be introduced below.

Decoding the bounding box from a query. We can simply decode the bounding box
from the positional vector. The center $(x_{B},y_{B})$ and the width and
height $w_{B}$ and $h_{B}$ of the indicated bounding box can be decoded:

$\displaystyle x_{B}=s_{\rm base}\cdot x,\hskip 16.99998pty_{B}=s_{\rm
base}\cdot y,$ (1) $\displaystyle w_{B}=s_{\rm base}\cdot 2^{z-r},h_{B}=s_{\rm
base}\cdot 2^{z+r},$ (2)

where $s_{\rm base}$ is the base downsampling stride offset and we set $s_{\rm
base}=4$ according to the stride of the largest feature map we use in the
experiments.

Figure 2: 3D feature sampling process. A query first obtains sampling points
in the 3D feature space and then perform 3D interpolation on these sampling
points.

Figure 3:  Adaptive mixing procedure between an object query and sampled
features. The object query first generates adaptive mixing weights and then
apply these weights to mix sampled features in the channel and spatial
dimension. Note that for clarity, we demonstrate adaptive mixing for one
sampling group.

###  3.3 Adaptive Location Sampling

As discussed in Section 3.1, the decoder should adaptively decide which
feature to sample regarding the query. That is, the decoder should decode
sampling locations with the consideration of both the positional vector
$(x,y,z,r)$ and content vector $\mathbf{q}$. Also, we argue that the decoder
must be adaptive not only over $(x,y)$ space but also be flexible in scales of
potential objects. Specifically, we can accomplish these goals by regarding
multi-scale features as a 3D feature space and adaptively sampling features
from it.

Multi-scale features as the 3D feature space. Given a feature map, indexed
$j$, with the downsampling stride $s^{\mathrm{feat}}_{j}$ from the backbone,
we first transform them by a linear layer to the same channel $d_{\rm feat}$
and compute its z-axis coordinate:

$\displaystyle z^{\mathrm{feat}}_{j}=\log_{2}(s^{\mathrm{feat}}_{j}/s_{\rm
base}).$ (3)

Then we virtually rescale the height and width of feature maps of different
strides to the same ones $H/s_{\rm base}$ and $W/s_{\rm base}$, where $H$ and
$W$ is the height and width of the input image, and put them aligned on x- and
y-axis in the 3D space as depicted in Figure 2. These feature maps are
supporting planes for the 3D feature space, whose interpolation is described
below.

Adaptive 3D feature sampling process. A query first generate $P_{\rm in}$ sets
of offset vectors to $P_{\rm in}$ points, $\\{(\Delta x_{i},\Delta
y_{i},\Delta z_{i})\\}_{P_{\rm in}}$, where each offset vector is indexed by
$i$, and depends on its content vector $\mathbf{q}$ by a linear layer:

$\displaystyle\\{(\Delta x_{i},\Delta y_{i},\Delta z_{i})\\}_{P_{\rm in}}={\rm
Linear}(\mathbf{q}).$ (4)

Then, these offsets are transformed to sampling locations according to the
positional vector of the query for every $i$:

$\displaystyle\begin{cases}\tilde{x}_{i}=x+\Delta x_{i}\cdot 2^{z-r},\\\
\tilde{y}_{i}=y+\Delta y_{i}\cdot 2^{z+r},\\\ \tilde{z}_{i}=z+\Delta
z_{i},\end{cases}$ (5)

It is worth noting that the area $\\{\Delta x_{i},\Delta
y_{i}\in[-0.5,0.5]\\}$ describes the bounding box decoded from the query. Our
offsets are not restricted to this range, meaning that a query can sample
features “out of the box”. Having obtained
$\\{(\tilde{x}_{i},\tilde{y}_{i},\tilde{z}_{i})\\}_{P_{\rm in}}$, our sampler
samples values given these points in the 3D space. In the current
implementation, the interpolation over the 3D space is in the compositional
manner: it first samples values given points by bilinear interpolation in the
$(x,y)$ space and then interpolates over the z-axis by gaussian weighting
given a sampling $\tilde{z}$, where the weight for the $j$-th feature map is:

$\displaystyle\tilde{w}_{j}={\exp(-(\tilde{z}-z^{\mathrm{feat}}_{j})^{2}/\tau_{z})\over\sum_{j}\exp(-(\tilde{z}-z^{\mathrm{feat}}_{j})^{2}/\tau_{z})},$
(6)

where $\tau_{z}$ is the softing coefficient for interpolating values over the
z-axis and we keep $\tau_{z}=2$ in this work. With the feature map of the
channel $d_{\rm feat}$, the shape of sampled feature matrix $\mathbf{x}$ is
$\mathbb{R}^{P_{\rm in}\times d_{\rm feat}}$. The adaptive 3D feature sampling
process eases the decoder learning by sampling features with explicit,
adaptive and coherent locations and scales regarding a query.

Group sampling. To sample as many points as possible, we introduce the group
sampling mechanism, analogous to multiple heads in attentional operators [42]
or groups in group convolution [47]. The group sampling first splits the
channel $d_{\rm feat}$ of the 3D feature space into $g$ groups, each with the
channel $d_{\rm feat}/g$, and performs 3D sampling individually for each
group. With the group sampling mechanism, the decoder can generate $g\cdot
P_{\rm in}$ offset vectors for a query to enrich the diversity of sampling
points and exploit richer spatial structure of these points. Sampled feature
matrix $\mathbf{x}$ now are of the shape $\mathbb{R}^{g\times P_{\rm
in}\times(d_{\rm feat}/g)}$. The grouping mechanism is also applied to the
adaptive mixing for efficiency as described below, and we term the group
sampling and mixing unified as the grouping mechanism.

Figure 4: Our decoder structure of the AdaMixer. There are two operator
streams on a query: one on its content vector $\mathbf{q}$ (the solid
horizontal line) and one on its positional vector $(x,y,z,r)$ (the dashed
horizontal line). Each operator on the content vector in the decoder is
followed by a residual addition and LayerNorm.

###  3.4 Adaptive Content Decoding

With features sampled, how to adaptively decode them is another key design in
our AdaMixer decoder. To capture correlation in spatial and channel dimension
of $\mathbf{x}$, we propose to efficiently decode the content in each
dimension separately. Specifically, we design a simplified and adaptive
variant of MLP-mixer [41], termed as adaptive mixing, with dynamic mixing
weights similar to dynamic filters in convolutions [20]. As shown in Figure 3,
the procedure contains sequentially the adaptive channel mixing and adaptive
spatial mixing to involve both adaptive channel semantics and spatial
structures under the guidance of a query.

Adaptive channel mixing. Given sampled feature matrix
$\mathbf{x}\in\mathbb{R}^{P_{\rm in}\times C}$ for a query in a group, where
$C=d_{\rm feat}/g$, the adaptive channel mixing (ACM) is to use the dynamic
weight based on $\mathbf{q}$ to transform features $\mathbf{x}$ on the channel
dimension to adaptively enhance channel semantics:

$\displaystyle M_{c}$ $\displaystyle={\rm
Linear}(\mathbf{q})\in\mathbb{R}^{C\times C}$ (7) $\displaystyle{\rm
ACM}(\mathbf{x})$ $\displaystyle={\rm ReLU}({\rm
LayerNorm}(\mathbf{x}M_{c})),$ (8)

where ${\rm ACM}(\mathbf{x})\in\mathbb{R}^{P_{\rm in}\times C}$ is the channel
mixed feature output and the linear layer is individual for each group. The
layer normalization [1] is applied to both dimensions of the mixed output.
Note that in this step, the dynamic weight is shared across different sampling
points in 3D space, analogous to adaptive $1\times 1$ convolution in [39] on
RoI features.

Adaptive spatial mixing. To enable the adaptability of a query to spatial
structures of sampled features, we introduce the adaptive spatial mixing (ASM)
process. As depicted in Figure 3, ASM can be described as firstly transposing
channel mixed feature matrix and applying the dynamic kernel to the spatial
dimension of it:

$\displaystyle M_{s}$ $\displaystyle={\rm
Linear}(\mathbf{q})\in\mathbb{R}^{P_{\rm in}\times P_{\rm out}}$ (9)
$\displaystyle{\rm ASM}(\mathbf{x})$ $\displaystyle={\rm ReLU}({\rm
LayerNorm}(\mathbf{x}^{T}M_{s})),$ (10)

where ${\rm ASM}(\mathbf{x})\in\mathbb{R}^{C\times P_{\rm out}}$ is the
spatial mixed output and $P_{\rm out}$ is the number of spatial mixing out
patterns. Note that the dynamic weight is shared across different channels. As
sampling points may be from different feature scales, ASM naturally involves
multi-scale interaction modeling, which is necessary for high performance
object detection.

The adaptive mixing procedure overall is depicted in Figure 3, where the
adaptive spatial mixing follows the adaptive channel mixing, both applied in a
sampling group. The final output of the shape $\mathbb{R}^{g\times C\times
P_{\rm out}}$ across group is flattened and transformed to the $d_{q}$
dimension by a linear layer to add back to the content vector.

###  3.5 Overall AdaMixer Detector

Like the decoder architecture in [4, 56], we place the self-attention between
queries, our proposed adaptive mixing, and feedforward-feed network (FFN)
sequentially in a stage of the decoder regarding the query content vector
$\mathbf{q}$, as shown in Figure 4. The query positional vector is updated by
another FFN at the end of each stage:

$\displaystyle x^{\prime}=x+\Delta x\cdot 2^{z},y^{\prime}=y+\Delta y\cdot
2^{z},$ (11) $\displaystyle z^{\prime}=z+\Delta z,\hskip
17.09996ptr^{\prime}=r+\Delta r,$ (12)

where $(\Delta x,\Delta y,\Delta z,\Delta r)$ is produced by the lower small
FFN block in Figure 4.

Position-aware multi-head self-attentions. Since we disentangle the content
and position for a query, the naive multi-head self-attention between the
content vectors of queries is not aware of what geometric relation between a
query and another query is, which is proven beneficial to suppress redundant
detections [4]. To achieve this, we embed positional information into the
self-attention. Our positional embedding for the content vector in the
sinusoidal form and every component of $(x,y,z,r)$ takes up a quarter of
channels. We also embed the intersection over foreground (IoF) as a bias to
the attention weight between queries to explicitly incorporate the relation of
being contained between queries. The attention for each head is

$\displaystyle{\rm Attn}(Q,K,V)={\rm Softmax}\left(QK^{T}/\sqrt{d_{q}}+\alpha
B\right)V,$ (13)

where $B_{ij}=\log\left({|{\rm box}_{i}\cap{\rm box}_{j}|/|{\rm
box}_{i}}|+\epsilon\right)$, $\epsilon=10^{-7}$, $Q,K,V\in\mathbb{R}^{N\times
d_{q}}$, standing for the query, key and value matrix in the self-attention
procedure, and $\alpha$ is a learnable scalar for each head. The $B_{ij}=0$
stands for the box $i$ being totally contained in the box $j$ and
$B_{ij}=\log\epsilon\ll 0$ indicates that there is no overlapping between $i$
and $j$.

Overall AdaMixer detector. The detection pipeline of AdaMixer is only composed
of a backbone and a AdaMixer decoder. It avoids adding explicit feature
pyramid networks or attentional encoders between backbone and decoder. The
AdaMixer directly gathers predictions of decoded queries as final object
detection results.

##  4 Experiments

In this section, we first elaborate on the implementation and training
details. Then we compare our models with other competitive detectors with
limited training epochs. Next, we perform ablation studies on the design of
our detector. We also align the training recipe to other query-based detectors
and compare our AdaMixer to them fairly.

###  4.1 Implementation Details

Dataset. We conduct extensive experiments on MS COCO 2017 dataset [26] in
mmdetection codebase [6]. Following the common practice, we use trainval35k
subset consisting up of 118K images to train our models and use minival subset
of 5K images as the validation set.

query dim $d_{q}$ |  feat. maps used |  feat. dim $d_{\rm feat}$ |  #stages in decoder |  #groups $g$ | $C$ | $P_{\rm in}$ | $P_{\rm out}$
---|---|---|---|---|---|---|---
256 | $C_{2}\sim C_{5}$ | 256 | 6 | 4 | 64 | 32 | 128

Table 2: Default configuration of our AdaMixer detector.

Configurations. The default hyper-parameters in our AdaMixer detector is
elaborated in Table 2. We configure the dimension of the query content vector
$d_{q}$ to $256$ following previous query-based work [4, 56, 39]. We use
feature maps $C_{2}\sim C_{5}$ from the backbone network. Multi-scale features
are processed by the linear transformation to the channel $d_{\rm feat}=256$
as supporting planes for the 3D feature space. The number of decoder stages is
set to 6 also following the common practice of query-based detectors. The
mixer grouping number is set to 4 as default. Accordingly, the channel of
sampled features per group is $C=d_{\rm feat}/g=64$. Also, the number of
sampling points $P_{\rm in}$ and mixing out patterns $P_{\rm out}$ per group
is set to $32$ and $128$. The hidden dimension of FFN on the content vector in
the decoder is set to 2048. The FFN dimension for classification and updating
positional vectors is set to 256.

Initializations. For training stability in early iterations, we initialize the
parameters of linear layers to produce dynamic parameters or sampling offsets
as follows: zeroing weights of these layers and initializing biases as
expected. This helps stable training by enforcing models to learn the
adaptability from zeros. The biases for the linear layer producing sampling
offest vectors is initialized in the way that $\Delta x_{i},\Delta y_{i}$ are
uniformly drawn in $[-0.5,0.5]$ and $\Delta z_{i}=-1$ for all $i$ to align
with the RoIAlign [17] level strategy. The bias in the linear layer to produce
mixing weights follows the default initialization in the PyTorch. We also
initialize all the query positional vectors into decoders such that the their
boxes and sampling points cover the whole image in the initial decoder stage
like [39]. Backbones are initialized from pre-trained models on the ImageNet
1K dataset [9].

detector | epochs | AP | AP50 | AP75 | APs | APm | APl
---|---|---|---|---|---|---|---
FCOS [40] | 12 | 38.7 | 57.4 | 41.8 | 22.9 | 42.5 | 50.1
Cascade R-CNN [3] | 12 | 40.4 | 58.9 | 44.1 | 22.8 | 43.7 | 54.0
GFocalV2 [22] | 12 | 41.1 | 58.8 | 44.9 | 23.5 | 44.9 | 53.3
BorderDet [33] | 12 | 41.4 | 59.4 | 44.5 | 23.6 | 45.1 | 54.6
Dynamic Head [8] | 12 | 42.6 | 60.1 | 46.4 | 26.1 | 46.8 | 56.0
DETR [4] | 12 | 20.0 | 36.2 | 19.3 | 6.0 | 20.5 | 32.2
Deformable DETR [56] | 12 | 35.1 | 53.6 | 37.7 | 18.2 | 38.5 | 48.7
Sparse R-CNN [39] | 12 | 37.9 | 56.0 | 40.5 | 20.7 | 40.0 | 53.5
AdaMixer ($N$=100) | 12 | 42.7 | 61.5 | 45.9 | 24.7 | 45.4 | 59.2
AdaMixer ($N$=300) | 12 | 44.1 | 63.4 | 47.4 | 27.0 | 46.9 | 59.5
AdaMixer ($N$=500) | 12 | 45.0 | 64.2 | 48.6 | 27.9 | 47.8 | 61.1

Table 3: $\mathbf{1}\times$ training scheme performance on COCO minival set
with different detectors and ResNet-50 as backbone.

Losses and optimizers. Following [56, 39], the training loss is the matching
loss consisting of the focal loss [25] with coefficient $\lambda_{\rm cls}=2$,
L1 bounding box loss with $\lambda_{L_{1}}=5$ and GIoU loss [37] with
$\lambda_{\rm giou}=2$. We use AdamW [30] as our optimizer with weight decay
0.0001. The initial learning rate is $2.5\times 10^{-5}$.

adaptive |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
loc. |  cont. |
|  |  35.7 |  55.2 |  37.8 |  20.1 |  38.1 |  48.8
✓ |  |  37.3 |  55.8 |  39.7 |  20.7 |  40.1 |  50.9
|  ✓ |  40.4 |  60.5 |  43.4 |  23.0 |  42.5 |  56.7
✓ |  ✓ |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2
(a)

mixing |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
ACM |  ACM |  41.5 |  60.5 |  44.3 |  23.5 |  44.1 |  57.4
ASM |  ASM |  39.8 |  58.8 |  42.6 |  22.8 |  42.4 |  56.1
ACM |  ASM |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2
ASM |  ACM |  41.5 |  60.4 |  44.5 |  23.9 |  44.4 |  57.1
(b)

pyramid |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
FPN [24] |  42.1 |  61.0 |  45.0 |  24.1 |  44.8 |  58.7
PAFPN [27] |  41.7 |  60.5 |  44.7 |  23.5 |  44.6 |  58.7
- |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2

(c)

$P_{\rm in}$ |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
8 |  41.2 |  60.3 |  44.1 |  24.0 |  43.9 |  57.2
16 |  41.8 |  60.9 |  44.5 |  24.5 |  44.6 |  58.4
32 |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2
64 |  42.7 |  61.5 |  46.1 |  24.9 |  45.5 |  59.3
(d)

$P_{\rm out}$ |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
32 |  41.1 |  60.0 |  44.0 |  24.5 |  43.6 |  57.2
64 |  42.1 |  61.2 |  45.0 |  24.0 |  44.8 |  57.8
128 |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2
256 |  42.4 |  61.4 |  45.5 |  24.4 |  45.0 |  58.7

(e)

pos. inf. |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
sinus. |  IoF |
|  |  41.2 |  59.6 |  44.2 |  23.6 |  43.5 |  57.9
✓ |  |  41.5 |  59.9 |  44.3 |  23.6 |  44.0 |  57.8
|  ✓ |  42.2 |  61.2 |  45.0 |  24.8 |  45.1 |  58.8
✓ |  ✓ |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2
(f)

Table 4: AdaMixer ablation experiments with ResNet-50 on MS COCO minival set.
Default choice for our model is colored gray

Training recipes. We adopt two versions of training recipes for a fair
comparison with different detectors. The first one adopts the classic
$\mathbf{1\times}$ training scheme, which includes a budget of 12 training
epochs with training images of shorter side resized to 800. This recipe
includes the random horizontal flipping as only the standard data augmentation
and allocates 100 learnable object queries to our AdaMixer detector, to
compare with popular and competitive detectors like FCOS [40] and Cascade
R-CNN [3] fairly. The second training recipe is to align with other query-
based detectors, which leverages more training epochs and performs crop and
multi-scale augmentation in [4, 56, 39]. Our second training recipe adopts the
same data augmentation and has a budget of 36 training epochs, namely
$\mathbf{3\times}$ training scheme. It uses 300 object queries to compare
fairly with [56, 39]. The learning rate is divided by a factor of 10 at epoch
8 and 11 in $1\times$ training scheme or at epoch 24 and 33 in $3\times$
training scheme, scaled proportionally.

The $1\times$ and $3\times$ training scheme for AdaMixer with ResNet-50
typically take about 9 and 29 hours on 8 V100 cards. During the inference
stage, we input images of the shorter size resized to 800 without data
augmentation. We leave more details about model training and inference and
visualizations in the supplementary material.

###  4.2 Fast Convergence with Limited Budgets

We first investigate our proposed AdaMixer with limited training epochs and
limited data augmentation, namely $1\times$ training scheme. For a fair
comparison, we disable the commonly-used crop and multi-scale data
augmentation in query-based detectors and allocate only 100 queries or
learnable proposals for these detectors. Experimental results are shown in
Table 3. AdaMixer with $N=100$ queries achieves 42.7 AP, outperforming state-
of-the-art traditional and query-based detectors with a limited training
budget. Moreover, if we increase the number of queries $N$ to 300 and 500, the
performance of the AdaMixer detector reaches 44.1 and 45.0 AP, especially with
27.0 and 27.9 APs in detecting small objects. It is worth noting that these
results are achieved with random flipping as the only data augmentation and
within 12 training epochs, showing that AdaMixer can be supervised efficiently
with training samples.

###  4.3 Ablation Studies

Due to the limited computational resource, we use ResNet-50 as the backbone
network and $1\times$ training scheme to perform ablation studies.

Decoding adaptability. We begin our ablations with the key component, the
adaptability design, in our AdaMixer detector. The adaptability in AdaMixer is
in two aspects: adaptive sampling for decoding locations and adaptive mixing
for decoding content. Table LABEL:tab:adaptabilityDecoding investigates the
performance under the condition of whether or not we enable the adaptability
in decoding sampling locations and decoding content. The cancellation for
adaptability on locations or content stands for enforcing weights of linear
layers producing sampling offsets or mixing weights all to zeros during the
training and inference. Only biases of these layers can be learned during the
training procedure, which are eventually not adaptive based on the query
content $\mathbf{q}$. In other words, all sampling offsets or mixing weights
are the same across different queries and different images with the
cancellation. As shown in Table LABEL:tab:adaptabilityDecoding, the
adaptability in both decoding sampling locations and sampled content is
essential to a good query-based object detector, which outperforms a non-
adaptive counterpart by 7.0 AP.

Adaptive mixing design. Moving forward, we compare different designs of our
adaptive mixing in Table LABEL:tab:mixingDesign. As shown in Figure 3, our
default design for adaptive mixing is to mix features first on the channel
dimension and then on the spatial dimension. We perform ablations by placing
only channel mixing, only spatial mixing, and the reversed order of our design
as three variants. The first adaptive channel mixing and then spatial one lead
to the best performance. This indicates that channel semantics and spatial
structures are both important to the mixing design. For the reversed mixing
variant, we suspect that the inferior result is due to the insufficient
channel semantics into spatial mixing as features are directly from the
backbone.

detector | backbone | encoder/pyramid net | #epochs | GFLOPs | AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---|---|---|---|---
DETR [4] | ResNet-50-DC5 | TransformerEnc | 500 | 187 | 43.3 | 63.1 | 45.9 | 22.5 | 47.3 | 61.1
SMCA [13] | ResNet-50 | TransformerEnc | 50 | 152 | 43.7 | 63.6 | 47.2 | 24.2 | 47.0 | 60.4
Deformable DETR [56] | ResNet-50 | DeformTransEnc | 50 | 173 | 43.8 | 62.6 | 47.7 | 26.4 | 47.1 | 58.0
Sparse R-CNN [39] | ResNet-50 | FPN | 36 | 174 | 44.5 | 63.4 | 48.2 | 26.9 | 47.2 | 59.5
Efficient DETR [49] | ResNet-50 | DeformTransEnc | 36 | 210 | 45.1 | 63.1 | 49.1 | 28.3 | 48.4 | 59.0
Conditional DETR [31] | ResNet-50-DC5 | TransformerEnc | 108 | 195 | 45.1 | 65.4 | 48.5 | 25.3 | 49.0 | 62.2
Anchor DETR [46] | ResNet-50-DC5 | DecoupTransEnc | 50 | 151 | 44.2 | 64.7 | 47.5 | 24.7 | 48.2 | 60.6
AdaMixer (ours) | ResNet-50 | - | 12 | 132 | 44.1 | 63.1 | 47.8 | 29.5 | 47.0 | 58.8
AdaMixer (ours) | ResNet-50 | - | 24 | 132 | 46.7 | 65.9 | 50.5 | 29.7 | 49.7 | 61.5
AdaMixer (ours) | ResNet-50 | - | 36 | 132 | 47.0 | 66.0 | 51.1 | 30.1 | 50.2 | 61.8
DETR [4] | ResNet-101-DC5 | TransformerEnc | 500 | 253 | 44.9 | 64.7 | 47.7 | 23.7 | 49.5 | 62.3
SMCA [13] | ResNet-101 | TransformerEnc | 50 | 218 | 44.4 | 65.2 | 48.0 | 24.3 | 48.5 | 61.0
Sparse R-CNN [39] | ResNet-101 | FPN | 36 | 250 | 45.6 | 64.6 | 49.5 | 28.3 | 48.3 | 61.6
Efficient DETR [49] | ResNet-101 | DeformTransEnc | 36 | 289 | 45.7 | 64.1 | 49.5 | 28.2 | 49.1 | 60.2
Conditional DETR [31] | ResNet-101-DC5 | TransformerEnc | 108 | 262 | 45.9 | 66.8 | 49.5 | 27.2 | 50.3 | 63.3
AdaMixer (ours) | ResNet-101 | - | 36 | 208 | 48.0 | 67.0 | 52.4 | 30.0 | 51.2 | 63.7
AdaMixer (ours) | ResNeXt-101-DCN | - | 36 | 214 | 49.5 | 68.9 | 53.9 | 31.3 | 52.3 | 66.3
AdaMixer (ours) | Swin-S | - | 36 | 234 | 51.3 | 71.2 | 55.7 | 34.2 | 54.6 | 67.3
Table 5: Different query-based detector performance on COCO minival set with
longer training scheme and single scale testing.

Extra pyramid networks. AdaMixer enjoys the simplicity for circumventing extra
attentional encoders or explicit pyramid networks. Instead, AdaMixer improves
the semantic and multi-scale modeling in the decoder. The adaptive 3D sampling
and following spatial mixing naturally enable multi-scale feature modeling and
enable queries to handle scale variations of objects. In Table
LABEL:tab:extraNeck, we investigate the performance of the AdaMixer detector
with introduction of extra pyramid networks. Models with these extra networks
might require a longer training time and more training samples to perform
well. These results are in favor of our AdaMixer design as a simplified query-
based detector.

Sampling points and spatial mixing out patterns. Table LABEL:tab:pin and
LABEL:tab:pout shows the ablation on the sampling points $P_{\rm in}$ and
spatial mixing out patterns $P_{\rm out}$ per group. The performance is
generally related to the number of sampling points $P_{\rm in}$ and spatial
mixing out patterns $P_{\rm out}$. A good balance between the complexity and
performance is $P_{\rm in}=32$ and $P_{\rm out}=128$, where the performance
saturates for $P_{\rm in}$ and decreases for $P_{\rm out}$ beyond this point.

Positional information in attention between queries. In Section 3.5, we
propose to embed the positional information into the self-attention between
the content vectors of queries. In addition to the regular sinusoidal
positional embedding, we also hardwire the intersection over foreground (IoF)
into the attention weight between boxes indicated by queries. We investigate
these two ingredients in Table LABEL:tab:positionalInformation. Results show
that combining these two ingredients notably increases the performance. The
individual effect of the IoF is also compelling. We argue that the IoF between
boxes, which describes the geometric relation of being contained directly for
corresponding queries, is important for the self-attention to imitate the NMS
procedure [4].

$g$ | 1 | 2 | 4 | 8
---|---|---|---|---
AP | 42.5 | 42.8 | 42.7 | 41.9
FLOPs | 111G | 106G | 104G | 106G
Params | 191M | 148M | 135M | 149M
Table 6: Grouping sampling and mixing with $g$ groups.

Mixer group number. The mixer grouping encourages the decoder sampler to
sample more diverse points. This also reduces total parameters and
computational costs by mixing divided groups of features. We here evaluate the
effect of the grouping mechanism with various $g$ in Table 6. The model
reaches the least FLOPs and number of parameters with $4$ mixer groups with
promising performance.

###  4.4 Comparison with Other Query-based Detectors

We present the final results of our AdaMixer and perform the comparison
between our AdaMixer and other state-of-the-art query-based detectors in Table
5. We use the $3\times$ training scheme to train our AdaMixer, which allocates
300 queries and includes the stronger data augmentation to align with the
common practice of other query-based methods. Specifically, we train our
AdaMixer with ResNet-50, ResNet-101, ResNeXt-101 [47] with deformable
convolution layers [55] and Swin-S [29] backbones. We also proportionally
stretch the training schedule for AdaMixer with ResNet-50 to investigate the
faster convergence speed, as depicted in Figure 1. AdaMixer with assorted
backbones significantly outperforms competitive query-based object detectors
with less computational cost. With bounding boxes as only supervising signals,
AdaMixer with Swin-S reaches 51.3 AP and 34.2 APs with the single scale
testing. Moreover, among these query-based detectors, only AdaMixer does not
require the extra attentional encoders and explicit pyramid networks. These
results demonstrate our AdaMixer is a simply-architected, effective, and fast-
converging query-based object detector.

##  5 Conclusion and Limitation

In this paper, we have presented a fast-converging query-based object
detection architecture, termed AdaMixer, to efficiently and effectively decode
objects from images. Our proposed AdaMixer improves the decoder of query-based
detectors with adaptive 3D sampling and adaptive channel and spatial mixing.
By improving query decoders, AdaMixer circumvents the requirement of extra
network modeling between backbone and decoder. Our AdaMixer achieves superior
performance, especially on small object detection, with less computational
cost compared to other query-based detectors. Moreover, it enables the fast
convergence speed with limited training budgets. We hope that AdaMixer can
serve as a strong baseline for future research.

The limitation in our AdaMixer is that though we have applied the grouping
mechanism, the total parameter number remains a little bit large. This is
mainly due to a large number of parameters in the linear layer to produce
dynamic mixing weights. We leave the question of how to further reduce the
number of parameters to the future work.

#### Acknowledgements.

This work is supported by National Natural Science Foundation of China
(No.62076119, No.61921006), Program for Innovative Talents and Entrepreneur in
Jiangsu Province, and Collaborative Innovation Center of Novel Software
Technology and Industrialization. Part of the work is done during the
internship of Ziteng at MYbank.

## Appendix

## A. 1. Detection Performance on COCO test set

backbone | AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
ResNet-50 | 47.2 | 66.3 | 51.4 | 28.3 | 49.3 | 60.6
ResNet-101 | 48.1 | 67.3 | 52.4 | 28.3 | 50.6 | 62.1
ResNeXt-101-DCN | 49.3 | 68.8 | 53.7 | 30.0 | 51.6 | 64.0
Swin-S | 51.3 | 71.3 | 55.9 | 31.4 | 53.9 | 66.3
Table 7: AdaMixer performance on COCO test-dev set with longer training scheme
(36 epochs training and 300 queries) and single model single scale testing.

The performance of AdaMixer on COCO minival set is reported in Table 5 in the
paper. Here we report the performance of these AdaMixer models on COCO test-
dev set in Table 7, where labels are not publicly available and evaluation is
done on the online server. Note that models here are exactly the same in Table
5 in the paper.

## A. 2. More Studies on Adaptive 3D Sampling

sampling |  AP |  AP50 |  AP75 |  APs |  APm |  APl
---|---|---|---|---|---|---
only C3 feature |  26.2 |  42.0 |  27.3 |  15.8 |  28.7 |  34.1
only C4 feature |  38.3 |  57.2 |  41.0 |  20.0 |  41.9 |  54.1
only C5 feature |  37.8 |  58.3 |  39.5 |  18.0 |  41.2 |  51.7
RoIAlign |  37.2 |  58.5 |  39.0 |  19.0 |  39.3 |  55.6
A2DS |  41.3 |  61.0 |  44.4 |  23.3 |  43.8 |  57.8
A3DS |  42.7 |  61.5 |  45.9 |  24.7 |  45.4 |  59.2
Table 8: Different feature sampling performance on MS COCO minival set with
$1\times$ training scheme. The “A3DS” is abbreviation for Adaptive 3D Sampling
used in AdaMixer. The “A2DS” is the 2D variant of adaptive 3D sampling, where
it disables learning adaptive z-axis offsets, namely, $\Delta z_{(\cdot)}=0$
through training and inference. Note that A2DS is still with multiple feature
maps and $z$ of a query still decides how feature maps are aggregated. The
RoIAlign is performed to extract features from feature maps individually
according to the bounding box indicated by a query. The row “only C(⋅)
feature” variants use a single feature map to perfom adaptive 2D sampling. No
FPNs used throughout this table.

We here also validate the effectiveness of our proposed 3D feature space and
adaptive 3D sampling in Table 8. Using single feature map leads to inferior
results for our AdaMixer. Note that there is no feature pyramid networks (FPN)
[24] used. The RoIAlign operator, which extracts features in a single level
map according to the bounding box, also performs inferior to our adaptive 3D
sampling approach. RoIAlign feature sampling locations are highly stricted
inner the bounding box and the sampling level (the $z$ coordinate in our work)
is also discretized, whereas locations and levels are adaptive on queries and
not stricted in our adaptive 3D sampling . This means that RoIAlign lacks
multi-level feature modeling and necessitates multi-scale feature interaction
necks.

Relations to other work. Our adaptive 3D sampling is in accord with deformable
convolutions[7, 55, 56] to explicitly utilize offsets to sample features
spatially to model deformation of objects. But our adaptive 3D sampling
extends the adaptive modeling to the scale dimension, namely z-axis, to cope
with the variation of potential objects. Also, our adaptive 3D sampling is
performed by queries in the sparse manner, whereas the deformable convolution
or attention are usually performed in the dense manner. Moreover, our method
naturally generalizes the scale-equalizing operators [23, 45] on discretized
feature maps with different scales to a continuous and interpolable one,
enjoying more flexible modeling for scale variations.

## A. 3. Model Details

Our AdaMixer is implemented on mmdetection framework [6]. In current
implementation, the model is fully based on PyTorch primitives without
customized CUDA codes.

detector | backbone |  AP${}_{\text{val}}$ |  AP${}_{\text{test}}$ |  #Param (M) |  flops (G) | FPS |  training hours
---|---|---|---|---|---|---|---
Faster R-CNN | R-50 | 41.8 | - | 42 | 207 | 17 |  $\sim$16
DETR | R-50-DC5 | 43.3 | - | 41 | 187 | 11 |  $\sim$204
Deformable DETR++ | R-50 | 46.2 | 46.9 | 40 | 173 | 12 |  $\sim$106
Sparse R-CNN | R-50 | 44.5 | - | 110 | 174 | 16 |  $\sim$30
AdaMixer | R-50 | 47.0 | 47.2 | 139 | 132 | 15 | 29
AdaMixer | R-101 | 48.0 | 48.1 | 158 | 208 | 12 | 37
AdaMixer | X-101-DCN | 49.5 | 49.3 | 160 | 214 | 8 | 65
AdaMixer | Swin-S | 51.3 | 51.3 | 164 | 234 | 10 | 51
Table 9:  Training and inference details about different models. AdaMixer
models here are ones in Table 5 of the paper with 300 queries.
AP${}_{\text{val}}$ and AP${}_{\text{test}}$ are the average precision on COCO
minival and test-dev set, respectively. FPS is calculated on a single Nvidia
V100 card. Training hours for other detectors are estimated on 8 V100 cards
when obtaining the speed of a few training iterations.

Initializations. We initialize all backbones from pre-trained model on
ImageNet 1K classification [9], including Swin-S [29]. Please refer to our
codebase for more detailed initializations in AdaMixer.

Training and inference speed. We evaluate AdaMixer training and inference
speed and compare it with other detectors in Table 9. Our AdaMixer shows
advantage not only on theoretical flops but on the actual training time and
inference FPS. Note that the current implementation, the adaptive 3D sampling
procedure is based on PyTorch primitive grid_sample, which can be optimized in
the future.

## A. 4. Visualizations

We also show visualizations of sampling points and final detections of our
AdaMixer detector in Figure 5 and 6. These visualizations are doned on samples
in COCO minival set. The first row shows the input image and detection results
and the other rows show sampling points for each stage. Figure 5 and 6 shows
two different query results111In a figure, the query index is consistent
through stages.. The sampling z-axis coordinate, $\tilde{z}$, is visualized as
the point size and a bigger point corresponds with the larger $\tilde{z}$.
Sampling points of different groups are colored with different colors. We can
see that AdaMixer actually sees out of the box and different groups have
preference for different semantics. Moreover, when comparing Figure 5 and 6,
we can find that sampling point patterns also vary across different queries,
indicating the enhanced sampling adaptability and flexibility in AdaMixer.
Counterintuitively, sampling points of a query do not gather together in a
monotonous manner across stages. Instead, they often stretch wider after
focusing on small regions and then focus again (stage
2$\to$3$\to$4$\to$5$\to$6). We suspect that this behavior is beneficial to
performing bounding box estimation more accurately. Please see visualizations
folder in our codebase for more visualizations.

Figure 5: Visualizations of detection results and sampling points of AdaMixer
with ResNet-50 as the backbone. Figure 6: Visualizations of detection results
and sampling points of AdaMixer with ResNet-50 as the backbone (cont’d).

## References

  * [1] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization.  arXiv, 2016.
  * [2] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis.  Soft-nms - improving object detection with one line of code.  In ICCV, 2017.
  * [3] Zhaowei Cai and Nuno Vasconcelos.  Cascade R-CNN: delving into high quality object detection.  In CVPR, 2018.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [5] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  Hybrid task cascade for instance segmentation.  In CVPR, 2019.
  * [6] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  Mmdetection: Open mmlab detection toolbox and benchmark.  In arXiv, 2019.
  * [7] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In ICCV, 2017.
  * [8] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang.  Dynamic head: Unifying object detection heads with attentions.  In CVPR, 2021.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [10] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In ICCV, 2019.
  * [11] Pedro F. Felzenszwalb, Ross B. Girshick, and David A. McAllester.  Cascade object detection with deformable part models.  In CVPR, 2010.
  * [12] Pedro F. Felzenszwalb, Ross B. Girshick, David A. McAllester, and Deva Ramanan.  Object detection with discriminatively trained part-based models.  IEEE Trans. Pattern Anal. Mach. Intell., 32(9):1627–1645, 2010\.
  * [13] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of DETR with spatially modulated co-attention.  In ICCV, 2021.
  * [14] Ziteng Gao, Limin Wang, and Gangshan Wu.  Mutual supervision for dense object detection.  In ICCV, 2021.
  * [15] Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, and Jian Sun.  OTA: optimal transport assignment for object detection.  In CVPR, 2021.
  * [16] Ross B. Girshick.  Fast R-CNN.  In ICCV, 2015.
  * [17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  In ICCV, 2017.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [19] Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang.  Bounding box regression with uncertainty for accurate object detection.  In CVPR, 2019.
  * [20] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool.  Dynamic filter networks.  In NIPS, 2016.
  * [21] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In ECCV, 2018.
  * [22] Xiang Li, Wenhai Wang, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss V2: learning reliable localization quality estimation for dense object detection.  In CVPR, 2021.
  * [23] Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang.  Scale-aware trident networks for object detection.  In arXiv, 2019.
  * [24] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [25] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [26] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In ECCV, 2014.
  * [27] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In CVPR, 2018.
  * [28] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: single shot multibox detector.  In ECCV, 2016.
  * [29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [30] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [31] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional DETR for fast training convergence.  In ICCV, 2021.
  * [32] Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin.  Libra R-CNN: towards balanced learning for object detection.  In CVPR, 2019.
  * [33] Han Qiu, Yuchen Ma, Zeming Li, Songtao Liu, and Jian Sun.  Borderdet: Border feature for dense object detection.  In ECCV, 2020.
  * [34] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, 2016.
  * [35] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv, 2018.
  * [36] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In NIPS, 2015.
  * [37] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In CVPR, 2019.
  * [38] Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun.  Overfeat: Integrated recognition, localization and detection using convolutional networks.  In ICLR, 2014.
  * [39] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo.  Sparse R-CNN: end-to-end object detection with learnable proposals.  In CVPR, 2021.
  * [40] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: fully convolutional one-stage object detection.  In ICCV, 2019.
  * [41] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  Mlp-mixer: An all-mlp architecture for vision.  In NeurIPS, 2021.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * [43] Paul A. Viola and Michael J. Jones.  Rapid object detection using a boosted cascade of simple features.  In CVPR, 2001.
  * [44] Jiaqi Wang, Kai Chen, Shuo Yang, Chen Change Loy, and Dahua Lin.  Region proposal by guided anchoring.  In CVPR, 2019.
  * [45] Xinjiang Wang, Shilong Zhang, Zhuoran Yu, Litong Feng, and Wayne Zhang.  Scale-equalizing pyramid convolution for object detection.  In CVPR, 2020.
  * [46] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun.  Anchor DETR: query design for transformer-based detector.  arXiv, 2021.
  * [47] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In CVPR, 2017.
  * [48] Tong Yang, Xiangyu Zhang, Zeming Li, Wenqiang Zhang, and Jian Sun.  Metaanchor: Learning to detect objects with customized anchors.  In NeurIPS, 2018.
  * [49] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang.  Efficient DETR: improving end-to-end object detector with dense prior.  ArXiv, 2021.
  * [50] Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas S. Huang.  Unitbox: An advanced object detection network.  In ACM Multimedia, 2016.
  * [51] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection.  In CVPR, 2020.
  * [52] Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye.  Freeanchor: Learning to match anchors for visual object detection.  In NeurIPS, 2019.
  * [53] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  ArXiv, 2019.
  * [54] Xingyi Zhou, Jiacheng Zhuo, and Philipp Krähenbühl.  Bottom-up object detection by grouping extreme and center points.  In CVPR, 2019.
  * [55] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In CVPR, 2019.
  * [56] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  In ICLR, 2020.


# BSDAR: Beam Search Decoding with Attention Reward in Neural Keyphrase
Generation

Iftitahu Ni’mah
Eindhoven University of Technology
Eindhoven, the Netherlands
<EMAIL_ADDRESS>
&Vlado Menkovski
Eindhoven University of Technology
Eindhoven, the Netherlands<EMAIL_ADDRESS>
&Mykola Pechenizkiy
Eindhoven University of Technology
Eindhoven, the Netherlands<EMAIL_ADDRESS>

###### Abstract

This study mainly investigates two decoding problems in neural keyphrase
generation: sequence length bias and beam diversity. We introduce an extension
of beam search inference based on word-level and n-gram level attention score
to adjust and constrain Seq2Seq prediction at test time. Results show that our
proposed solution can overcome the algorithm bias to shorter and nearly
identical sequences, resulting in a significant improvement of the decoding
performance on generating keyphrases that are present and absent in source
text.

_K_ eywords Seq2Seq $\cdot$ Beam Search $\cdot$ Attention Mechanism

##  1 Introduction

Neural keyphrase generation [1, 2], a conditioned Sequence-to-Sequence
(Seq2Seq) approach for automated keyphrase extraction, has recently shown a
promising result as another domain for exploring latent aspects of Seq2Seq
models [3, 4, 5, 6, 7, 8]. Given pairs of document and the corresponding
keyphrase references as ground truth labels, the task is to encode sequence of
words in the source document into contextual vector representation; and
accordingly generate sequence of target words, a ‘‘keyword’’ or ‘‘keyphrase’’
that retains core information of the source document.

Keyphrase generation shares a common objective with Seq2Seq-based document
summarization [6], i.e. to condense a document into a short document
abstraction. Consequently, both domains also share common challenges: the
generation algorithm needs to accommodate two mechanisms – to copy words from
source, and to generate semantically related words not featured in source
document. While the “copying” task is particularly easy for an unsupervised
keyword extractor (e.g. TfIdf), a generative model such as Seq2Seq Recurrent
Network has not been specifically trained on such task. The problem has been
addressed by incorporating copying mechanism [5] into Seq2Seq architecture,
resulting in models referred as CopyRNN [1] and CorrRNN [2]. However, there
has not been enough attention on addressing decoding issues in the current
keyphrase generation task, referred as “beam search curse” [9], which is also
listed as one of six challenges for NMT [10].

Figure 1: Beam Search Decoding Issues: sequence length bias and beam diversity
Figure 2: Beam Search Decoding Issues: generating Present (C) vs. Absent (A)
keyphrases

This work further studies the decoding issues in neural keyphrase generation.
We focus on two challenges of beam search decoding: (1) the algorithm bias to
shorter sequences; and (2) the algorithm tendency to produce nearly identical
sequences, causing the beam candidates to be less diverse.

Our empirical finding (fig. 2) shows an example of discrepancy between what
the network (CorrRNN) has learnt from training examples and what the network
generates at test time. _Here_ , the attention successfully annotates sequence
of words in the source document with a higher score (darker highlight),
matched against the corresponding ground truth references. On the contrary,
the decoding algorithm fails to include the corresponding sequences in a final
prediction set. This finding suggests that apart from improving the model
architecture to be more expressive to the corresponding “copying” task, the
overall generation performance is also conditioned on the decoding algorithm.
This example of decoding issue in the current task becomes our main motivation
to further utilizing the learnt attention scores as a mechanism to constrain
the sequence generation process at test time. We argue that constraining beam
search decoding based on “ _hints_ ” provided by attention network is
favourable in the current task, given a condition that most keyphrase
references are present in the document.

##  2 Beam Search Decoding Issues

###  2.1 Sequence length bias

By default, beam search decoding algorithm of Seq2Seq model [11] stops the
search when exactly $b$ completed candidates of target sequences found, i.e.
when the decoder network generates “$<$end$>$” token. As illustrated in fig.
1, although increasing beam size increases the possibility of more candidates
to explore, the likelihood for the model to find the “$<$end$>$” token after
the first decoding step is also high. Consequently, the beam mainly contains
short sequences ($1-$grams, $2-$grams), disregarding many potential $n-$grams
candidates with longer sequences ($n\geq 3$). This tendency of the decoding
algorithm to favour shorter sequences can hurt performance severely,
specifically in the current keyphrase generation task, where ground truth
references are in variable length ($n-$grams, $n=1,\ldots,5$). We further show
empirically in section 5 that solely utilizing a normalization technique does
not guarantee to solve the sequence length bias issues in the current task.

###  2.2 Beam diversity

Figure 1 also exemplifies diversity problem in the standard beam search
decoding of Seq2Seq model for the current task. The generation of nearly
identical beams, i.e. $>80\%$ of sequences started with word “internet”,
results in a low “informativeness” (based on Precision $P$ metric).
Consequently, it decreases the decoding performance in the current task.

##  3 Neural keyphrase generation

###  3.1 Seq2Seq Attention Model

Our Attention-based Seq2Seq model is constructed of: (1) encoder - decoder
architecture with attention mechanism [11] as a backbone architecture; (2)
copying mechanism [5, 6, 1]; and (3) coverage-review mechanism [6, 2]. We re-
implemented and modified CopyRNN [1] and CorrRNN [2] to be able to be trained
on two conditions of target vocabulary size: truncated vocabulary and a very
large target vocabulary [12]. In truncated vocabulary setting, the vocabulary
included in look-up dictionary is constrained to top-50K most frequent words,
while Out-of-Vocabulary (OOV) set is referred as “<unk>”. In this study,
Seq2Seq model with truncated vocabulary is referred as CorrRNN, while the
modified model for large target vocabulary is referred as CorrRNN-L.

####  3.1.1 CorrRNN

For both Seq2Seq models in this study (CorrRNN, CorrRNN-L), we use MLP
(concat) attention mechanism [11] as the attention scoring function
score$($Q,K$)=$VT tanh$($W$[$Q,K$])$ for incorporating copying, coverage, and
review mechanism into Seq2Seq architecture. Q corresponds to query-attention:
decoder state at one time step $s_{t}$; and K denotes keys-attention: a
sequence of encoder states $h_{1\ldots T_{x}}$ and coverage vector of
attention cov up to the current time step $t$. The latter corresponds to a
coverage mechanism [6, 2]. _Here_ , for CorrRNN, the model has two inputs: (1)
sequences with a truncated dictionary size referred as $x$; and (2) sequences
with extended vocabulary referred as $x^{(ext)}$. The additional oov
dictionary index size is set to $23914$, excluding the main vocabulary index
with top$-50K$ most frequent words.

####  3.1.2 CorrRNN-L

Since the vocabulary is not being truncated, CorrRNN-L model does not have
issues with OOV. The problem, however, is shifted to the complexity of
training and decoding due to large number of target words. This issue has been
addressed by incorporating sampled softmax approximation based on adaptive
sampling algorithm [12] as an approximate learning approach to a very large
target vocabulary size. _Here_ , we trained the model on $366730$ vocabulary
size. The number of vocabulary samples for the approximate softmax layer,
instead of _full_ softmax in CorrRNN model architecture, is set to $50K$.

##  4 Beam Search Decoding with Attention Reward (BSDAR)

###  4.1 Word-level Attention-Reward

We propose a word-level attention reward function WORD_ATT_REWARD as a
mechanism to augment Seq2Seq prediction, shown in alg. 1. For each decoding
time step $t$, the logits probability output of decoder network ($s_{t}$) is
augmented by an attention vector ($\alpha_{w}\in R^{Tx}$), which each value
corresponds to the attention weight of a word in source sequence $x$.
$\hat{\alpha}_{w}[w]$ (alg. 1 line 3) denotes a normalized (mean) attention
score of a particular word, given the possibility that the word occurs
multiple times in the source text. Since the attention vector $\alpha_{w}$ is
bounded to the sequential position of words in source input, the dictionary
look-up of word index and the corresponding position $f(w):K\xrightarrow{}V$
is also given during the decoding time as a reference to calculate mean
$\hat{\alpha}_{w}[w]$, where $K$ denotes a set of words in input sequence
$K=\\{w_{0}\ldots w_{N}\\}$ and $V$ corresponds to the position where the word
occurs $V=[w_{p_{1}}\ldots w_{p_{n}}]$ in input sequence. Correspondingly, the
augmentation of decoder logits only applies for words appear in source text.

To intensify the effect of logits augmentation, which is becoming critical in
a noisy decoding of Seq2Seq with large target vocabulary size (CorrRNN-L), we
use $\lambda$ and $\gamma$ (alg. 1 line 4) as an augmentation factor of
attention reward $\hat{\alpha}_{w}$. _Here_ , we set $\lambda=2$ and
$\gamma=\lambda*$ MAX $(\hat{\alpha}_{w})$.

Algorithm 1 WORD_ATT_REWARD

1:for $t=0,1,\ldots,$ MAX_STEPS do

2: Collect logits and attention weights

$s_{t}=$ DEC
$(y_{t},c_{t-1}^{(\texttt{cov})},c_{t-1}^{(\texttt{rev})},s_{t-1},\zeta_{t-1})$

$\alpha=\textbf{{MLP}}(s_{t-1},h,\texttt{cov})$

$s_{t}\in R^{V},\alpha\in R^{T_{x}}$

3: Compute normalized $\hat{\alpha}_{w}$, for each word

$\hat{\alpha}_{w}[w]=\texttt{{MEAN}}(\alpha[w_{p_{1}}\ldots w_{p_{n}}])$

4: Augment the logits, for words in $\hat{\alpha}_{w}$

$\widetilde{s}_{t}[w]=s_{t}[w]+(\lambda*\hat{\alpha}_{w}[w])+\gamma$

5: Get $b-$top most probable words

tokens $=$ ARGSORT $\\{\widetilde{s}_{t}\\}[:b]$

probs $=$ SORT $\\{\widetilde{s}_{t}\\}[:b]$

6: return BEAM-HYP $=$ (tokens, probs)

Algorithm 2 NGRAM_ATT_REWARD

1:while steps $<$ MAX_STEPS and Results $<$ BEAM_SIZE $b$ do

2: Run one step decoding with word attention reward

BEAM-HYP $=$WORD_ATT_REWARD()

3: Expand tree with new candidates HYPS $\frown$BEAM-HYP

4: Construct sequence candidates

SEQ $=$ HYPS.PREV_TOKENS $+$ HYPS.CURR_TOKENS

5: if SEQ found in annotated source ATT-ANNOT then

6: Reward candidates with $\hat{\alpha}_{p}$

AUG_PROB $=$ HYPS.CURR_PROB $+(\lambda*\hat{\alpha}_{p})+\gamma$

7: else if SEQ is partly composed of tokens in ATT-ANNOT then

8: Set probability into negative value (log “inf”) to penalize SEQ candidate

AUG_PROB $=-0.05$

9: else

10: Set logits probability without reward score

AUG_PROB $=$ HYPS.CURR_PROB

11: Re-sort beam candidates

12: Update tree with new candidates

13: Sort the candidates stored in memory (tree) based on normalized joint
probability

14: Expand Results with completed sequence candidates

Our proposed word-level attention reward shares a common intuition with actor-
critic algorithm for sequence prediction [13]. Intuitively, the
WORD_ATT_REWARD function corresponds to increasing the probability of
_actions_ that the encoder network gives a higher attention value; and
decreasing the probability of _actions_ that are being ignored by the encoder
network. The actions correspond to the Seq2Seq predictions for each decoding
step. The proposed reward mechanism, thus, bounds the actions with an
attention score, which each value represents a degree of importance of word in
source sequence based on what the Seq2Seq attention network has learnt during
training time.

###  4.2 $N-$gram-level attention reward

The proposed word-level attention reward in alg. 1 is based only on bag-of-
words assumption. As a potential trade-offs on the decoding performance, the
beam algorithm may then favour sequence candidates containing words with
higher attention score. Thus, it disregards whether the formed sequence is
sensible as a candidate of keyphrase. For instance, from the attention
visualization in fig. 2, a keyphrase candidate “inquiry honours” may also be
rewarded highly, considering the sequence is composed of words with a high
attention score (“inquiry”, “honours”). This leads to a noisy prediction and
can potentially decrease decoding performance.

To mitigate this issue, we also introduce the $n-$gram-level attention reward
function NGRAM_ATT_REWARD to further augment and penalize beam candidates
before adding them into the memory (alg. 2). For each decoding step $t$, given
the previous stored tokens in memory and the current beam candidate returned
by WORD_ATT_REWARD, an $n-$gram candidate SEQ were formed. For all SEQ
candidates matched against the extracted $n-$gram attention annotations ATT-
ANNOT, the logits of the last tokens of the SEQ candidates were added by the
corresponding $n-$gram attention score $\hat{\alpha}_{p}$ (alg. 2 line 6).

##### Automated attention-based annotation

The steps for acquiring both extracted n-gram attention annotations ATT-ANNOT
and the corresponding attention score are shown in alg. 3. The attention
annotation ATT-ANNOT was constructed during the first decoding step ($t=0$) by
a simple $n-$gram ($n=1,\ldots,5$) chunking method (i.e. sliding window of $n$
and $n+1$) of a filtered source document. The filtering of source document is
based on a percentile rank $10$ of sorted attention score $\alpha$ as a
cutting attention threshold $\tau$. Given the attention threshold $\tau$, the
ATT-ANNOT was extracted based on $n-$gram chunks of element-wise
multiplication between filtered attention score $\alpha^{\delta}$ (line 3) and
source sequence (line 4). The final result is a list of $n-$grams with the
corresponding attention score $\hat{\alpha}_{p}$. $\hat{\alpha}_{p}$ was
acquired by computing the mean of attention scores of words $p_{i}$ composing
the corresponding $n-$gram sequence $P$:
$\hat{\alpha}_{p}[P]=\frac{1}{N_{p}}\sum_{i=1}^{N_{p}}\hat{\alpha}[p_{1}]+\ldots+\hat{\alpha}[p_{i}]$.
The resulting extracted atention annotation from the first decoding step is
then used as a global reference for the following decoding time steps ($t>0$)
and is utilized as reference for NGRAM_ATT_REWARD function.

Algorithm 3 EXTRACT_ATT_ANNOT

1:Collect attention vector $\alpha$

2:Compute threshold $\tau$ based on percentile $10$ of $\alpha$

3:Binarize attention values based on threshold $\tau$

$\alpha^{\delta}_{t}=[1$ if $\alpha_{t}>\tau$ else $-1]$

4:Extract n-grams attention annotation

ATT-ANNOT = CHUNK$(\alpha^{\delta}*x_{1\ldots T_{x}})$

5:Extract $\alpha$ based on sequential position

SEQ-POS = CHUNK $(\alpha^{\delta}*$ $x-$pos$)$

$\hat{\alpha}[P]=\alpha[$SEQ-POS${}_{i}]$

6:Extract n-gram attention values $\hat{\alpha}_{p}$

$\hat{\alpha}_{p}=$ MEAN $(\hat{\alpha}[P_{1}\ldots P_{n}])$

7:Return {ATT-ANNOT $=f:K\xrightarrow{}V$, $K=\\{P_{1}\ldots P_{n}\\}$
$V=[\hat{\alpha_{p}}]$}

##### Penalty score

Penalty was given to the beam candidates that are partially composed of word
tokens found in ATT-ANNOT. These sequence candidates contain words with high
attention score, but are mainly _non-_ sensical. In this subset, the last
tokens of the sequence candidates were set with a negative probability
($-0.05$), shown in alg. 2 line 8. For all candidates with negative value of
probability in beam tree, the logits were then set to zero. This penalty
encourages the sequence candidates to have an extremely large log probability
values (“inf”), and correspondingly lower ranks during beam re-sorting stage.
For sequence candidates that do not contain words and phrases in ATT-ANNOT,
the logits output of decoder network were not augmented, kept _as is_ (line
10). Thus, the sequences not featured in source text were still considered as
candidates of Seq2Seq final prediction, but ranked after those found in the
source sequence. This last step intuitively aims to preserve the “abstractive”
ability of the model, i.e. the ability to produce sequences that do not
necessarily appear in the source but has a close semantic meaning with the
corresponding document.

###  4.3 Re-rank method

In addition to the proposed decoding with attention reward, a heuristic
approach to alleviate sequence length bias and diversity issues was employed.
We adopted a concept of _intra-_ and _inter-_ sibling rank of beam decoding
[14] into a simple implementation. We refer the heuristic approach adopted in
this study as (1) _pre-_ intra siblings rank; (2) _post-_ intra siblings re-
rank; and (3) _post-_ inter sibling re-rank. _Here_ , in _pre-_ intra siblings
ranking, for each decoding step, we only consider top-3 beams (word tokens) to
be added into the tree queue. Given completed sequences (i.e. sequences with
“<end>” as last token), in _post-_ intra siblings re-rank, given candidates
with the same parent node and sequence length, only top-1 beam candidates were
considered. Likewise, in _post-_ inter sibling re-rank, only top-5 candidates
were considered as final solution. While _pre-_ intra siblings rank was ranked
based on the probability scores of the last tokens of the sequence candidates,
the _post-_ intra siblings re-rank and _post-_ inter siblings re-rank were
sorted based on normalized joint probability score of the completed sequences.

##  5 Increasing traversal breadth and depth

Conventional wisdom for tackling the aforementioned beam search decoding
issues is by expanding the beam size (traversal breadth) and sequence length
(traversal depth). _Here_ , we show empirically that increasing both
parameters does not guarantee to result optimal solutions or significantly
increase the performance gain in the current study.

##### Expanding Beam Size

Figure (3(a) shows that there is no gain on increasing beam size (up to $95$
beams) and utilizing a simple length normalization technique. The different
trend of uni-gram-based evaluation in figure 3(b), however, indicates that the
beam solution includes potential candidates partially (i.e. contains partial
words in references), but fails to correctly form $n-$gram sequence candidate
longer than a uni-gram. Example of the decoding results is shown in table 1.

No Length Normalization | With Length Normalization
---|---
internet | internet
recommendations | internet analysis
support | recommendations
$<$unk$>$ | support
online |  $<$unk$>$
information | online
internet analysis | information
web | web
decision | decision
computer | computer

Table 1:  Decoding results. Beam size $b$ is set to 10.

(a) $n-$gram evaluation

(b) uni-gram evaluation

(c) MAX_SEQ vs. diversity score

Figure 3: The effect of changing beam size on decoding performance

##### Increasing sequence length

Increasing the traversal depth (MAX_SEQ), shown in figure 3(c), also does not
add any values on improving the diversity of beam candidates in the current
task. We define a diversity score (_y_ axis of figure 3(c)) as the “diversity”
of the first word tokens in prediction and ground truth set, as such.

$\texttt{{DIV\\_SCORE}}=\frac{\texttt{{COUNT}}(\\{w^{(1)}\\})}{\texttt{{COUNT}}(Y)}$
(1)

Where $\\{w^{(1)}\\}$ denotes a set of unique words, which each corresponds to
the first token of a sequence and $Y$ corresponds to a list of keyphrases. The
purpose of this diversity score metric is to measure the repetitiveness of
beam decoding based on the first word tokens of the generation results. The
higher the diversity score is (e.g. close to the diversity score of ground
truth references), the better of the decoding algorithm to overcome the beam
diversity issue.

##  6 Experiments and results

Seq2Seq model in this study was trained on KP20k corpus [1] ($\approx 530K$
documents, providing $\approx 2M$ training sets after preprocessing). For
training the model, both sources $x_{i}$ and the corresponding keyphrase
labels $P_{i}^{(n)}$ are represented as word sequences, such as
$x_{i}=\\{x_{1}^{(i)},x_{2}^{(i)},\cdots x_{T_{x}}^{(i)}\\}$ and
$P_{i,n}=\\{p_{1}^{(i,n)},p_{2}^{(i,n)},\cdots p_{T_{y}}^{(i,n)}\\}$, where
$T_{x}$ and $T_{y}$ are maximum sequence length of source document and target
keyphrase labels respectively. To be noted, each source document corresponds
to $n$ multiple keyphrase labels. By splitting the data sample
$\\{x_{i},P_{i}\\}$ into $n$ pairs
$\\{x_{i},P_{i}^{(1)}\\},\ldots,\\{x_{i},P_{i}^{(m)}\\}$, the training set is
presented as text-keyphrase pairs, each contains only one source text sequence
and target keyphrase sequence. In inference stage, standard evaluation data
sets for keyphrase extraction were used (Inspec [15] ($201$), Krapivin [16]
($201$), NUS [17] ($156$), Semeval-2010 [18] ($242$).

BS Decoding | CorrRNN
---|---
| Inspec (201)† | Krapivin (201)† | NUS (156)† | Semeval-2010 (242)†
| $R@50$ | $R@N$ | $R@50$ | $R@N$ | $R@50$ | $R@N$ | $R@50$ | $R@N$
BS | 0.098 | 0.098 | 0.073 | 0.073 | 0.093 | 0.093 | 0.056 | 0.056
BS++ | 0.091 | 0.091 | 0.094 | 0.094 | 0.098 | 0.098 | 0.059 | 0.059
BSDAR | 0.290 | 0.372 | 0.249 | 0.317 | 0.211 | 0.255 | 0.169 | 0.221

Table 2:  Comparison of Beam Search (BS) decoding algorithms on CorrRNN
performance. † size of test corpora

BS Decoding | CorrRNN-L
---|---
| Inspec (201)† | Krapivin (201)† | NUS (156)† | Semeval-2010 (242)†
| $R@50$ | $R$ | $R@50$ | $R$ | $R@50$ | $R$ | $R@50$ | $R$
BS | 0.045 | 0.045 | 0.038 | 0.038 | 0.043 | 0.043 | 0.026 | 0.026
BS++ | 0.033 | 0.047 | 0.038 | 0.05 | 0.0409 | 0.052 | 0.020 | 0.029
BSDAR | 0.268 | 0.327 | 0.193 | 0.226 | 0.152 | 0.182 | 0.139 | 0.166

Table 3:  Comparison of Beam Search (BS) decoding algorithms on CorrRNN-L
performance. † size of test corpora

Decoding | Inspec (199)† | Krapivin (185)† | NUS (148)† | Semeval-2010 (242)†
---|---|---|---|---
| CorrRNN | CorrRNN-L | CorrRNN | CorrRNN-L | CorrRNN | CorrRNN-L | CorrRNN | CorrRNN-L
| R | ROUGE-L | R | ROUGE-L | R | ROUGE-L | R | ROUGE-L | R | ROUGE-L | R | ROUGE-L | R | ROUGE-L | R | ROUGE-L
BS | 0.027 | 0.207 | 0.066 | 0.211 | 0.026 | 0.257 | 0.053 | 0.226 | 0.034 | 0.240 | 0.048 | 0.197 | 0.015 | 0.194 | 0.026 | 0.154
BS++ | 0.022 | 0.194 | 0.078 | 0.329 | 0.024 | 0.247 | 0.081 | 0.369 | 0.029 | 0.228 | 0.064 | 0.304 | 0.013 | 0.182 | 0.031 | 0.260
BSDAR | 0.038 | 0.249 | 0.079 | 0.331 | 0.071 | 0.300 | 0.064 | 0.348 | 0.037 | 0.260 | 0.065 | 0.310 | 0.031 | 0.225 | 0.041 | 0.253

Table 4:  Abstractive Performance. Scores are based on Micro-Average Recall
(R) and ROUGE-L average F1-score. †) size of test corpora after preprocessing.

Decoding | Inspec (138)† | Krapivin (85)† | NUS (93)† | Semeval-2010 (212)†
---|---|---|---|---
| CorrRNN | CorrRNN-L | CorrRNN | CorrRNN-L | CorrRNN | CorrRNN-L | CorrRNN | CorrRNN-L
BS | 0.236 | 0.189 | 0.236 | 0.183 | 0.215 | 0.150 | 0.196 | 0.142
BS++ | 0.249 | 0.328 | 0.248 | 0.344 | 0.234 | 0.279 | 0.199 | 0.269
BSDAR | 0.405 | 0.423 | 0.359 | 0.408 | 0.335 | 0.349 | 0.277 | 0.285

Table 5:  Abstractive Performance on longer sequences ($n-$grams, $n=3\ldots
5$). Scores are based on ROUGE-L average F1-score. †) size of test corpora
after preprocessing.

##### Beam Parameters

For the first decoding time step ($t=0$), beam size ($b$) is set to be a
larger number ($b=100$) than the remaining decoding time steps ($b=50$).
Number of hypothesis num_hyps representing the partial solutions (queue) of
sequence candidates to be added into the memory is set to $200$.

##### Evaluation Metrics

We employ Micro-average Recall (R), ROUGE-L average F1-score, and diversity
(section 5) metrics to evaluate performance of beam search decoding algorithms
in this study.

Model | Inspec-L (189)† | Krapivin-L (118)† | NUS-L (121)† | Semeval2010-N (234)†
---|---|---|---|---
| $R@10$ | $R@50$ | ROUGE-L | $R@10$ | $R@50$ | ROUGE-L | $R@10$ | $R50$ | ROUGE-L | $R@10$ | $R@50$ | ROUGE-L
TfIdf | 0.082 | 0.252 | 0.635 | 0.079 | 0.189 | 0.562 | 0.044 | 0.110 | 0.478 | 0.039 | 0.157 | 0.441
CorrRNN $+$ BS | 0. | 0. | 0.256 | 0. | 0. | 0.265 | 0. | 0. | 0.251 | 0. | 0. | 0.236
CorrRNN $+$ BS++ | 0.005 | 0.005 | 0.294 | 0.011 | 0.011 | 0.306 | 0.009 | 0.009 | 0.299 | 0.006 | 0.006 | 0.261
CorrRNN $+$ BSDAR | 0.007 | 0.130 | 0.643 | 0.009 | 0.123 | 0.560 | 0.009 | 0.070 | 0.4960 | 0.009 | 0.058 | 0.426
CorrRNN-L $+$ BS | 0. | 0. | 0.138 | 0. | 0. | 0.151 | 0. | 0. | 0.152 | 0. | 0. | 0.134
CorrRNN-L $+$ BS++ | 0. | 0.003 | 0.308 | 0.009 | 0.009 | 0.341 | 0.005 | 0.005 | 0.320 | 0.002 | 0.003 | 0.299
CorrRNN-L $+$ BSDAR | 0.102 | 0.342 | 0.664 | 0.057 | 0.219 | 0.572 | 0.035 | 0.164 | 0.4962 | 0.049 | 0.162 | 0.436

Table 6:  Comparison between models in subsets with longer keyphrases
(n-grams, $n=3\ldots 5$). †) size of test corpora after preprocessing.

###  6.1 Decoding performance

Table 2 and 3 shows the comparison between our proposed method (BSDAR) and
standard beam search decoding algorithms for Seq2Seq prediction. To show that
simply applying the heuristic rules (section 4.3) does not guarantee to solve
the decoding issues, we included the heuristic-based only beam decoding (BS++)
as a comparison. In general, the results for both CorrRNN and CorrRNN-L models
show that BSDAR can significantly improve the recall score of about $20-30\%$.

We also show that while expanding retrieval set (from $R@50$ to $R@N$, where
$N=50\ldots 200$) has no gain for the decoding based on standard beam decoding
(BS) and heuristic-based decoding (BS++), the performance of the proposed
decoding method significantly increases. This result indicates that the
solutions based on standard beam search (BS) and heuristic-based beam search
(BS++) mainly contain noises and non-relevant sequences, as compared to the
predictions resulted by BSDAR.

###  6.2 Abstractive performance

For a neural generation task, it is also important to maintain the model
ability to generate “abstractive” sequences. The results in table 4 show that
BSDAR is able to maintain a high performance on generating sequences not
featured in source text, given subsets with “absent” keyphrase references in a
variable sequence length ($n=1,\ldots,5$). We further evaluate the
“abstractive” performance of the decoding algorithms on longer sequences
(table 5). Since this task is generally challenging for any models, we use
ROUGE-L average $F_{1}-$score metric to match the prediction and ground truth
set. In addition to maintaining a high abstractive performance in the subset
with longer target sequences, BSDAR is also able to disclose the “actual”
intrinsic performance of CorrRNN-L. As compared to low decoding performance of
CorrRNN-L based on standard beam search (BS), the decoding performance of
CorrRNN-L based on BSDAR is higher. This indicates that the corresponding
model has actually learnt to attend on relevant words in source text, but the
decoding algorithm fails to include the words in the final generated
sequences. Understanding the intrinsic performance of a complex Seq2Seq model
exemplified by this empirical result is becoming essential, specifically to
further address a challenging “abstractive generation” task for any neural
sequence models.

###  6.3 On sequence length bias issue

To measure whether the proposed method in this study (BSDAR) can overcome the
algorithm bias to shorter sequences, we compare the Seq2Seq decoding
performance based on BSDAR with Tf-Idf unsupervised keyword extractor in a
subset with longer target sequences ($n-$grams, $n>3$). Intuitively, this
subset introduces challenges for both neural network and non-neural network
(Tf-Idf) approaches, since both models may suffer with sequence length bias
issues, resulting the prediction with shorter sequence length. The result in
table 7 shows that the proposed solution can improve the decoding performance
of Seq2Seq models, outperforming Tf-Idf in three data sets, as compared to
Seq2Seq with standard beam (BS) and heuristic-based only beam decoding (BS++).

###  6.4 On diversity issue

We also show that BSDAR can overcome the diversity issue of beam decoding
algorithm. Based on the result in table 7, BSDAR can maintain high diversity
score (section 5) close to the diversity measure of ground truth references.
Furthermore, as compared to the standard beam (BS) and heuristic beam (BS++),
BSDAR shows a reasonably better decoding performance for recalling uni-gram
($R^{(1)}$), bi-gram ($R^{(2)}$), and $n-$gram references ($R^{(n)},n\geq 3$).

Keyphrase set | DIV(n) | $R^{(1)}$ | $R^{(2)}$ | $R^{(n)}$
---|---|---|---|---
Ground truth | 0.942 | N/A | N/A | N/A
BS | 0.058 | 0.120 | 0.017 | 0.
BS++ | 0.661 | 0.112 | 0.030 | 0.004
BSDAR | 0.746 | 0.301 | 0.198 | 0.223

Table 7: Diversity measure of decoding algorithms across data sets

##  7 Conclusion

We present an approach to overcome two decoding issues in neural keyphrase
generation by incorporating attention vector as a re-scoring method of beam
search decoding algorithm. We show empirically that the proposed solution
(BSDAR) not only performs well on improving the generation of longer and
diverse sequences, but also maintains the Seq2Seq ability to predict
“abstractive” keyphrases, i.e. semantically relevant keyphrases that are not
present in source text.

## References

  * [1] R. Meng, S. Zhao, S. Han, D. He, P. Brusilovsky, and Y. Chi, “Deep keyphrase generation,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (Vancouver, Canada), pp. 582–592, Association for Computational Linguistics, July 2017.
  * [2] J. Chen, X. Zhang, Y. Wu, Z. Yan, and Z. Li, “Keyphrase generation with correlation constraints,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4057–4066, Association for Computational Linguistics, 2018.
  * [3] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in neural information processing systems, pp. 3104–3112, 2014.
  * [4] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014.
  * [5] J. Gu, Z. Lu, H. Li, and V. O. Li, “Incorporating copying mechanism in sequence-to-sequence learning,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (Berlin, Germany), pp. 1631–1640, Association for Computational Linguistics, August 2016.
  * [6] A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), (Vancouver, Canada), pp. 1073–1083, Association for Computational Linguistics, July 2017.
  * [7] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156–3164, 2015.
  * [8] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International conference on machine learning, pp. 2048–2057, 2015.
  * [9] Y. Yang, L. Huang, and M. Ma, “Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, (Brussels, Belgium), pp. 3054–3059, Association for Computational Linguistics, Oct.-Nov. 2018.
  * [10] P. Koehn and R. Knowles, “Six challenges for neural machine translation,” in Proceedings of the First Workshop on Neural Machine Translation, (Vancouver), pp. 28–39, Association for Computational Linguistics, Aug. 2017\.
  * [11] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, vol. abs/1409.0473, 2014.
  * [12] S. Jean, K. Cho, R. Memisevic, and Y. Bengio, “On using very large target vocabulary for neural machine translation,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), (Beijing, China), pp. 1–10, Association for Computational Linguistics, July 2015.
  * [13] D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. C. Courville, and Y. Bengio, “An actor-critic algorithm for sequence prediction,” CoRR, vol. abs/1607.07086, 2016.
  * [14] J. Li and D. Jurafsky, “Mutual information and diverse decoding improve neural machine translation,” CoRR, vol. abs/1601.00372, 2016.
  * [15] A. Hulth, “Improved automatic keyword extraction given more linguistic knowledge,” in Proceedings of the 2003 conference on Empirical methods in natural language processing, pp. 216–223, 2003.
  * [16] M. Krapivin, A. Autayeu, and M. Marchese, “Large dataset for keyphrases extraction,” Tech. Rep. DISI-09-055, DISI, Trento, Italy, May 2008.
  * [17] T. D. Nguyen and M.-Y. Kan, “Key phrase extraction in scientific publications,” in Proceeding of International Conference on Asian Digital Libraries, pp. 317–326, 2007.
  * [18] S. N. Kim, O. Medelyan, M.-Y. Kan, and T. Baldwin, “Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles,” in Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, (Stroudsburg, PA, USA), pp. 21–26, Association for Computational Linguistics, 2010.