#  Synthesizing Tensor Transformations for Visual Self-attention

Xian Wei∗, Xihao Wang1, Hai Lan2, JiaMing Lei3,
Yanhui Huang4, Hui Yu5, Jian Yang6
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding
Author

###### Abstract

Self-attention shows outstanding competence in capturing long-range
relationships while enhancing performance on vision tasks, such as image
classification and image captioning. However, the self-attention module highly
relies on the dot product multiplication and dimension alignment among query-
key-value features, which cause two problems: (1) The dot product
multiplication results in exhaustive and redundant computation. (2) Due to the
visual feature map often appearing as a multi-dimensional tensor, reshaping
the scale of the tensor feature to adapt to the dimension alignment might
destroy the internal structure of the tensor feature map. To address these
problems, this paper proposes a self-attention plug-in module with its
variants, namely, Synthesizing Tensor Transformations (STT), for directly
processing image tensor features. Without computing the dot-product
multiplication among query-key-value, the basic STT is composed of the tensor
transformation to learn the synthetic attention weight from visual
information. The effectiveness of STT series is validated on the image
classification and image caption. Experiments show that the proposed STT
achieves competitive performance while keeping robustness compared to self-
attention based above vision tasks.

##  1 Introduction

In recent years, the attention mechanism is generating considerable interest
in the domain of deep learning. The main breakthroughs of attention modules
were first appeared in Natural Language Processing (NLP) [33, 45, 41, 24, 43,
8, 32], and then also in computer vision domain [47, 20, 14]. These
achievements have demonstrated that the attention module provided a different
approach than Convolutional Neural Network (CNN) models to handle the task and
demonstrated a promising performance.

Generally, the attention module tends to exert a learnable weight on features
to distinguish importance from various perspectives. According to the
computing techniques and tasks, the attention modules can be categorized into
two types. Hard attention module is non-differentiable and requires
computation tricks [28, 1], while soft attention module is differentiable thus
having wider applications for visual tasks [39, 19, 42]. As a particular form
of attention, self-attention is applied as the core mechanism of the neural
network named transformer, which is widely employed in the field of computer
vision. The essential advantage of self-attention is that it focuses on the
relative importance of own content rather than weighting across multiple
contents as a general attention mechanism. As one of the critical roles in the
self-attention mechanism, dot product multiplication among the feature of
query, key, and value plays a significant role in learning self-alignment[35].
Precisely, employing the dot product between a single token and all other
tokens in the sequence could formulate the relative importance score. Through
pairwise dot product, the self-attention mechanism establishes a content-based
retrieval process.

However, do we really need to use dot product in the self-attention mechanism?
Is it the best choice to learn the self-alignment between the features of
query and key? We can not deny that attention-based architecture is one of the
most efficient models in machine learning, but it also has space to
improve[12]. Investigating the alternative plug-in module for current
architecture could help us to have a deeper understanding of the attention
mechanism. Undoubtedly, dot product multiplication has several benefits. But
it also induces exhaustive and redundant computation during exerting on the
features. Especially on large-scale processing data, the extra parameters and
memory caused by dot product operations enormously increase the burden of the
training process. Moreover, in the computer vision task, self-attention for
images tends to compute the similarity scores among visual features[47, 42,
27] rather than mapping for retrieval. In order to overcome the shortages of
dot production, it is necessary to search for another approach to improve the
performance of the self-attention mechanism. As the result, recent findings
regarding the NLP attention module have led to an alternative approach, named
Synthesizer Attention[35].

In our work, we provide a viable alternative to overcome the shortage brought
from the dot-production in the standard self-attention module, in order to
pursue a better performance of visual self-attention. Inspired by the NLP-like
synthesizer, we propose a plug-in synthesizer to replace traditional visual
self-attention modules, namely, the Visual Dense Synthesizing (VDS) module.
Compared with the NLP-like synthesizer, which contains one-dimensional natural
language vectors, our module applies the multi-dimensional tensor to adapt the
presentation of visual features. The module is basically composed of a tensor
transformation for self-alignment between query matrix and key matrix, without
computing the dot-product multiplication among the feature of query-key-value.
Removing dot product operations, the VDS gains some straightforward
advantages.

  1. 1.

Replacing dot production based computation with a learnable linear synthesizer
reduces dependency on the input, thus enhancing the robustness of the model
against external perturbations.

  2. 2.

The proposed linear synthesizer can be easily adapted to input data structure,
without reshaping the raw input. Thus, it is easier to preserve the underlying
structure of feature maps.

  3. 3.

Since the alignment transformations in the synthesizer are computed upon
various dimensions of feature maps, the attention is produced simultaneously
in both channel and spacial domain.

To further simplify the linear transformations in synthesizing, derived from
the regular VDS module, the paper further proposes a series of plug-and-play
modules, namely, Synthesizing Tensor Transformations (STT). The STT series are
finally applied for the image classification task and image caption task.
According to the result of our typical classification experiment, the STT
series demonstrates that it not only is a viable alternative of the
traditional self-attention module but also has better robustness.

##  2 Related Work

In this section, we will introduce the various applications of the self-
attention mechanism. Then, the dot product in self-attention would be
described as the preliminaries for our STT series.

###  2.1 Self-Attention Mechanism

The self-attention attracts considerable interest due to its versatile
application. For CNN-based models, self-attention mechanism has been used in
many modules, such as extra re-weighting modules for channels [18, 19, 39],
jointly spatial and channel attention module [4, 42, 15], or remold
convolution operation with self-attention [2, 29, 9, 46].

Another line of works arrange self-attention to be a component in a pipeline
for downstream tasks, such as augmentation of feature maps for classification
[2], object detection [47, 17, 3], and segmentation [40]. However, it is
difficult to directly apply the self-attention mechanism to pixel-wise data.
[30] restricted self-attention convolution within local neighborhoods for
query pixels, and [40, 16] restricted computation along individual axes. [5]
reduced image resolution and color space before transformer, and [13] directly
applied transformer on image sequence patches.

To reduce computational costs, some of the work adjust the structure of
attention. [21] divided and permuted the feature map thus smaller attention
map would be conquered and permuted back. [22] focused on the criss-cross
around the key points on the feature of query, key and value. [7] adopted
double attention to re-allocate features for channels and the total workload
shrinks after feature gathering. Differently, another way is to reduce the
length at certain dimension [41], or divide computation by locality sensitive
hashing [24].

(a)

(b)

(c)

(d)

Figure 1:  Classical visual self-attention vs. Synthesizers.

###  2.2 Dot-product in Self-attention

The regular self-attention operator based dot-product attention serves as a
basic building block of vision tasks [2, 29, 9, 46]. Let
$\mathcal{X}\in\mathbb{R}^{H\times W\times C}$ be a tensor input with the
height $H$, the width $W$, the number of channels $C$.

To feed the self-attention block, the 3D tensor
$\mathcal{X}\in\mathbb{R}^{H\times W\times C}$ is first flattened to a 2D
matrix $X=\in\mathbb{R}^{HW\times C}$. Then $X$ is projected to corresponding
representations $Q$, $K$ and $V$, with $Q=\phi_{Q}(X)$, $K=\phi_{K}(X)$ and
$V=\phi_{V}(X)$, respectively. $\phi_{Q},\phi_{K},\phi_{V}$ are trainable
transformations such as linear mappings or convolutions, and have matching
output dimensionality. For example, if we read $\phi_{Q},\phi_{K},\phi_{V}$
are linear transformations $W_{Q}\in\mathbb{R}^{C\times d}$,
$W_{K}\in\mathbb{R}^{C\times d}$ and $W_{V}\in\mathbb{R}^{C\times d}$,
respectively, the representations are $Q=XW_{Q}$, $K=XW_{K}$, $V=XW_{V}$, and
$m$ means the desired output dimension. Finally, the output value is weighted
average over input values,

$\displaystyle S$
$\displaystyle=\text{Softmax}\left(\dfrac{QK^{\top}}{\sqrt{d}}\right)$ (1)
$\displaystyle Y$ $\displaystyle=\text{Attention}(Q,K,V)=SV.$ (2)

Where $S$ denotes the attention coefficients that $S_{ij}$ measures the
similarity between the $i^{\text{th}}$ row of $Q$ and the $j^{\text{th}}$ row
of $K$. The detail of the dot-production is presented in the Figure 1(a).

Figure 2:  This figure visualizes an example of the $n-$mode product
$\mathcal{X}\times_{1}\mathbf{A}_{(H)}\times_{2}\mathbf{A}_{(W)}\times_{3}\mathbf{A}_{(C)}$,
i.e., the multiplication of a third-order tensor
$\mathcal{X}\in\mathbb{R}^{8\times 7\times 9}$ with matrices
$\mathbf{A}_{(H)}\in\mathbb{R}^{4\times 8}$,
$\mathbf{A}_{(W)}\in\mathbb{R}^{5\times 7}$,
$\mathbf{A}_{(C)}\in\mathbb{R}^{6\times 9}$.

Figure 3: Illustrations for tensor multiplication and decomposition.

##  3 Synthesizing Tensor Transformation

In order to introduce our Synthesizing Tensor Transformation, we first give a
fundamental model of tensor synthesizer. And then, the specific tensor
synthesizers would be provided as (a) Tensor Dense Synthesizer, (b) Tensor
Random Synthesizer, and (c) Tensor Factorized Synthesizer.

###  3.1 Preliminaries

##### Basic Tensor Synthesizer

As shown in Section 2.2, given a tensor input
$\mathcal{X}\in\mathbb{R}^{H\times W\times C}$ and its 2D reshaped matrix
$X\in\mathbb{R}^{HW\times C}$, the dot-product self-attention could be
interpreted as producing a dimension alignment from $\mathbb{R}^{HW\times C}$
to $\mathbb{R}^{HW\times HW}$. In the case of regular visual transformation
architecture, it will reshape the input feature $X$ as a defined dimensional
vector [13]. However, it may result in the loss of spatial information and a
prohibitively expensive computation when the dimension is high. Thus, in order
to directly process the tensor input $\mathcal{X}\in\mathbb{R}^{H\times
W\times C}$, we propose to synthesize tensor transformation for replacing the
dot-product self-attention in this section.

Formally, let $\mathcal{O}\in\mathbb{R}^{H\times W\times d}$ be the projected
features of $\mathcal{X}\in\mathbb{R}^{H\times W\times C}$ via the mapping
$\mathcal{F}=(\cdot)$, where $C$ and $d$ are the number of channels.

$Z=\mathcal{F}(\mathcal{O}),$ (3)

Intuitively, the $\mathcal{F}=(\cdot)$ could be a $3^{rd}$ tensor product. In
detail, the mapping $\mathcal{F}(\cdot)$ could be composed by three layered
feed-forward network, as followed:

$\mathcal{F}(\mathcal{O})=\mathcal{O}\times_{1}\mathbf{A}_{(H)}\times_{2}\mathbf{A}_{(W)}\times_{3}\mathbf{A}_{(C)}$
(4)

with $\mathbf{A}_{(H)}$, $\mathbf{A}_{(W)}$ , and $\mathbf{A}_{(C)}$ are three
tensors. For a detailed introduction of tensor products, we refer the
interested reader to [10, 36, 34].

##### Definition 1.

Given the tensor $\mathcal{X}\in\mathbb{R}^{I_{1}\times
I_{2}\times\cdots\times I_{N}}$ and the matrix
$\textbf{A}\in\mathbb{R}^{J_{n}\times I_{n}}$, the n-mode product is denoted
by

$\mathcal{X}\times_{n}\textbf{A}$ (5)

and results in a $I_{1}\times I_{2}\times\cdots\times I_{n-1}\times
J_{n}\times I_{n+1}\times\cdots\times I_{N}$ tensor. The entries of this
tensor are defined as

$\left(\mathcal{X}\times{}_{n}\mathbf{A}\right)_{i_{1}i_{2}\ldots
i_{n-1}j_{n}i_{n+1}\ldots i_{N}}=\sum_{i_{n}=1}^{I_{n}}x_{i_{1}i_{2}\ldots
i_{N}}\cdot a_{j_{n}i_{n}}$ (6)

for all $j_{n}=1,\cdots,J_{n}$.

For the convenience of understanding the tensor product, Figure 2 illustrates
an example of n-mode production. Besides, in order to obtain an appropriate
size of the matrix, a method named n-mode matrix unfolding is required.
Depending on this method, we are able to build the layered feed-forward
network as Eq. (4). The multiplication of a k-way tensor $\mathcal{X}$ along
all its modes with matrices $A_{1},\cdots,A_{k}$[34]

$\mathcal{Y}=\mathcal{X}\times{}_{1}A_{1}\times{}_{2}A_{2}\times{}_{3}\cdots\times_{k}A_{k}$
(7)

can be rewritten as the following linear system:

$\operatorname{vec}(\mathcal{Y})=\left(A_{k}\otimes\cdots\otimes A_{2}\otimes
A_{1}\right)\operatorname{vec}(\mathcal{X})$ (8)

where $vec(\cdot)$ means the vector space isomorphism vec:
$\mathbb{R}^{a\times b}\rightarrow\mathbb{R}^{ab}$, which denotes the operator
that stacks the columns of a matrix below one another[34]. Therein, $\otimes$
denotes the Kronecker product operator, with which the Kronecker product of
$\mathbf{A}$ and $\mathbf{B}$ is defined by

$\textbf{A}\otimes\textbf{B}=\begin{bmatrix}A_{11}\textbf{B}&\cdots&A_{1k_{1}}\textbf{B}\\\
\vdots&\ddots&\vdots\\\ A_{a1}\textbf{B}&\cdots&A_{ak_{1}}\textbf{B}\\\
\end{bmatrix}$ (9)

with $A_{ij}$ being an entry in A. Therefore, the mapping
$\mathcal{F}=(\cdot)$ could be presented as the tensor multiplication as the
Eq. 4. To offer a better understanding for this definition, Figure 3 shows the
process of the tensor multiplication with Kronecker product operator.

###  3.2 Synthesizer Tensor Model

##### Tensor Dense Synthesizer

To achieve mentioned dimension alignment on raw tensor $\mathcal{O}$ without
the dot-product self-attention mechanism, we define a tensor transformation to
compute attention coefficients by replacing the dot-product self-attention, as
follows,

$\displaystyle Z$
$\displaystyle=\mathcal{O}\times_{1}\mathbf{A}_{(H)}^{(l)}\times_{2}\mathbf{A}_{(W)}^{(l)}\times_{3}\mathbf{A}_{(C)}^{(l)},$
(10) $\displaystyle{S}$ $\displaystyle=\text{Softmax}\big{(}Z\big{)}$ (11)

with $\mathbf{A}_{(H)}^{(l)}\in\mathbb{R}^{HW\times H}$,
$\mathbf{A}_{(W)}^{(l)}\in\mathbb{R}^{HW\times W}$,
$\mathbf{A}_{(C)}^{(l)}\in\mathbb{R}^{1\times d}$, ${Z}\in\mathbb{R}^{HW\times
HW\times 1}$ and ${S}\in\mathbb{R}^{HW\times HW\times 1}$, and $l$ denotes the
layer number. The combination of Eq. (10) and Eq. (11) is referred to the VDS
module in Section 1. Then, the output is computed by

$Y=\text{Synthesizer}(Q,V)={S}V.$ (12)

This approach eliminates the dot product altogether by replacing $QK^{\top}$
in standard self-attention with the synthesizing function defined in Eq. (11).

##### Tensor Random Synthesizer

In the model of Tensor Dense Synthesizer, the features of input are projected
from $\mathcal{X}\in\mathbb{R}^{H\times W\times C}$ to $HW$ dimensions via the
$3^{rd}$ tensor production. Intuitively, the synthesizer implements the
conditioning on each feature independently. In contrast, the Tensor Random
Synthesizer utilizes the random value as initialized attention weights. Then,
the attention weights could follow the training process to update, or keep
fixed.

Since the synthesizer can be regarded as a self-alignment of dimensions, we
can make an attempt to set $Z$ as a random matrix. And the Random Synthesizer
is defined as:

$Y=\text{Synthesizer}(Q,V)=\text{Softmax}\big{(}R\big{)}V,$ (13)

with $R\in\mathbb{R}^{HW\times HW}$ being a randomly initialized matrix. The
attempt of utilizing the random initial matrix identifies the idea that it is
unnecessary to rely on any information from the individual input feature.
Therefore, the Random Tensor Synthesizer is tending to learn the task-specific
alignment through the training process. According to the token-token self-
attention, it is able to handle the cases that exceed length $H\times W$. In
our work, the factorized approach will address this case, and we will
introduce it in the Section 3.3.

##### Definition 2.

If a tensor
$\mathcal{Y}=\mathcal{X}\times_{1}\mathbf{A}_{(1)}\times_{2}\mathbf{A}_{(2)}\times_{3}\mathbf{A}_{(3)}$
with $\mathcal{Y}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}}$, it could be
assigned the dimension to three transformations. Therein,
$\mathcal{Y}_{1},\mathcal{Y}_{2},\mathcal{Y}_{3}$ are computed as follows.

$\mathcal{Y}_{1}=\mathcal{X}\times_{1}\mathbf{A}_{(1)}\times_{2}\mathbf{A}_{(2)}\times_{3}\mathbf{A}_{(3)}$

with $\mathbf{A}_{(1)}\in\mathbb{R}^{1\times I_{1}}$,
$\mathbf{A}_{(2)}\in\mathbb{R}^{I_{1}I_{2}\times I_{2}}$,
$\mathbf{A}_{(3)}\in\mathbb{R}^{I_{1}I_{2}\times I_{3}}$.

$\mathcal{Y}_{2}=\mathcal{X}\times_{1}\mathbf{A}_{(1)}\times_{2}\mathbf{A}_{(2)}\times_{3}\mathbf{A}_{(3)}$

with $\mathbf{A}_{(1)}\in\mathbb{R}^{I_{1}I_{2}\times I_{1}}$,
$\mathbf{A}_{(2)}\in\mathbb{R}^{1\times I_{2}}$,
$\mathbf{A}_{(3)}\in\mathbb{R}^{I_{1}I_{2}\times I_{3}}$,

$\mathcal{Y}_{3}=\mathcal{X}\times_{1}\mathbf{A}_{(1)}\times_{2}\mathbf{A}_{(2)}\times_{3}\mathbf{A}_{(3)}$

with $\mathbf{A}_{(1)}\in\mathbb{R}^{I_{1}I_{2}\times I_{1}}$,
$\mathbf{A}_{(2)}\in\mathbb{R}^{I_{1}I_{2}\times I_{2}}$,
$\mathbf{A}_{(3)}\in\mathbb{R}^{1\times I_{3}}$.

It is difficult to determine the dimensions of $\mathbf{A}_{(H)}^{(l)}$,
$\mathbf{A}_{(W)}^{(l)}$, $\mathbf{A}_{(C)}^{(l)}$, i.e.,how to assign the
dimensions $HW,H,W,1$ to three transformations.We note that all of the
proposed synthetic attention variants can be mixed in an additive fashion.
Depending on the definition, Eq. (11) can be expressed as:

$\begin{split}S&=\text{Softmax}\big{(}Z_{H}\big{)}V=\text{Softmax}\big{(}Z_{W}\big{)}V\\\
&=\text{Softmax}\big{(}Z_{C}\big{)}V.\end{split}$ (14)

For example, $Z_{H}$ could be computed as follows:

$Z_{H}=\mathcal{O}\times_{1}\mathbf{A}_{(H)}^{(l)}\times_{2}\mathbf{A}_{(W)}^{(l)}\times_{3}\mathbf{A}_{(C)}^{(l)}$

with $\mathbf{A}_{(H)}^{(l)}\in\mathbb{R}^{1\times H}$,
$\mathbf{A}_{(W)}^{(l)}\in\mathbb{R}^{HW\times W}$,
$\mathbf{A}_{(C)}^{(l)}\in\mathbb{R}^{HW\times d}$. In Figure 3, we offer the
visualization plot to describe this definition.

###  3.3 Multiple-level Factorized Tensor Synthesizer

We replace the dot-production in the standard self-attention module with our
synthesizer module in the Tensor Dense Synthesizer. However, the synthesizer
is too burdensome to learn when the size of parameters is too large. As shown
from Eq. (11), the basic $3^{rd}$ tensor synthesizer contains several
parameters $\mathbf{A}_{(H)}^{(l)}\in\mathbb{R}^{HW\times H}$,
$\mathbf{A}_{(W)}^{(l)}\in\mathbb{R}^{HW\times W}$,
$\mathbf{A}_{(C)}^{(l)}\in\mathbb{R}^{d\times 1}$. Therefore, there are
several factorized tensor synthesizer models introduced.

##### Tensor Dense Factorized Synthesizer

In order to reduce the size of parameters, partially avoiding the overfitting
while $HW$ being huge, the parameter matrices could be decomposed w.r.t the
Kronecker product[36]. The decomposition can be conducted on three dimensions
of the feature map, as can be seen in Figure 3.

For example, $\mathbf{A}_{(H)}^{(l)}$ could be approximated by $N$ small
matrices as

$\mathbf{A}_{(H)}^{(l)}=\mathbf{A}_{(H)_{1}}^{(l)}\otimes\mathbf{A}_{(H)_{2}}^{(l)}\otimes\cdots\otimes\mathbf{A}_{(H)_{N}}^{(l)},$
(15)

where $\mathbf{A}_{(H)_{i}}^{(l)}\in\mathbb{R}^{\alpha_{i}\times\beta_{i}}$
with $\alpha_{i},\beta_{i}$ being dimensions,
$HW=\alpha_{1}\alpha_{2}\cdots\alpha_{N},H=\beta_{1}\beta_{2}\cdots\beta_{N}$.
By using the Kronecker product operator, $\mathbf{A}_{(H)}^{(l)}$,
$\mathbf{A}_{(W)}^{(l)}$, $\mathbf{A}_{(C)}^{(l)}$ can be easily factorized to
several very small matrices. The Tensor Dense Factorized Synthesizer is
defined as:

$Y=\text{Synthesizer}(Q,V)=\text{Softmax}(C)V.$ (16)

Where
$C=\mathcal{F}_{H}(\mathbf{A}_{(H)}^{(l)})\times\mathcal{F}_{W}(\mathbf{A}_{(W)}^{(l)})\times\mathcal{F}_{C}(\mathbf{A}_{(C)}^{(l)})$,
$\mathcal{F}(\cdot)$ means the decomposition of the tensor and
$C\in\mathbb{R}^{HW\times HW}$.

##### Tensor Random Factorized Synthesizer

Similarly, the factorized synthesizer could also be applied into Tensor Random
Synthesizer. The idea is decomposing the matrix $R$ into several low rank
matrices $R_{i}\in\mathbb{R}^{\alpha_{i}\times\beta_{i}}$.

$Y=\text{Synthesizer}(Q,V)=\text{Softmax}(\mathcal{R})V,$ (17)

Where $\mathcal{R}=R_{1}\otimes R_{2}\otimes\cdots\otimes R_{N}$, and
$\mathcal{R}\in\mathbb{R}^{HW\times HW}$. About the low rank matrices $R_{i}$,
$\alpha_{1}\alpha_{2}\cdots\alpha_{N}=HW,\beta_{1}\beta_{2}\cdots\beta_{N}=HW$,
and $\alpha_{i},\beta_{i}<<HW$. With the reduced size of parameters, it is
able to avoid the overfitting problem.

##### Mixture Tensor Synthesizer

After introduced the above three kinds of tensor synthesizer models, it raises
a compounded model being the synthesizers mixed in additive fashion. For
instance:

$Y=\text{Softmax}(\theta_{1}S_{1}(\mathcal{O})+\cdots+\theta_{N}S_{N}(\mathcal{O}))V,$
(18)

Where $\theta_{i}$ are the learnable weights, and $\sum_{1}^{N}\theta_{i}=1$.
$S_{i}$ are the different synthesizer models. In the following experiment
part, the Mixture Tensor Synthesizer composes Tensor Random Factorized
Synthesizer and Tensor Dense Synthesizer together. The expression could be
outlined as follow:

$Y=\text{Softmax}(\theta_{1}\mathcal{R}+\theta_{2}Z)V,$ (19)

Where $Z$ is presented in Eq. (10). By employing more than one kind of
synthesizer, it is possible to obtain a better performance than a composed
individual synthesizer.

Model | None | CT | L | SD | SR | FSR | FSD | MS | STT | STTH | STTW
---|---|---|---|---|---|---|---|---|---|---|---
Accuracy | 65.56 | 66.1 | 66.21 | 67.12 | 66.97 | 66.79 | 66.57 | 66.21 | 66.69 | 66.82 | 66.27
Table 1: The Accuracy of Each Model. Noise | 0.01 | 0.02 | 0.03 | 0.05 | 0.04 | 0.06 | 0.07 | 0.08 | 0.09 | 0.1
---|---|---|---|---|---|---|---|---|---|---
None | 45.96 | 34.35 | 27.8 | 22.83 | 20.01 | 17.86 | 16.23 | 15.07 | 14.45 | 13.76
CT | 47.02 | 35.83 | 29.21 | 24.7 | 21.49 | 19.33 | 17.41 | 16.51 | 15.42 | 14.8
L | 41.51 | 28.54 | 21.69 | 18.56 | 16.16 | 14.53 | 13.57 | 12.41 | 12.17 | 11.5
SD | 48.71 | 37.79 | 31.54 | 27.3 | 23.7 | 22.23 | 20.18 | 18.92 | 17.45 | 17.03
SR | 45.93 | 34.02 | 27.06 | 23.01 | 19.91 | 17.36 | 16.02 | 14.93 | 13.93 | 13.4
FSR | 46.51 | 37.28 | 31.69 | 27.56 | 24.14 | 22.6 | 20.62 | 19.75 | 18.25 | 17.76
FSD | 68.9 | 37.5 | 30.42 | 25.89 | 22.85 | 20.7 | 19.33 | 17.91 | 16.87 | 16.07
STT | 47.29 | 36.81 | 31.98 | 28.11 | 25.49 | 23.75 | 22.39 | 21.44 | 20.09 | 19.54
STTH | 44.66 | 30.92 | 23.26 | 19.58 | 17.08 | 15.12 | 14.04 | 13.35 | 12.61 | 12.26
STTW | 42.2 | 30.21 | 23.65 | 19.92 | 17.53 | 16.41 | 15.18 | 14.65 | 14.33 | 13.64
MS | 48.2 | 36.8 | 29.55 | 25.13 | 21.45 | 19.34 | 17.74 | 16.61 | 15.76 | 15.26
Table 2: Classification Result on Images with Relatively Small Gaussian Noise. Rotation | 30 | 60 | 90 | 120 | 150 | 180 | 210 | 240 | 270 | 300 | 330
---|---|---|---|---|---|---|---|---|---|---|---
None | 37.58 | 26.51 | 23.07 | 21.12 | 19.4 | 19.39 | 18.68 | 18.89 | 18.27 | 17.74 | 17.4
CT | 36.54 | 36.57 | 23.27 | 21.25 | 19.75 | 19.44 | 19.7 | 18.61 | 18.19 | 18.46 | 17.75
L | 37.53 | 26.86 | 23.61 | 21.5 | 20.16 | 18.61 | 19.19 | 17.95 | 17.62 | 17.95 | 17.95
SD | 37.9 | 38.64 | 23.24 | 21.79 | 19.26 | 19.45 | 19.36 | 18.51 | 17.55 | 17.6 | 17.98
SR | 33.72 | 24.7 | 22.06 | 19.88 | 19 | 18.17 | 18.03 | 17.22 | 17.03 | 17.23 | 16.79
FSR | 37.04 | 26.86 | 23.1 | 20.69 | 19.72 | 18.69 | 18.7 | 18.27 | 17.69 | 17.17 | 17.08
FSD | 39.27 | 28.85 | 23.55 | 22.85 | 20.37 | 19.85 | 19.95 | 18.71 | 18.86 | 18.34 | 17.88
MS | 37.91 | 26.93 | 22.96 | 21.74 | 19.61 | 18.89 | 18.76 | 18.01 | 18.06 | 17.62 | 17.65
STT | 38.71 | 38.36 | 24.51 | 21.8 | 20.97 | 20.36 | 20.89 | 19.93 | 19.86 | 19.12 | 20
STTH | 42.32 | 30.44 | 26.28 | 24.53 | 22.17 | 22.18 | 22.37 | 21.83 | 20.85 | 20.07 | 20.16
STTW | 45.53 | 31.72 | 27.21 | 24 | 22.39 | 22.23 | 22.03 | 21.27 | 20.61 | 20.31 | 19.89
Table 3: Classification Result on Images with Rotations.

##  4 Experiments

In this section, we investigate the performance of our proposed STT module
series on basic image classification. The experiments focus on the performance
of the robustness of the model. For convenience, we abbreviate the proposed
STT series and comparison methods. The detail of information are listed in
Table 4.

ABBR | Combination
---|---
None | Convolution Baseline
CT |  CNN + Transformer [37]
L |  CNN + Linformer [41]
CNN | Double CNN
SD |  CNN + Matrix Dense Synthesizer [35]
STT | CNN + Tensor Dense Synthesizer
SR | CNN + Tensor Random Synthesizer
STTH | CNN + Tensor Height Synthesizer
STTW | CNN + Tensor Width Synthesizer
FSD | CNN + Tensor Dense Factorized Synthesizer
FSR | CNN + Tensor Random Factorized Synthesizer
MS | CNN + Mixture Tensor Synthesizers
Table 4: Abbreviation of Model Zoo. ABBR indicates abbreviations of our
proposed STT series, with Double CNN being the convolution baseline followed
by its repetition, Matrix Dense Synthesizer being the basic synthesizer
proposed in [35], and Tensor Dense Synthesizer being our basic tensor
synthesizer for vision tasks, Tensor Height Synthesizer being the tensor
synthesizer with an attention map full-rank in height, and Mixture Tensor
Synthesizer being the synthesizers mixed in additive fashion.

###  4.1 Image Classification

####  4.1.1 Implementation Details

STT series are validated on the benchmark image dataset CIFAR10[25] for
classification. The CIFAR10 is composed by 10 classes, and it has 50000 images
for training, 10000 for testing. To validate the functionality of the STT
series as attention modules applied to a deep learning framework, we employed
three different perturbations to test the model robustness of above all
approaches. Applied perturbations are, namely, Gaussian noise, image
rotations, and flipping.

  1. 1.

Gaussian noise: random noise is added on every pixel of the input and we want
to test whether the attention computed on the global scale out performs the
split ones.

  2. 2.

Rotations and flips: once the rotations or flips are exerted, the order among
channels would change correspondingly, while STT series treat the feature map
on the whole.

####  4.1.2 Ablation Study

To make fair comparisons with the traditional self-attention modules, we
choose a two-layer CNN as the baseline. The evaluation metric is the
classification accuracy. Besides, we select Transformer[37], Linformer[41],
and Matrix Dense Synthesizer[35] to test their perturbation robustness in the
same baseline structure.

About mentioned traditional self-attention modules, The Transformer and
Linformer modules employ the dot-product to compute the similarity between
token to token. And Matrix Dense Synthesizer also applies the synthesizer
mechanism to replace the dot production, but it doesn’t leverage the basic
tensor synthesizer in their module. Through the comparison, we could prove our
STT series satisfy the requirement to become a valuable alternative to the
dot-product-based self-attention model. Moreover, it has better robustness
performance.

####  4.1.3 Quantitative Analysis on Robust Classification

According to our proposed STT series, one of the advantages mentioned in
Section 1 is that they possess strong robustness against external
perturbations. The reason is that they calculate attention maps without
splitting the feature maps. Thus, the baseline model reduces the reliance on
the input, and the underlying structure of the feature map has more
possibility to preserved.

Our STT series has a more negligible effect from external perturbation on the
raw input than the dot-production-based self-attention modules. Hence, to
evaluate the robustness validation for our proposed STT series, we conduct the
image classification task to compare the STT series with traditional self-
attention modules.

##### Image Classification

Quantitative comparisons among various models are represented in the Table 1.
We examine all the models mentioned in the Model Zoo, the results of accuracy
are scarcely distinguishable from each model. According to the similar
accuracy, our experiment demonstrates that our STT series have equal
performance in the image classification task. Hence, the result proves that
the dot-production mechanism is not the best choice for the self-attention
module. The tensor synthesizer also shows the qualification to replace
traditional dot-production as a valuable alternative. Although the results are
similar under the no-noise situation, we focus on the ability to resist
perturbation disturbance. Because the no-noise test only proves that the
synthesizer module has the potential to replace the dot-production mechanism.
The following robustness experiment would illustrate the unique advantage of
the STT series.

##### Gaussian Noise

As a common statistical noise caused by sensor during sampling or
transmission, Gaussian noise is a widespread type of disturbance for digital
images. Varied levels of Gaussian noises are utilized to corrupt the images
for robustness evaluation, with detailed settings and results being collected
in Table 2.

As shown in Table 2, the accuracy of the classification decreases with the
Gaussian noise increasing. Under the disturbance of Gaussian noise, the STT
series doesn’t represent the worse accuracy than the models with dot-
production. In contrast, the STT series shows a more outstanding noise
resistance. During the noise is still small, the FSD leads to the best result.
With the noise becomeing large, the STT keeps the highest accuracy than other
modules. Among the STT series, the worse performance of STTH and STTW might
result from the inferior representation because the information is only
gathered from one side of the feature map compared to the transformer.
Compared with the modules employing dot-product, synthesizing modules appear
higher accuracy when Gaussian noise influences them.

##### Images with Rotations

Rotations are common in image augmentation process. We apply rotations to the
original images to test the robustness with the model pool. Detailed settings
and results are recorded in Table 3.

In Table 3, it is obvious that the classification performance of the whole
model zoo is impaired when rotation increases. However, the accuracy of the
STT series represents a similar decreasing tendency. It proves that the STT
series has the same performance as dot-production models when they are
suffering the rotation noise. Despite the STTH and STTW modules, other
approaches don’t reveal the resistance of rotation noise. While transformer
weakens the baseline CNN structure, STTH and STTW have a relatively better
accuracy performance. Based on such result, we make an assumption that the
full-rank attention map to give them a better robustness against the effect of
rotation. This assumption could also explain that the STT has a relatively
better result, because it also has a full-rank attention map.

Module | horizon | vertical | ho+ve
---|---|---|---
None | 64.74 | 25.18 | 25.08
CT | 64.93 | 25.52 | 25.45
L | 65.31 | 25.2 | 25.05
SD | 65.7 | 25.08 | 25.21
SR | 65.86 | 26.06 | 25.72
FSR | 65.08 | 24.63 | 25.14
FSD | 66.28 | 26.41 | 26.58
MS | 65.6 | 25.75 | 25.99
STT | 66.09 | 27.16 | 27.32
STTH | 66.09 | 29.29 | 28.9
STTW | 65.77 | 27.45 | 27.46
Table 5: Classification results on images with Flips, where he+ve means that both horizontal and vertical flips are applied. Meric | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE | CIDEr
---|---|---|---|---|---|---|---
LSTM | 0.7191 | 0.5153 | 0.3468 | 0.2339 | 0.2725 | 0.5699 | 0.8128
Transformer | 0.7077 | 0.5035 | 0.3392 | 0.2289 | 0.2692 | 0.5621 | 0.7970
SD | 0.7921 | 0.6566 | 0.5386 | 0.4457 | 0.3686 | 0.7211 | 1.2099
SR | 0.7888 | 0.6542 | 0.5359 | 0.4433 | 0.3688 | 0.7213 | 1.2040
MS | 0.7925 | 0.6569 | 0.5383 | 0.4451 | 0.3691 | 0.7225 | 0.2104
FSR | 0.7876 | 0.6505 | 0.5293 | 0.4345 | 0.3646 | 0.7159 | 1.1793
Table 6: Image captioning performance with Scaled-Dot-Product self-attention
being replaced by STT series.

##### Robustness against Flipping

The last perturbation is images flipping, which could be categorized into
horizon flipping, vertical flipping, and flipping both in horizon and
vertical. The results are recorded in Table 5.

From Table 5, we can see that accuracy decreases drastically once the vertical
transformation is applied. Compared with the STT series and the traditional
self-attention modules, their accuracy doesn’t appear a distinguished
difference. It demonstrates that the synthesizer modules are not more
vulnerable under the flipping noise. Even though all the models have severe
impair from vertical flipping noise, the STT series has slightly better
accuracy.

###  4.2 Image Captioning

####  4.2.1 Implementation Detail

We report our results on Microsoft COCO-2014 [6] dataset with $82,783$ images
for training, $40,504$ images for validation and $40,775$ images for testing.

##### Encoder-Decoder Architecture

To test the potency of STT series as a powerful alignment module and a
substitution of transformer, the comparison of the series is all implemented
with a common encoder-decoder architecture adopted in [44]. The only
difference is that [44] handles the sequential input with LSTM blocks.

##### Self-attention among Pixels

Sinusoidal positional encoding is adopted here for texts on all of the models.
As for visual feature maps, we form a 1024-dimensional vector to represent its
vertical position, and another 1024-dimensional vector for its horizontal
position, then concatenate these two vectors as final 2048-dimensional
position feature vector.

##### Pre-training and Beam Search

Firstly, rather than training a new model from scratch, we also adopt a
Resnet-101 pre-trained on Imagenet to be the image encoder, with the last two
layers being replaced by down-sampling convolution, and it will be fine-tuned
with COCO dataset. Secondly, to improve efficiency and preserve the results,
we take the pre-trained GLOVE [31] for word embedding. To be noticed, it’s not
a common NLP practice because the pre-training is not carried out on the
modules for alignment. Besides, Beam Search is adopted to alleviate
computation burden. It executes graph search on a limited sub-optimal set,
which saves a lot of time for all the models.

####  4.2.2 Metrics

The results are reported under standard evaluation protocol, where captioning
metrics including full set of BLEU [23], along with METEOR [11], CIDEr [38],
ROUGE [26]. Most of them are based on $n$-gram to evaluate the quality of the
generated sentences with ground truths, and they complement with each other to
yield fairer results.

####  4.2.3 Quantitative Analysis of Image Captioning

The performance quantified by the seven metrics is summarized in Table 6, from
which we can see that both of LSTM-based and transformer-based image
captioning framework perform worse than STT series for all seven metrics. It
is interesting that LSTM-based model appears to be inferior to transformer-
based models, given that transformer frees the model from sequential input
dependence. Furthermore, all of modules from STT series tend to improve the
results than traditional methods. Besides, MS (Tensor Mixture Synthesizer)
seems to achieve most of the best results among the model zoo. Moreover, in
comparison with baseline caption models with LSTM and Transformer, both SR
(Tensor Random Synthesizer) and FSR (Tensor Factorized Random Synthesizer)
yield better results with less time for testing, and fewer parameters. The
alignments in transformer are conducted on one side of the feature map while
dense synthesizer enables a competitive attention for three sides of the
feature tensor, which might result in the overall performance superiority of
STT against the baselines.

##  5 Conclusion

In this paper, we present a self-attention plug-in module with its variants,
named the series of STT. It employs the tensor transformation for self-
alignment to replace the traditional dot-product multiplication in the self-
attention module. Through this novel approach, the shortage of exhaustive and
redundant computation caused by dot-production is dramatically mitigated. In
our image caption experiments, the STT series achieve most of the best results
with less time in testing, and use fewer parameters. Moreover, because the
synthesizing mechanism notably reduces the input feature’s reliance, the
underlying structure is preserved. The STT series strongly facilitate the
robustness of the baseline model. In the image classification experiments, the
STT series represent the competitive robustness encountering multiple external
perturbations. It demonstrates the STT series have the potential to improve
the robustness, lower the overfitting, and the flexibility to be extended to
more deep learning architectures.

## References

  * [1] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu.  Multiple object recognition with visual attention.  In ICLR, 2015.
  * [2] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le.  Attention augmented convolutional networks.  In ICCV, pages 3286–3295, 2019.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, volume 12346, pages 213–229, 2020.
  * [4] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua.  Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning.  In CVPR, pages 5659–5667, 2017.
  * [5] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In ICML, pages 1691–1703, 2020.
  * [6] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng.  A^2-nets: Double attention networks.  In NIPS, pages 350–359, 2018.
  * [8] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller.  Rethinking attention with performers.  In ICLR, 2021.
  * [9] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi.  On the relationship between self-attention and convolutional layers.  In ICLR, 2020.
  * [10] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle.  A multilinear singular value decomposition.  SIAM journal on Matrix Analysis and Applications, 21(4):1253–1278, 2000.
  * [11] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014.
  * [12] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas.  Attention is not all you need: pure attention loses rank doubly exponentially with depth.  In ICML, volume 139, pages 2793–2803, 2021.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [14] Shahriar Rahman Fahim, Yeahia Sarker, Subrata K. Sarker, Md. Rafiqul Islam Sheikh, and Sajal K. Das.  Self attention convolutional neural network with time series imaging based feature extraction for transmission line fault detection and classification.  Electric Power Systems Research, 187:106437, 2020.
  * [15] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In CVPR, pages 3146–3154, 2019.
  * [16] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans.  Axial attention in multidimensional transformers.  arXiv preprint arXiv:1912.12180, 2019.
  * [17] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei.  Relation networks for object detection.  In CVPR, pages 3588–3597, 2018.
  * [18] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi.  Gather-excite: Exploiting feature context in convolutional neural networks.  In NIPS, pages 9423–9433, 2018.
  * [19] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, pages 7132–7141, 2018.
  * [20] Lun Huang, Wenmin Wang, Jie Chen, and Xiaoyong Wei.  Attention on attention for image captioning.  In ICCV, pages 4633–4642, 2019.
  * [21] Lang Huang, Yuhui Yuan, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  Interlaced sparse self-attention for semantic segmentation.  arXiv preprint arXiv:1907.12273, 2019.
  * [22] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In ICCV, pages 603–612, 2019.
  * [23] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In CVPR, pages 3128–3137, 2015.
  * [24] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In ICLR, 2020.
  * [25] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [26] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, pages 74–81, 2004.
  * [27] Chenxi Liu, Junhua Mao, Fei Sha, and Alan L. Yuille.  Attention correctness in neural image captioning.  In AAAI, pages 4176–4182, 2017.
  * [28] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.  Recurrent models of visual attention.  In NIPS, pages 2204–2212, 2014.
  * [29] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.  Stand-alone self-attention in vision models.  In NIPS, pages 68–80, 2019.
  * [30] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In ICML, pages 4055–4064, 2018.
  * [31] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In EMNLP, pages 1532–1543, 2014.
  * [32] Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.  Compressive transformers for long-range sequence modelling.  In ICLR, 2020.
  * [33] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.  Efficient content-based sparse attention with routing transformers.  Trans. Assoc. Comput. Linguistics, 9:53–68, 2021.
  * [34] Matthias Seibert, Julian Wörmann, Rémi Gribonval, and Martin Kleinsteuber.  Learning co-sparse analysis operators with separable structures.  IEEE Transactions on Signal Processing, 64(1):120–130, 2015.
  * [35] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng.  Synthesizer: Rethinking self-attention in transformer models, 2020.
  * [36] Charles F Van Loan.  The ubiquitous kronecker product.  Journal of computational and applied mathematics, 123(1-2):85–100, 2000.
  * [37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, pages 5998–6008, 2017.
  * [38] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In CVPR, pages 4566–4575, 2015.
  * [39] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In CVPR, 2017.
  * [40] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Axial-deeplab: Stand-alone axial-attention for panoptic segmentation, 2020\.
  * [41] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity, 2020.
  * [42] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In ECCV, 2018.
  * [43] Qingyang Wu, Zhenzhong Lan, Jing Gu, and Zhou Yu.  Memformer: The memory-augmented transformer, 2020.
  * [44] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, pages 2048–2057, 2015.
  * [45] Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big bird: Transformers for longer sequences.  In NIPS, 2020.
  * [46] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun.  Exploring self-attention for image recognition.  In CVPR, pages 10076–10085, 2020.
  * [47] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  In ICLR, 2021.