# Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped
Attention

Sitong Wu1, 2 Tianyi Wu1, 2 Haoru Tan3 Guodong Guo1, 2 Corresponding author.

###### Abstract

Recently, Transformers have shown promising performance in various vision
tasks. To reduce the quadratic computation complexity caused by the global
self-attention, various methods constrain the range of attention within a
local region to improve its efficiency. Consequently, their receptive fields
in a single attention layer are not large enough, resulting in insufficient
context modeling. To address this issue, we propose a Pale-Shaped self-
Attention (PS-Attention), which performs self-attention within a pale-shaped
region. Compared to the global self-attention, PS-Attention can reduce the
computation and memory costs significantly. Meanwhile, it can capture richer
contextual information under the similar computation complexity with previous
local self-attention mechanisms. Based on the PS-Attention, we develop a
general Vision Transformer backbone with a hierarchical architecture, named
Pale Transformer, which achieves 83.4%, 84.3%, and 84.9% Top-1 accuracy with
the model size of 22M, 48M, and 85M respectively for $224\times 224$
ImageNet-1K classification, outperforming the previous Vision Transformer
backbones. For downstream tasks, our Pale Transformer backbone performs better
than the recent state-of-the-art CSWin Transformer by a large margin on ADE20K
semantic segmentation and COCO object detection & instance segmentation. The
code will be released on https://github.com/BR-IDL/PaddleViT.

## Introduction

Inspired by the success of Transformer (Vaswani et al. 2017) on a wide range
of tasks in natural language processing (NLP) (McCann et al. 2017; Howard and
Ruder 2018), Vision Transformer (ViT) (Dosovitskiy et al. 2021) first employed
a pure Transformer architecture for image classification, which shows the
promising performance of Transformer architecture for vision tasks. However,
the quadratic complexity of the global self-attention results in expensive
computation costs and memory usage especially for high-resolution scenarios,
making it unaffordable for applications in various vision tasks.

A typical way to improve the efficiency is to replace the global self-
attention with local ones. A crucial and challenging issue is how to enhance
the modeling capability under the local settings. For example, Swin (Liu et
al. 2021) and Shuffle Transformer (Huang et al. 2021) proposed shifted window
and shuffled window, respectively (Figure 1(b)), and alternately used two
different window partitions (i.e., regular window and the proposed window) in
consecutive blocks to build cross-window connections. MSG Transformer (Fang et
al. 2021) manipulated the messenger tokens to exchange information across
windows. Axial self-attention (Wang et al. 2020) treated the local attention
region as a single row or column of the feature map (Figure 1(c)). CSWin (Dong
et al. 2021) proposed cross-shaped window self-attention (Figure 1(d)), which
can be regarded as a multiple row and column expansion of axial self-
attention. Although these methods achieve excellent performance and are even
superior to the CNN counterparts, the dependencies in each self-attention
layer are not rich enough for capturing sufficient contextual information.

(a) Global Self-Attention

Regular Window

Shifted Window

Shuffled Window

Messenger

(b) Window-based Self-Attention

(c) Axial Self-Attention (d) Cross-Shaped (e) Pale-Shaped

Window Self-Attention Self-Attention (ours)

Figure 1: Illustration of different self-attention mechanisms in Transformer
backbones. (a) is the standard global self-attention. (b) Window-based self-
attention mechanisms perform attention inside each window, and introduce
various strategies to build cross-window connections. Different colors in (b)
represent different windows. In (c), (d), and (e), the input features are
first split into multiple groups, one of which is illustrated by the shadow
area, and the self-attention is conducted within each group. Thus, for a
reference token denoted by the red dot, it can interact directly with the
tokens covered by the shadow area.

In this work, we propose a Pale-Shaped self-Attention (PS-Attention) to
capture richer contextual dependencies efficiently. Specifically, the input
feature maps are first split into multiple pale-shaped regions spatially. Each
pale-shaped region (abbreviating as pale) is composed of the same number of
interlaced rows and columns of the feature map. The intervals between adjacent
rows or columns are equal for all the pales. For example, the pink shadow in
Figure 1(e) indicates one of the pales. Then, self-attention is performed
within each pale. For any token, it can directly interact with other tokens
within the same pale, which endows our method with the capacity of capturing
richer contextual information in a single PS-Attention layer. To further
improve the efficiency, we develop a more efficient parallel implementation of
the PS-Attention. Benefit from the larger receptive fields and stronger
context modeling capability, our PS-Attention shows superiority to the
existing local self-attention mechanisms illustrated in Figure 1.

Based on the proposed PS-Attention, we design a general vision transformer
backbone with a hierarchical architecture, named Pale Transformer. We scale
our approach up to get a series of models, including Pale-T (22M), Pale-S
(48M), and Pale-B (85M), reaching significantly better performance than
previous approaches. Our Pale-T achieves 83.4% Top-1 classification accuracy
on ImageNet-1k, 50.4% single-scale mIoU on ADE20K (semantic segmentation),
47.4 box mAP (object detection) and 42.7 mask mAP (instance segmentation) on
COCO, outperforming the state-of-the-art backbones by +0.7%, +1.1%, +0.7, and
+0.5, respectively. Furthermore, our largest variant Pale-B is also superior
to the previous methods, achieving 84.9% Top-1 accuracy on ImageNet-1K, 52.2%
single-scale mIoU on ADE20K, 49.3 box mAP and 44.2 mask mAP on COCO.

## Related Work

ViT (Dosovitskiy et al. 2021), which takes the input image as a sequence of
patches, has paved a new way and shown promising performance for many vision
tasks dominated by CNNs over the years. A line of previous Vision Transformer
backbones mainly focused on the following two aspects to better adapt to
vision tasks: (1) Enhancing the locality of Vision Transformers. (2) Seeking a
better trade-off between performance and efficiency.

### Locally-Enhanced Vision Transformers

Different from CNNs, the inductive bias for local connections is not involved
in the original Transformer, which may lead to insufficient extraction of
local structures, such as lines, edges, and color conjunctions. Many works are
devoted to strengthening the local feature extraction of Vision Transformers.
The earliest approach is to replace the single-scale architecture of ViT with
a hierarchical one to obtain multi-scale features (Wang et al. 2021b). Such
design is followed by many works afterward (Liu et al. 2021; Huang et al.
2021; Yang et al. 2021; Dong et al. 2021). Another way is to combine CNNs and
Transformers. Mobile-Former (Chen et al. 2021b), Conformer (Peng et al. 2021)
and DS-Net (Mao et al. 2021) integrated the CNN and Transformer features by
the well-designed dual-branch structures. In contrast, Local ViT (Li et al.
2021b), CvT (Wu et al. 2021a) and Shuffle Transformer (Huang et al. 2021) only
inserted several convolutions into some components of Transformer. Besides,
some works obtain richer features by fusing the multi-branch with different
scales (Chen, Fan, and Panda 2021) or cooperating with local attention (Han et
al. 2021; Zhang et al. 2021; Chu et al. 2021a; Li et al. 2021a; Yuan et al.
2021b).

Figure 2: (a) The overall architecture of our Pale Transformer. (b) The
composition of each block. (c) Illustration of parallel implementation of PS-
Attention. For a reference token (red dot), it can directly interact with the
tokens within the shadow area.

### Efficient Vision Transformers

The mainstream research on improving the efficiency for Vision Transformer
backbones has two folds: reducing the redundant calculations via pruning
strategies and designing more efficient self-attention mechanisms.

##### Pruning Strategies for Vision Transformers.

For pruning, the existing methods can be divided into three categories: (1)
Token Pruning. DVT (Wang et al. 2021d) proposed a cascade Transformer
architecture to adaptively adjust the number of tokens according to the
hardness for classification of the input image. Considering that tokens with
irrelevant or even confusing information may be detrimental to image
classification, some works proposed to locate discriminative regions and
progressively drop less informative tokens by learnable sampling (Rao et al.
2021; Yue et al. 2021) and reinforcement learning (Pan et al. 2021)
strategies. However, such unstructured sparsity results in incompatibility
with dense prediction tasks. Some structure-preserving token selection
strategies were implemented via token pooling (Chen et al. 2021a) and a slow-
fast updating (Xu et al. 2021). (2) Channel Pruning. VTP (Zhu et al. 2021a)
presented a simple but effective framework to remove the reductant channels.
(3) Attention Sharing. Based on the observation that attention maps from
continuous blocks are highly correlated, PSViT (Chen et al. 2021a) was
proposed to reuse the attention calculation process between adjacent layers.

##### Efficient Self-Attention Mechanisms.

Considering that the quadratic computation complexity is caused by self-
attention, many methods are committed to improving its efficiency while
avoiding performance decay (Wang et al. 2021b; Zhu et al. 2021b; Liu et al.
2021; Huang et al. 2021). One way is to reduce the sequence length of key and
value. PVT (Wang et al. 2021b) proposed a spatial reduction attention to
downsample the scale of key and value before computing attention. Deformable
attention (Zhu et al. 2021b) used a linear layer to select several keys from
the full set, which can be regarded as a sparse version of global self-
attention. However, excessive downsampling will lead to information confusion,
and deformable attention relies heavily on a high-level feature map learned by
CNN and may not be directly used on the original input image. Another way is
to replace the global self-attention with local self-attention, which limits
the range of each self-attention layer into a local region. As shown in Figure
1(b), the feature maps are first divided into several non-overlapping square
regular windows (indicated with diverse colors), and the self-attention is
performed within each window individually. The key challenge for the design of
local self-attention mechanisms is to bridge the gap between local and global
receptive fields. A typical manner is to build connections across regular
square windows. For example, alternately using regular window and another
newly designed window partition manner (shifted window (Liu et al. 2021) or
shuffled window (Huang et al. 2021) in Figure 1(b)) in consecutive blocks, and
manipulating messenger tokens to exchange information across windows (Fang et
al. 2021). Besides, axial self-attention (Wang et al. 2020) achieves longer-
range dependencies in horizontal and vertical directions respectively by
performing self-attention in each single row or column of the feature map.
CSWin (Dong et al. 2021) proposed a cross-shaped window self-attention region
including multiple rows and columns. Although these existing local attention
mechanisms can provide opportunities for breaking through the local receptive
fields to some extent, their dependencies are not rich enough to capture
sufficient contextual information in a single self-attention layer, which
limits the modeling capacity of the whole network.

The most related to our work is CSWin (Dong et al. 2021), which developed a
cross-shaped window self-attention mechanism for computing self-attention in
the horizontal and vertical stripes, while our proposed PS-Attention computes
self-attention in the pale-shaped regions. Moreover, the receptive fields of
each token in our method are much wider than CSWin, which also endows our
approach with stronger context modeling capacity.

## Methodology

In this section, we first present our Pale-Shaped self-Attention (PS-
Attention) and its efficient parallel implementation. Then, the composition of
the Pale Transformer block is given. Finally, we describe the overall
architecture and variants configurations of our Pale Transformer backbone.

### Pale-Shaped self-Attention

For capturing dependencies varied from short-range to long-range, we propose
Pale-Shaped self-Attention (PS-Attention), which computes self-attention
within a pale-shaped region (abbreviating as pale). As shown in the pink
shadow of Figure 1(e), one pale contains $s_{r}$ interlaced rows and $s_{c}$
interlaced columns, which covers a region containing
$(s_{r}w+s_{c}h-s_{r}s_{c})$ tokens. We define $(s_{r},s_{c})$ as the pale
size. Given an input feature map $X\in\mathcal{R}^{h\times w\times c}$, we
first split it into multiple pales $\\{P_{1},...,P_{N}\\}$ with the same size
$(s_{r},s_{c})$, where $P_{i}\in\mathcal{R}^{(s_{r}w+s_{c}h-s_{r}s_{c})\times
c},i\in\\{1,2,...,N\\}$. The number of pales is equal to
$N=\frac{h}{s_{r}}=\frac{w}{s_{c}}$, which can be ensured by padding or
interpolation operation. For all pales, intervals between adjacent rows or
columns are the same. The self-attention is then performed within each pale
individually. As illustrated in Figure 1, the receptive field of PS-Attention
is significantly wider and richer than all the previous local self-attention
mechanisms, enabling more powerful context modeling capacity.

#### Efficient Parallel Implementation.

To further improve the efficiency, we decompose the vanilla PS-Attention
mentioned above into row-wise and column-wise attention, which perform self-
attention within row-wise and column-wise token groups, respectively.
Specifically, as shown in Figure 2(c), we first divide the input feature
$X\in\mathcal{R}^{h\times w\times c}$ into two independent parts
$X_{r}\in\mathcal{R}^{h\times w\times\frac{c}{2}}$ and
$X_{c}\in\mathcal{R}^{h\times w\times\frac{c}{2}}$ in the channel dimension,
which are then split into multiple groups for row-wise and column-wise
attention respectively.

$\displaystyle
X_{r}=[X_{r}^{1},...,X_{r}^{N_{r}}],X_{c}=[X_{c}^{1},...,X_{c}^{N_{c}}],$ (1)

where $N_{r}=h/s_{r}$, $N_{c}=w/s_{c}$, $X_{r}^{i}\in\mathcal{R}^{s_{r}\times
w\times c}$ contains $s_{r}$ interlaced rows, and
$X_{c}^{j}\in\mathcal{R}^{h\times s_{c}\times c}$ contains $s_{c}$ interlaced
columns.

Then, the self-attention is conducted within each row-wise and column-wise
token group, respectively. Similar to (Wu et al. 2021a), we use three
separable convolution layers $\phi_{Q}$, $\phi_{K}$, and $\phi_{V}$ to
generate the query, key, and value.

$\displaystyle Y_{r}^{i}={\rm
MSA}(\phi_{Q}(X_{r}^{i}),\phi_{K}(X_{r}^{i}),\phi_{V}(X_{r}^{i})),$ (2)
$\displaystyle Y_{c}^{i}={\rm
MSA}(\phi_{Q}(X_{c}^{i}),\phi_{K}(X_{c}^{i}),\phi_{V}(X_{c}^{i})),$

where $i\in\\{1,2,...,N\\}$, and MSA indicates the Multi-head Self-Attention
(Dosovitskiy et al. 2021).

Finally, the outputs of row-wise and column-wise attention are concatenated
along channel dimension, resulting in the final output
$Y\in\mathcal{R}^{h\times w\times c}$,

$\displaystyle Y={\rm Concat}(Y_{r},Y_{c}),$ (3)

where $Y_{r}=[Y_{r}^{1},...,Y_{r}^{N_{r}}]$ and
$Y_{c}=[Y_{c}^{1},...,Y_{c}^{N_{c}}]$.

Compared to the vanilla implementation of PS-Attention within the whole pale,
such a parallel mechanism has a lower computation complexity. Furthermore, the
padding operation only needs to ensure $h$ can be divisible by $s_{r}$ and $w$
can be divisible by $s_{c}$, rather than $\frac{h}{s_{r}}=\frac{w}{s_{c}}$.
Therefore, it is also conducive to avoiding excessive padding.

#### Complexity Analysis.

Given the input feature of size $h\times w\times c$ and pale size
$(s_{r},s_{c})$, the standard global self-attention has a computational
complexity of

$\mathcal{O}_{\text{Global}}=4hwc^{2}+2c(hw)^{2},$ (4)

however, our proposed PS-Attention under the parallel implementation has a
computational complexity of

$\mathcal{O}_{\text{Pale}}=4hwc^{2}+hwc(s_{c}h+s_{r}w+27)<<\mathcal{O}_{\text{Global}},$
(5)

which can obviously alleviate the computation and memory burden compared with
the global one, since $2hw>>(s_{c}h+s_{r}w+27)$ always holds. The detailed
derivations of Eq. (4) and Eq. (5) are provided in the supplementary material.

### Pale Transformer Block

As shown in Figure 2(b), our Pale Transformer block consists of three
sequential parts, the conditional position encoding (CPE) for dynamically
generating the positional embedding, the proposed PS-Attention module for
capturing contextual information, and the MLP module for feature projection.
The forward pass of the $l$-th block can be formulated as follows:

$\displaystyle\tilde{X}^{l}=X^{l-1}+\text{CPE}({X^{l-1}}),$ (6)
$\displaystyle\hat{X}^{l}=\tilde{X}^{l}+\text{PS-
Attention}\Big{(}\text{LN}(\tilde{X}^{l})\Big{)},$ (7) $\displaystyle
X^{l}=\hat{X}^{l}+\text{MLP\Big{(}LN}(\hat{X}^{l})\Big{)},$ (8)

where LN($\cdot$) refers to layer normalization (Ba, Kiros, and Hinton 2016).
The CPE (Chu et al. 2021b) is implemented as a simple depth-wise convolution,
which is widely used in previous works (Wu et al. 2021b; Chu et al. 2021a) for
its compatibility with an arbitrary size of input. The PS-Attention module
defined in Eq. (7) is constructed by sequentially performing Eq.
(LABEL:pale:split) to Eq. (LABEL:pale:concat). The MLP module defined in Eq.
(8) consists of two linear projection layers to expand and contract the
embedding dimension sequentially, which is the same as (Dosovitskiy et al.
2021) for fair comparisons.

### Overall Architecture and Variants

As illustrated in Figure 2(a), the Pale Transformer consists of four
hierarchical stages for capturing multi-scale features by following the
popular design in CNNs (He et al. 2016) and Transformers (Liu et al. 2021;
Dong et al. 2021). Each stage contains a patch merging layer and multiple Pale
Transformer blocks. The patch merging layer aims to spatially downsample the
input features by a certain ratio and expand the channel dimension by twice
for a better representation capacity. For fair comparisons, we use the
overlapping convolution for patch merging, the same as (Wu et al. 2021a; Dong
et al. 2021). Specifically, the spatial downsampling ratio is set to 4 for the
first stage and 2 for the last three stages, implementing by $7\times 7$
convolution with stride 4 and $3\times 3$ convolution with stride 2,
respectively. The outputs of the patch merging layer are fed into the
subsequent Pale Transformer blocks, with the number of tokens kept constant.
Following (Liu et al. 2021; Dong et al. 2021), we simply apply an average
pooling operation on the top of the last block to obtain a representative
token for the final classification head, which is composed of a single linear
projection layer.

#### Variants.

The definitions of model hyper-parameters for the $i$-th stage are listed
below:

  * •

$P_{i}$: the spatial reduction factor for patch merging layer,

  * •

$C_{i}$: the embedding dimension of tokens,

  * •

$S_{i}$: the pale size for the PS-Attention,

  * •

$H_{i}$: the head number for the PS-Attention,

  * •

$R_{i}$: the expansion ratio for the MLP module.

By varying the hyper-parameters $H_{i}$ and $C_{i}$ in each stage, we design
three variants of our Pale Transformer, named Pale-T (Tiny), Pale-S (Small),
and Pale-B (Base), respectively. Table 1 shows the detailed configurations of
all variants. Note that all variants have the same depth with $[2,2,16,2]$ in
four stages. In each stage of these variants, we set the pale size
$s_{r}=s_{c}=S_{i}=7$, and use the same MLP expansion ratio of $R_{i}=4$.
Thus, the main differences among Pale-T, Pale-S, and Pale-B lie in the
embedding dimension of tokens and the head number for the PS-Attention in four
stages, i.e., variants vary from narrow to wide.

Stage |  Output Stride | Layer | Pale-T | Pale-S | Pale-B
---|---|---|---|---|---
1 | 4 |  Patch Merging |  $P_{1}=4$ $C_{1}=64$ |  $P_{1}=4$ $C_{1}=96$ |  $P_{1}=4$ $C_{1}=128$
Pale Transformer Block | $\begin{bmatrix}S_{1}=7\\\ H_{1}=2\\\ R_{1}=4\end{bmatrix}\times 2$ | $\begin{bmatrix}S_{1}=7\\\ H_{1}=2\\\ R_{1}=4\end{bmatrix}\times 2$ | $\begin{bmatrix}S_{1}=7\\\ H_{1}=4\\\ R_{1}=4\end{bmatrix}\times 2$
2 | 8 |  Patch Merging |  $P_{2}=2$ $C_{2}=128$ |  $P_{2}=2$ $C_{2}=192$ |  $P_{2}=2$ $C_{2}=256$
Pale Transformer Block | $\begin{bmatrix}S_{2}=7\\\ H_{2}=4\\\ R_{2}=4\end{bmatrix}\times 2$ | $\begin{bmatrix}S_{2}=7\\\ H_{2}=4\\\ R_{2}=4\end{bmatrix}\times 2$ | $\begin{bmatrix}S_{2}=7\\\ H_{2}=8\\\ R_{2}=4\end{bmatrix}\times 2$
3 | 16 |  Patch Merging |  $P_{3}=2$ $C_{3}=256$ |  $P_{3}=2$ $C_{3}=384$ |  $P_{3}=2$ $C_{3}=512$
Pale Transformer Block | $\begin{bmatrix}S_{3}=7\\\ H_{3}=8\\\ R_{3}=4\end{bmatrix}\times 16$ | $\begin{bmatrix}S_{3}=7\\\ H_{3}=8\\\ R_{3}=4\end{bmatrix}\times 16$ | $\begin{bmatrix}S_{3}=7\\\ H_{3}=16\\\ R_{3}=4\end{bmatrix}\times 16$
4 | 32 |  Patch Merging |  $P_{4}=2$ $C_{4}=512$ |  $P_{4}=2$ $C_{4}=768$ |  $P_{4}=2$ $C_{4}=1024$
Pale Transformer Block | $\begin{bmatrix}S_{4}=7\\\ H_{4}=16\\\ R_{4}=4\end{bmatrix}\times 2$ | $\begin{bmatrix}S_{4}=7\\\ H_{4}=16\\\ R_{4}=4\end{bmatrix}\times 2$ | $\begin{bmatrix}S_{4}=7\\\ H_{4}=32\\\ R_{4}=4\end{bmatrix}\times 2$

Table 1:  Detailed configurations of Pale Transformer Variants.

Backbone | Params | FLOPs |  Top-1 (%)
---|---|---|---
RegNetY-4G (Radosavovic et al. 2020) | 21M | 4.0G | 80.0
DeiT-S (Touvron et al. 2021) | 22M | 4.6G | 79.8
PVT-S (Wang et al. 2021b) | 25M | 3.8G | 79.8
T2T-14 (Yuan et al. 2021a) | 22M | 6.1G | 80.7
DPT-S (Chen et al. 2021c) | 26M | 4.0G | 81.0
TNT-S (Han et al. 2021) | 24M | 5.2G | 81.3
Swin-T (Liu et al. 2021) | 29M | 4.5G | 81.3
Twins-SVT-S (Chu et al. 2021a) | 24M | 2.8G | 81.3
CvT-13 (Wu et al. 2021a) | 20M | 4.5G | 81.6
ViL-S (Zhang et al. 2021) | 25M | 4.9G | 82.0
PVTv2-B2 (Wang et al. 2021a) | 25M | 4.0G | 82.0
Focal-T (Yang et al. 2021) | 29M | 4.9G | 82.2
Shuffle-T (Huang et al. 2021) | 29M | 4.6G | 82.5
CSWin-T (Dong et al. 2021) | 23M | 4.3G | 82.7
LV-ViT-S⋆ (Jiang et al. 2021) | 26M | 6.6G | 83.3
Pale-T (ours) | 22M | 4.2G | 83.4
Pale-T⋆ (ours) | 22M | 4.2G | 84.2
RegNetY-8G (Radosavovic et al. 2020) | 39M | 8.0G | 81.7
PVT-M (Wang et al. 2021b) | 44M | 6.7G | 81.2
T2T-19 (Yuan et al. 2021a) | 39M | 9.8G | 81.4
DPT-M (Chen et al. 2021c) | 46M | 6.9G | 81.9
CvT-21 (Wu et al. 2021a) | 32M | 7.1G | 82.5
Swin-S (Liu et al. 2021) | 50M | 8.7G | 83.0
MViT-B-24 (Fan et al. 2021) | 54M | 10.9G | 83.1
Twins-SVT-B (Chu et al. 2021a) | 56M | 8.3G | 83.1
PVTv2-B3 (Wang et al. 2021a) | 45M | 6.9G | 83.2
ViL-M (Zhang et al. 2021) | 40M | 8.7G | 83.3
Focal-S (Yang et al. 2021) | 51M | 9.1G | 83.5
Shuffle-S (Huang et al. 2021) | 50M | 8.9G | 83.5
CSWin-S (Dong et al. 2021) | 35M | 6.9G | 83.6
Refined-ViT-S (Zhou et al. 2021) | 25M | 7.2G | 83.6
VOLO-D1⋆ (Yuan et al. 2021b) | 27M | 6.8G | 84.2
Pale-S (ours) | 48M | 9.0G | 84.3
Pale-S⋆ (ours) | 48M | 9.0G | 85.0
RegNetY-16G (Radosavovic et al. 2020) | 84M | 16.0G | 82.9
ViT-B/16‡ | 86M | 55.4G | 77.9
PVT-L (Wang et al. 2021b) | 61M | 9.8G | 81.7
DeiT-B (Touvron et al. 2021) | 86M | 17.5G | 81.8
T2T-24 (Yuan et al. 2021a) | 64M | 15.0G | 82.2
TNT-B (Han et al. 2021) | 66M | 14.1G | 82.8
ViL-B (Zhang et al. 2021) | 56M | 13.4G | 83.2
Swin-B (Liu et al. 2021) | 88M | 15.4G | 83.3
Twins-SVT-L (Chu et al. 2021a) | 99M | 14.8G | 83.3
PVTv2-B5 (Wang et al. 2021a) | 82M | 11.8G | 83.8
Focal-B (Yang et al. 2021) | 90M | 16.0G | 83.8
Shuffle-B (Huang et al. 2021) | 88M | 15.6G | 84.0
LV-ViT-M⋆ (Jiang et al. 2021) | 56M | 16.0G | 84.1
CSWin-B (Dong et al. 2021) | 78M | 15.0G | 84.2
Refined-ViT-M (Zhou et al. 2021) | 55M | 13.5G | 84.6
VOLO-D2⋆ (Yuan et al. 2021b) | 59M | 14.1G | 85.2
Pale-B (ours) | 85M | 15.6G | 84.9
Pale-B⋆ (ours) | 85M | 15.6G | 85.8

Table 2: Comparisons of different backbones on ImageNet-1K validation set. All
the approaches are trained and evaluated with the size of $224\times 224$,
except for the ViT-B‡ with size $384\times 384$. The superscript “${\star}$”
indicates employing MixToken and token labeling loss (Jiang et al. 2021)
during training.

Backbone | Params | FLOPs | Mask R-CNN (1x)
---|---|---|---
AP${}^{\text{box}}$ | AP${}_{50}^{\text{box}}$ | AP${}_{75}^{\text{box}}$ | AP${}^{\text{mask}}$ | AP${}_{50}^{\text{mask}}$ | AP${}_{75}^{\text{mask}}$
ResNet-50 (He et al. 2016) | 44M | 260G | 38.0 | 58.6 | 41.4 | 34.4 | 55.1 | 36.7
PVT-S (Wang et al. 2021b) | 44M | 245G | 40.4 | 62.9 | 43.8 | 37.8 | 60.1 | 40.3
ViL-S (Zhang et al. 2021) | 45M | 174G | 41.8 | 64.1 | 45.1 | 38.5 | 61.1 | 41.4
Twins-S (Chu et al. 2021a) | 44M | 228G | 42.7 | 65.6 | 46.7 | 39.6 | 62.5 | 42.6
DPT-S (Chen et al. 2021c) | 46M | - | 43.1 | 65.7 | 47.2 | 39.9 | 62.9 | 43.0
Swin-T (Liu et al. 2021) | 48M | 264G | 43.7 | 66.6 | 47.6 | 39.8 | 63.3 | 42.7
RegionViT-S+ (Chen, Panda, and Fan 2021) | 51M | 183G | 44.2 | 67.3 | 48.2 | 40.8 | 64.1 | 44.0
Focal-T (Yang et al. 2021) | 49M | 291G | 44.8 | 67.7 | 49.2 | 41.0 | 64.7 | 44.2
PVTv2-B2 (Wang et al. 2021a) | 45M | - | 45.3 | 67.1 | 49.6 | 41.2 | 64.2 | 44.4
CSWin-T (Dong et al. 2021) | 42M | 279G | 46.7 | 68.6 | 51.3 | 42.2 | 65.6 | 45.4
Pale-T (ours) | 41M | 306G | 47.4 | 69.2 | 52.3 | 42.7 | 66.3 | 46.2
ResNeXt-101-32 (He et al. 2016) | 63M | 340G | 41.9 | 62.5 | 45.9 | 37.5 | 59.4 | 40.2
PVT-M (Wang et al. 2021b) | 64M | 302G | 42.0 | 64.4 | 45.6 | 39.0 | 61.6 | 42.1
ViL-M (Zhang et al. 2021) | 60M | 261G | 43.4 | 65.9 | 47.0 | 39.7 | 62.8 | 42.1
DPT-M (Chen et al. 2021c) | 66M | - | 43.8 | 66.2 | 48.3 | 40.3 | 63.1 | 43.4
Twins-B (Chu et al. 2021a) | 76M | 340G | 45.1 | 67.0 | 49.4 | 41.1 | 64.1 | 44.4
RegionViT-B+ (Chen, Panda, and Fan 2021) | 93M | 307G | 45.4 | 68.4 | 49.6 | 41.6 | 65.2 | 44.8
PVTv2-B3 (Wang et al. 2021a) | 65M | - | 47.0 | 68.1 | 51.7 | 42.5 | 65.7 | 45.7
Focal-S (Yang et al. 2021) | 71M | 401G | 47.4 | 69.8 | 51.9 | 42.8 | 66.6 | 46.1
CSWin-S (Dong et al. 2021) | 54M | 342G | 47.9 | 70.1 | 52.6 | 43.2 | 67.1 | 46.2
Pale-S (ours) | 68M | 432G | 48.4 | 70.4 | 53.2 | 43.7 | 67.7 | 47.1
ResNeXt-101-64 (He et al. 2016) | 101M | 493G | 42.8 | 63.8 | 47.3 | 38.4 | 60.6 | 41.3
PVT-L (Wang et al. 2021b) | 81M | 364G | 42.9 | 65.0 | 46.6 | 39.5 | 61.9 | 42.5
ViL-B (Zhang et al. 2021) | 76M | 365G | 45.1 | 67.2 | 49.3 | 41.0 | 64.3 | 44.2
Twins-L (Chu et al. 2021a) | 120M | 474G | 45.2 | 67.5 | 49.4 | 41.2 | 64.5 | 44.5
PVTv2-B4 (Wang et al. 2021a) | 82M | - | 47.5 | 68.7 | 52.0 | 42.7 | 66.1 | 46.1
Focal-B (Yang et al. 2021) | 110M | 533G | 47.8 | 70.2 | 52.5 | 43.2 | 67.3 | 46.5
CSWin-B (Dong et al. 2021) | 97M | 526G | 48.7 | 70.4 | 53.9 | 43.9 | 67.8 | 47.3
Pale-B (ours) | 105M | 595G | 49.3 | 71.2 | 54.1 | 44.2 | 68.1 | 47.8

Table 3:  Comparisons on COCO val2017 with Mask R-CNN framework and 1x
training schedule for object detection and instance segmentation.

## Experiments

We first compare our Pale Transformer with the state-of-the-art Transformer
backbones on ImageNet-1K (Russakovsky et al. 2015) for image classification.
To further demonstrate the effectiveness and generalization of our backbone,
we conduct experiments on ADE20K (Zhou et al. 2019) for semantic segmentation
(Wu et al. 2021b, 2020; Zhang et al. 2019; Wu et al. 2021c), and COCO (Lin et
al. 2014) for object detection & instance segmentation. Finally, we dig into
the design of key components of our Pale Transformer to better understand the
method.

### Image Classification on ImageNet-1K

#### Settings.

All the variants are trained from scratch for 300 epochs on 8 V100 GPUs with a
total batch size of 1024. Both the training and evaluation are conducted with
the input size of $224\times 224$ on ImageNet-1K dataset. Detailed
configurations are provided in the supplementary material.

#### Results.

Table 2 compares the performance of our Pale Transformer with the state-of-
the-art CNNs and Vision Transformer backbones on ImageNet-1K validation set.
Compared to the advanced CNNs, our Pale variants are +3.4%, +2.6%, and +2.0%
better than the well-known RegNet (Radosavovic et al. 2020) models,
respectively, under the similar computation complexity. Meanwhile, our Pale
Transformer outperforms the state-of-the-art Transformer-based backbones, and
is +0.7% higher than the most related CSWin Transformer for all variants under
the similar model size and FLOPs. Note that LV-ViT (Jiang et al. 2021) and
VOLO (Yuan et al. 2021b), using additional MixToken augmentation and token
labeling loss (Jiang et al. 2021) for training, seem to be on par with our
approach. For fair comparisons, we use these two tricks on our Pale models,
labeled by ⋆ as the superscript. Pale-T⋆ achieves +0.9% gain than LV-ViT-S⋆
with fewer computation costs. Pale-S⋆ and Pale-B⋆ achieve 85.0% and 85.8%,
outperforming VOLO by +0.8% and +0.6%, respectively.

### Semantic Segmentation on ADE20K

#### Settings.

To demonstrate the superiority of our Pale Transformer for dense prediction
tasks, we conduct experiments on ADE20K with the widely-used UperNet (Xiao et
al. 2018) as decoder for fair comparisons to other backbones. Detailed
settings are described in the supplementary material.

#### Results.

Table 4 shows the comparisons of UperNet with various excellent Transformer
backbones on ADE20K validation set. We report both the single-scale (SS) and
multi-scale (MS) mIoU for better comparison. Our Pale variants are
consistently superior to the state-of-the-art method by a large margin.
Specifically, our Pale-T and Pale-S outperform the state-of-the-art CSWin by
+1.1% and +1.2% SS mIoU, respectively. Besides, our Pale-B achieves
52.5%/53.0% SS/MS mIoU, surpassing the previous best by +1.3% and +1.2%,
respectively. These results demonstrate the stronger context modeling capacity
of our Pale Transformer for dense prediction tasks.

Backbone |  Params |  FLOPs |  SS mIoU |  MS mIoU
---|---|---|---|---
DeiT-S (Touvron et al. 2021) | 52M | 1099G | - | 44.0
Swin-T (Liu et al. 2021) | 60M | 945G | 44.5 | 45.8
Focal-T (Yang et al. 2021) | 62M | 998G | 45.8 | 47.0
Shuffle-T (Huang et al. 2021) | 60M | 949G | 46.6 | 47.6
CrossFormer-S (Wang et al. 2021c) | 62M | 980G | 47.6 | 48.4
LV-ViT-S (Jiang et al. 2021) | 44M | - | 47.9 | 48.6
CSWin-T (Dong et al. 2021) | 60M | 959G | 49.3 | 50.4
Pale-T (ours) | 52M | 996G | 50.4 | 51.2
Swin-S (Liu et al. 2021) | 81M | 1038G | 47.6 | 49.5
Focal-S (Yang et al. 2021) | 85M | 1130G | 48.0 | 50.0
Shuffle-S (Huang et al. 2021) | 81M | 1044G | 48.4 | 49.6
VOLO-D1 (Yuan et al. 2021b) | - | - | - | 50.5
LV-ViT-M (Jiang et al. 2021) | 77M | - | 49.4 | 50.6
CrossFormer-B (Wang et al. 2021c) | 84M | 1090G | 49.7 | 50.6
CSWin-S (Dong et al. 2021) | 65M | 1027G | 50.0 | 50.8
Pale-S (ours) | 80M | 1135G | 51.2 | 52.2
Swin-B (Liu et al. 2021) | 121M | 1188G | 48.1 | 49.7
Shuffle-B (Huang et al. 2021) | 121M | 1196G | 49.0 | 50.5
Focal-B (Yang et al. 2021) | 126M | 1354G | 49.0 | 50.5
CrossFormer-L (Wang et al. 2021c) | 126M | 1258M | 50.4 | 51.4
CSWin-B (Dong et al. 2021) | 109M | 1222G | 50.8 | 51.7
LV-ViT-L (Jiang et al. 2021) | 209M | - | 50.9 | 51.8
Pale-B (ours) | 119M | 1311G | 52.2 | 53.0

Table 4: Comparisons of different backbones with UperNet as decoder on ADE20K
for semantic segmentation. All backbones are pretrained on ImageNet-1K with
the size of $224\times 224$. FLOPs are calculated with a resolution of
$512\times 2048$.

### Object Detection and Instance Segmentation on COCO

#### Settings.

We evaluate the performance of our Pale Transformer backbone on COCO benchmark
for object detection and instance segmentation, utilizing Mask R-CNN (He et
al. 2017) framework under 1x schedule (12 training epochs). Details can be
found in the supplementary material.

#### Results.

As shown in Table 3, for object detection, our Pale-T, Pale-S, and Pale-B
achieve 47.4, 48.4, and 49.2 box mAP for object detection, surpassing the
previous best CSWin Transformer by +0.7, +0.5, and +0.6, respectively.
Besides, our variants also have consistent improvement on instance
segmentation, which are +0.5, +0.5, and +0.3 mask mAP higher than the previous
best backbone.

### Ablation Study

We conduct ablation studies for the key designs of our Pale Transformer on
image classification and downstream tasks. All the experiments are performed
with the Tiny variant under the same training settings as mentioned above. We
also analyze the influence of position encoding in the supplementary material.

#### Effect of Pale Size.

The pale sizes of four stages $\\{S_{1},S_{2},S_{3},S_{4}\\}$ control the
trade-off between the richness of contextual information and computation
costs. As shown in Table 7, increasing the pale size (from 1 to 7) can
continuously improve performance across all tasks, while further up to 9 does
not bring obvious and consistent improvements but more FLOPs. Therefore, we
use $S_{i}=7,i\in\\{1,2,3,4\\}$ for all the tasks by default.

Pale size in four stages | ImageNet-1K | ADE20K | COCO
---|---|---|---
Top-1 (%) | SS mIoU (%) | AP${}^{\text{box}}$ | AP${}^{\text{mask}}$
1 1 1 1 | 82.4 | 47.9 | 46.1 | 41.5
3 3 3 3 | 82.9 | 49.4 | 46.7 | 42.3
5 5 5 5 | 83.1 | 49.7 | 46.8 | 42.4
7 7 7 7 | 83.4 | 50.4 | 47.4 | 42.7
9 9 9 9 | 83.3 | 50.6 | 47.4 | 42.6

Table 5:  Ablation study for different choices of pale size. The complete
table with parameters and FLOPs can be found in the supplementary material.

Attention mode | ImageNet-1K | ADE20K | COCO
---|---|---|---
Top-1 (%) | SS mIoU (%) | AP${}^{\text{box}}$ | AP${}^{\text{mask}}$
Axial | 82.4 | 47.9 | 46.1 | 41.5
Cross-Shaped | 82.8 | 49.0 | 46.6 | 42.2
Pale (vanilla) | 83.4 | 50.3 | 47.1 | 42.3
Pale (sequential) | 82.9 | 49.5 | 46.9 | 42.2
Pale (parallel) | 83.4 | 50.4 | 47.4 | 42.7

Table 6:  Ablation study for different attention modes. Params and FLOPs of
all the experiments are provided in the supplementary material.

#### Comparisons with Different Implementations of PS-Attention.

We compare three implementations of our PS-Attention. The vanilla PS-Attention
directly conducts self-attention within the whole pale region, which can be
approximated as two more efficient implementations, sequential and parallel.
The sequential one computes self-attention in row and column directions
alternately in consecutive blocks, while the parallel one performs row-wise
and column-wise attention in parallel within each block. As shown in Table 8,
the parallel PS-Attention achieves the best results on all tasks, even
slightly better than the vanilla one by +0.3/0.4 box/mask mAP on COCO. We
attribute this to that the excessive padding for the non-square input size in
vanilla PS-Attention will result in slight performance degradation.

#### Comparisons with other Axial-based Attentions.

In order to compare our PS-Attention with the most related axial-based self-
attention mechanisms directly, we replace the PS-Attention of our Pale-T with
the axial self-attention (Wang et al. 2020) and cross-shaped window self-
attention (Dong et al. 2021), respectively. As shown in Table 8, our PS-
Attention outperforms these two mechanisms obviously.

## Conclusion

This work presented a new effective and efficient self-attention mechanism,
named Pale-Shaped self-Attention (PS-Attention), which performs self-attention
in a pale-shaped region. PS-Attention can model richer contextual dependencies
than the previous local self-attention mechanisms. In order to further improve
its efficiency, we designed a parallel implementation for PS-Attention, which
decomposes the self-attention within the whole pale into row-wise and column-
wise attention. It is also conducive to avoiding excessive padding operations.
Based on the proposed PS-Attention, we developed a general Vision Transformer
backbone, called Pale Transformer, which can achieve state-of-the-art
performance on ImageNet-1K for image classification. Furthermore, our Pale
Transformer is superior to the previous Vision Transformer backbones on ADE20K
for semantic segmentation, and COCO for object detection & instance
segmentation.

## References

  * Ba, Kiros, and Hinton (2016) Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Chen et al. (2021a) Chen, B.; Li, P.; Li, B.; Li, C.; Bai, L.; Lin, C.; Sun, M.; Yan, J.; and Ouyang, W. 2021a.  PSViT: Better Vision Transformer via Token Pooling and Attention Sharing.  _arXiv preprint arXiv:2108.03428_.
  * Chen, Fan, and Panda (2021) Chen, C.-F.; Fan, Q.; and Panda, R. 2021.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  _arXiv preprint arXiv:2103.14899_.
  * Chen, Panda, and Fan (2021) Chen, C.-F.; Panda, R.; and Fan, Q. 2021.  RegionViT: Regional-to-Local Attention for Vision Transformers.  _arXiv preprint arXiv:2106.02689_.
  * Chen et al. (2021b) Chen, Y.; Dai, X.; Chen, D.; Liu, M.; Dong, X.; Yuan, L.; and Liu, Z. 2021b.  Mobile-Former: Bridging MobileNet and Transformer.  _arXiv preprint arXiv:2108.05895_.
  * Chen et al. (2021c) Chen, Z.; Zhu, Y.; Zhao, C.; Hu, G.; Zeng, W.; Wang, J.; and Tang, M. 2021c.  _DPT: Deformable Patch-Based Transformer for Visual Recognition_ , 2899–2907.  New York, NY, USA: Association for Computing Machinery.  ISBN 9781450386517.
  * Chu et al. (2021a) Chu, X.; Tian, Z.; Wang, Y.; Zhang, B.; Ren, H.; Wei, X.; Xia, H.; and Shen, C. 2021a.  Twins: Revisiting the design of spatial attention in vision transformers.  _arXiv preprint arXiv:2104.13840_ , 1(2): 3.
  * Chu et al. (2021b) Chu, X.; Tian, Z.; Zhang, B.; Wang, X.; Wei, X.; Xia, H.; and Shen, C. 2021b.  Conditional Positional Encodings for Vision Transformers.  _arXiv preprint arXiv:2102.10882_.
  * Cubuk et al. (2019) Cubuk, E. D.; Zoph, B.; Mane, D.; Vasudevan, V.; and Le, Q. V. 2019.  Autoaugment: Learning augmentation strategies from data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 113–123.
  * Dong et al. (2021) Dong, X.; Bao, J.; Chen, D.; Zhang, W.; Yu, N.; Yuan, L.; Chen, D.; and Guo, B. 2021\.  CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows.  _arXiv preprint arXiv:2107.00652_.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_.
  * Fan et al. (2021) Fan, H.; Xiong, B.; Mangalam, K.; Li, Y.; Yan, Z.; Malik, J.; and Feichtenhofer, C. 2021.  Multiscale Vision Transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 6824–6835.
  * Fang et al. (2021) Fang, J.; Xie, L.; Wang, X.; Zhang, X.; Liu, W.; and Tian, Q. 2021.  MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens.  _arXiv preprint arXiv:2105.15168_.
  * Han et al. (2021) Han, K.; Xiao, A.; Wu, E.; Guo, J.; Xu, C.; and Wang, Y. 2021.  Transformer in transformer.  _arXiv preprint arXiv:2103.00112_.
  * He et al. (2017) He, K.; Gkioxari, G.; Dollár, P.; and Girshick, R. 2017.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , 2961–2969.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Howard and Ruder (2018) Howard, J.; and Ruder, S. 2018.  Universal Language Model Fine-tuning for Text Classification.  In _ACL_.
  * Huang et al. (2016) Huang, G.; Sun, Y.; Liu, Z.; Sedra, D.; and Weinberger, K. Q. 2016.  Deep networks with stochastic depth.  In _European conference on computer vision_ , 646–661. Springer.
  * Huang et al. (2021) Huang, Z.; Ben, Y.; Luo, G.; Cheng, P.; Yu, G.; and Fu, B. 2021.  Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer.  _arXiv preprint arXiv:2106.03650_.
  * Jiang et al. (2021) Jiang, Z.; Hou, Q.; Yuan, L.; Zhou, D.; Shi, Y.; Jin, X.; Wang, A.; and Feng, J. 2021.  All Tokens Matter: Token Labeling for Training Better Vision Transformers.  _arXiv preprint arXiv:2104.10858_.
  * Li et al. (2021a) Li, J.; Yan, Y.; Liao, S.; Yang, X.; and Shao, L. 2021a.  Local-to-Global Self-Attention in Vision Transformers.  _arXiv preprint arXiv:2107.04735_.
  * Li et al. (2021b) Li, Y.; Zhang, K.; Cao, J.; Timofte, R.; and Van Gool, L. 2021b.  LocalViT: Bringing Locality to Vision Transformers.  _arXiv preprint arXiv:2104.05707_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , 740–755. Springer.
  * Liu et al. (2021) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021\.  Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 10012–10022.
  * Loshchilov and Hutter (2019) Loshchilov, I.; and Hutter, F. 2019.  Decoupled Weight Decay Regularization.  In _ICLR_.
  * Mao et al. (2021) Mao, M.; Zhang, R.; Zheng, H.; Gao, P.; Ma, T.; Peng, Y.; Ding, E.; and Han, S. 2021\.  Dual-stream Network for Visual Recognition.  _arXiv preprint arXiv:2105.14734_.
  * McCann et al. (2017) McCann, B.; Bradbury, J.; Xiong, C.; and Socher, R. 2017.  Learned in Translation: Contextualized Word Vectors.  In _NIPS_.
  * Pan et al. (2021) Pan, B.; Jiang, Y.; Panda, R.; Wang, Z.; Feris, R.; and Oliva, A. 2021.  IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers.  _arXiv preprint arXiv:2106.12620_.
  * Peng et al. (2021) Peng, Z.; Huang, W.; Gu, S.; Xie, L.; Wang, Y.; Jiao, J.; and Ye, Q. 2021.  Conformer: Local Features Coupling Global Representations for Visual Recognition.  _arXiv preprint arXiv:2105.03889_.
  * Radosavovic et al. (2020) Radosavovic, I.; Kosaraju, R. P.; Girshick, R.; He, K.; and Dollár, P. 2020\.  Designing network design spaces.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10428–10436.
  * Rao et al. (2021) Rao, Y.; Zhao, W.; Liu, B.; Lu, J.; Zhou, J.; and Hsieh, C.-J. 2021.  DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.  _arXiv preprint arXiv:2106.02034_.
  * Russakovsky et al. (2015) Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3): 211–252.
  * Szegedy et al. (2015) Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1–9.
  * Szegedy et al. (2016) Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2818–2826.
  * Touvron et al. (2021) Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and Jégou, H. 2021.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , 10347–10357. PMLR.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Wang et al. (2020) Wang, H.; Zhu, Y.; Green, B.; Adam, H.; Yuille, A.; and Chen, L.-C. 2020.  Axial-deeplab: Stand-alone axial-attention for panoptic segmentation.  In _European Conference on Computer Vision_ , 108–126. Springer.
  * Wang et al. (2021a) Wang, W.; Xie, E.; Li, X.; Fan, D.-P.; Song, K.; Liang, D.; Lu, T.; Luo, P.; and Shao, L. 2021a.  Pvtv2: Improved baselines with pyramid vision transformer.  _arXiv preprint arXiv:2106.13797_.
  * Wang et al. (2021b) Wang, W.; Xie, E.; Li, X.; Fan, D.-P.; Song, K.; Liang, D.; Lu, T.; Luo, P.; and Shao, L. 2021b.  Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 568–578.
  * Wang et al. (2021c) Wang, W.; Yao, L.; Chen, L.; Cai, D.; He, X.; and Liu, W. 2021c.  CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention.  _arXiv preprint arXiv:2108.00154_.
  * Wang et al. (2021d) Wang, Y.; Huang, R.; Song, S.; Huang, Z.; and Huang, G. 2021d.  Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length.  _arXiv preprint arXiv:2105.15075_.
  * Wu et al. (2021a) Wu, H.; Xiao, B.; Codella, N.; Liu, M.; Dai, X.; Yuan, L.; and Zhang, L. 2021a.  CvT: Introducing Convolutions to Vision Transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 22–31.
  * Wu et al. (2021b) Wu, S.; Wu, T.; Lin, F.; Tian, S.; and Guo, G. 2021b.  Fully Transformer Networks for Semantic Image Segmentation.  _arXiv preprint arXiv:2106.04108_.
  * Wu et al. (2020) Wu, T.; Tang, S.; Zhang, R.; Cao, J.; and Zhang, Y. 2020.  Cgnet: A light-weight context guided network for semantic segmentation.  _IEEE Transactions on Image Processing_ , 30: 1169–1179.
  * Wu et al. (2021c) Wu, T.; Tang, S.; Zhang, R.; and Guo, G. 2021c.  Consensus feature network for scene parsing.  _IEEE Transactions on Multimedia_.
  * Xiao et al. (2018) Xiao, T.; Liu, Y.; Zhou, B.; Jiang, Y.; and Sun, J. 2018.  Unified perceptual parsing for scene understanding.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 418–434.
  * Xu et al. (2021) Xu, Y.; Zhang, Z.; Zhang, M.; Sheng, K.; Li, K.; Dong, W.; Zhang, L.; Xu, C.; and Sun, X. 2021.  Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer.  _arXiv preprint arXiv:2108.01390_.
  * Yang et al. (2021) Yang, J.; Li, C.; Zhang, P.; Dai, X.; Xiao, B.; Yuan, L.; and Gao, J. 2021.  Focal Self-attention for Local-Global Interactions in Vision Transformers.  _arXiv preprint arXiv:2107.00641_.
  * Yuan et al. (2021a) Yuan, L.; Chen, Y.; Wang, T.; Yu, W.; Shi, Y.; Jiang, Z.-H.; Tay, F. E.; Feng, J.; and Yan, S. 2021a.  Tokens-to-Token ViT: Training Vision Transformers From Scratch on ImageNet.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 558–567.
  * Yuan et al. (2021b) Yuan, L.; Hou, Q.; Jiang, Z.; Feng, J.; and Yan, S. 2021b.  Volo: Vision outlooker for visual recognition.  _arXiv preprint arXiv:2106.13112_.
  * Yue et al. (2021) Yue, X.; Sun, S.; Kuang, Z.; Wei, M.; Torr, P. H.; Zhang, W.; and Lin, D. 2021.  Vision Transformer With Progressive Sampling.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 387–396.
  * Yun et al. (2019) Yun, S.; Han, D.; Oh, S. J.; Chun, S.; Choe, J.; and Yoo, Y. 2019.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6023–6032.
  * Zhang et al. (2018) Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2018.  mixup: Beyond Empirical Risk Minimization.  In _International Conference on Learning Representations_.
  * Zhang et al. (2021) Zhang, P.; Dai, X.; Yang, J.; Xiao, B.; Yuan, L.; Zhang, L.; and Gao, J. 2021.  Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2998–3008.
  * Zhang et al. (2019) Zhang, R.; Tang, S.; Zhang, Y.; Li, J.; and Yan, S. 2019.  Perspective-adaptive convolutions for scene parsing.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(4): 909–924.
  * Zhou et al. (2019) Zhou, B.; Zhao, H.; Puig, X.; Xiao, T.; Fidler, S.; Barriuso, A.; and Torralba, A. 2019.  Semantic understanding of scenes through the ade20k dataset.  _International Journal of Computer Vision_ , 127(3): 302–321.
  * Zhou et al. (2021) Zhou, D.; Shi, Y.; Kang, B.; Yu, W.; Jiang, Z.; Li, Y.; Jin, X.; Hou, Q.; and Feng, J. 2021.  Refiner: Refining Self-attention for Vision Transformers.  _arXiv preprint arXiv:2106.03714_.
  * Zhu et al. (2021a) Zhu, M.; Han, K.; Tang, Y.; and Wang, Y. 2021a.  Visual Transformer Pruning.  _arXiv preprint arXiv:2104.08500_.
  * Zhu et al. (2021b) Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2021b.  Deformable DETR: Deformable Transformers for End-to-End Object Detection.  In _International Conference on Learning Representations_.

##  Appendix A Appendix

In this appendix, we first provide the detailed experimental settings for
classification, semantic segmentation, object detection, and instance
segmentation, respectively. Then, we study the effect of the position encoding
method of our Pale Transformer, and provide more detailed comparisons of the
ablation studies in the body of our paper in terms of the model size and
computation costs. Finally, the detailed derivations of computation complexity
for the global self-attention and our PS-Attention are given.

##  Appendix B Detailed Experimental Settings

### Image Classification on ImageNet-1K

We follow most of the settings in DeiT (Touvron et al. 2021), Swin (Liu et al.
2021) and CSWin (Dong et al. 2021) for fair comparisons. In detail, we use
AdamW (Loshchilov and Hutter 2019) optimizer with a weight decay of 0.05. The
initial learning rate is set to 1e-3 and progressively decays after each
iteration by a cosine schedule. The linear warmup takes up 20 epochs. We use
the random horizontal flipping (Szegedy et al. 2015), color jitter, Mixup
(Zhang et al. 2018), CutMix (Yun et al. 2019) and AutoAugment (Cubuk et al.
2019) as data augmentation. We also adopt some common regularizations, such as
Label-Smoothing (Szegedy et al. 2016) and stochastic depth (Huang et al.
2016). The maximal stochastic depth rate is set to 0.1, 0.3, and 0.5 for
Pale-T, Pale-S, and Pale-B, respectively. All the variants are trained from
scratch for 300 epochs on 8 V100 GPUs with the input size of $224\times 224$
and a total batch size of 1024. During the evaluation, the images are first
resized to $256\times 256$ and then center-cropped to $224\times 224$.

### Semantic Segmentation on ADE20K

We conduct experiments on the widely-used and challenging ADE20K (Zhou et al.
2019) scene parsing dataset, which contains 20210, 2000, and 3352 images for
training, validation, and testing, respectively, with 150 fine-grained object
categories. For fair comparisons, we use our ImageNet-1k pretrained Pale
Transformer as backbone and UperNet as the decoder, and follow the same
training settings as (Liu et al. 2021). Specifically, all the models are
trained for total 160k iterations with a batch size of 16. The AdamW
(Loshchilov and Hutter 2019) optimizer with weight decay 0.01 is used. The
initial learning rate is set to 6e-5 and decay with a polynomial scheduler
after the 1500-iterations warmup. The stochastic depth rate is set to 0.3,
0.3, and 0.5 for Pale-T, Pale-S, and Pale-B, respectively. Both single-scale
and multi-scale inference are reported for performance comparison. For multi-
scale inference, factors vary from 0.75 to 1.75 with 0.25 as the interval.
Auxiliary losses are added to the output of stage 3 of the backbone with
factor 0.4, which is the same as the previous works (Liu et al. 2021; Dong et
al. 2021) for fair comparisons. For data augmentation during training, we
follow the default configurations of mmsegmentation, such as random crop,
random flipping, random rescaling (with ratio range from 0.5 to 2.0), and
random photometric distortion.

### Object Detection & Instance Segmentation on COCO

We compare the performance of our Pale Transformer backbone on COCO benchmark
for object detection and instance segmentation, with the typical Mask R-CNN
(He et al. 2017) framework. We follow the same training strategies as (Dong et
al. 2021). In detail, we train and evaluate our Pale Transformer under the
normal 1x schedule, and all the models are trained for 12 epochs on 8 GPUs
with the total batch size of 16 and single-scale input (shorter size is
resized to 800 and longer size is no more than 1333). We use AdamW (Loshchilov
and Hutter 2019) as the optimizer with a weight decay of 0.001 for Pale-T and
Pale-S and 0.05 for Pale-B. For all models, the learning rate is set to 0.0001
initially and decay at epoch 8 and 11 with a ratio of 0.1. We set the
stochastic depth rate to 0.2, 0.3, and 0.5 for Pale-T, Pale-S, and Pale-B,
respectively. FLOPs are compared under the input size of $1280\times 800$.

Pale size in four stages | ImageNet-1K | ADE20K | COCO
---|---|---|---
Params | FLOPs | Top-1 (%) | Params | FLOPs | SS mIoU (%) | Params | FLOPs | AP${}^{\text{box}}$ | AP${}^{\text{mask}}$
1 1 1 1 | 22M | 3.8G | 82.4 | 52M | 929G | 47.9 | 41M | 253G | 46.1 | 41.5
3 3 3 3 | 22M | 4.1G | 82.9 | 52M | 950G | 49.4 | 41M | 269G | 46.7 | 42.3
5 5 5 5 | 22M | 4.4G | 83.1 | 52M | 972G | 49.7 | 41M | 283G | 46.8 | 42.4
7 7 7 7 | 22M | 4.2G | 83.4 | 52M | 996G | 50.4 | 41M | 306G | 47.4 | 42.7
9 9 9 9 | 22M | 5.4G | 83.3 | 52M | 1021G | 50.6 | 41M | 322G | 47.4 | 42.6

Table 7:  Ablation study for different choices of pale size.

Attention mode | ImageNet-1K | ADE20K | COCO
---|---|---|---
Params | FLOPs | Top-1 (%) | Params | FLOPs | SS mIoU (%) | Params | FLOPs | AP${}^{\text{box}}$ | AP${}^{\text{mask}}$
Axial | 22M | 3.8G | 82.4 | 52M | 929G | 47.9 | 41M | 253G | 46.1 | 41.5
Cross-Shaped | 22M | 4.2G | 82.8 | 52M | 996G | 49.0 | 41M | 306G | 46.6 | 42.2
Pale (vanilla) | 22M | 5.4G | 83.4 | 52M | 2677G | 50.2 | 41M | 668G | 47.1 | 42.3
Pale (sequential) | 22M | 4.2G | 82.9 | 52M | 996G | 49.5 | 41M | 306G | 46.9 | 42.2
Pale (parallel) | 22M | 4.2G | 83.4 | 52M | 996G | 50.4 | 41M | 306G | 47.4 | 42.7

Table 8:  Ablation study for different attention modes.

##  Appendix C Further Ablation Study

In this section, we first provide the complete version of Table 5 and Table 6
in the body of our paper, including the parameters and FLOPs comparisons,
shown in Table 7 and Table 8. Then, we analyze the effect of different
position encoding methods for our Pale Transformer backbone.

Position Encoding | ImageNet-1K | ADE20K | COCO
---|---|---|---
Params | FLOPs | Top-1 (%) | Params | FLOPs | SS mIoU (%) | Params | FLOPs | AP${}^{\text{box}}$ | AP${}^{\text{mask}}$
no pos. | 22M | 4.2G | 82.5 | 51M | 996G | 48.3 | 41M | 306G | 46.2 | 41.5
APE | 22M | 4.2G | 82.9 | 59M | 996G | 49.6 | 49M | 306G | 46.8 | 42.2
CPE | 22M | 4.2G | 83.4 | 52M | 996G | 50.4 | 41M | 306G | 47.4 | 42.7

Table 9:  Ablation study for different position encoding methods.

#### Effect of Position Encoding.

The position encoding plays an important role in Transformers, as it can
introduce the spatial location awareness for feature aggregation of self-
attention. Here, we compare several widely-used position encoding methods,
e.g., no position encoding (no pos.), absolute position encoding (APE)
(Dosovitskiy et al. 2021) and conditional position encoding (CPE) (Chu et al.
2021b). As shown in Table 9, CPE performs best. Not using any position
encoding will cause serious performance degradation, which demonstrates the
effectiveness of the position encoding in Vision Transformer models.

##  Appendix D Derivations of the Computational Complexity

In this section, we derive the computation complexity of the global self-
attention and our PS-Attention in detail.

### Computational Complexity of Global Self-Attention

Supposing that the size of input feature map is denoted as $h\times w\times
c$. The global self-attention (Dosovitskiy et al. 2021) has three parts.
Firstly, the input feature $X\in\mathcal{R}^{h\times w\times c}$ is first sent
into three independent linear layers to generate query
$Q\in\mathcal{R}^{h\times w\times c}$, key $K\in\mathcal{R}^{h\times w\times
c}$, and value $V\in\mathcal{R}^{h\times w\times c}$, respectively. Thus, the
computational complexity of the generation of $Q$, $K$, and $V$ is

$\displaystyle\mathcal{O}_{\text{Global}}^{\text{qkv}}=3hwc^{2}.$ (9)

Secondly, the attention map $A$ is computed by
$\text{softmax}(QK^{T}/\sqrt{d})$. Then, the aggregated feature is obtained by
the matrix multiplication between the normalized attention map $A$ and the
value $V$. The computational complexity of these two processes is

$\displaystyle\mathcal{O}_{\text{Global}}^{\text{attn}}=2c(hw)^{2}.$ (10)

Finally, the aggregated feature also needs to pass through a linear projection
layer generally with the complexity of

$\displaystyle\mathcal{O}_{\text{Global}}^{\text{proj}}=hwc^{2}.$ (11)

Thus, the overall computational complexity of the global self-attention is

$\displaystyle\mathcal{O}_{\text{Global}}=\mathcal{O}_{\text{Global}}^{\text{qkv}}+\mathcal{O}_{\text{Global}}^{\text{attn}}+\mathcal{O}_{\text{Global}}^{\text{proj}}$
(12) $\displaystyle\quad\quad\ \ \ =4hwc^{2}+2c(hw)^{2}.$

### Computational Complexity of Our PS-Attention

Similarly, given the input feature of size $h\times w\times c$ and pale size
$(s_{r},s_{c})$, our PS-Attention(parallel) also contains three processes.
Firstly, the three individual $3\times 3$ separable convolutions are used to
generate the query $Q\in\mathcal{R}^{h\times w\times c}$, key
$K\in\mathcal{R}^{h\times w\times c}$, and value $V\in\mathcal{R}^{h\times
w\times c}$, respectively, with the complexity of

$\displaystyle\mathcal{O}_{\text{Pale}}^{\text{qkv}}=3(9hwc+hwc^{2})=27hwc+3hwc^{2}.$
(13)

Second, we decompose the self-attention within the whole pale region into row-
wise and column-wise self-attention. The computational complexity of these two
parallel branches are as follows

$\displaystyle\mathcal{O}_{\text{Pale}}^{\text{row}}=hw^{2}cs_{r},$ (14)
$\displaystyle\mathcal{O}_{\text{Pale}}^{\text{column}}=h^{2}wcs_{c}.$

Finally, the linear projection layer has the complexity of

$\displaystyle\mathcal{O}_{\text{Pale}}^{\text{proj}}=hwc^{2}.$ (15)

Therefore, the overall complexity of our parallel PS-Attention is

$\displaystyle\mathcal{O}_{\text{Pale}}=\mathcal{O}_{\text{Pale}}^{\text{qkv}}+\mathcal{O}_{\text{Pale}}^{\text{row}}+\mathcal{O}_{\text{Pale}}^{\text{column}}+\mathcal{O}_{\text{Pale}}^{\text{proj}}$
(16) $\displaystyle\quad\quad=4hwc^{2}+hwc(s_{c}h+s_{r}w+27).$

Compared with the global self-attention, our parallel PS-Attention has lower
complexity, since $2hw>>(s_{c}h+s_{r}w+27)$ always holds.


# Further study of Squeezing function Corresponding to Polydisk

Naveen Gupta  Department of Mathematics, University of Delhi, Delhi–110 007,
India<EMAIL_ADDRESS>and  Akhil Kumar  Department of Mathematics,
University of Delhi, Delhi–110 078, India<EMAIL_ADDRESS>
###### Abstract.

In the present article, we investigate some properties of squeezing function
corresponding to polydisk. We give some explicit expressions of squeezing
function corresponding to polydisk.

###### Key words and phrases:

squeezing function; extremal map; holomorphic homogeneous regular domain.

###### 2020 Mathematics Subject Classification:

32F45, 32H02

##  1\. Introduction

In 2012, F. Deng, Q. Guan, L. Zhang in their paper [3], introduced the notion
of squeezing function by following the work of K. Liu, X. Sun, S. T. Yau,
[15], [16] and S. K. Yeung [25].

######  Definition 1.1.

Let $\Omega\subset\mathbb{C}^{n}$ be a bounded domain. For $z\in{\Omega}$ and
a holomorphic embedding $f:\Omega\to\mathbb{B}^{n}$ with $f(z)=0$, define

$S_{\Omega}(f,z):=\sup\\{r:\mathbb{B}^{n}(0,r)\subseteq f(\Omega)\\},$

where $\mathbb{B}^{n}$ denotes unit ball in $\mathbb{C}^{n}$ and
$\mathbb{B}^{n}(0,r)$ denotes the ball centered at origin with radius $r$ in
$\mathbb{C}^{n}$. The squeezing function on $\Omega$, denoted by $S_{\Omega}$
is defined as

$S_{\Omega}(z):=\sup_{f}\\{r:\mathbb{B}^{n}(0,r)\subseteq f(\Omega)\\},$

where supremum is taken over all holomorphic embeddings
$f:\Omega\to\mathbb{B}^{n}$ with $f(z)=0.$

For other important properties of squeezing functions corresponding to unit
ball one can refer the following papers: [12], [11], [20], [26], [18], [19].
Recently [9] N. Gupta, S. K. Pant introduced squeezing function corresponding
to polydisk.

######  Definition 1.2.

[9] Let $\Omega\subset\mathbb{C}^{n}$ be a bounded domain. For $z\in{\Omega}$
and holomorphic embedding $f:\Omega\to\mathbb{D}^{n}$ with $f(z)=0$, define

$T_{\Omega}(f,z):=\sup\\{r:\mathbb{D}^{n}(0,r)\subseteq f(\Omega)\\},$

where $\mathbb{D}^{n}$ denotes unit polydisk and $\mathbb{D}^{n}(0,r)$ denotes
the polydisk centered at origin with radius $r$ in $\mathbb{C}^{n}$. The
squeezing function corresponding to polydisk on $\Omega$, denoted by
$T_{\Omega}$, is defined as

$T_{\Omega}(z):=\sup_{f}\\{r:\mathbb{D}^{n}(0,r)\subseteq f(\Omega)\\},$

where supremum is taken over all holomorphic embeddings
$f:\Omega\to\mathbb{D}^{n}$ with $f(z)=0.$

From definition, it is clear that for any bounded domain $\Omega$,
$T_{\Omega}$ is invariant under biholomorphic transformations, that is,
$T_{f(\Omega)}(f(z))=T_{\Omega}(z)$, where $f$ is any biholomorphic
transformation of $\Omega$. Also by [9, Lemma 1.2], holomorphic homogeneous
regular(HHR) domains are same in both the settings, that is, if squeezing
function corresponding to polydisk $T_{\Omega}$ on $\Omega$ admits positive
lower bound, then $\Omega$ is HHR.

Recently Rong and Yang [22] extended this notion to generalised squeezing
function by taking the embeddings in a bounded, balanced, convex domain. The
object, $T_{D}$(for a bounded domain $D$) is one particular example of
generalised squeezing function. N. Gupta, S. K. Pant in [10] introduced more
general notion $d$-balanced squeezing function by taking the embeddings in
bounded, $d$-balanced domain, convex domains. Now the question comes, what is
the importance of the object $T_{D}$, when there are more general set ups
available. It seems to us that the set up involving polydisk provides a
concrete model space to work with(note that $T_{D}$ is an example of a
generalised squeezing function). The better perspective is gained to deal with
difficulty arising in arbitrary balanced domains.

##  2\. Main theorems

We begin with the following observation: In [9], there is a statement which
says that for bounded homogeneous domain $\Omega\subset\mathbb{C}^{n}$, either
$S_{\Omega}(z)=\dfrac{1}{\sqrt{n}}T_{\Omega}(z)$ or
$T_{\Omega}(z)=\dfrac{1}{\sqrt{n}}S_{\Omega}(z)$. We see that this statement
is not true in general. To see this consider the following remark.

######  Remark 2.1.

Let
$\Omega=\mathbb{B}^{n}\times\mathbb{B}^{n}\times\ldots\times\mathbb{B}^{n}\subset\mathbb{C}^{n^{2}}$
be a bounded domain. Then for $z=(z_{1},z_{2},\ldots,z_{n})\in\Omega$ and
$n>1$(here $z_{i}=(z_{i1},z_{i2},\ldots,z_{in})$), we have

$S_{\Omega}(z)\neq\dfrac{1}{n}T_{\Omega}(z)\ \ \ \mbox{and}\ \ \
T_{\Omega}(z)\neq\dfrac{1}{n}S_{\Omega}(z).$

Since $\mathbb{B}^{n}$ is a classical symmetric domain, so by [3, Theorem 7.5]

$S_{\Omega}(z)=(S_{\mathbb{B}^{n}}(z_{1})^{-2}+S_{\mathbb{B}^{n}}(z_{2})^{-2}+\ldots+S_{\mathbb{B}^{n}}(z_{n})^{-2})^{-1/2}=\dfrac{1}{\sqrt{n}}.$

Also, by [9, Proposition 4.6], $T_{\Omega}(z)\geq\displaystyle\min_{1\leq
i\leq n}T_{\mathbb{B}^{n}}(z_{i})$ and since
$T_{\mathbb{B}^{n}}(z_{i})\equiv\dfrac{1}{\sqrt{n}}$ for each $i$, this
implies $T_{\Omega}(z)\geq\dfrac{1}{\sqrt{n}}$.
If $S_{\Omega}(z)=\dfrac{1}{n}T_{\Omega}(z)$, then $T_{\Omega}(z)=\sqrt{n}$
which is not possible.
If $T_{\Omega}(z)=\dfrac{1}{n}S_{\Omega}(z)$, then
$T_{\Omega}(z)=\dfrac{1}{n^{3/2}}<\dfrac{1}{\sqrt{n}}$ which is not possible.
It should be noted that
$\Omega=\mathbb{B}^{n}\times\mathbb{B}^{n}\times\ldots\times\mathbb{B}^{n}\subset\mathbb{C}^{n^{2}}$
is bounded homogeneous domain.

Before stating our next theorem we would like to fix notation for Poincaré
metric on unit disc. We consider positive real valued function
$\sigma:[0,1)\to\mathbb{R^{+}}$ defined as $\sigma(x)=\log\dfrac{1+x}{1-x}$.
It is clear that $\sigma$ is strictly increasing function and for
$z\in\mathbb{D}$, $\sigma(|z|)$ is the Poincaré distance between 0 and $z$.
One can observe that the inverse of this function can be expressed as
$\tanh\left(\dfrac{x}{2}\right)$.

######  Theorem 2.2.

Let
$\Omega=\Omega_{1}\times\Omega_{2}\times\ldots\times\Omega_{n}\subset\mathbb{C}^{n}$
be a bounded product domain such that for each $1\leq i\leq n$ there is a
singleton component $\\{p_{i}\\}$ of $\overline{\mathbb{C}}-\Omega_{i}$. Then
for $z=(z_{1},z_{2},\ldots,z_{n})\in\Omega$,
$T_{\Omega}(z)\leq\min\\{\sigma^{-1}(K_{\widetilde{\Omega}_{1}}(z_{1},p_{1})),\sigma^{-1}(K_{\widetilde{\Omega}_{2}}(z_{2},p_{2})),\ldots,\sigma^{-1}(K_{\widetilde{\Omega}_{n}}(z_{n},p_{n}))\\},$
where
$\widetilde{\Omega}_{i}=\Omega_{i}\cup\\{p_{i}\\}$ and
$K_{\widetilde{\Omega}_{i}}(\cdot,\cdot)$ is the Kobayashi distance on
$\widetilde{\Omega}_{i}$ for $i=1,2,\ldots,n$. In particular, $\Omega$ is not
holomorphic homogeneous regular domain.

###### Proof.

For $z=(z_{1},z_{2},\ldots,z_{n})\in\Omega$, let $T_{\Omega}(z)=c$. Then there
exist a holomorphic embedding
$f=(f_{1},f_{2},\ldots,f_{n}):\Omega\to\mathbb{D}^{n}$ such that $f(z)=0$ with
$\mathbb{D}^{n}(0,c)\subseteq f(\Omega)$. So there is a holomorphic embedding
$f_{i}:\Omega_{i}\to\mathbb{D}$ such that $f_{i}(z_{i})=0$ with
$\mathbb{D}(0,c)\subseteq f_{i}(\Omega_{i})$ for each $i=1,2,\ldots,n$.

For each $1\leq i\leq n$ , by Riemann’s removable singularity theorem there
exists a holomorphic function
$\widetilde{f}_{i}:\widetilde{\Omega}_{i}\to\mathbb{D}$ with
$\widetilde{f}_{i}(\zeta)=f_{i}(\zeta)$ for every $\zeta\in\Omega_{i}$. Now we
will show that $\widetilde{f}_{i}(p_{i})\notin f_{i}(\Omega_{i})$. For this,
let $\widetilde{f}_{i}(p_{i})\in f_{i}(\Omega_{i})$ and $\\{\zeta_{n}\\}$ be a
sequence in $\Omega_{i}$ converging to $p_{i}$, then by continuity of
$\widetilde{f}_{i}$ at $p_{i}$, $\\{\widetilde{f}_{i}(\zeta_{n})\\}$ converges
to $\widetilde{f}_{i}(p_{i})$. This implies
$\displaystyle\lim_{n\to\infty}f_{i}(\zeta_{n})=\widetilde{f}_{i}(p_{i})$. By
using continuity of $f_{i}^{-1}$,

$f_{i}^{-1}(\widetilde{f}_{i}(p_{i}))=f_{i}^{-1}\left(\lim_{n\to\infty}f_{i}(\zeta_{n})\right)=\lim_{n\to\infty}\zeta_{n},$

which is not possible. So

$\widetilde{f}_{i}(p_{i})\notin f_{i}(\Omega_{i})$ (2.1)

Also by decreasing property of Kobayashi distance

$\displaystyle
K_{\mathbb{D}}(\widetilde{f}_{i}(z_{i}),\widetilde{f}_{i}(p_{i}))$
$\displaystyle\leq K_{\widetilde{\Omega}_{i}}(z_{i},p_{i})$ $\displaystyle
K_{\mathbb{D}}(0,\widetilde{f}_{i}(p_{i}))$ $\displaystyle\leq
K_{\widetilde{\Omega}_{i}}(z_{i},p_{i})$
$\displaystyle\sigma(|\widetilde{f}_{i}(p_{i})|)$ $\displaystyle\leq
K_{\widetilde{\Omega}_{i}}(z_{i},p_{i}).$

By using (2.1)

$c\leq|\widetilde{f}_{i}(p_{i})|\leq\sigma^{-1}(K_{\widetilde{\Omega}_{i}}(z_{i},p_{i})).$

Since $i$ is fixed, the result follows. ∎

######  Theorem 2.3.

Let $\Omega=\Omega_{1}\times\Omega_{2}\subset\mathbb{C}^{2}$ be a bounded
product domain such that $\overline{\mathbb{C}}-\Omega_{1}$ has finite number
of components and none of them is singleton. Assume that $\Omega_{2}$ is a
simply connected domain in $\mathbb{C}$. Then for $z=(z_{1},z_{2})$

$\lim_{z_{1}\to\partial\Omega_{1}}T_{\Omega}(z)=1.$

###### Proof.

Let $E_{1},E_{2},\ldots,E_{n},\ n\geq 1$ bee connected components of
$\overline{\mathbb{C}}-\Omega_{1}$. Let
$\Omega_{1}^{\prime}=\overline{\mathbb{C}}\setminus E_{1}$ which is clearly a
simply connected domain with $\Omega_{1}\subseteq\Omega_{1}^{\prime}$. Since
$E_{1}$ is not a singleton, by Riemann mapping theorem there is a injective
holomorphic function from $\Omega_{1}^{\prime}$ onto $\mathbb{D}$. Note that
$f_{1}(\Omega_{1})$ is domain in $\mathbb{D}$ which we get after deleting
$n-1$ connected compact subsets, say $E_{2}^{\prime},\ldots,E_{n}^{\prime}$.

For $z=(z_{1},z_{2})\in\Omega$, let $l_{f_{1}(z_{1})}$ denotes the distance
between $f_{1}(z_{1})$ and $\cup_{i=2}^{n}E_{i}^{\prime}$ with respect to
Poincaré metric. Let $\mathbb{P}(f_{1}(z_{1}),l_{f_{1}(z_{1})})$ be the disc
centered at $f_{1}(z_{1})$ with radius $l_{f_{1}(z_{1})}$. Choose an injective
holomorphic function $g_{1}:\mathbb{D}\to\mathbb{D}$ with
$g_{1}(f_{1}(z_{1}))=0$. Since Poincaré metric is invariant under conformal
self map of $\mathbb{D}$, $g_{1}$ maps
$\mathbb{P}(f_{1}(z_{1}),l_{f_{1}(z_{1})})$ onto another disc centered at
origin with same radius $l_{f_{1}(z_{1})}$ with respect to Poincaré metric.

Also since $\Omega_{2}$ is simply connected bounded domain, there is an
injective holomorphic function $f_{2}$ from $\Omega_{2}$ onto $\mathbb{D}$
with $f_{2}(z_{2})=0$. Clearly the disc centered at $f_{2}(z_{2})$ with radius
$l_{f_{1}(z_{1})}$ with respect to Poincaré metric is contained in
$f_{2}(\Omega_{2})$. Let $g_{2}$ be identity function on $\mathbb{D}$. Choose
$g\circ f=(g_{1}\circ f_{1},g_{2}\circ f_{2})$ on $\Omega$, then
$\mathbb{D}^{2}(0,\sigma^{-1}(l_{f_{1}(z_{1})}))$ is contained in
$g(f(\Omega))$. For $f=(f_{1},f_{2})$,
$T_{f(\Omega)}(f(z))\geq\sigma^{-1}(l_{f_{1}(z_{1})})$. Since
$|f_{1}(z_{1})|\to 1$, $l_{f_{1}(z_{1})}\to\infty$ therefore
$\sigma^{-1}(l_{f_{1}(z_{1})})\to 1$. By invariant property of squeezing
function

$\lim_{z_{1}\to E_{1}}T_{\Omega}(z)=1.$

Similarly

$\lim_{z_{1}\to E_{i}}T_{\Omega}(z)=1,$

for each $i=2,3,\ldots,n$. Hence

$\lim_{z_{1}\to\partial\Omega_{1}}T_{\Omega}(z)=1.$

∎

##  3\. Examples

######  Example 3.1.

Let $\mathbb{D}\setminus\\{0\\}=\\{\zeta\in\mathbb{C}:0<|\zeta|<1\\}$ be
punctured disc in $\mathbb{C}$. Then for domain
$\Omega=\mathbb{D}\setminus\\{0\\}\times\mathbb{D}\setminus\\{0\\}\times\ldots\times\mathbb{D}\setminus\\{0\\}\subset\mathbb{C}^{n}$
and $z=(z_{1},z_{2},\ldots,z_{n})\in\Omega$,

$T_{\Omega}(z)=\min\\{|z_{1}|,|z_{2}|,\ldots,|z_{n}|\\}.$

For $z=(z_{1},z_{2},\ldots,z_{n})\in\Omega$ by [9, Proposition 4.6]

$\displaystyle T_{\Omega}(z)$
$\displaystyle\geq\min\\{T_{\mathbb{D}\setminus\\{0\\}}(z_{1}),T_{\mathbb{D}\setminus\\{0\\}}(z_{2}),\ldots,T_{\mathbb{D}\setminus\\{0\\}}(z_{n})\\}$
$\displaystyle=\min\\{S_{\mathbb{D}\setminus\\{0\\}}(z_{1}),S_{\mathbb{D}\setminus\\{0\\}}(z_{2}),\ldots,S_{\mathbb{D}\setminus\\{0\\}}(z_{n})\\}$
$\displaystyle=\min\\{|z_{1}|,|z_{2}|,\ldots,|z_{n}|\\}.$

By Theorem 2.1

$T_{\Omega}(z)\leq\min\\{|z_{1}|,|z_{2}|,\ldots,|z_{n}|\\}.$

######  Example 3.2.

For domain
$\Omega=\mathbb{D}\times\mathbb{D}\setminus\\{0\\}\subset\mathbb{C}^{2}$ and
$z=(z_{1},z_{2})\in\Omega$,

$T_{\Omega}(z)=|z_{2}|.$

For $z=(z_{1},z_{2})\in\Omega$ by [9, Proposition 4.6]

$\displaystyle T_{\Omega}(z)$
$\displaystyle\geq\min\\{T_{\mathbb{D}}(z_{1}),T_{\mathbb{D}\setminus\\{0\\}}(z_{2})\\}$
$\displaystyle=\min\\{S_{\mathbb{D}}(z_{1}),S_{\mathbb{D}\setminus\\{0\\}}(z_{2})\\}$
$\displaystyle=|z_{2}|.$

By similar process as in Theorem 2.1

$T_{\Omega}(z)\leq|z_{2}|.$

######  Example 3.3.

Let $A_{r}=\\{\zeta\in\mathbb{C}:r<|\zeta|<1,r>0\\}$ be annulus in
$\mathbb{C}$. Then for domain
$\Omega=A_{r}\times\mathbb{D}\subset\mathbb{C}^{2}$ and
$z=(z_{1},z_{2})\in\Omega$,

$T_{\Omega}(z)=\begin{cases}\dfrac{r}{|z_{1}|}&\text{if},r<|z_{1}|\leq\sqrt{r}\\\
|z_{1}|&\text{if},\sqrt{r}<|z_{1}|<1.\end{cases}$

For $z=(z_{1},z_{2})\in\Omega$ by [9, Proposition 4.6]

$\displaystyle T_{\Omega}(z)$
$\displaystyle\geq\min\\{T_{A_{r}}(z_{1}),T_{\mathbb{D}}(z_{2})\\}$
$\displaystyle=\min\\{S_{A_{r}}(z_{1}),S_{\mathbb{D}}(z_{2})\\}$
$\displaystyle=\begin{cases}\dfrac{r}{|z_{1}|}&\text{if},r<|z_{1}|\leq\sqrt{r}\\\
|z_{1}|&\text{if},\sqrt{r}<|z_{1}|<1\end{cases}.$

Let $T_{\Omega}(z)=c$. Then there exist a holomorphic embedding
$f=(f_{1},f_{2}):\Omega\to\mathbb{D}^{2}$ such that $f(z)=0$ with
$\mathbb{D}^{2}(0,c)\subseteq f(\Omega)$. So there is a holomorphic embedding
$f_{1}:\Omega_{1}\to\mathbb{D}$ such that $f_{1}(z_{1})=0$ with
$\mathbb{D}(0,c)\subseteq f_{1}(A_{r})$. Since holomorphic embedding maps
boundary to boundary, so either $f_{1}$ maps outer boundary of $A_{r}$ to
outer boundary of $f_{1}(A_{r})$ or $f_{1}$ maps inner boundary of $A_{r}$ to
outer boundary of $f_{1}(A_{r})$.

Assume $f_{1}$ maps outer boundary of $A_{r}$ to outer boundary of
$f_{1}(A_{r})$. Let $\Omega^{\prime}$ denotes the union of $f_{1}(A_{r})$ and
compact connected component of $\mathbb{D}\setminus f_{1}(A_{r})$, then
clearly $\Omega^{\prime}$ is simply connected domain. By Riemann mapping
theorem we can find a injective holomorphic function $g_{1}$ from $\mathbb{D}$
onto $\Omega^{\prime}$ with $g_{1}(0)=0$. Consider
$\widetilde{f_{1}}=g_{1}^{-1}\circ f_{1}$ which is a holomorphic embedding
from $A_{r}$ to $\mathbb{D}$ with $\widetilde{f_{1}}(z_{1})=0$ and
$\widetilde{f_{1}}(\partial\mathbb{D})=\partial\mathbb{D}$ such that
$g_{1}^{-1}(\mathbb{D}(0,c))\subseteq\widetilde{f_{1}}(A_{r})$. Also by
Schwarz lemma $g(\mathbb{D}(0,c))\subseteq\mathbb{D}(0,c)$. This implies
$\mathbb{D}(0,c))\subseteq\widetilde{f_{1}}(A_{r})$. Then by [19, Theorem 2],
$c\leq|z_{1}|$.

Assume $f_{1}$ maps inner boundary of $A_{r}$ to outer boundary of
$f_{1}(A_{r})$. Let $g_{1}(\zeta)=\dfrac{r}{\zeta}$ be reflection on $A_{r}$.
Consider $\widetilde{f_{1}}=f_{1}\circ g_{1}$ which is a holomorphic embedding
from $A_{r}$ to $\mathbb{D}$ with
$\widetilde{f_{1}}\left(\dfrac{r}{z_{1}}\right)=0$ and
$\widetilde{f_{1}}(\partial\mathbb{D})=\partial\mathbb{D}$ such that
$\mathbb{D}(0,c))\subseteq\widetilde{f_{1}}(A_{r})$. Again by [19, Theorem 2],
$c\leq r/|z_{1}|$.

######  Remark 3.4.

Domain
$\mathbb{D}\setminus\\{0\\}\times\mathbb{D}\setminus\\{0\\}\times\ldots\times\mathbb{D}\setminus\\{0\\}\subset\mathbb{C}^{n}$
is example of a Pseudoconvex domain in higher dimensions which is not HHR.
Also, if
$\Omega=\Omega_{1}\times\Omega_{2}\times\ldots\times\Omega_{n}\subset\mathbb{C}^{n}$
be a domain such that for some $i=1,2,3,\ldots,n$, $\Omega_{i}$ is punctured
somewhere, then $\Omega$ is not HHR.

## Ackowledgement

We are thankful to our research supervisor Sanjay Kumar Pant for reading its
previous draft and suggesting many changes.

## References

  * [1] H. Alexander, Extremal holomorphic imbeddings between the ball and polydisc, Proc. Amer. Math. Soc., 68(2) (1978), 200–202.
  * [2] T. J. Barth, The Kobayashi distance induces the standard topology, Proc. Amer. Math. Soc., 35(2) (1972), 439–441.
  * [3] F. Deng, Q. Guan, L. Zhang, Some properties of squeezing functions on bounded domains, Pacific Journal of Mathematics, 57(2) (2012), 319–342.
  * [4] F. Deng, Q. Guan, L. Zhang; Properties of squeezing functions and global transformations of bounded domains, Trans. Amer. Math. Soc., 368 (2016), 2679–2696.
  * [5] F. Deng, X. Zhang, Fridman’s invariants, squeezing functions and exhausting domains, Acta Math. Sin. (Engl. Ser.), 35(2019), 1723–1728.
  * [6] J. E. Fornæss, The squeezing function, Talk at Bulgaria academy of science national mathematics colloqium, 2019.
  * [7] B. L. Fridman, On the imbedding of a strictly pseudoconvex domain in a polyhedron, Dokl. Akad. Nauk SSSR, 249(1) (1979), 63–-67.
  * [8] B.L. Fridman, Biholomorphic invariants of a hyperbolic manifold and some applications, Trans. Amer. Math. Soc., 276 (1983), 685–698.
  * [9] N. Gupta, S. K. Pant, Squeezing function corresponding to polydisk, arXiv:2007.14363v2[math.cv].
  * [10] N. Gupta, S. K. Pant, Squeezing function for d-balanced domains, arXiv:2103.00526v1[math.cv].
  * [11] Seungro Joo, Kang-Tae Kim, On boundary points at which the squeezing function tends to one, The Journal of Geometric Analysis, (2016), 10.1007/s12220-017-9910-4
  * [12] K.T. Kim, L. Zhang, On the uniform squeezing property of bounded convex domains in $\mathbb{C}^{n}$, Pacific J.Math., 282(2) (2016), 341–358.
  * [13] Steven G. Krantz., Function theory of several complex variables, AMS Chelsea Publishing, Providence, Rhode Island, 1992.
  * [14] Y. Kubota, An extremal problem on the classical Cartan domain II, Kodai Math. J, 5 (1982), 218–224.
  * [15] K. Liu, X. Sun, S. T. Yau, Canonical metrics on the moduli space of Riemann surfaces, I, J. Differential Geom., 68(3) (2004), 571–637.
  * [16] K. Liu, X. Sun, and S.T. Yau, Canonical metrics on the moduli space of Riemann surfaces, II, J. Differential Geom., 69(1) (2005), 163–216.
  * [17] N.G. Lloyd, Remarks on generalising Rouche’s theorem, J. London Math. Soc. (2), 20 (1979), 259–272.
  * [18] P. Mahajan, K. Verma, A comparison of two biholomorphic invariants, Internat. J. Math., 30(1) (2019), 195–212.
  * [19] T.W. Ng, C.C. Tang, J. Tsai, The squeezing function on doubly-connected domains via the Loewner differential equation, Math. Ann. (2020). https://doi.org/10.1007/s00208-020-02046-w.
  * [20] N. Nikolov, L. Andreev, Boundary behavior of the squeezing functions of C-convex domains and plane domains, Internat. J. Math., 28(5)(2017), https://doi.org/10.1142/S0129167X17500318, 5 pp.
  * [21] N. Nikolov, K. Verma, On the squeezing function and Fridman invariants, J. Geom Anal., 30 (2019), 1218–1225.
  * [22] F. Rong, S. Yang, On Fridman invariants and generalized squeezing functions, Preprint (personal communication).
  * [23] W. Rudin, Function Theory in Polydiscs, Benjamin, New York, 1969 Springer-Verlag, New York-Berlin, 1980.
  * [24] W. Rudin, Function theory in the unit ball of $\mathbb{C}^{n}$, Berlin-Heidelberg-New York: Springer, 1980.
  * [25] S. K. Yeung, Geometry of domains with the uniform squeezing property, Adv. Math., 221(2) (2009), 547–569.
  * [26] A. Zimmer, A gap theorem for the complex geometry of convex domains, Trans. Amer. Math. Soc., 370 (2018), 7489–7509.