# S2-FPN: Scale-ware Strip Attention Guided Feature Pyramid Network for Real-
time Semantic Segmentation

Mohammed A. M. Elhassan1, Chenhui Yang1,*, Chenxi Huang1 , Tewodros Legesse
Munea1, Xin Hong2, Abuzar B. M. Adam3, Amina Benabid4
1 School of Informatics, Xiamen University,Xiamen, Fujian, China, 361005
2 College of Computer Science and Technology, Huaqiao University, Xiamen,
Fujian, China, 361021
3 Centre for Security, Reliability and Trust (SnT), University of Luxembourg,
1855 Luxembourg City, Luxembourg
4 School of Computer Science and Technology, Zhejiang Normal University,
Jinhua 321004, China E-mail addresses<EMAIL_ADDRESS>(Mohammed
A. M. Elhassan). E-mail addresses<EMAIL_ADDRESS>( Chenhui Yang).
E-mail addresses<EMAIL_ADDRESS>( Chenxi Huang) E-mail addresses:
<EMAIL_ADDRESS>( Tewodros Legesse Munea) E-mail addresses:
<EMAIL_ADDRESS>( Xin Hong) E-mail addresses: abuzar.babikir@uni.lu(Abuzar
B. M. Adam) E-mail addresses: amina.benabid@zjnu.edu.cn(Amina Benabid) *
Correspondence should be addressed to Chenhui Yang Manuscript received April
19, 2005; revised August 26, 2015.

###### Abstract

Modern high-performance semantic segmentation methods employ a heavy backbone
and dilated convolution to extract the relevant feature. Although extracting
features with both contextual and semantic information is critical for the
segmentation tasks, it brings a memory footprint and high computation cost for
real-time applications. This paper presents a new model to achieve a trade-off
between accuracy/speed for real-time road scene semantic segmentation.
Specifically, we proposed a lightweight model named Scale-aware Strip
Attention Guided Feature Pyramid Network (S2-FPN). Our network consists of
three main modules: Attention Pyramid Fusion (APF) module, Scale-aware Strip
Attention Module (SSAM), and Global Feature Upsample (GFU) module. APF adopts
an attention mechanisms to learn discriminative multi-scale features and help
close the semantic gap between different levels. APF uses the scale-aware
attention to encode global context with vertical stripping operation and
models the long-range dependencies, which helps relate pixels with similar
semantic label. In addition, APF employs channel-wise reweighting block (CRB)
to emphasize the channel features. Finally, the decoder of S2-FPN then adopts
GFU, which is used to fuse features from APF and the encoder. Extensive
experiments have been conducted on two challenging semantic segmentation
benchmarks, which demonstrate that our approach achieves better accuracy/speed
trade-off with different model settings. The proposed models have achieved a
results of 76.2%mIoU/87.3FPS, 77.4%mIoU/67FPS, and 77.8%mIoU/30.5FPS on
Cityscapes dataset, and 69.6%mIoU,71.0% mIoU,and 74.2% mIoU on Camvid
dataset.The code for this work will be made available at
https://github.com/mohamedac29/S2-FPN

###### Index Terms:

Semantic segmentation, Scale-aware strip attention,deep convolutional neural
networks, real-time semantic segmentation.

##  1 Introduction

Semantic segmentation is an essential high-level topic in computer vision and
has been widely used in various challenging problems such as, medical
diagnosis, [1, 2], autonomous vehicles [3], and scene analysis [4, 5]. Unlike
image classification, which aims to classify the whole image, semantic
segmentation predicts the per-pixel class for each image content. The current
frontiers in semantic segmentation methods are driven by the success of deep
convolution neural networks, specifically the fully convolution network (FCN)
Framework [6]. In the original FCN approach to obtain larger receptive fields,
we increase the depth of the network with more convolutional and downsampling
layers. However, simply increasing the number of downsampling operations leads
to reduced feature resolution, which causes spatial information loss. On the
other hand, the increased number of convolutional layers adds more challenges
to network optimization.

Figure 1: Accuracy/Speed performance comparison on the Cityscapes test set.
Our methods are presented in red dots while other methods are presented in
blue dots. Our approaches achieve state-of-the-art speed-accuracy trade-off

Over the years, various approaches have been proposed to tackle the drawbacks
of the typical FCN-based architecture. Here we mention the following: (i)
increasing the resolution of feature maps or maintaining a high-resolution
feature across stages, e.g., through decoder network (Figure 2 (2(a))) [1, 7,
8], dilated convolutions [9, 10] or high resolution [11]. (ii) improve the
segmentation performance with the subsequent approaches, e.g., PSPNet [5],
DeepLab [10, 12], PAN [13] DenseASPP [14] , RefineNet [15], PPANet [16], and
ParseNet [17]. These previous networks enlarge the receptive field to exploit
the rich contextual information. Nevertheless, the latter solution is
infeasible for real-time applications because it requires heavy networks and
expensive computation. (iii) some networks such as [18, 19] have added spatial
encoding path to preserve the spatial details as shown in Figure 2.2(b). To
perform fast segmentation with light computation cost, small size and
satisfactory segmentation accuracy, there are certain design philosophy that
could be followed: (i) incorporate a pre-trained lightweight classification
networks[20, 21, 22, 23] as a backbone to construct segmentation architecture
such as BiSeNet [19], STDC-Seg [24]. (ii) design a building block that is
suitable for low computation cost [25, 26]. (iii) put into consideration the
downsampling strategy and the depth of the network [27]. (iv) combine the low-
level and high-level features using the multi-path framework [19, 28, 18]. in
addition to that, restricting the input image size plays an important part in
increasing the inference speed.

Furthermore, many existing frameworks [29, 30, 31, 32, 33] have been developed
based on FPN [33] to exploit the inherent multi-scale feature representation
of deep convolutional networks. More specifically, FPN-based architectures
combine large-receptive field features, low-resolution- with small-receptive-
field features, high-resolution to capture objects at different scales. For
instance, [32, 33] utilize the lateral path to fuse adjacent features in a
top-down manner. This way FPN strengthen the learning of multi-scale
representation.Until now, feature pyramid network based approaches have
achieved state-of-arts performance in object detection. Nevertheless, the
fusion of adjacent features from different stages ignores the semantic gap
between these features, which degrades the multi-scale feature representation
(FPN-like model is shown in Figure 2 (2(c)). To alleviate this problem and
achieve a better accuracy/speed trade-off, we propose Scale-aware Strip
Attention guided Pyramid Network(S2-FPN), which can extract multi-
scale/multilevel representations while maintaining high computation
efficiency. Fig. 1 illustrates the accuracy (mIoU) and inference speed (FPS)
obtained by several state-of-the-art methods, including PSPNet[5],
DeepLab[10], ENet[34],ICNet[35], DABNet[6], DFANet-A[36], DFANet-B[36],
BiSeNet1[19], BiSeNet2[19], FasterSeg[37], TD4-Bise18[38], FANet-18[39],
FANet-34[39], LBN-AA[40], AGLNet[40], BiSeNetV2[28], BiSeNetV2-L[28],
HMSeg[41], TinyHMSeg[41], STDC2-Seg50[24], STDC2-Seg50[24], and our proposed
method, on the Cityscapes test dataset.

(a) Encoder-decoder

(b) Two Pathway

(c) FPN

(d) S2FPN

Figure 2: A comparison of an important semantic segmentation architectures,(a)
encoder-decoder model,(b) two-pathway model,(c) feature pyramid model,(d)
Scale-aware feature pyramid model (Ours).

The main contributions are summed as follows.

  1. 1.

We propose a Scale-ware strip attention module with low computation overhead.

  2. 2.

A new attention pyramid fusion(APF) module is proposed to address the semantic
gap in the main FPN. Our APF aggregates multi-scale representation
efficiently.

  3. 3.

Based on the two modules, we propose S2-FPN architecture, which effectively
improves the accuracy with high inference speed in complex environments.

  4. 4.

We report the results on a single-scale image on two datasets. S2-FPN achieves
77.8%, 77.4% and 76.2% mIoU on the Cityscapes dataset. 74.2%, 71.0, and 69.6
with backbones ResNet34-modified, ResNet34 and ResNet18, respectively.

  5. 5.

For image resolution of 512x1024, S2-FPN(ResNet18) obtains 87.3 FPS on NVIDIA
GTX1080Ti using PyTorch on the Cityscapes dataset, whereas S2-FPN (ResNet34)
and S2-FPN(ResNet34-modified) achieve 67 FPS and 30.5 FPS, respectively.

##  2 Related Work

###  2.1 Efficient Network Designs

In recent literature, real-time model design has become an essential part of
computer vision research development. Several lightweight architectures were
introduced as a trade-off between accuracy and latency SqueezeNet [20],
MobileNet [21, 42], ShuffleNet [43, 22],and Xception [23]. SqueezeNet reduces
the number of parameters through the squeeze and expansion module. MobileNetV1
implemented depth-wise separable convolutions to reduce the number of
parameters. MobileNetV2 proposed an inverted residual block to alleviate the
effect of depth-wise separable convolutions. ShuffleNet and Xception utilize
group convolution to reduce FLOPs. ResNet [44] adopted residual blocks to
obtain outstanding performance. These models were designed primarily for image
classification tasks. Later on, incorporated to work as backbones for semantic
segmentation networks to achieve real-time semantic segmentation.

###  2.2 Real-time Segmentation

Real-time semantic segmentation methods aim to predict dense pixels accurately
while maintaining high inference speed. Over the years there are different
approaches have been proposed in this regard. ENet [45], ERFNet [26], and
ESPNet [46, 47] proposed compact semantic segmentation networks that utilize
lightweight backbones. LEDNet [48], DABNet [25] introduced a lightweight model
to address the computation burden that limits the usage of deep convolutional
neural networks in mobile devices. The former used asymmetric encoder-decoder-
based architecture and implemented channel split and shuffle operations to
reduce the computation cost. The latter structured a depth-wise symmetric
building block to jointly extract local and global context information.
FDDWNet [49] utilized factorized depth-wise separable convolutions to reduce
the number of parameters and the dilation to learn multi-scale feature
representations. ICNet [50] proposes a compressed version of PSPNet based on
an image cascade network to increase the semantic segmentation speed. DFANet
[27] obtains the multi-scale feature propagation by utilizing a sub-stage and
sub-network aggregation.Tow-pathway networks such as BiSeNetV2 [28] and DSANet
[18] Fast-SCNN [51] introduce a shallow detailed branch to extract the spatial
details and a deep semantic branch to extract the semantic context. This paper
propose a lightweight model that utilizes (ResNet18 or ResNet34) as backbones.

###  2.3 Self-Attention Mechanism

Recent studies have shown that attention mechanisms can be successfully
applied to different tasks such as machine translation [52, 53], image
classification [54], object detection [55, 56] and semantic segmentation [57].
The attention mechanisms perform better than convolutional neural networks
when used to model the long-range dependencies. SENet [58] proposed the
squeeze and excitation module to re-calibrate the channel dependency
dynamically. Motivated by self-attention, DANet [59] improves feature
representation by applying channel attention and position attention, while
OCNet [60] uses self-attention to learn the object context. In this paper, to
construct an efficient and effective approach for modelling the long-range
dependencies, we introduce scale-aware attention with vertical striping
operation to reduce the computation cost.

###  2.4 Multi-scale Feature Fusion

One of the main difficulties in semantic segmentation is effectively
processing and representing multi-scale features. Earlier semantic
segmentation methods perform multilevel feature representations fusion [6, 7,
1]. Multi-branch architectures have been adopted to tackle the issue of multi-
scale fusion, e.g., BiSiNetV1 [19], BiSiNetV2 [28] and DSANet [18], add a
shallow branch to preserve the spatial details. ICNet fused branches with
different input resolutions. SPFNet [61] proposed a method for learning
separate spatial pyramid fusion for each feature subspace. Feature pyramid
network [33] is widely adopted in the object detection task [62, 29] to
address the problem of multi-scale feature representation. The feature pyramid
network architecture introduces a top-down routing to fuse features. Based on
FPN, NAS-FPN [29] utilizes the neural architecture search to obtain optimal
topology. EfficientDet [31] acquires more higher-level feature fusion using a
repeated bidirectional path. PANet [30] proposes a method that facilitates the
information flow through a bottom-up path augmentation. This work proposes the
Attention Pyramid Fusion module, which adopts a lightweight scale-aware
attention strategy to bridge the semantic gap between the different levels
more efficiently.

Figure 3: The detailed architecture of the proposed Scale-aware Strip
Attention Guided Pyramid Fusion model (S2-FPN). The model constructs from the
following modules: (a) Encoder, which incorporates ResNet18 or ResNet34, (b)
Attention Pyramid Fusion module, (c) illustrates the Global Feature
Upsample(GFU) module, (e) Feature adaptation block (FAB), and (e) Components
of Coarse Feature Generator block.

##  3 Methodology

This section describes the proposed Scale-aware Strip Attention Guided Feature
Pyramid Network (S2-FPN). We first give an overview of S2-FPN in Section 3.1.
Then, we introduce the different components of S2-FPN from Section 3.2 to
Section 4. Finally, we present the experimental results and analysis in
Section 5.

###  3.1 Overview

Figure 3. illustrates the proposed S2-FPN, which follows the feature pyramid
network and encoder-decoder architectures. S2-FPN consists of three main
parts: Feature Extractor or Encoder, Attention Pyramid Fusion (APF), and
Global Feature Upsample (GFU). In particular, Coarse Feature Generator Block
(CFGB), Feature Adaptation Block (FAB), and Scale-aware Strip Attention Module
(SSAM). S2-FPN takes ResNet18 and ResNet34 [44] as base models for feature
extraction while removing the global average pooling and the softmax. We
utilize $\\{F_{1},F_{2},F_{3},F_{4},F_{5}\\}$ as a feature hierarchy from the
base model with strides $\\{2,4,8,16,32\\}$. Then, we attached a coarse
feature generator block and feature adaptation block in parallel after
$F_{5}$. CFGB is constructed from a depth-wise convolution layer with stride 2
to generate coarse features for the attention pyramid fusion, while FAB is
produced through a depth-wise convolutional layer with stride 1 to project the
feature maps of the encoder into the global feature upsample module. Note that
the attention pyramid features have strides of $\\{4,8,16,23\\}$ pixel w.r.t
the input image. $\\{APF_{2},APF_{3},APF_{4},APF_{5}\\}$ are the top-down
features generated using the Attention Feature Pyramid (APF) module (Section
3.3).

###  3.2 Scale-aware Strip Attention Module

In order to capture the long-range dependencies and reduce the computation
cost, we introduce the Scale-aware Strip Attention module. In particular,
inspired by the results in [63, 59]; and to maintain the contextual
consistency, we apply striping operation to collect long-range context along
the vertical axis. Figure 4 illustrates the details structure of Scale-aware
Strip Attention module.
Let $\textbf{F}_{Refine}\in\mathbb{R}^{C\times H\times W}$ be an input feature
maps ($\textbf{F}_{Refine}$ is the output of feature refinement block (FRB) in
APF Figure 5) where C is the number of channels, W and H are the spatial
dimensions. SSAM first extracts height-wise contextual information from each
row of the refined feature maps $\textbf{F}_{Refine}$ by aggregating the
$C\times H\times W$ input representation into a
$\textbf{Z}^{max}\in\mathbb{R}^{C\times H\times 1}$,
$\textbf{Z}^{avg}\in\mathbb{R}^{C\times H\times 1}$ using Max Poling or
Average Pooling operation in parallel. $\textbf{Z}^{avg}$ and
$\textbf{Z}^{max}$ are illustrated in Equation 1 and 2, respectively.

$\textbf{Z}^{avg}=\textbf{G}_{pool}(\textbf{F}_{R})$ (1)

$\textbf{Z}^{max}=\textbf{G}_{pool}(\textbf{F}_{R})$ (2)

Where $\textbf{G}_{pool}(.)$ represents max or average pooling operations.
Then SSAM applies $1\times 1$ convolution layer with shared weights to
$\textbf{Z}^{max}$ and $\textbf{Z}^{avg}$ (Equations 3 and 3) to transfer the
height-wise context information of every feature within $\textbf{F}_{R}$, and
employs a softmax function to generate an attention map that highlights the
importance of corresponding feature maps in $\textbf{F}_{R}$ (Equation 5).

$\textbf{F}_{1}=f^{1s}(\textbf{Z}^{avg})$ (3)

$\textbf{F}_{2}=f^{1s}(\textbf{Z}^{max})$ (4)

$\textbf{A}=\sigma(\textbf{F}_{1}\odot\textbf{F}_{2})$ (5)

Where $f^{1s}(.)$ represents a convolutional layer with shared weights. The
attention mechanism can select the appropriate scale feature dynamically and
fuse feature of different scale by self-learning. Next, we have performed
element-wise multiplication operations ($\odot$) between attention maps and
$\textbf{F}_{1}$,$\textbf{F}_{2}$ to generate the scaled feature map
$\textbf{F}_{scale}$. Finally, an element-wise sum operation between
$\textbf{F}_{scale}$ and the input feature map F is employed with learnable
parameters $\alpha$ to obtain the final output $\textbf{F}_{SSAM}$ as in
Equation 7

$\textbf{F}_{scale}=\textbf{A}\odot\textbf{F}_{2}+\textbf{A}\odot\textbf{F}_{2}$
(6)

$\textbf{F}_{SSAM}=\alpha\textbf{F}_{scale}+(\alpha-1)\textbf{F}$ (7)

###  3.3 Attention Pyramid Fusion (APF) Module

Maintaining detailed information of an object when passing an image through
the encoder and decoder is a challenging problem. Feature pyramid network
[33], which was first introduced for object detection, has been used to
leverage the multi-level feature from the backbone (see Figure 2 2(c)).
However, inherent defects in FPN inhibit it from extracting sufficient
discriminative features. For instance, the extracted feature by the lateral
connection in FPN could suffer information loss in the high-level layers.
Another issue is that using bilinear interpolation operation and simple
element-wise addition of adjacent features ignores the semantic gaps of
different depths. Based on these observations, we propose the Attention
Pyramid Fusion (APF) module, which can learn discriminative multi-scale
features. The detailed structure of our Attention Pyramid Fusion block is
shown in Figure 5. In APF, the scale-aware strip attention guides the deeper
stages to aggregate high-resolution features, while channel-wise attention
employs channel reweighting. By combining these attention mechanisms, APF is
able to extract more detailed information in the final segmentation map.
In order to combine the coarse $\textbf{F}_{i}$ and low-level
$\textbf{F}_{i-1}$ adjacent feature maps, we first apply a 1x1 convolution on
$\textbf{F}_{i-1}$ followed by batch normalization and ReLU operation.
Meanwhile, we upsample the coarse feature map generated with CFGB to match the
low-level feature in terms of channel and spatial dimensions. Then the coarse
and the low-level feature maps are concatenated channel-wise to produce
$\textbf{F}_{concat}$, and the output of this operation is fed into a feature
refinement block (FRB) to reduce the aliasing effect. FRB is constructed by
stacking two convolutional layers, $1\times 1$ and $3\times 3$ each, followed
by batch normalization and ReLU.
Furthermore, we have developed two modules to reduce the gap between different
levels and enhance semantic consistency. Specifically, We first apply a
$3\times 3$ convolution followed by batch normalization and ReLU on the low-
level path, then multiplied by channel attention (Fig 5.b ) to generate a
feature reweighting (CRB) that filters the irrelevant information (see Fig 5).
CAM employs global average pooling to extract global context and learn the
weights for the channel sub-module. The final output of feature reweighting
branch $\textbf{X}_{A}$ is given in Equation 8.

Figure 4: The illustration of Scale-Aware Strip Attention module SSAM.

$\textbf{X}_{A}=f^{3}(f^{1}(\textbf{F}_{i-1}))\odot\textbf{F}_{C}(\textbf{F}_{R})$
(8)

where $f^{3}(\cdot)$ represents the convolution layer whose kernel size is
$3\times 3$, $f^{1}(\cdot)$ represents the convolution layer whose kernel size
is $1\times 1$. Secondly, we introduce scale-aware attention to aggregate
discriminative contextual information. The output of SSAM ($\textbf{F}_{S}$)
is multiplied by the output of the CFGB branch. The output of feature
reweighting branch $\textbf{X}_{B}$ is given in Equation 9.

$\textbf{X}_{B}=f^{3}(\mathbb{UP}(\textbf{F}_{i}))\odot\textbf{F}_{S}(\textbf{F}_{R})$
(9)

where $\mathbb{UP}(\cdot)$ represents the bilinear upsample operation. The two
branches $\textbf{X}_{A}$ and $\textbf{X}_{B}$ are fused with an element-wise
summation to produce $\textbf{F}_{out}$ . Finally, the output feature is
refined using a $3\times 3$ convolution, dropout, and projection layer to
produce the deep supervision in the four-level hierarchy. Similarly, the
output is passed into a feature refinement block consisting of $3\times 3$
convolution to generate the output for attention pyramid fusion of the next
stage.

$\textbf{F}_{out}=\textbf{X}_{A}+\textbf{X}_{B}$ (10)

Where $\textbf{F}_{out}$ represents the fused coarse and low-level features.

##  4 Global Feature Upsample (GFU)

In typical U-Net or encoder-decoder architectures, the extracted high-level
feature maps are directly upsampled in the decoding part to restore the
original resolution. However, if the high-level feature can be fully
propagated, it will obtain an accurate segmentation map. We applied a
depthwise convolution with stride to $F_{5}$ to obtain enough information to
produce a coarse semantic segmentation. On the other hand, the features from
the attention-based feature pyramid network are rich in details information
about different objects, which are beneficial in recognizing the boundaries
between objects. We can get an accurate semantic map by combining the details
and coarse semantic information. The proposed GFU module is shown in Fig.6.
For the high-level features, the proposed global feature upsampling module
firstly performs a bilinear upsampling on the input feature to sample it to
the same dimensions of AFPN features. We then applied $1\times 1$ convolution
and non-linearity followed by global average pooling to generate a global
context and sample it to the same channels of AFPN. In the meantime, we
performed $1\times 1$ convolution with batch normalization and relu on the
feature from AFPN and added high-level features. Finally, the output feature
is obtained by applying $1\times 1$ convolution with batch normalization and
relu. The proposed GFU can process and integrate features maps from different
stages more efficiently. The output of this block $\textbf{X}_{GFU}$ can be
summarized as follows:

$\textbf{X}_{GFU}=f^{1}(\textbf{G}_{pool}(f^{1}(ReLU(\mathbb{UP}(\textbf{X}_{F}))))+f^{1}(\textbf{X}_{P}))$
(11)

where ReLU is the Rectified Linear Unit, $\textbf{X}_{F}$ denotes the feature
map from the feature adaptation block (FAB), $\textbf{X}_{P}$ represents the
feature from the attention pyramid fusion (APF) module.

Figure 5: An overview of the Attention Pyramid Fusion Module. (a) APF module
architecture. (b) Components of the channel attention module (CAM). (c)
Components of the feature refinement block (FRB). TABLE I: THE PER-CLASS,
CLASS, AND CATEGORY IOU EVALUATION ON THE CITYSCAPES TEST SET."

Method | Road | S.Walk | Build | Wall | Fence | Pole | T-Light | T-Sign | Veg | Terrain | Sky | Person | Rider | Car | Truck | Bus | Tra | Motor | Bic | mIoU
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CRF-RNN[34] | 96.3 | 73.9 | 88.2 | 47.6 | 41.3 | 35.2 | 49.5 | 59.7 | 90.6 | 66.1 | 93.5 | 70.4 | 34.7 | 90.1 | 39.2 | 57.5 | 55.4 | 43.9 | 54.6 | 62.5
FCN[6] | 97.4 | 78.4 | 89.2 | 34.9 | 44.2 | 47.4 | 60.1.5 | 65.0 | 91.4 | 69.3 | 93.9 | 77.1 | 51.4 | 92.6 | 35.3 | 48.6 | 46.5 | 51.6 | 66.8 | 65.3
DeepLabv2[10] | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 70.4
Dilation10[36] | 97.6 | 79.2 | 89.9 | 37.3 | 47.6 | 53.2 | 58.6 | 65.2 | 91.8 | 69.4 | 93.7 | 78.9 | 55.0 | 3.3 | 45.5 | 53.4 | 47.7 | 52.2 | 66.0 | 67.1
AGLNet[64] | 97.8 | 80.1 | 91.0 | 51.3 | 50.6 | 58.3 | 63.0 | 68.5 | 92.3 | 71.3 | 94.2 | 80.1 | 59.6 | 93.8 | 48.4 | 68.1 | 42.1 | 52.4 | 67.8 | 70.1
BiSeNetV2/BiSeNetV2_L[28] | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 73.2
LBN-AA[40] | 98.2 | 84.0 | 91.6 | 50.7 | 49.5 | 60.9 | 69.0 | 73.6 | 92.6 | 70.3 | 94.4 | 83.0 | 65.7 | 94.9 | 62.0 | 70.9 | 53.3 | 62.5 | 71.8- | 73.6
S2-FPN18 | 98.2 | 84.7 | 92.0 | 50.2 | 54.9 | 62.9 | 71.3 | 75.6 | 93.0 | 70.0 | 94.7 | 84.6 | 67.4 | 95.2 | 65.1 | 79.4 | 68.7 | 65.1 | 73.9 | 76.2
S2-FPN34 | 98.4 | 85.5 | 92.7 | 56.2 | 57.6 | 64.7 | 72.7 | 76.2 | 93.3 | 70.8 | 94.9 | 85.2 | 69.0 | 95.52 | 66.7 | 80.8 | 69.0 | 66.4 | 74.3 | 77.4
S2-FPN34M | 98.1 | 84.5 | 92.7 | 51.5 | 57.3 | 68.5 | 76.5 | 79.3 | 93.5 | 71.4 | 94.5 | 86.4 | 68.4 | 95.3 | 61.5 | 80.4 | 72.6 | 68.0 | 76.5 | 77.8

Figure 6: Illustration of global feature upsampling module. TABLE II:
COMPARISON BETWEEN THE PROPOSED METHOD S2-FPN AND THE OTHER SOTA METHODS ON
THE CITYSCAPES TEST DATASET. WE REPORT THE BACKBONE, INPUT RESOLUTION, GPU
TYPE, NUMBER OF PARAMETERS (M), FLOPS (G), EVALUATION SPLIT (SET), ACHIEVED
ACCURACY (MIOU), AND THE INFERENCE SPEED (FPS)

Method Backbone Resolution GPU Parameters(M) Flops (G) test set mIoU FPS
PSPNet[5] ResNet101 713 $\times$713 - 250.8 412.2 ✓ 81.2 0.78 DeepLab[10]
VGG16 512$\times$1024 Titan X 262.1 457.8 ✓ 63.1 0.25 ENet[34] No 640
$\times$360 TitanX 0.4 3.8 ✓ 58.3 135.4 ICNet[35] PSPNet50 1024$\times$2048
TitanX 26.5 28.3 ✓ 69.5 30.3 DABNet[6] No 512$\times$1024 GTX 1080Ti 0.76 10.4
✓ 70.1 104 DFANet-A[36] XceptionA 1024$\times$1024 Titan X 7.8 3.4 ✓ 71.3 100
DFANet-B[36] XceptionB 1024$\times$1024 Titan X 4.8 2.1 ✓ 67.1 120
BiSeNet1[19] Xception39 768$\times$1536 NVIDIA Titan X 5.8 14.8 ✓ 68.4 105.8
BiSeNet2[19] ResNet18 768$\times$1536 NVIDIA Titan X 49 54.02 ✓ 74.8 65.5
FasterSeg[37] No 1024$\times$2048 GTX 1080Ti - ✓ 71.5 163.9 TD4-Bise18[38]
BiseNet18 1024$\times$2048 Titan Xp - ✓ 74.9 - FANet-18[39] ResNet18
1024$\times$2048 Titan X - 49 ✓ 74.4 72 FANet-34[39] ResNet34 1024$\times$2048
Titan X - 65 ✓ 75.5 58 LBN-AA[40] LBN-AA+MobileNetV2 488$\times$896 Titan X
6.2 49.5 ✓ 73.6 51.0 AGLNet[40] No 512$\times$1024 GTX 1080Ti 1.12 13.88 ✓
71.3 52.0 BiSeNetV2[28] No 512$\times$1024 GTX 1080Ti 49.0 21.2 ✓ 72.6 156
BiSeNetV2-L[28] No 512$\times$1024 GTX 1080Ti 118.5 ✓ 75.3 47.3 HMSeg[41] No
768$\times$1536 GTX 1080Ti 2.3 ✓ 74.3 83.2 TinyHMSeg[41] No 768$\times$1536
GTX 1080Ti 0.7 ✓ 71.4 172.4 STDC1-Seg50[24] STDC1 512$\times$1924 GTX 1080Ti
8.4 - ✓ 71.9 250.4 STDC2-Seg50[24] STDC2 512$\times$1024 GTX 1080Ti 12.5 - ✓
73.4 188.6 STDC1-Seg75[24] STDC1 768$\times$1536 GTX 1080Ti 8.4 - ✓ 75.3 126.7
STDC2-Seg75[24] STDC2 768$\times$1536 GTX 1080Ti 12.5 - ✓ 76.8 97.0 S2-FPN18
ResNet18 512$\times$1024 GTX 1080Ti 17.8 29.1 ✓ 76.2 87.3 S2-FPN34 ResNet34
512$\times$1024 GTX 1080Ti 27.9 48.4 ✓ 77.4 67 S2-FPN34M ResNet34
512$\times$1024 GTX 1080Ti 27.9 190 ✓ 77.8 30.5

##  5 Experimental Results

We evaluate the effectiveness of our proposed segmentation network on two
datasets: Cityscapes [65] and Camvid [66]. We first introduce the training
details. Then, we provide our full report of S2-FPN compared to other state-
of-the-arts methods on the two benchmarks. The following criteria are used for
the performance evaluation, i.e. accuracy,speed,backbone,input resolution and
parameters in Cityscapes. Experimental results demonstrate that S2-FPN obtains
a trade-off between accuracy and speed on both Camvid and Cityscapes datasets.
In the following subsection, we provide further details.

###  5.1 Training setting details

All the experiments are implemented with one Nvidia GTX 1080Ti GPU, on the
PyTorch platform [45]. We use ResNet with weights pre-trained on ImageNet [67]
as our backbone. The Adam optimizer [68] with initial learning rate of $3e-4$
is used to train the model on both Camvid and Cityscapes datasets. We use
weight decay of 5e-6. Following the previous methods settings [5, 10], we
decay the learning rate using polynomial learning rate scheduling using
equation $(1-\frac{iter}{max_{i}ter})^{power}$. The model has been trained for
500 epochs on Cityscapes dataset . For Camvid dataset we have train for 180
epochs when the model configured with ResNet18 and 150 when configured
modified ResNet34. All BatchNorm layers in our architecture replace by
InPlaceABN-Sync[69]. To avoid over-fitting, We apply a number of data
augmentations techniques as follows: random resize with scale range
[0.75,1.0,1.25,1.5,1.75, 2.0], random horizontal flipping and random cropping
images to a resolution of $512\times 1024$ for Cityscapes and $360\times 360$
for Camvid.
We adopt a well-known metrics to evaluate our model. Frame per-second (FPS)
and mean Intersection over Union (mIoU), which measure the semantic
segmentation latency and accuracy, respectively. Moreover, we evaluate the
computational complexity and memory consumption based on floating point
operations (FLOPs) and model parameters (Params), respectively.

###  5.2 Cityscapes Dataset

####  5.2.1 Ablation Study

In this subsection, we conduct a series of experiments to explore the effect
of each component (GFU, APF, SSAM, Supervision) in the proposed method. SS-FPN
is implemented with a different backbone setting (ResNet18, ResNet34) and we
have tested the model with the different settings.

TABLE III: COMPARISON BETWEEN S2-FPN16, S2-FPN34, AND S2-FPN34M IN TERMS OF
PARAMETERS, FRAME PER SECOND AND ACCURACY ON CITYSCAPES VALIDATION SET

Method Parameters(M) Flops FPS mIoU S2-FPN18 17.8 29.1 87.3 76.4 S2-FPN34 27.9
48.4 67 77.1 S2-FPN34M 27.9 190.0 30.5 77.5

####  5.2.2 Ablation Study With Modules

In this section, we conduct an empirical evaluation to ensure the
effectiveness of the different components in our architecture. We show the
importance of the scale-aware strip attention module (SSAM), attention pyramid
fusion module (APF), Global Feature (GFU) module, and Deep Supervision. The
training set is used to train the model, while the validation set is used for
evaluation. We first conduct the experiment with ResNet18 as a baseline model.
As illustrated in TableV, the baseline network obtains 65.7% mIoU . We further
evaluate the effect of global feature upsample module, Fig 3 by attaching it
to the encoder (Fig 3 (a). Essentially, the GFU is designed to attain category
information without using complex decoder blocks. The result in Table V shows
that adding GFM obtains 71.9 % mIoU, which add 6.2%mIoU improvement to the
baseline. Moreover, we add APF without supervision or scale-aware strip
attention module (replaced with $1\times 1$ convolution), it brings 3.7 %mIoU
performance gain. Using deep supervision has added 0.2 % mIoU improvement. To
capture the long-range dependencies, We further evaluate the significance of
the scale-aware attention module by adding it into APF with all the other
modules. Table V, shows this module improves the performance by 0.5%mIoU. The
ablation study and analyses highlighted the importance of each component in
our network. S2-FPN has improved the baseline by 10.7 % mIoU according to our
training settings. Thus, it presents an important strategy to design a network
with accuracy/speed trade-off.

(a) Original

(b) S2-FPN18

(c) S2-FPN34

(d) S2-FPN34M

(e) Ground Truth

Figure 7: Visual results of our method S2-FPN on Cityscapes dataset. TABLE IV:
COMPARISON CORSSENTROPY AND OHEM LOSS ON CITYSCAPES VALIDATION DATASET

Method Params Flops FPS mIoU CE 17.5 29.1 87 75.6 OHEM 17.5 29.1 87 76.4

Online Bootstrapping: Camvid and Cityscapes datasets contain easy and hard
pixels. One efficient way to train these hard pixels is to automate their
selection process during training. Following the previous works[], we utilize
OHEM or online hard example mining for hard training pixels. The examples with
a probability less than the threshold are considered hard training pixels. In
our experiments, k depends on the image crop size and the threshold is set to
0.7. Table IV shows that using OHEM increases the performance by 0.9 % mIoU
over Cross-entropy on Cityscapes validation dataset.

Ablation study with different baselines: We further detailed the performance
comparison of three settings for our proposed S2-FPN model on the Cityscapes
validation dataset. Specifically, we train the model using the following
baseline, ResNet18, ResNet34 and a modified version of ResNet34 as shown in
Table III. S2-FPN obtains accuracy of 76.4% mIoU and 87.3 FPS, which has a
better inference speed than using ResNet34 variants with minor inaccuracy.
Using the modified ResNet34 (S2-FPN34M) achieve the highest accuracy with 0.4
% mIoU increase over S2-FPN34 with the same number of parameters, but with
very high floating points. S2-FPN18 and S2-FPN34 seem to obtain a better
accuracy/speed trade-offs.

TABLE V: ABLATION STUDY FOR THE PROPOSED MODULES ON THE CITYSCAPES VALIDATION
SET,WHERE RESNET18 ARCHITECTURE SERVES AS THE BASELINE. WE SHOW THE
EFFECTIVENESS OF GFM: GLOBAL FUSION MODULE, AFP: ATTENTION PYRAMID FUSION ,
SSAM: SCALE-AWARE STRIP ATTENTION, AND SUPERVISION

Backbone GFU APF Supervision SSAM Params Flops FPS mIoU(%) ✓ 11.2 19 187 65.7
✓ ✓ 11.5 19.7 156.9 71.9 ✓ ✓ ✓ 17.8 29.2 93.8 75.6 ✓ ✓ ✓ ✓ 17.8 29.2 88 75.9 ✓
✓ ✓ ✓ ✓ 17.8 29.1 87 76.4

Effectiveness of Attention Pyramid Fusion (APF) module:. To verify the
effectiveness of the attention pyramid fusion module, we compare it with the
conventional feature pyramid network. As shown in Table VI, the attention
feature pyramid module achieves 73.9% mIoU, which is 4.8 % higher than the
feature pyramid network. The result shows that channel reweighing and
contextual information encoding are effectively incorporated in the proposed
attention pyramid fusion.

TABLE VI: COMPARISON BETWEEN FEATURE PYRAMID NETWORK (FPN) AND ATTENTION
PYRAMID FUSION (APF) ON THE CITYSCAPES VALIDATION DATASET

Method Params Flops FPS mIoU baseline 11.2 19 187 65.7 FPN 11.4 20.9 152 69.1
AFPN 17.5 28.6 94.3 73.9

###  5.3 Comparisons with State-of-the-Art Methods

In Table II, we present the comparisons between our S2-FPN and the state-of-
the-arts real-time semantic segmentation methods. Our method is tested on a
single GTX 1080Ti GPU with image resolution of 512 × 1024. We tested the speed
without any accelerating strategy and used only fine-data to train the model.
As stated in Table II, our proposed network setting get 76.2% mIoU with 102
FPS by using ResNet18 as the backbone network. As can be observed, the
accuracy is better than all other methods except for STD-Seg75, which has 0.2
% mIoU better than our small model. The inference speed is significantly
faster, even though we tested it using GTX1080 ti, this accuracy and inference
speed prove that even with light backbones our approach still obtains better
performance than other approaches. Besides, S2-FPN with ResNet34 base model
obtains 85 FPS inference speed with 77.4 % mIoU accuracy, which is state of
the art on speed/accuracy trade-offs on the Cityscapes benchmark and exceed
the best model STD-Seg75 by a margin of 0.6 % mIoU. Finally, we have modified
the ResNet34 backbone by changing the stride in stage 2 from 2 to 1 and run
the model (we name it S2-FPN34M), this modification increases the accuracy by
about 0.4 % mIoU but reduced the speed significantly to 30.5 FPS. We give a
detail of per-class accuracy of our three configurations Table I.

(a) Original

(b) S2-FPN18

(c) S2-FPN34

(d) S2-FPN34M

(e) Ground Truth

Figure 8: Visual results of our method S2-FPN on Camvid dataset.

###  5.4 Results on Camvid Dataset

As illustrated in Table VII, we evaluate the segmentation accuracy of our
proposed method on the Camvid dataset. We trained with an input image size of
360x480, with no use of external data. The training images used to train the
model and validation images to evaluate our models while the testing samples
are used to get the result for comparison with other methods. Overall, our
model has better accuracy. Our S2-FPN model with ResNet18 as a backbone
achieves a 69.6 % mIoU on the test set, while S2-FPN34 obtains accuracy of
71.0 % mIoU, which is a significant trade-off between speed and accuracy. The
last model configuration S2-FPN34M which replaces the stride at stage 2 from 2
to 1 achieves 74.2% mIoU . This further shows the validity and effectiveness
of our proposed method.

TABLE VII: INDIVIDUAL CATEGORY RESULTS ON CAMVID TEST SET IN TERMS OF MIOU FOR
11 CLASSES

Method Bui Tree Sky Car Sig Roa Ped Fen Pol Side Bic mIoU SegNet[7] 88.8 87.3
92.4 82.1 20.5 97.2 57.1 49.3 27.5 84.4 30.7 55.6 ENet[45] 74.7 77.8 95.1 82.4
51.0 95.1 67.2 51.7 35.4 86.7 34.1 51.3 BiSeNet1[19] 82.2 74.4 91.9 80.8 42.8
93.3 53.8 49.7 25.4 77.3 50.0 65.6 BiSeNet2[19] 83.0 75.8 92.0 83.7 46.5 94.6
58.8 53.6 31.9 81.4 54.0 68.7 NDNet45-FCN8-LF[70] 85.5 84.6 94.8 82.6 39.2
97.4 60.1 37.3 17.6 86.8 53.7 57.5 LBN-AA[40] 83.2 70.5 92.5 81.7 51.6 93.0
55.6 53.2 36.3 82.1 47.9 68.0 AGLNet[64] 82.6 76.1 91.0 87.0 45.3 95.4 61.5
39.5 39.0 83.1 62.7 69.4 BiSeNetV2/BiSeNetV2L[28] - - - - - - - - - - -
72.4/73.2 S2-FPN18 83. 77.2 91.8 88.9 48.2 95.7 56.4 43.4 32.4 84.8 62.5 69.6
S2-FPN34 85.3 77.4 91.7 91.2 49.6 95.7 59.1 46.8 33.2 85.4 66.5 71.0 S2-FPN34M
86.0 78.8 92.6 92.2 56.2 96.0 67.1 47.3 42.1 86.8 70.7 74.2

TABLE VIII: RESULTS OF THE MODEL ON CAMVID DATASET.$*$ INDICATES THE MODELS
PRE-TRAINED ON CITYSCAPES

Method Year Params M Speed (FPS) mIoU% Deeplab[10] 2017 262.1 4.9 61.6 PSPNet
[5] 2017 250.8 5.4 69.1 SegNet [7] 2015 29.5 46 55.6 ENet [45] 2016 0.36 -
61.3 DFANet-A [27] 2019 7.8 120 64.7 DFANet-B [27] 2019 4.8 160 59.3 BiSeNet1
[19] 2018 5.8 175 65.7 BiSeNet2 [19] 2018 49.0 116.3 68.7 ICNet [50] 2018 26.5
27.8 67.1 DABNet [25] 2019 0.76 66.4 AGLNet [64] 2020 1.12 90.1 69.4
NDNet45-FCN8-LF [70] 2020 1.1 - 57.5 LBN-AA [40] 2020 6.2 39.3 68.0
BiSeNetV2/BiSeNetV2L [28] 2021 - - 72.4/73.2 STDC1-Seg75 [24] 2021 8.4 197.6
73.0 STDC2-Seg75 [24] 2021 12.5 152.2 73.9 S2-FPN18 17.8 124.2 69.5 S2-FPN34
27.9 107.2 71.0 S2-FPN34M 27.9 55.5 74.2

##  6 Conclusion

In this work, we designed architecture for semantic segmentation that achieves
a better trade-off accuracy/speed and presents Scale-aware Strip Attention
S2-FPN for scene parsing in real-time. First, we introduce Scale-aware Strip
Attention and channel attention modules to model the long-range dependencies.
Specifically, by using scale-aware and vertical strip operations, our network
drastically reduced the computation cost of the attention mechanism.
Furthermore, we propose the Attention Pyramid Fusion module, which enables and
facilitates acquiring contextual information by utilizing the attention
mechanism. The ablation studies show the effectiveness of scale-ware strip
attention and attention pyramid fusion modules. Experimental results
demonstrate that S2-FPN achieves accuracy/speed trade-offs on Camvid and
Cityscapes.

## References

  * [1] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [2] M. Saha and C. Chakraborty, “Her2net: A deep framework for semantic segmentation and classification of cell membranes and nuclei in breast cancer evaluation,” _IEEE Transactions on Image Processing_ , vol. 27, no. 5, pp. 2189–2200, 2018.
  * [3] M. Siam, M. Gamal, M. Abdel-Razek, S. Yogamani, M. Jagersand, and H. Zhang, “A comparative study of real-time semantic segmentation for autonomous driving,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 587–597.
  * [4] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia, “Psanet: Point-wise spatial attention network for scene parsing,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 267–283.
  * [5] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.
  * [6] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [7] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [8] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2018, pp. 3–11.
  * [9] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” _arXiv preprint arXiv:1511.07122_ , 2015.
  * [10] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [11] K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5693–5703.
  * [12] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [13] H. Li, P. Xiong, J. An, and L. Wang, “Pyramid attention network for semantic segmentation,” _arXiv preprint arXiv:1805.10180_ , 2018.
  * [14] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang, “Denseaspp for semantic segmentation in street scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3684–3692.
  * [15] G. Lin, A. Milan, C. Shen, and I. Reid, “Refinenet: Multi-path refinement networks for high-resolution semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1925–1934.
  * [16] M. A. Elhassan, Y. Chen, Y. Chen, C. Huang, J. Yang, X. Yao, C. Yang, and Y. Cheng, “Ppanet: Point-wise pyramid attention network for semantic segmentation,” _Wireless Communications and Mobile Computing_ , vol. 2021, 2021.
  * [17] W. Liu, A. Rabinovich, and A. C. Berg, “Parsenet: Looking wider to see better,” _arXiv preprint arXiv:1506.04579_ , 2015.
  * [18] M. A. Elhassan, C. Huang, C. Yang, and T. L. Munea, “Dsanet: Dilated spatial attention for real-time semantic segmentation in urban street scenes,” _Expert Systems with Applications_ , vol. 183, p. 115090, 2021.
  * [19] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, “Bisenet: Bilateral segmentation network for real-time semantic segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 325–341.
  * [20] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size,” _arXiv preprint arXiv:1602.07360_ , 2016.
  * [21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [22] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 116–131.
  * [23] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [24] M. Fan, S. Lai, J. Huang, X. Wei, Z. Chai, J. Luo, and X. Wei, “Rethinking bisenet for real-time semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9716–9725.
  * [25] G. Li, I. Yun, J. Kim, and J. Kim, “Dabnet: Depth-wise asymmetric bottleneck for real-time semantic segmentation,” _arXiv preprint arXiv:1907.11357_ , 2019.
  * [26] E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo, “Erfnet: Efficient residual factorized convnet for real-time semantic segmentation,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 1, pp. 263–272, 2017.
  * [27] H. Li, P. Xiong, H. Fan, and J. Sun, “Dfanet: Deep feature aggregation for real-time semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9522–9531.
  * [28] C. Yu, C. Gao, J. Wang, G. Yu, C. Shen, and N. Sang, “Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation,” _International Journal of Computer Vision_ , pp. 1–18, 2021.
  * [29] G. Ghiasi, T.-Y. Lin, and Q. V. Le, “Nas-fpn: Learning scalable feature pyramid architecture for object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7036–7045.
  * [30] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8759–8768.
  * [31] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 781–10 790.
  * [32] A. Kirillov, R. Girshick, K. He, and P. Dollár, “Panoptic feature pyramid networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6399–6408.
  * [33] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [34] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr, “Conditional random fields as recurrent neural networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1529–1537.
  * [35] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” _arXiv preprint arXiv:1412.7062_ , 2014.
  * [36] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions (2015),” _arXiv preprint arXiv:1511.07122_ , 2016.
  * [37] W. Chen, X. Gong, X. Liu, Q. Zhang, Y. Li, and Z. Wang, “Fasterseg: Searching for faster real-time semantic segmentation,” _arXiv preprint arXiv:1912.10917_ , 2019.
  * [38] P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, and F. Perazzi, “Temporally distributed networks for fast video semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8818–8827.
  * [39] P. Hu, F. Perazzi, F. C. Heilbron, O. Wang, Z. Lin, K. Saenko, and S. Sclaroff, “Real-time semantic segmentation with fast attention,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 1, pp. 263–270, 2020.
  * [40] G. Dong, Y. Yan, C. Shen, and H. Wang, “Real-time high-performance semantic image segmentation of urban street scenes,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 6, pp. 3258–3274, 2020.
  * [41] P. Li, X. Dong, X. Yu, and Y. Yang, “When humans meet machines: Towards efficient segmentation networks.” in _BMVC_ , 2020.
  * [42] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [43] X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6848–6856.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [45] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, “Enet: A deep neural network architecture for real-time semantic segmentation,” _arXiv preprint arXiv:1606.02147_ , 2016.
  * [46] S. Mehta, M. Rastegari, A. Caspi, L. Shapiro, and H. Hajishirzi, “Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation,” in _Proceedings of the european conference on computer vision (ECCV)_ , 2018, pp. 552–568.
  * [47] S. Mehta, M. Rastegari, L. Shapiro, and H. Hajishirzi, “Espnetv2: A light-weight, power efficient, and general purpose convolutional neural network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9190–9200.
  * [48] Y. Wang, Q. Zhou, J. Liu, J. Xiong, G. Gao, X. Wu, and L. J. Latecki, “Lednet: A lightweight encoder-decoder network for real-time semantic segmentation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1860–1864.
  * [49] J. Liu, Q. Zhou, Y. Qiang, B. Kang, X. Wu, and B. Zheng, “Fddwnet: a lightweight convolutional neural network for real-time semantic segmentation,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 2373–2377.
  * [50] H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia, “Icnet for real-time semantic segmentation on high-resolution images,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 405–420.
  * [51] R. P. Poudel, S. Liwicki, and R. Cipolla, “Fast-scnn: Fast semantic segmentation network,” _arXiv preprint arXiv:1902.04502_ , 2019.
  * [52] J. Gehring, M. Auli, D. Grangier, and Y. N. Dauphin, “A convolutional encoder model for neural machine translation,” _arXiv preprint arXiv:1611.02344_ , 2016.
  * [53] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin, “Convolutional sequence to sequence learning,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1243–1252.
  * [54] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3156–3164.
  * [55] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European Conference on Computer Vision_. Springer, 2020, pp. 213–229.
  * [56] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” _arXiv preprint arXiv:2010.04159_ , 2020.
  * [57] L. Ye, M. Rochan, Z. Liu, and Y. Wang, “Cross-modal self-attention network for referring image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 502–10 511.
  * [58] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [59] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3146–3154.
  * [60] Y. Yuan, L. Huang, J. Guo, C. Zhang, X. Chen, and J. Wang, “Ocnet: Object context network for scene parsing,” _arXiv preprint arXiv:1809.00916_ , 2018\.
  * [61] M. A. Elhassan, C. Yang, C. Huang, and T. L. Munea, “Spfnet: Subspace pyramid fusion network for semantic segmentation,” _arXiv preprint arXiv:2204.01278_ , 2022.
  * [62] C. Guo, B. Fan, Q. Zhang, S. Xiang, and C. Pan, “Augfpn: Improving multi-scale feature learning for object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 595–12 604.
  * [63] Q. Hou, L. Zhang, M.-M. Cheng, and J. Feng, “Strip pooling: Rethinking spatial pooling for scene parsing,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4003–4012.
  * [64] Q. Zhou, Y. Wang, Y. Fan, X. Wu, S. Zhang, B. Kang, and L. J. Latecki, “Aglnet: Towards real-time semantic segmentation of self-driving images via attention-guided lightweight network,” _Applied Soft Computing_ , vol. 96, p. 106682, 2020.
  * [65] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [66] G. J. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes in video: A high-definition ground truth database,” _Pattern Recognition Letters_ , vol. 30, no. 2, pp. 88–97, 2009.
  * [67] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [68] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [69] S. Rota Bulò, L. Porzi, and P. Kontschieder, “In-place activated batchnorm for memory-optimized training of dnns,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [70] Z. Yang, H. Yu, M. Feng, W. Sun, X. Lin, M. Sun, Z.-H. Mao, and A. Mian, “Small object augmentation of urban scenes for real-time semantic segmentation,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 5175–5190, 2020.