11institutetext:  School of Electrical Engineering and Computer Science
Gwangju Institute of Science and Technology, South Korea
11email: (farzeen.munir, shoaibazam<EMAIL_ADDRESS><EMAIL_ADDRESS>
# ARTSeg: Employing Attention for Thermal images Semantic Segmentation

Farzeen Munir  Shoaib Azam  Unse Fatima  Moongu Jeon

###### Abstract

The research advancements have made the neural network algorithms deployed in
the autonomous vehicle to perceive the surrounding. The standard exteroceptive
sensors that are utilized for the perception of the environment are cameras
and Lidar. Therefore, the neural network algorithms developed using these
exteroceptive sensors have provided the necessary solution for the autonomous
vehicle’s perception. One major drawback of these exteroceptive sensors is
their operability in adverse weather conditions, for instance, low
illumination and night conditions. The useability and affordability of thermal
cameras in the sensor suite of the autonomous vehicle provide the necessary
improvement in the autonomous vehicle’s perception in adverse weather
conditions. The semantics of the environment benefits the robust perception,
which can be achieved by segmenting different objects in the scene. In this
work, we have employed the thermal camera for semantic segmentation. We have
designed an attention-based Recurrent Convolution Network (RCNN) encoder-
decoder architecture named ARTSeg for thermal semantic segmentation. The main
contribution of this work is the design of encoder-decoder architecture, which
employ units of RCNN for each encoder and decoder block. Furthermore, additive
attention is employed in the decoder module to retain high-resolution features
and improve the localization of features. The efficacy of the proposed method
is evaluated on the available public dataset, showing better performance with
other state-of-the-art methods in mean intersection over union (IoU).

###### Keywords:

Thermal Image Semantic segmentation Recurrent Convolution Neural Network
Attention .

##  1 Introduction

The last three decades have shown significant technological advancements that
reflects the development of efficient sensors. The applicability of these
sensors in the autonomous vehicle ensures its safe and secure services to the
urban environment, as indicated by SOTIF-
ISO/PAS-21448111https://www.daimler.com/innovation/case/autonomous/safety-
first-for-automated-driving-2.htm. Keeping safety as a priority in autonomous
vehicles, perception of the environment play a critical role along with
localization, planning and control module [1] [2] [3]. In the context of
perception, the most common exteroceptive sensors that are being deployed are
cameras (visible spectrum) and Lidar. The utilization of these sensor
modalities provides the necessary perception for the autonomous vehicle.
However, these sensors have limitations in adverse weather conditions at night
and low illumination environments. For instance, cameras (visible spectrum)
are operated in the visible spectrum domain and provide an in-depth
understanding of the environment. Still, environmental conditions like sun-
glare, low illumination affect the camera result, thus yielding low
performance in perception algorithms. On the other hand, as a surrogate to
$2$D information from cameras, Lidar gives the $3$D information about the
environment. Lidar provides 3D information of the environment by making a
point cloud map by projecting nearly thousands of laser beams to the
environment. Besides, Lidar effectiveness in providing the detailed $3$D
representation of the environment, its expensive cost and limitation in
adverse weather conditions are the prime concern of its detriment. On the
other hand, thermal cameras, contrary to cameras (visible-spectrum) and Lidar,
enable the perception algorithms to be utilized in adverse weather conditions,
such as at night or low illumination environmental conditions [4].

Thermal cameras operate in the infrared domain and capture the infrared
radiation emitted by the different entities in the environment having the
temperature above absolute zero [6]. This property of the thermal camera makes
it an optimal solution to be included in the sensor suite of the autonomous
vehicle for the perception of the environment in low illumination and night
conditions. Furthermore, besides thermal cameras applicability in adverse
weather conditions, the affordability of thermal cameras gives the potential
to be utilized in different perception tasks for the autonomous vehicle. These
perception tasks involve different computer vision applications such as object
detection [7] [8], visual tracking [9] and person re-identification [10]. In
order to determine the semantics of environment for the scene understanding
for the autonomous vehicle, semantic segmentation plays a vital role in the
perception of the autonomous vehicle. The capacity of thermal cameras to
operate at night and low illumination conditions motivate to use of the
thermal camera for semantic segmentation and investigate the semantic
segmentation problem using the thermal cameras.

In computer vision, semantic segmentation provides an in-depth understanding
of the scene by employing a semantic label to each pixel in the image. Despite
traditional machine learning approaches used to tackle semantic segmentation
problems, deep learning approaches gain unprecedented success. For instance,
most of these deep learning techniques (convolutional neural networks) are
applied using the RGB images for semantic segmentation. Thermal cameras
provide the grey-scale image that benefits the neural networks employed on
thermal images for different computer vision tasks.

This work explores the thermal image segmentation problem by designing a novel
encoder-decoder architecture called ARTSeg. The encoder consists of a residual
recurrent convolution block accompanied by a max-pooling layer. The encoder
outputs a latent representation of features which is fed to the decoder. The
decoder incorporates additive attention from the residual connections from the
encoder. The ARTSeg is evaluated on a public dataset and compared with state-
of-the-art methods. The main contributions of this work are as follows:

  1. 1.

We have designed a novel encoder-decoder architecture using a residual
recurrent convolution neural network.

  2. 2.

We have employed the additive attention mechanism to enhance the localization
of encoded features. Further, no post-processing steps are used because of
utilizing the attention for the thermal image segmentation.

  3. 3.

The proposed method’s efficacy is performed on available public datasets shows
better performance in terms of average accuracy and intersection over the
union in contrast to state-of-the-art methods.

The rest of the paper is organized as follows: Section 2 explains the related
work. The proposed method is described in the Section 3. Section 4 illustrate
the experimentation and results for the proposed method. Finally, section 5
concludes the paper.

##  2 Related Work

The concept of semantic segmentation plays a critical role in the robust
perception of the environment for autonomous driving. Deep neural networks,
especially convolution neural networks, are mostly used for semantic
segmentation using RGB cameras as a sensor modality. The early state-of-the-
art method that is employed for the semantic segmentation is Fully
Convolutional Network (FCN) proposed by [5]. In FCN, the fully connected
layers are replaced with the convolutional layers to output the full
resolution maps for the semantic segmentation using the backbone network of
VGG16 [23], and GoogleNet [11] architectures, respectively.

In literature, the encoder-decoder architecture is also used for semantic
segmentation. In the encoder-decoder architecture, the features are encoded to
give the latent representation and then decoded to provide the object details
and spatial resolution. [22] have proposed the encoder-decoder architecture
named SegNet. The encoder network consists of convolutional layers adopted
from the VGG16 [23] network to learn the encoded representation followed by
batch-normalization, rectified linear unit (ReLU) units and max-pooling
layers. The decoder follows the same encoder architecture for upsampling the
encoded features for semantic segmentation. Similarly, using the encoder-
decoder architecture, the encoder of the network is improved by using image
pyramids [28][29], conditional random fields [28], spatial pyramid pooling
[28] and atrous convolution [30] [31]. There is a trade-off between accuracy
and speed; in literature, some research is focused on improving the inference
speed using the encoder-decoder architecture; for instance, [13] have proposed
the encoder-decoder architecture ENet that is optimized for the fast inference
speed for semantic segmentation. In order to retain the feature
generalization, skip connections have been introduced for the semantic
segmentation as proposed by UNet [12].

Besides, using the single sensor modality, for instance, RGB images, in
literature, the fusion of thermal and visible-spectrum RGB domain is also
investigated for semantic segmentation. [15] have proposed FuseNet for the
semantic segmentation by incorporating the fusion of visible-spectrum RGB
domain with the thermal domain. Similarly, MFNet [19] and RTFNet [21] networks
follows the encoder-decoder architecture for the semantic segmentation using
the fusion of RGB and thermal data. [20] have proposed the FuseSeg using the
Bayesian fusion theory for the semantic segmentation using both RGB and
thermal data. In addition to the fusion techniques, some research is focused
on multi-spectral domain adaptation [32] [33] [34].

In contrast to the literature review, we have designed a novel encoder-decoder
architecture using the residual recurrent convolution layers followed by an
attention mechanism to retain the feature generalization for semantic
segmentation. Furthermore, we have explicitly used the thermal data for the
semantic segmentation without employing the fusion or domain adaption
techniques in this work.

Figure 1:  The overall framework of ARTSeg Network consists of an encoder-
decoder architecture.  Table 1: The detailed architecture of the proposed
ARTSeg

| Layer | Output size
---|---|---
| Input data |
Encoder | RRCNN-1 block | 32*256*256
Max pooling | 32*128*128
RRCNN-2 block | 64*128*128
Max pooling | 64*64*64
RRCNN-3 block | 128*64*64
Max pooling | 128*32*32
RRCNN-4 block | 256*32*32
Max pooling | 256*8*8
| RCNN-5 block | 256*8*8
Decoder | Up-Block | 128*32*32
Attention Module | 256*32*32
RRCNN-4 block | 128*32*32
Up-Block | 128*64*64
Attention Module | 256*64*64
RRCNN-3 block | 128*64*64
Up-Block | 64*128*128
Attention Module | 128*128*128
RRCNN-2 block | 64*128*128
| Up-Block | 32*256*256
| Attention Module | 64*256*256
| RRCNN-1 Block | 32*256*256
Output | Conv 1X1 | 5*256*256

##  3 Methods

This section presents the proposed method for the thermal image segmentation,
as illustrated in Fig.1. The proposed method is composed of encoder-decoder
architecture. The encoder module of the proposed method contains the residual
recurrent convolution block (RRCNN) followed by the max-pooling layer. The
residual recurrent convolutional block is comprised of recurrent convolutional
layers and skip connections from the encoder to decoder module. The encoder
module of the architecture receives the thermal image as input and generates
the encoded feature representation for the input image. Suppose $x_{l}$
represent the input image at the $l^{th}$ layer of the residual recurrent
convolutional block, and a pixel is located at $(u,v)$ at the input image on
the $p$ feature map of the recurrent convolution layer, then the output
$o_{uvp}$ is expressed in Eq.1.

$o_{uvp}^{l}=(w_{p}^{f})^{T}*x_{l}^{f}(u,v)(t)+(w_{p}^{r})^{T}*x_{l}^{r}(u,v)(t-1)+b_{p},$
(1)

where $w_{p}^{f}$ and $w_{p}^{r}$ are the weights of convolutional and
recurrent convolutional layer respectively and $b_{p}$ is the bias of the
network. The inputs to the convolution and $l^{th}$ layer recurrent
convolutional layer are represented as $x_{l}^{f}(u,v)(t)$ and
$x_{l}^{r}(u,v)(t-1)$ respectively. In Fig.1 the recurrent convolution layer
are represented in the RRCNN block with the “arrow” indicating the recurrence.
The output of the recurrent convolutional layer is batch-normalized, and fed
to the rectified linear activation function (ReLU) as shown in Eq.2.

$\mathfrak{F}(\mathbf{x_{l}},\boldsymbol{w_{l}})=f(o_{uvp}^{l}(t)))=max(0,o_{uvp}^{l}(t)),$
(2)

$\mathfrak{F}(\mathbf{x_{l}},\boldsymbol{w_{l}})$ represents the outcome of
the activation function from the $l^{th}$ layer recurrent convolutional
network. This is summed with the input $x_{l}$ in residual to give us the
output of RRCNN block as expressed as Eq.3.

$x_{l+1}=x_{l}+\mathfrak{F}(\mathbf{x_{l}},\boldsymbol{w_{l}}),$ (3)

where $x_{l+1}$ represent the outcome from the RRCNN block.

The decoder module follows the same architecture of the encoder module with
the addition of Upsample block and additive attention in the skip connections.
The decoder includes the same RRCNN block as the encoder module, but the RRCNN
module is used to upsampling the encoded features. Fig.1 shows the decoder
module with attention block. The output of the attention is calculated by the
element-wise multiplication of attention coefficient and input feature maps
from as shown in Eq.4.

$x_{l+1}^{\prime}=x_{l+1}\cdot\alpha,$ (4)

The purpose of the attention module is to focus on the salient region for the
thermal image segmentation. Mathematically, the attention of the network is
given by Eq.5. The term g represents the vector taken from the lowest layer of
the network. The detailed architecture is presented in Table-1.

$\displaystyle
s^{l}_{att}=\phi^{T}(\sigma(W^{T}_{x}x^{i}_{l+1}+W^{T}_{g}g^{i}+b_{g}))+b_{\phi},$
(5)
$\displaystyle\alpha^{l}=\varphi(s^{l}_{att}(x^{i}_{l+1},g^{i};\Theta_{att})),$

where $\sigma$ and $\varphi$ represent ReLU and sigmoid activation function
respectively. The $W_{x}$ and $W_{g}$ shows the linear transformation for the
attention network. In addition, the Upsample block includes an upsampling
layer followed by convolutional layer, batch normalization and the ReLU
activation unit. The semantic segmentation output is obtained by placing the
final convolution layer of kernel size $1\times 1$ at the final RRCNN block in
the decoder layer.

Table 2: The number of images in training and testing split of the dataset.

Dataset | Training set | Testing set
---|---|---
Day & Night time | 784 | 393
Day Time |  - | 205
Night Time |  - | 188

##  4 Experimentation and Results

This section explains the experimentation and results of the proposed method.
The performance of the proposed method is evaluated on the available public
dataset[19]. The efficacy of the proposed method is compared with the state-
of-the-art methods for thermal image semantic segmentation. For the
evaluation, we have adopted the standard metrics: Average class accuracy and
mean Intersection over Union (IoU). The following section discusses the
details of experimentation and results.

###  4.1 Thermal Semantic Segmentation Dataset

We have utilized the available public dataset for thermal image segmentation
provided by [19]. The dataset is comprised of thermal and visible-spectrum RGB
images. The dataset is collected using the InfRec R500 thermal camera,
simultaneously capturing the thermal spectrum and visible spectrum images. The
range of the thermal spectrum for collecting the images is $814\mu m$, giving
the images of resolution $480\times 640$. However, the field of view of the
visible-spectrum image and the thermal image is not identical. The visible
spectrum images have a horizontal field of view of $100^{\circ}$; besides, the
thermal spectrum images have a horizontal field of view of $32^{\circ}$. In
order to align the thermal and RGB spectrum images, the RGB images are cropped
and resized. The dataset provides the semantic labels for the urban
environment and is classified into nine classes: bicycle, person, car, curve,
car_stop, color_cone, guardrail, and background. The dataset has 1569 images
pairs of RGB and thermal altogether. There are 820 image pairs recorded during
the daytime and 749 recorded at nighttime. The dataset is split into the
training set and testing set. The testing set is further split into the day
and nighttime groups. The number of images in each group is shown in Table 2.

###  4.2 Training details

The proposed method ArtSeg is implemented using Pytorch deep learning library.
In training the proposed method, no pre-processing step is performed on the
input images. The proposed network ArtSeg is trained from scratch in an end-
to-end manner and does not employ any pre-trained weights. The network is
trained for a total of 500 epochs on Nvidia RTX $3090$ having $24GB$ memory.
The cross-entropy loss function is used for training the network given by
Eq.6.

$\displaystyle
Loss=-\frac{1}{M}\sum_{j=1}^{M}\sum_{c=1}^{C}S_{c,j}\ln(\hat{S}_{c,j})$ (6)

where $M$ is the total number of pixels in the ground truth label image, $C$
is the number of classes, in this case, $9$. $S_{c,j}$ denotes the ground
truth class label of each pixel, and $\hat{S}_{c,j}$ denotes the predicted
class of each pixel. We have used Adam optimizer for training the ARTSeg
network with the weight decay of $1\times 10^{-4}$; epsilon of $1\times
10^{-8}$ and learning rate of $5\times 10^{-4}$. The learning rate schedule
policy given by the Eq.7 is used to update the learning rate.

$LR=LR_{initial}\times(\frac{1-epoch}{epochs_{total}})^{p},$ (7)

The $p$ is set to $0.9$. The network is trained for 100 epochs. All the
parameter values are chosen empirically through grid search. These parameters
are kept constant in all the experiments. Moreover, training data is augmented
using the flip technique.

Table 3:  Comparison of the evaluation results of ARTSeg with other state-of-
the-art methods on the test set consisting of Day and Night time images. The
Class Accuracy (%), Avg.Acc (%) and $IOU$ (%) are the evaluation metrics. ”-”
indicates that the value is not available.

Model | Background | Car | Pedestrian | Bike | Curve | Car stop | Guardrail | Color cone | Bump | Avg.Acc | $IOU$
---|---|---|---|---|---|---|---|---|---|---|---
SegNet [19] | 96.90 | 83.30 | 72.10 | 76.80 | 58.30 | 31.90 | 0.00 | 0.00 | 63.90 | 53.70 | 58.30
SegNet_4ch [19] | 96.10 | 89.00 | 82.30 | 0.00 | 61.40 | 21.70 | 0.00 | 0.00 | 86.70 | 48.60 | 50.40
ENet [19] | 88.50 | 58.60 | 42.70 | 24.70 | 30.10 | 18.10 | 0.30 | 45.80 | 23.00 | 37.00 | 44.90
MFNet [19] | 96.80 | 82.90 | 85.20 | 74.20 | 61.50 | 27.30 | 0.00 | 60.70 | 43.30 | 59.10 | 64.90
FuseSeg [20] | - | 93.10 | 81.40 | 78.50 | 68.40 | 29.10 | 63.70 | 55.80 | 66.64 | 70.60 | 54.50
RTFNet [21] | - | 93.00 | 79.30 | 76.80 | 60.70 | 38.50 | 0.00 | 45.50 | 74.70 | 63.10 | 53.20
ARTSeg (Our) | 97.10 | 94.76 | 86.66 | 79.20 | 71.25 | 49.69 | 65.21 | 58.11 | 64.28 | 74.03 | 68.80

###  4.3 Evaluation metrics

In order to evaluate the performance of the proposed network, two evaluation
metrics are selected. The Average class accuracy and mean Intersection over
Union (IoU). The Average class accuracy is expressed as

$Avg.Acc=\frac{1}{N}\sum_{i=1}^{N}\frac{TP_{i}}{TP_{i}+FN_{i}},\\\ $ (8)

where $N$ is number of classes.$TP$ is true positive rate
($TP_{i}=\sum_{m=1}^{M}p_{ii}^{m}$) and $FN$ is false negative
rate(${FN}_{i}=\sum_{m=1}^{M}\sum_{j=1,j\neq i}^{N}p_{ij}^{m}$). The $IoU$ is
expressed as

$IoU=\frac{1}{N}\sum_{i=1}^{N}\frac{TP_{i}}{TP_{i}+FN_{i}+FP_{i}},\\\ $ (9)

here $FP$ is define as false positive
rate(${FP}_{i}=\sum_{m=1}^{M}\sum_{j=1,j\neq i}^{N}p_{ji}^{m}$). The $p_{ii}$
corresponds to the correct number of pixels classified for class $i$ having
the same class $i$ in the frame $m$, whereas $p_{ji}$ represents the number
incorrectly classified pixel of class $j$ as class $i$ in the frame $m$.
Similarly, the number of incorrectly classified pixel of class $i$ as class
$j$ is represented by $p_{ij}$ in the frame $m$.

###  4.4 Results

Figure 2:  The qualitative comparison between ARTSeg and other state-of-the-
art methods for semantic segmentation on thermal image dataset.  Table 4:
Quantitative analysis of ARTSeg with other state-of-the-art methods for day
time and night time test set. The evaluation is performed using Avg.acc (%)
and $IoU$ (%) metrics.

Methods | Day Time | Night Time
---|---|---
| Avg.Acc(%) |  $IOU$(%) | Avg.Acc(%) |  $IOU$(%)
SegNet [19] | 46.10 | 48.80 | 54.00 | 55.20
SegNet_4ch [19] | 50.50 | 51.10 | 50.50 | 51.10
MFNet [19] | 47.70 | 57.40 | 63.50 | 62.10
FuseSeg [20] | 62.10 | 47.80 | 67.30 | 54.60
RTFNet[21] | 60.00 | 45.80 | 60.70 | 54.80
ARTSeg (Our) | 65.58 | 64.08 | 70.85 | 65.56

The thermal image semantic segmentation dataset is benchmarked on existing
techniques, including MFRNet [19], FuseSeg[20], SegNet[22] and RTFNet[21].
Image segmentation using thermal images is an emerging research direction in
contrast to RGB image segmentation; given this, a few techniques have been
employed in the literature for thermal image segmentation. The MFRNet, FuseSeg
and RTFNet have fused the information from the visible spectrum and thermal
spectrum images. However, SegNet uses only thermal images for predicting
semantic segmentation. The proposed network ARTSeg is trained on only thermal
images. The evaluation of ARTSeg in comparison to the algorithms as mentioned
earlier is shown in Table 3. The Table shows the Accuracy measure of each
class and Avg.Acc and $IoU$ over all classes for both day and night time.
However, Table 4 shows the evaluation of the methods for daytime and nighttime
separately for all classes. ARTSeg has outperformed all existing methods and
achieved an improvement $3.43\%$ on Avg.Acc and $3.9\%$ in $IoU$ scores in
comparison to FuseSeg. Fig.2 manifests the qualitative results of the proposed
ARTSeg algorithm with other methods. In addition, we have also investigated
the effect of using different backbone networks to extract the features for
the encoder of the ARTSeg network.

Table 5: Quantitive analysis of ARTSeg with different backbone networks. The
evaluation is performed using Avg.acc (%) and $IOU$ (%) metrics on day and
night time test set.

Methods | DayTime | NightTime
---|---|---
| Avg.Acc(%) |  $IOU$ (%) | Avg.Acc(%) |  $IOU$(%)
ARTSeg-VGG16 | 54.21 | 46.80 | 59.12 | 48.20
ARTSeg-ResNet-18 | 57.50 | 47.60 | 61.89 | 51.10
ARTSeg-ResNet-50 | 63.25 | 55.40 | 65.50 | 57.23
ARTSeg-MobileNetv2 | 59.45 | 49.89 | 60.10 | 50.80
ARTSeg-DenseNet | 58.41 | 49.50 | 59.45 | 49.34
ARTSeg-ShuffleNet | 62.58 | 58.18 | 66.85 | 59.56
ARTSeg (Our) | 65.58 | 64.08 | 70.85 | 65.56

We utilize six different backbone network including VGG16 [23], ResNet-18
[24], ResNet-50 [24], MobileNetV2 [25], ShuffleNet [26] and DenseNet [27]. The
thermal image is input to the backbone network. The encoded features are then
passed to the decoder module. The backbone networks are trained with pre-
trained weights. The Table 5 shows the quantitative results with the backbone
networks. The results using different backbone networks in the encoder module
do not show any significant improvement compared to ARTSeg.

##  5 Conclusion

In this research article, we proposed ARTSeg, a novel encoder-decoder
architecture for thermal image semantic segmentation. The ARTSeg introduces
residual recurrent convolution block in the encoder, followed by a decoder
that utilizes additive attention in skip connection to refine full resolution
detection. The ARTSeg is evaluated on the public dataset in terms of Avg.Acc
and $IoU$. In comparison to other state-of-the-art methods, ARTSeg has a
higher Avg.Acc ($74.03\%$) and $IoU$ ($68.80\%$).

The inclusion of the thermal camera in the autonomous driving stack provide
help to understand the surroundings in low illumination conditions. In future
work, we aim to fuse information from the thermal camera and Lidar to improve
the autonomous vehicle’s perception.

## Acknowledgement

This work was partly supported by the ICT R$\&$D program of MSIP/IITP
(2014-3-00077, Development of global multitarget tracking and event prediction
techniques based on real-time large-scale video analysis), National Research
Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) (No.
2019R1A2C2087489), Ministry of Culture, Sports and Tourism (MCST), and Korea
Creative Content Agency (KOCCA) in the Culture Technology (CT) Research &
Development (R2020070004) Program 2021.

## References

  * [1] Azam, S., Munir, F., Sheri, A.M., Kim, J. and Jeon, M., 2020. System, design and experimental validation of autonomous vehicle in an unconstrained environment. Sensors, 20(21), p.5999.
  * [2] Munir, F., Azam, S., Hussain, M.I., Sheri, A.M. and Jeon, M., 2018, October. Autonomous vehicle: The architecture aspect of self driving car. In Proceedings of the 2018 International Conference on Sensors, Signal and Image Processing (pp. 1-5).
  * [3] Azam, S., Munir, F. and Jeon, M., 2020. Dynamic Control System Design for Autonomous Car. In VEHITS (pp. 456-463).
  * [4] Rosique, F., Navarro, P.J., Fernández, C. and Padilla, A., 2019. A systematic review of perception system and simulators for autonomous vehicles research. Sensors, 19(3), p.648.
  * [5] Long, Jonathan, Evan Shelhamer, and Trevor Darrell. ”Fully convolutional networks for semantic segmentation.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440. 2015.
  * [6] Vollmer, M. and Möllmann, K.P., 2017. Infrared thermal imaging: fundamentals, research and applications. John Wiley & Sons.
  * [7] Hwang, S., Park, J., Kim, N., Choi, Y. and So Kweon, I., 2015. Multispectral pedestrian detection: Benchmark dataset and baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1037-1045).
  * [8] Xu, D., Ouyang, W., Ricci, E., Wang, X. and Sebe, N., 2017. Learning cross-modal deep representations for robust pedestrian detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5363-5371).
  * [9] Li, C., Zhu, C., Huang, Y., Tang, J. and Wang, L., 2018. Cross-modal ranking with soft consistency and noisy labels for robust rgb-t tracking. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 808-823).
  * [10] Wu, A., Zheng, W.S., Yu, H.X., Gong, S. and Lai, J., 2017. RGB-infrared cross-modality person re-identification. In Proceedings of the IEEE international conference on computer vision (pp. 5380-5389).
  * [11] Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. ”Going deeper with convolutions.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9. 2015.
  * [12] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. ”U-net: Convolutional networks for biomedical image segmentation.” In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, Cham, 2015.
  * [13] Paszke, Adam, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. ”Enet: A deep neural network architecture for real-time semantic segmentation.” arXiv preprint arXiv:1606.02147 (2016).
  * [14] Song, Shuran, Samuel P. Lichtenberg, and Jianxiong Xiao. ”Sun rgb-d: A rgb-d scene understanding benchmark suite.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576. 2015.
  * [15] Hazirbas, Caner, Lingni Ma, Csaba Domokos, and Daniel Cremers. ”Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture.” In Asian conference on computer vision, pp. 213-228. Springer, Cham, 2016.
  * [16] Yu, Changqian, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. ”Learning a discriminative feature network for semantic segmentation.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1857-1866. 2018.
  * [17] Qiao, Yulong, Ziwei Wei, and Yan Zhao. ”Thermal infrared pedestrian image segmentation using level set method.” Sensors 17, no. 8 (2017): 1811.
  * [18] Li, Chenglong, Wei Xia, Yan Yan, Bin Luo, and Jin Tang. ”Segmenting objects in day and night: Edge-conditioned cnn for thermal image semantic segmentation.” IEEE Transactions on Neural Networks and Learning Systems (2020).
  * [19] Ha, Qishen, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. ”MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes.” In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5108-5115. IEEE, 2017.
  * [20] Sun, Yuxiang, Weixun Zuo, Peng Yun, Hengli Wang, and Ming Liu. ”FuseSeg: semantic segmentation of urban scenes based on RGB and thermal data fusion.” IEEE Transactions on Automation Science and Engineering (2020).
  * [21] Sun, Yuxiang, Weixun Zuo, and Ming Liu. ”Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes.” IEEE Robotics and Automation Letters 4, no. 3 (2019): 2576-2583.
  * [22] Badrinarayanan, Vijay, Alex Kendall, and Roberto Cipolla. ”Segnet: A deep convolutional encoder-decoder architecture for image segmentation.” IEEE transactions on pattern analysis and machine intelligence 39, no. 12 (2017): 2481-2495.
  * [23] Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.“ arXiv preprint arXiv:1409.1556 (2014).
  * [24] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep residual learning for image recognition.“ In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
  * [25] Sandler, Mark, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. “Mobilenetv2: Inverted residuals and linear bottlenecks.“ In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520. 2018.
  * [26] Ma, Ningning, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. “Shufflenet v2: Practical guidelines for efficient cnn architecture design.“ In Proceedings of the European conference on computer vision (ECCV), pp. 116-131. 2018.
  * [27] Huang, Gao, Zhuang Liu, and Kilian Q. Weinberger. “Densely connected convolutional networks. CoRR abs/1608.06993 (2016).“ arXiv preprint arXiv:1608.06993 (2016).
  * [28] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A.L., 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4), pp.834-848.
  * [29] Chen, L.C., Yang, Y., Wang, J., Xu, W. and Yuille, A.L., 2016. Attention to scale: Scale-aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3640-3649).
  * [30] Chen, L.C., Papandreou, G., Schroff, F. and Adam, H., 2017. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.
  * [31] Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X. and Cottrell, G., 2018, March. Understanding convolution for semantic segmentation. In 2018 IEEE winter conference on applications of computer vision (WACV) (pp. 1451-1460). IEEE.
  * [32] Kim, Y.H., Shin, U., Park, J. and Kweon, I.S., 2021. MS-UDA: Multi-Spectral Unsupervised Domain Adaptation for Thermal Image Semantic Segmentation. IEEE Robotics and Automation Letters.
  * [33] Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H. and Chandraker, M., 2018. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7472-7481).
  * [34] Kim, M. and Byun, H., 2020. Learning texture invariant representation for domain adaptation of semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12975-12984).