# DBAT: Dynamic Backward Attention Transformer for Material Segmentation with
Cross-Resolution Patches

Yuwen Heng, Srinandan Dasmahapatra, Hansung Kim
School of Electronics and Computer Science
University of Southampton, UK

###### Abstract

The objective of dense material segmentation is to identify the material
categories for every image pixel. Recent studies adopt image patches to
extract material features. Although the trained networks can improve the
segmentation performance, their methods choose a fixed patch resolution which
fails to take into account the variation in pixel area covered by each
material. In this paper, we propose the Dynamic Backward Attention Transformer
(DBAT) to aggregate cross-resolution features. The DBAT takes cropped image
patches as input and gradually increases the patch resolution by merging
adjacent patches at each transformer stage, instead of fixing the patch
resolution during training. We explicitly gather the intermediate features
extracted from cross-resolution patches and merge them dynamically with
predicted attention masks. Experiments show that our DBAT achieves an accuracy
of 86.85%, which is the best performance among state-of-the-art real-time
models. Like other successful deep learning solutions with complex
architectures, the DBAT also suffers from lack of interpretability. To address
this problem, this paper examines the properties that the DBAT makes use of.
By analysing the cross-resolution features and the attention weights, this
paper interprets how the DBAT learns from image patches. We further align
features to semantic labels, performing network dissection, to infer that the
proposed model can extract material-related features better than other
methods. We show that the DBAT model is more robust to network initialisation,
and yields fewer variable predictions compared to other models. The project
code is available at https://github.com/heng-yuwen/Dynamic-Backward-Attention-
Transformer.

###### Index Terms:

Material segmentation, image processing, scene understanding, neural networks,
network interpretability.

††publicationid: pubid: This paper is under consideration at Computer Vision
and Image Understanding

##  I Introduction

The dense material segmentation task aims to recognise the physical material
category (_e.g._ metal, plastic, stone, etc.) of each pixel in the input
image. The predicted material labels are instrumental in various applications
such as autonomous robots or entertainment systems. For example, the material
information is essential in robot manipulation [1, 2] and is also used for
immersive sound rendering by environment analysis [3, 4, 5, 6, 7]. Since the
appearance such as shape, colour and transparency of a specific material can
vary, identifying the materials from general RGB images remains a challenging
task [8, 9, 10]. In order to improve predictive accuracy, recent material
segmentation methods propose to combine both material and contextual features
[11, 8, 12, 13, 9]. Material characteristics such as texture and roughness
allow models to identify material categories without knowing all their varied
appearances. Contextual features including objects and scenes can limit the
possible categories of materials in the image.

However, existing methods have not thoroughly investigated different
strategies to combine the material and contextual features. The network
proposed by Schwartz _et al._[8, 12, 11] extracts these features with
independent branches and concatenates them to make the material decisions. The
material features are extracted from cropped image patches and contextual
features, including objects and scenes, are gathered from multiple pre-trained
branches targeting related tasks, such as object and scene classification. In
our preliminary work [9], we extended this multi-branch architecture with a
self-training strategy [14, 15] to boundary features between adjacent material
regions [16]. We found that the material features generalise better when
trained with a medium-sized dataset. However, the image patch resolution is
fixed, and this may not be optimal for extracting material features due to the
objects being at different distances to the camera. Ideally, small-resolution
patches should be applied to separate adjacent material regions, while large-
resolution patches can be used to cover as large a single piece of material as
possible to provide sufficient material information.

In our recent conference publication [17], we developed a cross-resolution
transformer architecture called DBAT. The DBAT extracts material features from
multi-resolution image patches rather than searching for an optimal patch
resolution. Initially, images are propagated through a transformer backbone
that processes 4x4 patches and gradually increases the patch size by merging
adjacent patches. Subsequently, we introduce a Dynamic Backward Attention
(DBA) module that aggregates the intermediate features extracted from cross-
resolution image patches. The proposed DBAT is evaluated by comparing its
performance on two material segmentation datasets, the Local Material Database
(LMD) and OpenSurfaces, with real-time state-of-the-art (SOTA) networks. The
experiments reveal that the DBAT achieves the highest accuracy and narrowest
uncertainty bounds. Notably, when trained with learning rate warm-up [18] and
polynomial decay [19], the average pixel accuracy (Pixel Acc) of the DBAT
reaches 86.85% when assessed on the LMD. This Pixel Acc represents a 21.21%
increase compared to the most recent publication [9] and outperforms the
second-best model ***Among the models that can serve real-time inference.
evaluated in the paper by 2.54%.

However, similar to other network-based methods, the DBAT faces challenges in
terms of interpretability. Ascertaining whether the network genuinely acquires
material features through numerical evaluation or segmentation visualisation
is a complex task. Therefore, in this extended paper, we further endeavour to
interpret the network behaviour of the DBAT using statistical and visual
tools, such as calculating the attention head equivalent patch size,
visualising attention masks, and assessing the Centred Kernel Alignment (CKA)
heatmap [20, 21]. In order to interpret the features with human-readable
concepts, we also employ the network dissection method [22, 23, 24, 25] to
identify the features learned by the network by aligning layer neurons with
semantic concepts. By analysing the semantic concepts of the extracted
features, this paper illustrates that the DBAT excels in extracting material-
related features, such as texture, which is an essential property for
distinguishing between various materials. By comparing the semantic concepts
of features extracted by other networks trained with either material or object
datasets, the results also indicate that the network architecture can
influence the extracted features, and the patch-based design is indeed
effective in compelling the networks to segment images based on material
features.

##  II Background

###  II-A Material Segmentation

Dense material segmentation aims to identify the material label for each
pixel. To predict the labels, recent segmentation studies tend to choose the
encoder-decoder architecture which down-samples the input image in the encoder
to extract features, and up-samples the features to the original resolution in
the decoder to make predictions [26, 27, 15, 28, 29, 30]. As out-of-the-box
segmentation architectures, these networks are proved to work well on many
datasets. However, for the material segmentation task, no convincing
performance has been achieved in the literature so far [12, 9, 1]. Their
reported per-pixel accuracy (Pixel Acc) is around 70%. According to Schwartz
_et al._[12, 8], annotating the materials in images is challenging due to the
considerable variation in material appearance. This difficulty results in
sparse annotations and unbalanced material labels in material datasets such as
LMD [11, 8], OpenSurfaces [31] and MINC [10]. This restricts the accuracy of
deep neural networks, whose success relies on the existence of large labelled
datasets. To achieve a satisfactory performance, Schwartz _et al._[8, 11]
proposed to extract features from image patches with the assumption that the
local material features can generalise the performance of the network to
unseen material regions. We further observed that the networks composed of
fast pooling and atrous convolution [28, 32, 33, 34] can overlook small
material regions due to the absence of local features [9]. Both works adopt
multi-branch networks to combine local material features with contextual
features about objects, scenes and material boundaries. Although those studies
improved the performance on material datasets, the patch resolution is fixed
to 48-by-48. Fixing the size is unlikely to be optimal for all images as the
areas that materials cover vary within and across images. Moreover, the multi-
branch architecture is computationally costly. As a single-branch network, our
DBAT is efficient enough to serve real-time applications. Furthermore, the
DBAT learns from patches with multiple resolutions and dynamically adjusts the
dependence on each resolution for each pixel of the feature map.

###  II-B Transformers in Vision Tasks

The transformer architecture exploits predictive correlations across multiple
segments of the input data in parallel. While these originated in capturing
word correlations in natural language processing [35, 36], similarly weighted
pixel correlations have been successfully deployed in vision tasks such as
classification [37, 38, 39] and segmentation [27, 40, 41]. The core component
in the transformer architecture is self-attention [42], which predicts the
attention score of each pixel against all other pixels, and calculates the
output by adding the score-weighted pixel values (V). The attention score is
obtained through the correlation between instance-specific queries (Q) and
keys (K), which are predicted from the same set of pixel features. Based on
the window size of the self-attention module, transformers can be categorised
as either global or local types. The global transformers represented by ViT
[37] and DeiT [43] apply the self-attention module to the whole feature map.
This design ensures that the network can have a global view at shallow layers
close to the input image. However, the quadratic complexity in attention
window size makes global transformers expensive to use. Furthermore, the
necessity of a global view at shallow layers is still under debate. A recent
study [21] shows that in models where global attention is employed, the
learned correlations can still focus on local regions at shallow layers. Their
work states that the information hidden in local regions is necessary for good
performance. Since material features, such as texture, are usually extracted
from local image regions, the local transformer architecture is chosen in this
paper. A typical example is the Swin transformer [27, 44], which applies the
self-attention module to small windowed regions. This local design ensures
that the network learns material features independently of contextual
information from the RGB images. By merging adjacent extracted patch features
at each attention stage, the local transformer progressively increases the
patch resolution during forward propagation. Consequently, the DBAT can
predict material labels by aggregating these cross-resolution features,
enabling it to handle materials that cover varying areas.

###  II-C Network Interpretability

The study of network interpretability aims to explain how a network combines
learned features to make predictions. For CNNs, the visualisation of the
convolutional kernel weights shows the pattern of features that the network
has extracted [45, 46]. For the attention-based network module, a simple yet
effective way is to plot the per-pixel attention masks on the input image and
the weights indicate the contribution to the final decision of each pixel [47,
48]. For transformers, however, interpreting the self-attention module remains
challenging due to the high dimensionality of the correlation mask and the
recursively connected attention modules. Carion _et al._[49] proposed to
reduce the dimensionality by visualising an attention mask for individual
pixels of the feature map one at a time. Chefer _et al._[50] reassigned a
trainable relevancy map to the input image and propagate it through all the
self-attention layers. However, these methods are designed for classification
tasks and they can only interpret the transformer behaviour for a specific
image. In this paper, we focus on the segmentation task and we prefer a
summary explanation of the whole dataset. Therefore, we choose to plot the CKA
heatmap [20, 21] and adopt the network dissection method [25, 23, 24]
explained in the following sections.

####  II-C1 Centered Kernel Alignment

The CKA matrix measures the layer similarity by normalising the Hilbert-
Schmidt independence criterion [51], as shown in Equation (1), where $X,Y$ are
the feature maps extracted from two network layers. The HSIC1 in Equation (2)
stands for the unbiased estimator of the Hilbert-Schmidt independence
criterion [51], which measures the distribution alignment between $K,L$. Here
$\tilde{K}\textnormal{ and }\tilde{L}$ are the matrix whose diagonal entries
are set to zero. Since HSIC${}_{1}=0$ indicates the independence of $K$ and
$L$, and CKA is robust to isotropic scaling, they together enable a meaningful
comparison of two networks [52, 21].

$\textnormal{CKA}(X,Y)=\frac{\textnormal{HSIC${}_{1}$}(XX^{T},YY^{T})}{\sqrt{\textnormal{HSIC${}_{1}$}(XX^{T},XX^{T})\textnormal{HSIC${}_{1}$}(YY^{T},YY^{T})}}$
(1)

$\begin{split}\textnormal{HSIC${}_{1}$}(K,L)=\frac{1}{m(m-3)}\bigg{[}tr(\tilde{K}\tilde{L})+\frac{1^{\top}\tilde{K}11^{\top}\tilde{L}1}{(m-1)(m-2)}-\\\
\frac{2}{m-2}1^{\top}\tilde{K}\tilde{L}1\bigg{]}\end{split}$ (2)

Since the CKA does not need to know the network architecture, it can be used
to compare the features that two arbitrary networks learn. By computing the
CKA of the elements in the Cartesian product of the layer sets of two
networks, we have the CKA matrix whose element CKAij indicates the similarity
between layer $i$ in network 1 and layer $j$ in network 2. In this paper, we
show how the DBAT modules alter the features extracted against those from the
backbone transformer. We show that the aggregated cross-resolution patch
features are different from its backbone features and this indicates that our
DBAT learns something new to improve performance.

####  II-C2 Network Dissection

Visualisation tools and CKA help to understand how the model combines specific
features to predict material labels and highlight the similarity or
independence of features acquired by different network layers. However, they
do not offer insight into the nature of these features. To address this issue,
we utilise techniques from the 'network dissection' literature [23, 22, 24,
25], which correlate neuron outputs to an independent set of human-
interpretable labels, such as objects, textures, or scenes.

To calculate the correlation, we require a densely labelled dataset containing
labels for the concepts. In this study, we use the Broden dataset proposed by
[23] to interpret the networks. First, the trained parameters are frozen.
Then, the output of each neuron in the last network layer is thresholded into
a mask to be compared with the corresponding concept labels in terms of mean
Intersection over Union (mIoU) [23]. The threshold is the value $a_{k}$
ensuring that 99.5% of the activation values are greater than it. A neuron is
assigned the interpretive label for which the mIoU score is the highest and
above 0.04. By measuring the number of neurons aligned with each concept, the
network dissection method indicates the features the network focuses on during
training.

This paper applies the network dissection method to compare the proposed DBAT
with selected networks. The results show that the DBAT is particularly good at
detecting local material features, such as texture, which may be the reason
why DBAT achieves the narrowest uncertainty bound across five runs.

It is worth noting that the network dissection method can only interpret
disentangled neurons. This means that only a fraction of the channels of a
network layer can be aligned with meaningful semantic concepts. The rest of
the neurons also detect useful features, but they cannot be explained. One of
the reasons is that these neurons are detecting mixed features (_e.g._
detecting both texture and object combinations). One related research topic is
'Interpretable Networks'. The idea is to disentangle the patterns that each
neuron learns so that the visualisation of the feature map becomes
interpretable. [53] proposed to separate the patterns that a network learns by
building an explanatory graph. The explanatory graph can be applied to trained
networks and summarise the extracted features into a few patterns. [54]
further introduced a filter loss term to regularise the features so that each
neuron contributes to one category with one consistent visual pattern.
However, their networks can only learn features from ball-like areas since the
filter loss is based on a regional template. [55] extended the interpretable
networks to learn disentangled patterns without shape or region limitations.
Their compositional network splits neurons into groups and makes the neurons
learn similar/different features within/across the groups. The trained
networks can produce meaningful feature maps with a slight sacrifice in
accuracy [55]. However, training an interpretable network is beyond the scope
of this paper and we will investigate this method in the future.

##  III Dynamic Backward Attention Transformer

This section explains the DBAT structure in detail. As shown in Fig
LABEL:net_overview, the DBAT consists of three modules: the backbone encoder,
the dynamic backward attention module, and the feature merging module. The
encoder is responsible for extracting cross-resolution feature maps with
window-based attention and patch merging. The proposed Dynamic Backward
Attention (DBA) module predicts per-pixel attention masks to aggregate the
cross-resolution feature maps extracted from the encoder. The feature merging
module guides the DBA module to extract features that are complementary to the
last stage encoder output, which holds a global view of the image. Finally,
the merged features, which have both enhanced cross-resolution features as
well as global features, are passed into a segmentation decoder to make the
material predictions.

###  III-A Dynamic Backward Attention Module

The DBA module depends on a backbone encoder to extract feature maps from
cross-resolution patches. There are multiple approaches to design the encoder.
One possible choice is to employ multiple branches that learn from varying-
sized patches, sharing the features during training [9], and tuning the number
of trainable parameters with the patch size. Another option is to utilise non-
overlapping convolutional kernels [56] and enlarge the patch size through a
pooling layer.

In this paper, we find the transformer as a suitable encoder to extract cross-
resolution patch features [9, 8], as it is inherently designed to process
image patches [37] and demonstrates promising results for vision tasks [37,
27, 44, 57]. Another reason is that, according to recent research [21], the
self-attention module can adapt its equivalent attention distance by assigning
weights to each pixel. In this study, the equivalent attention distance is
defined as the Euclidean distance between two pixels, weighted by the
attention weight. If the predictive accuracy on the training set improves by
increasing the attention weights of neighbouring pixels, the network is
considered to prefer local features. By allowing the network to choose from a
large number of patch sizes through attention weights, our DBAT encoder can
efficiently encode features at different resolutions, despite undergoing only
four stages, which will be discussed in Section VI-B.

Fig. LABEL:dbamodule shows the way our DBA module aggregates the cross-
resolution features. With the assumption that the features at each transformer
stage can preserve spatial location information [21], we propose to aggregate
these features through a weighted sum operation for each pixel of the feature
map. For stage $i$, the feature map spatial size can be computed as
$(\frac{H}{2\times 2^{i}},\frac{W}{2\times 2^{i}})$, where $H$ and $W$ are the
input image height and width. The attention weights $Attn_{i}$ are predicted
from the last feature map, $Map_{4}$ with a 1$\times$1 convolutional layer. To
perform the aggregation operation, it is necessary to normalise the attention
weights with softmax so that the weights across the masks sum to 1 at each
pixel location. It is worth noting that the spatial shapes of the feature maps
should be the same in order to perform the pixel-wise product between
$Map_{i}$ and $Attn_{i}$. Moreover, the shapes of $Attn_{i}$ should be the
same as well to normalise the per-pixel attention masks. In this paper, we set
$Attn_{i}$ to be the same size as $Map_{4}$, and down-sample $Map_{1,2,3}$ to
the shape of $Map_{4}$ so that the computation and memory overhead can be
minimised. The attention mechanism is expressed by Equation (3). Here the
product and sum operations are all performed element-wise.

$\displaystyle Aggregated\ Feature$
$\displaystyle=\sum_{i=1}^{i=4}Attn_{i}\odot Map_{i}$ (3)

###  III-B Feature Merging Module

Although the DBA module aggregates cross-resolution features, it is not
guaranteed that the aggregated features can improve network performance.
Moreover, it is not desired to drop all global or semi-global features since
they can limit the possible material categories in a given context [12].
Therefore, the feature merging module is proposed to guide the DBA module to
enhance the local features without harming the original backbone performance.
This module consists of a global-to-local attention as well as a residual
connection [58]. The simple residual connection, _e.g._ $Merged\
Feature=Map_{4}+Aggregated\ Feature$, can ensure the DBA module learns
complementary features. However, this simple addition operation would over-
emphasise $Map_{4}$ and break our DBA module which aggregates features
linearly. Therefore, we choose to bring in non-linear operations with the
global-to-local attention [59, 27, 60, 61], which identifies the relevant
information in the aggregated cross-resolution patch features through
attention mechanism, and merges it into $Map_{4}$, as illustrated in Fig.
LABEL:featuremerging. The query matrix Q is predicted from $Map_{4}$ and it is
applied to the key matrix K from the aggregated features. The matrix
multiplication (represented by the @ symbol) between Q and $\text{K}^{T}$
produces the attention alignment scores. Then the scores are normalised with
softmax to extract relevant information from the value matrix V. Here the
window-based attention in [27] is used as well. With the DBA module and the
feature merging module, the mechanism of DBAT can be described as enhancing
the material-relate local features by injecting cross-resolution patch
features into $Map_{4}$.

##  IV Experiment Setting

###  IV-A Material Segmentation Datasets

The present study evaluates the proposed DBAT using two datasets, namely the
LMD [12, 8, 9] and the OpenSurfaces [31]. The LMD comprises 5,845 low-
resolution images acquired from indoor and outdoor sources, which have been
manually labelled with 16 mutually exclusive material classes. On the other
hand, OpenSurfaces contains 25,352 high-resolution indoor images labelled with
45 material categories. However, both datasets suffer from sparsely or
coarsely labelled segments, as labelling images with material labels presents
a significant challenge [9, 12]. One of the key difficulties faced by
annotators is that materials are often treated as properties of objects [31,
12]. Consequently, material segments tend to be labelled within object
boundaries, which is undesirable as the material region should be marked
independently of its context. For instance, when annotating a scene depicting
a wooden bed on a wooden floor, the wood segment should ignore the object
boundary and cover all wood pixels. Additionally, OpenSurfaces is highly
unbalanced, with only 27 out of the 45 material classes having more than 60
samples. Among all the samples, 39.44% are segmented as 'wood' or 'painted'.
These limitations make the evaluation on OpenSurfaces less reliable compared
to that on LMD. Therefore, this study mainly focuses on LMD, and the
evaluation on OpenSurfaces will be presented as an additional piece of
evidence. Notably, a recent dataset, MCubeS [62], has been proposed to perform
material segmentation on outdoor images using multimodal data such as imaging
with near-infrared and polarised light. However, since this study concentrates
on material segmentation using indoor RGB images, the evaluation on MCubeS is
not included.

###  IV-B Evaluation metrics

In this paper, the networks are evaluated with three metrics: mean pixel
accuracy (Pixel Acc), mean class accuracy (Mean Acc), and mIoU. As discussed
in Section IV-A, the material annotations may not cover the whole material
region. As a consequence, the mIoU numerator would be much smaller than it
should be. This situation is especially severe for LMD which was annotated
sparsely on purpose [12, 8]. Therefore, the mIoU is not reported for LMD. In
addition to evaluating segmentation performance, we report the resources
required for each model, including the number of trainable parameters and the
number of floating-point operations (FLOPs) per forward propagation. To select
SOTA models for evaluation, this paper sets a selection criterion with the
frames per second (FPS). The model variants that can support real-time
inference (FPS larger than 24) are compared with the DBAT in this paper.

###  IV-C Implementation details

The networks reported in Section V are pre-trained on ImageNet [63]. This pre-
training step is expected to teach the network with prior knowledge about
contextual information such as scenes and objects. According to [12],
contextual information can reduce the uncertainty in material segmentation.
Therefore, the network should learn material features more efficiently with a
pre-training strategy. The details will be discussed in Section VI-C. For the
Swin backbone, its implementation follows the original paper and the window
size of the self-attention is set to 7. The decoder used in this paper is the
Feature Pyramid Network (FPN) [64] for the reason that the FPN can recognise
small material regions well [9]. The training images are resized first so that
the minimum borders are equal to 512. The resized images are then cropped into
$512\times 512$ patches to use batch training [65]. We use the AdamW optimiser
to train the networks with batch size 16, coefficients $\beta_{1}$ 0.9,
$\beta_{2}$ 0.999, and weight decay coefficient 0.01. Further, the learning
rate is warmed-up from 0 to 0.00006 with 1,500 training steps, and decreased
polynomially. The networks are trained on LMD for 200 epochs and on
OpenSurfaces for five days.

##  V Segmentation Performance Analysis

###  V-A Quantitative Analysis

Table I reports the segmentation performance of our DBAT as well as five other
models, ResNet-152 [58], ResNest-101 [66], EfficientNet-b5 [67], Swin-t [27],
and CAM-SegNet [9]. Their heaviest variants that can serve real-time inference
are evaluated apart from the CAM-SegNet. Although the CAM-SegNet does not meet
the real-time selection criterion, it is the most recent architecture for the
RGB-based material segmentation task. We notice that its architecture is
suitable for our DBA module since it has a dedicated local branch. Therefore,
in this paper, its local branch is equipped with the DBA module and its
performance is reported as the CAM-SegNet-DBA †††The CAM-SegNet-DBA is
implemented by replacing the original local branch [9] by a combination of
non-overlapping convolutional kernels and MLP. The patch resolution is
enlarged by concatenating features within the kernel size.. The evaluations
are reported across five independent runs. The metric differences are reported
in the order (Pixel Acc/Mean Acc/mIoU) with the additive method.

As shown in Table I, our DBAT achieves the best accuracy on the LMD among all
the real-time models in terms of Pixel Acc and Mean Acc. Specifically, our
DBAT achieves +0.85%/+1.50% higher than the second-best model, CAM-SegNet-DBA.
It is also +2.54%/+2.52% higher than its backbone encoder, Swin-t. As for the
OpenSurfaces, our DBAT beats the chosen models on Pixel Acc and mIoU.
Moreover, its performance is comparable to the multi-branch CAM-SegNet-DBA
(+0.50%/-0.62%/+0.09%) with 9.65 more FPS and 19.6G fewer FLOPs. It is worth
noting that compared with the performance reported in the original paper of
CAM-SegNet [9], our DBAT improves the Pixel Acc by 21.21%. Moreover, the per-
category analysis in Table LABEL:append_table_per_category shows that the DBA
module improves the recognition of materials that usually have uniform
appearances but varying shapes, such as paper, stone, fabric and wood. This
indicates that the cross-resolution features successfully learn from
distinguishable material features.

As shown in Fig. LABEL:uncertainty_lmd, our DBAT can segment the materials
consistently to achieve narrow uncertainty bounds, especially for the category
foliage from LMD. Unlike the other five models, almost all the reported runs
of our DBAT are within the upper and lower whiskers except for the category
asphalt and metal. For the evaluations on OpenSurfaces shown in Fig.
LABEL:uncertainty_opensurfaces, the uncertainty bounds of the proposed DBAT
are much narrower compared with other models
($\pm$$0.02\text{\,}\mathrm{\textup{p.p.}}$ for Pixel Acc,
$\pm$$0.08\text{\,}\mathrm{\textup{p.p.}}$ for Mean Acc and $\pm$
$0.06\text{\,}\mathrm{\textup{p.p.}}$ for mIoU)‡‡‡Here p.p. stands for the
percentage points.. This indicates that the DBAT is robust to the network
initialisation and can learn from image patches effectively. The CKA
similarity score, which is 0.9583 for our DBAT, is calculated and averaged for
every two checkpoints of the five individually trained networks to support
this deduction.

Datasets | LMD | OpenSurfaces | –
---|---|---|---
Architecture | Pixel Acc (%) | Mean Acc (%) | Pixel Acc (%) | Mean Acc (%) | mIoU (%) | #params (M) | #flops (G) | FPS
ResNet-152 | 80.68 $\pm$ 0.11 | 73.87 $\pm$ 0.25 | 83.11 $\pm$ 0.68 | 63.13 $\pm$ 0.65 | 50.98 $\pm$ 1.12 | 60.75 | 70.27 | 31.35
ResNeSt-101 | 82.45 $\pm$ 0.20 | 75.31 $\pm$ 0.29 | 84.75 $\pm$ 0.57 | 65.76 $\pm$ 1.32 | 53.74 $\pm$ 1.06 | 48.84 | 63.39 | 25.57
EfficientNet-b5 | 83.17 $\pm$ 0.06 | 76.91 $\pm$ 0.06 | 84.64 $\pm$ 0.34 | 65.41 $\pm$ 0.44 | 53.79 $\pm$ 0.54 | 30.17 | 20.5 | 27.00
Swin-t | 84.70 $\pm$ 0.26 | 79.06 $\pm$ 0.46 | 85.88 $\pm$ 0.27 | 69.74 $\pm$ 1.19 | 57.39 $\pm$ 0.54 | 29.52 | 34.25 | 33.94
CAM-SegNet-DBA | 86.12 $\pm$ 0.15 | 79.85 $\pm$ 0.28 | 86.00 $\pm$ 0.64 | 69.61 $\pm$ 1.08 | 57.52 $\pm$ 1.44 | 68.58 | 60.83 | 17.79
DBAT | 86.85 $\pm$ 0.08 | 81.05 $\pm$ 0.28 | 86.43 $\pm$ 0.02 | 69.18 $\pm$ 0.08 | 57.57 $\pm$ 0.06 | 56.03 | 41.23 | 27.44

TABLE I: Segmentation performance on the LMD and the OpenSurfaces. The FPS is
calculated by processing 1000 images with one NVIDIA 3060ti. The uncertainty
evaluation is reported by training the networks five times. The best
performance is shown in bold text and the second best is underlined.

###  V-B Qualitative Analysis

Fig. LABEL:B-4.1 shows the predicted material segmentation for three images.
In Fig. LABEL:B-4.1 (a), ResNet-152 [58], ResNest-101 [66], Swin-t [27], and
the modified CAM-SegNet-DBA segment the bed as fabric, the floor as plaster.
However, in this image, the floor appears to be covered with a carpet whose
material is fabric. One possible explanation is that for the scene bedroom,
the floor and wall are typically covered with plaster. These networks fail to
make predictions based on material features, but rely on contextual
information, so they tend to use plaster as a label for predictions. The
proposed DBAT and the EfficientNet-b5 [67] break the object boundary and
segment part of the floor as fabric. Moreover, there are fewer noisy pixels in
the DBAT segmented image when compared with the EfficientNet-b5. This
indicates that with cross-resolution patch features, the DBAT can identify
materials densely and precisely even if it is trained on a sparsely labelled
dataset.

The segmented materials in Fig. LABEL:B-4.1 (b) and Fig. LABEL:B-4.1 (c)
provide more evidence that the DBAT can segment the images well with features
extracted from cross-resolution patches. In Fig. LABEL:B-4.1 (b), the boundary
between the wooden window frame and glass-made windows in DBAT segmented image
is more adequate than the segments predicted by other networks. In Fig.
LABEL:B-4.1 (c), the segmented fabric aligns well with the ground truth with
no noisy predictions. Considering that the training of DBAT takes sparsely
labelled segments, it is reasonable to say that DBAT learns the difference
between materials from cross-solution patches.

###  V-C Ablation Study

This section studies the effectiveness of each component of the DBAT. In Table
II, the performance is reported after removing the feature merging module and
the DBA module in sequence. Without the feature merging module, the Pixel Acc
and Mean Acc decrease by $1.61\text{\,}\mathrm{\textup{p.p.}}$ and
$2.04\text{\,}\mathrm{\textup{p.p.}}$ respectively. This shows the importance
of the attention-based residual connection in improving performance. The
performance drops by another $0.72\text{\,}\mathrm{\textup{p.p.}}$ in Pixel
Acc and $0.13\text{\,}\mathrm{\textup{p.p.}}$ in Mean Acc after removing the
DBA module. This shows that the DBA module that learns complementary cross-
resolution features guided by the feature merging module can improve
performance effectively.

Architecture |  $\Delta$ Pixel Acc |  $\Delta$ Mean Acc
---|---|---
\- Feature merging | -1.61 | -2.04
\- Dynamic backward attention | -2.33 | -2.17
TABLE II: The ablation study to analyse each component of our DBAT. The
performance difference is reported in percentage points.

In order to justify the network design, this section further studies the
alternative implementations of the DBAT from three aspects: 1) how the
attention masks are predicted; 2) how the feature maps are down-sampled; 3)
how aggregated features are merged with $Map_{4}$. The performance differences
in Table III are reported by switching one of the implementations in DBAT to
its alternatives. As stated in Section III-A, the proposed DBAT adopts
convolutional kernels to generate the per-pixel attention masks [68]. By
replacing the kernels to dilated ones [69], the receptive field is enlarged
when predicting the masks. However, the DBAT performance decreases
significantly by
$-2.15\text{\,}\mathrm{\textup{p.p.}}$/$-2.67\text{\,}\mathrm{\textup{p.p.}}$.
This indicates that the local information is critical for the dynamic
attention module to work well. Section III-A describes that the cross-
resolution feature maps need to be down-sampled so that the fixed-size
attention masks can be applied. Originally DBAT uses the MLP to down-sample
the feature map. Instead of using such a trainable method, we can also use a
superficial non-parametric pooling layer, which decreases the performance by
$-0.88\text{\,}\mathrm{\textup{p.p.}}$/$-1.58\text{\,}\mathrm{\textup{p.p.}}$.
The slight drop in Pixel Acc and the significant drop in Mean Acc suggest that
the trainable down-sampling method can help balance the performance of
different material categories. As for the feature merging module, a simple
residual connection slightly reduces the performance by
$-0.58\text{\,}\mathrm{\textup{p.p.}}$/$-0.64\text{\,}\mathrm{\textup{p.p.}}$.
This highlights that DBAT needs a residual connection to guide the aggregation
of cross-resolution features and the representation capability of the feature
merging module may not be vital.

Implementation Choices |  $\Delta$ Pixel Acc |  $\Delta$ Mean Acc
---|---|---
| Generate
---
Attention Masks
CNN $-->$ Dilated CNN | -2.15 | -2.67
Down-sample | MLP $-->$ Average Pooling | -0.88 | -1.58
Feature Merging | Attention $-->$ Residual Connection | -0.58 | -0.64

TABLE III: The study of implementation choices in each component of our DBAT.
The performance difference is reported in percentage points.

##  VI Network Behaviour Analysis

Although our DBAT achieves the best accuracy with carefully designed
propagation, like other successful networks, its mechanism is still hidden in
a black box. In order to interpret the DBAT, we employ both statistical
methods such as reporting the average attention weights for each patch
resolution and feature interpretability methods such as the CKA analysis and
network dissection. We discover that our DBAT is particularly good at learning
material-related features such as texture, which justifies our assumption that
features extracted from image patches can boost performance in the material
segmentation task.

###  VI-A CKA Heatmap

Fig. LABEL:fig:ckamatrix (a) visualises the CKA heatmap of the DBAT and Fig.
LABEL:fig:ckamatrix (b) shows the heatmap measured between DBAT and Swin. The
brightness of the colour indicates the similarity of the features extracted by
two layers. The network layers are indexed by the forward-propagation order.
The DBAT and the Swin share the same network architecture before layer 106,
which explains the bright diagonal in Fig. LABEL:fig:ckamatrix (b) connecting
(0, 0) and (105, 105). When observed closely, the diagonal line becomes darker
as it approaches the point (105, 105). This suggests that at shallower layers,
the Swin backbone extracts features similar to when it is used alone, and it
gradually learns something new as it approaches the deeper layers. The dark
areas from layer 106 to layer 113 in Fig. LABEL:fig:ckamatrix (a) reflect the
attention masks predicted by the DBA module. By collecting the cross-
resolution feature maps, the aggregated features contain information from both
shallow and deep layers, illustrated by the bright region between layer 113
and 124. After layer 124, the feature merging module injects the relevant
information from the aggregated features into $Map_{4}$. This module produces
a feature map that differs from the features extracted by Swin, as shown by
the darker points near (140, 100) in Fig. LABEL:fig:ckamatrix (b).

###  VI-B Attention Analysis

This section analyses the dynamic attention module by visualising the
attention masks and calculating their descriptive statistics, including the
average attention weights as well as the equivalent attention patch
resolutions. Fig. LABEL:fig:attentionmasks shows the attention masks for
images in the LMD test set. The patch resolution increases from $Attn_{1}$ to
$Attn_{4}$. We notice that a prediction for a material covering multiple small
objects or a small area tends to depend on the features extracted from small
patch resolutions. For example, the first column images in Fig.
LABEL:fig:attentionmasks highlight the regions on where the network
concentrates. A brighter colour indicates higher attention weights. The wooden
area in the first row covers both the desk and the floor. The wooden chairs
and the fabric floor are mutually overlapping in the second row. Small patches
can isolate the objects and learn material features along the boundaries.

Fig. LABEL:fig:analysis_attn shows the average attention weight in (a) and the
equivalent patch size in (b, the box plot). The equivalent patch size, which
represents the attention distance of each attention head, is calculated by
transforming the attention diagonal distance in [21] to the side length of a
square, which suggests the patch resolution used to extract features from the
feature maps $Map_{1,2,3,4}$. As expected, the aggregated features mostly
(52.40%) depend on $Attn_{4}$, which is thoroughly processed by the whole
backbone encoder, with an average patch size of 74.31. Apart from $Attn_{4}$,
the aggregated features also depend on feature maps extracted from small patch
sizes. For example, $Attn_{3}$ is extracted from an average patch size of
31.68, and it contributes 30.50% to the aggregated features. Although
$Attn_{1}$ is gathered from a shallow stage of the network, the aggregated
features still depend on it to handle the overlapping material regions with a
patch size of 6.75 on average.

To further illustrate the effect of the dynamic backward attention module, the
similarity scores comparing one layer with $Map_{1,2,3,4}$ from the CKA matrix
is reported in Fig. LABEL:fig:analysis_attn. (b, the line plot). The blue line
compares $Map_{4}$ from the Swin encoder, and the orange line compares the
aggregated features from the DBAT. The increased similarity scores against
$Map_{1,2,3}$ clearly show that the aggregated feature depends on information
from shallow layers.

###  VI-C Network Dissection

The network dissection method aligns the disentangled neurons of one network
layer to semantic concepts [23, 24]. By counting the portion of neurons
aligned to each concept, it is possible to give an understanding of what
features the network learns. This paper studies local concepts such as colour,
texture and part, as well as global/semi-global concepts like object and
scene. The neurons of the last encoder layer are selected to be analysed since
they are more interpretable than shallow layers [23, 24]. Fig.
LABEL:netdissec-swindbat (a) depicts how pre-training influences the features
that the DBAT learns. Without pre-training, DBAT (the blue line in Fig.
LABEL:netdissec-swindbat) has shown a preference for texture features, and a
portion of its neurons can detect object and scene features. As shown by the
dotted orange line, the Swin backbone [27] trained with ImageNet [70] tends to
detect object features. The DBAT has more neurons aligned to texture and
object features when trained with a pre-trained backbone, shown as the purple
line. The observations indicate two aspects: (1) the DBAT relies on texture
features to solve the material segmentation task. (2) the pre-trained object
detectors reduce the uncertainty in identifying materials and ease the
training of texture detectors.

This paper further analyses the feature difference between material and object
tasks. In particular, Swin [27] is trained with two object-related datasets:
the ImageNet [63], an object classification dataset, and the ADE20K [71, 72],
an object-level segmentation dataset. As shown in Fig. LABEL:netdissec-
swindbat (b), the significant difference is the lack of texture features in
object-related tasks. This discovery highlights that enhancing features hidden
in patches is a valid heuristic to improve the network performance on the
material segmentation task.

Fig. LABEL:netdissec-comparemodels dissects three more networks: ResNet-152
[58], ResNest-101 [66], EfficientNet-b5 [67] on both material and object
tasks. An interesting discovery is that although these networks achieve
comparable performance, the features that they learn are different. For
example, the ResNet-152 relies on texture features on both tasks and it learns
more part-related features on material task compared with object task.
Although three of the networks in Fig. LABEL:netdissec-comparemodels learn
texture features on material task, a special exception is the EfficientNet-b5,
which knows almost nothing about texture for both tasks. This phenomenon goes
against the intuition that networks targeting material segmentation should
learn texture features well since texture describes the appearances of
materials. One reasonable explanation is that assigning material labels to
object or object parts can cover the labelled material region and achieves a
high accuracy since these material datasets are sparsely labelled. Therefore,
here we call for densely labelled material segmentation datasets for reliable
evaluation and analysis.

##  VII Conclusion

This paper proposed a single-branch network, the DBAT, to learn material-
related features extracted from image patches at multiple resolutions. The
features are aggregated dynamically based on predicted attention weights. The
network is designed to learn complementary features from the cross-resolution
patches with an attention-based residual connection. The DBAT outperforms all
chosen real-time models evaluated on two datasets. It also achieves comparable
performance with fewer FLOPs compared with the multi-branch network, the CAM-
SegNet-DBA. As for network analysis, this paper illustrates its inference
pipeline by visualising the attention masks as well as the CKA heatmaps. The
experiment shows that about half of the information comes from small image
patches, and those cross-resolution features can help the network learns from
overlapping materials. Moreover, the CKA heatmap shows that the aggregated
features carry the knowledge from shallow layers, which may be the key to
achieving good performance on material segmentation tasks. Instead of focusing
solely on achieving better performance on material segmentation tasks, we
moved a step further in this paper to interpret the material features that a
network learns with semantic concepts. The counted portions of neurons aligned
to each concept show that our DABT is particularly good at extracting texture-
related features. The analysis also reflects the potential unreliability of
evaluating sparse datasets, so we will seek for densely labelled material
datasets for reliable network evaluations in future work. One possible method
is to use the physically-based rendering to generate synthetic densely
annotated material segmentation datasets.

## Acknowledgments

This work was supported by the EPSRC Programme Grant Immersive Audio-Visual 3D
Scene Reproduction Using a Single 360 Camera (EP/V03538X/1). For the purpose
of open access, the author has applied a Creative Commons Attribution* (CC BY)
licence to any Author Accepted Manuscript version arising.

## References

  * [1] C. Zhao, L. Sun, and R. Stolkin, “Simultaneous material segmentation and 3d reconstruction in industrial scenarios,” _Frontiers in Robotics and AI_ , vol. 7, p. 52, 2020.
  * [2] N. Shrivatsav, L. Nair, and S. Chernova, “Tool substitution with shape and material reasoning using dual neural networks,” _arXiv preprint arXiv:1911.04521_ , 2019.
  * [3] A. McDonagh, J. Lemley, R. Cassidy, and P. Corcoran, “Synthesizing game audio using deep neural networks,” in _2018 IEEE Games, Entertainment, Media Conference (GEM)_. IEEE, 2018, pp. 1–9.
  * [4] H. Kim, L. Remaggi, P. J. Jackson, and A. Hilton, “Immersive spatial audio reproduction for vr/ar using room acoustic modelling from 360 images,” in _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2019, pp. 120–126.
  * [5] L. Chen, W. Tang, N. W. John, T. R. Wan, and J. J. Zhang, “Context-aware mixed reality: A learning-based framework for semantic-level interaction,” in _Computer Graphics Forum_ , vol. 39. Wiley Online Library, 2020, pp. 484–496.
  * [6] Y. Heng, S. Dasmahapatra, and H. Kim, “Material recognition for immersive interactions in virtual/augmented reality,” in _2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_. IEEE, 2023, pp. 577–578.
  * [7] A. Mona, W. Yihong, H. Yuwen, N. Mahesan, and K. Hansung, “Room acoustic properties estimation from a single 360° photo,” in _2022 30th European Signal Processing Conference (EUSIPCO)_. IEEE, 2022.
  * [8] G. Schwartz and K. Nishino, “Recognizing material properties from images,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 8, pp. 1981–1995, 2020.
  * [9] Y. Heng, Y. Wu, H. Kim, and S. Dasmahapatra, “Cam-segnet: A context-aware dense material segmentation network for sparsely labelled datasets,” in _17th International Conference on Computer Vision Theory and Applications (VISAPP)_ , vol. 5, 2022, pp. 190–201.
  * [10] S. Bell, P. Upchurch, N. Snavely, and K. Bala, “Material recognition in the wild with the materials in context database,” _Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * [11] G. Schwartz, _Visual Material Recognition_. Drexel University, 2018.
  * [12] G. Schwartz and K. Nishino, “Material recognition from local appearance in global context,” in _Biol. and Artificial Vision (Workshop held in conjunction with ECCV 2016)_ , 2016.
  * [13] S. Bell, P. Upchurch, N. Snavely, and K. Bala, “Material recognition in the wild with the materials in context database,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3479–3487.
  * [14] H. Cheng, C. Gu, and K. Wu, “Weakly-supervised semantic segmentation via self-training,” in _Journal of Physics: Conference Series_ , vol. 1487, no. 1. IOP Publishing, 2020, p. 012001\.
  * [15] B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Le, “Rethinking pre-training and self-training,” in _NeurIPS_ , 2020.
  * [16] A. Bokhovkin and E. Burnaev, “Boundary loss for remote sensing imagery semantic segmentation,” in _International Symposium on Neural Networks_. Springer, 2019, pp. 388–401.
  * [17] Y. Heng, Y. Wu, S. Dasmahapatra, and H. Kim, “Enhancing material features using dynamic backward attention on cross-resolution patches,” in _33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022_. BMVA Press, 2022. [Online]. Available: https://bmvc2022.mpi-inf.mpg.de/0004.pdf
  * [18] A. Gotmare, N. S. Keskar, C. Xiong, and R. Socher, “A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation,” in _International Conference on Learning Representations_ , 2018.
  * [19] P. Mishra and K. Sarawadekar, “Polynomial learning rate policy with warm restart for deep neural network,” in _TENCON 2019-2019 IEEE Region 10 Conference (TENCON)_. IEEE, 2019, pp. 2087–2092.
  * [20] T. Nguyen, M. Raghu, and S. Kornblith, “Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,” _International Conference on Learning Representations_ , 2020\.
  * [21] M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do vision transformers see like convolutional neural networks?” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [22] B. Zhou, D. Bau, A. Oliva, and A. Torralba, “Interpreting deep visual representations via network dissection,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 9, pp. 2131–2145, 2018.
  * [23] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, “Network dissection: Quantifying interpretability of deep visual representations,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6541–6549.
  * [24] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba, “Gan dissection: Visualizing and understanding generative adversarial networks,” in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * [25] D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba, “Understanding the role of individual units in a deep neural network,” _Proceedings of the National Academy of Sciences_ , vol. 117, no. 48, pp. 30 071–30 078, 2020.
  * [26] Y. Xing, L. Zhong, and X. Zhong, “An encoder-decoder network based fcn architecture for semantic segmentation,” _Wireless Communications and Mobile Computing_ , vol. 2020, 2020.
  * [27] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [28] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [29] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2881–2890.
  * [30] S.-W. Kim, H.-K. Kook, J.-Y. Sun, M.-C. Kang, and S.-J. Ko, “Parallel feature pyramid network for object detection,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 234–250.
  * [31] S. Bell, P. Upchurch, N. Snavely, and K. Bala, “OpenSurfaces: A richly annotated catalog of surface appearance,” _ACM Trans. on Graphics (SIGGRAPH)_ , vol. 32, no. 4, 2013.
  * [32] S. Qiao, L.-C. Chen, and A. Yuille, “Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 213–10 224.
  * [33] T. Panboonyuen, K. Jitkajornwanich, S. Lawawirojwong, P. Srestasathiern, and P. Vateekul, “Semantic labeling in remote sensing corpora using feature fusion-based enhanced global convolutional network with high-resolution representations and depthwise atrous convolution,” _Remote Sensing_ , vol. 12, no. 8, p. 1233, 2020.
  * [34] M. Xu, S. Zhong, C. Liu, S. Gong, Z. Wang, and Y. Xia, “Acclvos: Atrous convolution with spatial-temporal convlstm for video object segmentation,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 2089–2096.
  * [35] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [37] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2020\.
  * [38] C.-F. R. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-attention multi-scale vision transformer for image classification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 357–366.
  * [39] Y. Qing, W. Liu, L. Feng, and W. Gao, “Improved transformer net for hyperspectral image classification,” _Remote Sensing_ , vol. 13, no. 11, p. 2216, 2021.
  * [40] R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7262–7272.
  * [41] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr _et al._ , “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 6881–6890.
  * [42] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 076–10 085.
  * [43] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 10 347–10 357.
  * [44] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong _et al._ , “Swin transformer v2: Scaling up capacity and resolution,” _arXiv preprint arXiv:2111.09883_ , 2021.
  * [45] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012.
  * [46] Z. J. Wang, R. Turko, O. Shaikh, H. Park, N. Das, F. Hohman, M. Kahng, and D. H. P. Chau, “Cnn explainer: Learning convolutional neural networks with interactive visualization,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 27, no. 2, pp. 1396–1406, 2020.
  * [47] H. Fukui, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Attention branch network: Learning of attention mechanism for visual explanation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 10 705–10 714.
  * [48] N. Liu, J. Han, and M.-H. Yang, “Picanet: Learning pixel-wise contextual attention for saliency detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3089–3098.
  * [49] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [50] H. Chefer, S. Gur, and L. Wolf, “Transformer interpretability beyond attention visualization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 782–791.
  * [51] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt, “Feature selection via dependence maximization.” _Journal of Machine Learning Research_ , vol. 13, no. 5, 2012.
  * [52] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3519–3529.
  * [53] Q. Zhang, R. Cao, F. Shi, Y. N. Wu, and S.-C. Zhu, “Interpreting cnn knowledge via an explanatory graph,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [54] Q. Zhang, Y. N. Wu, and S.-C. Zhu, “Interpretable convolutional neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8827–8836.
  * [55] W. Shen, Z. Wei, S. Huang, B. Zhang, J. Fan, P. Zhao, and Q. Zhang, “Interpretable compositional convolutional neural networks,” in _Proceedings of the International Joint Conference on Artificial Intelligence_ , 2021.
  * [56] N. Yamanakkanavar and B. Lee, “Using a patch-wise m-net convolutional neural network for tissue segmentation in brain mri images,” _IEEE Access_ , vol. 8, pp. 120 946–120 958, 2020.
  * [57] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [58] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [59] Y. Shen, E. M.-K. Lai, and M. Mohaghegh, “Effects of similarity score functions in attention mechanisms on the performance of neural question answering systems,” _Neural Processing Letters_ , pp. 1–20, 2022.
  * [60] B. Tu, W. He, W. He, X. Ou, and A. Plaza, “Hyperspectral classification via global-local hierarchical weighting fusion network,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 15, pp. 184–200, 2021.
  * [61] M. Xu, D. F. Wong, B. Yang, Y. Zhang, and L. S. Chao, “Leveraging local and global patterns for self-attention networks,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, jul 2019, pp. 3069–3075.
  * [62] Y. Liang, R. Wakaki, S. Nobuhara, and K. Nishino, “Multimodal material segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 800–19 808.
  * [63] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [64] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [65] L. Mu, “Efficient mini-batch training for stochastic optimization,” _ACM_ , 2014.
  * [66] H. Zhang, C. Wu, Z. Zhang, Y. Zhu, H. Lin, Z. Zhang, Y. Sun, T. He, J. Mueller, R. Manmatha _et al._ , “Resnest: Split-attention networks,” _arXiv preprint arXiv:2004.08955_ , 2020.
  * [67] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [68] Y. Chen, X. Dai, M. Liu, D. Chen, L. Yuan, and Z. Liu, “Dynamic convolution: Attention over convolution kernels,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 030–11 039.
  * [69] Y. Wei, H. Xiao, H. Shi, Z. Jie, J. Feng, and T. S. Huang, “Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7268–7277.
  * [70] M. Á. Martínez-Domingo, E. M. Valero, J. Hernández-Andrés, S. Tominaga, T. Horiuchi, and K. Hirai, “Image processing pipeline for segmentation and material classification based on multispectral high dynamic range polarimetric images,” _Optics express_ , vol. 25, no. 24, pp. 30 073–30 090, 2017.
  * [71] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 633–641.
  * [72] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ade20k dataset,” _International Journal of Computer Vision_ , vol. 127, no. 3, pp. 302–321, 2019.