# A Label Propagation Strategy for CutMix in Multi-Label Remote Sensing Image
Classification

Tom Burgert, Tim Siebert, Kai Norman Clasen, and Begüm Demir T. Burgert, T.
Siebert, K. Clasen, and B. Demir are with the Berlin Institute for the
Foundations of Learning and Data (BIFOLD) and with the Faculty of Electrical
Engineering and Computer Science, Technische Universität Berlin, 10623 Berlin,
Germany (emails<EMAIL_ADDRESS><EMAIL_ADDRESS>k.clasen@tu-
berlin.de, demir@tu-berlin.de).

###### Abstract

The development of supervised deep learning-based methods for multi-label
scene classification (MLC) is one of the prominent research directions in
remote sensing (RS). Yet, collecting annotations for large RS image archives
is time-consuming and costly. To address this issue, several data augmentation
methods have been introduced in RS. Among others, the data augmentation
technique CutMix, which combines parts of two existing training images to
generate an augmented image, stands out as a particularly effective approach.
However, the direct application of CutMix in RS MLC can lead to the erasure or
addition of class labels (i.e., label noise) in the augmented (i.e., combined)
training image. To address this problem, we introduce a label propagation (LP)
strategy that allows the effective application of CutMix in the context of MLC
problems in RS without being affected by label noise. To this end, our
proposed LP strategy exploits pixel-level class positional information to
update the multi-label of the augmented training image. We propose to access
such class positional information from reference maps associated to each
training image (e.g., thematic products) or from class explanation masks
provided by an explanation method if no reference maps are available.
Similarly to pairing two training images, our LP strategy carries out a
pairing operation on the associated pixel-level class positional information
to derive the updated multi-label for the augmented image. Experimental
results show the effectiveness of our LP strategy in general and its
robustness in the case of various simulated and real scenarios with noisy
class positional information in particular.

###### Index Terms:

CutMix, data augmentation, label propagation strategy, multi-label scene
classification (MLC), remote sensing (RS)

##  I Introduction

The growing interest in Earth observation has led to significant technological
advancements, contributing to a recent surge in the availability of remote
sensing (RS) image archives. Typically, an RS image scene consists of multiple
classes and, thus, can simultaneously be associated with different land use
land cover (LULC) class labels (i.e., multi-labels). Therefore, the
development of multi-label scene classification (MLC) methods that aim to
automatically assign multiple labels to each image in an archive attracts
growing research interest in the RS community [1], [2], [3]. In particular,
MLC gains popularity due to its capacity of facilitating a quick
categorization and understanding of the image scenes. Aligned with the
advances in machine learning, deep neural networks play a crucial role for
many state-of-the-art results in RS MLC. As a result of costly and time-
consuming annotation processes, an important factor to improve the
generalization performance of these networks is the use of data augmentation
techniques. Data augmentation refers to the process of creating slightly
modified versions of existing training images by applying stochastic
transformations to the images that preserve their semantic characteristics.
Consequently, deep neural networks trained with augmented images are less
prone to overfit on individual images and, simultaneously, gain a diversified
understanding of individual classes as the effective amount of training images
is increased. The data augmentation techniques proposed in the literature
typically function in a task-agnostic manner. Most of the existing techniques
are initially presented in the context of single-label image classification
(assigning a single high-level category label to each image), but are also
applied in other related image recognition tasks such as MLC, object
detection, or semantic segmentation (i.e., pixel-based classification). While
a wide range of data augmentation techniques can also be directly applicable
to RS MLC, its current utilization is limited to simple techniques like image
rotations, translations, and flipping [4].

Among different data augmentation techniques, CutMix [5] stands out as a
particularly effective approach. It is inspired by the success of the box
augmentation strategy CutOut [6], [7] that removes areas from an image and
replaces them with zeros or random noise to enhance model robustness against
partially invisible class features (e.g., occlusion, cut-off parts). CutMix
takes this concept further by filling informative parts from different images
into the uninformative cutout areas and updating the label accordingly.
Consequently, CutMix not only ensures model robustness to partially invisible
class features but also introduces an additional learning signal, enriching
the model’s understanding of the data. Due to its effectiveness, CutMix has
been successfully adopted for RS tasks, such as single-label scene
classification [8], ship object detection [9], and semantic segmentation [10].

However, in contrast to simple image transformations that preserve the whole
training image (e.g., rotation, flipping), the use of CutMix for MLC problems
remains challenging. Unlike object detection and semantic segmentation, there
is no direct access to pixel-level class positional information in MLC to
correctly update the multi-labels. Applying CutMix to multi-label images
without a proper label update strategy can result in the introduction of
multi-label noise [11]. If a class is only present in the cutout area, the
application of CutMix can lead to additive noise, as the multi-label would
incorrectly indicate the presence of this class. On the other hand, if a new
class is contained in the pasted area originating from a different image, the
application of CutMix can lead to subtractive noise as the multi-label would
incorrectly indicate the absence of the class.

To overcome the problem of introducing multi-label noise when directly
applying CutMix in RS MLC, we propose a label propagation (LP) strategy that
enables the correct update of the multi-label of a CutMix-augmented image. In
addition to pairing two images, the LP strategy carries out a pairing
operation on associated pixel-level class positional information to derive the
updated multi-label vector for the augmented image. In detail, we consider two
scenarios associated to the availability of pixel-level class positional
information. In the first scenario, we assume that the multi-labels of the
images originate from associated reference maps (i.e., each class label
annotated in the reference map is indicated as present in the multi-label). In
other words, for each training image class labels are not only available at
image-level but also at pixel-level. These reference maps may comprise
different levels of unreliability depending on the considered way of obtaining
the map. In the case that the associated reference maps are obtained via
manual annotation of each pixel (e.g., DeepGlobe dataset [12]), unreliability
in class positional information can arise from labeling errors in the manual
pixel annotation of the images. In the case that the associated reference maps
are obtained through the inference of a segmentation model (i.e., pixel-based
classification model) (e.g., Dynamic World dataset [13]), unreliability in
class positional information can originate from inaccurate predictions. In the
case that the associated reference maps are obtained from thematic products
[14], [15], [16], [17] (e.g., BigEarthNet [18]), unreliability in class
positional information can have various causes, such as (i) different
resolutions between thematic product and the considered image, (ii) imprecise
co-registration of thematic product and the considered image, (iii) temporal
difference between thematic product and the considered image, or (iv) labeling
error in the thematic product. In the second scenario, we assume that
associated reference maps are not available. Thus, each training image is only
annotated with image-level multi-labels. In this case, we propose to leverage
post hoc explanation methods to create class explanation masks that attribute
each class that is indicated as present in the multi-label of an image with
the respective relevant pixels. Likewise, the second scenario may also include
unreliability due to imprecise explanations. Experimental results obtained on
three diverse datasets show the effectiveness of our LP strategy in general
and robustness to not fully reliable (i.e., noisy) pixel-level class
positional information in particular.

In summary, our contributions are the following:

  * •

We propose an LP strategy that allows the effective application of CutMix to
RS MLC without being affected by multi-label noise.

  * •

To the best of our knowledge, we are the first to explore the benefits of
thematic products and explanation methods to provide pixel-level class
positional information for correctly updating the multi-labels of a CutMix-
augmented image in RS MLC.

  * •

We conduct extensive experiments considering class positional information with
varying levels of reliability. In particular, we show the effectiveness of our
LP strategy for: (i) reliable class positional information; (ii) a variety of
six simulated types of noisy class positional information; (iii) a scenario in
which the noisy class positional information is provided by thematic products;
and (iv) a scenario in which the noisy class positional information is
provided by explanation methods.

The rest of the paper is organized as follows. Section II surveys the related
data augmentation techniques presented in computer vision (CV) and RS
communities. In Section III, our LP strategy that allows the application of
CutMix in the framework of MLC problems in RS is introduced. Section IV
describes the considered multi-label datasets and the experimental setup,
while the experimental results are presented in Section V. Finally, in Section
VI, the conclusion of the work is drawn.

##  II Related Work

According to our knowledge, there is no data augmentation techniques
specifically designed and developed for MLC problems. Most of the existing
data augmentation techniques are considered to be task-agnostic, and thus,
directly used in MLC problems. Therefore, in this section, we provide a
general overview of data augmentation techniques developed for different image
recognition tasks in the CV and RS communities.

In detail, data augmentation techniques play a crucial role for state-of-the-
art results in supervised [19], [20], [21], semi-supervised [22], [23], [24],
and self-supervised learning [25], [26], [27] in CV. Notably, these techniques
demonstrate their effectiveness in single-label classification (SLC) but also
in related tasks like object detection [28], and semantic segmentation [29].
Similarly to CV, data augmentation techniques significantly improve the
performance and generalization capabilities for RS tasks. Recently,
researchers have explored various approaches to tailor well-established CV
data augmentation techniques for RS tasks, such as SLC [8], [30], [31], [32],
[33], object detection [9], [34], [35], [36], [37], [38], or semantic
segmentation tasks [10], [39]. Joined in their purpose to alleviate
overfitting and to serve as a form of regularization in small data regimes,
the approaches for data augmentation can be broadly divided into three
categories: (i) image modification (Section II-A); (ii) image generation
(Section II-B); or (iii) image pairing (Section II-C) that also includes
CutMix-based approaches.

###  II-A Image Modification based Data Augmentation

The earliest data augmentation techniques developed are based on image
modifications [40]. These techniques aim to encode invariances to data
distortions. Image modifications can be grouped into two variants: (i)
geometric transformations and (ii) channel transformations. Geometric
transformations involve operations like horizontal flips, random crops, pixel
shifts, distortions to orientations, or random erasing (i.e., CutOut) [6],
[7]. Conversely, channel transformations include operations like color
distortions, contrast and brightness adjustments, or histogram equalization.
Both types of transformations are applied successfully to standard CV
benchmarks such as CIFAR [41] and ImageNet [42] (e.g., [43], [44], [45]).

Many of these image modification techniques, in particular geometric
transformations, also belong to the standard training repertoire for various
RS benchmarks [46], [47], [48]. In addition, Ding et al. [30] find
augmentations of translation, pose synthesis, and speckle noise to be
beneficial for the task of vehicle classification on synthetic aperture radar
(SAR) data. Wei Zhang and Yungang Cao [33] address the differences of object
structures in RS SLC in comparison to CV SLC (e.g., smaller objects, less
connected objects, less central objects) by utilizing class activation maps
(CAMs) [49]. CAMs are used to constrain the geometric transformations of
cropping and translation by guaranteeing a minimum area of the relevant pixels
to be retained. Further, Gaussian noise injection is constrained by weighting
it by the inverse of the contribution of pixels to the final prediction.

###  II-B Image Generation based Data Augmentation

In the context of image generation, various data augmentation techniques are
proposed to enhance generalization performance by leveraging synthetically
generated images. These approaches have in common that they employ a
generative adversarial network (GAN) for image generation. In particular, GAN-
based data augmentation techniques show beneficial effects under scarce data
regimes [50], [51] or for few-shot learning tasks [52]. Additionally, Geirhos
et al. [53] improve the robustness of neural networks towards adversarial
examples with a GAN-based ImageNet stylizing approach that guides the network
to prioritize shape over texture.

In RS, image generation approaches are mainly proposed for object detection
tasks focused on vehicles like aircraft, ships, or ground vehicles, in which
images are commonly rare and costly to annotate. While Yan et al. create new
images by integrating existing 3D models of different ships [34] or aircraft
[54] into real RS scenes, Zheng et al. [35] synthesize realistic ground
vehicles by making use of a GAN. Furthermore, Xiao et al. [32] extend the
integration of 3D models of ships by applying a neural style transfer (NST)
network termed Sim2RealNet to eliminate the domain gap between real and
simulated images for single-label ship classification.

###  II-C Image Pairing based Data Augmentation

Data augmentation techniques aligned with the third category involve mixing
existing images with appropriate changes to ground reference labels. Mixup
[55] most prominently creates convex combinations of pairs of images and their
corresponding labels, favoring simple linear behavior in between training
images. Until now, Mixup still forms the basis of many state-of-the-art
approaches in semi-supervised [23] and label noise robust learning [56].
Inspired by the image pairing approach of Mixup, CutMix extends the box
augmentation technique of random erasing (i.e., CutOut) by pasting
corresponding bounding boxes of other training images into the erased areas.
Further extensions of CutMix exist that paste multiple boxes based on the most
attentive regions [57]. The equivalent for CutMix in object detection consists
of copying object instances (i.e., bounding boxes) and paste-blending them
onto diverse backgrounds [58]. On the other hand, in semantic segmentation,
CutMix-inspired approaches commonly termed as copy-paste mechanisms operate at
pixel-level [29]. Moreover, in object detection, Mosaic augmentation further
extends CutMix by stitching four resized images together, followed by a random
crop that retains parts of every image.

Likewise, the majority of adoptions of image pairing approaches in RS is based
on the concepts of the image blending technique Mixup [55], or the box
augmentation technique CutMix [5]. In SLC, Lu et al. [8] propose RSI-Mix, a
variant of CutMix that selects two images of the same category and copy-pastes
a selected bounding box from one image to another. Moreover, Yan et al. [31]
beneficially integrate Mixup into a semi-supervised framework for SLC. In
object detection, Suo et al. [9] elaborate CutMix for artificially increasing
the object density by pasting existing bounding box annotations of ships from
different images onto a single target image. Zhao et al. [37] introduce a
variant of the Mosaic data augmentation technique that puts special emphasis
on rare or hard-to-detect class objects and tackles the problem of class
imbalances via a joint data augmentation technique that makes use of Mixup,
Mosaic, and SMOTE [59]. Further, Zhang et al. [36] propose Object First Mixup,
an alteration to Mixup that adjusts the weights of segmented objects and their
background in semi-supervised object detection. Finally, in semantic
segmentation, You et al. [39] incorporate the copy-paste mechanism (which is
the pixel annotation-based equivalent of CutMix) into a semi-supervised
framework in which the ground reference pixels of annotated roads are pasted
onto unlabeled images. Concurrently, the framework RanPaste introduced by Wang
et al. [10] exploits color jitter and random Gaussian noise for consistency
regularization while making use of a CutMix-like pasting mechanism that pastes
a bounding box (in contrast to pixels of a segment) of a labeled image onto an
unlabeled image. Consequently, a pseudo-label mask for the paired image is
inferred from the predictions of a student model.

In contrast to image modification and image generation approaches as well as
Mixup, the application of CutMix in MLC tasks can lead to the erasure or
addition of class labels in the augmented (i.e., paired) image. To address
this problem, we propose an LP strategy that enables the correct update of the
multi-label of the augmented image for RS MLC.

##  III Proposed Strategy

In this section we introduce our label propagation (LP) strategy that aims to
allow an effective application of CutMix for MLC problems in RS. Let
$\mathcal{D}\coloneqq\\{(\mathbf{x}_{1},\mathbf{y}_{1}),\cdots,(\mathbf{x}_{N},\mathbf{y}_{N})\\}$
be a multi-label training set, where $N\in\mathbb{N}$ is the number of
training images. Each pair $(\mathbf{x}_{i},\mathbf{y}_{i})$ in $D$ consists
of an image $\mathbf{x}_{i}\in\mathbb{R}^{C\times H\times W}$, where $C$ is
the number of image bands with a height $H$ and a width $W$, and its multi-
label vector $\mathbf{y}_{i}\in\\{0,1\\}^{L}$, where $L\in\mathbb{N}$ defines
the number of classes. Each entry $(\mathbf{y}_{i})_{l}$ relates to the
presence (i.e., 1) or absence (i.e., 0) of the respective class $l$. We aim to
train a deep neural network $h$ such that it generalizes well to classify
unseen images correctly. We assume that $|D|$ is not sufficiently large and
that the data annotation process is costly and time-consuming. To overcome
this issue, we adopt the CutMix algorithm due to its proven success into our
training procedure for improving the performance of $h$. However, as mentioned
before, the labels in MLC do not explicitly indicate class positional
information. Therefore, applying CutMix to training images annotated with
multi-labels can lead to the potential erasure or addition of class labels in
the paired image and, thus, may lead to multi-label noise. To address this
problem, we propose an LP strategy that aims to preserve the correct class
information within the multi-label vector of the augmented image to enable an
effective application of CutMix in MLC. In detail, for accessing pixel-level
class positional information that is incorporated by our LP strategy into the
CutMix process, we consider two scenarios: (i) the usage of reference maps
that correspond to the multi-labels, or (ii) the usage of class explanation
masks provided by explanation methods if no reference maps are available. In
the following, we initially define the algorithm for CutMix in MLC.
Thereafter, our LP strategy for CutMix in MLC is introduced for both
scenarios. Finally, the section is concluded by a specification of the box
creation algorithm as well as the application procedure of CutMix with our LP
strategy to an MLC task.

###  III-A Definition of the CutMix Algorithm

Let a box (i.e., rectangular region) $R=(r_{a},r_{b},r_{c},r_{d})$ be
specified through the left lower corner $(r_{a},r_{b})$ and the right upper
corner $(r_{c},r_{d})$, respectively. Then, the area of the box is denoted as
$A_{R}=(r_{c}-r_{a})(r_{d}-r_{b})$. Further, the binary mask
$B_{R}=(b_{j_{1}j_{2}})_{j_{1}j_{2}}\in\mathbb{R}^{H\times W}$ corresponding
to a box $R$ is defined, where $b_{j_{1}j_{2}}$ is defined for
$j_{1},j_{2}=1,...,n$ as follows:

$\displaystyle b_{j_{1}j_{2}}=\begin{cases}1,\textrm{ if }r_{a}\leq j_{1}\leq
r_{c}\textrm{ and }r_{b}\leq j_{2}\leq r_{d}\\\ 0,\textrm{
otherwise.}\end{cases}$ (1)

The goal of CutMix is to generate an augmented training image
$\mathbf{\tilde{x}}$ with a corresponding multi-label vector
$\mathbf{\tilde{y}}$ by combining two existing training images
$\mathbf{x}_{1},\mathbf{x}_{2}$ and its corresponding label vectors
$\mathbf{y}_{1},\mathbf{y}_{2}$. Therefore, the two boxes $R_{1}$ and $R_{2}$,
and their corresponding binary masks $B_{1}$ and $B_{2}$ are generated. To
erase $R_{1}$ from $\mathbf{x}_{1}$, the inverse binary mask $(1-B_{1})$ is
combined with $\mathbf{x}_{1}$ through the Hadamard product, as
$(1-B_{1})\odot\mathbf{x}_{1}$. Further, the box $R_{2}$ is extracted from
$\mathbf{x}_{2}$ by applying the Hadamard product between $\mathbf{x}_{2}$ and
the binary mask $B_{2}$, as $B_{2}\odot\mathbf{x}_{2}$. To fill the erased
area in $\mathbf{x}_{1}$ with the box selected from $\mathbf{x}_{2}$, the two
boxes have to be spatially aligned. This alignment is carried out by the image
shift operator $T^{x}_{R_{2}R_{1}}:\mathbb{R}^{C\times H\times
W}\to\mathbb{R}^{C\times H\times W}$. To explain in detail the working
principle of the image shift operator for a single image band (i.e., when
$C=1$), let us assume that the unaligned matrices $B_{1}\odot\mathbf{x}_{1}$
and $B_{2}\odot\mathbf{x}_{2}$ have the following block matrix structures:

$\displaystyle B_{1}\odot\mathbf{x}_{1}$ $\displaystyle=\begin{pmatrix}0&0\\\
0&\mathbf{x}_{R_{1}}\end{pmatrix}$ (2) $\displaystyle
B_{2}\odot\mathbf{x}_{2}$
$\displaystyle=\begin{pmatrix}\mathbf{x}_{R_{2}}&0\\\ 0&0\end{pmatrix}.$ (3)

Then, $T_{R_{2}R_{1}}^{x}$ shifts $\mathbf{x}_{R_{2}}$ to the position of
$\mathbf{x}_{R_{1}}$ as:

$\displaystyle
T_{R_{2}R_{1}}^{x}(B_{2}\odot\mathbf{x}_{2})=\begin{pmatrix}0&0\\\
0&\mathbf{x}_{R_{2}}\end{pmatrix}.$ (4)

This shift can be represented by (channel-wise) matrix multiplication with
block-identity matrices. To generate the corresponding multi-label vector
$\mathbf{\tilde{y}}$, $\mathbf{y}_{1}$ and $\mathbf{y}_{2}$ are weighted by
the respective remaining sizes of the paired image parts of $\mathbf{x}_{1}$
and $\mathbf{x}_{2}$. Then, the augmented pair
$(\mathbf{\tilde{x}},\mathbf{\tilde{y}})$ is defined by the two pairing
operations:

$\displaystyle\mathbf{\tilde{x}}$
$\displaystyle=(1-B_{1})\odot\mathbf{x}_{1}+T_{R_{2}R_{1}}^{x}(B_{2}\odot\mathbf{x}_{2})$
(5) $\displaystyle\mathbf{\tilde{y}}$
$\displaystyle=(1-A_{R})\mathbf{y}_{1}+A_{R}\mathbf{y}_{2}.$ (6)

It is worth noting that there exist multiple versions of CutMix. In this work,
CutMix with unaligned positions of the boxes $R_{1}$ and $R_{2}$ is considered
to increase the diversity of the augmented images. For further details, we
refer the reader to the original CutMix paper [5].

###  III-B Proposed Label Propagation Strategy for CutMix in Multi-Label
Remote Sensing Image Classification

\begin{overpic}[width=390.25534pt,clip,trim=0.0pt 0.0pt 0.0pt
28.45274pt]{images/method/src_schematic_illustra.pdf}
\small\put(10.6,-1.0){Step (1)} \put(45.5,-1.0){Step (2)} \put(82.5,-1.0){Step
(3)} \end{overpic}

Figure 1:  Schematic illustration of the LP strategy for CutMix in MLC
leveraging reference maps. Step (1): the training images and associated
reference maps $(\mathbf{x}_{1},\mathbf{m}_{1})$,
$(\mathbf{x}_{2},\mathbf{m}_{2})$ are selected. Step (2): the binary mask
$B_{1}$ erases a box from $(\mathbf{x}_{1},\mathbf{m}_{1})$, and the binary
mask $B_{2}$ extracts a box of the same size from
$(\mathbf{x}_{2},\mathbf{m}_{2})$, respectively. Step (3): the augmented
training image $\mathbf{\tilde{x}}$ and reference map $\mathbf{\tilde{m}}$ are
created by leveraging the shift operators to replace the erased area of
$(\mathbf{x}_{1},\mathbf{m}_{1})$ with the extracted box from
$(\mathbf{x}_{2},\mathbf{m}_{2})$. The reference map $\mathbf{\tilde{m}}$ is
then used to derive the updated multi-label vector $\mathbf{\tilde{y}}$ for
$\mathbf{\tilde{x}}$ through the function $\phi$.

The proposed LP strategy aims to preserve all class labels in the augmented
image by leveraging pixel-level class positional information. In detail, we
consider two scenarios in which each training image is associated with such
class positional information. In the first scenario, the training images are
associated with reference maps that directly correspond to the multi-label of
the image. In the second scenario, the training images are associated with
class explanation masks that are obtained by an explanation method if no
reference maps are available. Analogously to the application of the pairing
operation for two images in (5), our LP strategy additionally applies this
operation to the pixel-level class positional information of the two images
and derives the multi-label vector from the paired information. In the
following, the variants of our LP strategy for the respective two scenarios to
effectively apply CutMix in MLC are described.

Scenario 1: Label Propagation based on Reference Maps

In the first scenario, we assume the existence of associated reference maps.
Following the general notation of
$[k_{1},k_{2}]_{\mathbb{N}}\coloneqq\\{j\in\mathbb{N}\;|\;k_{1}\leq j\leq
k_{2}\\}$ for $k_{1},k_{2}\in\mathbb{N}$, for each of the pairs
$(\mathbf{x}_{i},\mathbf{y}_{i})$ in $D$, let
$\mathbf{m}_{i}\in[0,L]_{\mathbb{N}}^{H\times W}$ be a pixel-level reference
map associated with the image $\mathbf{x}_{i}$, where each entry in
$\mathbf{m}_{i}$ indicates the class label of one of the classes present in
$\mathbf{y}_{i}$. In this scenario, we apply a pairing operation analogous to
the pairing of $\mathbf{x}_{1},\mathbf{x}_{2}$ for the two reference maps
$\mathbf{m}_{1},\mathbf{m}_{2}$. Therefore, we define the reference map shift
operator $T^{m}_{R_{2}R_{1}}:[0,L]_{\mathbb{N}}^{H\times
W}\to[0,L]_{\mathbb{N}}^{H\times W}$. Further, a read-out function
$\phi:[0,L]_{\mathbb{N}}^{H\times W}\to\\{0,1\\}^{L}$ is defined to extract
the class positional information of the updated reference map
$\mathbf{\tilde{m}}$ to generate $\mathbf{\tilde{y}}$. In particular, $\phi$
returns the set over all entries (class labels) of the reference map
$\mathbf{\tilde{m}}$. Therefore, integrating the LP strategy for CutMix yields
the augmented pair $(\mathbf{\tilde{x}},\mathbf{\tilde{y}})$, where

$\displaystyle\mathbf{\tilde{x}}$
$\displaystyle=(1-B_{1})\odot\mathbf{x}_{1}+T_{R_{2}R_{1}}^{x}(B_{2}\odot\mathbf{x}_{2})$
(7) $\displaystyle\mathbf{\tilde{m}}$
$\displaystyle=(1-B_{1})\odot\mathbf{m}_{1}+T_{R_{2}R_{1}}^{m}(B_{2}\odot\mathbf{m}_{2})$
(8) $\displaystyle\mathbf{\tilde{y}}$
$\displaystyle=\phi(\mathbf{\tilde{m}}).$ (9)

An illustration of the proposed LP strategy that allows the application of
CutMix in multi-label classification problems can be found in Figure 1. First,
two training images $\mathbf{x}_{1},\mathbf{x}_{2}$ with their associated
reference maps $\mathbf{m}_{1},\mathbf{m}_{2}$ are selected (Step 1). Then,
analogously to CutMix in SLC, two boxes $R_{1}$ and $R_{2}$ and their
corresponding binary masks $B_{1}$ and $B_{2}$ are generated to extract boxes
from $(\mathbf{x}_{2},\mathbf{m}_{2})$ and erase the inverse counterpart boxes
from $(\mathbf{x}_{1},\mathbf{m}_{1})$ (Step 2). The pairing of the masked
images and reference maps is carried out by the shift operators
$T^{x}_{R_{2}R_{1}}$ and $T^{m}_{R_{2}R_{1}}$. Finally, the multi-label vector
$\mathbf{\tilde{y}}$ is read-out by $\phi$ from the updated reference map
$\mathbf{\tilde{m}}$ to yield the augmented pair
$(\mathbf{\tilde{x}},\mathbf{\tilde{y}})$ (Step 3).

Scenario 2: Label Propagation based on Class Explanation Masks

In the second scenario, we assume that there is no associated reference maps
available. To address this issue, we propose to use explanation methods to
provide the source for pixel-level class positional information that can be
leveraged by our LP strategy. Let each of the pairs
$(\mathbf{x}_{i},\mathbf{y}_{i})$ in $D$ be associated with a set of class
explanation masks $\mathbf{e}_{i}\in\\{0,1\\}^{L\times H\times W}$ generated
by an explanation method. An entry of one in
$(\mathbf{e}_{i})_{l}\in\\{0,1\\}^{H\times W}$ corresponds to a pixel (i.e.,
activating pixel) in the image that is attributed importance for predicting
class $l$. To enable the pairing operation for two sets of class explanation
masks $\mathbf{e}_{1},\mathbf{e}_{2}$, we define the class explanation mask
shift operator $T^{e}_{R_{2}R_{1}}:\\{0,1\\}^{L\times H\times
W}\to\\{0,1\\}^{L\times H\times W}$. Further, a read-out function
$\psi:\\{0,1\\}^{L\times H\times W}\to\\{0,1\\}^{L}$ is defined to derive
$\tilde{\mathbf{y}}$ from the updated class explanation mask
$\mathbf{\tilde{e}}$. To address the higher uncertainty between activating and
non-activating pixels in border regions of the class explanation masks, this
function returns a class $l$ as present in the updated multi-label vector
$\mathbf{\tilde{y}}$ if the total number of activating pixels is greater than
a minimum pixel threshold $t_{\text{map}}$. Note that if a class does not
belong to the actual multi-label of the corresponding image, the whole
explanation mask of this class is set to zero. Thus, it is ensured to only
derive reasonable classes in the updated multi-label vector
$\mathbf{\tilde{y}}$. Consequently, when leveraging class explanation masks,
the augmentation procedure to generate $\tilde{\mathbf{x}}$ is equivalent to
(7), while the notation to derive $\tilde{\mathbf{y}}$ from a set of updated
class explanation mask $\tilde{\mathbf{e}}$ is similar to (8) and (9), as

$\displaystyle\mathbf{\tilde{e}}$
$\displaystyle=(1-B_{1})\odot\mathbf{e}_{1}+T_{R_{2}R_{1}}^{e}(B_{2}\odot\mathbf{e}_{2})$
(10) $\displaystyle\mathbf{\tilde{y}}$
$\displaystyle=\psi(\mathbf{\tilde{e}}).$ (11)

It is worth noting that the class explanation masks can be provided by any
explanation method. While some explanation methods directly output class
explanations in the form of binary masks, other methods only provide class
explanation heatmaps, where each pixel is associated with a probability of
contributing to the prediction of the respective class, e.g., DeepLift [60] or
GradCAM [61]. In this case, we propose to convert the heatmaps into a binary
mask using a threshold $t_{\text{cam}}$.

Generation of Boxes in CutMix

It is necessary to generate different boxes with various side lengths and
areas throughout the CutMix process. Thus, to meet the requirement of
controlling the box sizes, the implementation of CutMix needs to incorporate
an input of a minimal area $A_{\min}>0$ and a maximal area $A_{\max}\geq
A_{\min}>0$ of the box, denoted as the box size range $A_{min}-A_{max}$. Thus,
for a box $R=(r_{a},r_{b},r_{c},r_{d})$ the points are required to fulfill the
inequality:

$\displaystyle A_{\min}\leq(r_{c}-r_{a})(r_{d}-r_{b})\leq A_{\max}.$ (12)

We propose an algorithm for creating the points that satisfy (12). It operates
batch-wise by creating coordinates $(r_{a},r_{b},r_{c},r_{d})$ for $N_{R}$
boxes, which is the number of boxes needed for the batch-wise application of
CutMix. The first step is to sample a list of box coordinates $\mathcal{B}$
uniformly from $[0,H]_{\mathbb{N}}\times[0,W]_{\mathbb{N}}$. Thus, it is
iterated until $N_{R}$ points satisfy (12). If there are less than $N_{R}$
points, every pair of points violating the condition is re-sampled. The
pseudo-code of the algorithm is presented in Algorithm 1.

Algorithm 1 Creation of Boxes

1:$A_{\min}>0$; $A_{\max}\geq A_{\min}>0$; $N_{R},H,W\in\mathbb{N}$.

2:List of $N_{R}$ valid random boxes.

3:Initialize empty list of boxes $\mathcal{B}$

4:while $|\mathcal{B}|<N_{R}$ do

5: Sample $\bar{r}_{a}$, $\bar{r}_{c}$ uniformly from $[0,H]_{\mathbb{N}}$

6: Sample $\bar{r}_{b}$, $\bar{r}_{d}$ uniformly from $[0,W]_{\mathbb{N}}$

7: $r_{a}=\min(\bar{r}_{a},\bar{r}_{c})$;
$r_{c}=\max(\bar{r}_{a},\bar{r}_{c})$

8: $r_{b}=\min(\bar{r}_{b},\bar{r}_{d})$;
$r_{d}=\max(\bar{r}_{b},\bar{r}_{d})$

9: Calculate area $A=(r_{c}-r_{a})(r_{d}-r_{b})$

10: if $A_{\min}\leq A\leq A_{\max}$ then

11: Append ($r_{a}$, $r_{b}$, $r_{c}$, $r_{d}$) to $\mathcal{B}$

12: end if

13:end while

14:return $\mathcal{B}$

Multi-Label Scene Classification with CutMix and the proposed Label
Propagation Strategy

In the training phase of a deep neural network $h$, CutMix is applied to the
selected images with a probability $p$. In each training step, after loading a
batch from $\mathcal{D}$, random images $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$
with their respective multi-label vectors $\mathbf{y}_{1}$ and
$\mathbf{y}_{2}$ and reference maps $\mathbf{m}_{1}$ and $\mathbf{m}_{2}$ (or
class explanation masks $\mathbf{e}_{1}$ and $\mathbf{e}_{2}$) are selected
for the LP strategy enhanced augmentation process. The process yields a
replacement of $\mathbf{x}_{1}$ and $\mathbf{y}_{1}$ by probability $p$ with
the augmented image $\mathbf{\tilde{x}}$ and its updated multi-label vector
$\mathbf{\tilde{y}}$ that is derived by our LP strategy based on a pairing
operation of the reference maps. It is worth noting that the one-to-one
replacement ensures that the batch size is persistent. After training the deep
neural network $h$ with CutMix-augmented images together with our proposed LP
strategy, the predicted multi-labels are assigned to images in the considered
archive.

##  IV Dataset Description and Experimental Design

(a)

(b)

(c)

Figure 2: Example images of the datasets: 2(a) DeepGlobe-ML, 2(b) BigEarthNet-S2, and 2(c) TreeSatAI. Table I: Characteristics of the multi-label scene classification datasets used in our experiments. Dataset | $|\mathcal{D}|$ |  $L$ | Avg. $L$ |  $C$ |  Image Sizes (Spatial Resolution) | Pixel-Level Reference Maps
---|---|---|---|---|---|---
per Image |
DeepGlobe-ML | $18.185$ | $6$ | $1.71$ | $3$ |  120$\times$120 (0.5m) | Manually Annotated
BigEarthNet-S2 [18] | $250.249$ | $19$ | $2.95$ | $10$ |  120$\times$120 (10m), 60$\times$60 (20m) | Thematic Product
TreeSatAI [62] | $40.332$ | $15$ | $1.59$ | $4$ |  304$\times$304 (0.2m) | Not available

In the experiments, we have used three RS multi-label datasets: 1) DeepGlobe-
ML, which is a multi-label dataset that we constructed from the DeepGlobe Land
Cover Classification Challenge dataset [12], 2) BigEarthNet-S2 [18], and 3)
TreeSatAI [62]. For each dataset, example images are shown in Figure 2. A
comparison of the main characteristics of the considered datasets is provided
in Table I, while the datasets are briefly described in the following.

###  IV-A Datasets

####  IV-A1 DeepGlobe-ML

The original DeepGlobe Land Cover Classification Challenge dataset [12] is
collected from the DigitalGlobe +Vivid Images dataset that contains 1949 RGB
tiles of size $2448\times 2448$ pixels with a spatial resolution of
$0.5\text{\,}\mathrm{m}$ acquired over Thailand, Indonesia, and India. Each
tile is associated with a manually annotated ground reference map. The classes
comprise urban, agriculture, rangeland, forest, water, barren, and unknown. To
ensure one MLC dataset associated with reliable reference maps in this paper,
we constructed the DeepGlobe-ML from the original pixel-level classification
dataset. To this end, all tiles were divided into a grid of $120\times
120$-pixel patches. The respective multi-labels were then derived from the
label information of the associated $120\times 120$-pixel reference maps. We
discarded all patches containing the class unknown. To account for an
appropriate MLC dataset, we considered only $20\text{\,}\mathrm{\char
37\relax}$ of the patches with a single present class and all patches with
more than one present class, resulting in an average of $1.71$ present classes
per patch (cf., $1.28$ present classes without discarding patches). In total,
DeepGlobe-ML includes $30,443$ patches and is split into a training set
($60\text{\,}\mathrm{\char 37\relax}$), a validation set
($20\text{\,}\mathrm{\char 37\relax}$), and a test set
($20\text{\,}\mathrm{\char 37\relax}$).

####  IV-A2 BigEarthNet-S2

BigEarthNet-S2 [18] is a multi-label dataset that includes $590,326$
Sentinel-2 multispectral images acquired over ten countries in Europe. The
LULC class annotations are obtained from reference maps that originate from
the publicly available thematic product CORINE Land Cover Map inventory of
2018 [15]. Following the LULC class nomenclature proposed in [63], each image
is annotated with a subset of $19$ LULC classes, including different types of
forests, water, or complex urban or agricultural classes. The dataset contains
an average of $2.95$ present classes per image. In the experiments, a filtered
subset that excludes patches with seasonal snow, clouds, and cloud shadows is
used. The selected subset is divided into a training set
($50\text{\,}\mathrm{\char 37\relax}$), a validation set
($25\text{\,}\mathrm{\char 37\relax}$), and a test set
($25\text{\,}\mathrm{\char 37\relax}$). Furthermore, only $10$ bands (out of
the originally $13$ bands) with a spatial resolution of
$10\text{\,}\mathrm{m}$ ($120\times 120$-pixel) and $20\text{\,}\mathrm{m}$
($60\times 60$-pixel) are considered. It is worth noting that a preliminary
version of the currently ongoing label update is used in our experiments.

####  IV-A3 TreeSatAI

The TreeSatAI dataset [62] is a multi-label tree species classification
benchmark dataset that consists of $50,381$ image-triplets of a high-
resolution aerial image, a Sentinel-1 SAR and a Sentinel-2 multispectral image
acquired over the German federal state of Lower Saxony. In our experiments, we
only use the aerial images. Each of the Each of the $304\times 304$-pixel
aerial images includes the RGB and a near-infrared band with a spatial
resolution of $0.5\text{\,}\mathrm{m}$ and is associated with multi-labels
representing different species of trees from 15 genus classes. The labels were
derived from forest administration data of the federal state of Lower Saxony
that was annotated in the field or by photo interpretation. For the
experiments, we chose a threshold of $5\text{\,}\mathrm{\char 37\relax}$
coverage of the pixels for a class to be annotated as present in the image.
Further, we updated the original train-test split by also considering a
validation set, yielding a training set of $80\text{\,}\mathrm{\char
37\relax}$, a validation set of $10\text{\,}\mathrm{\char 37\relax}$, and a
test set of $10\text{\,}\mathrm{\char 37\relax}$ of the images. On average,
there are $1.59$ different tree species present in each image. It is worth
noting that there exist no further pixel-level reference maps associated with
this dataset. The application of our LP strategy is only possible when
leveraging explanation methods (i.e., generating class explanation masks).

###  IV-B Experimental Setup

For all our experiments, we use a randomly initialized ResNet18 from the
pytorch-image-models library [64]. We choose a batch size of 300 and the
cosine annealing learning rate scheduling with a start rate set to 5e-4 and
warm-up iterations based on the number of steps. Each model is trained for 120
epochs. For reproducibility reasons, the global random state is fixed to seeds
in the range of 42 to 46 implemented by the seed-everything functionality in
PyTorch lightning [65]. We conduct every experiment with these five seeds and
report the average of the test set performance in mean average precision macro
(denoted as mAP macro) of the best model determined by the validation set. All
experiments are carried out on an A100 SXM4 80GB GPU.

##  V Experimental Results

The experimental results are organized as follows. In Section V-A, we assess
the effectiveness of our LP strategy for CutMix in RS MLC when reliable pixel-
level reference maps are available. Subsequently, in Section V-B, we conduct
an extensive ablation study on the effect of noisy reference maps on our LP
strategy. To this end, we define and simulate noise types that can be related
to different types of labeling errors. To further show the benefits in real
use-cases beyond simulated noisy reference maps, in Section V-C, we study the
effects of our LP strategy in a scenario in which the class positional
information is extracted from noisy pixel-level reference maps derived from a
thematic product. Finally, in Section V-D, we show the advantage of our LP
strategy in a scenario in which we do not have access to reference maps and
instead leverage the class positional information provided by an explanation
method.

###  V-A Label Propagation based on Reliable Reference Maps

To evaluate the potential of leveraging pixel-level reference maps to
correctly update the multi-labels during the application of CutMix in RS MLC,
we initially study the effects of our proposed LP strategy when reliable
reference maps are available. To this end, we report the results on the
DeepGlobe-ML dataset. Therefore, we compare three different box augmentation
techniques: i) CutOut; ii) CutMix without our proposed LP strategy (denoted as
CutMix w/o LP); and CutMix with our proposed LP strategy (denoted as CutMix w
LP). As these techniques may favor different sizes of boxes, we conduct
experiments for the value of the box size range $A_{min}-A_{max}$ as $0.10.3$,
$0.10.5$, $0.10.7$, $0.30.5$, and $0.30.7$. All techniques are applied with a
probability $p$ of 0.5. The results are shown in Table II. It can be noted
that our proposed LP strategy is effective under all tested box sizes. In
addition, for each box size, it improves an application of CutMix without LP.
In particular, it can be observed that the larger the selected box sizes
become, the more beneficial it is to apply our LP strategy. While for the box
size range of $0.30.7$ CutMix w LP results in the best overall performance of
all methods with $82.30\text{\,}\mathrm{\char 37\relax}$ in mAP macro, the box
size range also yields one of the largest improvements to CutMix w/o LP. This
improvement can be attributed to the highest probability of deleting class
labels from one of the paired images. Further, in contrast to both CutMix
techniques, it is worth noting that the performance of CutOut is more or less
constant (i.e., invariant) for all evaluated box sizes except for very small
box sizes (i.e., $0.10.3$).

Table II: Results in mAP macro ($\mathrm{\char 37\relax}$) for DeepGlobe-ML
with associated reliable reference maps. Box Augmentation
---
| Box Size Range
Method | $0.10.3$ | $0.10.5$ | $0.10.7$ | $0.30.5$ | $0.30.7$
CutOut [6] | $79.56$ | $79.86$ | $79.88$ | $80.10$ | $80.20$
CutMix w/o LP | $78.93$ | $79.17$ | $79.76$ | $80.25$ | $80.49$
CutMix w LP | 80.33 | 80.74 | 80.72 | 82.21 | 82.30
No Augmentation
Baseline | $78.40$ |  |  |  |

###  V-B Label Propagation based on Simulated Noisy Reference Maps

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: Examples of simulated noise in pixel-level reference maps: 3(a)
original image, 3(b) original reference map, 3(c) Mask Shift, 3(d)
Dilation/Erosion, 3(e) Rectify Borders, 3(f) Border Deformation, 3(g) Segment
Swap, and 3(h) Class Swap.

Only in rare cases there exist reliable pixel-level reference maps. In many
scenarios, multi-labels are based on not fully reliable reference maps, and
sometimes, even the estimation of class positional information via explanation
methods is necessary. To better understand the effectiveness of our LP
strategy, we conduct an extensive ablation study for six different types of
noise in reference maps that simulate real scenarios. To this end, we define
the following noise types: (i) Mask Shift, (ii) Dilation/Erosion, (iii)
Rectify Borders, (iv) Border Deformation, (v) Segment Swap, and (vi) Class
Swap. These noise types are studied to show the generalizability of our LP
strategy to any arbitrary source of noisy class positional information. In the
following, we illustrate intuitive examples per noise type. Nonetheless, many
of the noise types can be associated with various scenarios. An illustration
of the different noise types is shown in Figure 3. The example image is
depicted in Figure 3(a) and the associated reference map in Figure 3(b).

Mask Shift. Mask Shift is associated with an imperfect alignment of the
reference map with the considered image (see Figure 3(c)). The most prominent
example relates to an imprecise co-registration of image and label source if
the class positional information is derived from a thematic product. To
simulate this noise type, for a fraction $f$ (in $\mathrm{\char 37\relax}$) of
reference maps associated with the training images, we apply a shift in a
uniformly sampled direction with a uniformly sampled pixel shift taken from
the interval of 0 and a maximum pixel shift $\alpha_{\text{pixel\\_shift}}$.

Dilation/Erosion. Dilation/Erosion is associated with an inaccurate border of
one consecutive class segment (i.e., an area that covers an individual class)
that is proportionally enlarged or downsized (see Figure 3(d)). The most
prominent example relates to the temporal difference between the image and
label source if the class positional information is derived from a thematic
product, e.g., a partially dried-out lake or a burned forest. To simulate this
noise type, for a fraction $f$ (in $\mathrm{\char 37\relax}$) of reference
maps associated with the training images, we uniformly select one class
segment and either dilate or erode the segment for
$\alpha_{\text{iterations}}$ iterations.

Rectify Borders. Rectify Borders simulates pixel-level reference maps that are
highly coarse (see Figure 3(e)). The most prominent example relates to
different resolutions of image and label source if the class positional
information is derived from a thematic product. To simulate this noise type,
for a fraction $f$ (in $\mathrm{\char 37\relax}$) of reference maps associated
with the training images, we first downsample its resolution by a magnitude of
$\alpha_{\text{magnitude}}$ and then resize it back to its original
resolution.

Border Deformation. Border Deformation is associated with inaccurate borders
between class segments of the reference map (see Figure 3(f)). The most
prominent examples relate to (i) human annotation errors in the thematic
product or the pixel-wise annotation of the considered image, (ii) inaccurate
predictions of class segments if the class positional information is derived
from the inference of a segmentation model (i.e., pixel-based classification
model), or (iii) temporal differences between the considered image and
thematic product, e.g., an urban area that was extended by a few buildings. To
simulate this noise type, for a fraction $f$ (in $\mathrm{\char 37\relax}$) of
reference maps associated with the training images, we place
$\alpha_{\text{max\\_def}}$ small bounding boxes of varying size onto each
reference map. If a bounding box overlays the border between two or more class
segments, the whole bounding box gets assigned to one of the bordering classes
selected at random. Otherwise, the bounding box gets assigned to the class it
overlays (i.e., no alteration to the reference map is applied).

Segment Swap. Segment Swap is associated with an inaccurate positional
assignment of a class (see Figure 3(g)). The most prominent example relates to
a case in which the multi-labels are not associated with any reference maps
and explanation methods are employed to estimate the class positional
information, e.g., the class explanation masks associate the wrong parts of
the image with a specific present class. To simulate this noise type, for a
fraction $f$ (in $\mathrm{\char 37\relax}$) of reference maps associated with
the training images, we remove one class segment and randomly place it in an
aleatory polygon shape at a different location in the same reference map.

Class Swap. Class Swap is associated with inaccurate class assignments to
class segments (see Figure 3(h)). The most prominent example relates to human
annotation errors in the thematic product or in the pixel-wise annotation of
the considered image, e.g., an ambiguous class is confused by the annotator.
Another example relates to the incorrect prediction of class segments if the
class positional information is derived from the inference of a segmentation
model (i.e., pixel-based classification model). To simulate this noise type,
we assign a uniformly sampled new class for a fraction $f$ (in $\mathrm{\char
37\relax}$) of all class segments contained in all reference maps of the
training set. It is worth noting that this noise type leads to multi-label
noise. Thus, in this setting, we also update all multi-labels that are
associated with noisy reference maps when comparing the performance of CutMix
with and without our LP strategy.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4: Results of simulated noise in pixel-level reference maps for
DeepGlobe-ML: 4(a) Mask Shift, 4(b) Dilation/Erosion, 4(c) Rectify Borders,
4(d) Border Deformation, 4(e) Segment Swap, 4(f) Class Swap.

Figure 4 shows the results associated with different types of noise in the
reference maps for the DeepGlobe-ML dataset. All our experiments for CutMix w
LP are carried out with the best box size range of $0.30.7$ and applied with a
probability $p$ of 0.5. The results are compared to CutMix w/o LP for the same
box size range (see Section V-A). For Mask Shift, Dilation/Erosion, Rectify
Border, and Border Deformation we applied the noise with varying intensity to
different fractions of the reference maps ($25\text{\,}\mathrm{\char
37\relax}$, $50\text{\,}\mathrm{\char 37\relax}$, $100\text{\,}\mathrm{\char
37\relax}$). For Mask Shift, in all the considered settings, our proposed LP
strategy yields better performance than CutMix without LP (see Figure 4(a)).
While a maximum shift of 12 and 24 pixels ($10\text{\,}\mathrm{\char
37\relax}$ and $20\text{\,}\mathrm{\char 37\relax}$ of the height) barely
impacts model performance under any fraction of affected reference maps, only
a strong noise induction of a maximum pixel shift of 36
($30\text{\,}\mathrm{\char 37\relax}$ of the height) has an observable
negative impact on the overall performance if at least half of the reference
maps are affected. Nonetheless, in comparison to the same setup without
simulated noise (for same box size range, see Table II) it still results in a
performance that is $1.5\text{\,}\mathrm{\char 37\relax}$ better in mAP macro.
In contrast, the dilation or erosion of class segments significantly decreases
the CutMix w LP performance already at lighter intensities (see Figure 4(b)).
We observe a linear decrease in performance for the parameter of the fraction
of reference maps as well as the iterations of applying dilation/erosion to a
class segment, leading to the lowest mAP macro performance of
$80.75\text{\,}\mathrm{\char 37\relax}$ for $\alpha_{\text{iteration}}$ of 36
iterations. This setting yields similar performance to CutMix w/o LP, while
our LP strategy yields better results under any other setting. A similar
linear trend can be observed for noise in the form of a different resolution
of reference maps than the images (see Figure 4(c)). However, its negative
impact is less significant than Dilation/Erosion and the performance decrease
is only minor. Under all tested settings, our LP strategy leads to higher
performance than CutMix w/o LP. On the other hand, border deformations (see
Figure 4(d)) do not seem to have an impact on model performance at all. While
the performances of different parameter settings follow no clear pattern, none
of them yields a deviation from the results under reliable reference maps of
more than $0.2\text{\,}\mathrm{\char 37\relax}$ in mAP macro. This can be
explained by the preservation of a high intersection over union between the
reliable and the noisy reference maps as only the border regions are
alternated. When entire class segments are moved to different areas in the
reference map in a distinct shape, a linear trend can be observed between the
fraction of segments moved and the performance of the model (see Figure 4(e)).
Nevertheless, even for $30\text{\,}\mathrm{\char 37\relax}$ of the reference
maps affected by segment swaps, leveraging our proposed LP strategy still
improves CutMix w/o LP by about $1.5\text{\,}\mathrm{\char 37\relax}$ in mAP
macro. For Class Swap, as changing the annotated class for segments leads to
multi-label noise, each of the box size range settings for CutMix w/o LP had
to be retrained. In line with the results without noise, for all three tested
fractions of swapped class segments, the improvement of employing CutMix with
our proposed LP strategy stays constant at around $1.5\text{\,}\mathrm{\char
37\relax}$ in mAP macro (see Figure 4(f)). Under incorrectly annotated
classes, the application of CutMix with our proposed LP strategy remains more
effective than without it.

In summary, it was observed that inaccurate borders due to border deformation
as well as annotation errors in terms of swapped class segments, do not impact
the effectiveness of our proposed LP strategy at all. On the other hand, our
experiments revealed some sensitivity to the shift of reference maps, low-
resolution reference maps, the dilation/erosion, and the replacement of
individual class segments. In particular, strong dilation or erosion of class
segments that affect more than $50\text{\,}\mathrm{\char 37\relax}$ of the
reference maps may lead to similar performance to CutMix w/o LP when our LP
strategy is employed. However, in all other tested settings, the decrease in
performance due to noisy reference maps is not substantial. Thus, it is still
beneficial to employ our proposed LP strategy over CutMix without a proper
label update for all simulated noise types. This is due to the fact that the
robustness of our LP strategy to noisy class positional information can be
attributed to the fact that rough estimates are sufficient to correctly update
the multi-labels. If a class covers a larger area in one of the remaining
parts of the paired images, a single correctly annotated pixel in the
reference map is enough to effectively update the multi-label of the augmented
image using our LP strategy. The results indicate that the cases of wrong
updates are less significant than the loss in information when our LP strategy
is not used.

###  V-C Label Propagation based on Noisy Reference Maps from Thematic
Products

To evaluate the effectiveness of our LP strategy in a scenario in which class
positional information is provided in the form of a thematic product, we adopt
the same setup from Section V-A for the BigEarthNet-S2 dataset. In line with
the findings of our ablation study on simulated noisy reference maps, the
usage of our LP strategy results in an improvement over CutMix without an
effective label update (see Table III). Under all box size ranges tested,
CutMix w LP consistently outperforms CutMix w/o LP. It is worth noting that
the largest improvements are similar to those obtained for DeepGlobe-ML (see
V-A) (about $2.0\text{\,}\mathrm{\char 37\relax}$ in mAP macro) even though
the class positional information contains more noise due to the usage of
reference maps derived from a thematic product. The noisy class positional
information originating from a thematic product does not impact the
effectiveness of our LP strategy. Further, unlike DeepGlobe-ML, the box size
range does not seem to play an important role in the generalization
performance. CutMix w LP achieves the best results of about
$76.7\text{\,}\mathrm{\char 37\relax}$ in mAP macro for the box size ranges
$0.10.3$, $0.30.5$, and $0.30.7$; a clear trend in the reliance on smaller or
larger boxes cannot be observed.

Table III: Results in mAP macro ($\mathrm{\char 37\relax}$) for BigEarthnet-S2
with associated noisy reference maps derived from a thematic product. Box
Augmentation
---
| Box Size Range
Method | $0.10.3$ | $0.10.5$ | $0.10.7$ | $0.30.5$ | $0.30.7$
CutOut [6] | $73.32$ | $74.03$ | $73.85$ | $74.59$ | $74.68$
CutMix w/o LP | $75.42$ | $74.90$ | $74.69$ | $74.52$ | $75.39$
CutMix w LP | 76.72 | 76.32 | 76.05 | 76.76 | 76.72
No Augmentation
Baseline | $71.46$ |  |  |  |

###  V-D Label Propagation based on Class Explanation Masks

Finally, to show the generalization capability of our proposed LP strategy to
scenarios in which no associated pixel-level reference maps are available, we
conduct experiments on the TreeSatAI dataset. In this experiment, we apply the
LP strategy to class explanation masks derived from an explanation method. To
this end, we initially obtain the attribution of pixel importance from CAMs
generated by the method Grad-CAM [61]. However, any other post hoc explanation
method that is able to attribute multiple labels with the input pixels can be
used. In this scenario, the workflow is two-fold. First, a deep neural network
is trained on the task at hand. Then, a post hoc explanation method is applied
to obtain the class explanation masks for each image. As CAMs comprise
heatmaps with continuous values, a threshold is applied to obtain binary masks
that fulfill the requirements defined in Section III-B to be leveraged by our
LP strategy. For all experiments, we set the threshold $t_{\text{cam}}$ as
0.1, while we set the second threshold for minimum activating pixels
$t_{\text{map}}$ as 10 to smooth uncertainty in border regions between
activating and non-activating pixels. Nonetheless, it is worth noting that
many parameter combinations for both threshold parameters can lead to
reasonable performance. We applied a grid search that indicated almost
neglectable differences. Three exemplary CAMs generated by GradCAM are shown
in Figure 5.

Regarding box sizes and application probability of the investigated box
augmentation techniques, we followed the same setup as previous experiments.
From Table IV it can be observed that the results of TreeSatAI exhibit similar
trends as DeepGlobe-ML. Even though TreeSatAI is the most challenging dataset
and the class positional information is derived from noisy CAMs, the
observations from DeepGlobe-ML can be generalized to scenarios in which no
associated reference maps are available. In line with the results of V-C, not
fully reliable class positional information does not reduce the effectiveness
of our proposed LP strategy. For all box size ranges tested, CutMix w LP
outperforms its counterpart without LP. Similarly, as observed in DeepGlobe-
ML, the more larger boxes are used, the better CutMix w LP becomes, leading to
the best mAP macro performance of $68.91\text{\,}\mathrm{\char 37\relax}$ for
a box size range of $0.30.7$. It is worth noting that it yields the largest
improvement over CutMix w/o LP of all scenarios tested with
$4.2\text{\,}\mathrm{\char 37\relax}$ in mAP macro. Consequently, our LP
strategy demonstrates effectiveness also for scenarios in which explanation
methods are leveraged to obtain class positional information to allow the
effective use of CutMix in MLC.

Table IV: Results in mAP macro ($\mathrm{\char 37\relax}$) for TreesSatAI with
associated class explanation masks obtained by thresholded CAMs. Box
Augmentation
---
| Box Size Range
Method | $0.10.3$ | $0.10.5$ | $0.10.7$ | $0.30.5$ | $0.30.7$
CutOut [6] | $56.42$ | $56.71$ | $57.51$ | $56.12$ | $57.07$
CutMix w/o LP | $57.42$ | $58.78$ | $60.19$ | $63.82$ | $64.67$
CutMix w LP | 62.04 | 63.51 | 64.48 | 68.77 | 68.91
No Augmentation
Baseline | $53.44$ |  |  |  |

(a)

(b)

(c)

(d)

Figure 5: Examples for CAMs generated by Grad-CAM: 5(a) original image, 5(b)
activation map for class Betula, 5(c) activation map for class Pinus, and 5(d)
activation map for class Quercus.

##  VI Conclusion

In this article, we have proposed a label propagation (LP) strategy that
allows the effective application of CutMix in the framework of multi-label
image classification (MLC) in remote sensing (RS). The direct application of
CutMix to training images annotated with multi-labels can lead to the
potential erasure or addition of class labels in the paired image and, thus,
may lead to multi-label noise. The proposed LP strategy overcomes this problem
by preserving the correct class information within the multi-label vector of
the augmented image to enable an effective application of CutMix in MLC. In
detail, it exploits pixel-level class positional information such that the
preservation of the correct multi-labels of erased and copied areas can be
guaranteed. The pixel-level class positional information can either originate
from associated pixel-level reference maps or class explanation masks
generated by explanation methods. Similarly to the pairing operation of CutMix
for two images, our proposed LP strategy pairs the associated pixel-level
class positional information from both images to derive the updated multi-
label vector for the augmented image. The results show the effectiveness of
the LP strategy under reliable associated reference maps and under various
simulated noisy reference maps with less reliable class positional
information. In line with the observations from the simulated noisy reference
maps, our proposed LP strategy proves to be effective in real scenarios when
leveraging noisy reference maps derived from thematic products, as well as
integrating class explanation masks when no associated reference maps are
available. The reason for the robustness of our LP strategy under not fully
accurate class positional information is that for assigning high-level
categories in an MLC task rough estimates of class positional information can
be sufficient to update the multi-label vector of the augmented image.
Consequently, integrating our proposed LP strategy allows the application of
CutMix in RS MLC without being affected by multi-label noise.

It is noteworthy that, in theory, our proposed LP strategy is applicable to
any data augmentation technique that omits parts of the augmented image.
Nonetheless, experiments with CutOut or random cropping showed no significant
improvement when using our LP strategy. We attribute this to the different
effects of multi-label noise. CutOut and random cropping only delete parts of
an image without adding parts of a second image. The deletion of parts leads
to less problematic additive label noise if the multi-label is not updated.
The authors of the study on the effects of multi-label noise [11] suggest that
only subtractive label noise has a clear negative effect on a multi-label
classification task. Since the direct application of CutMix in MLC also
introduces subtractive noise by filling out the cutout parts in the augmented
image with parts of a different image, its effective application is allowed by
the proposed LP strategy that ensures a correct update of the multi-label.

As a final remark, we would like to note that the proposed strategy is
promising for possible operational applications in which associated class
positional information might not be fully reliable due to both its general
properties and its simplicity in the implementation. However, label imbalances
can affect the performance in MLC tasks, particularly for underrepresented
classes. The CutMix algorithm with our LP strategy does not change the
expected label distribution. As a future work, we plan to adapt CutMix to
account for label imbalances in multi-label training sets to further improve
the generalization capabilities of deep neural networks trained on multi-label
RS image classification tasks.

##  VII Acknowledgement

This work is supported by the European Research Council (ERC) through the
ERC-2017-STG BigEarth Project under Grant 759764.

## References

  * [1] G. Sumbul and B. Demir, “A deep multi-attention driven approach for multi-label remote sensing image classification,” in IEEE Access, vol. 8, pp. 95934–95946, 2020.
  * [2] Y. Li, R. Chen, Y. Zhang, M. Zhang, and L. Chen, “Multi-label remote sensing image scene classification by combining a convolutional neural network and a graph neural network,” in Remote Sensing, vol. 23, p. 4003, 12, 2020\.
  * [3] H. Yessou, G. Sumbul, and B. Demir, “A comparative study of deep learning loss functions for multi-label remote sensing image classification,” in IEEE International Geoscience and Remote Sensing Symposium, pp. 1349–1352, 2020.
  * [4] R. Stivaktakis, G. Tsagkatakis, and P. Tsakalides, “Deep learning for multilabel land cover scene categorization using data augmentation,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 7, pp. 1031–1035, 2019\.
  * [5] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “CutMix: Regularization strategy to train strong classifiers with localizable features,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032, 2019.
  * [6] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” in arXiv preprint arXiv:1708.04552, 2017.
  * [7] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13001–13008, 2020.
  * [8] X. Lu, C. Zhang, Q. Ye, C. Wang, C. Yang, and Q. Wang, “RSI-Mix: Data augmentation method for remote sensing image classification,” in 7th International Conference on Intelligent Computing and Signal Processing, pp. 1982–1985, 2022.
  * [9] Z. Suo, Y. Zhao, S. Chen, and Y. Hu, “BoxPaste: An effective data augmentation method for SAR ship detection,” Remote Sensing, vol. 14, no. 22, 2022.
  * [10] J.-X. Wang, S.-B. Chen, C. H. Q. Ding, J. Tang, and B. Luo, “RanPaste: Paste consistency and pseudo label for semisupervised remote sensing image semantic segmentation,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022.
  * [11] T. Burgert, M. Ravanbakhsh, and B. Demir, “On the effects of different types of label noise in multi-label remote sensing image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022\.
  * [12] I. Demir, K. Koperski, D. Lindenbaum, G. Pang, J. Huang, S. Basu, F. Hughes, D. Tuia, and R. Raskar, “Deepglobe 2018: A challenge to parse the earth through satellite images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 172–181, 2018\.
  * [13] C. F. Brown, S. P. Brumby, B. Guzder-Williams, T. Birch, S. B. Hyde, J. Mazzariello, W. Czerwinski, V. J. Pasquarella, R. Haertel, S. Ilyushchenko, K. Schwehr, M. Weisse, F. Stolle, C. Hanson, O. Guinan, R. Moore, and A. M. Tait, “Dynamic World, Near real-time global 10 m land use land cover mapping,” Scientific Data, vol. 9, no. 1, p. 251, 2022\.
  * [14] C. Paris and L. Bruzzone, “A novel approach to the unsupervised extraction of reliable training samples from thematic products,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 3, pp. 1930–1948, 2020.
  * [15] G. Büttner, J. Feranec, G. Jaffrain, L. Mari, G. Maucha, and T. Soukup, “The CORINE land cover 2000 project,” EARSeL eProceedings, vol. 3, no. 3, pp. 331–346, 2004.
  * [16] O. Arino, J. J. R. Perez, V. Kalogirou, S. Bontemps, P. Defourny, and E. V. Bogaert, “Global land cover map for 2009 (GlobCover 2009),” 2012.
  * [17] E. Bartholomé, A. Belward, A. Frédéric, S. Bartalev, C. Carmona-Moreno, H. Eva, S. Fritz, J. Grégoire, P. Mayaux, and H.-J. Stibig, GLC 2000: Global land cover mapping for the year 2000.  2002\.
  * [18] G. Sumbul, M. Charfuelan, B. Demir, and V. Markl, “Bigearthnet: A large-scale benchmark archive for remote sensing image understanding,” in IEEE International Geoscience and Remote Sensing Symposium, pp. 5901–5904, 2019.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016.
  * [20] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning, pp. 6105–6114, 2019.
  * [21] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986, 2022.
  * [22] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687–10698, 2020.
  * [23] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [24] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” Advances in Neural Information Processing Systems, vol. 33, pp. 596–608, 2020.
  * [25] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in Proceedings of the 37th International Conference on Machine Learning, vol. 119, pp. 1597–1607, July 2020.
  * [26] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020.
  * [27] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660, 2021.
  * [28] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.
  * [29] G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph, “Simple copy-paste is a strong data augmentation method for instance segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2918–2928, 2021.
  * [30] J. Ding, B. Chen, H. Liu, and M. Huang, “Convolutional neural network with data augmentation for SAR target recognition,” IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 3, pp. 364–368, 2016.
  * [31] P. Yan, F. He, Y. Yang, and F. Hu, “Semi-supervised representation learning for remote sensing image classification based on generative adversarial networks,” IEEE Access, vol. 8, pp. 54135–54144, 2020.
  * [32] Q. Xiao, B. Liu, Z. Li, W. Ni, Z. Yang, and L. Li, “Progressive data augmentation method for remote sensing ship image classification based on imaging simulation system and neural style transfer,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 14, pp. 9176–9186, 2021.
  * [33] W. Zhang and Y. Cao, “A new data augmentation method of remote sensing dataset based on Class Activation Map,” Journal of Physics: Conference Series, vol. 1961, no. 1, p. 012023, 2021.
  * [34] Y. Yan, Z. Tan, and N. Su, “A data augmentation strategy based on simulated samples for ship detection in RGB remote sensing images,” ISPRS International Journal of Geo-Information, vol. 8, no. 6, 2019.
  * [35] K. Zheng, M. Wei, G. Sun, B. Anas, and Y. Li, “Using vehicle synthesis generative adversarial networks to improve vehicle detection in remote sensing images,” ISPRS International Journal of Geo-Information, vol. 8, no. 9, 2019.
  * [36] Z. Zhang, Z. Feng, and S. Yang, “Semi-supervised object detection framework with object first mixup for remote sensing images,” in IEEE International Geoscience and Remote Sensing Symposium, pp. 2596–2599, 2021.
  * [37] B. Zhao, Y. Wu, X. Guan, L. Gao, and B. Zhang, “An improved aggregated-mosaic method for the sparse object detection of remote sensing imagery,” Remote Sensing, vol. 13, no. 13, 2021.
  * [38] G. Chen, G. Pei, Y. Tang, T. Chen, and Z. Tang, “A novel multi-sample data augmentation method for oriented object detection in remote sensing images,” in IEEE 24th International Workshop on Multimedia Signal Processing, pp. 1–7, 2022.
  * [39] Z.-H. You, J.-X. Wang, S.-B. Chen, J. Tang, and B. Luo, “FMWDCT: Foreground mixup into weighted dual-network cross training for semisupervised remote sensing road extraction,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 5570–5579, 2022.
  * [40] D. Ciregan, U. Meier, and J. Schmidhuber, “Multi-column deep neural networks for image classification,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 3642–3649, 2012.
  * [41] A. Krizhevsky, G. Hinton, and others, “Learning multiple layers of features from tiny images,” 2009.
  * [42] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.
  * [43] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, vol. 25, 2012.
  * [44] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in Proceedings of the British Machine Vision Conference, pp. 87.1–87.12, 2016.
  * [45] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710, 2018.
  * [46] M. Neumann, A. S. Pinto, X. Zhai, and N. Houlsby, “In-domain representation learning for remote sensing,” arXiv preprint arXiv:1911.06721, 2019.
  * [47] Y. Bazi, L. Bashmal, M. M. A. Rahhal, R. A. Dayil, and N. A. Ajlan, “Vision transformers for remote sensing image classification,” Remote Sensing, vol. 13, no. 3, 2021.
  * [48] D. Wang, J. Zhang, B. Du, G.-S. Xia, and D. Tao, “An empirical study of remote sensing pretraining,” IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1–20, 2023.
  * [49] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Is object localization for free?-weakly-supervised learning with convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 685–694, 2015.
  * [50] L. Sixt, B. Wild, and T. Landgraf, “RenderGAN: Generating realistic labeled data,” Frontiers in Robotics and AI, vol. 5, 2018.
  * [51] Q. Wu, Y. Chen, and J. Meng, “DCGAN-based data augmentation for tomato leaf disease identification,” IEEE Access, vol. 8, pp. 98716–98728, 2020.
  * [52] A. Antoniou, A. Storkey, and H. Edwards, “Data augmentation generative adversarial networks,” in International Conference on Learning Representations, 2018.
  * [53] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness,” in International Conference on Learning Representations, 2018.
  * [54] Y. Yan, Y. Zhang, and N. Su, “A novel data augmentation method for detection of specific aircraft in remote sensing RGB images,” IEEE Access, vol. 7, pp. 56051–56061, 2019.
  * [55] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: beyond empirical risk minimization,” in International Conference on Learning Representations, 2018.
  * [56] J. Li, R. Socher, and S. C. Hoi, “DivideMix: Learning with noisy labels as semi-supervised learning,” in International Conference on Learning Representations, 2019.
  * [57] D. Walawalkar, Z. Shen, Z. Liu, and M. Savvides, “Attentive Cutmix: An enhanced data augmentation approach for deep learning based image classification,” in IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3642–3646, 2020.
  * [58] D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 1301–1310, 2017.
  * [59] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, no. 1, pp. 321–357, 2002.
  * [60] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” in International Conference on Machine Learning, pp. 3145–3153, PMLR, 2017.
  * [61] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626, 2017.
  * [62] S. Ahlswede, C. Schulz, C. Gava, P. Helber, B. Bischke, M. Förster, F. Arias, J. Hees, B. Demir, and B. Kleinschmit, “TreeSatAI benchmark archive: A multi-sensor, multi-label dataset for tree species classification in remote sensing,” Earth System Science Data Discussions, vol. 2022, pp. 1–22, 2022\.
  * [63] G. Sumbul, A. de Wall, T. Kreuziger, F. Marcelino, H. Costa, P. Benevides, M. Caetano, B. Demir, and V. Markl, “BigEarthNet-MM: A large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [software and data sets],” IEEE Geoscience and Remote Sensing Magazine, vol. 9, no. 3, pp. 174–180, 2021.
  * [64] R. Wightman, “PyTorch Image Models,” 2019.  Publication Title: GitHub repository.
  * [65] W. Falcon and The PyTorch Lightning team, “PyTorch Lightning,” Mar. 2019\.