(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Universidad Autonoma de Madrid

# Gradient-based Class Weighting for Unsupervised Domain Adaptation in Dense
Prediction Visual Tasks

Roberto Alcover-Couso 11 Marcos Escudero-Viñolo 11 Juan C. SanMiguel 11 Jesus
Bescos 11

###### Abstract

In unsupervised domain adaptation (UDA), where models are trained on source
data (e.g., synthetic) and adapted to target data (e.g., real-world) without
target annotations, addressing the challenge of significant class imbalance
remains an open issue. Despite considerable progress in bridging the domain
gap, existing methods often experience performance degradation when confronted
with highly imbalanced dense prediction visual tasks like semantic and
panoptic segmentation. This discrepancy becomes especially pronounced due to
the lack of equivalent priors between the source and target domains, turning
class imbalanced techniques used for other areas (e.g., image classification)
ineffective in UDA scenarios. This paper proposes a class-imbalance mitigation
strategy that incorporates class-weights into the UDA learning losses, but
with the novelty of estimating these weights dynamically through the loss
gradient, defining a Gradient-based class weighting (GBW) learning. GBW
naturally increases the contribution of classes whose learning is hindered by
large-represented classes, and has the advantage of being able to
automatically and quickly adapt to the iteration training outcomes, avoiding
explicitly curricular learning patterns common in loss-weighing strategies.
Extensive experimentation validates the effectiveness of GBW across
architectures (convolutional and transformer), UDA strategies (adversarial,
self-training and entropy minimization), tasks (semantic and panoptic
segmentation), and datasets (GTA and Synthia). Analysing the source of
advantage, GBW consistently increases the recall of low represented classes.

###### Keywords:

Unsupervised Domain Adaptation Class Imbalance Semantic segmentation

(a) Color Image

(b) Ground truth

(c) SOTA

(d) GBW

Figure 1: UDA driven by MIC [18] ($1^{st}$ row) and EDAPS [34] ($2^{nd}$ and
$3^{rd}$ rows) methods is biased toward more populated (frequent or larger)
classes on the source dataset (c), miss-classifying instances of less frequent
or smaller classes: false positives examples include instances of train, car
and person miss-classified as truck, road and car respectively. (d) GBW
improves the classification of under-represented classes.

##  1 Introduction

Dense prediction visual tasks entail a major challenge for the annotation of
the required training datasets, as the annotation of a single image for
semantic or panoptic segmentation is estimated to take more than an hour for
an average human [8, 12]. A bypass solution has been the use of synthetic
datasets, as simulated environments can significantly reduce annotation costs.
However, a network trained on a synthetic source dataset is expected to
underperform when applied to the real target dataset, due to the data
distribution co-variate shifts between domains. To mitigate this domain gap,
unsupervised domain adaptation (UDA) is defined as the training of a deep
learning model focused on generalizing to unlabeled target domains by
leveraging labeled data from a source domain [47, 43, 16, 34].

UDA methods have remarkably progressed in the last few years and they still
are a more effective solution for specific domains than foundational models
[5] mainly due to their domain specificity. However, there is still a
noticeable performance gap compared to supervised learning [2, 52, 57]. One of
the factors contributing to this gap is the complexity of handling class
imbalance. As an inherent factor of the data distribution, class imbalance is
also subjected to covariant shifts and changes between the source and target
domains. Strategies to handle class imbalance can be roughly divided into [41,
20]: data-level and algorithmic-level techniques. Data-level techniques [16,
17, 18, 34] can not cope with class-imbalance in dense visual tasks, as they
cannot yield uniform class samplings due to the imbalance nature of these
tasks, where context is key for prediction and the context of low populated
classes is prone to be composed of large populated classes. Therefore,
sampling more from the low populated classes entails including (eventually
more) samples from large populated ones, keeping or even enlarging the
imbalance. Algorithmic-level techniques assign different weights to the loss
of different classes according to the target class distribution generally
under a class-weighting strategy [20]. Class-weighing has the effect of
differently pondering the contribution to the global training loss of the
different classes [22]. Class-weighting techniques have been proven
ineffective for UDA due to the lack of target domain class-frequencies [54,
55] and the discredited assumption that target domain class-frequency is
equivalent to that of source domain [54, 55]. Therefore, the only feasible
alternative if to estimate the target priors on every model update using
pseudo-labels [54, 55], inquiring in excessive computational costs, and
demotivating the use of algorithmic-level techniques [22].

UDA methods for image-classification have reported the advantages of assigning
different relevance to the training samples through importance weighting [53].
Importance weighting prioritizes data points with high impact on the learning
process by focusing on data that generates large gradients during optimization
[21]. Adaptive weighting incorporates importance weighting in the training
process by giving more importance to certain samples on the fly. This dynamic
prioritization changes the learning target, but helps the learning process
focusing on examples that have a bigger impact on how the model improves [37].
Although per-sample weighting is a key stage in dense visual prediction UDA
methods to prevent the adaptation over-fit to fuzzy pseudo-label predictions
[14], general per-sample adaptive weighting has not been previously explored
for dense prediction visual tasks. This is probably due to the memory and
computation requirements in estimating a relevance for each pixel.

To address the class-imbalance challenge in UDA, this paper describes a new
algorithm-level technique: a gradient-based class-loss weighting (GBW) for UDA
that can be applied to dense prediction visual tasks. GBW firstly leverages
adaptive weighting for dense visual prediction tasks by considering the
gradients of the per-class loss in each iteration as a proxy to the class
learning and pondering the learning of the classes such that the overall loss
gradient (and hence—we hypothesize, the learning) is maximized. GBW can be
straightforwardly integrated into various UDA methods across different visual
recognition tasks, making it highly valuable in practice. The overall effect
in UDA of GBW is the alleviation of class imbalance, consistently improving
the classification of low populated classes as well as the overall
performance. Figure 1 shows a visual comparison of the effect of using GBW in
state-of-the-art (SOTA) methods HRDA [17] for semantic segmentation and EDAPS
[34] for panoptic segmentation.

(a) Coarse classes

(b) Pedestrian and rider

Figure 2: Averaged per-class weights ($v_{c}$) of GBW on the GTA-Cityscapes
framework for HRDA method throughout the UDA training process. 2(a) shows the
evolution for some coarse classes and 2(b) depicts the complementary evolution
for person and rider.

Leveraging the gradient rather than class frequencies permits the application
to both source and target domains, enabling its use for UDA. Performing
adaptive weighting at the class rather than at the pixel level, not only
effectively enables the use of adaptive weighting techniques for dense visual
prediction tasks but also naturally adapts the training to each class learning
landscape. This enables the learning process to evolve different for each
class based on its nature, interactions with other classes, and the training
class-wise estimates (see examples of per-class weights evolution in Figure
2). Experimental evaluation reports significant and consistent performance
improvements for different UDA methods (including adversarial training,
entropy-minimization, and self-training) on multiple dense prediction visual
tasks (semantic and panoptic segmentation), and using different network
architectures (CNNs and Vision Transformers).

##  2 Related Work

###  2.1 Unsupervised Domain Adaptation: UDA

The goal of UDA is to provide robustness to co-variate shifts on an unlabeled
target domain by leveraging training on a labeled source domain. Due to the
universality of domain gaps, the use of UDA has been explored in multiple
computer vision problems, including image classification [30, 18], semantic
segmentation [16, 17] and panoptic segmentation [34].

UDA methods can be classified into three non-exclusive principal categories
[39]: Input space adaptation [16, 17, 18, 43, 28], feature space adaptation
[44, 47] and output space adaptation [1, 44, 47, 16]. Input space adaptation
methods reduce the domain gap by modifying the style of images following a
data transformation protocol based on data augmentations [28, 43], the mixture
of images [43] or style transfer [51, 49]. These transformations can be
applied to either the source or target domains. Feature space adaptation
methods follow the hypothesis that minimizing a discrepancy metric between the
source and target model outputs will produce domain-agnostic features.
Thereby, a classifier capable of generalizing to both domains
indistinguishably can be trained. This alignment is typically driven by
adversarial training [44, 47] or by minimizing a suitable metric between
features such as L2 distance [7, 16] or the Maximum Mean Discrepancy (MMD)
[13, 55, 54]. Output space adaptation methods rely on the model’s predictions
during training, often utilizing a process known as self-training, where the
model adapts based on its own tentative predictions. This adaptation is
typically achieved through the assignment of pseudo-labels, often under some
confidence paradigm [16, 17, 18, 43]. However, as the model is expected to
provide false positives, a prominent challenge raises: concept drifting [4,
36], which is the event where a model trained on pseudo-labels over-fits to
false positives, consequently reducing the performance on the target domain.

Transversal to all these categories is the problem of class imbalance: Input
space adaptation methods employ multiple versions of source images
(reinforcing the source class-biases), feature space adaptation aims to
generate source agnostic features based on source-biased features, and output
space adaptation employ source-biased predictions to generate pseudo-labels.
Thus, all UDA stages may benefit from better techniques to handle class
imbalance.

###  2.2 Handling class imbalance in UDA

In real-world scenarios, accurately classifying examples from an infrequent
class poses a significant challenge known as the class imbalance problem. For
image classification, competitive datasets, such as CIFAR-10/100 [25],
ImageNet [9], Caltech-101/256[15], and MIT-67 [31], deliberately hampers class
imbalance by ensuring all classes have a minimum representation [22].
Unfortunately, this balanced-samples design becomes infeasible in the context
of dense tasks such as semantic segmentation. Certain classes, regardless of
the number of available images, inevitably exhibit significantly smaller
representations compared to broader classes. Furthermore, for some domains it
is not possible to obtain more images from certain classes, such as rare
illnesses for medical tasks. Therefore, mitigating the adverse effects of
imbalanced class distributions emerges as a transversal research problem [22,
29, 35].

Techniques to handle class imbalance in machine learning can be broadly
classified into two main categories [41, 20]: data-level and algorithmic-level
techniques. Data-level techniques try to balance the data by reducing the
likelihood of selecting majority class samples for training (undersampling) or
increasing the likelihood of minority class sampling (oversampling)[41]. On
the other hand, algorithmic-level techniques modify the learning process to
tackle the bias produced by the imbalanced data. These techniques are
typically implemented with a weight schema assigning penalties to each class
[22, 20, 24, 27]. Alternatively, hard class-weighting strategies have been
proposed for semantic segmentation so that all classes contribute equally in
the loss [42, 56, 60]. However, these strategies are not well suited for
densely populated images, as result in small objects accounting equally in the
loss to classes composing most of the scene.

In the context of UDA for dense tasks, data-level techniques are the default
and typically the only imbalance techniques employed [17, 16, 18, 34]. This is
mainly because standard data-level techniques rely on the source domain class
frequency and algorithmic-based techniques are not effective without target
domain class-frequency [54, 55]. Although a few works have explored
algorithmic-level techniques, these either require the estimation of the
target dataset class-frequency at every model update [54, 55] or are focused
on relaxing the filtering criteria of pseudo-labels for low-populated
classes[61, 38]. The former becomes unpractical for dense tasks, as such
estimation must be performed at every model update making the process very
time consuming. The latter does not tackle the root problem because, as the
model is trained with the source biases, the pseudo-labels are more prone to
classify least represented classes as other densely populated classes in the
source set. Altogether, current state-of-the-art UDA methods for dense tasks
rely solely on the employment of sampling techniques [17, 16, 18, 34],
discarding the possible benefits of joint training with data and algorithmic
imbalance techniques [41, 20].

This paper describes a new algorithmic-level technique complementary with
data-level techniques that can be applied to source and target domain, not
requiring target domain information, by firstly tailoring adaptive weighting
[37] to a class-wise paradigm and incorporating it into the UDA learning
process.

##  3 Method

###  3.1 The learning of dense prediction visual tasks

Let $\hat{\mathbf{y}}_{i,t}=G(x_{i};\theta_{t})$ be the output of a deep
learning model parameterized by $\theta$, at the $t^{th}$ training step, given
an input sample $\mathbf{x}_{i}$. In the context of classification,
$\hat{\mathbf{y}}_{i,t}$ is typically a per-class probability vector. Note
that for dense tasks such as semantic and panoptic segmentation, a sample is
considered to be a pixel from the image.

The learning is typically driven by maximizing the probability of the expected
class $y_{i}$. This can be segregated on a per-class basis:

$\mathcal{L}(\hat{\mathbf{y}}_{i,t},y_{i})=\sum_{c=1}^{C}\frac{l_{c}(\hat{\mathbf{y}}_{i,t},y_{i})}{|y_{i}=c|},$
(1)

where $C$ is the number of classes, $L(\hat{\mathbf{y}}_{i,t},y_{i})$ is the
aggregated loss, $l_{c}$ the probability distance of the prediction to a given
class and $|\cdot|$ the cardinal function. For source domain labels and target
domain pseudo-labels, $l_{c}$ is typically the cross-entropy loss:
$l_{c}(\hat{\mathbf{y}}_{i,t},y_{i})=-\delta[c-y_{i}]log(\hat{y}_{i,t})$.
Furthermore, for the target domain some regimes also minimize the entropy of
the predictions[47]:
$l_{c}(\hat{\mathbf{y}}_{i,t})=-\hat{y}_{i,t}^{c}log(\hat{y}_{i,t}^{c})$.

Including into the formulation a set of per-class weights
$\mathbf{v}\in\mathbb{R}^{C}$, the loss for each sample is defined by:

$\mathcal{L}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v})=\sum_{c=1}^{C}v_{c}\frac{l_{c}(\hat{\mathbf{y}}_{i,t},y_{i})}{|y_{i}=c|}.$
(2)

When no class weighting is applied $\mathbf{v}$ is a vector of one value
elements of size $C$.

The learning of $\theta$ is driven by the minimization of the loss
$\mathcal{L}$ over $N$ samples:

$\theta^{*}=arg\min_{\theta}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(\hat{\mathbf{y}}_{i,t},y_{i}).$
(3)

This optimization is often solved through stochastic gradient descent (SGD)
with mini-batches, leading to the update equation:

$\theta_{t+1}=\theta_{t}-\eta\frac{1}{M}\sum_{i=1}^{M}\nabla_{\theta_{t}}\mathcal{L}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t}),$
(4)

where $\eta$ represents the learning rate, $t$ the optimization step, and the
samples $i=1,...,M$ are selected randomly from the training set.

###  3.2 Gradient based weighting (GBW)

Figure 3: Semantic segmentation using GBW. Given a forward pass of an image
and the respective per-class loss ($l_{c}$). First, the gradients
($||\nabla_{\theta_{t}}\overline{l_{c}}||^{2}$) are estimated with the
gradient of the last layer (shadowed in orange) to compute the per-class
weights $\mathbf{v}$. Second, the backward pass is performed wrt the weighted
cross entropy $\mathcal{L}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})$.

We adapt [37] by following the assumption that minimizing a class loss should
make the model more prone to classify that class, therefore, by dynamically
setting optimal weights to decrease the overall gradient, the model should
avoid over-fitting to broad classes. Additionally, this avoids manually
setting class weights which could lead to over-classifying small classes. To
that end, GBW estimates class weights at each learning step given the average
per-class loss ($\overline{l_{c}}$) gradients:

$\mathbf{v}^{*}=arg\min_{\mathbf{v}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,(t+1)},y_{i};\mathbf{v}_{t})-\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t}).$
(5)

Using linear approximation, one can approximate the term
$\overline{l_{c}}(\hat{\mathbf{y}}_{i,(t+1)},y_{i};\mathbf{v}_{t})$:

$\overline{l_{c}}(\hat{\mathbf{y}}_{t+1},y_{i};\mathbf{v}_{t})\leq\\{\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})+(\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t}))^{T}(\theta_{t+1}-\theta_{t})\\}$
(6)

Note that
$\nabla_{\theta_{t}}\overline{l_{c}}(\cdot;\mathbf{v}_{t})=\frac{1}{C}\sum_{c=1}^{C}v_{c}\nabla_{\theta_{t}}l_{c}$,
making the operation linearly dependant on the per-class losses.

$\begin{split}&\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})-\overline{l_{c}}(\hat{\mathbf{y}}_{i,(t+1)},y_{i};\mathbf{v}_{t})\\\
&\text{including Eq \ref{eq:taylor}}\\\
&\begin{split}\leq\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})&-\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})-(\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i})^{T}(\theta_{t+1}-\theta_{t})\end{split}\\\
&=-(\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})^{T}(\theta_{t+1}-\theta_{t})\\\
&\text{including Eq \ref{eq:optW} and simplifying}\\\
&=(\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})^{T}(\eta\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t}))\\\
&=\eta||\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})||^{2}.\end{split}$
(7)

Class weights can be calculated through the following optimization problem:

$\begin{split}\mathbf{v}^{*}=arg\min_{\mathbf{v}}-||\nabla_{\theta_{t}}\overline{l_{c}}(\hat{\mathbf{y}}_{i,t},y_{i};\mathbf{v}_{t})||^{2}\\\
\text{subject to }v_{c}\geq 0,\forall c\in[1,C]\\\ \end{split}$ (8)

This optimization can be efficiently computed by expressing the problem as a
quadratic programming problem [45]:

$\begin{split}\text{minimize
}\textbf{v}_{t}^{T}Q\textbf{v}_{t}-(||\nabla_{\theta_{t}}\overline{l_{c}}||^{2})^{T}\textbf{v}_{t}\\\
\text{subject to }-I\textbf{v}_{t}\leq\textbf{0},\end{split}$ (9)

where 0 is a vector filled with zeros and $I$ is the identity matrix. The only
problem left is defining the $Q$ matrix. Through initial experimentation, we
found that this optimization would lead to highly variable weights which
disrupt learning. Therefore, we opt to define $Q$ as a regularization matrix:
$Q=\lambda I$, where $\lambda$ is a regularization parameter. Specifically, we
impose an $L_{2}$ regularization on $\textbf{v}_{t}$:

$\begin{split}\mathbf{v}^{*}=arg\min_{\mathbf{v}}-(||\nabla_{\theta_{t}}\overline{l_{c}}||^{2})^{T}\textbf{v}_{t}+\lambda||\textbf{v}_{t}||^{2}\\\
\text{subject to }v_{c}\geq 0,\forall c\in[1,C]\\\ \end{split}$ (10)

Additionally, to maintain a similar scale to the original loss, we impose an
additional constraint that ensures the weights sum up to the number of
classes.

$\begin{split}\mathbf{v}^{*}=arg\min_{\mathbf{v}}-(||\nabla_{\theta_{t}}\overline{l_{c}}||^{2})^{T}\textbf{v}_{t}+\lambda||\textbf{v}_{t}||^{2}\\\
\text{subject to }v_{c}\geq 0,\forall c\in[1,C]\\\ \text{ and
}\sum_{c=1}^{C}v_{c}=C,\end{split}$ (11)

the last constraint also promotes that all batches are weighted similarly in
the loss, as without it, we seldom noticed large discrepancies for some
instances and training iterations that were precluding a smooth learning
process.

Overall, the complexity of GBW scales with the number of classes: $O(C)^{2}$,
making its resolution feasible for large datasets. However, to further
facilitate the computation of the gradient norms without performing back-
propagation throughout the entire model, the gradient norms are further
approximated by calculating the gradient of the loss function with respect to
the pre-activation outputs of the last layer in the model. The use of these
pre-activation gradients, as demonstrated in previous studies [37, 21],
significantly simplifies the weight computation process and substantially
reduces computational complexity, while preserving a significant correlation
to the true gradients. Figure 3 provides a visual summary of our GBW
methodology for semantic segmentation.

#### Combination with previous per-sample weighting

In UDA, pseudo-labels are commonly employed to reinforce the classification
performance on the target data. These pseudo-labels usually are weighted at a
sample level to avoid over-fitting to fuzzy predictions [16, 17, 18, 43, 14,
50, 40, 26]. Notably, neglecting these pseudo-labels in the optimization goal
leads to the class-weights counteracting the confidence weighting in the loss.
To prevent this, we modify the optimization goal in Equation 11 by
incorporating the pseudo-label confidence $p_{i}$:

$\mathbf{v}^{*}=arg\min_{\mathbf{v}}-(||\nabla_{\theta_{t}}\sum_{i=1}^{M}p_{i}\cdot
l_{c}(\hat{\mathbf{y}}_{i,t},y_{i})||^{2})^{T}+\lambda||\textbf{v}_{t}||^{2}.$
(12)

##  4 Experimental Exploration

###  4.1 Setup

#### Semantic Segmentation

We explore the benefits of the proposed GBW method on two popular UDA
scenarios defined by the adaptation for semantic segmentation of the source
datasets GTA[32] and Synthia [33] to the target dataset Cityscapes [8]. GTA is
a synthetic dataset comprising 25K images and sharing 19 semantic classes with
Cityscapes. Synthia is a synthetic urban scenes dataset for semantic
segmentation, composed of 9.5K images and with 16 common semantic classes with
Cityscapes. Cityscapes is a real-image dataset with urban scenes generated by
filming with a camera inside of a car while driving through different German
cities. It consists of 3K images for training and 0.5K images for validation.
For evaluation, we rely on the common metric used for semantic segmentation:
the per-class intersection over union (IoU) [11], between the model prediction
and the ground-truth label. IoU measures at pixel-level the relationship
between True Positives (TP), False Positives (FP) and False Negatives (FN):
$IoU=\frac{TP}{TP+FP+FN}$.

#### Panoptic Segmentation

Similarly to Semantic segmentation, we measure the benefits of GBW for UDA
between the synthetic source dataset Synthia [33] and the real target dataset
Cityscapes [8]. These are evaluated through common Panoptic segmentation
metrics: mean Segmentation Quality (mSQ), mean Recognition Quality (mRQ) and
mean Panoptic Quality (mPQ) [23]. The mSQ measures the closeness of the
predicted segments with their ground truths, mRQ is equivalent to the F1 score
and the mPQ presents a global quality of the panoptic segmentation by
combining at a per-class level the SQ and RQ: PQ = SQ$\times$RQ. Note that
improving the recall of an algorithm mainly affects mRQ, as mSQ is mostly
driven by the segmentation quality of the TP [23].

#### Training parameters

The architectures employed and all training parameters but our proposed weight
regularization in equation 11, are as in the original papers [16, 17, 18, 43,
44, 47, 34].

###  4.2 GBW for UDA in Semantic Segmentation

Architecture | UDA Method | data-level | GTA | Synthia
---|---|---|---|---
|  |  | w/o GBW | w/ GBW | w/o GBW | w/ GBW
Convolutional | Output [47] | $\times$ | 43.8 | 46.1 | 38.1 | 41.2
| Feature [44] | $\times$ | 42.4 | 45.5 | 46.7* | 50.7*
| Input [43] | $\times$ | 52.1 | 55.4 | 48.3 | 50.1
Transformer | Output + Input[16] | $\checkmark$ | 68.3 | 69.2 | 60.9 | 62.3
| Output + Input[17] | $\checkmark$ | 73.8 | 74.7 | 65.8 | 66.8
| Output + Input[18] | $\checkmark$ | 75.9 | 76.4 | 67.5 | 68.8

Table 1: Comparison of UDA methods for the GTA-Cityscapes and Synthia-
Cityscapes semantic segmentation frameworks for different architectures and
methods.

#### Incorporation of GBW into UDA methods

Table 1 measures the performance gain of key state-of-the-art UDA methods in
the GTA-Cityscapes and Synthia-Cityscapes scenarios when GBW is incorporated.
Methods are arranged according to the use (marked with a ✓) or not (marked
with a $\times$) of some form of data-level class imbalance techniques. The
results show that incorporating our per-class weighting strategy consistently
leads to performance gains across various architectures and methods. Notably,
the largest gains are observed for methods that do not employ data-level class
imbalance techniques, with improvements of up to 3.3 points in mIoU for GTA-
Cityscapes and 4.0 for Synthia-Cityscapes. BW provides moderate gains also for
methods that already utilize data-level class imbalance techniques,
underlining its complementary nature to these strategies.

Architecture | Method | GTA-Cityscapes | Synthia-Cityscapes
---|---|---|---
Convolutional | CRA[50] | 46.7 | 49.3
| Zhang et al. [58] | 48.2 | 48.4
| GBW (DACS) | 55.4 | 50.1
Transformer | ExpCons[48] | 69.6 | 61.5
| DiGA[40] | 70.0 | 62.1
| MIC[18] | 75.9 | 67.5
| GBW (MIC) | 76.4 | 68.8

Table 2: Comparison of leading GTA-Cityscapes and Synthia-Cityscapes UDA
semantic segmentation frameworks for convolutional- and transformer-based
architectures.

#### Comparison with state-of-the-art

Table 2 provides a comprehensive comparison of the currently state-of-the-art
convolutional-based and transformer-based models. Notably, the inclusion of
GBW in the learning of MIC [18] results in performance gains compared to the
other analysed state-of-the-art methods for both scenarios. Importantly, GBW
achieves these results without the need for computationally expensive
transformations such as style transfer [50], or feature prototyping [28, 58].

###  4.3 GBW for UDA in Panoptic segmentation

For panoptic segmentation UDA, we explore the benefits of including GBW in the
learning process of the state-of-the-art method EDAPS [34]. In the Synthia-
Cityscapes scenario (Tab. 3), the use of GBW provides improvements across all
metrics. Specifically of +0.5 mSQ, +1.8 mPQ and +2.3 mRQ. As mSQ measures the
segmentation quality of the true positives, it is mostly driven by the
segmentation capabilities of the network, thus, the gain in terms of mSQ is
narrower. To the best of our knowledge, this GBW-enhanced EDAPS setup is the
current state-of-the-art for UDA in panoptic segmentation across all metrics.

Method |  mPQ |  mSQ |  mRQ
---|---|---|---
CVRN [19] |  32.1 |  66.6 |  40.9
UniDAPS [59] |  33.0 |  64.7 |  42.2
EDAPS [34] |  41.2 |  72.7 |  53.6
GBW (EDAPS) |  43.0 |  73.2 |  55.9

Table 3: Performance comparison of state-of-the-art Synthia-Cityscapes UDA
panoptic segmentation frameworks. Best results indicated in bold.

###  4.4 Analysis of GBW

#### Combination with data-level class imbalance techniques

Currently, the default technique for handling class imbalance in UDA is rare
class sampling [16]. Therefore, we propose to analyze the impact of GBW when
used in conjunction and separated from the standard data-level technique.
Table 4(a) presents a comparison of various configurations incorporating per-
class weights and class-uniform sampling. The results emphasize the
substantial impact of these techniques in the performance of the DAFormer
method for semantic segmentation UDA [16]: the combined approach yields +7.5
points increase in terms of mIoU.

| None | w/GBW | w/S | w/GBW+S
---|---|---|---|---
mIoU | 61.7 | 65.2 | 68.3 | 69.2
Gain | 0 | 3.5 | 6.6 | 7.5
(a) Data-level techniques on DAFormer [16].

Loss | wo | w/IF | w/PF | w/LBW | w/GBW
---|---|---|---|---|---
CE | 73.8 | 71.3 | 30.7 | 72.9 | 74.7
FL [27] | 68.9 | 68.5 | 64.0 | 68.9 | 70.1

(b) Algorithmic-level techniques on HRDA [17].

Table 4: Performance comparison with alternative data and algorithmic level
class imbalance techniques on the GTA-Cityscapes semantic segmentation
framework [16]. Key.S: Sampling. CE: Cross-entropy. FL: Focal-loss. IF: image
frequency. PF: pixel frequency. LBW: loss-based weighting.

#### Effect on different segmentation losses

Table 4(b) compiles the performances of incorporating in the learning process
state-of-the-art class weighting strategies, including regular class weights
[46]: the inverse frequency on the source set, either considering the pixel-
level frequency (PF) [3] or the image-level frequency (IF) [6], and a class
weighting based on the per class loss rather than their gradient (LBW) [10].
Reported performance corroborate the previous evidence that weighting the loss
based on source domain priors do not transfer well to the target domain due to
the distribution shift. Regarding the frequency source, PF results in
drastically higher weights for underrepresented classes compared to IF,
turning the model prone to classify broad structures as low-represented
categories, thus presenting a significant drop in performance. Meanwhile, IF
weighting assigns more uniform weights, thus, presenting better performance
than PF, but dropping the performance of broad categories compared to GBW. The
use of LBW results in second performance, but suffers from a stable tendency
that prevents it from quickly adapting to the learning outcomes as GBW.

#### Per-class improvements

Figure 4(a), depicts the disparity between the state-of-the-art model HRDA
confusion matrix [17] and the confusion matrix of the modified HRDA after the
inclusion of GBW. Introducing GBW notably enhances recall, as evidenced by the
predominantly positive values along the diagonal (15 out of 19). Note that the
gain in recall is specially significant for the least represented classes in
the source dataset (GTA): train, motocycle and bicycle. This is evidenced by
the true positives on the diagonal. Although GBW-enhanced model shows
improvements for most classes, it is observed that some classes as sidewalk
and motorcycle do not benefit from it. This is further analyzed in the
Supplementary material.

(a) Relative discrepancy matrix (green/red indicates a rise/drop in
performance, yellow indicates equal performance) between our GBW-enhanced HRDA
model and the reported results of HRDA [17]. The vertical axis corresponds to
the ground truth labels, while the horizontal axis represents the predicted
classes. Diagonal is depicted with bold colors for visualization. Classes are
sorted from least to most number of samples.

(b) Per-class gain/downgrade compared to the baseline (i.e., without
incorporating GBW) using different $\lambda$ values. Change is measured in
mIoU points. Results for the GTA-Cityscapes scenario for HRDA method [17].

Figure 4: GBW per-class performance analysis.

Figure 5 includes a qualitative comparison between models trained with and
without GBW. The GBW-model exhibits enhanced capability in accurately
classifying less populated classes. See examples for fence (second row), bus
and sign (fourth row). See Supplementary material for more visual examples.

Figure 5: Qualitative comparison. Column-wise: image, ground truth, model
trained with and without GBW. Semantic segmentation ($1^{st}$ row). Panoptic
segmentation ($2^{nd}$ row).

#### Regularization weight

This section examines the impact of the regularization factor when GBW is
incorporated into the HRDA model [17] for semantic segmentation, as shown in
Table 5. Note that $\lambda$ represents the weight of the regularization term
in Equation 11, with higher values enforcing uniform weights, which is
equivalent to not applying any weighting at all. This is clearly observed in
the performance when $\lambda=10$, which matches the performance of a model
trained without GBW. Conversely, extremely small $\lambda$ values may cause
the weights of certain classes approach to zero, effectively disregarding them
during training. This accounts for the suboptimal performance observed when
$\lambda=0.01$. Figure 4(b) visualize disentangles this effect in per-class
performance. GBW drastically affects train and visually similar classes (bus,
truck) performance, with improvements of over +10 IoU. We believe this to be
because train is the second least represented class in the GTA [32] dataset
and models trained on GTA tend to classify it as bus and truck, thus improving
also their IoU by reducing FPs.

$\lambda$ | - | 0.01 | 0.1 | 0.5 | 1 | 2 | 10
---|---|---|---|---|---|---|---
mIoU | 73.2 | 56 | 73.5 | 73.8 | 74.7 | 73.8 | 73.2
Gain | - | -16.8 | 0.3 | 0.6 | 1.5 | 0.6 | 0.0

Table 5: Ablation study for the $\lambda$ weight on the GTA-Cityscapes
framework embedded into the state-of-the-art UDA method HRDA[17]

#### Weight distribution

The main hypothesis is that a GBW-enhanced model tends to promote the learning
of under-represented classes. To visually inspect this, Figure 6(a) compares
the percentage of pixels per class in the source dataset with the maximum
weight assigned by GBW, showcasing that highly represented classes tend to be
given a smaller weight than less represented classes. The Figure only depicts
the maximum weight per class along the training, as weights are made to be
very variable between training iterations depending on the current state of
the model. It can be observed that classes that presented a large variation in
weights, such as train and sign, are associated with larger improvements in
performance, with increases of over 12 points in mIoU. Figure 6(b) illustrates
the relationship between weight variance and mIoU. Note how classes with
performance improvements highly correlate with the variability of the assigned
weights. Additionally, highly represented classes as road and sidewalk,
exhibit more uniform weights, indicated by a low standard deviation.

(a) Visual comparison between class prevalence and GBW maximum assigned
weights. Prevalence measured by negative log percentage of pixels in the
source dataset.

(b) Relationship between performance improvements of GBW and the standard
deviation of the GBW assigned during training.

Figure 6: Analysis of the GBW weights assigned throughout the training with
HRDA [17].

#### Weight evolution

Figures 2(a) and 2(b) present the averaged evolution of the per-class weights
assigned by GBW. Coarse and high populated classes (as shown in Figure 2(a))
start the training process with higher weights. However, as training
progresses, these weights gradually decrease and are distributed to other
classes. Specifically, looking at two similar classes: person (highly
populated) and rider (under-populated) GBW adeptly handles them by assigning
higher weights to person at the onset of training, while lowering the weight
for rider (see Figure 2(b)). As the training proceeds, the weight assigned to
rider gradually increases, while decreasing simultaneously the weight for
person suggesting an interesting capacity of GBW in adapting to the training
outcomes. This, together with previous evidences, suggests that the success of
GBW relies on the algorithm’s capacity to dynamically assign appropriate
weights to each class at different stages of the learning process naturally
and dynamically integrating evidences from all classes.

##  5 Conclusion

Despite the extensive literature in UDA, the impact of statistical
discrepancies—such as different class imbalances between the source and target
dataset—is still an open topic. Due to the lack of target labels, algorithmic-
level techniques to handle class imbalance are deemed unsuited for UDA in
dense prediction visual tasks. In this work, class imbalance is handled by an
adaptive weighting scheme based on the loss gradients at each training step.
Per-class weights are updated on-the-fly at each training step to compensate
for the magnitudes of the loss gradient. Experimental results show
improvements in the recall of scarcely represented classes. The benefits of
GBW are exhaustively explored by means of its introduction into six different
state-of-the-art UDA methods for semantic segmentation and panoptic
segmentation. The results consistently show that incorporating GBW leads to
performance gains across different architectures and datasets for both
semantic and panoptic segmentation. We hope that GBW can be used on future UDA
methods by serving as a building block for narrowing the gap between UDA and
supervised methods.

## References

  * [1] A.K., D., Sanodiya, R.K., Jose, B.R., Mathew, J.: Visual domain adaptation through locality information. Engineering Applications of Artificial Intelligence 123, 106172 (2023). https://doi.org/https://doi.org/10.1016/j.engappai.2023.106172
  * [2] Alcover-Couso, R., Escudero-Vinolo, M., SanMiguel, J.C., Martinez, J.M.: Soft labelling for semantic segmentation: Bringing coherence to label down-sampling (2023)
  * [3] Alonso, I., Sabater, A., Ferstl, D., Montesano, L., Murillo, A.C.: Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2021)
  * [4] Cascante-Bonilla, P., Tan, F., Qi, Y., Ordonez, V.: Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In: AAAI Conference on Artificial Intelligence (2020)
  * [5] Chen, J., Yang, Z., Zhang, L.: Semantic segment anything (2023)
  * [6] Chen, M., Xue, H., Cai, D.: Domain adaptation for semantic segmentation with maximum squares loss. In: IEEE Int. Conf. Comput. Vis. (ICCV). pp. 2090–2099 (October 2019). https://doi.org/10.1109/ICCV.2019.00218
  * [7] Chen, Y., Li, W., Van Gool, L.: Road: Reality oriented adaptation for semantic segmentation of urban scenes. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2017)
  * [8] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 3212–3223 (2016). https://doi.org/10.1109/CVPR.2016.350
  * [9] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
  * [10] Escudero-Viñolo, M., López-Cifuentes, A.: Ccl: Class-wise curriculum learning for class imbalance problems. In: 2022 IEEE Int. Conf. Image Process. (ICIP). pp. 1476–1480 (2022). https://doi.org/10.1109/ICIP46576.2022.9897273
  * [11] Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge (2010)
  * [12] Fernández-Moreno, M., Lei, B., Holm, E.A., Mesejo, P., Moreno, R.: Exploring the trade-off between performance and annotation complexity in semantic segmentation. Engineering Applications of Artificial Intelligence 123, 106299 (2023). https://doi.org/https://doi.org/10.1016/j.engappai.2023.106299
  * [13] Gao, L., Zhang, J., Zhang, L., Tao, D.: Dsp: Dual soft-paste for unsupervised domain adaptive semantic segmentation. In: ACM Int. Conf. Multimedia (MM). p. 2825–2833 (2021). https://doi.org/10.1145/3474085.3475186
  * [14] Gong, R., Wang, Q., Danelljan, M., Dai, D., Van Gool, L.: Continuous pseudo-label rectified domain adaptive semantic segmentation with implicit neural representations. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 7225–7235 (June 2023)
  * [15] Griffin, G., Holub, A., Perona, P.: Caltech 256 (2022). https://doi.org/10.22002/D1.20087
  * [16] Hoyer, L., Dai, D., Van Gool, L.: DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 9924–9935 (2022)
  * [17] Hoyer, L., Dai, D., Van Gool, L.: HRDA: Context-aware high-resolution domain-adaptive semantic segmentation. In: IEEE Eur. Conf. Comput. Vis. (ECCV). p. 372–391 (2022)
  * [18] Hoyer, L., Dai, D., Wang, H., Van Gool, L.: MIC: Masked image consistency for context-enhanced domain adaptation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2023)
  * [19] Huang, J., Guan, D., Xiao, A., Lu, S.: Cross-view regularization for domain adaptive panoptic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2021)
  * [20] Johnson, J.M., Khoshgoftaar, T.M.: Survey on deep learning with class imbalance. In: Journal of Big Data (2019). https://doi.org/10.1186/s40537-019-0192-5
  * [21] Katharopoulos, A., Fleuret, F.: Not all samples are created equal: Deep learning with importance sampling. In: International Conference on Machine Learning (2018)
  * [22] Khan, S.H., Hayat, M., Bennamoun, M., Sohel, F.A., Togneri, R.: Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Transactions on Neural Networks and Learning Systems 29(8), 3573–3587 (2018). https://doi.org/10.1109/TNNLS.2017.2732482
  * [23] Kirillov, A., He, K., Girshick, R., Rother, C., Dollar, P.: Panoptic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2019)
  * [24] Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. In: Progress in Artificial Intelligence (2016). https://doi.org/10.1007/s13748-016-0094-0
  * [25] Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (canadian institute for advanced research). Kaggle (2010)
  * [26] Li, T., Roy, S., Zhou, H., Lu, H., Lathuilière, S.: Contrast, stylize and adapt: Unsupervised contrastive learning framework for domain adaptive semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) Workshops. pp. 4868–4878 (June 2023)
  * [27] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: IEEE Conf. Comput. Vis. (ICCV) (2017)
  * [28] Nam, K.D., Nguyen, T.M., Dieu, T.V., Visani, M., Nguyen, T.O., Sang, D.V.: A novel unsupervised domain adaption method for depth-guided semantic segmentation using coarse-to-fine alignment. IEEE Access 10, 101248–101262 (2022). https://doi.org/10.1109/ACCESS.2022.3205414
  * [29] Pihur, V., Datta, S., Datta, S.: Weighted rank aggregation of cluster validation measures: a monte carlo cross-entropy approach. Bioinformatics pp. 1607–1615 (2007). https://doi.org/10.1093/bioinformatics/btm158
  * [30] Qu, S., Zou, T., Röhrbein, F., Lu, C., Chen, G., Tao, D., Jiang, C.: Upcycling models under domain and category shift. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2023)
  * [31] Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 413–420. IEEE (2009)
  * [32] Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: IEEE Eur. Conf. Comput. Vis. (ECCV). pp. 102–118 (2016). https://doi.org/10.1007/978-3-319-46475-6_7
  * [33] Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) pp. 3234–3243 (2016)
  * [34] Saha, S., Hoyer, L., Obukhov, A., Dai, D., Gool, L.V.: Edaps: Enhanced domain-adaptive panoptic segmentation. IEEE Int. Conf. Comp. Vis. (ICCV) (2023)
  * [35] Salehi, S.S.M., Erdogmus, D., Gholipour, A.: Tversky loss function for image segmentation using 3d fully convolutional deep networks. In: Machine Learning in Medical Imaging. pp. 379–387 (2017)
  * [36] Sammut, C., Harries, M.: Concept drift. In: Encyclopedia of Machine Learning. pp. 202–205 (2010). https://doi.org/10.1007/978-0-387-30164-8_153
  * [37] Santiago, C., Barata, C., Sasdelli, M., Carneiro, G., Nascimento, J.C.: Low: Training deep neural networks by learning optimal sample weights. Pattern Recognition 110, 107585 (2021). https://doi.org/https://doi.org/10.1016/j.patcog.2020.107585
  * [38] Saporta, A., Vu, T.H., Cord, M., Pérez, P.: Esl: Entropy-guided self-supervised learning for domain adaptation in semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. Workshop (CVPRW) (2020)
  * [39] Schwonberg, M., Niemeijer, J., Termöhlen, J.A., schäfer, J.P., Schmidt, N.M., Gottschalk, H., Fingscheidt, T.: Survey on unsupervised domain adaptation for semantic segmentation for visual perception in automated driving. IEEE Access 11, 54296–54336 (2023). https://doi.org/10.1109/ACCESS.2023.3277785
  * [40] Shen, F., Gurram, A., Liu, Z., Wang, H., Knoll, A.: Diga: Distil to generalize and then adapt for domain adaptive semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 15866–15877 (2023)
  * [41] Spelmen, V.S., Porkodi, R.: A review on handling imbalanced data. In: Int. Conf. Current Trends towards Converging Technologies (ICCTCT). pp. 1–11 (2018). https://doi.org/10.1109/ICCTCT.2018.8551020
  * [42] Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (2017)
  * [43] Tranheden, W., Olsson, V., Pinto, J., Svensson, L.: Dacs: Domain adaptation via cross-domain mixed sampling. IEEE Winter Conf. App. Comp. Vis. (WACV) pp. 1378–1388 (2020)
  * [44] Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 7472–7481 (2018). https://doi.org/10.1109/CVPR.2018.00780
  * [45] Vavasis, S.A.: Complexity theory: quadratic programmingComplexity Theory: Quadratic Programming, pp. 451–454 (2009)
  * [46] Veropoulos, K., Campbell, C., Cristianini, N.: Controlling the sensitivity of support vector machines. International Joint Conference Artificial Intelligence (IJCAI) (1999)
  * [47] Vu, T.H., Jain, H., Bucher, M., Cord, M., Pérez, P.: Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 2512–2521 (2019). https://doi.org/10.1109/CVPR.2019.00262
  * [48] Wang, K., Donghyun, K., Feris, R., Saenko, K., Betke, M.: Exploring consistency in cross-domain transformer for domain adaptive semantic segmentation. IEEE Int. Conf. Comp. Vis. (ICCV) (2023)
  * [49] Wang, S., Zhao, X., Chen, J.: Discovering latent target subdomains for domain adaptive semantic segmentation via style clustering. Multimedia Tools and Applications pp. 3234–3243 (2023). https://doi.org/10.1007/s11042-023-15620-6
  * [50] Wang, Z., Liu, X., Suganuma, M., Okatani, T.: Unsupervised domain adaptation for semantic segmentation via cross-region alignment. Computer Vision and Image Understanding p. 103743 (2023). https://doi.org/https://doi.org/10.1016/j.cviu.2023.103743
  * [51] Wu, X., Wu, Z., Lu, Y., Ju, L., Wang, S.: Style mixing and patchwise prototypical matching for one-shot unsupervised domain adaptive semantic segmentation. In: AAAI Conf. Art. Int. (2022)
  * [52] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Álvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
  * [53] Xu, X., He, H., Zhang, H., Xu, Y., He, S.: Unsupervised domain adaptation via importance sampling. IEEE Transactions on Circuits and Systems for Video Technology 30(12), 4688–4699 (2020). https://doi.org/10.1109/TCSVT.2019.2963318
  * [54] Yan, H., Ding, Y., Li, P., Wang, Q., Xu, Y., Zuo, W.: Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 945–954 (2017). https://doi.org/10.1109/CVPR.2017.107
  * [55] Yan, H., Li, Z., Wang, Q., Li, P., Xu, Y., Zuo, W.: Weighted and class-specific maximum mean discrepancy for unsupervised domain adaptation. IEEE Transactions on Multimedia PP, 1–1 (2019). https://doi.org/10.1109/TMM.2019.2953375
  * [56] Yeung, M., Sala, E., Schönlieb, C.B., Rundo, L.: Unified focal loss: Generalising dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Computerized Medical Imaging and Graphics p. 102026 (2022). https://doi.org/https://doi.org/10.1016/j.compmedimag.2021.102026
  * [57] Yuan, Y., Chen, X., Wang, J.: Object-contextual representations for semantic segmentation. In: IEEE Eur. Conf. Comput. Vis. (ECCV). pp. 173–190 (2020)
  * [58] Zhang, J., Huang, J., Jiang, X., Lu, S.: Black-box unsupervised domain adaptation with bi-directional atkinson-shiffrin memory. IEEE Int. Conf. Comp. Vis. (ICCV) (2023)
  * [59] Zhang, J., Huang, J., Zhang, X., Lu, S.: Unidaformer: Unified domain adaptive panoptic segmentation transformer via hierarchical mask calibration. In: IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). pp. 11227–11237 (2023). https://doi.org/10.1109/CVPR52729.2023.01080
  * [60] Zhao, R., Qian, B., Zhang, X., Li, Y., Wei, R., Liu, Y., Pan, Y.: Rethinking dice loss for medical image segmentation. In: IEEE Int. Conf. on Data Mining (ICDM). pp. 851–860 (2020). https://doi.org/10.1109/ICDM50108.2020.00094
  * [61] Zou, Y., Yu, Z., Kumar, B.V., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: IEEE Eur. Conf. Comput. Vis. (ECCV) (2018)


A Multi-loudspeaker Binaural Room Impulse Response Dataset with High-
Resolution Translational and Rotational Head Coordinates in a Listening Room
Yue Qiao∗, Ryan Miguel Gonzales, and Edgar Choueiri
Correspondence∗: Yue Qiao<EMAIL_ADDRESS>

##  1 Introduction

A binaural room impulse response (BRIR) describes characteristics of acoustic
wave interactions from a sound source in a room to the torso, head, and ears
of a listener. The use of BRIRs has been ubiquitous in many audio
applications. For example, in spatial audio reproduction with headphones,
BRIRs are used as audio filters to simulate or reproduce an immersive and
perceptually plausible sounding environment; in loudspeaker-based
applications, the frequency-domain counterparts of BRIRs are equivalent to the
acoustic transfer functions between the loudspeakers and the listener’s ears,
based on which audio filters are designed for tasks such as crosstalk
cancellation ([1],[2],[3]), room correction/loudspeaker equalization
([4],[5]), and personal sound zones ([6],[7],[8]). In addition to audio
reproduction and rendering, BRIRs have also played an important role in other
audio-related tasks, such as sound source localization [9], sound source
separation [10], and audio-visual learning [11].

As implied by its name, a BRIR is dependent on both the listener’s
anthropometric features (e.g., ear size and shape) and the room’s geometry and
acoustic properties. Due to the complex acoustic interactions, such as sound
reflections in the room and scattering off the listener, a BRIR varies with
both the listener’s position and orientation in the room. This is unlike room
impulse response (RIR) which only depends on the position, or anechoic head-
related impulse response (HRIR) which, in the far-field case, only depends on
the orientation. Although there have been multiple HRIR and RIR datasets
([12], [13], [14]) available, and it is possible to synthesize BRIRs from
HRIRs and RIRs using methods such as the image source model [15], synthesized
BRIRs lose physical accuracy and can only maintain perceptual plausibility.
While this is sufficient for some applications, such as headphone-based
auralization, it is not appropriate for others, such as crosstalk cancellation
and personal sound zones, where measured BRIRs are required. Moreover, the
lack of a high-resolution BRIR dataset, compared to the existing ones measured
at sparse listener positions and orientations ([16], [17], [18]), has limited
the study of BRIR modeling and interpolation at high frequencies and the
development of machine learning-based audio applications.

In this paper, we introduce a dataset that contains BRIRs measured in an
acoustically treated listening room using multiple loudspeakers and at high-
resolution translational and rotational head coordinates. Although the dataset
is only measured in one specific room, it is expected to be useful for a wide
range of applications as it faithfully captures the spatial dependency of
BRIRs on listener positions and orientations. For example, it can be directly
applied to studies of BRIR modeling and interpolation and applications that
require multi-loudspeaker BRIRs, such as crosstalk cancellation and personal
sound zones. It has been shown [19] that the spatial sampling resolution of
the dataset is adequate for rendering personal sound zones with continuous
listener movements within a certain frequency. With proper interpolation
between the measured BRIRs, the dataset can also be used to simulate binaural
audio for headphone-based auralization with continuous listener movements. In
addition, the dataset can be used for either data augmentation or performance
evaluation in a wide range of machine learning-based tasks that require
binaural audio with listener movements in multiple degrees of freedom.

##  2 Methods

###  2.1 Data Collection

We measured BRIRs in an irregular-shaped listening room with a near-shoebox
shape (see Fig. 2 for the exact dimensions). The listening room has a
$RT_{60}$ of 0.24 s averaged in the range between 1300 and 6300 Hz. The room
floor is covered with carpet, and the walls and ceiling are partially covered
with acoustic panels. Fig. 2 also shows the setup and dimensions of the
measurement system. A linear array of eight loudspeakers was used as sound
sources, and each is a Focal Shape 40 4-inch Flax woofer. The loudspeaker
array layout was initially intended for sound field control applications, such
as rendering personal sound zones. A Brüel & Kjær Head and Torso Simulator
(HATS, Type 4100) was used as the mannequin listener, with its built-in
microphones replaced with a pair of in-ear binaural microphones (Theoretica
Applied Physics BACCH-BM Pro). The microphones were calibrated and free-field
equalized before the measurement. A custom-made, computer-controlled
mechanical translation platform was applied to enable translational movements,
and a turntable (Outline ET250-3D) was mounted on top of the platform for
rotational movements in the azimuth.

The BRIR measurement grid has a range of [0.5, 1.0] m in the $y$ direction
(front/back) and [-0.5, 0.5] m in the $x$ direction (left/right), with a 0.05
m spacing between adjacent grid points. The distances are relative to the
center of the loudspeaker array. At each grid point, the BRIRs were measured
at 37 different azimuth angles from the listener facing left to facing right,
with a 5∘ spacing between adjacent angles. In total, there are 68376 (= 11
$y$-translations $\times$ 21 $x$-translations $\times$ 37 azimuthal rotations
$\times$ 8 loudspeakers) BRIRs measured.

We measured BRIRs by playing back exponential sine sweep (ESS) signals from
the loudspeakers and recording the signals received with binaural microphones.
Each sine sweep signal has a length of 500 ms at a 48 kHz sampling rate and is
generated using the synchronized ESS method [20], with a start frequency of
100 Hz and an end frequency of 24 kHz. The synchronized ESS method is a
variant of the traditional ESS method [21], with the advantage of correctly
estimating higher harmonic frequency responses. All eight loudspeakers were
triggered in series with no overlapping between the ESS signals.

The entire data collection process was split into multiple measurement
sessions. For each session, we manually fixed the distance from the listener
to the array in the $y$ direction and automated the movements in the $x$
direction and the azimuthal rotations. The measurement automation, signal
generation, and data collection were implemented in Cycling ’74 Max 8. The
BRIR post-processing was performed in MATLAB. Each session lasted for
approximately 2 hours, and the entire data collection process took 9 days.

###  2.2 Data Processing

The BRIRs were obtained by first deconvolving the recorded signals with the
ESS signal in the frequency domain, with 32768-length FFT at a 48 kHz sampling
rate. Then, a fourth-order highpass Butterworth filter with a cutoff frequency
of 100 Hz was applied to the deconvolved signals to remove the low-frequency
noise that was present during the measurement. Finally, the deconvolved
signals were truncated to the first 16384 samples (corresponding to 341.3 ms)
and globally normalized. No loudspeaker equalization was applied to the BRIRs
as the loudspeaker-specific information, such as directivity, is an integral
part of the BRIR and therefore is difficult to compensate for. The processed
BRIRs, together with the corresponding listener position and orientation
coordinates, were saved as separate files corresponding to different $y$
translations in the SOFA (spatially-oriented format for acoustics, [22])
format, following the AES69-2022 (SOFA 2.1) Standard. The dataset was
generated using SOFA Toolbox for MATLAB/Octave version 2.2.0.

##  3 Data Visualization

We examine the dataset by visualizing 1) the time index of the BRIR onset, 2)
the peak amplitude of the BRIR, and 3) the interaural time difference (ITD) of
the BRIR, as functions of the listener position. Both the onset and the ITD of
BRIRs were calculated in a thresholding approach (see [23] as an example).
Fig. 3 shows the onset, peak amplitude, and ITD of the BRIRs measured at the
fourth loudspeaker (counting from the left) and with the listener facing
forward. The colors in the figures were interpolated using the “interp” option
of the MATLAB “pcolor” function. All three figures show clear spatial
dependency on the listener’s position, with the onset increasing and the peak
amplitude decreasing as the listener moves away from the loudspeaker. The ITD
is nearly zero when the listener is on-axis with the loudspeaker and increases
as the listener moves to off-axis positions. Note that the maxima of the peak
amplitude in Fig. 3 are not on-axis with the loudspeaker, which is due to the
occlusion effect of the listener’s head and ear.

## Data Availability Statement

The dataset is publicly available at https://doi.org/10.34770/6gc9-5787.

## Funding

This work was supported by a research grant from Masimo Corporation.

## Acknowledgments

We wish to thank K. Tworek, R. Sridhar, and J. Tylka for their prior work on
developing the control software for the translation platform and the
turntable. We also wish to thank L. Guadagnin for his support on the
loudspeaker array.

## References

  * Cooper and Bauck [1989] Duane H Cooper and Jerald L Bauck.  Prospects for transaural recording.  _Journal of the Audio Engineering Society_ , 37(1/2):3–19, 1989.
  * Gardner [1998] William G Gardner.  _3-D audio using loudspeakers_ , volume 444.  Springer Science & Business Media, 1998.
  * Choueiri [2018] Edgar Choueiri.  Binaural audio through loudspeakers.  In Agnieszka Roginska and Paul Geluso, editors, _Immersive sound: the art and science of binaural and multi-channel audio_ , chapter 6, pages 124–179. Taylor & Francis, 2018.
  * Karjalainen et al. [1999] Matti Karjalainen, Esa Piirilä, Antti Järvinen, and Jyri Huopaniemi.  Comparison of loudspeaker equalization methods based on dsp techniques.  _Journal of the Audio Engineering Society_ , 47(1/2):14–31, 1999.
  * Lindfors et al. [2022] Joel Lindfors, Juho Liski, and Vesa Välimäki.  Loudspeaker equalization for a moving listener.  _Journal of the Audio Engineering Society_ , 70(9):722–730, 2022.
  * Druyvesteyn and Garas [1997] WF Druyvesteyn and J. Garas.  Personal sound.  _J. Audio Eng. Soc._ , 45(9):685–701, Sep. 1997\.
  * Betlehem et al. [2015] Terence Betlehem, Wen Zhang, Mark A Poletti, and Thushara D Abhayapala.  Personal sound zones: Delivering interface-free audio to multiple listeners.  _IEEE Signal Processing Magazine_ , 32(2):81–91, 2015.
  * Qiao and Choueiri [2023a] Yue Qiao and Edgar Choueiri.  The effects of individualized binaural room transfer functions for personal sound zones.  _Journal of the Audio Engineering Society_ , 71(12):849–859, 2023a.
  * Shinn-Cunningham et al. [2005] Barbara G Shinn-Cunningham, Norbert Kopco, and Tara J Martin.  Localizing nearby sound sources in a classroom: Binaural room impulse responses.  _The Journal of the Acoustical Society of America_ , 117(5):3100–3115, 2005.
  * Yu et al. [2016] Yang Yu, Wenwu Wang, and Peng Han.  Localization based stereo speech source separation using probabilistic time-frequency masking and deep neural networks.  _EURASIP Journal on Audio, Speech, and Music Processing_ , 2016:1–18, 2016.
  * Younes et al. [2023] Abdelrahman Younes, Daniel Honerkamp, Tim Welschehold, and Abhinav Valada.  Catch me if you hear me: Audio-visual navigation in complex unmapped environments with moving sounds.  _IEEE Robotics and Automation Letters_ , 8(2):928–935, 2023.
  * Sridhar et al. [2017] Rahulram Sridhar, Joseph G Tylka, and Edgar Choueiri.  A database of head-related transfer functions and morphological measurements.  In _Audio Engineering Society Convention 143_. Audio Engineering Society, 2017.
  * Brinkmann et al. [2019] Fabian Brinkmann, Manoj Dinakaran, Robert Pelzer, Peter Grosche, Daniel Voss, and Stefan Weinzierl.  A cross-evaluated database of measured and simulated hrtfs including 3d head meshes, anthropometric features, and headphone impulse responses.  _Journal of the Audio Engineering Society_ , 67(9):705–718, 2019.
  * Koyama et al. [2021] Shoichi Koyama, Tomoya Nishida, Keisuke Kimura, Takumi Abe, Natsuki Ueno, and Jesper Brunnström.  Meshrir: A dataset of room impulse responses on meshed grid points for evaluating sound field analysis and synthesis methods.  In _2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , pages 1–5. IEEE, 2021.
  * Wendt et al. [2014] Torben Wendt, Steven Van De Par, and Stephan D Ewert.  A computationally-efficient and perceptually-plausible algorithm for binaural room impulse response simulation.  _Journal of the Audio Engineering Society_ , 62(11):748–766, 2014.
  * Jeub et al. [2009] Marco Jeub, Magnus Schafer, and Peter Vary.  A binaural room impulse response database for the evaluation of dereverberation algorithms.  In _2009 16th International Conference on Digital Signal Processing_ , pages 1–5. IEEE, 2009.
  * Kayser et al. [2009] Hendrik Kayser, Stephan Dieter Ewert, Jörn Anemüller, Thomas Rohdenburg, Volker Hohmann, and Birger Kollmeier.  Database of multichannel in-ear and behind-the-ear head-related and binaural room impulse responses.  _EURASIP Journal on advances in signal processing_ , 2009:1–10, 2009.
  * Erbes et al. [2015] Vera Erbes, Matthias Geier, Stefan Weinzierl, and Sascha Spors.  Database of single-channel and binaural room impulse responses of a 64-channel loudspeaker array.  In _Audio Engineering Society Convention 138_. Audio Engineering Society, 2015.
  * Qiao and Choueiri [2023b] Yue Qiao and Edgar Choueiri.  Optimal spatial sampling of plant transfer functions for head-tracked personal sound zones.  In _Audio Engineering Society Convention 154_. Audio Engineering Society, 2023b.
  * Novak et al. [2015] Antonin Novak, Pierrick Lotton, and Laurent Simon.  Synchronized swept-sine: Theory, application, and implementation.  _Journal of the Audio Engineering Society_ , 63(10):786–798, 2015.
  * Farina [2000] Angelo Farina.  Simultaneous measurement of impulse response and distortion with a swept-sine technique.  In _Audio engineering society convention 108_. Audio Engineering Society, 2000.
  * AES69- [2022] AES69-2022.  Aes standard for file exchange-spatial acoustic data file format.  Standard, Audio Engineering Society, 2022.
  * Katz and Noisternig [2014] Brian FG Katz and Markus Noisternig.  A comparative study of interaural time delay estimation methods.  _The Journal of the Acoustical Society of America_ , 135(6):3530–3540, 2014.

(a) Photo of the measurement system. Note that the tweeter loudspeaker arrays
in the photo were not used in the data collection.

(b) Schematic of the listening room. Left: side view. Right: top view.

(a) Schematic diagram of the measurement system.

(b) Dimensions of the loudspeaker array.

Figure 2: Illustrations of the measurement system. Figure 3: Three example
figures showing the properties of the BRIRs measured at the fourth loudspeaker
(counting from the left) and with the listener facing forward. The $x$ axis of
the figures is aligned with the loudspeaker array. The dashed line in the
figure indicates the center of the fourth loudspeaker. Top figure: Onsets of
the left-ear BRIRs. Middle figure: Peak amplitude of the left-ear BRIRs.
Bottom figure: Calculated ITD of the BRIRs.