(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: University of Padova, Italy
11email<EMAIL_ADDRESS>

# When Cars meet Drones: Hyperbolic Federated Learning for Source-Free Domain
Adaptation in Adverse Weather

Giulia Rizzoli∗ 0000-0002-1390-8419 University of Padova, Italy
11email<EMAIL_ADDRESS>
Matteo Caligiuri∗ 0009-0006-2928-1047 University of Padova, Italy
11email<EMAIL_ADDRESS>
Donald Shenaj  0000-0002-6501-9437 University of Padova, Italy
11email<EMAIL_ADDRESS>
Francesco Barbato  0000-0001-9893-5813 University of Padova, Italy
11email<EMAIL_ADDRESS>
Pietro Zanuttigh  0000-0002-9502-2389 0000-0002-1390-8419 0009-0006-2928-1047
0000-0002-6501-9437 0000-0001-9893-5813 0000-0002-9502-2389 University of
Padova, Italy
11email<EMAIL_ADDRESS>

###### Abstract

In Federated Learning (FL), multiple clients collaboratively train a global
model without sharing private data. In semantic segmentation, the Federated
source Free Domain Adaptation (FFreeDA) setting is of particular interest,
where clients undergo unsupervised training after supervised pretraining at
the server side. While few recent works address FL for autonomous vehicles,
intrinsic real-world challenges such as the presence of adverse weather
conditions and the existence of different autonomous agents are still
unexplored. To bridge this gap, we address both problems and introduce a new
federated semantic segmentation setting where both car and drone clients co-
exist and collaborate. Specifically, we propose a novel approach for this
setting which exploits a batch-norm weather-aware strategy to dynamically
adapt the model to the different weather conditions, while hyperbolic space
prototypes are used to align the heterogeneous client representations.
Finally, we introduce FLYAWARE, the first semantic segmentation dataset with
adverse weather data for aerial vehicles.

**footnotetext: These authors contributed equally to this work.

FL

    Federated Learning
FFreeDA

    Federated source Free Domain Adaptation
GAN

    Generative Adversarial Network
OD

    Object Detection

##  1 Introduction

The field of autonomous driving has evolved significantly beyond the
traditional concept of cars navigating city streets. In the contemporary
landscape, autonomous navigation encompasses various perspectives, including
drones and robots, serving a wide range of applications like autonomous
delivery and surveillance. The demand for intelligent systems that can
seamlessly generalize across these varying viewpoints is quickly increasing,
to enhance safety, efficiency, and adaptability. A key challenge lies in
achieving this goal while preserving confidentiality and data security. The
importance of robust privacy measures in developing autonomous systems cannot
be overstated, as these systems often rely on substantial amounts of data to
function effectively. The traditional method of gathering and analyzing data
in a centralized manner raises relevant security concerns. Federated learning
allows tackling this issue by introducing a cutting-edge paradigm that enables
the collaborative training of machine learning models without sharing the
private data of the participants (_i.e_., clients). In this paper, we
introduce a novel federated learning approach tailored to the specific
challenges encountered in driving scenarios, where autonomous agents have
different viewpoints, specifically the street-view vision for cars and the
aerial for UAVs. Multi-viewpoint paradigms have been proven beneficial in
various tasks among which localization and segmentation [35, 31, 18], where
jointly training the model with images from different viewpoints improves the
objective task. Consequently, we enhance our framework by integrating aerial
data. Moreover, we exploit this strategy in combination with the recent
federated learning paradigm proposed in [37] that exploits a supervised server
pretraining to enable unsupervised training on the client side. Additionally,
our approach not only enables knowledge sharing between clients with different
identities - either cars or drones - but also dynamically adapts to ever-
changing weather conditions, aligning with the real-world challenges in this
dynamic field. In this scenario (shown in Fig. 1), several key challenges
arise:

  1. 1.

Clients operating without supervision tend to become overly confident in their
predictions, causing the model to diverge.

  2. 2.

Clients exhibit differing distributions, and varying weather conditions can
significantly impact performance (i.e., clients are non-iid, heterogeneous).

  3. 3.

Clients are unbalanced, reflecting real-world scenarios where certain types of
agents may be less common (e.g., fewer drones compared to cars).

Figure 1: Overview of the considered setting: after a supervised pretraining
on synthetic data, a heterogeneous set of unsupervised clients collaboratively
optimize their model in a federated fashion. Notably, adapting the model also
to varying weather conditions.

To address these challenges, we present several novel contributions:

  * •

We introduced an unsupervised federated learning system for autonomous cars
that can benefit from aerial view integration and seamlessly switch between
the two perspectives.

  * •

We introduced HyperFLAW(Hyperbolic Federated Learning in Adverse Weather),
which addresses domain shifts in diverse weather conditions using weather-
aware batch normalization layers and integrates the first semantic
segmentation dataset with adverse weather imagery for aerial vehicles.

  * •

We exploited prototype-based learning in the hyperbolic space to ensure
consistent training across the clients in this very challenging setting.

##  2 Related Work

Domain Adaptation (DA) primarily focuses on adapting a model from a source to
a target domain, typically in scenarios where the target data lacks labels,
referred to as Unsupervised Domain Adaptation (UDA) [36, 43]. Historically, DA
methods aimed to bridge the gap by quantifying domain divergence [45, 33].
Adversarial learning has also been widely used [44, 24, 30]. In the
contemporary landscape, advanced approaches [54, 15, 28] leverage self-
learning techniques. Several other approaches have been used to align the
domains such as image-to-image translation [47, 1, 6], pseudo-label prototypes
[51, 23, 50], and confidence thresholding [28, 54]. Within the objective of
adapting to adverse weather conditions, Bruggemann et al. [3] propose a
solution that requires sunny daytime references. To tackle nighttime images,
Xiao et al. [46] employed a data augmentation method leveraging multiple
modalities. Finally, Source Free Domain Adaptation (SFDA) presents a distinct
advantage in addressing the performance gap by allowing adaptation without
access to labeled source data [19].

Federated Learning has captured the interest of the research community due to
its practical applicability in various real-world scenarios and its potential
to manage sensitive data [27]. In literature, it has been applied to different
vision tasks [38], initially with a large emphasis on classification [21, 20],
but more recently also focusing on other tasks including semantic segmentation
for Autonomous Driving [5, 11, 37]. Most existing works in FL assume labeled
data on remote clients [5], which is an unrealistic assumption, especially in
the driving scenario. Notably, Shenaj et al. [37] introduce the Federated
source Free Domain Adaptation (FFreeDA) setting, where only the server
accesses a labeled source dataset. We address this challenge by incorporating
the management of diverse viewpoints, accommodating various client types, and
dealing with the influence of adverse weather conditions.

Hyperbolic Prototypical Learning.  Working in Euclidean spaces, implicitly or
explicitly assume that data is best represented on regular grids. While this
model offers an intuitive and grounded underlying manifold, its properties may
not be the most suitable for all types of data [29]. Inspired by developments
in other fields, deep learning in hyperbolic space has recently been exploited
in computer vision using different isomorphic models, such as the Poincaré
model. Several works have proposed prototypes-based hyperbolic embeddings for
few-shot learning for classification [17, 14, 26], where hyperbolic space
offers advantages over the Euclidean one. In particular, Khrulkov et al. [17]
demonstrated its competitiveness on simple ConvNets while Gao et al. [12] show
the benefits achievable with different curvatures. Atigh et al. [2] introduced
approximations for faster computations tailored for use in semantic
segmentation. Facing these insights, hyperbolic prototypical learning seems
suitable for learning features robust to domain shift and varying weather
conditions even when employing different curvatures and lightweight networks,
as those used on clients in federated scenarios. Building upon these
foundations, our approach exploits hyperbolic learning to align features from
diverse viewpoints while reducing domain shift.

##  3 Problem Formulation

Most works in Federated Learning operate under the assumption of labeled data
being available on remote clients [5]. Providing semantic segmentation ground
truth is extremely time-consuming and costly, thus it is quite unrealistic to
expect labeled data on real-world clients, especially during the deployment
phase. Therefore, our objective is to develop a model that can seamlessly
adapt from labeled synthetic data to unlabeled real-world images in a
distributed paradigm. For this reason, we chose to deploy our setting on top
of the pre-existing FFreeDA scenario [37]. Moreover, we discard the assumption
that the clients are homogeneous in nature and consider a setting where
different autonomous agents work cooperatively in fluctuating atmospheric
conditions (_i.e_., clear sky, night, rain and fog). In the following, we will
first formalize the problem and then move to a detailed description of the
methodologies proposed to tackle it in Sec. 4.

Source-Free Domain Adaptation (SFDA). Consider an image
$x\\!\in\\!\\!\mathcal{X}\\!\subset\\!\\!\mathbf{R}^{|\mathcal{I}|x3}$ and its
corresponding label $y\in\mathcal{Y}\subset\mathbf{R}^{C}$, where
$|\mathcal{I}|=H\times W$ are the image dimensions and $C$ is the number of
classes. SFDA entails a pretraining phase on a source dataset
$\mathcal{D}^{S}=\\{(x^{s},y^{s})\\}^{N^{s}}$, made of pairs of images and
labels, and an adaptation stage on a target dataset
$\mathcal{D}^{T}=\\{(x^{T})\\}^{N^{T}}$, composed of unlabeled images.
Specifically, compared to standard DA: 1) the two training phases are
distinct; 2) $\mathcal{D}^{S}$ is unavailable at the adaptation phase.

Federated source-Free Domain Adaptation (FFreeDA). This setting deals with a
distributed setup comprising a central server and a set of clients
$\mathcal{K}$ with $\left|\mathcal{K}\right|=K$. The training datasets are
organized as follows: the primary dataset $\mathcal{D}^{S}$, resides on the
server side. Concurrently, each of the $K$ client has a distinct training
dataset denoted as $\mathcal{D}^{T}_{k}=\\{x_{k,i}^{T}\in\mathcal{X}:\forall
i\in|\mathcal{D}^{T}_{k}|\\}$. Notably, these client datasets are locally
stored and managed, ensuring that
$\mathcal{D}^{T}_{1}\cap\mathcal{D}^{T}_{2}\cap\ldots\cap\mathcal{D}^{T}_{K}=\varnothing$.
In distributed settings, the parameter $K$ is reasonably large, and the local
datasets vary in terms of statistics, _i.e_. size and distribution. However,
they typically possess a much smaller size compared to the source dataset
($N^{S}\\!\gg N^{T}$). FFreeDA assumes that the local datasets are drawn from
the same meta-distribution, meaning that $\mathcal{D}_{k}^{T}$ comprises
images solely from one latent domain. The objective is to achieve optimal
segmentation performance on the target data distribution with the model
$M(\theta):\mathcal{X}\rightarrow\mathbf{R}^{|\mathcal{I}|\times|\mathcal{Y}|}$,
parameterized by $\theta$. This goal is pursued by minimizing an appropriate
loss function, expressed as:

$\theta^{*}=\arg\min_{\theta}\sum_{k\in[K]}\frac{|\mathcal{D}_{k}^{T}|}{|\mathcal{D}^{T}|}\mathcal{L}_{k}(\theta)$
(1)

where $\mathcal{L}_{k}$ represents the local loss function, and
$\mathcal{D}^{T}=\bigcup_{k\in\mathcal{K}}\mathcal{D}^{T}_{k}$.

Multi-viewpoint FFreeDA in Adverse Weather. Aiming to assist agents lacking
large training datasets or capabilities, such as drones, we consider a setting
that enables scene understanding for unsupervised autonomous navigation robust
to multiple viewpoints. First of all, we consider
$\mathcal{D}^{S}=\mathcal{D}^{S}_{car}\cup\mathcal{D}^{S}_{drone}$. Accounting
for adverse weather as a heterogeneity factor, our dataset
$\mathcal{D}^{S}=\\{(\mathbf{x}^{S},\mathbf{y}^{S},w^{S})\\}^{N^{S}}$ also
includes $w^{S}$, denoting the atmospheric condition for each frame, together
with the color images $\mathbf{x}^{S}$ and the ground truth segmentation maps
$\mathbf{y}^{S}$. Similarly, the client can accommodate either car or drone
agents: $\mathcal{D}^{T}=\mathcal{D}^{T}_{car}\cup\mathcal{D}^{T}_{drone}$,
although clients datasets are still local and disjoint. As in the previous
setting, each client $k\in\mathcal{K}$ have its own set of unsupervised images
both in terms of semantics and weather:
$\mathcal{D}^{T}_{k}=\\{(\mathbf{x}^{T}_{k})\\}$. Notably, the number of
samples of the drones’ target dataset ($N^{T}_{drone}$) is typically lower
than in the target datasets for cars ($N^{T}_{car}$). Furthermore, the set of
semantic classes could differ between car and aerial vehicles datasets (in our
case $C_{drone}\subset C_{car}=C^{T}=C^{S}$). The identity of each client is
fixed, meaning it is designated as either a car or a drone and does not change
during training, however, there is no such limitation on the weather
conditions, which can change dynamically during the training process. The goal
is to minimize the objective function of Eq. 1, where the target data
distribution holds into its heterogeneity different viewpoints and weathers.
In Fig. 2 we present our approach, which will be detailed in Sec. 4.

##  4 Training Procedure

The training is organized in a set of rounds $r=0,\ldots,R$, where $r=0$
corresponds to the pretraining phase. At each round $r$, we denote by
$M^{r}_{G}$ the global model at the server side, while we refer to $M^{r}_{k}$
as the model of the $k-th$ client.
Pretraining. We train the initial model $M^{0}_{G}$ on a large-scale source
synthetic dataset $\mathcal{D}^{S}$, taking advantage of the server’s
computing capabilities. By sampling batches of cross-agent we maximize the
standard cross entropy loss $\mathcal{L}(y^{S},M_{G}^{0}(x^{S}))$. Moreover,
an additional component $M_{W}$ consisting of 3 convolutional layers is
trained to recognize weather conditions by optimizing
$\mathcal{L}(w^{S},M_{W}(x^{S}))$. This classifier is a lightweight module
intended to provide weather information for clients and it is not trained
after this phase. Differently from [37], we do not assume to have information
on the clients’ distribution (i.e., in [37] the average Fourier domain
coefficients from $\mathcal{D}^{T}$ are used, requiring priors on clients’
data).
Adaptation. At each round $r$, we consider that, among all the set of clients
$\mathcal{K}$, only a subset denoted as $\mathcal{K}^{r}$ participate in the
training. This assumption is common in FL where not all the clients might be
concurrently active and reflects well on our scenario (e.g. a car is parked).
Then, the server transmits the global model from the previous round
$M_{G}^{r-1}$. Note that the weather classifier $M_{W}$ is only transmitted
once as it is fixed. The training for the active clients will be performed
separately on $M_{k}^{r},k\in\mathcal{K}^{r}$. Furthermore, a prototype
representation $\mathbf{p_{c,k}^{r}}$ for each class (i.e., the centroids of
the class samples in feature space) is also built and transmitted to the
server together with the manifold curvature $\gamma_{k}$ (refer to Sec. 4.3).

Figure 2: The two-step training process of HyperFLAW: server pretraining on
synthetic data and client-side real-world adaptation. We highlight the active
modeling of weather conditions using weather-aware batch norm layers and the
use of hyperbolic space for feature alignment via prototypical learning,
ensuring consistency in features extracted by car and drone agents across
various atmospheric conditions. At the server, the aggregation with the queue
of previous global models reduces the instability introduced by aggregating
unlabeled clients.

Finally, the models are aggregated at the server side with the approach of
Sec. 4.4 obtaining the global model ${M}_{G}^{r}$ that is used as input for
the next step. Prototypical representations are also aggregated at the server
side obtaining a set of global class prototypes $\mathbf{p^{r}_{G,c}}$.

###  4.1 Client Self-Training (ST)

Self-training involves iteratively enhancing a model’s performance by using
its own predictions. In FL, the clients can rely on the server model as the
teacher, as ideally, the server aggregates the global knowledge from the
targets. In our framework, we update the teacher model after the round is
over. This implies that the global model received by the clients at the
beginning of each round $r$, $M_{G}^{r-1}$, serves also as the current
teacher. This approach offers the advantage of eliminating the necessity to
transmit two models per round as in [37] where also a model from $R_{u}$ steps
before was used, thus requiring only a single model transmission and avoiding
the extra hyperparameter $R_{u}$. The local model of the $k-th$ client is
trained using the weak supervision of the pseudo-labels produced by the
teacher model $M_{G}^{r-1}$ and its local model prediction $M^{r}_{k}$ by
optimizing the self-training loss obtained via cross-entropy as
$\mathcal{L}_{st}(M_{G}^{r-1}(x_{k}^{T}),M^{r}_{k}(x_{k}^{T}))$. Moreover, to
achieve a training that is consistent across the heterogeneous clients and a
model able to generalize to different types of agents and weather conditions
the self-training loss is used in combination with the prototype-based one,
i.e.,

$\mathcal{L}_{k}(\theta)=\mathcal{L}_{st}(\theta)+\lambda_{cl}\mathcal{L}_{cl}(\theta)$
(2)

where $\theta$ are the model parameters, $\mathcal{L}_{cl}$ is the clustering
loss computed using prototypical representation (see Sec. 4.3), and
$\lambda_{cl}$ balances the two losses.

###  4.2 Weather Batch Normalization (BN)

Inspired by [22] where the update of the batch normalization layers is not
synchronized within the global model, we personalize the training by
considering a different set of batch-normalization (BN) parameters for each
weather condition $w$, defined as $\\{\theta^{BN}_{w}\\}$. After the
pretraining ($r=0$), all the weather-specific BNs inherit the global BN
statistics $\theta^{BN}$, while in each subsequent round, the statistics of
the BNs are averaged only over those of the same weather condition.
Specifically, on the client side, after obtaining the prediction
$\bar{w}=M_{W}(x^{T})$ from the classifier, the client batch-norm parameters
are optimized using Eq. 2 only on the selected weather-BNs
$\theta^{BN}_{\bar{w}}$. This allows us to enforce a more general feature
extraction from the convolutional layers while exploiting the domain-specific
nuances of road scenes under different weather conditions.

###  4.3 Prototypical Learning

While the various devices explore road scenes, their distinct viewpoints force
the local models to conform more closely to the features within each specific
perspective. In the pretraining phase, drone and car samples are provided
jointly, however, this simultaneity does not apply to clients, thereby the
network predictions could diverge, making the subsequent aggregation on the
server side more challenging. To ensure the ongoing alignment of features
across the two data types, we propose the exploitation of joint class
prototypes as a viable solution. For each client, we compute a set of class
prototypes $\mathcal{P}$, where each prototype $\mathbf{p}_{c}\in\mathcal{P}$
is defined as a function of the set $\mathcal{F}_{c}$ of all feature vectors
$\textbf{f}_{c}$ from the encoder output that are associated to class $c$:

$\mathbf{p_{c}}=g(\mathcal{F}_{c}),\;\mathcal{F}_{c}\\!=\
\\{\mathbf{f}=E(\mathbf{x})(h,w):\;\hat{\mathbf{y}}(h,w)=c\\}$ (3)

where $\hat{\mathbf{y}}$ are the teacher pseudo-labels subsampled at the
feature space resolution and the function $g$ computes either the mean or the
midpoint depending on the considered manifold: Euclidean or Poincaré Ball,
respectively.

At the beginning of each round $r$, a participating client $k$ initializes its
local prototypes $\mathbf{p_{k,c}^{r}}$ using the previous round ones received
from the server: $\mathbf{p_{k,c}^{r}}=\mathbf{p_{G,c}^{r-1}}$ . Then, the
prototypes are updated at training time using exponential smoothing with rate
$\beta=0.85$ to stabilize their evolution as follows:

$\mathbf{p_{c}}=\beta\;\mathbf{p^{\prime}_{c}}+(1-\beta)\;\mathbf{p_{c}}$ (4)

where $\mathbf{p^{\prime}_{c}}$ is the prototype from the previous step. After
each round, prototypes $\mathbf{p_{k,c}^{r}},\forall c\in\mathcal{C}$ computed
by each client are sent back to the server for aggregation.

Hyperbolic Protoypes. As mentioned before, to further improve performance, we
exploited the capacity of hyperbolic space - specifically of the Poincaré ball
model - to fit complex feature geometries better and used it to embed the
prototypes. We refer to [29, 40] for further discussion. The hyperbolic space
represented in Riemannian geometry, with dimension $n$ and constant negative
curvature $-\gamma$ is expressed as:

$\mathbb{B}_{\gamma}^{n}=\\{\mathbf{x}\in\mathbb{R}^{n}:\|\mathbf{x}\|^{2}<1/\gamma\\}$
(5)

On such space, the distance between two points $\mathbf{x}$,
$\mathbf{y}\in\mathbb{B}_{\gamma}^{n}$ can be computed as:

$d_{\gamma}(\mathbf{x},\mathbf{y})=\left(\frac{2}{\sqrt{\gamma}}\right)\tanh^{-1}\left(\sqrt{\gamma}\|\mathbf{-x}\oplus_{\gamma}\mathbf{y}\|\right)$
(6)

where $\mathbf{x}\oplus_{\gamma}\mathbf{y}$ is the Möbius addition:

$\mathbf{x}\oplus_{\gamma}\mathbf{y}=\frac{(1+2\gamma\langle\mathbf{x},\mathbf{y}\rangle+\gamma\|\mathbf{y}\|^{2})\mathbf{x}+(1-\gamma\|\mathbf{x}\|^{2})\mathbf{y}}{1+2\gamma\langle\mathbf{x},\mathbf{y}\rangle+\gamma^{2}\|\mathbf{x}\|^{2}\|\mathbf{y}\|^{2}}$
(7)

The translation from the Euclidean to the hyperbolic space is obtained given
the exponential mapping in the direction of a tangent vector
$\mathbf{v}\in\mathcal{T}\mathbb{B}_{\gamma}^{n}$:

$\exp_{\mathbf{x}}^{\gamma}(\mathbf{v})=\mathbf{x}\oplus_{\gamma}\left(\tanh\left(\frac{\sqrt{\gamma}\|\mathbf{v}\|}{1-\gamma\|\mathbf{v}\|^{2}}\right)\frac{\mathbf{v}}{\sqrt{\gamma}\|\mathbf{v}\|}\right)$
(8)

The prototypes are determined as the midpoints of the hyperbolic feature
vectors $\\{\mathbf{f}_{i}:=\mathbf{f}^{hyp}_{i}\in\mathbb{B}_{c}^{n}\\}$,
using an approximation of the Frechet mean:

$\mathbf{p}=\frac{1}{2}\otimes_{\gamma}\frac{\sum_{\mathbf{f}\in\mathcal{F}}\lambda_{\mathbf{f}}^{\gamma}\mathbf{f}}{\sum_{\mathbf{f}\in\mathcal{F}}(\lambda_{\mathbf{f}}^{\gamma}-1)}$
(9)

where the Möbius scalar multiplication is:

$r\otimes_{\gamma}\mathbf{x}=\frac{1}{\sqrt{\gamma}}\tanh\left(r\tanh^{-1}(\sqrt{\gamma}\|\mathbf{x}\|)\right)\frac{\mathbf{x}}{\|\mathbf{x}\|}$
(10)

Since the curvature $\gamma$ is a scalar value, we can convert it to a
learnable parameter of the network and optimize it during the training process
[41]. This is done on the client side using self-supervision and prototype
loss, then, at the end of each round, the curvature of the active clients is
aggregated using FedAvg algorithm, i.e., the average is weighted by the number
of seen samples.

Prototypical Loss. Due to the distribution mismatch between source and target
domains and the different viewpoints, feature vectors from these domains
become misaligned, leading to inaccuracies in target representations and a
subsequent decline in segmentation accuracy. To address this issue, we adopt a
clustering objective in the latent space by leveraging prototypical
representations:

$\mathcal{L}_{cl}=\frac{1}{|\hat{\mathcal{C}}|}\sum_{c\in\hat{\mathcal{C}}}\frac{1}{|\mathcal{F}_{c}|}\sum_{\mathbf{f_{c}\in\mathcal{F}_{c}}}d(\mathbf{f}_{c},\mathbf{p}_{c})$
(11)

where
$\hat{\mathcal{C}}\;\coloneqq\;\\{c\in\mathcal{C}:\;|\mathcal{F}_{c}|>0\\}$ is
the set of active classes in the current batch.

###  4.4 Server Aggregation (AG)

Regularization. As mentioned earlier, the wide range of data variability
across client devices, caused by various types of autonomous agents and
changing weather conditions, makes aggregation at the server side challenging.
Moreover, unrestricted self-training on the client side can lead to
instability (see training curve when only $\mathcal{L}_{st}$ is active in Fig.
5) and to a decrease in performance after several rounds. To stabilize the
results with respect to standard averaging [27], we used a combination of the
models received from the clients with the global ones available at server side
from previous steps. In this way, during the first rounds, there is a smoother
adaptation from the pretrained model. More in detail, the aggregated model on
the server side is computed as:

$M_{G}^{r}=\frac{\frac{\sum_{k\in\mathcal{K}^{R}}M^{r}_{k}}{|\mathcal{K}^{R}|}+\sum_{j=1}^{Q}M_{G}^{r-j}}{Q+1}$
(12)

i.e., we get the output of federated averaging and smooth it by averaging with
the $Q$ previous rounds, thus making the learning more stable [4, 39]. Other
approaches [37] tackle regularization by exploiting the global model from
several snapshots ago, _i.e_. $M_{G}^{0}$. The idea behind this is to distill
knowledge from past predictions, where the server has higher confidence.
However, this introduces additional transmission requirements at each round or
necessitates clients to allocate resources for storing $M_{G}^{0}$. Moreover,
clients must perform additional inference and computation for each image. We
address these challenges by implementing a strategy that only requires effort
from the server side, _i.e_., by maintaining a queue of previous model
snapshots ($Q$). We ensure that the burden falls on the server, where memory
and computational capabilities are less constrained. We will denote this
aggregation strategy as AG in the results to distinguish it from standard
FedAvg.
Prototype aggregation. Prototypes are also aggregated on the server side. To
this aim, we used a combination of the previously computed prototypes and the
ones received by the clients. To determine the updates, we calculated a
weighted average of the prototypes received by the clients and smoothed the
result by considering the prototypes from the previous round:

$\mathbf{p}_{G,c}^{r+1}=\beta^{\prime}\mathbf{p}_{G,c}^{r-1}+(1-\beta^{\prime})\sum_{k\in\mathcal{K}^{r}}N_{k,c}^{r}\mathbf{p}_{k,c}^{r}$
(13)

where $N_{k,c}^{r}=|\mathcal{F}_{k,c}|$ at round $r$ and
$N_{c}^{r,tot}=\sum_{k\in\mathcal{K}}N_{k,c}$. We used the same smoothing
parameter of Eq. 4, i.e., $\beta^{\prime}=\beta=0.85$.

Table 1: Data distribution ($\dagger$: with Cityscapes as clear, see Suppl.
Mat. )

|  |  | Clear | Night | Rain | Fog
---|---|---|---|---|---|---
SYNTH | SELMA | Train | 24735 | 24735 | 24735 | 24735
Test | 3087 | 3087 | 3087 | 3087
FLYAWARE-S | Train | 24771 | 24731 | 24808 | 24631
Test | 3147 | 3139 | 3128 | 2935
REAL | ACDC† | Train | 1060 | 400 | 400 | 400
Test | 214 | 100 | 100 | 100
FLYAWARE-R | Train | 94 | 35 | 35 | 35
Test | 18 | 18 | 17 | 17
FLYAWARE-R-XL | Train | 565 | 565 | 565 | 565
Test | 112 | 53 | 52 | 52

FA-S

FA-R

Day

Night

Rain

Fog

Figure 4: Samples from different datasets under varying weather conditions.

##  5 The FLYAWARE Adverse Dataset

In standard autonomous driving for semantic segmentation, there is a
sufficient number of datasets providing a diversity of adverse conditions [42,
34]. However, in the case of aerial viewpoints, a scarcity of such datasets is
evident, thus making the development of a dataset of this type a priority to
unlock further research on the topic. To this aim, we introduce a new
benchmark FLYAWARE - FLYing in Adverse Weather: Artificial and REal data. In
the following section, we outline the structure of the synthetic and real
aerial datasets, $\mathcal{D}_{drone}^{S}$ and $\mathcal{D}_{drone}^{T}$,
respectively. They contain over 100K for which we contribute with 108180
generated images and 1287 domain-translated ones. In Tab. 1, we present the
datasets used in this study. For additional information regarding the datasets
for autonomous cars, please consult the Suppl. Mat.

FLYAWARE-S: Adverse Synthetic Dataset. Recently released, Syndrone [32] is a
synthetic dataset based on the CARLA simulator [8]. While the dataset is
richly annotated with 28 semantic classes, it currently lacks imagery in
different weather conditions. We extended their work starting from the
codebase provided in [32] by generating images in 3 adverse weather conditions
(_i.e_., rain, fog, and night), while maintaining the capability of the system
to produce images from multiple viewpoints and heights. The dataset includes
drone views at heights ranging from 20 to 80 meters and with angles varying
from 30 to 90 degrees, all with a resolution of 1920x1080. Moreover, we also
generated 3D data (depth maps and LiDAR) for future usage in multimodal
architectures.

FLYAWARE-R: Real Dataset Translation. Since no adverse weather dataset for
aerial vehicles is available we opted for using image translation over
standard drone datasets to build the adverse weather imagery. Addressing the
task of converting daytime images into adverse conditions requires the
availability of images of clear weather samples like sunny or cloudy
conditions, and of adverse samples occurring in specific conditions of
interest, such as rainy weather. Inspired by [13] which performs adverse
domain translations for autonomous driving in the context of depth estimation,
we opted for the use of Generative Adversarial Networks. For the rain and
night adverse conditions, we employed the ForkGAN model [52] to translate
clear-day training samples from the UAVid dataset [25]. While for the fog
samples, we have used a combination of Omnidata [10] and FoHIS [49] methods.
In the following, we will present an in-depth explanation of how we have
performed the aforementioned conversion.

Day2Night. To convert daylight into nighttime, we trained ForkGAN over $14K$
images, half of which represent daytime and the other half nighttime. All the
used images were sampled from Visdrone [53], and UAVDT [9], two datasets
designed for Object Detection. We have decided to sample the training data
from different datasets to increase the overall number of nighttime images.
This was necessary to ensure that the GAN architecture reaches good enough
reconstruction performance. The training phase lasted $40$ epochs. After the
training, for all the nighttime samples in the UAVid dataset, we convert
clear-day images into their nighttime counterparts using the pretrained model.

Day2Rain. Training the GAN models directly on drone data proved advantageous
as it helped to mitigate domain shifts. In the rain case however there are no
available drone datasets - not even for other tasks as in the case of night
images -, therefore we had to train ForkGAN using diverse datasets that
included adverse weather scenarios, like BDD100K [48], ACDC [34] and
RainCityscapes [16] (which are datasets for automotive applications). For
daytime to rainy conversion, we followed the same strategy applied for the
conversion to nighttime training the ForkGAN model over $9K$ images for $40$
epochs and we applied the pretrained model to convert images into their rainy
counterparts.

Day2Fog. To perform daytime to foggy conversion we first estimated a depth map
for each image in the UAVid dataset using Omnidata [10], then, we exploited
the estimated depth as input to the FoHIS method [49] to apply fog.

Federated Splits. We consider two scenarios: the first is an unbalanced setup
with $40$ clients, corresponding to $32$ automotive clients and $8$ drones;
the second is more balanced having $32$ automotive clients and $32$ drones,
however the training and test set of the drones come from two different
domains. Aligned with previous works [37], the clients have a different number
of samples associated with them, in particular, the number of samples in each
client is in the range $[69,72]$ for cars and $[24,25]$ for drones. To
simulate real scenarios, we assumed that the most common weather would be
clear daytime, while clients can have different combinations of adverse
weather. Refer to the Suppl. Mat. for the weather distribution details on
clients.

##  6 Implementation Details

Datasets. In this study, we leveraged two synthetic driving datasets - one
collected from cars and one from drones - each encompassing a range of
challenging illumination and weather conditions: the driving dataset SELMA
[42] and the FLYAWARE-S dataset introduced in this work. Both datasets are
based on the CARLA simulator [8] for image generation, making them ideal
choices for our research due to their shared framework. Then, our method is
adapted in a federated fashion to real-world datasets: for cars we selected
the ACDC dataset, supplemented by Cityscapes [7] for the clear samples as
described in the Suppl. Mat. As the drone counterpart, we used our adverse-
weather translated dataset FLYAWARE-R, which offers labeled frames captured
from aerial views. Additionally, to enhance the robustness of our approach, we
performed also tests using the VisDrone object detection dataset for clients’
training as training at client side is unsupervised. In this setting the
entire FLYAWARE-R dataset can be used for testing purposes (_i.e_. XL) for
more reliable results. The adverse translation of FLYAWARE-R is described in
Sec. 5. All the extended datasets and splits are referenced in Fig. 4 and will
be available at the link: [soon]
Server pretraining. We chose DeepLabV3 architecture with MobileNetV2 as in
[37]. We trained our model on the source synthetic dataset using a decreasing
power-law learning rate $\eta$, starting at $\eta=5e-3$ with a power of 0.9.
The optimization used SGD with momentum 0.9 and no weight decay, lasting for 5
epochs with batch size 16. For the weather classifier, we trained a 3-layer
ConvNet with batch size 88, 8 epochs and SGD optimizer with learning rate
$1e-4$.
Clients Adaptation. For the target dataset, experiments were run with a fixed
learning rate of $\eta=1e-4$ with SGD optimizer. The training involved 5
clients per round for a total of $R=100$ rounds, with $\lambda_{cl}=140$. The
pseudo-label teacher model was updated at the end of each round. For the
optimizer of the manifolds, we use RiemmanianAdam as in [40] with $\gamma=0.1$
as initialization, learning rate equal to $1e-3$ and weight decay of $4e-4$.

Table 2: mIoU on the target datasets (ACDC and FLYAWARE-R (FA-R)).

Setting | Train | Test | Method | Car | Drone | All
---|---|---|---|---|---|---
centralized | FA-S+SELMA | ACDC+FA-R | Source Only | 19.96 | 33.11 | 22.70
centralized | ACDC+FA-R | ACDC+FA-R | Target Only | 60.17 | 50.14 | 58.08
federated | ACDC+FA-R | ACDC+FA-R | Fine-Tuning | 34.35 | 39.10 | 35.34
FFreeDA | ACDC+FA-R | ACDC+FA-R | FedAvg [27]+ST | 25.00 | 30.25 | 26.10
FFreeDA | ACDC+FA-R | ACDC+FA-R | LADD [37] | 25.95 | 30.79 | 26.96
FFreeDA | ACDC+FA-R | ACDC+FA-R | HyperFLAW | 27.45 | 34.69 | 28.96
centralized | FA-S+SELMA | ACDC+FA-R-XL | Source Only | 19.96 | 32.61 | 22.61
FFreeDA | ACDC+VisDrone | ACDC+FA-R-XL | FedAvg [27]+ST | 22.76 | 31.55 | 24.59
FFreeDA | ACDC+VisDrone | ACDC+FA-R-XL | LADD [37] | 26.32 | 28.93 | 26.86
FFreeDA | ACDC+VisDrone | ACDC+FA-R-XL | HyperFLAW | 27.10 | 35.45 | 28.84

##  7 Experimental Results

The results are detailed in Tab. 2, which reports the average mIoU for cars,
drones and a combined score. The latter is computed using the average for the
classes present in both datasets and the car mIoU otherwise. For comparison,
we initially examined 3 supervised settings: 1) the source only approach,
which relies exclusively on synthetic labeled data from the source domains for
training; 2) the fully centralized approach, where all available data is used
to train the network specifically on the target domain ($250$ epochs,
$lr=5e-3$); 3) the upper limit scenario for the federated model, where it
undergoes fine-tuning with full supervision on the clients ($R=250$). In the
unsupervised setting, (i.e., in the context of FFreeDA), we explore two
strategies: the standard FedAvg [27] combined with our self-training approach,
and the main competitor, LADD [37].

Table 7 and 7:  Motivation of our approach: viewpoints generalization (Sec.
7); generalization to different weather conditions (Sec. 7).

Pretrain on |  Source Only | Adapted
---|---|---
Separately | Jointly
Car+Drone | 22.70 | 28.95 | 28.96
Car only | 18.23 | 25.17 | 25.24
Drone only | 15.25 | 14.37 | 13.68

Pretrain | Adapt | Clear | Night | Rain | Fog | All
---|---|---|---|---|---|---
Clear | - | 23.43 | 6.43 | 24.93 | 25.50 | 18.28
All | - | 25.41 | 13.62 | 26.72 | 26.08 | 22.69
Clear | Clear | 26.87 | 1.02 | 8.02 | 9.86 | 14.36
Clear | All | 25.97 | 7.87 | 25.70 | 23.16 | 18.11
All | Clear | 30.96 | 10.85 | 22.22 | 26.38 | 25.03
All | All | 31.73 | 14.18 | 34.19 | 29.86 | 28.96

Notably, unlike our strategy, LADD conducts dataset stylization from the
target domain, assuming prior knowledge of the client data distribution during
pretraining. We pretrain LADD on our synthetic adverse weather dataset to
ensure comparability, maintaining the target’s average style as outlined in
[37].

The large domain shift between the source and target domains is clear from the
mIoU of $22.70\%$ achieved by the source only training compared to $58.08\%$
of the target training. Although the performance increases by almost $13\%$ by
fine-tuning the source only pretraining, assuming labeled data on the client
side is not very realistic. In the unsupervised scenario, the baseline
approach of using self-teaching with pseudo-labels together with FedAvg leads
to a score of $26.10\%$, with a relatively small improvement from the source
only proving that simple FL strategies struggle in this challenging setting.
LADD [37] allows to improve performances to $26.96\%$, an improvement of
around $1\%$ compared to FedAvg. Our approach, in comparison to FedAvg and
LADD, demonstrates a better improvement over the source only baseline reaching
a mIoU of $28.96\%$. In particular notice how the gain is also well-
distributed across car and drone agents, meaning that our strategy can
effectively align with the intrinsic distribution shift that different
viewpoints introduce. Notably, our approach is the only able to improve the
source only results on both cars and drones, while competitors struggle with
drone imagery since they are not able to generalize well to both and trade-off
the better performances on cars with a reduction of the accuracy on drones.
Qualitative results are reported in the Suppl. Mat.

Generalization capabilities of the proposed approach. We perform further tests
to evaluate the capability of our approach to adapt to different viewpoints
and weather conditions. Specifically, in Sec. 7 we can underline the
following: 1) Jointly pretraining on both car and drones improves the overall
mIoU of $4.47\%$ compared to car-only and $7.45\%$ to drone-only; 2) Adapting
to car and drones together leads to roughly the same performances of adapting
separately two models (one for cars and one for drones), i.e., the model
generalize well across the two agents even when only one viewpoint is
provided; 3) In the drone-only setting, the limited number of clients results
in poor generalization and overconfidence in predictions. In Sec. 7, we
underline that: 1) When pretraining with synthetic data from all weather
conditions the model improves on all, especially the night; moreover, adapting
from clear weather only results in poor generalization when adapting to real
data; 2) Although adapting with only clear weather data increases the mIoU on
clear weather itself, it worsens the performance on almost all the other
conditions.

Table 3: Ablation of the proposed method components.

$\mathcal{L}_{st}$ | AG |  | $\mathcal{L}_{cl}$ | BN | Car | Drone | All
---|---|---|---|---|---|---|---
|  |  |  |  | 19.96 | 33.11 | 22.70
✓ |  |  |  |  | 25.00 | 30.25 | 26.10
✓ | ✓ |  |  |  | 24.75 | 33.49 | 26.57
✓ | ✓ |  |  | ✓ | 24.91 | 34.55 | 26.92
✓ | ✓ |  | ✓ |  | 27.17 | 33.60 | 28.51
✓ | ✓ |  | ✓ | ✓ | 27.45 | 34.69 | 28.96

Figure 5: Training curves of the ablation.

Ablation studies. Tab. 3 shows an extensive ablation of the various modules of
the proposed approach together and Fig. 5 the corresponding training curves.
The aggregation strategy of Eq. (12) stabilizes the self-training and avoids
performance decrease on unbalanced clients (_i.e_. drones). The batch
normalization effect is visible in cars and drones ($+3.4\%$ overall). Observe
that the testing set of drones is more balanced in terms of weather categories
(¼ each), making it more noticeable. The hyperbolic prototypes allow for
improving the generalization of the feature spaces compared to the baseline,
achieving $28.51\%$. In Suppl. Mat. a comprehensive examination of the
advantages over the standard Euclidean space is provided. Finally, adding each
component leads to the final score of $28.96\%$. Overall, the contributions of
all the components jointly produce a gain of over $6.26\%$ of mIoU on the
source only. Further ablation on the hyperparameters is in the Suppl. Mat.

##  8 Conclusion

In this paper we introduced a new federated learning scenario, addressing the
challenges of heterogeneous autonomous agents in adverse weather conditions.
The contributions include a realistic federated setting where clients are
unsupervised and a versatile model seamlessly integrating aerial views. To
achieve these results, we introduced the FLYAWARE dataset for semantic
segmentation in adverse weather. The proposed method, HyperFLAW, employs
weather-aware batch normalization layers to mitigate domain shifts across
diverse weather conditions. Additionally, it leverages prototype-based
learning in the hyperbolic space, contributing a novel method for consistent
training in this challenging federated setting. By extending the scope to
different types of vehicles and addressing real-world challenges such as
adverse weather and privacy preservation, our contributions lay a foundation
for future advancements in autonomous driving. Further research will be
devoted to the exploitation of multi-modal data and to more advanced ad-hoc
adaptation strategies for adverse weather conditions.

### Acknowledgment

This work was supported in part by the European Union through the Italian
National Recovery and Resilience Plan (NRRP) of NextGenerationEU, a
partnership on “Telecommunications of the Future” (Program “RESTART”) under
Grant PE0000001.

## References

  * [1] Araslanov, N., Roth, S.: Self-supervised augmentation consistency for adapting semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15384–15394 (2021)
  * [2] Atigh, M.G., Schoep, J., Acar, E., Van Noord, N., Mettes, P.: Hyperbolic image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4453–4462 (2022)
  * [3] Brüggemann, D., Sakaridis, C., Truong, P., Van Gool, L.: Refign: Align and refine for adaptation of semantic segmentation to adverse conditions. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3174–3184 (2023)
  * [4] Caldarola, D., Caputo, B., Ciccone, M.: Window-based model averaging improves generalization in heterogeneous federated learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2263–2271 (2023)
  * [5] Chellapandi, V.P., Yuan, L., Brinton, C.G., Żak, S.H., Wang, Z.: Federated learning for connected and automated vehicles: A survey of existing approaches and challenges. IEEE Transactions on Intelligent Vehicles (2023)
  * [6] Choi, J., Kim, T., Kim, C.: Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6830–6840 (2019)
  * [7] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [8] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: An open urban driving simulator. In: Conference on robot learning. pp. 1–16. PMLR (2017)
  * [9] Du, D., Qi, Y., Yu, H., Yang, Y., Duan, K., Li, G., Zhang, W., Huang, Q., Tian, Q.: The unmanned aerial vehicle benchmark: Object detection and tracking. In: Proceedings of the European conference on computer vision (ECCV). pp. 370–386 (2018)
  * [10] Eftekhar, A., Sax, A., Malik, J., Zamir, A.: Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10786–10796 (2021)
  * [11] Fantauzzo, L., Fanì, E., Caldarola, D., Tavera, A., Cermelli, F., Ciccone, M., Caputo, B.: Feddrive: Generalizing federated learning to semantic segmentation in autonomous driving. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 11504–11511. IEEE (2022)
  * [12] Gao, Z., Wu, Y., Jia, Y., Harandi, M.: Curvature generation in curved spaces for few-shot learning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8691–8700 (2021)
  * [13] Gasperini, S., Morbitzer, N., Jung, H., Navab, N., Tombari, F.: Robust monocular depth estimation under challenging conditions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8177–8186 (2023)
  * [14] Guo, Y., Wang, X., Chen, Y., Yu, S.X.: Clipped hyperbolic classifiers are super-hyperbolic classifiers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11–20 (2022)
  * [15] Hoyer, L., Dai, D., Van Gool, L.: Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9924–9935 (2022)
  * [16] Hu, X., Fu, C.W., Zhu, L., Heng, P.A.: Depth-attentional features for single-image rain removal. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 8022–8031 (2019)
  * [17] Khrulkov, V., Mirvakhabova, L., Ustinova, E., Oseledets, I., Lempitsky, V.: Hyperbolic image embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6418–6428 (2020)
  * [18] Klinghoffer, T., Philion, J., Chen, W., Litany, O., Gojcic, Z., Joo, J., Raskar, R., Fidler, S., Alvarez, J.M.: Towards viewpoint robustness in bird’s eye view segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8515–8524 (2023)
  * [19] Li, J., Yu, Z., Du, Z., Zhu, L., Shen, H.T.: A comprehensive survey on source-free domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
  * [20] Li, Q., He, B., Song, D.: Model-contrastive federated learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10713–10722 (2021)
  * [21] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2, 429–450 (2020)
  * [22] Li, X., Jiang, M., Zhang, X., Kamp, M., Dou, Q.: Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 (2021)
  * [23] Liu, Y., Deng, J., Gao, X., Li, W., Duan, L.: Bapa-net: Boundary adaptation and prototype alignment for cross-domain semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8801–8811 (2021)
  * [24] Luo, Y., Zheng, L., Guan, T., Yu, J., Yang, Y.: Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2507–2516 (2019)
  * [25] Lyu, Y., Vosselman, G., Xia, G.S., Yilmaz, A., Yang, M.Y.: Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing 165, 108–119 (2020)
  * [26] Ma, R., Fang, P., Drummond, T., Harandi, M.: Adaptive poincaré point to set distance for few-shot classification. In: Proceedings of the AAAI conference on artificial intelligence. vol. 36, pp. 1926–1934 (2022)
  * [27] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics (AISTATS). pp. 1273–1282. PMLR (2017)
  * [28] Mei, K., Zhu, C., Zou, J., Zhang, S.: Instance adaptive self-training for unsupervised domain adaptation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16. pp. 415–430. Springer (2020)
  * [29] Mettes, P., Atigh, M.G., Keller-Ressel, M., Gu, J., Yeung, S.: Hyperbolic deep learning in computer vision: A survey. arXiv preprint arXiv:2305.06611 (2023)
  * [30] Michieli, U., Biasetton, M., Agresti, G., Zanuttigh, P.: Adversarial learning and self-teaching techniques for domain adaptation in semantic segmentation. IEEE Transactions on Intelligent Vehicles 5(3), 508–518 (2020)
  * [31] Ren, H., Yang, Y., Wang, H., Shen, B., Fan, Q., Zheng, Y., Liu, C.K., Guibas, L.J.: Adela: Automatic dense labeling with attention for viewpoint shift in semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8079–8089 (2022)
  * [32] Rizzoli, G., Barbato, F., Caligiuri, M., Zanuttigh, P.: Syndrone-multi-modal uav dataset for urban scenarios. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2210–2220 (2023)
  * [33] Saito, K., Watanabe, K., Ushiku, Y., Harada, T.: Maximum classifier discrepancy for unsupervised domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3723–3732 (2018)
  * [34] Sakaridis, C., Dai, D., Van Gool, L.: Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10765–10775 (2021)
  * [35] Sarlin, P.E., Trulls, E., Pollefeys, M., Hosang, J., Lynen, S.: Snap: Self-supervised neural maps for visual positioning and semantic understanding. Advances in Neural Information Processing Systems 36 (2024)
  * [36] Schwonberg, M., Niemeijer, J., Termöhlen, J.A., Schäfer, J.P., Schmidt, N.M., Gottschalk, H., Fingscheidt, T.: Survey on unsupervised domain adaptation for semantic segmentation for visual perception in automated driving (2023)
  * [37] Shenaj, D., Fanì, E., Toldo, M., Caldarola, D., Tavera, A., Michieli, U., Ciccone, M., Zanuttigh, P., Caputo, B.: Learning across domains and devices: Style-driven source-free domain adaptation in clustered federated learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 444–454 (2023)
  * [38] Shenaj, D., Rizzoli, G., Zanuttigh, P.: Federated learning in computer vision. IEEE Access (2023)
  * [39] Shenaj, D., Toldo, M., Rigon, A., Zanuttigh, P.: Asynchronous federated continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5054–5062 (2023)
  * [40] van Spengler, M., Berkhout, E., Mettes, P.: Poincare resnet. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5419–5428 (October 2023)
  * [41] van Spengler, M., Wirth, P., Mettes, P.: Hypll: The hyperbolic learning library. arXiv preprint arXiv:2306.06154 (2023)
  * [42] Testolina, P., Barbato, F., Michieli, U., Giordani, M., Zanuttigh, P., Zorzi, M.: Selma: Semantic large-scale multimodal acquisitions in variable weather, daytime and viewpoints. IEEE Transactions on Intelligent Transportation Systems (2023)
  * [43] Toldo, M., Maracani, A., Michieli, U., Zanuttigh, P.: Unsupervised domain adaptation in semantic segmentation: a review. Technologies 8(2), 35 (2020)
  * [44] Tsai, Y.H., Hung, W.C., Schulter, S., Sohn, K., Yang, M.H., Chandraker, M.: Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7472–7481 (2018)
  * [45] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  * [46] Xia, R., Zhao, C., Zheng, M., Wu, Z., Sun, Q., Tang, Y.: Cmda: Cross-modality domain adaptation for nighttime semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21572–21581 (2023)
  * [47] Yang, Y., Soatto, S.: Fda: Fourier domain adaptation for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4085–4095 (2020)
  * [48] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T.: Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2636–2645 (2020)
  * [49] Zhang, N., Zhang, L., Cheng, Z.: Towards simulating foggy and hazy images and evaluating their authenticity. In: International Conference on Neural Information Processing. pp. 405–415. Springer (2017)
  * [50] Zhang, P., Zhang, B., Zhang, T., Chen, D., Wang, Y., Wen, F.: Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12414–12424 (2021)
  * [51] Zhang, Q., Zhang, J., Liu, W., Tao, D.: Category anchor-guided unsupervised domain adaptation for semantic segmentation. Advances in neural information processing systems 32 (2019)
  * [52] Zheng, Z., Wu, Y., Han, X., Shi, J.: Forkgan: Seeing into the rainy night. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. pp. 155–170. Springer (2020)
  * [53] Zhu, P., Wen, L., Du, D., Bian, X., Fan, H., Hu, Q., Ling, H.: Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(11), 7380–7399 (2021)
  * [54] Zou, Y., Yu, Z., Kumar, B., Wang, J.: Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of the European conference on computer vision (ECCV). pp. 289–305 (2018)

When Cars meet Drones: Hyperbolic Federated Learning for Source-Free Domain
Adaptation in Adverse Weather
Supplementary Material

Giulia Rizzoli* Matteo Caligiuri* Donald Shenaj Francesco Barbato Pietro
Zanuttigh

**footnotetext: These authors contributed equally to this work.

This document contains the supplementary material for the paper When Cars meet
Drones: Hyperbolic Federated Learning for Source-Free Domain Adaptation in
Adverse Weather. We start by introducing additional details on the employed
datasets. We then proceed with ablations supporting the usage of the weather
batch-normalization and of hyperbolic prototypes. Finally, we show further
experimental data including per-class accuracy scores and qualitative results.

##  S1 Data Selection and Distribution

In the main paper, we introduced the FLYAWARE aerial dataset since it is a
novel contribution. Conversely, we offered fewer details for car driving
datasets, as we relied on existing datasets. Here we will first describe the
technical choices adopted to enable distributed learning on car agents in
adverse weather. Then, some additional details on weather distribution among
clients are provided.

###  S1.1 Driving Datasets

For the synthetic source dataset for cars, we used the SELMA dataset [42]. It
offers a comprehensive set of 27 weather and daytime conditions, resulting in
a vast dataset of over 20M samples. To better align with the weather scenarios
considered in real data [34], we opted not to use the standard SELMA split,
but downloaded over 24k samples in the ClearNoon, ClearNight, HardRainNoon,
MidFoggyNoon splits. In total, for this work, we employed almost 100k SELMA
samples from the Desk Cam point of view that match the one used in the real-
world dataset. As the real counterpart, we used the ACDC dataset, from which
we selected the 3 domains — night, rain, and fog — that match our pretraining.
As ACDC lacks images in clear weather, we supplemented it with daytime images
coming from the Cityscapes dataset [7], which is the most similar to ACDC.
Note that, the ACDC dataset has been built to create an adverse condition
version of Cityscapes, and shares the same class-set. Cityscapes provides more
samples than those in the thematic splits by ACDC. Therefore, we subsampled
its training set to match the sizes. Since the goal was to select clear
weather conditions, we devised an automatic way of assigning a sunlit level
(Eq. S1) to each image and selected the needed images by sorting them
according to this metric:

$\text{sunlit}(\textbf{X})=\sum\limits_{c\in\\{r,g,b\\}}\frac{1}{HW}\sum\limits_{i=0}^{W}\sum\limits_{i=0}^{W}\textbf{X}[i,j,c]$
(S1)

As shown in Tab. 1 in the main paper, we selected a set of images to match the
total count of all other conditions combined, aiming for a ratio of three
clear sky images to one adverse condition. This decision was driven by the
desire to retain “clear sky” as the most probable scenario in real-world
contexts. In Fig. S1 we report some examples of samples with their rating (a
bright one, a dark one and a mid-range one).

Figure S1: Cityscapes samples and corresponding sunlit score. Decreasing from
the left, high/mid/low-score ($\sim 300/200/100$).

###  S1.2 Weather Heterogeneity

Fig. S2 illustrates the weather distribution among clients in the
configuration ACDC+ FLYAWARE-R. We deliberately supplied clear day samples to
each client to emulate a common scenario where clear daylight conditions
prevail, with fewer instances of challenging weather conditions. This
introduces an inherent challenge as adverse weather data samples are less
prevalent. Additionally, the imbalance among clients further complicates the
scenario.

Figure S2: Distribution of weather conditions across clients (ACDC and
FLYAWARE-R).

Fig. S3 shows the weather distribution for clients in the ACDC+FLYAWARE-R-XL
setup. While this configuration exhibits better data balance across clients,
it introduces new complexities: some drone clients exclusively operate in
adverse conditions, and the training and testing domains originate from
different datasets (VisDrone [53] and UAVid [25], respectively). Notably,
VisDrone images offer real night data, whereas we utilized domain-translated
images for fog and rain using the same strategies described in the main paper.

Figure S3: Distribution of weather conditions across clients (ACDC and
FLYAWARE-R-XL).

##  S2 Impact of Weather-Batch Normalization

As shown in Tab. 3 of the main paper, the inclusion of ad-hoc batch norms
(BNs) enhances performance in comparison to utilizing a non-personalized
network. First of all, in Fig. S4, we show in detail the architecture of the
weather classifier. Although the classifier model is simple, it achieves an
accuracy of $98.96\%$ on the source synthetic datasets used to train it and of
$72.37\%$ on the real world target ones.

Figure S4: Weather classifier architecture model. Table S1: Effect of the different modules on the weather. Modules | Clear | Night | Rain | Fog
---|---|---|---|---
Pretrain | 25.41 | 13.62 | 26.72 | 26.08
ST + AG w/o BN | 29.27 | 13.47 | 29.47 | 27.86
ST + AG w BN | 26.81 | 13.50 | 30.19 | 28.86
ALL w/o BN | 31.58 | 14.04 | 30.53 | 28.93
ALL w BN | 31.73 | 14.18 | 34.19 | 29.86

We further examine in Tab. S1 the advantage of adapting the system to
accommodate different weather conditions as a remarkable feature for the
decision-making of autonomous driving agents. The basic self-training
strategy, coupled with the proposed server-side aggregation scheme, has shown
stability and an overall improvement over pretraining. However, due to the
prevalence of clear-day images in most clients’ samples, the network tends to
learn more about this weather condition. Introducing the personalized weather
BNs helps to mitigate this, improving performances in adverse weather
conditions, with an mIoU increase of $3\%$ on Rain and $1.78\%$ on Fog.
Similar results are evident when comparing the full method with and without
the weather BNs, showcasing mIoU increases of $3.66\%$ on Rain and $0.93\%$ on
Fog. As a side note, the most difficult scenario remains the night where the
improvement over the pretraining is just $0.56\%$ of mIoU. Overall, we remark
on the significance of incorporating personalized weather BNs for mitigating
bias towards clear day images and enhancing performance across various weather
conditions.

As a final note, considering that modern vehicles often have access to real-
time weather information through several sensors (e.g., automatic windshield
wipers for rain detection, clock or optical light detection for night and
fog), and the weather information might be directly obtained.

##  S3 Analysis of class prototypes effectiveness

Hyperbolic vs. Euclidean Prototypes

Figure S5: Euclidean vs hyperbolic prototypes.

In Fig. S5, we show the training curves for both Euclidean prototypes and
hyperbolic ones employing different values of the weighting parameter for the
prototype loss $\lambda_{cl}$. Notably, hyperbolic space prototypes
consistently allow for an increase in performances across different
$\lambda_{cl}$ values. The key strength of the hyperbolic space lies in its
ability to effectively represent distances between feature vectors, a crucial
factor in tasks like clustering where having a distance function capturing
meaningful relationships among data points is essential.

Setting of the $\lambda_{cl}$ parameter Fig. S6 shows the mIoU for different
values of the $\lambda_{cl}$ parameter. Best performances are achieved in the
range 140-180, with a quite stable maxima.

Figure S6: Tuning on $\lambda_{cl}$ weight.

##  S4 Per-class Results

Tab. S2 contains a more detailed version of the results in Tab. 3 of the main
paper that shows also the class-by-class accuracy. As common on the employed
datasets, the mIoU is higher in the common classes (e.g., road or building)
and lower in the rare and more challenging ones. However, it can be seen that
by adding the various components of the model, results tend to increase
consistently in most classes, even if a few challenging ones remain hard to
detect. Some classes like traffic sign, car or bicycle show impressive
improvements. On the other side, there are a few cases in which our approach
does not improve the results on some under-represented classes such as terrain
and rider.

Table S2: Per class IoU for the proposed approach and its ablated versions.

ST |  AG |  |  CL |  BN |  road |  sidewalk |  building |  wall |  fence |  pole |  traffic light |  traffic sign |  vegetation |  terrain |  sky |  person |  rider |  car |  truck |  bus |  train |  motorcycle |  bicycle |  All
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  | 35.4 | 5.7 | 55.8 | 4.6 | 4.2 | 14.2 | 13.3 | 10.9 | 61.9 | 6.3 | 43.7 | 26.0 | 17.2 | 11.1 | 3.3 | 3.8 | 1.4 | 8.9 | 10.7 | 22.7
✓ |  |  |  |  | 43.7 | 22.0 | 57.2 | 6.6 | 4.7 | 15.8 | 9.8 | 13.9 | 50.2 | 5.7 | 68.4 | 24.3 | 11.4 | 34.6 | 8.9 | 4.7 | 3.5 | 16.0 | 14.9 | 26.1
✓ | ✓ |  |  |  | 43.9 | 23.2 | 59.2 | 7.1 | 5.9 | 16.2 | 11.1 | 14.8 | 55.3 | 6.1 | 58.5 | 26.0 | 11.9 | 34.2 | 9.2 | 4.3 | 3.0 | 15.8 | 13.5 | 26.6
✓ | ✓ |  |  | ✓ | 43.8 | 23.6 | 59.7 | 6.9 | 5.9 | 15.9 | 10.7 | 14.8 | 56.5 | 5.1 | 60.6 | 25.4 | 11.9 | 37.7 | 9.0 | 4.5 | 2.8 | 15.1 | 13.0 | 26.9
✓ | ✓ |  | ✓ |  | 46.8 | 22.4 | 59.9 | 7.4 | 6.7 | 17.6 | 17.0 | 22.4 | 56.6 | 7.1 | 62.6 | 25.4 | 10.5 | 35.5 | 9.5 | 6.9 | 3.7 | 16.6 | 25.7 | 28.5
✓ | ✓ |  | ✓ | ✓ | 47.2 | 22.7 | 60.6 | 7.0 | 6.7 | 17.3 | 16.8 | 22.7 | 57.7 | 5.9 | 64.9 | 25.4 | 10.6 | 36.9 | 9.3 | 6.8 | 3.5 | 16.8 | 25.5 | 29.0

##  S5 Qualitative Results

Fig. S7 shows some qualitative results for car and drone samples in both clear
and adverse weather. FedAvg [27] and LADD [37] appear to be overconfident on
the road and car classes, respectively. While our model adeptly captures the
structure of the street better than other models in both adverse and non-
adverse conditions, the adverse conditions images remain a challenging aspect.
For drones, the discrepancies between the predicted classes and the ground
truth arise also due to the fact that the network is trained on a superset of
classes beyond those present in the drone dataset. Notice that sometimes the
network assigns “fine” classes that share a semantic affiliation with the
respective “coarse” classes, e.g., predict terrain instead of vegetation, but
on the drone dataset labeling terrain is not present and terrain samples have
ground truth set to vegetation. The same thing happens for the sidewalk and
road or rider and person. For this reason, we remapped the 19 classes of the
full set into the 5 drone ones using the mapping proposed in [32]. The figure
shows both the original prediction maps from the network and the ones obtained
after remapping the 19 classes into the 5 of the drone datasets. The
comparison with the ground truth shows that some predictions not matching are
just due to the different types of labeling in drone and car datasets.

| RGB | GT | FedAvg[27]+ST | LADD[37] | HyperFLAW
---|---|---|---|---|---
CAR CLEAR |  |  |  |  |
DRONE CLEAR |  |  |  |  |
DRONE CLEAR (remapped) |  |  |
CAR ADVERSE |  |  |  |  |
DRONE ADVERSE |  |  |  |  |
DRONE ADVERSE (remapped) |  |  |

Road | Sidewalk | Building | Wall | Fence | Pole | T. Light | T. Sign | Vegetation | Terrain
---|---|---|---|---|---|---|---|---|---
Sky | Person | Rider | Car | Truck | Bus | Train | Motorbike | Bicycle | Unlabeled

Figure S7: Qualitative results.

  *[FL]: Federated Learning
  *[FFreeDA]: Federated source Free Domain Adaptation
  *[GAN]: Generative Adversarial Network