# One-Shot Sequential Federated Learning for Non-IID Data by Enhancing Local
Model Diversity

Naibo Wang  National University of SingaporeSingaporeSingapore
<EMAIL_ADDRESS>,  Yuchen Deng  National University of
SingaporeSingaporeSingapore<EMAIL_ADDRESS>,  Wenjie Feng  National
University of SingaporeSingaporeSingapore<EMAIL_ADDRESS>,  Shichen
Fan  Xidian UniversityXi’anChina<EMAIL_ADDRESS>,  Jianwei Yin
Zhejiang UniversityHangzhouChina<EMAIL_ADDRESS>and  See-Kiong Ng
National University of SingaporeSingaporeSingapore<EMAIL_ADDRESS>
###### Abstract.

Traditional federated learning mainly focuses on parallel settings (PFL),
which can suffer significant communication and computation costs. In contrast,
one-shot and sequential federated learning (SFL) have emerged as innovative
paradigms to alleviate these costs. However, the issue of non-IID (Independent
and Identically Distributed) data persists as a significant challenge in one-
shot and SFL settings, exacerbated by the restricted communication between
clients. In this paper, we improve the one-shot sequential federated learning
for non-IID data by proposing a local model diversity-enhancing strategy.
Specifically, to leverage the potential of local model diversity for improving
model performance, we introduce a local model pool for each client that
comprises diverse models generated during local training, and propose two
distance measurements to further enhance the model diversity and mitigate the
effect of non-IID data. Consequently, our proposed framework can improve the
global model performance while maintaining low communication costs. Extensive
experiments demonstrate that our method exhibits superior performance to
existing one-shot PFL methods and achieves better accuracy compared with
state-of-the-art one-shot SFL methods on both label-skew and domain-shift
tasks (e.g., 6%+ accuracy improvement on the CIFAR-10 dataset).

Sequential Federated Learning, One-Shot Federated Learning

††copyright: none

##  1\. Introduction

Federated learning (FL) [McMahan et al., 2017] is a promising paradigm which
enables collaborative machine learning [Wang et al., 2022, Balkus et al.,
2022] amongst multiple clients to build a consensus global model without the
need to access others’ datasets. This paradigm offers salient benefits such as
preservation of privacy [Gao et al., 2021], security of data [Ma et al.,
2020], and the facility for different clients to derive a model exhibiting a
higher degree of inference capability compared to individual client-based
training [Yang et al., 2019].

As shown in Fig. 1, two federated learning paradigms are parallel FL (PFL)
[Bonawitz et al., 2019, Qu et al., 2022] and sequential FL (SFL) [Li and Lyu,
2024]. PFL synchronizes model training across clients in parallel, such as
FedAvg [McMahan et al., 2017], while SFL adopts a client-by-client training
sequence that is widely applied in many scenarios such as healthcare [Chen et
al., 2023, Chang et al., 2018]. Compared with PFL, SFL exhibits significant
advantages in training efficiency, as evidenced by a reduction in training
rounds [Zaccone et al., 2022]. SFL also shows robust performance with limited
datasets [Kamp et al., 2023] and offers enhanced data privacy protection due
to its decentralized architecture [Hegedűs et al., 2021].

One main concern of SFL is communication costs. To mitigate communication
costs, one-shot federated learning [Guha et al., 2018] has been proposed where
only one communication round is needed for the clients to interact with the
server or other clients. However, current one-shot FL works [Zhang et al.,
2022, Heinbaugh et al., 2023] mainly focus on PFL (Fig. 1 (a)) which typically
requires a central server to facilitate model training. This can lead to
numerous limitations including risk of privacy leakage [Kairouz et al., 2021]
and server node bottlenecks [Hard et al., 2021]. To address these issues, the
decentralized structure [Su et al., 2023] has been incorporated into one-shot
PFL which typically utilizes a mesh-topology network for clients to
disseminate their models to others. Nonetheless, this setting still leads to
significant communication overhead than SFL. In contrast, one-shot SFL,
wherein each client only needs to communicate with its adjacent client once,
can greatly reduce communication costs. However, existing studies on one-shot
SFL [Sheller et al., 2020, Chang et al., 2018] struggle with handling the non-
IID data, a common challenge in FL that significantly impairs the performance
of federated models [Shoham et al., 2019]. How to tackle the non-IID data in
one-shot SFL is still an open problem.

|
---|---
(a) Parallel FL framework |  (b) Sequential FL framework
Figure 1. Two federated learning settings.

(a) Training loss surface on local dataset $D_{i}$.

(b) Test error surface on the whole test set.

Figure 2. An illustration of our training solution on client $i$. Based on
previously trained models in the model pool, every new model $m_{k}$ starts
training from $\theta_{k}^{s}=f(\\{\theta_{i}\\}_{i=0}^{k-1})$ to improve
training diversity ($f$ is the average function in our paper). During
training, optimization of $\theta_{k}$ is constrained within a specific region
(the non-shadow areas). $\theta_{k}$ is required to maintain a certain
distance ($d_{1}$) from existing models $\\{\theta_{i}\\}_{i=0}^{k-1}$ to
enhance model diversity, and should not diverge significantly ($d_{2}$) from
the initial model $\theta_{0}$ to prevent deviation from the globally optimal
solution caused by non-IID data. After training, all trained models
$\\{\theta_{i}\\}_{i=1}^{k}$ display similar training losses on the local
dataset $D_{i}$ (a) but have different test errors on the whole test set (b).
Meanwhile, the averaged model $\theta_{avg}$ of all models in the model pool
achieves a lower test error than any single model (b).

The key question in one-shot SFL is how to effectively transfer knowledge
given the limited communication, especially with the non-IID data. Our insight
is to utilize the diversity of models. Existing studies have shown that
combining multiple networks can notably enhance the model’s performance due to
the inherent diversity amongst model weights [Wortsman et al., 2022, Cha et
al., 2021, Izmailov et al., 2018]. This observation has also been solidly
backed by theoretical support from Rame et al. [Rame et al., 2022]. However,
directly applying such a diversity strategy in SFL is not feasible, since SFL
restricts each client to receive only one model from its adjacent client. This
limitation could lead to insufficient diversity during model training, which
can potentially undermine the performance of the final trained model. To
overcome this challenge within the one-shot SFL framework, we propose a
diversity-enhanced mechanism for model training. This mechanism is designed to
augment the diversity of locally trained models. By generating a broader
spectrum of models, we facilitate enriched knowledge transfer between adjacent
clients, which also serves to mitigate the impact of data distribution
disparities across clients.

In this paper, we present a novel one-shot sequential Federated learning
framework by Enhancing Local Model diversitY (FedELMY). Specifically, we
improve the local model training diversity by constructing a model pool
consisting of various models for each client. We introduce two distance
regularization terms during the local training process to enhance model
training diversity while mitigating the impact of non-IID data. Fig. 2
presents a case study illustrating our core concept. Compared with
conventional federated learning methods, our approach effectively minimizes
communication costs, safeguards data privacy, and mitigates the detrimental
effects of non-IID data, thereby enhancing overall model performance.
Experiments show that our method can outperform existing one-shot SFL methods
on both label-skew and domain-shift datasets.

The main contributions of this paper are as follows:

  * •

We tackle the novel and practical problem of one-shot sequential federated
learning. To the best of our knowledge, this is the first work to
systematically investigate the one-shot communication setting in sequential
federated learning.

  * •

We introduce a novel framework FedELMY to reduce the communication cost and
improve the global model performance by enhancing model diversity during local
training.

  * •

We conduct extensive experiments on four datasets, considering both feature
and label distribution shifts. Our method achieves superior performance
compared with existing one-shot PFL and SFL methods.

##  2\. Related Work

###  2.1. Parallel Federated Learning

One well-known PFL method is FedAvg [McMahan et al., 2017], whose performance
is hindered due to the dispersed nature of the data (non-IID). Methods such as
FedProx [Li et al., 2020], FedDyn [Acar et al., 2021], Astraea [Duan et al.,
2020], pFedMe [T Dinh et al., 2020], and FedCurv [Shoham et al., 2019] use a
measure of global parameter stiffness to tackle data heterogeneity in
federated learning. Additionally, approaches like FedDC [Gao et al., 2022] and
SCAFFOLD [Karimireddy et al., 2020] use a global gradient adjustment mechanism
to manage local variations of the data. Other methods like FedGMM [Wu et al.,
2023], FCCL [Huang et al., 2022], FedBN [Li et al., 2021b] and ADCOL [Li et
al., 2023] utilize the personalized model instead of a shared global model to
improve the performance of federated learning. These methods all necessitate a
central server for model training.

Another PFL scenario is the decentralized federated learning [Yuan et al.,
2023, Beltrán et al., 2023, Roy et al., 2019] when edge devices conduct
training without a central server. Sun et al. [Sun et al., 2022] proposed the
decentralized FedAvg with momentum (DFedAvgM) algorithm to improve the
performance of trained models in the decentralized federated learning setting.
Shi et al. [Shi et al., 2023] applied the SAM [Foret et al., 2021] optimizer
to improve the model consistency of decentralized federated learning. This
setting can better protect privacy than centralized PFL but incurs more
communication costs.

###  2.2. Sequential Federated Learning

Recently, Sequential Federated Learning (SFL) started to gain attention in the
FL community. Micah J et al. [Sheller et al., 2020] proposed a basic SFL
framework to facilitate multi-institutional collaborations without sharing
patient data; Li et al. [Li and Lyu, 2024] proved that the convergence
guarantees of SFL on heterogeneous data are better than PFL for both full and
partial client participation, and validated that SFL outperforms PFL on
extremely heterogeneous data in cross-device settings; Chen et al. [Chen et
al., 2023] proposed MetaFed, an SFL scheme with cyclic knowledge distillation
for personalized healthcare; Cho et al. [Cho et al., 2023] also provided
convergence analysis of sequential federated averaging and proves that it can
achieve a faster asymptotic convergence rate than vanilla FedAvg with uniform
client participation under suitable conditions. However, existing works still
lack the ability to deal with non-IID data and exhibit poor performance in
real-world applications.

###  2.3. One-shot Federated Learning

One-shot federated learning [Duan, 2023, Guha et al., 2018, Su et al., 2023,
Feng et al., 2023], which aims to train a global model with only one round of
communication between the clients and the server, has been proposed to reduce
the communication cost and simultaneously enhance data privacy protection in
federated learning. FedDISC [Yang et al., 2023] made progress in one-shot
semi-supervised federated learning, employing a pre-trained diffusion model.
Investigations by Diao et al. [Diao et al., 2022] and Joshi et al.
[Jhunjhunwala et al., 2023] have independently explored one-shot federated
learning, viewing the problem through the respective perspectives of the open-
set problem and Fisher information. Moreover, the proliferation of pre-trained
models has sparked interest in collaborative, model-centric machine learning
[Bommasani et al., 2021, Wang et al., 2023]. Instances such as FedKT [Li et
al., 2021a] and DENSE [Zhang et al., 2022] are two one-shot federated learning
schemes that choose to send models instead of gradients to the central server
and generate a final global model through knowledge distillation. Despite
their advancements, these one-shot methods are not designed for sequential
federated learning, which still results in a higher risk of privacy leakage.
Thus, devising a one-shot sequential federated learning framework is still
under-explored.

##  3\. Method

In this section, we introduce our problem setting and present our proposed
algorithm FedELMY. The overview of our method is demonstrated in Fig. 3.

###  3.1. Problem Formulation

Assume there are $N$ different clients (parties), each client has its own
private dataset $D_{i}=\\{(x_{k},y_{k})\\}_{k=1}^{n_{i}}$ with size $n_{i}$.
The principal goal of sequential federated learning is to develop a global
model $m$ across the dataset $\mathcal{D}=\\{D_{i}\\}_{i=1}^{N}$, by
minimizing the error on training data. The optimization objective is
formulated as:

(1) $\mathop{\min}\limits_{m}\sum\nolimits_{i=1}^{N}\mathbb{E}_{(x,y)\sim
D_{i}}[L(m;x,y)],$

where $L(m;x,y)$ is the loss function evaluated on the private dataset $D_{i}$
from client $i$ ($c_{i}$) with model $m$. The training procedure of sequential
federated learning is described as follows:

1\. At the start of each training round $r$, the client’s training sequence
$\\{c_{\pi_{1}},c_{\pi_{2}},\cdots,c_{\pi_{N}}\\}$ is determined by randomly
selecting indices $\\{\pi_{1},\pi_{2},\cdots,\pi_{N}\\}$ without replacement
from set $\\{1,2,\cdots,N\\}$.

2\. At the very beginning of training when round $r=1$, randomly initialize
global model $m^{(0)}$.

3\. For the $i$-th client ($c_{\pi_{i}}$) in round $r$, initialize its model
$m^{(r)}_{\pi_{i},0}$ with the latest model:

(2) $m^{(r)}_{\pi_{i},0}=\begin{cases}m^{(r-1)},&\text{if }i=1\\\
m^{(r)}_{\pi_{i-1},E_{local}},&\text{if }i>1\end{cases}$

where $m^{(r-1)}$ is the global model received from round $r-1$, and
$m^{(r)}_{\pi_{i-1},E_{local}}$ is the model received from the $(i-1)$-th
client ($c_{\pi_{i-1}}$) in round $r$ after it trained its model for
$E_{local}$ epochs.

4\. Update model $m^{(r)}_{\pi_{i},0}$ for $E_{local}$ epochs based on
$D_{\pi_{i}}$ and send $m^{(r)}_{\pi_{i},E_{local}}$ to the $(i+1)$-th client
($c_{\pi_{i+1}}$), if SGD [Bottou, 2010] is chosen as the optimizer, then the
update can be described by:

(3) $m^{(r)}_{\pi_{i},k+1}=m^{(r)}_{\pi_{i},k}-\eta\cdot g^{(r)}_{\pi_{i},k}$

where $m^{(r)}_{\pi_{i},k+1}$ denotes the local model of client $\pi_{i}$
after $k$ local training steps in round $r$, $\eta$ is the learning rate, and
$g^{(r)}_{\pi_{i},k}$ represents the gradient of the loss function based on
$D_{\pi_{i}}$ at step $k$.

5\. At the end of round $r$, i.e., after all clients finished their training,
we can get the global model $m^{(r)}$ of round $r$ as:

(4) $m^{(r)}=m^{(r)}_{\pi_{N},E_{local}}$

For one-shot sequential federated learning, only 1 training round is required,
i.e., $r\equiv 1$, thus $m^{(1)}$ will be the final global model.

_Analysis._ Although the communication cost is low, the information exchanged
in SFL is limited, especially under the one-shot setting. Therefore, it is a
big challenge to aggregate diverse and useful information to get a better
global model. In other words, the diversity of distributed data under the one-
shot SFL paradigm has not been well utilized. Meanwhile, the non-IID data in
SFL increases the risk of models getting stuck in a local minimum, which
further limits the enhancement of the generalization ability of the global
model [Li et al., 2020]. Hence, it is essential to explore methods to increase
the diversity during model training and mitigate the impact of non-IID data to
enhance the model’s performance in one-shot SFL.

Figure 3. Overview of our method. Every client $i$ receives a model
$m_{avg}^{i-1}$ from its previous client $i-1$ and sends model $m_{avg}^{i}$
to its next client $i+1$ after training (I). For each client $i$, we train $S$
models and put them into its model pool $\mathcal{M}^{i}$ (II). Every new
model $m^{i}_{j}$ is initialized to the average of the existing models in
$\mathcal{M}^{i}$ and trained under the control of $d_{1}$ and $d_{2}$ (III).

###  3.2. Local Model Diversity Enhancement

The authors of SWAD [Cha et al., 2021] demonstrate that solely minimizing
empirical loss in a single model is typically insufficient for achieving good
generalization. Furthermore, they argue that the performance markedly improves
through averaging diverse models trained by various hyperparameters, even when
the training losses of these models are similar. Therefore, we propose to
enhance diversity by a Model Pool, which is essentially a collection of models
maintained by a client during its local training. Here, every client $i$
possesses a model pool $\mathcal{M}^{i}$ initially consisting of a solitary
model, denoted as $m_{0}^{i}$. For the first client when $i=1$, the model
$m_{0}^{1}$ undergoes the process of random initialization, followed by a
warm-up process over $E_{\mathrm{w}}$ epochs on its local dataset $D_{1}$. As
for the subsequent clients indexed from $i=2:N$, each model, $m_{0}^{i}$, is
assigned to be the average model $m_{avg}^{i-1}$ which is efficiently derived
from the model pool $\mathcal{M}^{i-1}$ of the previous client $i-1$:

(5)
$m_{avg}^{i-1}\leftarrow\frac{\sum_{t=0}^{|\mathcal{M}^{i-1}|-1}m_{t}^{i-1}}{|\mathcal{M}^{i-1}|},$

The parameters to be sent to the next client have been improved to an average
of a local diverse model pool instead of a single model.

Then, for each individual client $i$, we train an additional series of $S$
models, denoted as $\\{m_{j}^{i}\\}_{j=1}^{S}$, on the local dataset $D_{i}$
to probe a broader range of diverse models. Each new model, $m_{j}^{i}$, stems
from the existing models in the established model pool, $\mathcal{M}^{i}$.
More specifically, initialization of every new model $m_{j}^{i}$ is achieved
by averaging all the models currently in circulation within the model pool
$\mathcal{M}^{i}$, with the first model $m_{0}^{i}$ included; that is,

(6)
$m_{j}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum\nolimits_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i},$

In this way, we craft a unique launching point for the training of each new
model, ensuring a departure from all earlier models within our pool
$\mathcal{M}^{i}$. Consequently, a diverse array of starting points is
harnessed, granting us access to a more comprehensive set of potential
solutions that evade the restrictions of a singular training trajectory and
its attendant limitations.

After initialization, the model $m_{j}^{i}$ is then trained for $E_{local}$
epochs, employing the same hyperparameters as with preceding models. Upon
finishing training, $m_{j}^{i}$ will be added to the model pool
$\mathcal{M}^{i}$. After training all $S$ models for client $i$, the client
will calculate the final averaged model $m_{avg}^{i}$, which is the average of
all models in $\mathcal{M}^{i}$, and send $m_{avg}^{i}$ to the subsequent
client, $i+1$. This process will be repeated until client $i=N$ concludes its
local training. Upon completion, the final client $i=N$ will output the final
model $m_{final}$, which is the average of all models in its model pool
$\mathcal{M}^{N}$, i.e.,
$m_{final}\leftarrow\frac{1}{|\mathcal{M}^{N}|}\sum_{t=0}^{|\mathcal{M}^{N}|-1}m_{t}^{N}$.

To further enhance training diversity and prevent the convergence of the newly
generated model $m_{j}^{i}$ towards the states of existing models in
$\mathcal{M}^{i}$—thereby diminishing diversity—we aim to maintain the current
exploration direction of the model as distinct as possible from those of
existing models, throughout the training process. Consequently, we introduce a
new distance control term, $d_{1}$, in the training process, which is defined
as:

(7)
$\begin{split}d_{1}&=\frac{1}{|\mathcal{M}^{i}|}\sum\nolimits_{t=0}^{|\mathcal{M}^{i}|-1}dist(m_{j}^{i},m_{t}^{i})\\\
&=\frac{1}{|\mathcal{M}^{i}|}\sum\nolimits_{t=0}^{|\mathcal{M}^{i}|-1}||m_{j}^{i}-m_{t}^{i}||_{2},\end{split}$

where $m_{j}^{i}$ represents the current model being trained for client $i$,
and $m_{t}^{i}$ signifies a model in the model pool $\mathcal{M}^{i}$. This
distance is calculated by averaging the $L_{2}-norm$ between the current model
and all existing models in the model pool. The loss function incorporates and
deducts this distance during training, allowing the model to maximize its
distance from the existing models, therefore promoting training diversity.

###  3.3. Mitigation of Non-IID Data Impact

In non-IID scenarios, variations in local objectives may risk deviating from
the globally optimal solution across multiple local iterations, thereby
impeding convergence [Li et al., 2020]. To prevent significant deviation of
the model from the global solution, we further refine the model training
process by introducing an additional regularization term $d_{2}$, defined as
follows:

(8) $d_{2}=dist(m_{j}^{i},m_{0}^{i})=||m_{j}^{i}-m_{0}^{i}||_{2},$

where $m_{0}^{i}$ represents the first model in the model pool
$\mathcal{M}^{i}$ for client $i$. The loss function also incorporates this
term to ensure that the model maintains reasonable proximity to the initial
model $m_{0}^{i}$ (global solution of previous clients) in the pool during
local updates. It helps mitigate the effect of non-IID data distribution while
accommodating system heterogeneity.

The total loss function $\mathcal{L}$ for model $m_{j}^{i}$ is formed as
follows:

(9) $\mathcal{L}(m_{j}^{i})=\ell(m_{j}^{i};D_{i})-\alpha\cdot d_{1}+\beta\cdot
d_{2},$

where $\ell$ denotes the original loss function, $D_{i}$ is the local dataset
for client $i$, and $\alpha$ and $\beta$ are two hyperparameters that govern
the effect of both distances on model training. Our aim is to achieve a
balance that enhances model training diversity while mitigating the impact of
non-IID data.

Input: Local datasets $\mathcal{D}=\\{D_{i}\\}_{i=1}^{N}$, warm-up epoch
$E_{\mathrm{w}}$, learning rate $\eta$, number of local iterations
$E_{local}$, model number to be trained per client $S$, scale hyperparameters
$\alpha$, $\beta$

Output: The final model $m_{final}$

1

2Initialization: For client 1, warm up a randomly initialized model
$m_{avg}^{0}$ for $E_{\mathrm{w}}$ epochs

3for  _client $i=1:N$ _ do

4

5 Only for client $i=2:N$, receive $m_{avg}^{i-1}$ from client $i-1$

6 // Initialize model pool $\mathcal{M}^{i}$ for client $i$

7 $\mathcal{M}^{i}=\\{m_{0}^{i}\\}$ with $m_{0}^{i}\leftarrow$ $m_{avg}^{i-1}$

8 for  _$j=1:S$_ do

9

10 // Initialize $m_{j}^{i}$

11
$m_{j}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i}$

12 // Local training for $m_{j}^{i}$

13 for  _$k=1:E_{local}$_ do

14

15 $\mathcal{L}(m_{j}^{i})=\ell(m_{j}^{i};D_{i})-\alpha\cdot d_{1}+\beta\cdot
d_{2}$

16 $m_{j}^{i}\leftarrow m_{j}^{i}-\eta\nabla_{m}\mathcal{L}(m_{j}^{i})$

17 end for

18

19 $\mathcal{M}^{i}\leftarrow\mathcal{M}^{i}\cup\\{m_{j}^{i}\\}$

20 end for

21
$m_{avg}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i}$

22 Only for client $i=1:N-1$, send $m_{avg}^{i}$ to client $i+1$

23 end for

24

25// For the final client $i=N$, output model $m_{final}$

$m_{final}\leftarrow m_{avg}^{N}$

Algorithm 1 Our proposed FedELMY

###  3.4. Implementation Details

The details of our method are depicted in Algorithm 1. The training procedure
begins at client 1, which acts as the starting point for training with a warm-
up phase (line 1). Then, each client $i$ will create and initialize a model
pool $\mathcal{M}^{i}$ (lines 3-5). The process of involving a new model
$m_{j}^{i}$ in $\mathcal{M}^{i}$ is presented in line 8. Lines 10-13 elaborate
on the local training process per model based on our customized loss function
$\mathcal{L}$, as delineated in Sec. 3.3. Once all models for client $i$ have
completed training, the average model $m_{avg}^{i}$ derived from
$\mathcal{M}^{i}$ is sent to the next client $i+1$ (lines 16-17). The final
global model $m_{final}$ is obtained as the average of the models in model
pool $\mathcal{M}^{N}$ of the final client $N$ (line 20).

Communication cost. In our method, each client will send a model to its
adjacent client only once, thus only $N-1$ model exchanges are required.
Therefore, the overall communication cost for all $N$ parties is $O(NM)$,
where $M$ represents the size of the model.

Computation cost. In our approach, every client will train $S$ models, thus
the overall computation cost for all $N$ parties is $O(NSE_{local})$.

##  4\. Experiments

###  4.1. Experimental Setup

Datasets and Data Partition. As shown in Fig. 4, in federated learning
applications, non-IID data distribution is a common scenario, often arising in
two forms: label-skew [Diao et al., 2022] and feature-skew (also known as
domain-shift) [Huang et al., 2023]. We seek to corroborate the effectiveness
of our method under these two distinctive non-IID conditions. We select two
datasets associated with label-skew setting, namely CIFAR-10 [Krizhevsky et
al., 2009] and Tiny-ImageNet [Le and Yang, 2015], along with two datasets
linked to domain-shift setting, specifically PACS [Li et al., 2017] and
Office-Caltech-10 [Gong et al., 2012].

The training datasets for CIFAR-10 and Tiny-ImageNet are partitioned into
$N=10$ clients with Dirichlet distribution $\beta=0.5$. Meanwhile, with
respect to the PACS and Office-Caltech-10 datasets, they intrinsically contain
four distinct domain samples, each allocated to a single client, yielding
$N=4$ clients in total. Each client is provided with a randomly selected 90%
of the local training dataset for training purposes, with the remaining
serving as the validation set. We test the final model $m_{final}$ on all test
data from all clients, providing a global test performance measurement.

(a) Label-skew distribution.

(b) Domain-shift distribution.

Figure 4. Two data distributions across clients. For the label-skew distribution, the color depth of every square represents the number of samples of the corresponding class on that client; for the domain-shift (feature-skew) distribution, every client possesses a specific domain with all classes. Table 1. Test accuracy (%, mean$\pm$std) comparison of our FedELMY method to other baselines on both label-skew and domain-shift (feature-skew) tasks. MetaFed, FedSeq and our FedELMY are SFL methods, while other baselines are all PFL methods. Distribution | Label-Skew | Domain-Shift
---|---|---
Dataset | CIFAR-10 | Tiny-Imagenet | PACS | Office-Caltech-10
$E_{local}$ | 100 | 200 | 100 | 200 | 100 | 200 | 100 | 200
DFedAvgM | 19.32$\pm$2.13 | 18.59$\pm$1.65 | 2.72$\pm$0.52 | 2.02$\pm$0.20 | 20.49$\pm$1.07 | 21.58$\pm$2.28 | 10.18$\pm$1.56 | 10.01$\pm$0.70
DFedSAM | 17.12$\pm$0.44 | 18.51$\pm$1.28 | 2.58$\pm$0.29 | 3.15$\pm$0.29 | 19.01$\pm$1.09 | 20.79$\pm$1.36 | 14.74$\pm$1.63 | 15.09$\pm$1.04
FedOV | 36.32$\pm$7.56 | 38.06$\pm$7.40 | 1.18$\pm$0.10 | 1.29$\pm$0.30 | 12.09$\pm$4.00 | 22.15$\pm$1.40 | 9.67$\pm$4.09 | 10.01$\pm$3.72
DENSE | 61.76$\pm$0.43 | 64.59$\pm$1.04 | 1.27$\pm$0.45 | 1.49$\pm$0.05 | 17.34$\pm$2.14 | 15.60$\pm$5.40 | 33.12$\pm$5.16 | 36.68$\pm$4.87
MetaFed | 71.46$\pm$0.95 | 71.31$\pm$0.73 | 23.52$\pm$0.29 | 24.76$\pm$0.24 | 35.67$\pm$0.27 | 38.73$\pm$2.61 | 41.78$\pm$3.24 | 42.26$\pm$3.77
FedSeq | 72.92$\pm$0.62 | 73.59$\pm$0.95 | 25.50$\pm$0.34 | 25.08$\pm$0.89 | 43.68$\pm$1.29 | 46.53$\pm$0.75 | 32.78$\pm$3.99 | 37.07$\pm$5.78
FedELMY | 79.03$\pm$0.74 | 80.28$\pm$1.18 | 32.84$\pm$0.22 | 30.42$\pm$0.10 | 46.08$\pm$1.82 | 47.74$\pm$1.68 | 44.64$\pm$1.98 | 45.26$\pm$3.93

Baselines. We compare our method with baselines under various federated
learning settings: DFedAvgM [Sun et al., 2022], a decentralized adaptation of
the extensively studied FedAvg [McMahan et al., 2017] method in federated
learning; DFedSAM [Shi et al., 2023], an application of the SAM [Foret et al.,
2021] optimizer in decentralized PFL; FedOV [Li et al., 2022], a centralized
one-shot federated learning method employing the open-set to bolster the final
model; DENSE [Zhang et al., 2022], a one-shot federated learning framework
also reliant on a central server to deliver a global model via knowledge
distillation. Lastly, we compare our method to two sequential federated
learning methods: MetaFed [Chen et al., 2023], a personalized sequential
federated learning approach used for healthcare, which requires at least two
rounds of communication; FedSeq [Li and Lyu, 2024], which is the state-of-the-
art sequential federated learning algorithm.

In an effort to ensure a fair comparison, we adapted the decentralized PFL
methods DFedAvgM and DFedSAM to the one-shot setting, and adjusted these
methods to select all clients for training and communication to fit the
setting. In the remaining parts, unless otherwise specified, we set the local
training epoch $E_{local}$ to 200. For more implementation details about our
method and all baseline methods, such as the optimizer, learning rate and
batch size we employed, please refer to Sec. 1 of the supplementary material.

###  4.2. Effectiveness

####  4.2.1. One-shot Setting

Table 1 presents the test accuracy of different methods in two training
settings ($E_{local}=100$ and $200$) to verify the robustness of our method.
Results show that our FedELMY method outperforms all other methods across all
datasets including both label-skew and domain-shift tasks. Specifically,
FedELMY surpasses existing SFL methods by over 6% in accuracy, and outperforms
current PFL methods by more than 15% on the CIFAR-10 (label-skew) dataset.
Additionally, FedELMY achieves a 25% higher accuracy over PFL methods and at
least an 1.5% improvement over SFL methods on the PACS (domain-shift) dataset.
These results consistently demonstrate the effectiveness of our method.

It is noteworthy that for the Tiny-ImageNet dataset, due to its large number
of classes (200), most PFL baselines fail to deliver effective performance
such as DENSE (close to random guesses), suggesting that they cannot
effectively handle datasets with too many classes. However, our method works
well and also achieves improvement compared to other SFL methods. Such
improvements are attributed to the diverse local training so that our method
can learn better feature representations.

In addition, it is not always the case that the test accuracy of
$E_{local}=200$ is better than $100$, indicating that simply increasing the
number of training rounds does not always improve model performance –
enhancing diversity is an essential factor.

Table 2. Few-Shot performance comparison for PACS dataset. Shot | 1 | 3 | 5 | 7
---|---|---|---|---
MetaFed | 41.62% | 44.75% | 46.11% | 46.64%
FedSeq | 47.31% | 49.30% | 50.96% | 50.97%
FedELMY | 49.14% | 56.35% | 57.05% | 57.13%

####  4.2.2. Few-Shot Setting

Although FedELMY is designed for the one-shot setting, here we explore the
performance of FedELMY in few-shot scenarios to validate the scalability of
our framework, i.e., when the final client $N$ sends model $m_{avg}^{N}$ to
the first client $1$, thereby starting a new cycle of model training. As we
can see from Table 2, even under few-shot settings, FedELMY consistently
outperforms the SFL baselines. Such observations underscore the effectiveness
and scalability of FedELMY. Meanwhile, we observe that as the number of
training rounds increases to a certain extent, the overall performance does
not significantly improve, indicating that the model has reached a state of
convergence. This concludes that blindly increasing the number of training
rounds will not significantly enhance the model performance.

Figure 5. Communication cost comparison of different algorithms for CIFAR-10
dataset when the number of clients $N=10$ with model ResNet-18.

###  4.3. Efficiency

In this section, we will evaluate our method’s efficiency by comparing its
communication cost and model performance under different computation costs
with other baselines.

####  4.3.1. Communication cost.

Fig. 5 shows the communication expenses of different methods with $N=10$
clients on the CIFAR-10 dataset with the Resnet-18 model, whose size is $M=$
46.2 MB. The communication cost of FedELMY is restricted to $(N-1)\times M=$
415.8 MB, with only the FedSeq method displaying the same cost as ours;
however, it notably underperforms in performance (Table. 1). MetaFed requires
$(2N-1)\times M=$ 877.8 MB of communication cost since it requires at least
two rounds of communication (common knowledge accumulation and
personalization). Central server-dependent methods, such as DENSE and FedOV,
require an expenditure of $N\times M=$ 462 MB to transfer models to the
server. Other methods mandating communication with their neighbors like
DFedAvgM and DFedSAM, will require even higher communication costs.
Accordingly, our method effectively diminishes the communication burden of FL
and thus strengthens data privacy.

Figure 6. Test accuracy comparison with different computation costs on
CIFAR-10 dataset, where FedELMY ($S=5,E_{local}=40$) means we will train 5
models on every client, wherein each model undergoes 40 training epochs in our
framework; FedSeq ($E_{local}=1000$) denotes that we will train only 1 model
for 1000 epochs for every client in FedSeq.

####  4.3.2. Computation cost.

Given that FedELMY requires training $S$ models at each client, it involves
$S\times E_{local}$ epochs of training per client. Compared to FedSeq, in
which each client trains a single model for $E_{local}$ epochs, and MetaFed,
which necessitates $2\times E_{local}$ epochs of training per client, our
method appears to demand a higher computational expense. However, FedELMY
significantly enhances the model performance without increasing the
communication cost (Table. 1). Thus, our method strikes a balance (trade-off)
between computational cost (training time) and performance gains.

Meanwhile, we conducted experiments to align the computational costs of
different methods for a fair comparison (Fig. 6). First, we align the
computation cost of our method to the baselines. By setting
{$S=5,E_{local}=40\\}$, FedELMY maintains the same computational cost as
baselines. As we can see, even under this setting, our method still
outperforms the baselines; then, we align the computation cost of the
baselines to FedELMY by increasing $E_{local}$ to 1000, Fig. 6 shows that
FedSeq displays worse performance and MetaFed shows negligible variation in
its performance. This can be attributed to the phenomenon of overfitting due
to excessive local training, which increases the model’s generalization error
and hence, compromises their performance. Contrastingly, our approach, which
equally included $S\times E_{local}$ training rounds, exhibited superior
performance. These experiments all validate the efficacy of our model training
procedures by the diversity-enhanced mechanism.

Table 3. Ablation studies on different regularization terms. Method | $\mathcal{M}$ | $d1$ | $d2$ | CIFAR10 | PACS
---|---|---|---|---|---
MetaFed |  |  |  | 71.29% | 41.62%
FedSeq |  |  |  | 73.54% | 47.31%
FedELMY | $\checkmark$ |  |  | 78.92% | 48.11%
FedELMY | $\checkmark$ | $\checkmark$ |  | 79.62% | 49.04%
FedELMY | $\checkmark$ |  | $\checkmark$ | 79.77% | 48.94%
FedELMY | $\checkmark$ | $\checkmark$ | $\checkmark$ | 80.08% | 49.14%

###  4.4. Ablation study

In this section, we examine the impact of distance metrics, hyperparameters,
and diversity control measures on our method’s performance, as well as the
influence of client orders.

####  4.4.1. Effects of distance terms.

We conducted ablation experiments to verify the effectiveness of the
components in our method. Since the model pool $\mathcal{M}$ is indispensable,
we examined the impact of the two distance regularization terms, $d_{1}$ and
$d_{2}$, referenced in Eq. 7 and Eq. 8 respectively. Table 3 illustrates the
outcomes of respective schemes. As we can see, the introduction of either
$d_{1}$ or $d_{2}$ independently improves our method’s performance compared to
solely using the model pool $\mathcal{M}$, and further enhancement is achieved
when both distance terms are incorporated. This confirms that integrating both
distance terms will strengthen the model’s performance. It is worth
emphasizing that even in the absence of $d_{1}$ and $d_{2}$, relying solely on
the model pool $\mathcal{M}$ for training, our approach still outperforms the
baselines. This reaffirms the essential paradigm for fostering diversity
within the system.

(a) Effect of warm-up epoch $E_{\mathrm{w}}$.

(b) Effect of model quantity $S$.

(c) Effect of scale parameter $\alpha$.

(d) Effect of scale parameter $\beta$.

Figure 7. Grid search results for CIFAR-10 dataset to investigate the
sensitivity of FedELMY to various hyperparameters.

####  4.4.2. Hyperparameter sensitivity.

We further investigate the sensitivity of our method to different choices of
hyperparameters, as shown in Fig. 7. The optimal combination of these
parameters was then employed in subsequent training. It is evident from the
figure that regardless of the adopted search strategies, the performance
robustness of our method remains relatively undisturbed across all four
examined parameters. This evidence suggests that the performance of our method
is robust to the change of hyperparameters.

(a) FedELMY

(b) MetaFed

(c) FedSeq

Figure 8. T-SNE comparison between FedELMY and other SFL baselines on PACS dataset with model ResNet-18. Table 4. Performance comparison of PACS dataset for different domain training orders, where the order “PACS” means we train models by domain order “Photo (client 1) $\rightarrow$ Art-Painting (client 2) $\rightarrow$ Cartoon (client 3) $\rightarrow$ Sketch (client 4)”. Order | PACS | ACPS | SCPA | CSPA | Average
---|---|---|---|---|---
MetaFed | 41.62% | 42.75% | 31.65% | 40.64% | 39.17%
FedSeq | 47.31% | 45.21% | 40.96% | 33.71% | 41.80%
FedELMY | 49.14% | 46.74% | 43.65% | 41.46% | 45.25%
Figure 9. Test accuracy comparison with different diversity control measures,
where FedELMY (L2-Norm) means we will use the L2-norm (Euclidean) distance as
our diversity control measure to train the models.

(a) CIFAR-10 ($S=5$)

(b) PACS ($S=10$)

Figure 10. Heatmap of pairwise L2-norm (Euclidean) distances of all trained
models within the final client’s model pool $\mathcal{M}^{N}$ with size $S+1$.

####  4.4.3. Client order.

To explore the influence of the order in which domains are trained in domain-
shift tasks, we applied our method to the PACS dataset with various domain
training orders, as depicted in Table 4. As we can see, irrespective of the
order chosen for domain training, our method consistently surpasses the
performance of the baseline methods. This signifies our method’s robustness in
handling diverse domain order training.

####  4.4.4. Diversity control measures.

Fig. 9 presents the impact of various diversity control measures applied with
different distances. Notably, the L2-norm emerges as the best measure, yet,
the remaining measures also exceed the performance of baseline methods in most
cases. This superiority of the L2-norm is attributed to its ability to measure
the true distance between points directly and precisely, offering a more
sensitive and effective constraint on point dispersion in the space, thereby
yielding superior results.

###  4.5. Case study

####  4.5.1. Classification result visualization

Fig. 8 illustrates the t-SNE plots of the feature maps for our method and
other SFL baselines on the PACS dataset with model ResNet-18. As we can see,
in comparison to MetaFed and FedSeq, FedELMY more effectively clusters samples
of the same class, indicating a clear separation in the feature space for
class 0, 1, 2, 3, and 4—a capability not observed in other baselines.
Consequently, by refining the diverse local training process, our approach is
adept at learning better feature representations, thereby improving overall
model performance.

####  4.5.2. Local model diversities.

Fig. 10 illustrates the L2-norm distance matrix for each pair of models within
the final client’s model pool $\mathcal{M}^{N}$, after the completion of
training. It is evident that the pairwise distances among all trained models
in the model pool display a remarkable variation, without an apparent
correlation or trend, such as monotonically increasing/decreasing. This
confirms the significant diversity among the models within the pool, attesting
to the efficacy of our method in fostering model diversity. By strategically
enhancing the disparities between models, we can substantively improve the
performance of our approach.

##  5\. Conclusion

In this paper, we highlight the significance of one-shot sequential federated
learning for alleviating the communication burdens of current collaborative
machine learning paradigms and address the challenge posed by non-IID data. We
present a novel one-shot SFL framework with the local model diversity
enhancement strategy to reduce communication costs and effectively improve the
global model. In particular, we design the local model pool with two
regularization terms as a diversity-enhanced mechanism to improve model
performance and mitigate the effect of non-IID data. The effectiveness of our
method was demonstrated with superior performance on extensive experiments
across several datasets including both label-skew and domain-shift tasks. In
the future, we will consider integrating more advanced privacy protection
measures, adapting to more federated learning settings, and dealing with real-
time data under online learning environments to further enhance the
feasibility and scalability of the proposed framework.

## References

  * Acar et al. [2021] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama.  Federated learning based on dynamic regularization.  _arXiv preprint arXiv:2111.04263_ , 2021.
  * Balkus et al. [2022] Salvador V Balkus, Honggang Wang, Brian D Cornet, Chinmay Mahabal, Hieu Ngo, and Hua Fang.  A survey of collaborative machine learning using 5g vehicular communications.  _IEEE Communications Surveys & Tutorials_, 24(2):1280–1303, 2022.
  * Beltrán et al. [2023] Enrique Tomás Martínez Beltrán, Mario Quiles Pérez, Pedro Miguel Sánchez Sánchez, Sergio López Bernal, Gérôme Bovet, Manuel Gil Pérez, Gregorio Martínez Pérez, and Alberto Huertas Celdrán.  Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges.  _IEEE Communications Surveys & Tutorials_, 2023.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Bonawitz et al. [2019] Keith Bonawitz, Brendan McMahan, et al.  Towards federated learning at scale: System design.  _Proceedings of Machine Learning and Systems_ , 1:374–388, 2019.
  * Bottou [2010] Léon Bottou.  Large-scale machine learning with stochastic gradient descent.  In _Proceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers_ , pages 177–186. Springer, 2010.
  * Cha et al. [2021] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  _Advances in Neural Information Processing Systems_ , 34:22405–22418, 2021.
  * Chang et al. [2018] Ken Chang, Niranjan Balachandar, Carson Lam, Darvin Yi, James Brown, Andrew Beers, Bruce Rosen, Daniel L Rubin, and Jayashree Kalpathy-Cramer.  Distributed deep learning networks among institutions for medical imaging.  _Journal of the American Medical Informatics Association_ , 25(8):945–954, 2018.
  * Chen et al. [2023] Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang, and Xing Xie.  Metafed: Federated learning among federations with cyclic knowledge distillation for personalized healthcare.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * Cho et al. [2023] Yae Jee Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, and Tong Zhang.  On the convergence of federated averaging with cyclic client participation.  In _International Conference on Machine Learning_ , pages 5677–5721. PMLR, 2023.
  * Diao et al. [2022] Yiqun Diao, Qinbin Li, and Bingsheng He.  Towards addressing label skews in one-shot federated learning.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Duan [2023] Moming Duan.  Towards open federated learning platforms: Survey and vision from technical and legal perspectives.  _arXiv preprint arXiv:2307.02140_ , 2023.
  * Duan et al. [2020] Moming Duan, Liang Liang, et al.  Self-balancing federated learning with global imbalanced data in mobile systems.  _IEEE Transactions on Parallel and Distributed Systems_ , 32(1):59–71, 2020.
  * Feng et al. [2023] Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo.  Learning federated visual prompt in null space for mri reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Foret et al. [2021] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=6Tm1mposlrM.
  * Gao et al. [2022] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu.  Feddc: Federated learning with non-iid data via local drift decoupling and correction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10112–10121, 2022.
  * Gao et al. [2021] Wei Gao, Shangwei Guo, Tianwei Zhang, Han Qiu, Yonggang Wen, and Yang Liu.  Privacy-preserving collaborative learning with automatic transformation search.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 114–123, 2021.
  * Gong et al. [2012] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.  Geodesic flow kernel for unsupervised domain adaptation.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 2066–2073. IEEE, 2012.
  * Guha et al. [2018] Neel Guha, Ameet Talwalkar, and Virginia Smith.  One-shot federated learning.  In _NeurIPS 2018 Workshop on Machine Learning on the Phone and other Consumer Devices_ , 2018.
  * Hard et al. [2021] Andrew Hard, Kurt Partridge, Rajiv Mathews, and Sean Augenstein.  Jointly learning from decentralized (federated) and centralized data to mitigate distribution shift.  2021\.
  * Hegedűs et al. [2021] István Hegedűs, Gábor Danner, and Márk Jelasity.  Decentralized learning works: An empirical comparison of gossip learning and federated learning.  _Journal of Parallel and Distributed Computing_ , 148:109–124, 2021.
  * Heinbaugh et al. [2023] Clare Elizabeth Heinbaugh, Emilio Luz-Ricca, and Huajie Shao.  Data-free one-shot federated learning under very high statistical heterogeneity.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Huang et al. [2022] Wenke Huang, Mang Ye, and Bo Du.  Learn from others and be yourself in heterogeneous federated learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10143–10153, 2022.
  * Huang et al. [2023] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du.  Rethinking federated learning with domain shift: A prototype view.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16312–16322. IEEE, 2023.
  * Izmailov et al. [2018] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  _arXiv preprint arXiv:1803.05407_ , 2018.
  * Jhunjhunwala et al. [2023] Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi.  Towards a theoretical and practical understanding of one-shot federated learning with fisher information.  In _Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities_ , 2023.
  * Kairouz et al. [2021] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ , 14(1–2):1–210, 2021.
  * Kamp et al. [2023] Michael Kamp, Jonas Fischer, and Jilles Vreeken.  Federated learning from small datasets.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=hDDV1lsRV8.
  * Karimireddy et al. [2020] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh.  Scaffold: Stochastic controlled averaging for federated learning.  In _International conference on machine learning_ , pages 5132–5143. PMLR, 2020.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Le and Yang [2015] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7):3, 2015.
  * Li et al. [2017] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5542–5550, 2017.
  * Li et al. [2021a] Qinbin Li, Bingsheng He, and Dawn Song.  Practical one-shot federated learning for cross-silo setting.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, (IJCAI)_ , pages 1484–1490. International Joint Conferences on Artificial Intelligence Organization, 8 2021a.  doi: 10.24963/ijcai.2021/205.  URL https://doi.org/10.24963/ijcai.2021/205.
  * Li et al. [2023] Qinbin Li, Bingsheng He, and Dawn Song.  Adversarial collaborative learning on non-iid features.  2023\.
  * Li et al. [2022] Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao.  Learning to collaborate in decentralized learning of personalized models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9766–9775, 2022.
  * Li et al. [2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  _Proceedings of Machine learning and systems_ , 2:429–450, 2020.
  * Li et al. [2021b] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou.  Fedbn: Federated learning on non-iid features via local batch normalization.  _arXiv preprint arXiv:2102.07623_ , 2021b.
  * Li and Lyu [2024] Yipeng Li and Xinchen Lyu.  Convergence analysis of sequential federated learning on heterogeneous data.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ma et al. [2020] Chuan Ma, Jun Li, Ming Ding, Howard H Yang, Feng Shu, Tony QS Quek, and H Vincent Poor.  On safeguarding privacy and security in the framework of federated learning.  _IEEE network_ , 34(4):242–248, 2020.
  * McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pages 1273–1282. PMLR, 2017.
  * Qu et al. [2022] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin.  Rethinking architecture design for tackling data heterogeneity in federated learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10061–10071, 2022.
  * Rame et al. [2022] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord.  Diverse weight averaging for out-of-distribution generalization.  _Advances in Neural Information Processing Systems_ , 35:10821–10836, 2022.
  * Roy et al. [2019] Abhijit Guha Roy, Shayan Siddiqui, Sebastian Pölsterl, Nassir Navab, and Christian Wachinger.  Braintorrent: A peer-to-peer environment for decentralized federated learning.  _arXiv preprint arXiv:1905.06731_ , 2019.
  * Sheller et al. [2020] Micah J Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al.  Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data.  _Scientific reports_ , 10(1):12598, 2020.
  * Shi et al. [2023] Yifan Shi, Li Shen, Kang Wei, Yan Sun, Bo Yuan, Xueqian Wang, and Dacheng Tao.  Improving the model consistency of decentralized federated learning.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023.
  * Shoham et al. [2019] Neta Shoham, Tomer Avidor, Aviv Keren, Nadav Israel, Daniel Benditkis, Liron Mor-Yosef, and Itai Zeitak.  Overcoming forgetting in federated learning on non-iid data.  _arXiv preprint arXiv:1910.07796_ , 2019.
  * Su et al. [2023] Shangchao Su, Bin Li, and Xiangyang Xue.  One-shot federated learning without server-side training.  _Neural Networks_ , 164:203–215, 2023.  doi: 10.1016/j.neunet.2023.04.035.
  * Sun et al. [2022] Tao Sun, Dongsheng Li, and Bao Wang.  Decentralized federated averaging.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):4289–4301, 2022.
  * T Dinh et al. [2020] Canh T Dinh, Nguyen Tran, and Josh Nguyen.  Personalized federated learning with moreau envelopes.  _Advances in Neural Information Processing Systems_ , 33:21394–21405, 2020.
  * Wang et al. [2022] Junbo Wang, Amitangshu Pal, Qinglin Yang, Krishna Kant, Kaiming Zhu, and Song Guo.  Collaborative machine learning: Schemes, robustness, and privacy.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Wang et al. [2023] Naibo Wang, Wenjie Feng, Fusheng Liu, Moming Duan, and See-Kiong Ng.  Data-free diversity-based ensemble selection for one-shot federated learning in machine learning model market.  _arXiv preprint arXiv:2302.11751_ , 2023.
  * Wortsman et al. [2022] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pages 23965–23998. PMLR, 2022.
  * Wu et al. [2023] Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng.  Personalized federated learning under mixture of distributions.  In _Proceedings of the 40th International Conference on Machine Learning (ICML)_ , pages 37860–37879, 2023.
  * Yang et al. [2023] Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue.  Exploring one-shot semi-supervised federated learning with a pre-trained diffusion model.  _arXiv preprint arXiv:2305.04063_ , 2023.
  * Yang et al. [2019] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong.  Federated machine learning: Concept and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 10(2):1–19, 2019.
  * Yuan et al. [2023] Liangqi Yuan, Lichao Sun, Philip S Yu, and Ziran Wang.  Decentralized federated learning: A survey and perspective.  _arXiv preprint arXiv:2306.01603_ , 2023.
  * Zaccone et al. [2022] Riccardo Zaccone, Andrea Rizzardi, Debora Caldarola, Marco Ciccone, and Barbara Caputo.  Speeding up heterogeneous federated learning with sequentially trained superclients.  In _2022 26th International Conference on Pattern Recognition (ICPR)_ , pages 3376–3382. IEEE, 2022.
  * Zhang et al. [2022] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu.  Dense: Data-free one-shot federated learning.  _Advances in Neural Information Processing Systems_ , 35:21414–21428, 2022.

Appendix

##  Appendix A Experimental Setup

Adaptation Details In an effort to ensure a fair comparison, we adapted the
decentralized algorithms DFedAvgM, DFedSAM, and FedSeq to the one-shot
setting, and adjusted these methods to select all clients for both training
and communication to fit the one-shot setting. This modification means we
operate these methods for only one round of communication and select all
clients for training and model distribution instead of just a portion,
ensuring that all the clients’ local data are utilized.

Implementation Details In all methods, for local model training, we select the
model with the highest validation accuracy as the final model. We employed the
Adam optimizer (excluding DFedSAM, which utilized the SAM optimizer) with a
learning rate of $\eta=5\times 10^{-5}$ applicable for the CIFAR-10, PACS, and
Tiny-ImageNet datasets. Due to the relatively smaller size of the Office-
Caltech-10 dataset, we adopted the Adam optimizer with a learning rate of
$\eta=0.001$. For the MA-Echo method, we used the learning rate of 0.01 for
all datasets as specified in their paper. In all cases, the Adam optimizer’s
weight decay was fixed at $1\times 10^{-4}$. As for the other hyperparameters
of the baselines, we adhered to the values suggested in their papers, which
were kept constant and chosen based on optimal validation accuracy. We applied
the optimal hyperparameters obtained through the grid-search strategy for our
method. Specifically, we set $E_{\mathrm{w}}$ to $30$ for the CIFAR-10
dataset, $20$ for all other datasets; we set $S$ to $5$ for the CIFAR-10
dataset, $10$ for the PACS and Office-Caltech-10 dataset, and $3$ for the
Tiny-ImageNet dataset; we set $\alpha$ to $0.06$ for the CIFAR-10 dataset, $1$
for the PACS and Tiny-ImageNet dataset, $0.001$ for the Office-Caltech-10
dataset; we set $\beta$ to $1$ for the CIFAR-10, PACS and Tiny-ImageNet
dataset, $0.001$ for the Office-Caltech-10 dataset. During the local training
phase of every client, in addition to utilizing scale hyperparameters $\alpha$
and $\beta$, our method also included control over the magnitudes of the
distance parameters $d_{1}$ and $d_{2}$ to one order of magnitude smaller than
the original loss $\ell$, which was achieved through the logarithmic scaling.
For instance, assuming the initial loss $\ell$ is 6.02 and the original
distance $d_{1}$ is 45, we would first normalize $d_{1}$ to 0.45. Following
this, we multiply this normalized distance by the scaling factor $\alpha$ and
subsequently integrate it into the comprehensive loss function $\mathcal{L}$.
This calibration ensures the original loss maintains primacy in influencing
the training process, while the distance regularization terms play an
ancillary role, thereby preventing their dominance over the central objective.

Environment All our experiments were conducted on a single machine with 1TB
RAM and 256-core AMD EPYC 7742 64-Core Processor @ 3.4GHz CPU. The GPU we used
is NVIDIA A100 SXM4 with 40GB memory. The software environment settings are:
Python 3.7.4, PyTorch 1.13.1 with CUDA 11.6 on Ubuntu 20.04.4 LTS. All the
experimental results are the average over three trials.

##  Appendix B Few-Shot Adaptation

Alg. 2 illustrates the implementation details and overview of our method in
few-shot scenarios. In this scenario, after the local training of the final
client $N$, it will send its averaged model $m_{avg}^{N}$ to the first client
$1$ to start a new cycle of model training, this process will be repeated for
multiple rounds. After completing $r=T$ iterations, we regard $m_{avg}^{N}$ as
the final global model $m_{final}$.

Input: Local datasets $\mathcal{D}=\\{D_{i}\\}_{i=1}^{N}$, warm-up epoch
$E_{\mathrm{w}}$, learning rate $\eta$, number of local iterations
$E_{local}$, model number to be trained per client $S$, scale hyperparameters
$\alpha$, $\beta$

Output: The final model $m_{final}$

1

2Initialization: For client 1, warm up a randomly initialized model
$m_{avg}^{0}$ for $E_{\mathrm{w}}$ epochs

3for  _shot $r=1:T$_ do

4 for  _client $i=1:N$ _ do

5 Receives $m_{avg}^{i-1}$ from client $i-1$ (for client $1$ from shot $r>1$,
receives from client $N$)

6 // Initialize model pool $\mathcal{M}^{i}$ for client $i$

7 $\mathcal{M}^{i}=\\{m_{0}^{i}\\}$ with $m_{0}^{i}\leftarrow$ $m_{avg}^{i-1}$

8 for  _$j=1:S$_ do

9

10 // Initialize $m_{j}^{i}$

11
$m_{j}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i}$

12 // Local training for $m_{j}^{i}$

13 for  _$k=1:E_{local}$_ do

14

15 $\mathcal{L}(m_{j}^{i})\leftarrow\ell(m_{j}^{i};D_{i})-\alpha\cdot
d_{1}+\beta\cdot d_{2}$

16 $m_{j}^{i}\leftarrow m_{j}^{i}-\eta\nabla_{m}\mathcal{L}(m_{j}^{i})$

17 end for

18

19 $\mathcal{M}^{i}\leftarrow\mathcal{M}^{i}\cup\\{m_{j}^{i}\\}$

20 end for

21
$m_{avg}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i}$

22 Sends $m_{avg}^{i}$ to client $i+1$ (for client $N$, sends to client $1$)

23 end for

24

25 end for

26// For the final client $i=N$, outputs model $m_{final}$

$m_{final}\leftarrow m_{avg}^{N}$

Algorithm 2 Few-Shot Adaptation of FedELMY

##  Appendix C Adaptation to Decentralized Parallel Federated Learning

Input: Local datasets $\mathcal{D}=\\{D_{i}\\}_{i=1}^{N}$, warm-up epoch
$E_{\mathrm{w}}$, learning rate $\eta$, number of local iterations
$E_{local}$, model number to be trained per client $S$, scale hyperparameters
$\alpha$, $\beta$

Output: The final model $m_{final}$

1

2Initialization: For every client $i$, warm up a randomly initialized model
$m_{0}^{i}$ for $E_{\rm{w}}$ epochs

3for  _client $i=1:N$ in parallel _ do

4

5 // Initialize model pool $\mathcal{M}^{i}$ for client $i$

6 $\mathcal{M}^{i}=\\{m_{0}^{i}\\}$

7 for  _$j=1:S$_ do

8

9 // Initialize $m_{j}^{i}$

10
$m_{j}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i}$

11 // Local training for $m_{j}^{i}$

12 for  _$k=1:E_{local}$_ do

13

14 $\mathcal{L}(m_{j}^{i})\leftarrow\ell(m_{j}^{i};D_{i})-\alpha\cdot
d_{1}+\beta\cdot d_{2}$

15 $m_{j}^{i}\leftarrow m_{j}^{i}-\eta\nabla_{m}\mathcal{L}(m_{j}^{i})$

16 end for

17

18 $\mathcal{M}^{i}\leftarrow\mathcal{M}^{i}\cup\\{m_{j}^{i}\\}$

19 end for

20
$m_{avg}^{i}\leftarrow\frac{1}{|\mathcal{M}^{i}|}\sum_{t=0}^{|\mathcal{M}^{i}|-1}m_{t}^{i}$

21 Sends $m_{avg}^{i}$ to all other clients $j=1:N$ , ${j}\neq{i}$

22 end for

23

24// For any client $i=1:N$, output model $m_{final}$

$m_{final}\leftarrow\frac{1}{N}\sum_{i=1}^{N}m_{avg}^{i}$

Algorithm 3 FedELMY for Decentralized PFL

Our method can also be adapted to the decentralized Parallel Federated
Learning (PFL) setting. Alg. 3 demonstrates the implementation details of our
method under the decentralized PFL environment. As we can see, under this
setting, clients will train their models concurrently (in parallel). During
the training process, each client begins with a randomly initialized model
$m_{0}^{i}$. Upon completion of local training, each client will
simultaneously transmit the locally averaged model $m_{avg}^{i}$ to all other
clients. Once a client $i$ receives all models from every neighbor, it will
calculate an aggregated model by averaging its own averaged model
$m_{avg}^{i}$ with those received averaged models as the final model
$m_{final}$. Please note that our framework can also be adjusted for a
centralized PFL environment with a central server. In this setup, each client
will send its averaged model $m_{avg}^{i}$ to the central server. The overall
performance will be consistent with that of the decentralized PFL adaptation
since the server will finally average all models in a manner identical to the
decentralized setting. Experimental results of such an adaptation are shown in
Sec. D.

##  Appendix D Additional Experiments

In the remaining parts, unless otherwise specified, we set the local training
epoch $E_{local}$ to 200, use the ResNet-18 model structure and follow the
Dirichlet distribution $Dir(0.5)$ for the label-skew tasks in our experiments.

Table 5. Test accuracy (%) comparison on more public datasets, including both
label-skew and domain-shift datasets.

Distrubution | Label-Skew | Domain-Shift
---|---|---
Dataset | MNIST | SVHN | Office31 |  Office- Home
DFedAvgM | 11.35$\pm$4.01 | 18.60$\pm$5.66 | 3.47$\pm$1.11 | 1.71$\pm$0.23
DFedSAM | 12.53$\pm$3.93 | 22.97$\pm$4.98 | 3.49$\pm$1.29 | 1.25$\pm$0.32
FedOV | 73.10$\pm$0.33 | 37.47$\pm$1.48 | 3.39$\pm$0.46 | 2.31$\pm$0.15
DENSE | 95.88$\pm$1.59 | 73.76$\pm$1.25 | 3.15$\pm$0.58 | 1.68$\pm$0.17
MetaFed | 98.07$\pm$0.39 | 76.66$\pm$1.78 | 2.97$\pm$0.43 | 2.40$\pm$0.50
FedSeq | 98.31$\pm$0.65 | 76.18$\pm$2.45 | 3.46$\pm$0.22 | 2.47$\pm$0.29
FedELMY | 98.91$\pm$0.23 | 79.55$\pm$4.12 | 3.57$\pm$0.09 | 2.70$\pm$0.13

D.1 Comparison on more datasets. We compare our method to other baselines on
four more public datasets, including both label-skew and domain-shift datasets
to enrich our experimental results. As shown in Table 5, our method
consistently outperforms all other baselines on both label-skew and domain-
shift scenarios, which further validates the effectiveness of our approach.

D.2 Hyperparameter sensitivity. We investigate the sensitivity of our method
to different choices of hyperparameters for the PACS dataset as a supplement,
as shown in Fig. 11. The findings show that our method maintains consistent
robustness irrespective of the search strategies implemented, as observed
across all four evaluated hyperparameters (when $S$ grew to around 10, the
performance started to become stable). Such evidence underscores the method’s
limited reliance on hyperparameter selection.

(a) Effect of warm-up epoch $E_{\mathrm{w}}$.

(b) Effect of model quantity $S$.

(c) Effect of scale parameter $\alpha$.

(d) Effect of scale parameter $\beta$.

Figure 11. Grid search results for PACS dataset to investigate the sensitivity
of our method to different hyperparameters.

D.3 Impact of computation cost. We show another experiment about the effect of
different computation costs on the PACS dataset with Resnet-18 structure. As
we can see from Fig. 12, our method can still get the best performance under
different computation cost settings, which validates the efficacy of our model
training procedures by the diversity-enhanced mechanism.

Figure 12. Test accuracy comparison with different computation costs on PACS dataset. Table 6. Test accuracy (%) comparison of our FedELMY method to other baselines on different numbers of clients $N$. For the domain-shift tasks like the PACS dataset, the number of clients $N=8$ means we split the whole dataset into 8 parts and allocate them to the clients following the order of “Photo (client 1) $\rightarrow$ Art-Painting (client 2) $\rightarrow$ Cartoon (client 3) $\rightarrow$ Sketch (client 4) $\rightarrow$ Photo (client 5) $\rightarrow$ Art-Painting (client 6) $\rightarrow$ Cartoon (client 7) $\rightarrow$ Sketch (client 8)” for subsequent training. Distribution | Label-Skew | Domain-Shift
---|---|---
Dataset | CIFAR-10 | Tiny-ImageNet | PACS | Office-Caltech-10
$N$ | 5 | 20 | 50 | 100 | 5 | 20 | 50 | 100 | 8 | 20 | 40 | 8 | 20 | 40
DFedAvgM | 21.27 | 18.22 | 19.50 | 24.48 | 2.64 | 1.96 | 1.99 | 3.09 | 17.55 | 16.37 | 16.62 | 10.65 | 9.73 | 8.94
DFedSAM | 30.62 | 18.75 | 10.00 | 10.01 | 4.34 | 2.67 | 0.64 | 0.51 | 16.56 | 16.48 | 16.58 | 14.88 | 11.58 | 11.52
FedOV | 41.49 | 30.93 | 31.93 | 38.42 | 1.33 | 1.23 | 1.11 | 1.08 | 18.59 | 17.73 | 17.32 | 15.78 | 10.56 | 9.34
DENSE | 67.54 | 52.77 | 48.38 | 19.04 | 3.88 | 3.03 | 3.12 | 3.01 | 19.18 | 9.24 | 14.53 | 20.72 | 9.52 | 11.07
MetaFed | 72.53 | 57.73 | 47.47 | 32.13 | 31.44 | 20.46 | 16.16 | 13.04 | 27.33 | 20.35 | 16.76 | 19.43 | 11.33 | 11.45
FedSeq | 73.84 | 64.05 | 53.62 | 35.82 | 31.97 | 23.38 | 14.92 | 15.27 | 33.01 | 17.29 | 12.20 | 18.40 | 12.23 | 9.78
FedELMY | 80.99 | 68.18 | 64.31 | 40.01 | 38.43 | 29.52 | 19.93 | 20.83 | 36.10 | 25.19 | 23.60 | 24.45 | 12.48 | 12.21
Table 7. Test accuracy (%) comparison of our FedELMY method to other baselines with the CNN model structure. Dataset | CIFAR-10 |  Tiny- ImageNet | PACS |  Office- Caltech-10
---|---|---|---|---
DFedAvgM | 15.08 | 3.56 | 20.56 | 22.84
DFedSAM | 13.25 | 0.49 | 15.79 | 9.20
FedOV | 53.49 | 0.69 | 12.17 | 22.65
DENSE | 61.42 | 6.72 | 27.16 | 34.88
MetaFed | 56.11 | 6.68 | 28.46 | 29.99
FedSeq | 66.08 | 14.49 | 38.36 | 38.22
FedELMY | 69.49 | 17.05 | 39.89 | 39.77

D.4 Impact of client numbers. We assessed the performance of various methods
across all datasets by changing the number of clients $N$, as shown in Table
6. It is evident that there is a general trend toward diminishing accuracy for
all methods as the client number $N$ increases, aligning with findings
reported in other literature on traditional federated learning. However, it is
noteworthy that despite the influence of client quantity on one-shot federated
learning, our approach consistently surpasses other baseline methods in
performance. To conclude, these experiments consistently confirm the
effectiveness and robustness of our proposed method to deal with one-shot
sequential federated learning.

D.5 Impact of model structure. To evaluate the scalability of our approach, we
modified the model structure to a three-layer convolutional neural network
(CNN) and replicated the corresponding experiments across all datasets. The
results of these tests, which compare the test accuracy of our method to
various baseline methods under different scenarios, are presented in Table 7,
consistent with the primary analyses reported in the main paper. These
findings affirm the robustness and scalability of our proposed method.

Table 8. Test accuracy (%) comparison of FedELMY to other baselines for different Dirichlet distributions. Dataset | CIFAR-10 | Tiny-ImageNet
---|---|---
$Dir(\cdot)$ | 0.1 | 0.3 | 0.5 | 0.1 | 0.3 | 0.5
DFedAvgM | 10.33 | 16.14 | 18.50 | 1.07 | 1.68 | 2.04
DFedSAM | 15.62 | 21.84 | 19.48 | 1.74 | 2.80 | 3.41
FedOV | 25.92 | 31.93 | 45.69 | 0.94 | 1.23 | 1.59
DENSE | 26.53 | 61.73 | 64.33 | 1.94 | 2.37 | 1.48
MetaFed | 27.18 | 57.24 | 71.29 | 9.96 | 19.45 | 24.53
FedSeq | 27.41 | 57.91 | 73.54 | 15.00 | 22.23 | 24.74
FedELMY | 29.13 | 66.32 | 80.08 | 16.76 | 26.85 | 30.49

D.6 Impact of data distribution. To investigate the label-skew scenarios using
the CIFAR-10 and Tiny-ImageNet datasets, we performed experiments across
varying levels of skew by manipulating the Dirichlet distribution, denoted as
$Dir(\cdot)$. The results, detailed in Table 8, reveal that our approach
outperforms other competing methods in all scenarios, which demonstrates
considerable advantages of our method, particularly in terms of scalability
and privacy preservation, marking it as a more robust solution in these
contexts.

Table 9. Test accuracy (%, mean$\pm$std) comparison of our method under the
decentralized PFL setting to other similar baselines with the Resnet-18 model
structure.

Dataset | CIFAR-10 |  Tiny- ImageNet | PACS |  Office- Caltech-10
---|---|---|---|---
MA-Echo | 10.03$\pm$1.38 | 0.51$\pm$0.02 | 16.70$\pm$0.60 | 11.67$\pm$2.39
DFedAvgM | 18.59$\pm$1.65 | 2.02$\pm$0.20 | 21.58$\pm$2.28 | 10.01$\pm$0.70
DFedSAM | 18.51$\pm$1.28 | 3.15$\pm$0.29 | 20.79$\pm$1.36 | 15.09$\pm$1.04
FedELMY (PFL) | 24.71$\pm$2.62 | 4.12$\pm$0.94 | 24.13$\pm$1.57 | 12.66$\pm$1.36

D.7 PFL Adaptation. Although our method is not exclusively designed for PFL,
we have nevertheless adjusted our algorithm to fit the decentralized PFL setup
for a fair comparison. As shown in Table 9, it is evident that our method,
even when adapted to the decentralized PFL structure, secures optimal results
on most of the datasets relative to other baselines (we compare with MA-Echo
[Su et al., 2023], which represents the state-of-the-art one-shot
decentralized PFL method; DENSE and FedOV which do not conform to the
decentralized structure, are excluded from the comparison). It also surpasses
most existing methods on the Office-Caltech-10 dataset. Therefore, our method
successfully adapts to various settings and data distributions, highlighting
its broad applicability.