Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Structured Federated Learning through

Clustered Additive Modeling

Jie MA\({}^{1}\), Tianyi Zhou\({}^{2}\), Guodong Long\({}^{1}\), Jing Jiang\({}^{1}\), Chengqi Zhang\({}^{1}\)

\({}^{1}\)Australian Artificial Intelligence Institute, FEIT, University of Technology Sydney

\({}^{2}\)University of Maryland

jie.ma-5@student.uts.edu.au, tianyi@umd.au

{guodong.long, jing.jiang, chengqi.zhang}@uts.edu.au

###### Abstract

Heterogeneous federated learning without assuming any structure is challenging due to the conflicts among non-identical data distributions of clients. In practice, clients often comprise near-homogeneous clusters so training a server-side model per cluster mitigates the conflicts. However, FL with client clustering often suffers from "clustering collapse", i.e., one cluster's model excels on increasing clients, and reduces to single-model FL. Moreover, cluster-wise models hinder knowledge sharing between clusters and each model depends on fewer clients. Furthermore, the static clustering assumption on data may not hold for dynamically changing models, which are sensitive to cluster imbalance/initialization or outliers. To address these challenges, we propose "**Clustered Additive Modeling (CAM)**", which applies a globally shared model \(\Theta_{g}\) on top of the cluster-wise models \(\Theta_{1:K}\), i.e., \(y=h(x;\Theta_{g})+f(x;\Theta_{k})\) for clients of cluster-\(k\). The global model captures the features shared by all clusters so \(\Theta_{1:K}\) are enforced to focus on the difference among clusters. To train CAM, we develop a novel **Fed-CAM** algorithm that alternates between client clustering and training global/cluster models to predict the residual of each other. We can easily modify any existing clustered FL methods by CAM and significantly improve their performance without "clustering collapse" in different non-IID settings. We also provide a convergence analysis of Fed-CAM algorithm.

## 1 Introduction

Federated learning (FL) trains a global model over distributed clients and enforces data localization, i.e., data only stay local for model training at the client side while the server periodically averages client models' weights to update a global model and broadcast it to all clients. When the local data distributions are identical across clients, one global model suffices to serve all clients [1] However, non-identical distributions across clients (i.e., statistical heterogeneity) [2] are more common in practical FL scenarios, which leads to conflicts between the global objective and local ones. Instead of applying one model to all the \(m\) clients, an ideal case for non-IID settings would be training a local model per client without any interference from others. However, the local data are usually insufficient so a global model trained on heterogeneous clients exploiting all their data can still be helpful to local training. Hence, non-IID FL methods [3, 4, 5] usually struggle to find a sweet spot between the global consensus and local personalization. Without any assumptions on the structure among clients, all clients' distributions can be equally different from each other so a global model is impacted by the conflicts of all clients and may provide limited guidance to their local training.

**Clustered Federated Learning.** That being said, non-IID clients in practice usually have rich structures that have not been explored by most existing FL methods. A common structure is clusters, i.e., heterogeneous clients can be grouped into several near-homogeneous clusters each composed ofclients with similar distributions. In practice, clusters might be associated with geographic/age/income groups, affiliations, etc. Hence, we can train a server-side model for each cluster, hence mitigating the conflicts caused by heterogeneity. Unfortunately, clients' cluster memberships are usually undefined or inaccessible due to sensitive/private information and have to be jointly optimized with cluster-wise models, as recent clustered FL [6, 7, 8, 9] approaches do. They maintain \(K\) models \(\Theta_{1:K}\) for \(K\) clusters and assigns one \(\Theta_{k}\) to each client-\(i\) (with local data \(X_{i}\) and local model \(\theta_{i}\)), e.g., by min-loss (\(\Theta_{k}\) with the minimum loss on \(X_{i}\)) or K-means (the nearest \(\Theta_{k}\) to \(\theta_{i}\)) criterion. Hence, \(1\leq K\leq m\) models can accommodate more heterogeneity than single-model FL but also allows knowledge sharing among similar clients, which is lacking if training \(m\) client models independently. Hence, it may reach a better trade-off between global consensus and local personalization in non-IID settings.

However, compared to the general non-IID assumption, _clustered FL's assumption might be too restrictive since it prohibits inter-cluster knowledge sharing and enforces every cluster-wise model's training to only depend on a few clients_. This is contradictory to the widely studied strategy that different tasks or domains can benefit from sharing low-level or partial representations. It is due to the gap between the assumption of "clustered data distributions" and the algorithms of "clustering models (represented by loss vectors or model weights)": they are not equal and the latter is more restrictive. In other words, clients of different clusters can still benefit from feature/parameter sharing.

Moreover, _clustered FL usually suffers from optimization instability because dynamically changing models_ can violate the static clustering assumption and lead to imbalanced cluster assignment, which affects \(\Theta_{1:K}\) and local training in the future. In particular: (1) _Clustering collapse_, i.e., the clients assigned to one cluster keeps growing so "the rich becomes richer (i.e., the cluster-wise model becomes even stronger)" until reducing to single-model FL. This happens because most clients tend to first learn shared features before focusing on client-specific ones; (2) _Fragile to outliers_ such as malicious clients that may dominate some clusters and push all other benign ones to one or a few clusters; (3) _Sensitive to initialization_. The process highly depends on initial and earlier cluster assignments since they determine which clients' local training starts from the same model.

**Main Contributions.** To overcome the above problems of clustered FL, we propose a novel structured FL model termed "Clustered **A**dditive **M**odeling (**CAM**)". Compared to clustered FL, CAM trains a global model \(\Theta_{g}\) on top of the \(K\) clusters' models \(\Theta_{1:K}\). Its prediction for client-\(i\) combines the outputs of \(\Theta_{g}\) and the associated cluster \(c(i)\)'s model, i.e., \(y=h(x;\Theta_{g})+f(x;\Theta_{c(i)})\). This simple additive model removes the restriction of clustered FL by letting all clients share a base model \(\Theta_{g}\). It enforces \(\Theta_{1:K}\) to focus on learning the different features between clusters, hence mitigating "clustering collapse". Moreover, CAM tends to learn balanced clusters and determine the number of clusters automatically (by starting from more clusters and then zeroing out some of them), as demonstrated in Fig 1. Furthermore, CAM is less vulnerable to outliers, which can be mainly captured by \(\Theta_{1:K}\) and have less impact on the global model \(\Theta_{g}\). In addition, interactions between \(\Theta_{1:K}\) and \(\Theta_{g}\) make CAM less sensitive to initial cluster assignments since updating \(\Theta_{g}\) also changes the clustering.

Figure 1: Cluster sizes during IFCA vs. IFCA+CAM in client/cluster-wise non-IID settings on CIFAR-10. Legend: cluster ID (cluster size) in the last round. **CAM effectively mitigates clustering collapse/imbalance.**

Figure 2: Test accuracy and macro-F1 (mean\(\pm\)std) of IFCA/FeSEM (w/o CAM) and IFCA/FeSEM (CAM) in cluster/client-wise non-IID settings on CIFAR-10 dataset. “IFCA(5)” represents IFCA with \(K=5\) clusters. **CAM consistently brings substantial improvement to IFCA/FeSEM on both metrics and in both settings.**

CAM is a general model-agnostic method that can modify any existing clustered FL methods. As examples, we apply CAM to two representative methods, i.e., IFCA [6] and FeSEM [8]. In the optimization of CAM, \(\Theta_{1:K}\) and \(\Theta_{g}\) aim to fit the residual of each other's prediction. To this end, we propose an efficiently structured FL algorithm "**Fed-CAM**", which alternates between cluster assignment (server), local training (clients), and update of \(\Theta_{1:K}\) and \(\Theta_{g}\) (server). In experiments on several benchmarks in different non-IID settings, CAM significantly improves SOTA clustered FL methods, as shown in Fig 2. Moreover, we provide a convergence analysis of Fed-CAM algorithm.

## 2 Related Work

**Non-IID FL** aims to tackle statistical heterogeneity across clients. FedAvg [1] is designed for the IID setting, so it suffers from client drift and slow convergence with non-IID clients [2]. To address this challenge, FedDANE [10] proposed a federated Newton-type optimization method by adapting a method for classical distributed optimization, i.e., DANE, to the FL setting. Instead of synchronizing all clients' models to be the same global model periodically, FedProx [3] only adds a proximal term to the local training objective that discourages the local model from drifting away from the global model and thus preserves the heterogeneity. [11] applies adaptive learning rates to clients and [12] conducts attention-based adaptive weighting to aggregate clients' models. [13] studies the convergence of the FedAvg in non-IID scenarios. Recent work also studies client-wise personalized FL [14; 15; 16; 17; 18; 19; 20], which aim to address the non-IID challenge by training a personalized model per client with the help of the shared global model. Their objectives focus on training local models rather than the server-side model.

**Clustered FL** assumes that non-IID clients can be partitioned into several groups and clients in each group share a cluster-wise model. It jointly optimizes the cluster assignments and the clusters' models. K-means-based methods [8] assign clusters to clients according to their model parameters' distance. CFL [21] divides clients into two partitions based on the cosine similarity between client gradients and then checks whether a partition is congruent according to the gradient norm. IFCA [6] and HypCluster [7] assign to each client the cluster whose model achieves the minimum loss on the client's data. Few-shot clustering has been introduced to clustered FL by [22; 23]. FedP2P [24] allows communication between clients in the same cluster. [25] uses cluster-based contexts to enhance the fine-tuning of personalized FL models. [9] proposed the first cluster-wise non-IID setting and a bi-level optimization framework unifying most clustered FL methods.

**Additive modeling in FL** trains multiple models and adds their outputs together as its prediction. It was introduced to FL very recently. Federated residual learning [26] proposed an FL algorithm to train an additive model for regression tasks. [27] applies additive modeling to combining the outputs of a shared model and a local model in a partial model personalization framework. However, additive modeling has not been studied for clustered FL.

## 3 Clustered Additive Modeling (CAM)

In this section, we introduce clustered additive modeling (CAM), which combines a global model and cluster-wise model prediction in FL. CAM conducts a joint optimization of the global and cluster-wise models defined by a cluster assignment criterion. In particular, we provide two examples of CAM using different cluster assignment criteria, i.e., min-loss and K-means, which have been adopted respectively by two SOTA clustered-FL methods, i.e., IFCA and FeSEM. For each of them, we derive alternating optimization procedures (i.e., IFCA-CAM and FeSEM-CAM) that can be implemented in FL setting using two parallel threads of local model training. At the end of this section, we unify both algorithms in a structured FL algorithm Fed-CAM.

**Notations.** We assume that there are \(m\) clients and \(K\) clusters, where client-\(i\) has \(n_{i}\) examples and all clients have \(n=\sum_{i=1}^{m}n_{i}\) examples. On the server side, we have a global model \(\Theta_{g}\) and \(K\) cluster-wise models \(\Theta_{1:K}\). On the client side, we train \(m\) cluster models \(\Theta_{1:m}^{0}\) used to update the global model \(\Theta_{g}\) in FL and \(\theta_{1:m}\) used to update the cluster-wise model \(\Theta_{c(i)}\) assigned to each client-\(i\), where \(c(i)\) is its cluster label determined by the cluster assignment criterion \(c(\cdot)\). We further define \(C_{k}\triangleq\{i\in[m]:c(i)=k\}\) as the set of clients in cluster-\(k\). For simplicity, we will use \(X_{i}\) and \(Y_{i}\) to respectively represent the local training data on client-\(i\) and their ground truths, and \(\ell(Y_{i},F(X_{i}))\) denotes the batch loss of model \(F(\cdot)\) on \((X_{i},Y_{i})\). A CAM model for client-\(i\) can be

\[F_{i}(\cdot)=h(\cdot;\Theta_{g})+f(\cdot;\Theta_{c(i)})). \tag{1}\]For classification, \(F_{i}(\cdot)\) produces logits and we can apply softmax to get the class probabilities.

### IFCA-CAM: model performance-driven clustering

We extend the min-loss criterion used in IFCA [6] to CAM for cluster assignment, i.e., each client-\(i\) is assigned to the cluster-\(k\) whose model \(\Theta_{k}\) leads to the minimal loss of CAM on client-\(i\)'s data, i.e.,

\[c(i)=\arg\min_{k\in[K]}\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i};\Theta_{k})). \tag{2}\]

IFCA-CAM optimizes \(\Theta_{g}\) and \(\Theta_{1:K}\) for minimizing the above minimal loss over all the \(m\) clients, i.e.,

\[\text{IFCA-CAM: }\min_{\Theta_{g},\Theta_{1:K}}\sum_{i=1}^{m}\frac{n_{i}}{n} \min_{k\in[K]}\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i};\Theta_{k})), \tag{3}\]

where the inner minimization performs the min-loss assignment in Eq. (2). We solve Eq. (3) by the following alternating minimization of cluster membership, cluster-wise models, and the global model.

**(i)** Cluster assignment by applying Eq. (2) to the latest \(\Theta_{g}\) and \(\Theta_{1:K}\). This yields \(c(\cdot)\) and \(C_{1:K}\).

**(ii)** Fixing \(\Theta_{g}\), we can optimize the cluster-wise models \(\Theta_{1:K}\) by gradient descent:

\[\Theta_{k}\leftarrow\Theta_{k}-\eta\sum_{i\in C_{k}}\frac{n_{i}}{n}\nabla_{ \Theta_{k}}\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i};\Theta_{k})),\;\forall\;k\in [K]. \tag{4}\]

In FL, the gradient can be approximated by aggregating the model updates of local models \(\theta_{i}\) from clients, whose training on the client side is: (1) initializing \(\theta_{i}\leftarrow\Theta_{c(i)}\); (2) starting from the initialization, running \(E\) local epochs updating \(\theta_{i}\) by

\[\theta_{i}\leftarrow\theta_{i}-\eta\nabla_{\theta_{i}}\ell(Y_{i},h(X_{i}; \Theta_{g})+f(X_{i};\theta_{i})),\;\forall\;i\in[m]; \tag{5}\]

and (3) aggregating the local model update \(\theta_{i}-\Theta_{k}\) from client \(i\in C_{k}\) to update \(\Theta_{k}\), i.e.,

\[\Theta_{k}\leftarrow\left(1-\sum_{i\in C_{k}}\frac{n_{i}}{n}\right)\Theta_{k} +\sum_{i\in C_{k}}\frac{n_{i}}{n}\theta_{i}. \tag{6}\]

**(iii)** Fixing \(\Theta_{1:K}\), we can optimize the global model \(\Theta_{g}\) by gradient descent:

\[\Theta_{g}\leftarrow\Theta_{g}-\eta\sum_{i\in[m]}\frac{n_{i}}{n}\nabla_{\Theta _{g}}\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i};\Theta_{c(i)})). \tag{7}\]

In FL, this gradient step can be approximated by aggregating the local models \(\theta_{i}^{0}\) (similar to FedAvg): (1) initializing \(\theta_{i}^{0}\leftarrow\Theta_{g}\); (2) running \(E\) local epochs training \(\theta_{i}^{0}\) by

\[\theta_{i}^{0}\leftarrow\theta_{i}^{0}-\eta\nabla_{\theta_{i}^{0}}\ell(Y_{i}, h(X_{i};\theta_{i}^{0})+f(X_{i};\Theta_{c(i)})),\;\forall\;i\in[m]; \tag{8}\]

and (3) aggregating the updated local models \(\theta_{i}^{0}\) of all the \(m\) clients to update \(\Theta_{g}\), i.e.,

\[\Theta_{g}\leftarrow\sum_{i\in[m]}\frac{n_{i}}{n}\theta_{i}^{0}. \tag{9}\]

We can run two parallel threads of local training for \(\theta_{i}^{0}\) and \(\theta_{i}\) for each client-\(i\) because their training in Eq. (8) and Eq. (5) does not depend on each other (but they both depend on the cluster assignments in (i)). This is analogous to the simultaneous update algorithm (FedSim) in [27]. One may also consider an alternative update algorithm (which may enjoy a slightly faster convergence) that iterates (i)\(\rightarrow\)(ii)\(\rightarrow\)(i)\(\rightarrow\)(iii) in each round. However, it doubles the communication rounds ((i) requires one communication round) and does not allow parallel local training. Since the alternative update does not show a significant empirical improvement over FedSim in [27], we mainly focus on the parallel one in the remainder of this paper.

### FeSEM-CAM: parameter similarity-based clustering

We follow a similar procedure of IFCA-CAM to derive FeSEM-CAM, which applies a K-means style clustering to the client models \(\theta_{1:m}\), whose objective is minimizing the sum of squares of client-cluster distance, i.e.,

\[\min_{\Theta_{1:K}}\sum_{i=1}^{m}\frac{n_{i}}{n}\min_{j\in[K]}\| \theta_{i}-\Theta_{j}\|_{2}^{2}. \tag{10}\]

Hence, similar to FeSEM [8], FeSEM-CAM assigns the nearest cluster-wise model to each client and updates the cluster-wise models as the cluster centroids (i.e., K-means algorithm), i.e.,

\[c(i)=\arg\min_{k\in[K]}\|\theta_{i}-\Theta_{k}\|_{2}^{2},\ \ \Theta_{k} \leftarrow\sum_{i\in C_{k}}\frac{n_{i}}{\sum_{i\in C_{k}}n_{i}}\theta_{i}. \tag{11}\]

We iterate the above K-means steps for a few times until convergence in practice. FeSEM-CAM applies the K-means objective in Eq. (10) as a regularization to the loss of CAM model \(\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i};\theta_{i}))\), i.e.,

\[\text{FeSEM-CAM:}\ \ \min_{\Theta_{g},\Theta_{1:K},\theta_{1:m}} \sum_{i=1}^{m}\frac{n_{i}}{n}\left[\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i}; \theta_{i}))+\frac{\lambda}{2}\min_{j\in[K]}\|\theta_{i}-\Theta_{j}\|_{2}^{2} \right], \tag{12}\]

where the minimization w.r.t. \(\Theta_{1:K}\) (with \(\theta_{1:m}\) fixed) recovers the (weighted) K-means objective in Eq. (10). Unlike IFCA-CAM, where client model \(\theta_{i}\) is an auxiliary/latent variable for FL not showing in the objective of Eq. (3), it is explicitly optimized in Eq. (12). Similar to IFCA-CAM, we solve Eq. (3) by iterating the following alternating minimization steps (i)-(iii).

**(i)** K-means clustering that iterates Eq. (11) for a few steps until convergence, which yields \(c(\cdot)\), \(C_{1:K}\), and \(\Theta_{1:K}\). The update of \(\Theta_{1:K}\) is analogous to Eq. (6).

**(ii)** Fixing \(\Theta_{g}\), we optimize \(\theta_{1:m}\) by client-side local gradient descent:

\[\theta_{i}\leftarrow(1-\eta\lambda)\theta_{i}+\eta\lambda\Theta_ {c(i)}-\eta\frac{n_{i}}{n}\nabla_{\theta_{i}}\ell(Y_{i},h(X_{i};\Theta_{g})+f (X_{i};\theta_{i})),\ \forall\ i\in[m]. \tag{13}\]

The first two terms in Eq. (13) compute a linear interpolation between \(\theta_{i}\) and its assigned cluster's model \(\Theta_{c(i)}\). This is a result of the K-means regularization term in Eq. (12) and keeps \(\theta_{i}\) close to \(\Theta_{c(i)}\).

**(iii)** Fixing \(\theta_{1:m}\), we can optimize \(\Theta_{g}\) by gradient descent:

\[\Theta_{g}\leftarrow\Theta_{g}-\eta\sum_{i\in[m]}\frac{n_{i}}{n} \nabla_{\Theta_{g}}\ell(Y_{i},h(X_{i};\Theta_{g})+f(X_{i};\theta_{i})). \tag{14}\]

In FL, this gradient step can be approximated by aggregating the local models \(\theta_{i}^{0}\) (similar to FedAvg): (1) initializing \(\theta_{i}^{0}\leftarrow\Theta_{g}\); (2) running \(E\) local epochs training \(\theta_{i}^{0}\) by

\[\theta_{i}^{0}\leftarrow\theta_{i}^{0}-\eta\nabla_{\theta_{i}^{0} }\ell(Y_{i},h(X_{i};\theta_{i}^{0})+f(X_{i};\theta_{i})),\ \forall\ i\in[m]; \tag{15}\]

and (3) aggregating the updated local models \(\theta_{i}^{0}\) of all the \(m\) clients to update \(\Theta_{g}\) by Eq. (9).

### Algorithm

In Algorithm 1, we propose a structured FL algorithm for CAM, i.e., Fed-CAM, which can unify the derived optimization procedures for IFCA-CAM and FeSEM-CAM and can be easily extended to other clustered FL and clustering criteria.

**Warmup.** As an alternating optimization framework, it would be unstable if both \(\Theta_{g}\) and \(\Theta_{1:K}\) are randomly initialized and jointly optimized in parallel since they may capture overlapping information and result in an inefficient competitive game. To encourage them to learn complementary knowledge, warmup training for one of them before the joint optimization is helpful. For example, a few rounds of FedAvg can produce a "warm" \(\Theta_{g}\), whose predictions' residuals are more informative to train \(\Theta_{1:K}\). Another warmup strategy could be to run a few local training epochs and extract warm \(\Theta_{1:K}\) by clustering the lightly-trained local models \(\theta_{1:m}\). In Fed-CAM, we can apply the former warmup to IFCA-CAM and the latter to FeSEM-CAM.

## 4 Convergence Analysis

Based on the convergence analysis presented in [27], which aims to minimize the following objective:

\[\min_{u,V}F(u,V):=\frac{1}{n}\sum_{i=1}^{m}F_{i}(u,v_{i}), \tag{16}\]

where \(u\) represents the shared parameters and \(V=v_{1},v_{2},\cdots,v_{m}\) denotes the personalized parameters. If we map \(\Theta_{g}\) to \(u\), and \(\Theta_{1:K}\) to \(V\) respectively, this appears strikingly similar to our methods as illustrated in Equations 3 and 12. Provided that the clustering remains stable, we can employ the theoretical framework of [27]. And firstly, we make some standard assumptions for the convergence analysis as below.

**Assumption 1**.: (Smoothness). For \(i=1,\cdots,m\), the loss function \(l\) is continuously differentiable, and there exist constants \(L\) that \(\nabla_{\Theta_{g}}\ell(\Theta_{g},\Theta_{k})\) is L-Lipschitz with respect to \(\Theta_{g}\) and \(\Theta_{k}\), and \(\nabla_{\Theta_{k}}\ell(\Theta_{g},\Theta_{k})\) is L-Lipschitz with respect to \(\Theta_{g}\) and \(\Theta_{k}\).

**Assumption 2**.: (Unbiased gradients and bounded variance). The stochastic gradients are unbiased and have bounded variance. For all \(\Theta_{g}\) and \(\Theta_{k}\),

\[\mathbb{E}[\widetilde{\nabla}_{\Theta_{g}}\ell(\Theta_{g},\Theta_{k})]=\nabla _{\Theta_{g}}\ell(\Theta_{g},\Theta_{k}),\ \ \mathbb{E}[\widetilde{\nabla}_{ \Theta_{k}}\ell(\Theta_{g},\Theta_{k})]=\nabla_{\Theta_{k}}\ell(\Theta_{g}, \Theta_{k}),\]

and

\[\mathbb{E}[\|\widetilde{\nabla}_{\Theta_{g}}\ell(\Theta_{g},\Theta_{k})-\nabla _{\Theta_{g}}\ell(\Theta_{g},\Theta_{k})\|^{2}]\leq\sigma_{g}^{2},\ \ \mathbb{E}[\| \widetilde{\nabla}_{\Theta_{k}}\ell(\Theta_{g},\Theta_{k})-\nabla_{\Theta_{k} }\ell(\Theta_{g},\Theta_{k})\|^{2}]\leq\sigma_{k}^{2}.\]

**Assumption 3**.: (Partial gradient diversity). There exists a constant for all \(\theta_{i}^{0}\) and \(\Theta_{g}\), \(\theta_{i}\) and \(\Theta_{k}\),

\[\sum_{i=1}^{m}\frac{n_{i}}{n}\|\nabla_{\Theta_{g}}\ell(\Theta_{g },\theta_{i})-\nabla_{\Theta_{g}}\ell(\Theta_{g},\Theta_{k})\|^{2}\leq\delta^{2}\] \[\sum_{i\in[k]}\frac{n_{i}}{\sum_{i\in[k]}n_{i}}\|\nabla_{\Theta_ {k}}\ell(\theta_{i}^{0},\Theta_{k})-\nabla_{\Theta_{k}}\ell(\Theta_{g},\Theta _{k})\|^{2}\leq\delta^{2}.\]

**Assumption 4**.: (Convexity of cluster models). Fix \(\Theta_{g}\), assume \(\ell(\Theta_{k})\) is convex.

**Theorem 1**.: _(Convergence of Fed-CAM). Let Assumptions 1, 2, 3 and 4 hold, and learning rates chosen as \(\eta=\tau/(LE)\) for a \(\tau\) depending on the parameters \(L,\sigma_{g}^{2},\sigma_{k}^{2},\delta^{2},s,m,T\), provided clustering stable, we have (ignoring absolute constants),_

\[\frac{1}{T}\sum_{t=1}^{T}(\frac{1}{L}\mathbb{E}[\|\nabla_{\Theta _{g}}\ell(\Theta_{g},\Theta_{k})\|^{2}]+\frac{s}{mL}\frac{1}{m}\sum_{i=1}^{m} \mathbb{E}[\|\nabla_{\Theta_{c(i)}}\ell(\Theta_{g},\Theta_{c(i)})\|^{2}]) \tag{17}\] \[\leq \frac{(\triangle\iota\sigma_{sim,1}^{2})^{1/2}}{T^{1/2}}+\frac{( \triangle_{\iota}^{2}\sigma_{sim,2}^{2})^{1/3}}{T^{2/3}}+O(\frac{1}{T}), \tag{18}\]_where \(\triangle_{\ell}=\ell_{0}-\ell^{*}\), and we define effective variance terms,_

\[\sigma^{2}_{sim,1} =\frac{2}{L}(\delta^{2}(1-\frac{s}{m})+\frac{\sigma^{2}_{g}}{L}+ \frac{\sigma^{2}_{k}s}{m})) \tag{19}\] \[\sigma^{2}_{sim,2} =\frac{2}{L}(\delta^{2}+\sigma^{2}_{g}+\sigma^{2}_{k})(1-\frac{1 }{E}). \tag{20}\]

_Remark 1_.: It is straightforward to prove that the clustering of both IFCA-CAM and FeSEM-CAM converges, as evidenced by Ma et al. (2022). However, proving the stability of these clustering methods is more challenging due to the oscillation phenomenon often seen in K-means. The stability of clustering will be further demonstrated through experimental analysis in Section 5.2.

_Remark 2_.: Besides the clustering structure, there is a distinct difference between FedSim [27] and Fed-CAM. In Fed-CAM, we need to aggregate both \(\Theta_{g}\) and \(\Theta_{1:K}\), while in FedAlt [27], only \(\Theta_{g}\) requires aggregation. The \(\sigma^{2}_{sim,1}\) and \(\sigma^{2}_{sim,2}\) reflect the impact of sample number \(s\) and local steps \(E\). Larger \(s\) or smaller \(E\), better convergence rate. According to the results presented in [27], alternative gradient descent surpasses simultaneous gradient descent in terms of convergence rate. The asymptotic \(1/\sqrt{T}\) rate is achieved when each device is seen at least once on average, and the \(1/T\) term is dominated by the \(1/\sqrt{T}\) term, a situation that occurs when (ignoring absolute constants)

\[T\geq\frac{\triangle_{\ell}}{\sigma^{2}_{sim,1}}\max\{(1-\frac{1}{E})\frac{m}{ s},2\}.\]

## 5 Experiments

**Benchmark datasets and partitions.** The proposed methods have been validated using several benchmark datasets. While detailed results for PathMNIST and TissueMNIST from the MedMNIST [28] are provided in the supplementary material, the other datasets used for validation include:

* **Fashion-MNIST**[29] includes 70,000 labeled fashion images (28\(\times\)28 grayscale) in 10 classes, such as T-shirts, Trouser, and Bag, with others.
* **CIFAR-10**[30] consists of 60,000 images (32\(\times\)32 color) in 10 classes, including airplane, automobile, bird, and truck, among others. The divergence among classes in CIFAR-10 is relatively higher than in other datasets from the MNIST family.

Each dataset is split among 200 clients and we create the following non-IID scenarios:

* **Client-wise non-IID by Dirichlet distribution (\(\alpha=0.1\))**: This approach uses the Dirichlet distribution to control the randomness of non-IID data, as proposed by [31]. This is a standard method used in most personalized FL methods, which are usually client-wise non-IID.
* **Cluster-wise non-IID by Dirichlet distribution (\(\alpha=(0.1,10)\))**: This strategy divides the dataset into \(K\) clusters with \(\alpha=0.1\) to generate substantial variance in cluster-wise non-IID. Then, each cluster is divided into \(m/K\) clients with \(\alpha=10\) to control the non-IID across clients.
* **Client-wise non-IID by n-class (2)**: This method randomly selects \(n\)-class out of all classes in the dataset for each client, as proposed by [1], and then samples instances from these datasets.
* **Cluster-wise non-IID by n-class (3, 2)**: This approach randomly assigns \(3\) classes to each cluster, ensuring a relatively balanced number of instances per class. It then assigns \(2\) classes to each client.

**Baselines.** We select baseline methods from four categories as follows:

* **Single model-based FL:** We choose FedAvg [1] and FedProx [3] with a coefficient of 230 and a regularization of 0.95 as the baselines.
* **Ensemble FL:** We train FedAvg and FedProx \(K\) times and then learn an ensemble model via soft voting to serve all clients, which are named FedAvg+ and FedProx+, respectively.
* **Clustered FL:** We choose FeSEM [8] and IFCA [6], which is similar to HypCluster [7].
* **Clustered FL with additive modeling:** We integrate CAM with IFCA and FeSEM, denoting them as IFCA-CAM and FeSEM-CAM, respectively.

**Learning-related hyperparameters.** We use the Convolutional Neural Network (CNN) [32] as the basic model architecture for each client, as detailed in the supplementary material. For optimization, we employ SGD with a learning rate of 0.001 and momentum of 0.9 to train the model, and the batch size is 32. We evaluate the performance using both **micro accuracy** (%) and **macro F1-score** (%) on the client-wise test datasets to better capture the non-IID nature per client.

**FL system settings.** We conduct 100 global communication rounds in the FL system, including 30 warm-up rounds if applicable. Each communication involves 10 local steps. For the clustering process of FeSEM-CAM, we measure distance on the flattened parameters of the fully-connected layers, and use K-Means as the clustering algorithm. The coefficient \(\lambda\) is chosen from \(0.001,0.01,0.1\) based on the best performance.

### Main Results and Comparisons

**Cluster-wise non-IID** scenarios make the assumption that there are underlying clustering structures among clients. Table 1 compares the methods using two benchmark datasets, namely Fashion-MNIST and CIFAR-10. Results using two biomedical datasets are presented in the appendix. The following are some notable observations and analyses:

* The application of the ensemble mechanism to FedAvg and FedProx yields minor improvements. This is because the server-side model in FedAvg or FedProx is already a relatively strong model, while ensemble mechanisms usually excel with weak models.
* The introduction of CAM significantly enhances the performance of IFCA, which typically struggles with clustering collapse in cluster-wise non-IID scenarios. Notably, CAM decomposes the shared components into a global model and personalized parts into cluster models. Thus, the clustering collapse is mitigated by isolating the dominant shared knowledge.
* FeSEM generally exhibits robust performance on cluster-wise non-IID without outliers. Implementing CAM in FeSEM further improves the Macro-F1 performance. The clustering process in FeSEM tends to overfit the label distribution (imbalanced classes) of clients to achieve higher accuracy. However, the application of CAM introduces a global model with a balanced label distribution by averaging all clients, thereby boosting the Macro-F1 performance while preserving the cluster-wise non-IID for high accuracy.
* With an increase in the number of clusters \(K\), the CAM-based methods show substantial improvements in Macro-F1. The decomposition of shared knowledge and cluster-wise non-IID characteristics benefit from a reasonably larger \(K\), which facilitates fine-grained, cluster-wise personalization.

**Client-wise non-IID** Table 2 presents comparative results under client-wise non-IID scenarios using two benchmark datasets: Fashion-MNIST and CIFAR-10. Interestingly, IFCA maintains stable performance under client-wise non-IID conditions, primarily because it cannot form a single dominant cluster model - a primary cause of clustering collapse - in a highly heterogeneous environment. The application of CAM to IFCA and FeSEM shows a significant enhancement, particularly on the CIFAR-10 dataset. This improvement is likely due to FeSEM's typical restriction on knowledge sharing across clusters. In contrast, CAM utilizes a global model to capture more useful common

\begin{table}
\begin{tabular}{c c c|c c c c|c c c c} \hline \hline \multicolumn{2}{c}{} & \multicolumn{4}{c}{Fashion-MNIST} & \multicolumn{4}{c}{CIFAR-10} \\ \hline \multicolumn{2}{c}{} & \multicolumn{2}{c}{Non-IID setting} & \multicolumn{2}{c}{Dirichlet \(\alpha=(0.1,10)\)} & \multicolumn{2}{c}{n-class \((3,2)\)} & \multicolumn{2}{c}{Dirichlet \(\alpha=(0.1,10)\)} & \multicolumn{2}{c}{n-class \((3,2)\)} \\ \hline \multicolumn{2}{c}{**\#Cluster**} & \multicolumn{2}{c}{Methods} & \multicolumn{1}{c}{Accuracy} & \multicolumn{1}{c}{Macro-F1} & \multicolumn{1}{c}{Accuracy} & \multicolumn{1}{c}{Macro-F1} & \multicolumn{1}{c}{Accuracy} & \multicolumn{1}{c}{Macro-F1} & \multicolumn{1}{c}{Accuracy} & \multicolumn{1}{c}{Macro-F1} \\ \hline \multirow{3}{*}{**1**} & FedAvg & 86.08\(\pm\)0.70 & 57.24\(\pm\)2.26 & 86.33\(\pm\)0.44 & 46.09\(\pm\)1.08 & 24.38\(\pm\)3.30 & 11.69\(\pm\)3.15 & 21.33\(\pm\)3.83 & 9.00\(\pm\)0.58 \\  & FedProx & 86.32\(\pm\)0.70 & 58.03\(\pm\)0.38 & 86.42\(\pm\)0.38 & 45.86\(\pm\)1.42 & 24.73\(\pm\)3.68 & 11.28\(\pm\)2.35 & 22.66\(\pm\)1.33 & 9.23\(\pm\)0.78 \\ \hline \multirow{6}{*}{**5**} & FedAvg+ & 87.61 & 9.48 & 86.95 & 65.61 & 25.97 & 12.16 & 24.35 & 9.66 \\  & FedProx+ & 87.94 & 59.83 & 86.52 & 65.73 & 26.05 & 12.53 & 24.83 & 9.31 \\  & ICA & 84.60\(\pm\)2.22 & 62.03\(\pm\)3.01 & 84.91\(\pm\)2.54 & 66.90\(\pm\)4.43 & 34.10\(\pm\)1.79 & 22.12\(\pm\)21.21 & 29.80\(\pm\)4.49 & 17.90\(\pm\)0.26 \\  & IFCA-CAM & 93.33\(\pm\)0.95 & 79.64\(\pm\)4.09 & 95.83\(\pm\)0.49 & 77.56\(\pm\)1.14 & 58.13\(\pm\)3.82 & 28.09\(\pm\)3.68 & 54.56\(\pm\)3.58 & 27.27\(\pm\)1.06 \\  & FeSEM & 94.61\(\pm\)5.14 & 82.90\(\pm\)2.08 & 84.20\(\pm\)1.96 & 77.70\(\pm\)6.05 & 90.90\(\pm\)2.34 & 23.73\(\pm\)25.85 & 33.75\(\pm\)2.54 \\  & FeSEM-CAM & **95.13\(\pm\)1.18** & **85.10\(\pm\)3.17** & **95.60\(\pm\)1.08** & **75.82\(\pm\)1.47** & **64.83\(\pm\)2.33** & **30.83\(\pm\)1.77** & **65.88\(\pm\)1.21** & **36.83\(\pm\)1.17** \\ \hline \multirow{6}{*}{**10**} & FedAvg+ & 89.42 & 67.83 & 86.91 & 63.01 & 63.04 & 28.33 & 13.97 & 27.28 & 9.81 \\  & FedProx+ & 89.85 & 68.02 & 86.73 & 63.42 & 28.33 & 13.64 & 2.664 & 9.64 \\  & ICA & 82.10\(\pm\)0.45 & 62.62\(\pm\)8.28 & 56.58\(\pm\)4.97 & 28.98\(\pm\)1.16 & 27.89\(\pm\)3.99 & 34.66\(\pm\)2.60 & 18.70\(\pm\)1.31 \\  & IRCA-CAM & 94.52\(\pm\)2.54 & 88.45\(\pm\)5.46 & 95.00\(\pm\)0.87 & 82.89\(\pm\)1.16 & 70.90\(\pm\)1.18 & 40.88\(\pm\)1.28 & 68.46\(\pm\)4.08 & 41.45\(\pm\)4.00 \\  & FeSEM & 95.73\(\pm\)1.28 & 89.34\(\pm\)1.57 & 95.54\(\pm\)0.74 & 84.81\(\pm\)3.28 & 66.69\(\pm\)2.18 & 38.35\(\pm\)4.17 & 71.62\(\pm\)1.23 & 49.72\(\pm\)3.34 \\  & FeSEM-CAM & **96.19\(\pm\)1.20** & **92.37\(\pm\)1.85** & **98.07\(\pm\)1.46** & **92.43\(\pm\)2.70** & **78.45\(\pm\)1.71** & **49.50\(\pm\)1.13** & **75.04\(\pm\)1.17** & **55.50\(\pm\)2.07** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Test results (mean\(\pm\)std) in **cluster**-wise non-IID settings on Fashion-MNIST & CIFAR-10.

knowledge across clusters, thereby substantially enhancing the generalization capability of each cluster. Furthermore, CIFAR-10, being a relatively complex dataset with a diversity of images, underscores the importance of sharing common knowledge.

### Visualization: CAM combats clustering collapse

Figures 1 and 3 demonstrate the effectiveness of applying CAM to mitigate clustering collapse in IFCA and FeSEM under both cluster-wise and client-wise non-IID scenarios, using the CIFAR-10 dataset with \(K=10\). Each color represents a cluster, and the X-axis represents the iteration rounds.

In the case of IFCA, we observe a severe clustering collapse issue in cluster-wise non-IID scenarios. A single cluster can encompass all clients in the client-wise non-IID setting and up to \(50\%\) of clients in the cluster non-IID setting. Furthermore, the clustering remains unstable throughout the process. However, when CAM is applied in IFCA-CAM, it quickly identifies some clustering structures within a few rounds, and this structure closely approximates the ground truth.

As for FeSEM, while the phenomenon of clustering collapse is not as pronounced, a single cluster can still dominate up to \(25\%\) of all clients if there are no outliers. CAM can expedite the clustering convergence, sometimes achieving it in just one round. Moreover, under client-wise non-IID settings, the application of CAM results in lower variance and more uniform cluster size. In the case of cluster-wise non-IID settings, FeSEM-CAM can easily identify the ground truth.

## 6 Conclusions

We propose a novel structured FL model "clustered additive modeling (CAM)" and an efficient FL algorithmic framework Fed-CAM to address non-IID FL challenges with clustering structure. CAM is a general mode-agnostic tool that can improve various existing non-IID FL methods. It can capture more general non-IID structures with global knowledge sharing among clients than clustered FL and overcome several weaknesses such as clustering collapse, vulnerability to cluster imbalance/initialization, etc. Theoretically, Fed-CAM is capable of achieving an asymptotic convergence rate of \(O(1/\sqrt{T})\). Extensive experiments show that CAM brings substantial improvement to existing clustered FL methods, improves cluster balance, and effectively mitigates clustering collapse.

\begin{table}
\begin{tabular}{c c|c c c c c c c c} \hline \hline \multicolumn{3}{c}{Datasets} & \multicolumn{4}{c}{CIFAR-10} \\ \hline \multicolumn{3}{c}{No-IID setting} & \multicolumn{4}{c}{Dritic \(\alpha=0.1\)} & \multicolumn{4}{c}{p-scales (2)} & \multicolumn{4}{c}{Dritic \(\alpha=0.1\)} & \multicolumn{4}{c}{n-class (2)} \\ \hline \multirow{3}{*}{**\#Cluster**} & Methods & Accuracy & More-Fl & Accuracy & More-Fl & Accuracy & More-Fl & Accuracy & More-Fl \\ \cline{2-11}  & FedAvg & 85.90\(\pm\)0.46 & 54.52\(\pm\)2.66 & 86.17\(\pm\)0.25 & 44.48\(\pm\)1.24 & 25.62\(\pm\)3.47 & 11.38\(\pm\)2.02 & 24.30\(\pm\)3.53 & 8.56\(\pm\)0.64 \\ \cline{2-11}  & FedPox & 86.03\(\pm\)0.58 & 54.69\(\pm\)3.32 & 86.47\(\pm\)0.23 & 44.89\(\pm\)1.38 & 25.72\(\pm\)3.29 & 11.14\(\pm\)1.49 & 24.19\(\pm\)2.45 & 8.69\(\pm\)0.74 \\ \hline \multirow{6}{*}{**5**} & FedAvg* & 86.12 & 61.07 & 86.55 & 45.39 & 25.71 & 12.45 & 24.83 & 8.74 \\  & FedAvg* & 86.39 & 56.56 & 86.15 & 45.33 & 25.88 & 12.43 & 25.88 & 8.55 \\  & IPCA & 90.13\(\pm\)61 & 68.47\(\pm\)5.23 & 9.41\(\pm\)5.44 & 72.30\(\pm\)5.32 & 42.71\(\pm\)0.28 & 20.28 & 26.77\(\pm\)1.48 & 45.64\(\pm\)12.78 & 17.78\(\pm\)1.29 \\  & IPCA-CAM & 97.32\(\pm\)14 & 90.71\(\pm\)5.17 & 92.24\(\pm\)1.02 & 22.24\(\pm\)3.43 & 54.21\(\pm\)2.55 & 41.88\(\pm\)1.54 & 59.24\(\pm\)1.51 & 25.20\(\pm\)1.05 \\  & FeSEM & 91.51\(\pm\)29 & 73.78\(\pm\)9.88 & 91.83\(\pm\)1.34 & 71.05\(\pm\)6.83 & 54.30\(\pm\)4.58 & 24.78\(\pm\)6.01 & 55.55\(\pm\)4.83 & 33.80\(\pm\)4.18 \\  & FeSEM+CAM & **97.44\(\pm\)1.04** & **75.12\(\pm\)15.82** & **93.14\(\pm\)2.60** & **76.98\(\pm\)21.77** & **59.71\(\pm\)28.09** & **40.45\(\pm\)33.35** & **56.70\(\pm\)1.46** & **34.52\(\pm\)1.44** \\ \hline \multirow{6}{*}{**10**} & FedAvg* & 86.81 & 60.43 & 86.91 & 47.12 & 27.83 & 13.65 & 27.71 & 9.65 \\  & FedPox* & 86.34 & 56.2 & 86.78 & 42.83 & 25.86 & 12.84 & 26.16 & 9.94 \\  & IPCA & 91.04\(\pm\)4.33 & 68.66\(\pm\)67.47 & 92.45\(\pm\)16.79 & 24.56 & 72.98\(\pm\)5.80 & 47.62\(\pm\)10.15 & 23.36\(\pm\)24.87 & 47.96\(\pm\)10.19 & 17.88\(\pm\)1.04 \\  & IPCA-CAM & **95.70\(\pm\)19** & 79.71\(\pm\)1.91 & 92.75\(\pm\)2.63 & 76.31\(\pm\)4.39 & 72.54\(\pm\)7.27 & 42.86\(\pm\)4.36 & 61.01\(\pm\)2.41 & 31.63\(\pm\)2.7 \\  & FeSEM & 93.32\(\pm\)30 & 80.41\(\pm\)10.16 & 93.75\(\pm\)1.39 & 73.96\(\pm\)5.67 & 67.1\(\pm\)5.7 & 31.96\(\pm\)5.23 & 63.61\(\pm\)6.51 & 47.29\(\pm\)6.08 \\  & FeSEM-CAM & 95.25\(\pm\)19 & 81.55\(\pm\)22.24 & **95.15\(\pm\)1.48** & **86.16\(\pm\)3.19** & **80.11\(\pm\)1.82** & **59.19\(\pm\)4.67** & **69.88\(\pm\)1.77** & **49.5\(\pm\)1.42** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Test results (mean\(\pm\)std) in **client**-wise non-IID settings on Fashion-MNIST & CIFAR-10.

Figure 3: Cluster sizes during FeSEM vs. FeSEM+CAM in client/cluster-wise non-IID settings on CIFAR-10. Legend: cluster ID (cluster size) in the last round. **CAM effectively mitigates clustering collapse/imbalance**.

## References

* [1]B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. Cited by: SS1.
* [2]P. Kairouz, H. D. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. (2021) Advances and open problems in federated learning. Foundations and Trends(r) in Machine Learning14 (1-2), pp. 1-210. Cited by: SS1.
* [3]T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2020) Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems2, pp. 429-450. Cited by: SS1.
* [4]H. Zhu, J. Xu, S. Liu, and Y. Jin (2021) Federated learning on non-iid data: a survey. arXiv preprint arXiv:2106.06843. Cited by: SS1.
* [5]D. Gao, X. Yao, and Q. Yang (2022) A survey on heterogeneous federated learning. arXiv preprint arXiv:2210.04505. Cited by: SS1.
* [6]A. Ghosh, J. Chung, D. Yin, and K. Ramchandran (2020) An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems33, pp. 19586-19597. Cited by: SS1.
* [7]Y. Mansour, M. Mohri, J. Ro, and A. Suresh (2020) Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619. Cited by: SS1.
* [8]M. Xie, G. Long, T. Shen, T. Zhou, X. Wang, J. Jiang, and C. Zhang (2021) Multi-center federated learning. arXiv preprint arXiv:2108.08647. Cited by: SS1.
* [9]J. Ma, G. Long, T. Zhou, J. Jiang, and C. Zhang (2022) On the convergence of clustered federated learning. arXiv preprint arXiv:2202.06187. Cited by: SS1.
* [10]T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2019) FedDane: a federated newton-type method. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pp. 1227-1231. Cited by: SS1.
* [11]S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konecny, S. Kumar, and H. Brendan (2020) Adaptive federated optimization. arXiv preprint arXiv:2003.00295. Cited by: SS1.
* [12]J. Jiang, S. Ji, and G. Long (2020) Decentralized knowledge acquisition for mobile internet applications. World Wide Web23 (5), pp. 2653-2669. Cited by: SS1.
* [13]X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang (2019) On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189. Cited by: SS1.
* [14]A. Z. Tan, H. Yu, L. Cui, and Q. Yang (2022) Toward personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems. Cited by: SS1.
* [15]A. Z. Fallah, A. Mokhtari, and A. Ozdaglar (2020) Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Advances in Neural Information Processing Systems33, pp. 3557-3568. Cited by: SS1.
* [16]A. Shamsian, A. Navon, E. Fetaya, and G. Chechik (2021) Personalized federated learning using hypernetworks. In International Conference on Machine Learning, pp. 9489-9502. Cited by: SS1.
* [17]Y. Deng, M. Mahdi Kamani, and M. Mahdavi (2020) Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461. Cited by: SS1.
* [18]L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai (2021) Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pp. 2089-2099. Cited by: SS1.
* [19]C. T. Dinh, N. Tran, and J. Nguyen (2020) Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems33, pp. 21394-21405. Cited by: SS1.
* [20]K. Singhal, H. Sidahmed, Z. Garrett, S. Wu, J. Rush, and S. Prakash (2021) Federated reconstruction: partially local federated learning. Advances in Neural Information Processing Systems34, pp. 11220-11232. Cited by: SS1.
* [21]F. Sattler, K. Muller, and W. Samek (2020) Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems. Cited by: SS1.
* [22]D. Kurian Dennis, T. Li, and V. Smith (2021) Heterogeneity for the win: one-shot federated clustering. Cited by: SS1.
* [23]P. Awasthi and O. Sheffet (2012) Improved spectral-norm bounds for clustering. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, pp. 37-49. Cited by: SS1.

* [24] Li Chou, Zichang Liu, Zhuang Wang, and Anshumali Shrivastava. Efficient and less centralized federated learning. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_, pages 772-787. Springer, 2021.
* [25] Xueyang Tang, Song Guo, and Jingcai Guo. Personalized federated learning with contextualized generalization. In Lud De Raedt, editor, _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_, pages 2241-2247. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.
* [26] Alekh Agarwal, John Langford, and Chen-Yu Wei. Federated residual learning. _arXiv preprint arXiv:2003.12880_, 2020.
* [27] Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In _International Conference on Machine Learning_, pages 17716-17758. PMLR, 2022.
* [28] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. _arXiv preprint arXiv:2110.14795_, 2021.
* [29] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. _arXiv preprint arXiv:1708.07747_, 2017.
* [30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
* [31] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. _arXiv preprint arXiv:1909.06335_, 2019.
* [32] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. _nature_, 521(7553):436-444, 2015.

Response:
## Review

### Summary

This paper presents a novel approach to clustered federated learning called Clustered Additive Modeling (CAM), which addresses critical issues such as clustering collapse and sensitivity to outliers. By integrating predictions from a global model with those from cluster-specific models, CAM enhances performance across various non-IID settings. The authors demonstrate the effectiveness of their method through extensive experiments, showing significant improvements over existing clustered federated learning techniques. This contribution is positioned to have a substantial impact on the study of statistical heterogeneity and client structures in federated learning.

### Strengths

- 1. The proposed method is simple yet effective, likely to establish a new baseline in clustered federated learning.
- 2. The paper tackles the inherent challenge of clustering collapse in federated learning, providing a clear motivation and solution.
- 3. The overall flow and clarity of the paper are good, with a well-structured theoretical analysis.
- 4. The claims made are well-supported by experimental results across diverse datasets and settings.
- 5. The paper includes a comprehensive comparison with existing methods, enhancing the validity of its claims.

### Weaknesses

- 1. Readability could be improved due to the use of many abbreviations and symbols without clear definitions.
- 2. Some descriptions lack clarity, particularly regarding the linkage of figures to the text.
- 3. The proposed method requires additional model training, raising questions about computational efficiency and complexity.
- 4. The convergence analysis is limited, lacking details that could strengthen the theoretical foundation.
- 5. The paper does not adequately discuss the impact of outliers on the global model and the implications for clustering dynamics.

### Questions

- 1. How does the proposed method handle scenarios with a significant number of outliers?
- 2. What is the rationale for using different symbols for the global and cluster-specific models in the equations?
- 3. Can the authors elaborate on the cost of the warmup stage and its effect on performance?
- 4. Is there a need for additional comparisons with personalized federated learning methods?
- 5. How can the proposed method be adapted for graph data or other complex data structures?

### Soundness

**Score:** 3

**Description:** 3 = good; the methodology is sound and the theoretical foundations are mostly well-laid, though some details require further elaboration to fully support the claims.

### Presentation

**Score:** 3

**Description:** 3 = good; while the paper is generally clear and structured, improvements in readability and clarity of certain sections are needed.

### Contribution

**Score:** 3

**Description:** 3 = good; the paper introduces a significant advancement in the field, addressing a critical problem, though further validation through comparisons with other models would strengthen its contribution.

### Rating

**Score:** 7

**Description:** 7 = accept, but needs minor improvements; the paper is technically solid and impactful but requires some refinement in presentation and additional discussions regarding limitations.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper offers a novel solution to a significant problem in federated learning, demonstrating sound methodology and promising empirical results. While there are minor weaknesses in presentation and clarity, the overall contribution is valuable and impactful, warranting acceptance.