Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

More specifically, FedCVAE-KD aggregates the models using a lightweight knowledge distillation procedure; client decoders are teachers, and the server decoder is the student.

The review comment to be evaluated is:

### Strength

* The proposed method of one-shot FL is very interesting and novel.
* The proposed method doesn't require auxiliary public dataset.
* Both security and privacy extensions are considered in the paper.
* It can be applied to heterogeneous local models with similar generative capabilities.

### Weaknesses
* One potential issue of the method is training a CVAE locally. In the paper, the images used in the dataset are very small, and the amount of data each client has is large. However, in the real-world application of FL like medical imagings, with very large images and very limited number of data samples, training a CVAE locally can be difficult even just for one class of data.
* The training task of local models is different from the original task. This limits the re-use of pre-trained local model from original task. I think one of the reasons that people want to do one-shot FL is to prevent any re-training locally. But the proposed method still requires to train some new model.

The evidence to be evaluated is:Knowledge distillation (KD) aims to extract information from a trained _teacher_ model to train a separate lightweight _student_ model (Bucilua et al., 2006; Hinton et al., 2015). Typically, the student model is trained by minimizing the discrepancy between student and teacher logits generated using a suitable auxiliary dataset (Hinton et al., 2015); KL-divergence is often chosen as the measure of discrepancy.

Several works have integrated KD into FL to mitigate privacy risks, reduce upload costs, or regularize local learning using an ensemble-of-teachers approach (Lin et al., 2020; Zhu et al., 2021; Guha et al., 2019).

We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

FedCVAE-KD trains a CVAE \(f_{\mathbf{w}^{k}}(\cdot)\) for every client \(k\in\mathcal{C}\) to convergence on their private local dataset by solving Equation 1; this CVAE is parameterized by \(\mathbf{w}^{k}=[\phi^{k},\mathbf{\theta}^{k}]\), with encoder \(E_{\mathbf{\phi}^{k}}(\cdot)\) and decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\). Then, clients communicate their decoder weights \(\mathbf{\theta}^{k}\) and label distributions \(\bar{p}^{k}(\mathbf{y})\) to the server; in practice, this simply requires upload of client label counts as in Zhu et al. (2021). This completes the single communication round.

The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space to a single student server decoder, parameterized by \(\mathbf{\theta}^{S}\). To achieve this, we generate \(n_{D}\) total KD training samples, with \(\mathcal{D}_{\text{Ens}}\) defined as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k}, \mathbf{z}_{i}^{k})\}_{i=1}^{\lfloor n_{D}/\rfloor}\). Then, we train the server to match the teacher's mapping of \((\mathbf{z}_{i}^{k},\mathbf{y}_{i}^{k})\) to \(\hat{\mathbf{x}}_{i}^{k}\) by minimizing a reconstruction loss: \[\ell_{\text{KD}}(\mathbf{\theta}^{S};\mathbf{z}^{k},\mathbf{y}^{k},\hat{\mathbf{x }}^{k})=g(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k}),\hat{\mathbf{x }}^{k}), \tag{2}\] which penalizes the dissimilarity \(g(\cdot)\) in data space between the synthetic data sample generated by the server decoder \(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k})\) and the client decoder sample \(\hat{\mathbf{x}}^{k}\).

While we do not explore the extended communication setting, we note that FedCVAE-KD extends naturally by communicating the server decoder parameters obtained through KD to all clients and repeating the outlined procedure for non-terminal communication rounds.

In the case of FedCVAE-Ens and FedCVAE-KD, an attacker who intercepts all client decoders and local label distributions should not be able to generate the high quality samples necessary to train a high quality classifier.

The server decoder for FedCVAE-KD has the same architecture as the client CVAEs by default, although this is not strictly necessary because FedCVAE-KD supports heterogeneous CVAE architectures.

CVAE training uses binary cross entropy and mean squared error for the reconstruction term of the objective for grayscale and RGB images, respectively; we use the same reconstruction objective for the KD loss (\(g(\cdot)\) in Equation 2).

The proposed KD aggregation method used in FedCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure 8 in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C). As expected, FedCVAE-Ens bounds the performance of FedCVAE-KD, but KD-based aggregation still preserves client ensemble performance well across levels of \(\alpha\) for all datasets (Table 1).

The server decoder matches that which is described in Section 4.1 but can be chosen arbitrarily.

Notably, our KD procedure for FedCVAE-KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge to a single decoder.

Other works have used KD to circumvent issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary data (Lin et al., 2020; Sattler et al., 2021) or a generative model (Zhu et al., 2021; Zhang et al., 2022) to compactly capture client ensemble learning.

Guha et al. (2019) originally proposed one-shot FL and introduced two methods, one based on heuristic selection methods for client inclusion in the final ensemble and another that used KD with an auxiliary dataset for ensemble aggregation. Li et al. (2021) extended the use of KD with a hierarchical KD procedure with wide applicability to a variety of local model types.

Zhang et al. (2021) proposed a data-free KD procedure based on a generator network trained using the ensemble of client classifiers, showing promising performance even under heterogeneous local models.

In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security.

1:procedureServer 2:for each client \(k\in\mathcal{C}\) in parallel do 3:\(\mathbf{\theta}^{k},\;\hat{p}^{k}(\mathbf{y})\leftarrow\)ClientLocalUpdate(\(k\), \(T_{L}\)) 4: Generate samples from each client \(\mathcal{D}_{\text{Ens}}^{k}:=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k})\}_ {i=1}^{\lfloor nC/m\rfloor}\) using client decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\) and label distribution \(\hat{p}^{k}(\mathbf{y})\) 5: Combine client subsets into an IID labeled dataset \(\mathcal{D}_{\text{Ens}}:=\mathcal{D}_{\text{Ens}}^{1}\cup\mathcal{D}_{\text{ Ens}}^{2}\cup...\cup\mathcal{D}_{\text{Ens}}^{m}\)

While parameter averaging may seem like a reasonable approach for client decoder aggregation in FedCVAE-KD, it generates qualitatively poor samples (Figure 8) and fails to train a high-accuracy classifier (Table 6). The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input: Figure 1: Motivating our proposed methods, FEDCVAE-ENS and FEDCVAE-KD, using the MNIST dataset as an example. In cases of very high statistical heterogeneity, each client will only observe one or two of the ten available classes, as seen on the left where the size of each dot is proportional to the number of samples. For example, client 2 only observes 4’s and 7’s, resulting in a client decoder that can expertly generate these digits. Note that the columns are shown in order of conditioning class (digits 0-9). Similarly, client 4 is an expert in 3’s and 6’s. In FEDCVAE-KD, our lightweight knowledge distillation training procedure compacts local learning into a single server decoder, as evidenced by the high-quality samples from all available classes (digits 0-9). This server decoder can then be used for any downstream task, e.g., classification.

Input: Figure 2: The full pipeline for one of our proposed methods, FEDCVAE-KD. Here, E, D, and C represent “encoder,” “decoder,” and “classifier” models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses uploaded client decoder parameters and local label distributions to train a server decoder using knowledge distillation (KD). Finally, synthetic labeled samples from the server decoder are used to train a classifier.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "FedCVAE-KD aggregates the models using a lightweight knowledge distillation procedure; client decoders are teachers, and the server decoder is the student" appears to be well-stated and accurately reflects the evidence without exaggeration. The evidence thoroughly documents how client decoders act as teachers to train a student server decoder through KD, with clear explanations of the process and mathematical formulation (Equation 2). The method's effectiveness is demonstrated through comparative results showing superior performance over parameter averaging, as shown in Figure 8 and associated performance metrics.

<score>1</score>

Justification:
1. The evidence directly supports all components of the claim:
- KD procedure is clearly outlined in the methodology
- Client-teacher/server-student relationship is explicitly stated
- Results validate the effectiveness of the approach
2. The wording is precise and measured, without overstatement
3. The review comments, while noting some practical limitations, do not contradict the specific claim about the KD procedure
4. Visual evidence in Figures 1-2 illustrates the claimed architecture


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Thorough experiments on multiple benchmark datasets (MNIST, FashionMNIST, SVHN) demonstrate the superiority of FedCVAE-Ens and FedCVAE-KD over other relevant one-shot FL methods in the high statistical heterogeneity setting.

The evidence to be evaluated is:Notably, existing one-shot FL methods either ignore the issue of statistical heterogeneity (i.e., Guha et al. (2019)), fail to comprehensively explore the effect of statistical heterogeneity on performance (i.e., Shin et al. (2020) and Li et al. (2021)), or degrade substantially at even moderate levels of statistical heterogeneity (i.e., Zhou et al. (2020) and Zhang et al. (2021)). We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

To simulate statistical heterogeneity, we use the Dirichlet distribution to generate disjoint non-IID client training datasets as in Hsu et al. (2019) and Lin et al. (2020). Specifically, we sample \(\mathbf{p}^{k}\sim Dir(\alpha)\) and allocate a \(\mathbf{p}^{k}_{i}\) proportion of class \(i\) to client \(k\). The parameter \(\alpha\) controls the level of non-IID-ness, with a lower \(\alpha\) inducing more skewed label distributions across clients.

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

There are recent FL methods that are not appropriate in our proposed setting. The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

Unless otherwise stated, we use \(m=10\) clients, \(\alpha=0.01\) (very heterogeneous), and report average test accuracy across 5 seeded parameter initializations \(\pm\) one standard deviation. The data partition is fixed unless otherwise stated. Following Zhu et al. (2021), we distribute \(50\%\) of the available training data to clients for MNIST and FashionMNIST, and \(100\%\) for SVHN. All available test data is used to evaluate the final server classifier (or ensemble for FedOneShot).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the evidence provided:

The claim appears to be well-stated based on the comprehensive experimental evidence presented. The tables and figures show:

1. Systematic evaluation across three standard benchmark datasets (MNIST, FashionMNIST, SVHN)
2. Clear superiority of FedCVAE-Ens and FedCVAE-KD over baselines (FedAvg and FedOneShot) in high heterogeneity settings (α=0.001, 0.01)
3. Consistent performance across varying numbers of clients
4. Quantitative results showing significant improvements (1.75x-2.75x accuracy gains)
5. Visual evidence demonstrating better sample quality through knowledge distillation

The experimental methodology is thorough with proper statistical validation (5 seeds, standard deviations reported) and clear parameter settings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Thorough experiments on multiple benchmark datasets (MNIST, FashionMNIST, SVHN) demonstrate the superiority of FedCVAE-Ens and FedCVAE-KD over other relevant one-shot FL methods in the high statistical heterogeneity setting.

The review comment to be evaluated is:

Strength:

-- The proposed solutions seem effective under the one-shot FL setting. The efficacy and efficiency are both demonstrated in the extensive experiments.
-- The writing is clear and easy to follow. The tables and figures are well-organized.

Weaknesses:

-- Constructing a generative model to generate samples for FL model training is not new. There are a few works utilizing a similar strategy in FL, for example:
Data-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021
Fine-tuning global model via data-free knowledge distillation for non-iid federated learning, CVPR 2022
Please make a comparison with those works

-- Some details are not clear enough for me. For example, in Eq 2, what is the formulation for g(.)?  Is this dissimilarity measured in the data-space or feature space?

The evidence to be evaluated is:Notably, existing one-shot FL methods either ignore the issue of statistical heterogeneity (i.e., Guha et al. (2019)), fail to comprehensively explore the effect of statistical heterogeneity on performance (i.e., Shin et al. (2020) and Li et al. (2021)), or degrade substantially at even moderate levels of statistical heterogeneity (i.e., Zhou et al. (2020) and Zhang et al. (2021)). We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

To simulate statistical heterogeneity, we use the Dirichlet distribution to generate disjoint non-IID client training datasets as in Hsu et al. (2019) and Lin et al. (2020). Specifically, we sample \(\mathbf{p}^{k}\sim Dir(\alpha)\) and allocate a \(\mathbf{p}^{k}_{i}\) proportion of class \(i\) to client \(k\). The parameter \(\alpha\) controls the level of non-IID-ness, with a lower \(\alpha\) inducing more skewed label distributions across clients.

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

There are recent FL methods that are not appropriate in our proposed setting. The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

Unless otherwise stated, we use \(m=10\) clients, \(\alpha=0.01\) (very heterogeneous), and report average test accuracy across 5 seeded parameter initializations \(\pm\) one standard deviation. The data partition is fixed unless otherwise stated. Following Zhu et al. (2021), we distribute \(50\%\) of the available training data to clients for MNIST and FashionMNIST, and \(100\%\) for SVHN. All available test data is used to evaluate the final server classifier (or ensemble for FedOneShot).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is generally well-stated with minimal exaggeration. The evidence provides comprehensive experimental results across MNIST, FashionMNIST, and SVHN datasets, showing superior performance of FedCVAE-Ens and FedCVAE-KD over baselines (FedAvg and FedOneShot) in high statistical heterogeneity settings. The results are thoroughly documented in Tables 1 and 2 with clear statistical significance. However, the reviewer points out that similar generative model strategies exist in FL, suggesting some overstatement in claiming complete novelty. Overall, the claim is supported by robust empirical evidence while acknowledging some limitations.

<score>2</score>

Justification:
1. The evidence strongly supports superior performance:
- At α=0.001 (extreme heterogeneity), FedCVAE-KD achieves >1.75x accuracy on MNIST, >2x on FashionMNIST, and nearly 2.75x on SVHN compared to baselines
- Tables 1 and 2 demonstrate consistent outperformance across different heterogeneity levels and numbers of clients
- Results show statistical significance with reported standard deviations

2. Minor overstatement factors:
- Reviewer notes similar existing approaches using generative models in FL
- Some experimental details (like Eq 2 formulation) could be clearer
- The claim could better acknowledge these limitations

3. Strong methodological support:
- Clear experimental setup with multiple seeds and fixed data partitions
- Comprehensive evaluation across heterogeneity levels
- Visual evidence of improved sample generation (Figure 8)

The score of 2 reflects that while there is robust evidence supporting the claim's core message, there could be slightly better acknowledgment of related work and limitations.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Thorough experiments on multiple benchmark datasets (MNIST, FashionMNIST, SVHN) demonstrate the superiority of FedCVAE-Ens and FedCVAE-KD over other relevant one-shot FL methods in the high statistical heterogeneity setting.

The review comment to be evaluated is:

The paper presents a relatively simple idea and is mostly clearly written. As far as I am aware, this method is novel in the context of FL. The authors also provide a reasonable amount of information, which should make FedCVAE reproducible. The authors seem to mostly tackle the cross-silo setting, where each client has enough data. One thing I noticed is that the authors mention that the conditional VAE (CVAE) objective at eq. 1 bounds the marginal likelihood $\log p(x)$; this is not correct, as the specific objective at eq. 1 bounds the conditional marginal likelihood $\log p(x|y)$.

Besides that, there are a couple of things that are, in my opinion, not properly motivated. More specifically, the authors propose to sample from the generative model and train a classifier at the server. Why is this necessary and why you do not simply use the generative classifier  $p(y|x) = \frac{\int p(x| z, y)p(z)dz p(y)}{p(x)}$? To me it seems that the target for the auxiliary classifier is nothing else, but to approximate this generative classifier. The loss for the auxiliary classifier (in the limit where you constantly sample from the generative model) can be obtained by minimising the average KL-divergence between the true CVAE generative posterior $p(y|x)$ and the classifier predictive $q(y|x)$. More specifically: $$E_p(x)[KL(p(y|x) || q(y|x))] = - E_p(y,x) [\log q(y|x)] + const = -E_p(y,z,x)[\log q(y|x)]  + const = -E_p(y)p(z)p(x|y,z)[\log q(y|x)] + const$$ In this sense, it seems that 1) it might not be necessary to train an auxiliary classifier and 2) the modelling choices made in the generative model itself are important for the performance of the actual downstream classifier. I believe a discussion about this is warranted to be added in the paper. Apart from that, when the authors want to sample datapoints from the model they actually use a different distribution over the latents than the one the model was actually trained; e.g., using a $z \sim U[-1, 1]$ instead of $z\sim N(0, I)$. They mention that it is beneficial to stay at a high density region of the prior, but how does the performance of the algorithm fare when you use the correct prior? Why not use e.g., $z \sim N(0, \sigma)$ with $\sigma < 1$ which is closer to the correct prior?

Another thing that could be better explained are the security and privacy aspects of this method, especially given the FL scenario. FedCVAE uploads the CVAE decoder along with $p(y)$ to the server, which is inherently less private than FedAvg. While it is true that with DP you can bound memorisation and get privacy, at the moment the authors do not use anything and the gains in performance when doing DP relatively to e.g., DP-FedAvg are not clear. In fact, given a sufficient powerful generative model, this method should be indistinguishable from sending the actual data. Furthermore, the arguments about security by shifting $\mu$, need to be more formal. Are there some guarantees that you can give, given a sufficient powerful adversary? For example, given $p(y)$, and $p(x|z, y)$, couldn’t the attacker just try to infer a global mean for $z$? For example, by visual inspection of the samples or an optimization procedure? And a small comment; $U[-10, 10]$ is not an exceptionally good guess for $N(0, I)$ (which is what the text mentions at the beginning of page 9) as 99% of the probability of $N(0,I)$ is in $[-3, 3]$, so 70% of the mass of the uniform proposal is on almost zero probability mass under $N(0, I)$.

Furthermore, this method requires building good generative models which is generally a harder task than building a good discriminative model. How does this work on more complex datasets, such as CIFAR10 and CIFAR100, where $p(x|y, z)$ might not be as sharp? Furthermore, while non-iid settings are nice to have, to me it seems that this method might suffer in less non-iid / iid settings (e.g, $\alpha=1.0$, $\alpha=\inf$) ; the generative distribution is more diverse and thus it would be harder to model accurately with a, hardware constrained, VAE model. This can also sort of be seen when comparing FedCVAE-KD to FedCVAE-Ens; the former has worse performance that the latter, given that the entire training distribution needs to be modelled by a single model. How does the performance of the method scale with higher values of $\alpha$? Besides that, it seems that the evaluation against the baselines is not entirely apples-to-apples; for example, FedOneShot does not need to reveal to the server the labels and the models trained on the local data, so it has fundamentally different security / privacy tradeoffs.

The evidence to be evaluated is:Notably, existing one-shot FL methods either ignore the issue of statistical heterogeneity (i.e., Guha et al. (2019)), fail to comprehensively explore the effect of statistical heterogeneity on performance (i.e., Shin et al. (2020) and Li et al. (2021)), or degrade substantially at even moderate levels of statistical heterogeneity (i.e., Zhou et al. (2020) and Zhang et al. (2021)). We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

To simulate statistical heterogeneity, we use the Dirichlet distribution to generate disjoint non-IID client training datasets as in Hsu et al. (2019) and Lin et al. (2020). Specifically, we sample \(\mathbf{p}^{k}\sim Dir(\alpha)\) and allocate a \(\mathbf{p}^{k}_{i}\) proportion of class \(i\) to client \(k\). The parameter \(\alpha\) controls the level of non-IID-ness, with a lower \(\alpha\) inducing more skewed label distributions across clients.

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

There are recent FL methods that are not appropriate in our proposed setting. The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

Unless otherwise stated, we use \(m=10\) clients, \(\alpha=0.01\) (very heterogeneous), and report average test accuracy across 5 seeded parameter initializations \(\pm\) one standard deviation. The data partition is fixed unless otherwise stated. Following Zhu et al. (2021), we distribute \(50\%\) of the available training data to clients for MNIST and FashionMNIST, and \(100\%\) for SVHN. All available test data is used to evaluate the final server classifier (or ensemble for FedOneShot).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find this claim to be moderately overstated. While the experimental results do show FedCVAE-Ens and FedCVAE-KD outperforming baselines on MNIST, FashionMNIST, and SVHN datasets in high heterogeneity settings, there are important limitations noted in the review that aren't acknowledged in the claim. These include: 1) lack of testing on more complex datasets like CIFAR10/100, 2) potential performance issues in less heterogeneous settings, and 3) privacy/security tradeoffs compared to baselines. The term "superiority" is too strong given these limitations.

<score>4</score>

Justification:

1. The experimental evidence shows strong performance in high heterogeneity settings:
- Tables 1 and 2 demonstrate better accuracy compared to baselines
- Consistent performance across different numbers of clients
- Lower standard deviations indicating stability

2. However, several limitations make the claim overstated:
- Reviewer points out lack of testing on more complex datasets: "How does this work on more complex datasets, such as CIFAR10 and CIFAR100"
- Security/privacy tradeoffs not fully addressed: "FedCVAE uploads the CVAE decoder along with p(y) to the server, which is inherently less private than FedAvg"
- Potential weaknesses in less heterogeneous settings: "this method might suffer in less non-iid / iid settings"
- Limited comparison scope: "evaluation against the baselines is not entirely apples-to-apples"

3. The claim would be better stated by:
- Specifying performance advantages are primarily in high heterogeneity settings
- Acknowledging dataset complexity limitations
- Noting the different privacy/security tradeoffs compared to baselines
- Using more measured language than "superiority"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Thorough experiments on multiple benchmark datasets (MNIST, FashionMNIST, SVHN) demonstrate the superiority of FedCVAE-Ens and FedCVAE-KD over other relevant one-shot FL methods in the high statistical heterogeneity setting.

The review comment to be evaluated is:

The paper is very well written and easy to follow. The numerical experiments are well executed and show promising performance achieved by the proposed methods.

That being said, there are some downsides that, in my opinion, slightly outweigh the strengths of the paper. On the problem formulation side, I believe the one-shot federated learning (FL) problem is not very well motivated; the point of FL is to enable training models at different clients by leveraging each others' data through communication without explicitly sharing them. In a one-shot scenario, the problem essentially reduces to having a set of models, which need to be aggregated into a single, universal model, so I am still not convinced how the "federated" aspect of the problem is of the essence.

On the contributions side, the main issue I have with the paper is a lack of a central, key novelty. In its current form, the paper is mostly a collection of different ideas, namely two algorithms that are almost independent of each other, as well as a security aspect, which is again mostly orthogonal to the proposed methods. To me, what would make this paper much stronger is to focus on only one of these algorithms/aspects throughout the paper. For example, since the performance of FEDCVAE-KD is better than baselines, I believe it would be better to focus on this algorithm in the main body of the paper, and add the ensemble version of the algorithm (which, as the authors allude to, requires a higher amount of memory that scales with the number of clients) as an extension, in a separate discussion section toward the end of the paper and/or in the appendix. Moreover, the security and privacy considerations are definitely important, but including them in the main body of the paper stretches it too thin, in my opinion.

Further comments:

- Aside from the solid empirical performance, are there any intuitive/theoretical explanations as to why the proposed methods are well suited for i) *one-shot* FL, and ii) *high heterogeneity* FL? As presented, I cannot see a clear motivation for why the proposed methods should perform better in the considered problem formulation.
- Please discuss whether or not sharing local label distributions with the server violates the privacy of clients' data. How do the proposed methods perform if this information is unavailable at the server?
- Presenting the flowchart figure for only one of the proposed methods (FEDCVAE-KD) in Figure 2 and not including any of the algorithms for the two proposed methods makes the paper slightly incoherent, without switching back and forth between the body of the paper and the appendix. I suggest rearranging the contents to remedy this issue as best as possible by at least having both the figure and algorithm for one of the methods or having the figures for both methods in the main body of the paper.
- In the first footnote on page 4, why is a uniform distribution used for sampling $\mathbf{z}$? Why not, for example, use a truncated Gaussian distribution?
- In my opinion, including FedAVG is slightly unfair in the experiments. Under FedAVG, a set of client model parameters $\\{\mathbf{w}^k\\}_{k=1}^m$ are aggregated using a simple average, i.e., $\mathbf{w}=\frac{1}{m} \sum_\{k=1\}^m\mathbf{w}^k$. In a very heterogeneous case, each of these model parameters is the minimizer of a very different objective function over an entirely different optimization landscape, so it is completely expected that a linear averaging of these model parameters should lead to inferior performance, especially if the averaging operation is done only once.
- Following the above comment, I believe your primary competing baseline is FedONESHOT, so I suggest adding more details on this algorithm's main idea.
- Do you have any explanation as to why the standard deviation of the performance of the proposed methods is, on average, larger under SVHN in Table 2?

The evidence to be evaluated is:Notably, existing one-shot FL methods either ignore the issue of statistical heterogeneity (i.e., Guha et al. (2019)), fail to comprehensively explore the effect of statistical heterogeneity on performance (i.e., Shin et al. (2020) and Li et al. (2021)), or degrade substantially at even moderate levels of statistical heterogeneity (i.e., Zhou et al. (2020) and Zhang et al. (2021)). We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

To simulate statistical heterogeneity, we use the Dirichlet distribution to generate disjoint non-IID client training datasets as in Hsu et al. (2019) and Lin et al. (2020). Specifically, we sample \(\mathbf{p}^{k}\sim Dir(\alpha)\) and allocate a \(\mathbf{p}^{k}_{i}\) proportion of class \(i\) to client \(k\). The parameter \(\alpha\) controls the level of non-IID-ness, with a lower \(\alpha\) inducing more skewed label distributions across clients.

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

There are recent FL methods that are not appropriate in our proposed setting. The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

Unless otherwise stated, we use \(m=10\) clients, \(\alpha=0.01\) (very heterogeneous), and report average test accuracy across 5 seeded parameter initializations \(\pm\) one standard deviation. The data partition is fixed unless otherwise stated. Following Zhu et al. (2021), we distribute \(50\%\) of the available training data to clients for MNIST and FashionMNIST, and \(100\%\) for SVHN. All available test data is used to evaluate the final server classifier (or ensemble for FedOneShot).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find it to be moderately overstated. While the experimental results do show that FedCVAE-Ens and FedCVAE-KD outperform the baselines (FedAvg and FedOneShot) on MNIST, FashionMNIST, and SVHN datasets in high heterogeneity settings, there are some important caveats. The reviewer points out valid concerns about the one-shot FL problem formulation and notes that FedAvg may not be a fair baseline comparison. Additionally, the higher standard deviation in SVHN results and the degrading performance of FedCVAE-KD with increasing clients on SVHN suggest some limitations not acknowledged in the broad claim of "superiority."

<score>4</score>

Justification:
1. The experimental evidence does support better performance compared to baselines:
- Tables 1 and 2 show consistently higher accuracy for both proposed methods
- Results show robustness across different heterogeneity levels (α values)
- Lower standard deviations in results compared to baselines

2. However, several factors make the claim overstated:
- The reviewer notes FedAvg is not a fair baseline comparison since simple averaging is expected to perform poorly in heterogeneous settings
- Limited comparison with only two baselines, while other relevant methods were excluded
- Performance degradation of FedCVAE-KD on SVHN with increased clients is not addressed in the claim
- Reviewer highlights lack of theoretical/intuitive explanation for why these methods work better
- The "superiority" claim doesn't acknowledge these limitations or specify conditions where performance may degrade

3. From the review comment:
"Following the above comment, I believe your primary competing baseline is FedONESHOT" - suggesting comparison against FedAvg may inflate perceived advantages
"Do you have any explanation as to why the standard deviation of the performance of the proposed methods is, on average, larger under SVHN in Table 2?" - highlighting unexplained performance variations

The claim would be better stated if it acknowledged these limitations and specified the conditions under which the methods show advantages.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Thorough experiments on multiple benchmark datasets (MNIST, FashionMNIST, SVHN) demonstrate the superiority of FedCVAE-Ens and FedCVAE-KD over other relevant one-shot FL methods in the high statistical heterogeneity setting.

The review comment to be evaluated is:

### Strength

* The proposed method of one-shot FL is very interesting and novel.
* The proposed method doesn't require auxiliary public dataset.
* Both security and privacy extensions are considered in the paper.
* It can be applied to heterogeneous local models with similar generative capabilities.

### Weaknesses
* One potential issue of the method is training a CVAE locally. In the paper, the images used in the dataset are very small, and the amount of data each client has is large. However, in the real-world application of FL like medical imagings, with very large images and very limited number of data samples, training a CVAE locally can be difficult even just for one class of data.
* The training task of local models is different from the original task. This limits the re-use of pre-trained local model from original task. I think one of the reasons that people want to do one-shot FL is to prevent any re-training locally. But the proposed method still requires to train some new model.

The evidence to be evaluated is:Notably, existing one-shot FL methods either ignore the issue of statistical heterogeneity (i.e., Guha et al. (2019)), fail to comprehensively explore the effect of statistical heterogeneity on performance (i.e., Shin et al. (2020) and Li et al. (2021)), or degrade substantially at even moderate levels of statistical heterogeneity (i.e., Zhou et al. (2020) and Zhang et al. (2021)). We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

To simulate statistical heterogeneity, we use the Dirichlet distribution to generate disjoint non-IID client training datasets as in Hsu et al. (2019) and Lin et al. (2020). Specifically, we sample \(\mathbf{p}^{k}\sim Dir(\alpha)\) and allocate a \(\mathbf{p}^{k}_{i}\) proportion of class \(i\) to client \(k\). The parameter \(\alpha\) controls the level of non-IID-ness, with a lower \(\alpha\) inducing more skewed label distributions across clients.

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

There are recent FL methods that are not appropriate in our proposed setting. The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

Unless otherwise stated, we use \(m=10\) clients, \(\alpha=0.01\) (very heterogeneous), and report average test accuracy across 5 seeded parameter initializations \(\pm\) one standard deviation. The data partition is fixed unless otherwise stated. Following Zhu et al. (2021), we distribute \(50\%\) of the available training data to clients for MNIST and FashionMNIST, and \(100\%\) for SVHN. All available test data is used to evaluate the final server classifier (or ensemble for FedOneShot).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be largely well-stated but slightly overstated. While the experimental results in Tables 1 and 2 clearly demonstrate superior performance of FedCVAE-Ens and FedCVAE-KD over baseline methods across multiple datasets and heterogeneity levels, there are some limitations noted in the reviewer comments regarding real-world applicability with large images and limited data. The experiments are thorough on the stated benchmark datasets, but the claim could better acknowledge these practical limitations.

<score>2</score>

Justification:
1. Supporting evidence:
- Comprehensive experiments across MNIST, FashionMNIST, and SVHN datasets
- Consistent superior performance shown in Tables 1 and 2
- Testing across varying levels of statistical heterogeneity (α=0.05, 0.01, 0.001)
- Multiple client scenarios tested (m=5,10,20,50)
- Clear performance advantages: Up to 1.75x-2.75x accuracy improvements

2. Qualifying factors:
- Reviewer points out limitations with large images and limited data samples
- Datasets used are relatively simple (28x28 or 32x32 images)
- Local CVAE training requirements may be challenging in real applications
- Additional local model training needed, contrary to ideal one-shot scenario

The claim is mostly accurate but would benefit from acknowledging these practical constraints while maintaining its core findings about superior performance on the tested benchmarks.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, FedCVAE-Ens and FedCVAE-KD obtain more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and more than \(2.75\times\) the accuracy for SVHN under extreme statistical heterogeneity (i.e., clients only observe one or two classes).

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs.

Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6). \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim and evidence:

The claim states that FedCVAE-Ens and FedCVAE-KD achieve more than 1.75x, 2x, and 2.75x accuracy compared to baselines for MNIST, FashionMNIST, and SVHN respectively under extreme statistical heterogeneity.

Looking at Table 1 (α=0.001 case which represents extreme heterogeneity):
- MNIST: Best baseline is 45.12%, FedCVAE-KD gets 82.24% (1.82x) and FedCVAE-Ens gets 95.37% (2.11x)
- FashionMNIST: Best baseline is 32.77%, FedCVAE-KD gets 69.53% (2.12x) and FedCVAE-Ens gets 76.04% (2.32x)
- SVHN: Best baseline is 20.31%, FedCVAE-KD gets 55.48% (2.73x) and FedCVAE-Ens gets 65.52% (3.22x)

The claim is well-stated and supported by the numerical evidence shown in the tables. The multipliers claimed (1.75x, 2x, and 2.75x) are actually conservative compared to the actual improvements shown in the data.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, FedCVAE-Ens and FedCVAE-KD obtain more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and more than \(2.75\times\) the accuracy for SVHN under extreme statistical heterogeneity (i.e., clients only observe one or two classes).

The review comment to be evaluated is:

Strength:

-- The proposed solutions seem effective under the one-shot FL setting. The efficacy and efficiency are both demonstrated in the extensive experiments.
-- The writing is clear and easy to follow. The tables and figures are well-organized.

Weaknesses:

-- Constructing a generative model to generate samples for FL model training is not new. There are a few works utilizing a similar strategy in FL, for example:
Data-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021
Fine-tuning global model via data-free knowledge distillation for non-iid federated learning, CVPR 2022
Please make a comparison with those works

-- Some details are not clear enough for me. For example, in Eq 2, what is the formulation for g(.)?  Is this dissimilarity measured in the data-space or feature space?

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs.

Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

**Benchmark Datasets.** MNIST and FashionMNIST contain \(28\times 28\) grayscale images of handwritten digits and clothing/accessories, respectively, with \(60,000\) train samples and \(10,000\) test samples. SVHN contains \(32\times 32\) RGB image crops of street-view house numbers, with \(73,257\) train samples and \(26,032\) test samples.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6). \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and claim, I find that the claim about achieving "more than 1.75x, 2x, and 2.75x accuracy" improvements is generally well-supported by the data shown in Table 1 (α=0.001 row) for MNIST, FashionMNIST, and SVHN respectively. The actual numbers from Table 1 show:

MNIST: 95.37/11.90 = 8.01x improvement
FashionMNIST: 76.04/10.00 = 7.60x improvement
SVHN: 65.52/15.94 = 4.11x improvement

The claim is actually understated compared to the actual improvements shown in the data. The claim is conservative in its representation of the performance gains and stays well within the bounds of what the evidence demonstrates.

<score>1</score>