Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a method that shifts the center of the CVAE prior distribution and experimentally demonstrate that this promotes security, and show how either method can incorporate heterogeneous local models.

The review comment to be evaluated is:

The paper presents a relatively simple idea and is mostly clearly written. As far as I am aware, this method is novel in the context of FL. The authors also provide a reasonable amount of information, which should make FedCVAE reproducible. The authors seem to mostly tackle the cross-silo setting, where each client has enough data. One thing I noticed is that the authors mention that the conditional VAE (CVAE) objective at eq. 1 bounds the marginal likelihood $\log p(x)$; this is not correct, as the specific objective at eq. 1 bounds the conditional marginal likelihood $\log p(x|y)$.

Besides that, there are a couple of things that are, in my opinion, not properly motivated. More specifically, the authors propose to sample from the generative model and train a classifier at the server. Why is this necessary and why you do not simply use the generative classifier  $p(y|x) = \frac{\int p(x| z, y)p(z)dz p(y)}{p(x)}$? To me it seems that the target for the auxiliary classifier is nothing else, but to approximate this generative classifier. The loss for the auxiliary classifier (in the limit where you constantly sample from the generative model) can be obtained by minimising the average KL-divergence between the true CVAE generative posterior $p(y|x)$ and the classifier predictive $q(y|x)$. More specifically: $$E_p(x)[KL(p(y|x) || q(y|x))] = - E_p(y,x) [\log q(y|x)] + const = -E_p(y,z,x)[\log q(y|x)]  + const = -E_p(y)p(z)p(x|y,z)[\log q(y|x)] + const$$ In this sense, it seems that 1) it might not be necessary to train an auxiliary classifier and 2) the modelling choices made in the generative model itself are important for the performance of the actual downstream classifier. I believe a discussion about this is warranted to be added in the paper. Apart from that, when the authors want to sample datapoints from the model they actually use a different distribution over the latents than the one the model was actually trained; e.g., using a $z \sim U[-1, 1]$ instead of $z\sim N(0, I)$. They mention that it is beneficial to stay at a high density region of the prior, but how does the performance of the algorithm fare when you use the correct prior? Why not use e.g., $z \sim N(0, \sigma)$ with $\sigma < 1$ which is closer to the correct prior?

Another thing that could be better explained are the security and privacy aspects of this method, especially given the FL scenario. FedCVAE uploads the CVAE decoder along with $p(y)$ to the server, which is inherently less private than FedAvg. While it is true that with DP you can bound memorisation and get privacy, at the moment the authors do not use anything and the gains in performance when doing DP relatively to e.g., DP-FedAvg are not clear. In fact, given a sufficient powerful generative model, this method should be indistinguishable from sending the actual data. Furthermore, the arguments about security by shifting $\mu$, need to be more formal. Are there some guarantees that you can give, given a sufficient powerful adversary? For example, given $p(y)$, and $p(x|z, y)$, couldn’t the attacker just try to infer a global mean for $z$? For example, by visual inspection of the samples or an optimization procedure? And a small comment; $U[-10, 10]$ is not an exceptionally good guess for $N(0, I)$ (which is what the text mentions at the beginning of page 9) as 99% of the probability of $N(0,I)$ is in $[-3, 3]$, so 70% of the mass of the uniform proposal is on almost zero probability mass under $N(0, I)$.

Furthermore, this method requires building good generative models which is generally a harder task than building a good discriminative model. How does this work on more complex datasets, such as CIFAR10 and CIFAR100, where $p(x|y, z)$ might not be as sharp? Furthermore, while non-iid settings are nice to have, to me it seems that this method might suffer in less non-iid / iid settings (e.g, $\alpha=1.0$, $\alpha=\inf$) ; the generative distribution is more diverse and thus it would be harder to model accurately with a, hardware constrained, VAE model. This can also sort of be seen when comparing FedCVAE-KD to FedCVAE-Ens; the former has worse performance that the latter, given that the entire training distribution needs to be modelled by a single model. How does the performance of the method scale with higher values of $\alpha$? Besides that, it seems that the evaluation against the baselines is not entirely apples-to-apples; for example, FedOneShot does not need to reveal to the server the labels and the models trained on the local data, so it has fundamentally different security / privacy tradeoffs.

The evidence to be evaluated is:This joint can be factorized as \(p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z})=p_{\mathbf{\theta}}(\mathbf{x}[\mathbf{z}] p(\mathbf{z})\), where the prior \(p(\mathbf{z})\) is usually chosen to be a multivariate standard normal distribution, i.e., \(p(\mathbf{z})=\mathcal{N}(\mathbf{0},\mathbf{I})\).

Conditional VAEs (CVAEs) extend the basic VAE by conditioning the encoder and decoder on the one-hot encoding of the class label \(\mathbf{y}\in\mathcal{Y}=\{1,2,...,K\}\) corresponding to data sample \(\mathbf{x}\), resulting in the conditional encoder \(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\) and conditional decoder \(p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z},\mathbf{y})\).

\[\mathcal{L}(\mathbf{\phi},\mathbf{\theta};\mathbf{x},\mathbf{y})=-D_{\text{KL}}(q_{ \mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\mid\mid p(\mathbf{z}))+\mathbb{ E}_{q_{\mathbf{\theta}}(\mathbf{z}|\mathbf{x},\mathbf{y})}[\log p_{\mathbf{\theta}}( \mathbf{x}|\mathbf{z},\mathbf{y})], \tag{1}\]

They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

We discuss privacy- and security-promoting extensions in Appendix B and Section 3.3, respectively. FedCVAE-KD trains a CVAE \(f_{\mathbf{w}^{k}}(\cdot)\) for every client \(k\in\mathcal{C}\) to convergence on their private local dataset by solving Equation 1; this CVAE is parameterized by \(\mathbf{w}^{k}=[\phi^{k},\mathbf{\theta}^{k}]\), with encoder \(E_{\mathbf{\phi}^{k}}(\cdot)\) and decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\). Then, clients communicate their decoder weights \(\mathbf{\theta}^{k}\) and label distributions \(\bar{p}^{k}(\mathbf{y})\) to the server; in practice, this simply requires upload of client label counts as in Zhu et al. (2021). This completes the single communication round.

In practice, we find it useful to focus on the highest density region of the prior and instead sample from a truncated standard normal distribution with tight symmetric bounds. The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space to a single student server decoder, parameterized by \(\mathbf{\theta}^{S}\). To achieve this, we generate \(n_{D}\) total KD training samples, with \(\mathcal{D}_{\text{Ens}}\) defined as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k}, \mathbf{z}_{i}^{k})\}_{i=1}^{\lfloor n_{D}/\rfloor}\). Then, we train the server to match the teacher's mapping of \((\mathbf{z}_{i}^{k},\mathbf{y}_{i}^{k})\) to \(\hat{\mathbf{x}}_{i}^{k}\) by minimizing a reconstruction loss: \[\ell_{\text{KD}}(\mathbf{\theta}^{S};\mathbf{z}^{k},\mathbf{y}^{k},\hat{\mathbf{x }}^{k})=g(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k}),\hat{\mathbf{x }}^{k}), \tag{2}\] which penalizes the dissimilarity \(g(\cdot)\) in data space between the synthetic data sample generated by the server decoder \(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k})\) and the client decoder sample \(\hat{\mathbf{x}}^{k}\).

Because FedCVAE-KD ensembles client decoders to create a labeled dataset \(\mathcal{D}_{\text{Ens}}\) to train the server decoder, each client can have a unique CVAE model architecture, accommodating each client's computational limitations.

CVAEs use a prior distribution over latent space to train the encoder and decoder models. While a multivariate standard normal distribution is typically used for convenience, any normal distribution is acceptable. To promote security, we propose to shift the center of the prior distribution \(\mathbf{\mu}\) to a random position in real space (i.e., \(\mathbf{\mu}\in\mathbb{R}^{d}\)), which can be communicated offline or via encryption methods between server and clients (Zhou et al., 2020). As shown in Figure 7 (Appendix B), sampling latent vectors too far from the center of the normal prior produces qualitatively poor data samples, deterring eavesdropping attackers who have no knowledge of \(\mathbf{\mu}\).

The server decoder for FedCVAE-KD has the same architecture as the client CVAEs by default, although this is not strictly necessary because FedCVAE-KD supports heterogeneous CVAE architectures.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

The proposed KD aggregation method used in FedCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure 8 in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C). As expected, FedCVAE-Ens bounds the performance of FedCVAE-KD, but KD-based aggregation still preserves client ensemble performance well across levels of \(\alpha\) for all datasets (Table 1).

**Heterogeneous Local Models.** To simulate clients with diverse computational resources, we train both FedCVAE-Ens and FedCVAE-KD using two local CVAE architectures: the first as described in Section 4.1 and the second with one convolutional/deconvolutional block removed for the encoder/decoder, respectively.

Because the two architectures demonstrate similar generative capabilities, final server classifier accuracy is very similar when comparing homogeneous against heterogeneous local architectures (Figure 4).4 Notably, our KD procedure for FedCVAE-KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge to a single decoder.

**Promoting Security.** We verify the effectiveness of our proposed distribution shift extension for securing the uploaded information. Suppose an eavesdropping attacker is able to intercept the label distributions, decoder weights, and decoder architectures from all clients during upload. Without knowledge of the shared center of the multivariate normal prior \(\mathbf{\mu}\), we show that training a performant classifier is infeasible because it is difficult to extract high-quality samples from the client decoders. In particular, even when the attacker samples latent vectors \(\mathbf{z}\) from a broad region which overlaps with the high-density region of the prior (i.e., a uniform distribution centered on \(\mathbf{\mu}\)), the accuracy of the classifier trained on the resulting samples degrades sharply as the sampling region grows (Figure 5).

Therefore, this simple FedCVAE-Ens and FedCVAE-KD extension reduces eavesdropping attackers' capacity to extract high-quality samples from uploaded decoders or train a performant downstream model, reducing communication risks.

Statistical heterogeneity quickly emerged as a core issue within FL, with many studies focusing on maintaining high performance under very non-IID data.

Other works have used KD to circumvent issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary data (Lin et al., 2020; Sattler et al., 2021) or a generative model (Zhu et al., 2021; Zhang et al., 2022) to compactly capture client ensemble learning.

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security.

During training, the CVAE will largely observe latent vectors in the highest density region of the prior distribution; for a standard normal distribution, this is near the center \(\mathbf{\mu}=\mathbf{0}\). Latent vectors distant from the center will not generate high-quality samples when used with the trained decoder. As a demonstration, we train a centralized CVAE and sample both close to the center of the prior (i.e., \(\mathbf{z}_{i}\sim\mathcal{U}(-1,1)\)) and distant from the center of the prior (i.e., \(\mathbf{z}_{i}\sim\mathcal{U}(5,20)\)).

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 7: CVAE samples generated using latent vectors distant from the center of the multivariate normal prior (top row) and close to the center (bottom row).

Input: Figure 4: Results with heterogeneous local models. “Homogeneous” uses the same CVAE architecture for all clients, whereas “heterogeneous” uses two architectures with similar generative capabilities.

Input: Figure 5: Results for our proposed security-promoting extension at high heterogeneity (α = 0.05). We show the accuracy for a classifier trained on samples from the intercepted client decoders and using client label distributions when the center of the normal prior is unknown. The sampling bounds represent the parameters of the uniform distribution used for latent vector sampling, i.e., ±100 represents the distribution U(−100, 100). The prior is a multivariate standard normalN (0, I).

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be somewhat overstated, primarily regarding the security aspects. While the evidence shows that sampling far from the prior's center produces poor quality samples (Figure 7) and classifier accuracy degrades with broader sampling bounds (Figure 5), this doesn't conclusively demonstrate security - as noted in the review, an attacker could potentially infer the global mean. Additionally, while the evidence supports that the method can incorporate heterogeneous models (Figure 4 shows similar performance), this is demonstrated only with two architectures of similar generative capabilities, making the claim broader than what's firmly established.

<score>4</score>

Justification: The claim overstates the security benefits without acknowledging limitations and potential vulnerabilities noted in the review. The heterogeneous model support is demonstrated but more limited in scope than the broad claim suggests. The evidence shows degraded sample quality with shifted distributions but lacks formal security guarantees mentioned in the review feedback.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a method that shifts the center of the CVAE prior distribution and experimentally demonstrate that this promotes security, and show how either method can incorporate heterogeneous local models.

The review comment to be evaluated is:

The paper is very well written and easy to follow. The numerical experiments are well executed and show promising performance achieved by the proposed methods.

That being said, there are some downsides that, in my opinion, slightly outweigh the strengths of the paper. On the problem formulation side, I believe the one-shot federated learning (FL) problem is not very well motivated; the point of FL is to enable training models at different clients by leveraging each others' data through communication without explicitly sharing them. In a one-shot scenario, the problem essentially reduces to having a set of models, which need to be aggregated into a single, universal model, so I am still not convinced how the "federated" aspect of the problem is of the essence.

On the contributions side, the main issue I have with the paper is a lack of a central, key novelty. In its current form, the paper is mostly a collection of different ideas, namely two algorithms that are almost independent of each other, as well as a security aspect, which is again mostly orthogonal to the proposed methods. To me, what would make this paper much stronger is to focus on only one of these algorithms/aspects throughout the paper. For example, since the performance of FEDCVAE-KD is better than baselines, I believe it would be better to focus on this algorithm in the main body of the paper, and add the ensemble version of the algorithm (which, as the authors allude to, requires a higher amount of memory that scales with the number of clients) as an extension, in a separate discussion section toward the end of the paper and/or in the appendix. Moreover, the security and privacy considerations are definitely important, but including them in the main body of the paper stretches it too thin, in my opinion.

Further comments:

- Aside from the solid empirical performance, are there any intuitive/theoretical explanations as to why the proposed methods are well suited for i) *one-shot* FL, and ii) *high heterogeneity* FL? As presented, I cannot see a clear motivation for why the proposed methods should perform better in the considered problem formulation.
- Please discuss whether or not sharing local label distributions with the server violates the privacy of clients' data. How do the proposed methods perform if this information is unavailable at the server?
- Presenting the flowchart figure for only one of the proposed methods (FEDCVAE-KD) in Figure 2 and not including any of the algorithms for the two proposed methods makes the paper slightly incoherent, without switching back and forth between the body of the paper and the appendix. I suggest rearranging the contents to remedy this issue as best as possible by at least having both the figure and algorithm for one of the methods or having the figures for both methods in the main body of the paper.
- In the first footnote on page 4, why is a uniform distribution used for sampling $\mathbf{z}$? Why not, for example, use a truncated Gaussian distribution?
- In my opinion, including FedAVG is slightly unfair in the experiments. Under FedAVG, a set of client model parameters $\\{\mathbf{w}^k\\}_{k=1}^m$ are aggregated using a simple average, i.e., $\mathbf{w}=\frac{1}{m} \sum_\{k=1\}^m\mathbf{w}^k$. In a very heterogeneous case, each of these model parameters is the minimizer of a very different objective function over an entirely different optimization landscape, so it is completely expected that a linear averaging of these model parameters should lead to inferior performance, especially if the averaging operation is done only once.
- Following the above comment, I believe your primary competing baseline is FedONESHOT, so I suggest adding more details on this algorithm's main idea.
- Do you have any explanation as to why the standard deviation of the performance of the proposed methods is, on average, larger under SVHN in Table 2?

The evidence to be evaluated is:This joint can be factorized as \(p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z})=p_{\mathbf{\theta}}(\mathbf{x}[\mathbf{z}] p(\mathbf{z})\), where the prior \(p(\mathbf{z})\) is usually chosen to be a multivariate standard normal distribution, i.e., \(p(\mathbf{z})=\mathcal{N}(\mathbf{0},\mathbf{I})\).

Conditional VAEs (CVAEs) extend the basic VAE by conditioning the encoder and decoder on the one-hot encoding of the class label \(\mathbf{y}\in\mathcal{Y}=\{1,2,...,K\}\) corresponding to data sample \(\mathbf{x}\), resulting in the conditional encoder \(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\) and conditional decoder \(p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z},\mathbf{y})\).

\[\mathcal{L}(\mathbf{\phi},\mathbf{\theta};\mathbf{x},\mathbf{y})=-D_{\text{KL}}(q_{ \mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\mid\mid p(\mathbf{z}))+\mathbb{ E}_{q_{\mathbf{\theta}}(\mathbf{z}|\mathbf{x},\mathbf{y})}[\log p_{\mathbf{\theta}}( \mathbf{x}|\mathbf{z},\mathbf{y})], \tag{1}\]

They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

We discuss privacy- and security-promoting extensions in Appendix B and Section 3.3, respectively. FedCVAE-KD trains a CVAE \(f_{\mathbf{w}^{k}}(\cdot)\) for every client \(k\in\mathcal{C}\) to convergence on their private local dataset by solving Equation 1; this CVAE is parameterized by \(\mathbf{w}^{k}=[\phi^{k},\mathbf{\theta}^{k}]\), with encoder \(E_{\mathbf{\phi}^{k}}(\cdot)\) and decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\). Then, clients communicate their decoder weights \(\mathbf{\theta}^{k}\) and label distributions \(\bar{p}^{k}(\mathbf{y})\) to the server; in practice, this simply requires upload of client label counts as in Zhu et al. (2021). This completes the single communication round.

In practice, we find it useful to focus on the highest density region of the prior and instead sample from a truncated standard normal distribution with tight symmetric bounds. The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space to a single student server decoder, parameterized by \(\mathbf{\theta}^{S}\). To achieve this, we generate \(n_{D}\) total KD training samples, with \(\mathcal{D}_{\text{Ens}}\) defined as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k}, \mathbf{z}_{i}^{k})\}_{i=1}^{\lfloor n_{D}/\rfloor}\). Then, we train the server to match the teacher's mapping of \((\mathbf{z}_{i}^{k},\mathbf{y}_{i}^{k})\) to \(\hat{\mathbf{x}}_{i}^{k}\) by minimizing a reconstruction loss: \[\ell_{\text{KD}}(\mathbf{\theta}^{S};\mathbf{z}^{k},\mathbf{y}^{k},\hat{\mathbf{x }}^{k})=g(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k}),\hat{\mathbf{x }}^{k}), \tag{2}\] which penalizes the dissimilarity \(g(\cdot)\) in data space between the synthetic data sample generated by the server decoder \(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k})\) and the client decoder sample \(\hat{\mathbf{x}}^{k}\).

Because FedCVAE-KD ensembles client decoders to create a labeled dataset \(\mathcal{D}_{\text{Ens}}\) to train the server decoder, each client can have a unique CVAE model architecture, accommodating each client's computational limitations.

CVAEs use a prior distribution over latent space to train the encoder and decoder models. While a multivariate standard normal distribution is typically used for convenience, any normal distribution is acceptable. To promote security, we propose to shift the center of the prior distribution \(\mathbf{\mu}\) to a random position in real space (i.e., \(\mathbf{\mu}\in\mathbb{R}^{d}\)), which can be communicated offline or via encryption methods between server and clients (Zhou et al., 2020). As shown in Figure 7 (Appendix B), sampling latent vectors too far from the center of the normal prior produces qualitatively poor data samples, deterring eavesdropping attackers who have no knowledge of \(\mathbf{\mu}\).

The server decoder for FedCVAE-KD has the same architecture as the client CVAEs by default, although this is not strictly necessary because FedCVAE-KD supports heterogeneous CVAE architectures.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

The proposed KD aggregation method used in FedCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure 8 in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C). As expected, FedCVAE-Ens bounds the performance of FedCVAE-KD, but KD-based aggregation still preserves client ensemble performance well across levels of \(\alpha\) for all datasets (Table 1).

**Heterogeneous Local Models.** To simulate clients with diverse computational resources, we train both FedCVAE-Ens and FedCVAE-KD using two local CVAE architectures: the first as described in Section 4.1 and the second with one convolutional/deconvolutional block removed for the encoder/decoder, respectively.

Because the two architectures demonstrate similar generative capabilities, final server classifier accuracy is very similar when comparing homogeneous against heterogeneous local architectures (Figure 4).4 Notably, our KD procedure for FedCVAE-KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge to a single decoder.

**Promoting Security.** We verify the effectiveness of our proposed distribution shift extension for securing the uploaded information. Suppose an eavesdropping attacker is able to intercept the label distributions, decoder weights, and decoder architectures from all clients during upload. Without knowledge of the shared center of the multivariate normal prior \(\mathbf{\mu}\), we show that training a performant classifier is infeasible because it is difficult to extract high-quality samples from the client decoders. In particular, even when the attacker samples latent vectors \(\mathbf{z}\) from a broad region which overlaps with the high-density region of the prior (i.e., a uniform distribution centered on \(\mathbf{\mu}\)), the accuracy of the classifier trained on the resulting samples degrades sharply as the sampling region grows (Figure 5).

Therefore, this simple FedCVAE-Ens and FedCVAE-KD extension reduces eavesdropping attackers' capacity to extract high-quality samples from uploaded decoders or train a performant downstream model, reducing communication risks.

Statistical heterogeneity quickly emerged as a core issue within FL, with many studies focusing on maintaining high performance under very non-IID data.

Other works have used KD to circumvent issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary data (Lin et al., 2020; Sattler et al., 2021) or a generative model (Zhu et al., 2021; Zhang et al., 2022) to compactly capture client ensemble learning.

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security.

During training, the CVAE will largely observe latent vectors in the highest density region of the prior distribution; for a standard normal distribution, this is near the center \(\mathbf{\mu}=\mathbf{0}\). Latent vectors distant from the center will not generate high-quality samples when used with the trained decoder. As a demonstration, we train a centralized CVAE and sample both close to the center of the prior (i.e., \(\mathbf{z}_{i}\sim\mathcal{U}(-1,1)\)) and distant from the center of the prior (i.e., \(\mathbf{z}_{i}\sim\mathcal{U}(5,20)\)).

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 7: CVAE samples generated using latent vectors distant from the center of the multivariate normal prior (top row) and close to the center (bottom row).

Input: Figure 4: Results with heterogeneous local models. “Homogeneous” uses the same CVAE architecture for all clients, whereas “heterogeneous” uses two architectures with similar generative capabilities.

Input: Figure 5: Results for our proposed security-promoting extension at high heterogeneity (α = 0.05). We show the accuracy for a classifier trained on samples from the intercepted client decoders and using client label distributions when the center of the normal prior is unknown. The sampling bounds represent the parameters of the uniform distribution used for latent vector sampling, i.e., ±100 represents the distribution U(−100, 100). The prior is a multivariate standard normalN (0, I).

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and reviewer comments, I'll analyze the claim: "We propose a method that shifts the center of the CVAE prior distribution and experimentally demonstrate that this promotes security, and show how either method can incorporate heterogeneous local models."

The evidence strongly supports both parts of this claim:
1. Security promotion is demonstrated through experiments showing degraded sample quality and classifier accuracy when latent vectors are sampled far from the shifted center (Figure 5, Figure 7)
2. Heterogeneous model support is verified through experiments showing similar performance between homogeneous and heterogeneous architectures (Figure 4)

The claim is well-stated and accurately reflects the evidence without exaggeration, making appropriate qualifications.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We propose a method that shifts the center of the CVAE prior distribution and experimentally demonstrate that this promotes security, and show how either method can incorporate heterogeneous local models.

The review comment to be evaluated is:

### Strength

* The proposed method of one-shot FL is very interesting and novel.
* The proposed method doesn't require auxiliary public dataset.
* Both security and privacy extensions are considered in the paper.
* It can be applied to heterogeneous local models with similar generative capabilities.

### Weaknesses
* One potential issue of the method is training a CVAE locally. In the paper, the images used in the dataset are very small, and the amount of data each client has is large. However, in the real-world application of FL like medical imagings, with very large images and very limited number of data samples, training a CVAE locally can be difficult even just for one class of data.
* The training task of local models is different from the original task. This limits the re-use of pre-trained local model from original task. I think one of the reasons that people want to do one-shot FL is to prevent any re-training locally. But the proposed method still requires to train some new model.

The evidence to be evaluated is:This joint can be factorized as \(p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z})=p_{\mathbf{\theta}}(\mathbf{x}[\mathbf{z}] p(\mathbf{z})\), where the prior \(p(\mathbf{z})\) is usually chosen to be a multivariate standard normal distribution, i.e., \(p(\mathbf{z})=\mathcal{N}(\mathbf{0},\mathbf{I})\).

Conditional VAEs (CVAEs) extend the basic VAE by conditioning the encoder and decoder on the one-hot encoding of the class label \(\mathbf{y}\in\mathcal{Y}=\{1,2,...,K\}\) corresponding to data sample \(\mathbf{x}\), resulting in the conditional encoder \(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\) and conditional decoder \(p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z},\mathbf{y})\).

\[\mathcal{L}(\mathbf{\phi},\mathbf{\theta};\mathbf{x},\mathbf{y})=-D_{\text{KL}}(q_{ \mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\mid\mid p(\mathbf{z}))+\mathbb{ E}_{q_{\mathbf{\theta}}(\mathbf{z}|\mathbf{x},\mathbf{y})}[\log p_{\mathbf{\theta}}( \mathbf{x}|\mathbf{z},\mathbf{y})], \tag{1}\]

They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

We discuss privacy- and security-promoting extensions in Appendix B and Section 3.3, respectively. FedCVAE-KD trains a CVAE \(f_{\mathbf{w}^{k}}(\cdot)\) for every client \(k\in\mathcal{C}\) to convergence on their private local dataset by solving Equation 1; this CVAE is parameterized by \(\mathbf{w}^{k}=[\phi^{k},\mathbf{\theta}^{k}]\), with encoder \(E_{\mathbf{\phi}^{k}}(\cdot)\) and decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\). Then, clients communicate their decoder weights \(\mathbf{\theta}^{k}\) and label distributions \(\bar{p}^{k}(\mathbf{y})\) to the server; in practice, this simply requires upload of client label counts as in Zhu et al. (2021). This completes the single communication round.

In practice, we find it useful to focus on the highest density region of the prior and instead sample from a truncated standard normal distribution with tight symmetric bounds. The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space to a single student server decoder, parameterized by \(\mathbf{\theta}^{S}\). To achieve this, we generate \(n_{D}\) total KD training samples, with \(\mathcal{D}_{\text{Ens}}\) defined as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k}, \mathbf{z}_{i}^{k})\}_{i=1}^{\lfloor n_{D}/\rfloor}\). Then, we train the server to match the teacher's mapping of \((\mathbf{z}_{i}^{k},\mathbf{y}_{i}^{k})\) to \(\hat{\mathbf{x}}_{i}^{k}\) by minimizing a reconstruction loss: \[\ell_{\text{KD}}(\mathbf{\theta}^{S};\mathbf{z}^{k},\mathbf{y}^{k},\hat{\mathbf{x }}^{k})=g(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k}),\hat{\mathbf{x }}^{k}), \tag{2}\] which penalizes the dissimilarity \(g(\cdot)\) in data space between the synthetic data sample generated by the server decoder \(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k})\) and the client decoder sample \(\hat{\mathbf{x}}^{k}\).

Because FedCVAE-KD ensembles client decoders to create a labeled dataset \(\mathcal{D}_{\text{Ens}}\) to train the server decoder, each client can have a unique CVAE model architecture, accommodating each client's computational limitations.

CVAEs use a prior distribution over latent space to train the encoder and decoder models. While a multivariate standard normal distribution is typically used for convenience, any normal distribution is acceptable. To promote security, we propose to shift the center of the prior distribution \(\mathbf{\mu}\) to a random position in real space (i.e., \(\mathbf{\mu}\in\mathbb{R}^{d}\)), which can be communicated offline or via encryption methods between server and clients (Zhou et al., 2020). As shown in Figure 7 (Appendix B), sampling latent vectors too far from the center of the normal prior produces qualitatively poor data samples, deterring eavesdropping attackers who have no knowledge of \(\mathbf{\mu}\).

The server decoder for FedCVAE-KD has the same architecture as the client CVAEs by default, although this is not strictly necessary because FedCVAE-KD supports heterogeneous CVAE architectures.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

The proposed KD aggregation method used in FedCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure 8 in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C). As expected, FedCVAE-Ens bounds the performance of FedCVAE-KD, but KD-based aggregation still preserves client ensemble performance well across levels of \(\alpha\) for all datasets (Table 1).

**Heterogeneous Local Models.** To simulate clients with diverse computational resources, we train both FedCVAE-Ens and FedCVAE-KD using two local CVAE architectures: the first as described in Section 4.1 and the second with one convolutional/deconvolutional block removed for the encoder/decoder, respectively.

Because the two architectures demonstrate similar generative capabilities, final server classifier accuracy is very similar when comparing homogeneous against heterogeneous local architectures (Figure 4).4 Notably, our KD procedure for FedCVAE-KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge to a single decoder.

**Promoting Security.** We verify the effectiveness of our proposed distribution shift extension for securing the uploaded information. Suppose an eavesdropping attacker is able to intercept the label distributions, decoder weights, and decoder architectures from all clients during upload. Without knowledge of the shared center of the multivariate normal prior \(\mathbf{\mu}\), we show that training a performant classifier is infeasible because it is difficult to extract high-quality samples from the client decoders. In particular, even when the attacker samples latent vectors \(\mathbf{z}\) from a broad region which overlaps with the high-density region of the prior (i.e., a uniform distribution centered on \(\mathbf{\mu}\)), the accuracy of the classifier trained on the resulting samples degrades sharply as the sampling region grows (Figure 5).

Therefore, this simple FedCVAE-Ens and FedCVAE-KD extension reduces eavesdropping attackers' capacity to extract high-quality samples from uploaded decoders or train a performant downstream model, reducing communication risks.

Statistical heterogeneity quickly emerged as a core issue within FL, with many studies focusing on maintaining high performance under very non-IID data.

Other works have used KD to circumvent issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary data (Lin et al., 2020; Sattler et al., 2021) or a generative model (Zhu et al., 2021; Zhang et al., 2022) to compactly capture client ensemble learning.

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security.

During training, the CVAE will largely observe latent vectors in the highest density region of the prior distribution; for a standard normal distribution, this is near the center \(\mathbf{\mu}=\mathbf{0}\). Latent vectors distant from the center will not generate high-quality samples when used with the trained decoder. As a demonstration, we train a centralized CVAE and sample both close to the center of the prior (i.e., \(\mathbf{z}_{i}\sim\mathcal{U}(-1,1)\)) and distant from the center of the prior (i.e., \(\mathbf{z}_{i}\sim\mathcal{U}(5,20)\)).

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 7: CVAE samples generated using latent vectors distant from the center of the multivariate normal prior (top row) and close to the center (bottom row).

Input: Figure 4: Results with heterogeneous local models. “Homogeneous” uses the same CVAE architecture for all clients, whereas “heterogeneous” uses two architectures with similar generative capabilities.

Input: Figure 5: Results for our proposed security-promoting extension at high heterogeneity (α = 0.05). We show the accuracy for a classifier trained on samples from the intercepted client decoders and using client label distributions when the center of the normal prior is unknown. The sampling bounds represent the parameters of the uniform distribution used for latent vector sampling, i.e., ±100 represents the distribution U(−100, 100). The prior is a multivariate standard normalN (0, I).

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Evaluation Summary:
The claim about shifting the CVAE prior distribution to promote security and incorporating heterogeneous models is fairly well-supported by the evidence. Figure 5 demonstrates that sampling from incorrect regions degrades classifier accuracy, supporting the security claims. Figure 4 and accompanying results show similar performance between homogeneous and heterogeneous architectures, validating that claim. However, the security benefits are somewhat overstated - while the approach makes attacks more difficult, it's presented as more definitive than the evidence suggests, making this lean slightly toward overstated.

<score>2</score>

Detailed Justification:

1. Security Claims:
- Evidence shows degraded sample quality when sampling far from the prior center (Figure 7)
- Figure 5 demonstrates declining classifier accuracy with broader sampling bounds
- However, security benefits are presented more definitively than the experimental results warrant

2. Heterogeneous Model Support:
- Figure 4 directly demonstrates comparable performance between heterogeneous and homogeneous architectures
- Results show "similar generative capabilities" between different architectures
- Table 2 shows consistent performance across different client numbers
- This aspect is well-supported by multiple pieces of evidence

The claim earns a 2 rather than 1 mainly due to slightly overstating the security benefits, though the heterogeneous model claims are well-supported.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines.

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the statement about doubling accuracy is supported by the data, but slightly overstated in its generalization. While Table 1 shows that at α=0.001 (highest heterogeneity), FedCVAE-KD achieves >1.75x accuracy for MNIST, >2x for FashionMNIST, and nearly 2.75x for SVHN compared to baselines, and FedCVAE-Ens performs even better, the claim's use of "typically" suggests this happens consistently across all conditions, which isn't always true. At lower heterogeneity levels (α=0.05), the improvement is notable but doesn't always reach 2x.

<score>2</score>

Justification:
1. The data supports significant improvements at high heterogeneity:
- MNIST (α=0.001): 95.37% (FedCVAE-Ens) vs 45.12% (FedAvg)
- FashionMNIST (α=0.001): 76.04% vs 32.77%
- SVHN (α=0.001): 65.52% vs 20.31%

2. However, the claim's limitations include:
- "Typically" implies consistency across conditions, but the 2x improvement isn't universal
- At α=0.05 (lower heterogeneity), improvements are smaller
- The term "baselines" is vague when specific comparisons are made to FedAvg and FedOneShot

3. The claim would be more accurate if it specified:
- The specific heterogeneity level (α=0.001)
- The exact baseline methods being compared
- The variation in improvement across different conditions


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that HRBP (HRBP++) can almost match the accuracy of unstructured sparse training methods while achieving a huge acceleration on hardware.

The review comment to be evaluated is:

Weaknesses:
1. From my point of view, the most important matter of the submission is whether $dW=dY  X^\top$ can be applied in backward directly. Though it is claimed and visualized in the content (for example, Figure 1) that the resulting $W^{'}$ is not $W^\top$, I have conducted gradient derivation by manually multiplying gradient of output and activation and my experience implied that dW can be arrived by matrix multiplication between dY and X. My past work may not be fully solid. However, one very obvious question will be: why padding is necessary in backward ? In another words, I am still confused about the submission's foundational motivation.

The evidence to be evaluated is:Even with the help of the dedicated sparse matrix representation technique such as CSR format (Buluc et al., 2009), the unstructured sparsity still expects to have over \(85\%\) sparsity ratio to acquire limited acceleration since the irregular weight distribution causes significant computation overhead due to poor data locality (Yuan et al., 2021). The recent weight regrouping (reorganization) (Rumi et al., 2020) can accelerate the unstructured sparse weights on hardware by extracting multiple smaller dense blocks in a large sparse matrix, which can improve the throughput with GEMM.

The regrouping algorithm (Rumi et al., 2020) finds similar rows and columns from the sparse weights matrix \(\mathbf{W}\odot m\) and brings them together into several dense blocks. In detail, it firstly clusters \(C_{O}\) rows of \(m\) into several groups based on the Jaccard similarity among non-zeros columns. For each group, it then picks out columns with the most non-zeros weights from all \(C_{I}K_{h}K_{w}\) columns and generates one dense block. For example, in Fig. 2(a), we can group filter \(\mathbf{K}_{0}\), \(\mathbf{K}_{1}\), and \(\mathbf{K}_{3}\) together, and select columns with at least two non-zero (_i.e._, \(0^{th}\), \(2^{th}\), and \(5^{th}\) columns), which leads to the dense block with orange color in Fig. 2(b).

The vanilla HRBP inevitably introduces too many zero kernels, which may result in layer collapse and a heavy accuracy drop (Tanaka et al., 2020). Thus, we further propose HRBP++ to reduce the number of zero kernels.

To this end, HRBP++ allows for zero values within kernels, resulting in fewer entirely discarded kernels for the same overall sparsity ratio.

With HRBP(++), we can extract dense blocks from sparse weights and accelerate both forward and backward of sparse training. For simplicity, we apply HRBP(++) to a random pruning mask, which randomly chooses connections to satisfy the sparsity ratio requirements. To this end, we design a strategy to create a random pruning mask in HRBP (++) style, named _HRBP(++)-based Random Mask_.

We implement our method with CUDA and measure the acceleration rate on GPU. In detail, we rewrite the nn.Conv2d module of Pytorch with CUDA to run CNN with sparse weights. The thread is the basic programmable unit that allows GPU programmers to use massive numbers of CUDA cores. CUDA threads are grouped at different levels such as warp, block, and grid. Given the dense blocks found by HRBP/HRBP++, the original convolution computation will be decomposed into several computations corresponding to dense blocks, and these computations will be performed in parallel by threads.

We follow the experimental settings in (Wang et al., 2020; Yuan et al., 2021) and conduct experiments on CIFAR-10, CIFAR-100 and ImageNet-1K (Deng et al., 2009). As the common setting in sparse training, we apply the WideResNet-32-2 and VGG-19 (Simonyan and Zisserman, 2014) for CIFAR-10/100 and ResNet-50 (He et al., 2016) for ImageNet.

We evaluate our method on NVIDIA Ampere A100 (108 SMs, 40GB).

We report the overall training time acceleration rate compared with the dense baselines, which includes both forward and backward calculations on hardware.

We compare HRBP(++) on static sparse training against random unstructured pruning (RU), random channel-wise pruning (RC) with an ERK ratio, and unstructured pruning with well-defined criteria such as SNIP (Lee et al., 2019) and GraSP (Wang et al., 2020).

Following (Wang et al., 2020), we set the sparsity ratio to \(90\%\) and \(95\%\).

From the accuracy perspective, HRBP++ achieves similar performance to unstructured pruning methods and much better performance than channel-wise pruning. For example, on CIFAR-10 with ResNet-32, HRBP++ achieves an accuracy of \(90.84\%\) at \(95\%\) sparsity, which is \(3.65\%\) higher than the RC and just \(0.17\%\) negligible drop to SNIP (Lee et al., 2019).

From the training time acceleration perspective, unstructured pruning cannot be accelerated due to its irregularity of non-zero weights. However, our HRBP can almost match the acceleration of channel-wise pruning, with the benefit of our optimized convolution implementation. Moreover, HRBP obtains more acceleration compared to the convention regrouping (Rumi et al., 2020), as the latter can only accelerate the forward pass. In summary, our HRBP(++) can successfully achieve comparable accuracy as unstructured pruning and bring training time acceleration on hardware.

We take VGG-19 with \(90\%\) overall sparsity on CIFAR-10 as an example and report the speedup of each layer in Figure 5. At shallow layers with a small sparsity ratio, channel-wise pruning achieves a slightly better acceleration. However, at deeper layers, HRBP achieves comparable or even better results, with at most \(7.5\)x acceleration on Conv14.

We further explore the dynamic sparse training with HRBP++, which prunes and grows the connections based on the proposed block-wise updating during the training time. We compare DST with HRBP++ to several unstructured DST methods, including DeepR (Bellec et al., 2018), DSR (Mostafa and Wang, 2019), SET (Mocanu et al., 2018), and RigL (Evci et al., 2020).

As the speedup is similar to static sparse training, we only show the accuracy comparison in Table 2 for simplicity. Noticeably, our HRBP++ can still match the accuracy of unstructured DST methods.

We evaluate the static sparse training with HRBP(++) on ImageNet.

As shown in Table 3, HRBP++ can also achieve similar performance as well-designed unstructured pruning methods. Meanwhile, it provides \(1.26\times\) and \(1.44\times\) training time acceleration on hardware when the pruning ratio is \(80\%\) and \(90\%\) respectively. To this end, our HRBP++ is still effective on complicated tasks like ImageNet.

As in Figure 7, our method can achieve comparable accuracy as unstructured pruning methods at all sparsity levels and all kinds of networks. This suggests the generalization ability of our method. For the acceleration rate on both training and inference, please see Figure 14 in Appendix.

We then propose the HRBP, aiming to accelerate the sparse training at both forward and backward passes by extracting dense blocks from the unstructured mask. Extensive results suggest that sparse training with HRBP(++) can achieve comparable accuracy with unstructured pruning methods. Meanwhile, it brings significant training acceleration. We believe our method can integrate the advantage of both unstructured pruning and structured pruning and make sparse training more hardware-friendly.

Input: unstructured mask \(m\), density ratio of mask \(r\), number of clusters \(t\), density ratio of kernel \(s\), number of input channel \(C_{I}\), the number of output channel \(C_{O}\), the width of the kernel \(K_{w}\), the height of the kernel \(K_{h}\), and the minimum number of rows of each group \(B_{1}\) Output: Dense groups

With HRBP, we can achieve \(72.7\%\) accuracy with \(60\%\) sparsity.

Our method can also achieve comparable or even better accuracy than DessiLBI at the same sparsity level. Note that, DessiLBI follows the LTH setting which still needs dense training at the first step, while our method directly trains the sparse subnetwork from scratch without dense training. We by default apply our method to random unstructured sparse mask with ERK sparse ratio distribution (Evci et al., 2020) of each layer.

We then apply HRBP++ to these unstructured masks and show the results in Figure 13. Noticeably, with SNIP and GraSP masks, HRBP++ can also achieve comparable accuracy at the same sparsity level.

This is consistent with the conclusion in (Evci et al., 2020) that ERK sparsity is beneficial.

Input: Table 3: Static sparse training on ImageNet. The dense ResNet-50 has 75.70% top-1 accuracy.

Input: Figure 5: The layer-wise speedup of VGG-19.

Input: Figure 7: The trade-off between sparsity and accuracy. All models are evaluated on CIFAR-10.

Input: Table 2: Comparison of dynamic sparse training methods on CIFAR-10 and CIFAR-100.

Input: Figure 14: Training and inference acceleration on hardware of ResNet-32 on CIFAR-10.

Input: Figure 13: Ablation study with different types of mask initialization of ResNet-32 on CIFAR-10.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that while HRBP/HRBP++ does show comparable accuracy to unstructured methods in most cases, the claim about "huge acceleration" is somewhat overstated. The evidence shows moderate speedups:
- 1.26x-1.44x acceleration on ImageNet (Table 3)
- Variable layer-wise speedups on VGG-19, maxing at 7.5x for Conv14 (Fig 5)
- Training acceleration around 1.5x at 90% sparsity (Fig 14)

While these improvements are meaningful, describing them as "huge" overstates their magnitude. The accuracy matching part is well-supported by extensive experimental results across different datasets and architectures.

<score>3</score>

The claim receives a score of 3 because while the accuracy comparison part is well-stated and supported by comprehensive evidence, the characterization of acceleration as "huge" is an overstatement given the moderate speedups shown in the actual results.

The review comment about gradient computation fundamentals does not directly relate to the acceleration and accuracy claims being evaluated here.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that HRBP (HRBP++) can almost match the accuracy of unstructured sparse training methods while achieving a huge acceleration on hardware.

The review comment to be evaluated is:

(1) Can you provide some intuitions on whether the proposed form of sparsity is applicable to other neural network operations? DepthWise convolution? Self-Attention? etc.?

(2) Prior work studied other neural networks (recurrent, deeper resnets). Have you considered applying your method on these neural network to better show its generality? Or is this a limitation of your method that can only be applied to reasonably small-sized models?

(3) While it is good to use a broad terms for the paper title, however I would suggest the title of the paper to clearly focus on the target goals of the paper. The paper reads as it is a technique that is general enough for different NN operations, but as the method and results demonstrate, it mainly targets convolution operations.

(4) The results in Table 1 demonstrate that HRBP is seemingly more robust across multiple iterations of training (less variations) -- I agree that it may not be notable difference, but do you have any intuition why it is the case?

(5) The paper covers reasonably high sparsity regime (e.g. 90% and 95% sparsity). What are the benefits of your method for other sparsity ratios? An ablation study---similar to prior work which you have cited---would provide insights about the trade-offs between accuracy-speedup. This is critically important because not all neural networks can be sparsified to the degree that you have studied.

The evidence to be evaluated is:Even with the help of the dedicated sparse matrix representation technique such as CSR format (Buluc et al., 2009), the unstructured sparsity still expects to have over \(85\%\) sparsity ratio to acquire limited acceleration since the irregular weight distribution causes significant computation overhead due to poor data locality (Yuan et al., 2021). The recent weight regrouping (reorganization) (Rumi et al., 2020) can accelerate the unstructured sparse weights on hardware by extracting multiple smaller dense blocks in a large sparse matrix, which can improve the throughput with GEMM.

The regrouping algorithm (Rumi et al., 2020) finds similar rows and columns from the sparse weights matrix \(\mathbf{W}\odot m\) and brings them together into several dense blocks. In detail, it firstly clusters \(C_{O}\) rows of \(m\) into several groups based on the Jaccard similarity among non-zeros columns. For each group, it then picks out columns with the most non-zeros weights from all \(C_{I}K_{h}K_{w}\) columns and generates one dense block. For example, in Fig. 2(a), we can group filter \(\mathbf{K}_{0}\), \(\mathbf{K}_{1}\), and \(\mathbf{K}_{3}\) together, and select columns with at least two non-zero (_i.e._, \(0^{th}\), \(2^{th}\), and \(5^{th}\) columns), which leads to the dense block with orange color in Fig. 2(b).

The vanilla HRBP inevitably introduces too many zero kernels, which may result in layer collapse and a heavy accuracy drop (Tanaka et al., 2020). Thus, we further propose HRBP++ to reduce the number of zero kernels.

To this end, HRBP++ allows for zero values within kernels, resulting in fewer entirely discarded kernels for the same overall sparsity ratio.

With HRBP(++), we can extract dense blocks from sparse weights and accelerate both forward and backward of sparse training. For simplicity, we apply HRBP(++) to a random pruning mask, which randomly chooses connections to satisfy the sparsity ratio requirements. To this end, we design a strategy to create a random pruning mask in HRBP (++) style, named _HRBP(++)-based Random Mask_.

We implement our method with CUDA and measure the acceleration rate on GPU. In detail, we rewrite the nn.Conv2d module of Pytorch with CUDA to run CNN with sparse weights. The thread is the basic programmable unit that allows GPU programmers to use massive numbers of CUDA cores. CUDA threads are grouped at different levels such as warp, block, and grid. Given the dense blocks found by HRBP/HRBP++, the original convolution computation will be decomposed into several computations corresponding to dense blocks, and these computations will be performed in parallel by threads.

We follow the experimental settings in (Wang et al., 2020; Yuan et al., 2021) and conduct experiments on CIFAR-10, CIFAR-100 and ImageNet-1K (Deng et al., 2009). As the common setting in sparse training, we apply the WideResNet-32-2 and VGG-19 (Simonyan and Zisserman, 2014) for CIFAR-10/100 and ResNet-50 (He et al., 2016) for ImageNet.

We evaluate our method on NVIDIA Ampere A100 (108 SMs, 40GB).

We report the overall training time acceleration rate compared with the dense baselines, which includes both forward and backward calculations on hardware.

We compare HRBP(++) on static sparse training against random unstructured pruning (RU), random channel-wise pruning (RC) with an ERK ratio, and unstructured pruning with well-defined criteria such as SNIP (Lee et al., 2019) and GraSP (Wang et al., 2020).

Following (Wang et al., 2020), we set the sparsity ratio to \(90\%\) and \(95\%\).

From the accuracy perspective, HRBP++ achieves similar performance to unstructured pruning methods and much better performance than channel-wise pruning. For example, on CIFAR-10 with ResNet-32, HRBP++ achieves an accuracy of \(90.84\%\) at \(95\%\) sparsity, which is \(3.65\%\) higher than the RC and just \(0.17\%\) negligible drop to SNIP (Lee et al., 2019).

From the training time acceleration perspective, unstructured pruning cannot be accelerated due to its irregularity of non-zero weights. However, our HRBP can almost match the acceleration of channel-wise pruning, with the benefit of our optimized convolution implementation. Moreover, HRBP obtains more acceleration compared to the convention regrouping (Rumi et al., 2020), as the latter can only accelerate the forward pass. In summary, our HRBP(++) can successfully achieve comparable accuracy as unstructured pruning and bring training time acceleration on hardware.

We take VGG-19 with \(90\%\) overall sparsity on CIFAR-10 as an example and report the speedup of each layer in Figure 5. At shallow layers with a small sparsity ratio, channel-wise pruning achieves a slightly better acceleration. However, at deeper layers, HRBP achieves comparable or even better results, with at most \(7.5\)x acceleration on Conv14.

We further explore the dynamic sparse training with HRBP++, which prunes and grows the connections based on the proposed block-wise updating during the training time. We compare DST with HRBP++ to several unstructured DST methods, including DeepR (Bellec et al., 2018), DSR (Mostafa and Wang, 2019), SET (Mocanu et al., 2018), and RigL (Evci et al., 2020).

As the speedup is similar to static sparse training, we only show the accuracy comparison in Table 2 for simplicity. Noticeably, our HRBP++ can still match the accuracy of unstructured DST methods.

We evaluate the static sparse training with HRBP(++) on ImageNet.

As shown in Table 3, HRBP++ can also achieve similar performance as well-designed unstructured pruning methods. Meanwhile, it provides \(1.26\times\) and \(1.44\times\) training time acceleration on hardware when the pruning ratio is \(80\%\) and \(90\%\) respectively. To this end, our HRBP++ is still effective on complicated tasks like ImageNet.

As in Figure 7, our method can achieve comparable accuracy as unstructured pruning methods at all sparsity levels and all kinds of networks. This suggests the generalization ability of our method. For the acceleration rate on both training and inference, please see Figure 14 in Appendix.

We then propose the HRBP, aiming to accelerate the sparse training at both forward and backward passes by extracting dense blocks from the unstructured mask. Extensive results suggest that sparse training with HRBP(++) can achieve comparable accuracy with unstructured pruning methods. Meanwhile, it brings significant training acceleration. We believe our method can integrate the advantage of both unstructured pruning and structured pruning and make sparse training more hardware-friendly.

Input: unstructured mask \(m\), density ratio of mask \(r\), number of clusters \(t\), density ratio of kernel \(s\), number of input channel \(C_{I}\), the number of output channel \(C_{O}\), the width of the kernel \(K_{w}\), the height of the kernel \(K_{h}\), and the minimum number of rows of each group \(B_{1}\) Output: Dense groups

With HRBP, we can achieve \(72.7\%\) accuracy with \(60\%\) sparsity.

Our method can also achieve comparable or even better accuracy than DessiLBI at the same sparsity level. Note that, DessiLBI follows the LTH setting which still needs dense training at the first step, while our method directly trains the sparse subnetwork from scratch without dense training. We by default apply our method to random unstructured sparse mask with ERK sparse ratio distribution (Evci et al., 2020) of each layer.

We then apply HRBP++ to these unstructured masks and show the results in Figure 13. Noticeably, with SNIP and GraSP masks, HRBP++ can also achieve comparable accuracy at the same sparsity level.

This is consistent with the conclusion in (Evci et al., 2020) that ERK sparsity is beneficial.

Input: Table 3: Static sparse training on ImageNet. The dense ResNet-50 has 75.70% top-1 accuracy.

Input: Figure 5: The layer-wise speedup of VGG-19.

Input: Figure 7: The trade-off between sparsity and accuracy. All models are evaluated on CIFAR-10.

Input: Table 2: Comparison of dynamic sparse training methods on CIFAR-10 and CIFAR-100.

Input: Figure 14: Training and inference acceleration on hardware of ResNet-32 on CIFAR-10.

Input: Figure 13: Ablation study with different types of mask initialization of ResNet-32 on CIFAR-10.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim based on the provided evidence and review comments:

The claim appears moderately overstated for several reasons:

1. While the evidence shows comparable accuracy to unstructured methods on the tested datasets, the review comments highlight limited testing on other neural network architectures (like DepthWise convolution, Self-Attention, deeper ResNets).

2. The "huge acceleration" claim is not fully supported - the evidence shows modest speedups (1.26× and 1.44× for ImageNet at 80% and 90% sparsity). Figure 14 shows acceleration rates mostly below 1.5× for training.

3. The method was mainly tested on specific CNN architectures and sparsity ratios (90-95%), with reviewers noting the lack of comprehensive ablation studies across different sparsity ratios.

4. The generalizability claim needs more evidence, as highlighted by reviewer comments about limited testing on other neural operations and network architectures.

<score>4</score>