Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy.

The review comment to be evaluated is:

This paper proposes to increase training costs to compensate for the reduced costs from multi-scale CNN training by either increasing batch size (and therefore lowering the number of iterations per epoch) or increasing the number of augmented versions (duplicates) of the same samples within a batch. The former allows for smaller total training costs than conventional single-scale training, while the latter maintains the total training costs but improves the final performance. Several training improvement methods are introduced to improve the multi-scale training.

Paper's strengths
- The paper is quite well-written.
- Code and models are provided for reproducibility.
- Gradient smoothing is a nice way to mitigate the variability of gradient statistics resulted from different input sizes. As far as I know, this is quite novel, particularly in the context of multi-scale training.

Paper's weaknesses
- Multi-scale training is a common practice in many computer vision tasks especially in object detection** (less common in image classification). This paper also does multi-scale training but only introduces some minor improvements that are neither breakthroughs nor that they provide any interesting insight.
> - Bag of Freebies for Training Object Detection Neural Networks. arXiv.
> - YOLO9000: Better, Faster, Stronger. CVPR 2017.
> - MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.

- For "step-wise size sampling", it seems like that conclusion to use this variant of sampling is heuristically chosen and totally ignores the existing practice in other computer vision tasks. One of the straightforward ways to do multi-scale training in object detection is to select different input sizes even for the images within the same batch (by padding zeros for the smaller images). Alternatively, one could sample different input sizes for different GPU batches (all images within a GPU share the same size) while doing multiple-GPU training.

- The three training improvements (step-wise size sampling, gradient smoothing and batch-norm calibration) are what separate this paper from prior work but they are not extensively evaluated. Some of them are briefly evaluated in the appendix and some of their effects are just briefly mentioned in the method section. They ought to appear in the experimental section of the main paper. Gradient smoothing seems like a nice idea but it is unclear how important it is given that there is only one figure (Fig.7) showing its impact on the performance.

- This paper strives to increase the number of batch samples given a fixed budget of computational and time resources for per iteration step. I wonder why this should be limited to the cost of an iteration step but not the entire training cost from all epochs/iterations. It makes more sense to measure the cost for the entire training process which accurately tells how much is spent to train the model until convergence.

- In Sec. 5.1, the size ratios for different datasets are carefully chosen based on cross-validation. This makes it hard to directly apply MixSize to other datasets or settings without going through this step. It also adds additional computations which defeat the purpose of increasing batch size to maintain the same training cost as conventional single-scale training.

- Using separate BatchNorm statistics for multi-scale inputs/features has been explored in the following papers (published at least few months before ICLR deadline). They should be cited and compared against MixSize:
> - Learning to Learn Parameterized Classification Networks for Scalable Input Images. ECCV 2020.
> - Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks. CVPR 2018.

Overall, this paper shows good performance and has some good ideas (e.g., gradient smoothing) for improving multi-scale training. But it fails to give more emphasis to or do a deeper dive into the potentially novel aspects of the work. The current performance improvements may come from doing just trivial multi-scale training which was already widely-explored in prior work.

The evidence to be evaluated is:Recently, Tan and Le (2019) showed a computation-vs-accuracy trade-off in applying scaling to the image size used for convolutional networks training and evaluation.

In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence.

Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed. This procedure reduced the train-test discrepancy caused by the change in image size and allowed faster training time and improved accuracy -- at the cost of additional fine-tuning procedure and additional computations at inference time. In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

As it is a common practice to use different image size for evaluation than that used for training (He et al., 2017; Lin et al., 2017; Redmon and Farhadi, 2018), it was observed by Touvron et al. (2019) and empirically verified (Figure 1) that classification accuracy may degrade above or below a certain size threshold.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\).

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

Furthermore, given a fixed budget of computational and time resources (per step), we can now modify our regime along spatial and batch axes.

MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size. This behavior may require hyper-parameter tuning and may also affect size-dependent layers such as batch normalization (Ioffe and Szegedy, 2015).

We observed progressive resizing to cause noticeable accuracy deterioration when switching one size for another (Appendix E), which may be attributed to the network tendency to adapt to a specific size.

We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy.

For example, in Table 2, the gradient variance is larger when computed over a small image size (unsurprisingly). This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training.

Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

\[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant.

While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\).

We present the final test accuracy, as well as the number of training steps for each model in Table 1. We see that our MixSize regimes yield two possible improvements:* Reduced number of training steps to achieve a similar test accuracy using \(B^{+}\) regime.  * Better test accuracy using \(D^{+}\) regime (enabling BA with same computational budget).

\(B+\) enables larger batches on average, hence the reduction in training steps, while \(D+\) leverages benefits of batch-augmentation to achieve better accuracy within the same compute.

We are first interested in accelerating the time needed for convergence of the tested models using our \(B^{+}\) scheme. We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F). As described earlier, this model requires the same time and computational resources as the baseline model. However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size.

Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

As the number of floating-point operations (flops) grows linearly with the number of pixels, using a mixed regime significantly improves accuracy per compute at evaluation.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size.

Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size.

Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases. We used 6 common networks (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017; Mahajan et al., 2018; Sandler et al., 2018) pretrained on the ImageNet dataset at a resolution of \(224\times 224\) and measured their accuracy on varying sizes.

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We can see that gradient smoothing helps to reduce gap between gradient norms at difference batch sizes and improves training and validation accuracy. We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

In both CIFAR10 (Figure 8) and ImageNet (Figure 9) sampling methods exhibit better accuracy than a progressive resizing regime. This is most apparent when the final size used at training is larger than target size for inference (on which test accuracy is measured). We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 1: Top-1 test accuracy per image size, models trained on specific sizes (ResNet50, ImageNet).

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Table 1: Test accuracy (Top-1) results for CIFAR and ImageNet. Each row represents models trained using the same computational and memory budget per step. Steps and accuracy are reported at the completion of a fixed epoch budget (i.e., 90 epochs for ResNet on ImageNet, 200 for ResNet on CIFAR). Accuracy is reported for model’s original size (32 for CIFAR, 224 for ImageNet).

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy" appears to be well-stated. The evidence strongly supports this through:

- Table 1 demonstrates that B+ regime (with reduced average image size) requires fewer training steps while maintaining similar accuracy
- Figure 2 shows B+ regime required ~2.7x fewer steps with comparable accuracy
- The evidence comprehensively documents how manipulating image sizes affects both training time and accuracy across multiple experiments and datasets
- The trade-off is clearly demonstrated without overstating or exaggerating the findings

The claim cautiously presents the trade-off without overgeneralizing or making unsupported assertions about the magnitude of effects.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy.

The review comment to be evaluated is:

== Summary ==

The paper proposes to use different image resolutions during the training of a deep neural network image classifier, and varying the batch size or number of data augmented versions of the images, keeping the computational cost per step roughly constant. The authors apply this approach to several architectures and three datasets, and show that they can achieve or improve the same accuracy as the baselines but much faster; or achieve better results with the same computational budget.

== Pros ==

+ The authors conducted their experiments using three different datasets (Cifar10, Cifar100 and ImageNet), and six different architectures (ResNet44, ResNet50, WideResNet, DenseNet and EfficientNet).

+ The proposed approach outperforms the baselines consistently across the 8 pairs of (dataset, architecture) that they have studied. In addition, MixSize can be easily implemented (the authors also provide a PyTorch implementation).

+ The authors investigate different "tricks" to apply during training when using MixSize to stabilize training or achieving better results. For instance, they compared randomly sampling the size from a distribution (as they propose) versus increasing the image resolution as training progresses, and showed that randomly sampling yields slightly better results.

+ Figure 3a seems to suggest that MixSize yields more robust classifiers under a wide range of image sizes. The area under the "Mixed S=144" curve seems to be larger than the area under the "Fixed S=224". However, further experimentation is needed to confirm this, since the area of the "Mixed S=208" seems closer to "Fixed S=224", and in any case the maximum image size was capped around 415.

== Cons ==

- On of the claimed contributions is: "Faster training vs. high accuracy. We show that reducing the average image size at train- ing leads to a trade-off between the time required to train the model and its final accuracy.". However, I would not consider this a novel contribution, since the trade-off between speed and accuracy is well-known. In fact, the authors cite Huang et al. (2018) and Szegedy et al. (2016) which already showed this. EfficientNet is another well-known architecture that takes advantage of this fact.

- In the intro, the authors claim "Touvron et al. (2019) demonstrated that networks trained on specific image resolutions perform poorly on other image sizes at evaluation time, as confirmed in Figure 1". This is inaccurate, since Touvron et al. (2019) actually show that slightly increasing the test resolution improves accuracy, due to the discrepancy in object sizes introduced by data augmentation (cropping). In fact, Figure 1 shows the same effect (model trained with 224 res, achieves best results with 284 eval image size). The statement in the introduction is again contradicted at the end of the first paragraph in Section 2.1.

- The authors do not report any statistical significance metric. Some datasets have very close results, so it's hard to tell whether the improvements are (statistically) significant or not.

- Poor captions in figures and tables. For instance, the difference between solid lines and dashed lines is only explained at the very last figure in the appending (Figure 7). Also, the caption of Figure 3 reads "Test accuracy on validation set" which is ambiguous: Is it a typo, or is it that the authors report the results on the 50k validation set of ImageNet (and use some smaller subset of the training set as validation)?

== Reasons for score ==

Although the proposed approach is simple and consistently improves the baseline results, I'm not convinced that the originality and significance of the work is enough for it to be accepted.

Regarding originality, there is a plethora of works exploring the trade-offs between image size and accuracy. The most similar works are Howard (2018) and Karras et al. (2017), which increase the image size through training. It's not clear that random sampling offers a much better result, judging from Figure 8 in Appending E, if one compares the accuracy of "Small->Large" strategy at 125k steps (possibly before the last increase in size).

Regarding significance, if one restricts the analysis to the best architectures in each dataset, the increase in accuracy does not seem to be very large. Cifar10: 98.16% -> 98.32% (AmoebaNet), ImageNet: 76.32% -> 76.53% (EfficientNet-B0). Cifar100 shows a larger improvement, but the authors did not use AmoebaNet (which worked best in Cifar10) for some unknown reason. The fact that no statistical significance metrics are reported, does not help to discern whether the improvements are meaningful or not.

The evidence to be evaluated is:Recently, Tan and Le (2019) showed a computation-vs-accuracy trade-off in applying scaling to the image size used for convolutional networks training and evaluation.

In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence.

Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed. This procedure reduced the train-test discrepancy caused by the change in image size and allowed faster training time and improved accuracy -- at the cost of additional fine-tuning procedure and additional computations at inference time. In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

As it is a common practice to use different image size for evaluation than that used for training (He et al., 2017; Lin et al., 2017; Redmon and Farhadi, 2018), it was observed by Touvron et al. (2019) and empirically verified (Figure 1) that classification accuracy may degrade above or below a certain size threshold.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\).

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

Furthermore, given a fixed budget of computational and time resources (per step), we can now modify our regime along spatial and batch axes.

MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size. This behavior may require hyper-parameter tuning and may also affect size-dependent layers such as batch normalization (Ioffe and Szegedy, 2015).

We observed progressive resizing to cause noticeable accuracy deterioration when switching one size for another (Appendix E), which may be attributed to the network tendency to adapt to a specific size.

We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy.

For example, in Table 2, the gradient variance is larger when computed over a small image size (unsurprisingly). This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training.

Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

\[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant.

While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\).

We present the final test accuracy, as well as the number of training steps for each model in Table 1. We see that our MixSize regimes yield two possible improvements:* Reduced number of training steps to achieve a similar test accuracy using \(B^{+}\) regime.  * Better test accuracy using \(D^{+}\) regime (enabling BA with same computational budget).

\(B+\) enables larger batches on average, hence the reduction in training steps, while \(D+\) leverages benefits of batch-augmentation to achieve better accuracy within the same compute.

We are first interested in accelerating the time needed for convergence of the tested models using our \(B^{+}\) scheme. We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F). As described earlier, this model requires the same time and computational resources as the baseline model. However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size.

Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

As the number of floating-point operations (flops) grows linearly with the number of pixels, using a mixed regime significantly improves accuracy per compute at evaluation.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size.

Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size.

Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases. We used 6 common networks (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017; Mahajan et al., 2018; Sandler et al., 2018) pretrained on the ImageNet dataset at a resolution of \(224\times 224\) and measured their accuracy on varying sizes.

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We can see that gradient smoothing helps to reduce gap between gradient norms at difference batch sizes and improves training and validation accuracy. We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

In both CIFAR10 (Figure 8) and ImageNet (Figure 9) sampling methods exhibit better accuracy than a progressive resizing regime. This is most apparent when the final size used at training is larger than target size for inference (on which test accuracy is measured). We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 1: Top-1 test accuracy per image size, models trained on specific sizes (ResNet50, ImageNet).

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Table 1: Test accuracy (Top-1) results for CIFAR and ImageNet. Each row represents models trained using the same computational and memory budget per step. Steps and accuracy are reported at the completion of a fixed epoch budget (i.e., 90 epochs for ResNet on ImageNet, 200 for ResNet on CIFAR). Accuracy is reported for model’s original size (32 for CIFAR, 224 for ImageNet).

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the review and evidence provided, the claim "We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy" appears to be overstated.

While the evidence does show that varying image sizes affects training time and accuracy, the relationship isn't a simple trade-off. In fact, the authors demonstrate that with their MixSize approach, they can achieve both faster training AND improved accuracy in many cases. Table 1 shows improved accuracy with fewer training steps across multiple architectures and datasets. Additionally, the reviewer specifically points out that this trade-off was already well-known from previous work (citing Huang et al. 2018 and Szegedy et al. 2016).

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy.

The review comment to be evaluated is:

This paper presents a mixed-size CNN training scheme, using several different input image sizes for one single model training. The authors assume the training budget, represented as S_i^2*B_i*D_i (i.e., spatial sample size, the number of batched distinct samples and the duplicates for each distinct sample), to be a fixed constant during training step i.  Under such an assumption, two mixed-size training scenarios are considered, one for training acceleration and the other for improved model generalization ability. The authors additionally use step-wise image size sampling, gradient smoothing, and per-size BN calibration to enhance the model performance under the above two mixed-size training scenarios. Experimental validation is performed on CIFAR and ImageNet datasets using diverse CNN structures.

Mixed-size training is a critical problem and the methods proposed in this paper are interesting. My main concerns to this paper are as follows.

--- Critical related works and comparison are missing.

Mixed-size training of CNNs for image classification are not new. Here are some recent works, however they are missed by the authors.

“Resolution Adaptive Networks for Efficient Inference”, in CVPR 2020

 “Resolution Switchable Networks for Runtime Efficient Image Recognition”, in ECCV 2020.

“MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution”, in ECCV 2020.

Besides, as described by the authors, NeurIPS 2019 paper “Fixing the train-test resolution discrepancy” also considers how to enhance the performance of CNN models when applying them with different input image sizes. But a performance comparison is missing.

To show the advantages of this paper, thorough discussion and performance comparison with the above works are necessary. Taking ResNet-50 trained on ImageNet as an instance, I notice that the proposed method shows obviously worse accuracy compared to some of these works.

---Another critical baseline is also missing.

In page 8, “We note that for original fixed-size regimes this calibration procedure resulted, with degraded results and so we report accuracy without calibration for these models”. To my understanding, this is somewhat weird. Why BN calibration does not work on the other image sizes when the model is trained with a fixed image size? It is not clear. Furthermore, in NeurIPS 2019 paper “Fixing the train-test resolution discrepancy”, this line of methods work pretty well. Such a BN calibration should server as another baseline for more fair comparison.

---Regarding B+ design

How about the wall-clock training cost (in hours) instead of the number of iterations/epochs?
How about the performance of applying “scale the learning rate linearly” to train the baseline model?

---How about model transfer ability?

Only image classification tasks are considered in the experiments. How about the performance of the trained backbone models, when transferring them in the downstream tasks, such as object detection and semantic segmentation? Can performance gain be transferred?

---Others

Is there any principled way regarding the size sampling strategy? The current strategy is based on manual setting, which limits its use in real applications.

I suggest the authors to also provide precise accuracy numbers, etc. regarding some figures (e.g., figure 1, figure 4) shown in the paper.

Generally, I am on the fence, to this paper. I encourage the authors to address the questions I raised above.

The evidence to be evaluated is:Recently, Tan and Le (2019) showed a computation-vs-accuracy trade-off in applying scaling to the image size used for convolutional networks training and evaluation.

In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence.

Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed. This procedure reduced the train-test discrepancy caused by the change in image size and allowed faster training time and improved accuracy -- at the cost of additional fine-tuning procedure and additional computations at inference time. In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

As it is a common practice to use different image size for evaluation than that used for training (He et al., 2017; Lin et al., 2017; Redmon and Farhadi, 2018), it was observed by Touvron et al. (2019) and empirically verified (Figure 1) that classification accuracy may degrade above or below a certain size threshold.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\).

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

Furthermore, given a fixed budget of computational and time resources (per step), we can now modify our regime along spatial and batch axes.

MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size. This behavior may require hyper-parameter tuning and may also affect size-dependent layers such as batch normalization (Ioffe and Szegedy, 2015).

We observed progressive resizing to cause noticeable accuracy deterioration when switching one size for another (Appendix E), which may be attributed to the network tendency to adapt to a specific size.

We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy.

For example, in Table 2, the gradient variance is larger when computed over a small image size (unsurprisingly). This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training.

Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

\[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant.

While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\).

We present the final test accuracy, as well as the number of training steps for each model in Table 1. We see that our MixSize regimes yield two possible improvements:* Reduced number of training steps to achieve a similar test accuracy using \(B^{+}\) regime.  * Better test accuracy using \(D^{+}\) regime (enabling BA with same computational budget).

\(B+\) enables larger batches on average, hence the reduction in training steps, while \(D+\) leverages benefits of batch-augmentation to achieve better accuracy within the same compute.

We are first interested in accelerating the time needed for convergence of the tested models using our \(B^{+}\) scheme. We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F). As described earlier, this model requires the same time and computational resources as the baseline model. However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size.

Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

As the number of floating-point operations (flops) grows linearly with the number of pixels, using a mixed regime significantly improves accuracy per compute at evaluation.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size.

Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size.

Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases. We used 6 common networks (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017; Mahajan et al., 2018; Sandler et al., 2018) pretrained on the ImageNet dataset at a resolution of \(224\times 224\) and measured their accuracy on varying sizes.

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We can see that gradient smoothing helps to reduce gap between gradient norms at difference batch sizes and improves training and validation accuracy. We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

In both CIFAR10 (Figure 8) and ImageNet (Figure 9) sampling methods exhibit better accuracy than a progressive resizing regime. This is most apparent when the final size used at training is larger than target size for inference (on which test accuracy is measured). We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 1: Top-1 test accuracy per image size, models trained on specific sizes (ResNet50, ImageNet).

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Table 1: Test accuracy (Top-1) results for CIFAR and ImageNet. Each row represents models trained using the same computational and memory budget per step. Steps and accuracy are reported at the completion of a fixed epoch budget (i.e., 90 epochs for ResNet on ImageNet, 200 for ResNet on CIFAR). Accuracy is reported for model’s original size (32 for CIFAR, 224 for ImageNet).

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and claim, I find that the claim "reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy" is well-stated and supported by the evidence. The paper demonstrates this trade-off through:

1. Table 1 showing B+ regime reduces training steps while maintaining/slightly improving accuracy
2. Figure 2 and experimental results showing ~2.7x fewer training steps with comparable accuracy
3. Detailed empirical evidence across multiple datasets and architectures

The claim is carefully worded to reflect what is actually shown in the results, without overstating the benefits or making sweeping generalizations.

<score>1</score>

The reviewer's comments, while raising valid points about missing comparisons and additional experiments, don't directly challenge this specific claim about the training time vs accuracy trade-off. The evidence provides clear numerical support for this fundamental trade-off relationship.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address these issues, we jointly propose FedCVAE-Ens and FedCVAE-KD, two novel data-free one-shot FL models that reframe the local learning task using conditional variational autoencoders (CVAE).

The evidence to be evaluated is:A variational autoencoder (VAE) is a probabilistic generative model that attempts to learn the distribution of data samples (Kingma & Welling, 2014). A VAE is a latent variable method that models the joint distribution \(p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z})\) of a data sample \(\mathbf{x}\in\mathcal{X}\subseteq\mathbb{R}^{D}\) and latent variable \(\mathbf{z}\in\mathcal{Z}\subseteq\mathbb{R}^{d}\), where usually \(d<<D\).

Conditional VAEs (CVAEs) extend the basic VAE by conditioning the encoder and decoder on the one-hot encoding of the class label \(\mathbf{y}\in\mathcal{Y}=\{1,2,...,K\}\) corresponding to data sample \(\mathbf{x}\), resulting in the conditional encoder \(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\) and conditional decoder \(p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z},\mathbf{y})\).

\[\mathcal{L}(\mathbf{\phi},\mathbf{\theta};\mathbf{x},\mathbf{y})=-D_{\text{KL}}(q_{ \mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\mid\mid p(\mathbf{z}))+\mathbb{ E}_{q_{\mathbf{\theta}}(\mathbf{z}|\mathbf{x},\mathbf{y})}[\log p_{\mathbf{\theta}}( \mathbf{x}|\mathbf{z},\mathbf{y})], \tag{1}\] which bounds the conditional marginal likelihood of the data \(\log p(\mathbf{x}|\mathbf{y})\). Here, \(D_{\text{KL}}(\cdot)\) represents KL-divergence.

Knowledge distillation (KD) aims to extract information from a trained _teacher_ model to train a separate lightweight _student_ model (Bucilua et al., 2006; Hinton et al., 2015).

Several works have integrated KD into FL to mitigate privacy risks, reduce upload costs, or regularize local learning using an ensemble-of-teachers approach (Lin et al., 2020; Zhu et al., 2021; Guha et al., 2019). However, KD approaches in FL usually require an auxiliary public dataset with similar properties as the distributed dataset, which may be difficult to obtain in practice (Zhu et al., 2021).

We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space to a single student server decoder, parameterized by \(\mathbf{\theta}^{S}\). To achieve this, we generate \(n_{D}\) total KD training samples, with \(\mathcal{D}_{\text{Ens}}\) defined as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k}, \mathbf{z}_{i}^{k})\}_{i=1}^{\lfloor n_{D}/\rfloor}\). Then, we train the server to match the teacher's mapping of \((\mathbf{z}_{i}^{k},\mathbf{y}_{i}^{k})\) to \(\hat{\mathbf{x}}_{i}^{k}\) by minimizing a reconstruction loss: \[\ell_{\text{KD}}(\mathbf{\theta}^{S};\mathbf{z}^{k},\mathbf{y}^{k},\hat{\mathbf{x }}^{k})=g(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k}),\hat{\mathbf{x }}^{k}), \tag{2}\] which penalizes the dissimilarity \(g(\cdot)\) in data space between the synthetic data sample generated by the server decoder \(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k})\) and the client decoder sample \(\hat{\mathbf{x}}^{k}\).

Because FedCVAE-KD ensembles client decoders to create a labeled dataset \(\mathcal{D}_{\text{Ens}}\) to train the server decoder, each client can have a unique CVAE model architecture, accommodating each client's computational limitations.

FedCVAE-KD can be applied to any task that requires a labeled dataset, which is more general than classification; there is no commitment to a particular terminal task before learning occurs.

CVAEs use a prior distribution over latent space to train the encoder and decoder models. While a multivariate standard normal distribution is typically used for convenience, any normal distribution is acceptable. To promote security, we propose to shift the center of the prior distribution \(\mathbf{\mu}\) to a random position in real space (i.e., \(\mathbf{\mu}\in\mathbb{R}^{d}\)), which can be communicated offline or via encryption methods between server and clients (Zhou et al., 2020).

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

We base our CVAE architecture on Higgins et al. (2017).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

The proposed KD aggregation method used in FedCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure 8 in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C). As expected, FedCVAE-Ens bounds the performance of FedCVAE-KD, but KD-based aggregation still preserves client ensemble performance well across levels of \(\alpha\) for all datasets (Table 1). The upper bound accuracy established by FedCVAE-Ens is promising, but successful decoder aggregation may require a stronger KD procedure (e.g., feature-based KD as described in Gou et al. (2021)).

To simulate clients with diverse computational resources, we train both FedCVAE-Ens and FedCVAE-KD using two local CVAE architectures: the first as described in Section 4.1 and the second with one convolutional/deconvolutional block removed for the encoder/decoder, respectively. The server decoder matches that which is described in Section 4.1 but can be chosen arbitrarily. Because the two architectures demonstrate similar generative capabilities, final server classifier accuracy is very similar when comparing homogeneous against heterogeneous local architectures (Figure 4).4 Notably, our KD procedure for FedCVAE-KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge to a single decoder.

Therefore, this simple FedCVAE-Ens and FedCVAE-KD extension reduces eavesdropping attackers' capacity to extract high-quality samples from uploaded decoders or train a performant downstream model, reducing communication risks.

FL was originally proposed by McMahan et al. (2017) as a paradigm for decentralized distributed learning. Statistical heterogeneity quickly emerged as a core issue within FL, with many studies focusing on maintaining high performance under very non-IID data. Many approaches have experimented with augmenting FedAvg by adding proximal terms to the local objective as an attempt to restrain local updates (Li et al., 2020; Karimireddy et al., 2020; Wang et al., 2020; Acar et al., 2021; Li et al., 2021). Other works have used KD to circumvent issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary data (Lin et al., 2020; Sattler et al., 2021) or a generative model (Zhu et al., 2021; Zhang et al., 2022) to compactly capture client ensemble learning. Recent approaches focus on improving client selection strategy (Tang et al., 2022), reducing catastrophic forgetting to balance global knowledge against local learning (Huang et al., 2022), or improving the generality of client models (Mendieta et al., 2022). However, all of these methods are designed for standard FL and rely heavily on local regularization through substantial iterative communication, which is not feasible in the one-shot FL setting where communication is limited to a single round.

Methods in one-shot FL have demonstrated strong performance under substantial communication constraint. Guha et al. (2019) originally proposed one-shot FL and introduced two methods, one based on heuristic selection methods for client inclusion in the final ensemble and another that used KD with an auxiliary dataset for ensemble aggregation.

Although these methods obtain high accuracy, they rely on an auxiliary public dataset to support KD, which is inapplicable in data-free FL.

Zhang et al. (2021) proposed a data-free KD procedure based on a generator network trained using the ensemble of client classifiers, showing promising performance even under heterogeneous local models. However, existing approaches in one-shot FL either do not experiment with high statistical heterogeneity or degrade even under moderate levels of heterogeneity, unlike our proposed methods.

A few studies have experimented with using VAEs in FL.

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

* Zhang et al. (2021) Jie Zhang, Chen Chen, Bo Li, Linguuan Lyu, Shuang Wu, Jianghe Xu, Shouhong Ding, and Chao Wu. A practical data-free approach to one-shot federated learning with heterogeneity. _arXiv preprint arXiv:2112.12371_, 2021.

* Zhang et al. (2022) Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 10174-10183, June 2022.

FedCVAE-Ens follows the same procedure as FedCVAE-KD, but alternatively defines \(\mathcal{D}_{\text{Ens}}\) as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}:=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k})\}_ {i=1}^{c/m}\) and uses \(\mathcal{D}_{\text{Ens}}\) to directly train the server classifier. Figure 6 visualizes the full pipeline for FedCVAE-Ens and Algorithm 2 details the full procedure. Algorithm 3 shows the local client training procedure, which is the same for FedCVAE-Ens and FedCVAE-KD.

1:procedureServer 2:for each client \(k\in\mathcal{C}\) in parallel do 3:\(\mathbf{\theta}^{k},\;\hat{p}^{k}(\mathbf{y})\leftarrow\)ClientLocalUpdate(\(k\), \(T_{L}\)) 4: Generate samples from each client \(\mathcal{D}_{\text{Ens}}^{k}:=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k})\}_ {i=1}^{\lfloor nC/m\rfloor}\) using client decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\) and label distribution \(\hat{p}^{k}(\mathbf{y})\) 5: Combine client subsets into an IID labeled dataset \(\mathcal{D}_{\text{Ens}}:=\mathcal{D}_{\text{Ens}}^{1}\cup\mathcal{D}_{\text{ Ens}}^{2}\cup...\cup\mathcal{D}_{\text{Ens}}^{m}\) 6:for classifier epoch \(i=1\) to \(T_{C}\)do

```**Algorithm 2** - FedCVAE-Ens in the one-shot FL setting. \(T_{L}\) represents the number of local training epochs. The server classifier parameters are \(\mathbf{w}_{C}^{S}\), with training epochs \(T_{C}\), number of training samples \(n_{C}\), classification loss \(\ell_{C}(\cdot)\), and learning rate \(\eta_{C}\). \(\mathcal{C}\) is the set of clients.

van den Burg and Williams (2021) define a probabilistic generative model's propensity to reproduce samples observed in the training data as memorization, and prove that ensuring a particular level of differential privacy (DP) can bound memorization in probabilistic generative models (including CVAEs). Thus, beyond the normal privacy guarantees attributable to DP, employing FL DP-estimation techniques (i.e., Geyer et al. (2017)) also ensures low memorization across the FedCVAE-Ens and FedCVAE-KD pipelines, further enhancing the privacy of our proposed methods.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6). \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 6: The full pipeline for one of our proposed methods, FEDCVAE-ENS. Here, E, D, and C represent “encoder,” “decoder,” and “classifier” models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses the ensemble of uploaded client decoders and corresponding local label distributions to generate a labeled dataset of synthetic samples to train a classifier.

Input: Figure 4: Results with heterogeneous local models. “Homogeneous” uses the same CVAE architecture for all clients, whereas “heterogeneous” uses two architectures with similar generative capabilities.

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input: Figure 1: Motivating our proposed methods, FEDCVAE-ENS and FEDCVAE-KD, using the MNIST dataset as an example. In cases of very high statistical heterogeneity, each client will only observe one or two of the ten available classes, as seen on the left where the size of each dot is proportional to the number of samples. For example, client 2 only observes 4’s and 7’s, resulting in a client decoder that can expertly generate these digits. Note that the columns are shown in order of conditioning class (digits 0-9). Similarly, client 4 is an expert in 3’s and 6’s. In FEDCVAE-KD, our lightweight knowledge distillation training procedure compacts local learning into a single server decoder, as evidenced by the high-quality samples from all available classes (digits 0-9). This server decoder can then be used for any downstream task, e.g., classification.

Input: Figure 7: CVAE samples generated using latent vectors distant from the center of the multivariate normal prior (top row) and close to the center (bottom row).

Input: Figure 2: The full pipeline for one of our proposed methods, FEDCVAE-KD. Here, E, D, and C represent “encoder,” “decoder,” and “classifier” models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses uploaded client decoder parameters and local label distributions to train a server decoder using knowledge distillation (KD). Finally, synthetic labeled samples from the server decoder are used to train a classifier.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, this appears to be a well-stated claim that accurately reflects what the research demonstrates. The evidence comprehensively shows that the authors developed two novel CVAE-based methods (FedCVAE-Ens and FedCVAE-KD) for data-free one-shot federated learning. The extensive experimental results, mathematical formulations, and performance comparisons support that these methods effectively reframe local learning using CVAEs. The evidence demonstrates superior performance across different datasets, heterogeneity levels, and client numbers compared to baselines. The claim avoids exaggeration and matches the scope and limitations presented in the evidence.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address these issues, we jointly propose FedCVAE-Ens and FedCVAE-KD, two novel data-free one-shot FL models that reframe the local learning task using conditional variational autoencoders (CVAE).

The review comment to be evaluated is:

Strength:

-- The proposed solutions seem effective under the one-shot FL setting. The efficacy and efficiency are both demonstrated in the extensive experiments.
-- The writing is clear and easy to follow. The tables and figures are well-organized.

Weaknesses:

-- Constructing a generative model to generate samples for FL model training is not new. There are a few works utilizing a similar strategy in FL, for example:
Data-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021
Fine-tuning global model via data-free knowledge distillation for non-iid federated learning, CVPR 2022
Please make a comparison with those works

-- Some details are not clear enough for me. For example, in Eq 2, what is the formulation for g(.)?  Is this dissimilarity measured in the data-space or feature space?

The evidence to be evaluated is:A variational autoencoder (VAE) is a probabilistic generative model that attempts to learn the distribution of data samples (Kingma & Welling, 2014). A VAE is a latent variable method that models the joint distribution \(p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z})\) of a data sample \(\mathbf{x}\in\mathcal{X}\subseteq\mathbb{R}^{D}\) and latent variable \(\mathbf{z}\in\mathcal{Z}\subseteq\mathbb{R}^{d}\), where usually \(d<<D\).

Conditional VAEs (CVAEs) extend the basic VAE by conditioning the encoder and decoder on the one-hot encoding of the class label \(\mathbf{y}\in\mathcal{Y}=\{1,2,...,K\}\) corresponding to data sample \(\mathbf{x}\), resulting in the conditional encoder \(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\) and conditional decoder \(p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z},\mathbf{y})\).

\[\mathcal{L}(\mathbf{\phi},\mathbf{\theta};\mathbf{x},\mathbf{y})=-D_{\text{KL}}(q_{ \mathbf{\phi}}(\mathbf{z}|\mathbf{x},\mathbf{y})\mid\mid p(\mathbf{z}))+\mathbb{ E}_{q_{\mathbf{\theta}}(\mathbf{z}|\mathbf{x},\mathbf{y})}[\log p_{\mathbf{\theta}}( \mathbf{x}|\mathbf{z},\mathbf{y})], \tag{1}\] which bounds the conditional marginal likelihood of the data \(\log p(\mathbf{x}|\mathbf{y})\). Here, \(D_{\text{KL}}(\cdot)\) represents KL-divergence.

Knowledge distillation (KD) aims to extract information from a trained _teacher_ model to train a separate lightweight _student_ model (Bucilua et al., 2006; Hinton et al., 2015).

Several works have integrated KD into FL to mitigate privacy risks, reduce upload costs, or regularize local learning using an ensemble-of-teachers approach (Lin et al., 2020; Zhu et al., 2021; Guha et al., 2019). However, KD approaches in FL usually require an auxiliary public dataset with similar properties as the distributed dataset, which may be difficult to obtain in practice (Zhu et al., 2021).

We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

The trained client decoders act as the teacher models, conveying their aggregate knowledge of how to map from latent space to data space to a single student server decoder, parameterized by \(\mathbf{\theta}^{S}\). To achieve this, we generate \(n_{D}\) total KD training samples, with \(\mathcal{D}_{\text{Ens}}\) defined as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k}, \mathbf{z}_{i}^{k})\}_{i=1}^{\lfloor n_{D}/\rfloor}\). Then, we train the server to match the teacher's mapping of \((\mathbf{z}_{i}^{k},\mathbf{y}_{i}^{k})\) to \(\hat{\mathbf{x}}_{i}^{k}\) by minimizing a reconstruction loss: \[\ell_{\text{KD}}(\mathbf{\theta}^{S};\mathbf{z}^{k},\mathbf{y}^{k},\hat{\mathbf{x }}^{k})=g(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k}),\hat{\mathbf{x }}^{k}), \tag{2}\] which penalizes the dissimilarity \(g(\cdot)\) in data space between the synthetic data sample generated by the server decoder \(D_{\mathbf{\theta}^{S}}(\mathbf{z}^{k};\mathbf{y}^{k})\) and the client decoder sample \(\hat{\mathbf{x}}^{k}\).

Because FedCVAE-KD ensembles client decoders to create a labeled dataset \(\mathcal{D}_{\text{Ens}}\) to train the server decoder, each client can have a unique CVAE model architecture, accommodating each client's computational limitations.

FedCVAE-KD can be applied to any task that requires a labeled dataset, which is more general than classification; there is no commitment to a particular terminal task before learning occurs.

CVAEs use a prior distribution over latent space to train the encoder and decoder models. While a multivariate standard normal distribution is typically used for convenience, any normal distribution is acceptable. To promote security, we propose to shift the center of the prior distribution \(\mathbf{\mu}\) to a random position in real space (i.e., \(\mathbf{\mu}\in\mathbb{R}^{d}\)), which can be communicated offline or via encryption methods between server and clients (Zhou et al., 2020).

To validate FedCVAE-Ens and FedCVAE-KD, we conduct experiments on three image datasets that are standard in the FL literature: MNIST (Lecun et al., 1998), Fashion-MNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011).3

We compare the performance of FedCVAE-Ens and FedCVAE-KD in the one-shot data-free FL setting against two existing methods: FedAvg (McMahan et al., 2017) and a method proposed in Guha et al. (2019), which we call FedOneShot.

The one-shot methods proposed in Li et al. (2021) and Shin et al. (2020) are not applicable because of their reliance on public auxiliary data for server-side training or fine-tuning. Similarly, many standard FL methods are not appropriate because they depend on an auxiliary dataset (i.e., FedDF (Lin et al., 2020)) or focus on regularization (i.e., FedGen (Zhu et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2020), FedNova (Wang et al., 2020)), which is incompatible with the one-shot setting.

We base our CVAE architecture on Higgins et al. (2017).

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN.

The proposed KD aggregation method used in FedCVAE-KD substantially improves on aggregation via parameter averaging, generating qualitatively more realistic samples (Figure 8 in Appendix C) and achieving higher server classifier accuracy (Table 6 in Appendix C). As expected, FedCVAE-Ens bounds the performance of FedCVAE-KD, but KD-based aggregation still preserves client ensemble performance well across levels of \(\alpha\) for all datasets (Table 1). The upper bound accuracy established by FedCVAE-Ens is promising, but successful decoder aggregation may require a stronger KD procedure (e.g., feature-based KD as described in Gou et al. (2021)).

To simulate clients with diverse computational resources, we train both FedCVAE-Ens and FedCVAE-KD using two local CVAE architectures: the first as described in Section 4.1 and the second with one convolutional/deconvolutional block removed for the encoder/decoder, respectively. The server decoder matches that which is described in Section 4.1 but can be chosen arbitrarily. Because the two architectures demonstrate similar generative capabilities, final server classifier accuracy is very similar when comparing homogeneous against heterogeneous local architectures (Figure 4).4 Notably, our KD procedure for FedCVAE-KD still performs well using heterogeneous models, indicating diverse architectures can organize latent space similarly enough to successfully translate this knowledge to a single decoder.

Therefore, this simple FedCVAE-Ens and FedCVAE-KD extension reduces eavesdropping attackers' capacity to extract high-quality samples from uploaded decoders or train a performant downstream model, reducing communication risks.

FL was originally proposed by McMahan et al. (2017) as a paradigm for decentralized distributed learning. Statistical heterogeneity quickly emerged as a core issue within FL, with many studies focusing on maintaining high performance under very non-IID data. Many approaches have experimented with augmenting FedAvg by adding proximal terms to the local objective as an attempt to restrain local updates (Li et al., 2020; Karimireddy et al., 2020; Wang et al., 2020; Acar et al., 2021; Li et al., 2021). Other works have used KD to circumvent issues associated with parameter averaging under non-IID data, focusing on leveraging auxiliary data (Lin et al., 2020; Sattler et al., 2021) or a generative model (Zhu et al., 2021; Zhang et al., 2022) to compactly capture client ensemble learning. Recent approaches focus on improving client selection strategy (Tang et al., 2022), reducing catastrophic forgetting to balance global knowledge against local learning (Huang et al., 2022), or improving the generality of client models (Mendieta et al., 2022). However, all of these methods are designed for standard FL and rely heavily on local regularization through substantial iterative communication, which is not feasible in the one-shot FL setting where communication is limited to a single round.

Methods in one-shot FL have demonstrated strong performance under substantial communication constraint. Guha et al. (2019) originally proposed one-shot FL and introduced two methods, one based on heuristic selection methods for client inclusion in the final ensemble and another that used KD with an auxiliary dataset for ensemble aggregation.

Although these methods obtain high accuracy, they rely on an auxiliary public dataset to support KD, which is inapplicable in data-free FL.

Zhang et al. (2021) proposed a data-free KD procedure based on a generator network trained using the ensemble of client classifiers, showing promising performance even under heterogeneous local models. However, existing approaches in one-shot FL either do not experiment with high statistical heterogeneity or degrade even under moderate levels of heterogeneity, unlike our proposed methods.

A few studies have experimented with using VAEs in FL.

We are the first to apply CVAEs to one-shot FL with a focus on high statistical heterogeneity. In this paper, we proposed FedCVAE-Ens and FedCVAE-KD, data-free one-shot FL methods that reframe the local learning task using CVAEs. Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

* Zhang et al. (2021) Jie Zhang, Chen Chen, Bo Li, Linguuan Lyu, Shuang Wu, Jianghe Xu, Shouhong Ding, and Chao Wu. A practical data-free approach to one-shot federated learning with heterogeneity. _arXiv preprint arXiv:2112.12371_, 2021.

* Zhang et al. (2022) Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 10174-10183, June 2022.

FedCVAE-Ens follows the same procedure as FedCVAE-KD, but alternatively defines \(\mathcal{D}_{\text{Ens}}\) as the combination of client subsets \(\mathcal{D}_{\text{Ens}}^{k}:=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k})\}_ {i=1}^{c/m}\) and uses \(\mathcal{D}_{\text{Ens}}\) to directly train the server classifier. Figure 6 visualizes the full pipeline for FedCVAE-Ens and Algorithm 2 details the full procedure. Algorithm 3 shows the local client training procedure, which is the same for FedCVAE-Ens and FedCVAE-KD.

1:procedureServer 2:for each client \(k\in\mathcal{C}\) in parallel do 3:\(\mathbf{\theta}^{k},\;\hat{p}^{k}(\mathbf{y})\leftarrow\)ClientLocalUpdate(\(k\), \(T_{L}\)) 4: Generate samples from each client \(\mathcal{D}_{\text{Ens}}^{k}:=\{(\hat{\mathbf{x}}_{i}^{k},\mathbf{y}_{i}^{k})\}_ {i=1}^{\lfloor nC/m\rfloor}\) using client decoder \(D_{\mathbf{\theta}^{k}}(\cdot)\) and label distribution \(\hat{p}^{k}(\mathbf{y})\) 5: Combine client subsets into an IID labeled dataset \(\mathcal{D}_{\text{Ens}}:=\mathcal{D}_{\text{Ens}}^{1}\cup\mathcal{D}_{\text{ Ens}}^{2}\cup...\cup\mathcal{D}_{\text{Ens}}^{m}\) 6:for classifier epoch \(i=1\) to \(T_{C}\)do

```**Algorithm 2** - FedCVAE-Ens in the one-shot FL setting. \(T_{L}\) represents the number of local training epochs. The server classifier parameters are \(\mathbf{w}_{C}^{S}\), with training epochs \(T_{C}\), number of training samples \(n_{C}\), classification loss \(\ell_{C}(\cdot)\), and learning rate \(\eta_{C}\). \(\mathcal{C}\) is the set of clients.

van den Burg and Williams (2021) define a probabilistic generative model's propensity to reproduce samples observed in the training data as memorization, and prove that ensuring a particular level of differential privacy (DP) can bound memorization in probabilistic generative models (including CVAEs). Thus, beyond the normal privacy guarantees attributable to DP, employing FL DP-estimation techniques (i.e., Geyer et al. (2017)) also ensures low memorization across the FedCVAE-Ens and FedCVAE-KD pipelines, further enhancing the privacy of our proposed methods.

While FedAvg and FedOneShot are very sensitive to dataset split (high standard deviation), the results in Table 5 for both FedCVAE-Ens and FedCVAE-KD are not only consistent with Table 2, but also exhibit reasonable stability (low standard deviation).

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6). \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 6: The full pipeline for one of our proposed methods, FEDCVAE-ENS. Here, E, D, and C represent “encoder,” “decoder,” and “classifier” models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses the ensemble of uploaded client decoders and corresponding local label distributions to generate a labeled dataset of synthetic samples to train a classifier.

Input: Figure 4: Results with heterogeneous local models. “Homogeneous” uses the same CVAE architecture for all clients, whereas “heterogeneous” uses two architectures with similar generative capabilities.

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input: Figure 1: Motivating our proposed methods, FEDCVAE-ENS and FEDCVAE-KD, using the MNIST dataset as an example. In cases of very high statistical heterogeneity, each client will only observe one or two of the ten available classes, as seen on the left where the size of each dot is proportional to the number of samples. For example, client 2 only observes 4’s and 7’s, resulting in a client decoder that can expertly generate these digits. Note that the columns are shown in order of conditioning class (digits 0-9). Similarly, client 4 is an expert in 3’s and 6’s. In FEDCVAE-KD, our lightweight knowledge distillation training procedure compacts local learning into a single server decoder, as evidenced by the high-quality samples from all available classes (digits 0-9). This server decoder can then be used for any downstream task, e.g., classification.

Input: Figure 7: CVAE samples generated using latent vectors distant from the center of the multivariate normal prior (top row) and close to the center (bottom row).

Input: Figure 2: The full pipeline for one of our proposed methods, FEDCVAE-KD. Here, E, D, and C represent “encoder,” “decoder,” and “classifier” models, respectively. First, clients train CVAEs on their private local datasets. Then, the server uses uploaded client decoder parameters and local label distributions to train a server decoder using knowledge distillation (KD). Finally, synthetic labeled samples from the server decoder are used to train a classifier.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears well-stated. The paper thoroughly demonstrates how FedCVAE-Ens and FedCVAE-KD reframe local learning using CVAEs in one-shot FL. The evidence shows:
- Detailed mathematical formulation of the CVAE approach
- Comprehensive experimental results showing superior performance over baselines
- Clear diagrams explaining the model architecture and pipeline
- Extensive validation across multiple datasets and heterogeneity levels
- Acknowledgment of limitations and potential improvements

The authors avoid overstatement by providing concrete evidence for each aspect of their claim and acknowledging where improvements could be made.

<score>1</score>