Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves \(20\%-40\%\) reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss.

The review comment to be evaluated is:

Pros:
1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications.

2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks.

Cons:
1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a "speed" column in every table of this submission, to compare the actual inference speed among the methods.

2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?

3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models?

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).

Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Many adaptive computation methods have been recently proposed with the goal of improving efficiency (Bengio et al., 2015, 2013; Veit and Belongie, 2018; Wang et al., 2018; Graves, 2016; Meng et al., 2021). Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Yu et al., 2018; Figurnov et al., 2017; McGill and Perona, 2017; Teerapittayanon et al., 2016) Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training. Veit et al. (Veit and Belongie, 2018) propose a block skipping method controlled by samples from a Gumbel softmax, while Wu et al. (Wu et al., 2018) develop a reinforcement learning approach to achieve this goal. Adaptive computation time for recurrent neural networks is also presented in (Graves, 2016). SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D).

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation.

As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We evaluate the performance of our approach using three video action recognition datasets, namely Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira and Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB-21 (Jhuang et al., 2013). Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, here's my evaluation:

The claim states a "20%-40% reduction in computation (FLOPs) without any performance loss" across multiple datasets and tasks. The evidence partially supports this through tables showing FLOPs reductions:

- Table 10: ~28% reduction (20.0 vs 27.7 GFLOPs) for 8-frame with slight performance gain
- Table 5: ~27% reduction for R(2+1)D model (40.3 vs 55.2 GFLOPs) with maintained/improved accuracy
- Table 11: ~26% reduction (7.8 vs 10.6 GFLOPs) with maintained IoU

However, the reviewer raises a valid concern that FLOPs alone isn't a reliable speed metric. The claim would be more precise if it specified "FLOPs reduction" rather than implying general "computation reduction."

<score>2</score>

Justification:
1. The 20-40% FLOPs reduction is supported by data across multiple models and tasks
2. Performance is maintained or improved in most cases
3. The claim could be more precise by specifically mentioning "FLOPs reduction" rather than general "computation"
4. Multiple datasets are indeed tested as claimed
5. The experimental evidence is thorough and well-documented
6. The reviewer's concern about FLOPs vs actual speed is valid but doesn't invalidate the specific FLOPs-related results

The claim is slightly overstated but largely accurate, earning a score of 2.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves \(20\%-40\%\) reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss.

The review comment to be evaluated is:

Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).

Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Many adaptive computation methods have been recently proposed with the goal of improving efficiency (Bengio et al., 2015, 2013; Veit and Belongie, 2018; Wang et al., 2018; Graves, 2016; Meng et al., 2021). Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Yu et al., 2018; Figurnov et al., 2017; McGill and Perona, 2017; Teerapittayanon et al., 2016) Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training. Veit et al. (Veit and Belongie, 2018) propose a block skipping method controlled by samples from a Gumbel softmax, while Wu et al. (Wu et al., 2018) develop a reinforcement learning approach to achieve this goal. Adaptive computation time for recurrent neural networks is also presented in (Graves, 2016). SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D).

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation.

As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We evaluate the performance of our approach using three video action recognition datasets, namely Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira and Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB-21 (Jhuang et al., 2013). Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim "Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves 20%-40% reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss."

Summary:
The claim is somewhat overstated. While the evidence shows significant FLOP reductions across multiple datasets and tasks (Tables 5, 10, 11), the exact 20-40% range isn't consistently demonstrated across all comparisons. For example:
- R(2+1)D: ~27% reduction (40.3 vs 55.2 GFLOPs) for 16-frame
- I3D: ~43% reduction (32.1 vs 56.0 GFLOPs) for 16-frame
- X3D: ~16% reduction (5.38 vs 6.42 GFLOPs)

The "without any performance loss" claim is generally supported by the data, but there are slight variations in performance metrics. The review comment also points out that the core idea isn't entirely novel, referencing AR-Net as a predecessor.

<score>3</score>

This moderate score (3) reflects that while the claim has merit and is supported by substantial evidence, the specific percentage range claimed (20-40%) isn't consistently demonstrated across all experiments, and the novelty aspect is questioned in the review comments.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The set of generated images provides an intuitive visual summary of the tabular data that can be quickly and easily communicated and understood by the user.

The evidence to be evaluated is:We aim to generate images describing the underlying segment/trend of the given table.

Our goal can hence be described as, given a tabular input \(y\in\mathcal{L}^{m\times n}\), we aim to generate an informative and high-quality image \(\widehat{x}\in\mathbb{N}^{h\times w\times d}\), where \(h,w,d\) is the height, width and depth that resembles real-world images and contains relevant information about \(y\). In order to achieve our goal, one possible approach is to assume that the real-world images \(x\) and the tables \(y\) follow a joint distribution \(p(x,y)\), therefore given a table \(y\), we can model the mapping rule \(h:y\to x\) by the conditional distribution \(p(x|y)\), which can be learned by constructing a training dataset \(\{(y_{i},x_{i})\}_{i=1}^{n}\). After training on the dataset, we thus obtain a table-to-image model \(\hat{h}:y\to x\) approximating the true conditional distribution, and we can then use it to infer a set of images for any new unseen tabular dataset \(y_{\text{new}}\) as \(\widehat{x}=\hat{h}(y_{\text{new}})\).

With this procedure, we defined 300 table-image pairs across 6 categories: meat, wine, sweet, fish, gold, fruit, each with 50 table-image pairs.

For a given table \(y\in\mathcal{L}^{m\times n}\) generated based on the process described before, we directly input the pivot column names to DallE (Ramesh et al., 2021, 2022) and select the top images generated based on their CLIP score with respect to the input column name. For example, given a table whose pivot column name is "MntFruits", we will directly input "MntFruits" into DallE to generate images.

To understand the utility and effectiveness of the proposed approaches for this new problem, we designed experiments to quantitatively evaluate the quality and diversity of the images generated from the various approaches. Furthermore, we also show the actual images generated from the proposed models and provide a detailed discussion on some of the interesting insights observed.

However, DALLE-Tabbie is able to generate images that are significantly more diverse as shown in Figure 4. Notably, the images are even diverse in terms of style, that is, realistic images as well as those that appear to be drawn, to even those that are a hybrid with some realistic and artistic features and colors.

Input: Figure 4: Comparison of the diversity of the images generated by the proposed approaches.

Input: Figure 3: Qualitative comparison of the images generated from the different approaches.

Input: Figure 1: Tabular Dataset to Image Generation for Interactive Exploratory Data Analysis. See text for detailed discussion.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, this appears to be somewhat overstated. While the evidence does show that the system can generate images related to tabular data, several key limitations exist:

1. The evidence only demonstrates basic image generation for simple column names/categories (like "fish" or "fruits"), not complex tabular relationships or trends.
2. The paper shows limited evaluation of user understanding or communication effectiveness.
3. No user studies or metrics are presented to support claims about "quick and easy" comprehension.
4. The sample size of 300 table-image pairs across just 6 categories is relatively small.
5. The images shown vary in style and quality, which could affect clarity of communication.

The claim goes beyond what the evidence directly supports regarding intuitiveness and ease of understanding.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The set of generated images provides an intuitive visual summary of the tabular data that can be quickly and easily communicated and understood by the user.

The review comment to be evaluated is:

The main idea of the paper is interesting. But the motivation what we could benefit from the generated images is not very clear.

There are some questions:
1. The authors need to provide more examples when the generated images help. The reviewer is confusing that we can generate an image for each sample or only for a whole set of tabular data.
2. The authors may consider providing more illustration examples in the paper. The current illustration images seem to be subjective.
3. Does the ability of the method depend on the quality of the pre-trained model?
4. In some real-world tasks, we may not know the exact meaning of the column, and it is difficult to find the pre-trained model in the specific area. For example, if the table is related to housing prices or climate change, could the authors show the corresponding images?

The evidence to be evaluated is:We aim to generate images describing the underlying segment/trend of the given table.

Our goal can hence be described as, given a tabular input \(y\in\mathcal{L}^{m\times n}\), we aim to generate an informative and high-quality image \(\widehat{x}\in\mathbb{N}^{h\times w\times d}\), where \(h,w,d\) is the height, width and depth that resembles real-world images and contains relevant information about \(y\). In order to achieve our goal, one possible approach is to assume that the real-world images \(x\) and the tables \(y\) follow a joint distribution \(p(x,y)\), therefore given a table \(y\), we can model the mapping rule \(h:y\to x\) by the conditional distribution \(p(x|y)\), which can be learned by constructing a training dataset \(\{(y_{i},x_{i})\}_{i=1}^{n}\). After training on the dataset, we thus obtain a table-to-image model \(\hat{h}:y\to x\) approximating the true conditional distribution, and we can then use it to infer a set of images for any new unseen tabular dataset \(y_{\text{new}}\) as \(\widehat{x}=\hat{h}(y_{\text{new}})\).

With this procedure, we defined 300 table-image pairs across 6 categories: meat, wine, sweet, fish, gold, fruit, each with 50 table-image pairs.

For a given table \(y\in\mathcal{L}^{m\times n}\) generated based on the process described before, we directly input the pivot column names to DallE (Ramesh et al., 2021, 2022) and select the top images generated based on their CLIP score with respect to the input column name. For example, given a table whose pivot column name is "MntFruits", we will directly input "MntFruits" into DallE to generate images.

To understand the utility and effectiveness of the proposed approaches for this new problem, we designed experiments to quantitatively evaluate the quality and diversity of the images generated from the various approaches. Furthermore, we also show the actual images generated from the proposed models and provide a detailed discussion on some of the interesting insights observed.

However, DALLE-Tabbie is able to generate images that are significantly more diverse as shown in Figure 4. Notably, the images are even diverse in terms of style, that is, realistic images as well as those that appear to be drawn, to even those that are a hybrid with some realistic and artistic features and colors.

Input: Figure 4: Comparison of the diversity of the images generated by the proposed approaches.

Input: Figure 3: Qualitative comparison of the images generated from the different approaches.

Input: Figure 1: Tabular Dataset to Image Generation for Interactive Exploratory Data Analysis. See text for detailed discussion.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the provided evidence and review comments, I'll evaluate the claim that "the set of generated images provides an intuitive visual summary of the tabular data that can be quickly and easily communicated and understood by the user."

The evidence shows significant limitations and gaps that make this claim overstated:

1. Limited example scope - Only 300 table-image pairs across 6 specific categories were used, making generalizability questionable.

2. The reviewer points out valid concerns about:
- Lack of clarity on whether images are generated per sample or for whole datasets
- Subjective nature of current illustrations
- Uncertainty about performance in real-world tasks with ambiguous column meanings
- Dependency on pre-trained models' quality

3. The paper focuses more on technical implementation than demonstrating actual user understanding/communication benefits.

While the figures show the system can generate diverse images, there's insufficient evidence that these images actually improve user understanding or communication of tabular data.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines.

The review comment to be evaluated is:

Strength:

-- The proposed solutions seem effective under the one-shot FL setting. The efficacy and efficiency are both demonstrated in the extensive experiments.
-- The writing is clear and easy to follow. The tables and figures are well-organized.

Weaknesses:

-- Constructing a generative model to generate samples for FL model training is not new. There are a few works utilizing a similar strategy in FL, for example:
Data-Free Knowledge Distillation for Heterogeneous Federated Learning, ICML 2021
Fine-tuning global model via data-free knowledge distillation for non-iid federated learning, CVPR 2022
Please make a comparison with those works

-- Some details are not clear enough for me. For example, in Eq 2, what is the formulation for g(.)?  Is this dissimilarity measured in the data-space or feature space?

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines" appears to be well-stated. The evidence from Table 1 shows that at α=0.001 (highest heterogeneity), FedCVAE-KD achieves >1.75x accuracy for MNIST, >2x for FashionMNIST, and nearly 2.75x for SVHN compared to baselines. FedCVAE-Ens performs even better. The claim is conservative in stating "typically more than double" when in some cases the improvement is nearly triple.

<score>1</score>

Justification:
1. The claim is directly supported by quantitative evidence from Table 1
2. The wording "typically more than double" is actually understated given the evidence shows:
   - MNIST: 82.24% vs 45.12% (FedAvg) - ~1.82x improvement
   - FashionMNIST: 69.53% vs 32.77% - ~2.12x improvement
   - SVHN: 55.48% vs 20.31% - ~2.73x improvement
3. The review comments do not challenge this specific performance claim
4. The claim acknowledges the context ("at the highest levels of statistical heterogeneity") rather than making a blanket statement
5. The results are consistent across multiple datasets and experiments, strengthening the validity


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines.

The review comment to be evaluated is:

The paper presents a relatively simple idea and is mostly clearly written. As far as I am aware, this method is novel in the context of FL. The authors also provide a reasonable amount of information, which should make FedCVAE reproducible. The authors seem to mostly tackle the cross-silo setting, where each client has enough data. One thing I noticed is that the authors mention that the conditional VAE (CVAE) objective at eq. 1 bounds the marginal likelihood $\log p(x)$; this is not correct, as the specific objective at eq. 1 bounds the conditional marginal likelihood $\log p(x|y)$.

Besides that, there are a couple of things that are, in my opinion, not properly motivated. More specifically, the authors propose to sample from the generative model and train a classifier at the server. Why is this necessary and why you do not simply use the generative classifier  $p(y|x) = \frac{\int p(x| z, y)p(z)dz p(y)}{p(x)}$? To me it seems that the target for the auxiliary classifier is nothing else, but to approximate this generative classifier. The loss for the auxiliary classifier (in the limit where you constantly sample from the generative model) can be obtained by minimising the average KL-divergence between the true CVAE generative posterior $p(y|x)$ and the classifier predictive $q(y|x)$. More specifically: $$E_p(x)[KL(p(y|x) || q(y|x))] = - E_p(y,x) [\log q(y|x)] + const = -E_p(y,z,x)[\log q(y|x)]  + const = -E_p(y)p(z)p(x|y,z)[\log q(y|x)] + const$$ In this sense, it seems that 1) it might not be necessary to train an auxiliary classifier and 2) the modelling choices made in the generative model itself are important for the performance of the actual downstream classifier. I believe a discussion about this is warranted to be added in the paper. Apart from that, when the authors want to sample datapoints from the model they actually use a different distribution over the latents than the one the model was actually trained; e.g., using a $z \sim U[-1, 1]$ instead of $z\sim N(0, I)$. They mention that it is beneficial to stay at a high density region of the prior, but how does the performance of the algorithm fare when you use the correct prior? Why not use e.g., $z \sim N(0, \sigma)$ with $\sigma < 1$ which is closer to the correct prior?

Another thing that could be better explained are the security and privacy aspects of this method, especially given the FL scenario. FedCVAE uploads the CVAE decoder along with $p(y)$ to the server, which is inherently less private than FedAvg. While it is true that with DP you can bound memorisation and get privacy, at the moment the authors do not use anything and the gains in performance when doing DP relatively to e.g., DP-FedAvg are not clear. In fact, given a sufficient powerful generative model, this method should be indistinguishable from sending the actual data. Furthermore, the arguments about security by shifting $\mu$, need to be more formal. Are there some guarantees that you can give, given a sufficient powerful adversary? For example, given $p(y)$, and $p(x|z, y)$, couldn’t the attacker just try to infer a global mean for $z$? For example, by visual inspection of the samples or an optimization procedure? And a small comment; $U[-10, 10]$ is not an exceptionally good guess for $N(0, I)$ (which is what the text mentions at the beginning of page 9) as 99% of the probability of $N(0,I)$ is in $[-3, 3]$, so 70% of the mass of the uniform proposal is on almost zero probability mass under $N(0, I)$.

Furthermore, this method requires building good generative models which is generally a harder task than building a good discriminative model. How does this work on more complex datasets, such as CIFAR10 and CIFAR100, where $p(x|y, z)$ might not be as sharp? Furthermore, while non-iid settings are nice to have, to me it seems that this method might suffer in less non-iid / iid settings (e.g, $\alpha=1.0$, $\alpha=\inf$) ; the generative distribution is more diverse and thus it would be harder to model accurately with a, hardware constrained, VAE model. This can also sort of be seen when comparing FedCVAE-KD to FedCVAE-Ens; the former has worse performance that the latter, given that the entire training distribution needs to be modelled by a single model. How does the performance of the method scale with higher values of $\alpha$? Besides that, it seems that the evaluation against the baselines is not entirely apples-to-apples; for example, FedOneShot does not need to reveal to the server the labels and the models trained on the local data, so it has fundamentally different security / privacy tradeoffs.

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Looking at the claim and evidence carefully:

The claim states that FedCVAE-Ens and FedCVAE-KD "typically more than double the accuracy of the baselines" at the highest levels of statistical heterogeneity.

From Table 1, at α=0.001 (highest heterogeneity):
- MNIST: FedCVAE-KD (82.24%) vs best baseline (45.12%) = 1.82x
- FashionMNIST: FedCVAE-KD (69.53%) vs best baseline (32.77%) = 2.12x
- SVHN: FedCVAE-KD (55.48%) vs best baseline (20.31%) = 2.73x

The evidence supports that performance improvements are consistently above or close to 2x across datasets. The claim is well-stated and accurately reflects the empirical results without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines.

The review comment to be evaluated is:

The paper is very well written and easy to follow. The numerical experiments are well executed and show promising performance achieved by the proposed methods.

That being said, there are some downsides that, in my opinion, slightly outweigh the strengths of the paper. On the problem formulation side, I believe the one-shot federated learning (FL) problem is not very well motivated; the point of FL is to enable training models at different clients by leveraging each others' data through communication without explicitly sharing them. In a one-shot scenario, the problem essentially reduces to having a set of models, which need to be aggregated into a single, universal model, so I am still not convinced how the "federated" aspect of the problem is of the essence.

On the contributions side, the main issue I have with the paper is a lack of a central, key novelty. In its current form, the paper is mostly a collection of different ideas, namely two algorithms that are almost independent of each other, as well as a security aspect, which is again mostly orthogonal to the proposed methods. To me, what would make this paper much stronger is to focus on only one of these algorithms/aspects throughout the paper. For example, since the performance of FEDCVAE-KD is better than baselines, I believe it would be better to focus on this algorithm in the main body of the paper, and add the ensemble version of the algorithm (which, as the authors allude to, requires a higher amount of memory that scales with the number of clients) as an extension, in a separate discussion section toward the end of the paper and/or in the appendix. Moreover, the security and privacy considerations are definitely important, but including them in the main body of the paper stretches it too thin, in my opinion.

Further comments:

- Aside from the solid empirical performance, are there any intuitive/theoretical explanations as to why the proposed methods are well suited for i) *one-shot* FL, and ii) *high heterogeneity* FL? As presented, I cannot see a clear motivation for why the proposed methods should perform better in the considered problem formulation.
- Please discuss whether or not sharing local label distributions with the server violates the privacy of clients' data. How do the proposed methods perform if this information is unavailable at the server?
- Presenting the flowchart figure for only one of the proposed methods (FEDCVAE-KD) in Figure 2 and not including any of the algorithms for the two proposed methods makes the paper slightly incoherent, without switching back and forth between the body of the paper and the appendix. I suggest rearranging the contents to remedy this issue as best as possible by at least having both the figure and algorithm for one of the methods or having the figures for both methods in the main body of the paper.
- In the first footnote on page 4, why is a uniform distribution used for sampling $\mathbf{z}$? Why not, for example, use a truncated Gaussian distribution?
- In my opinion, including FedAVG is slightly unfair in the experiments. Under FedAVG, a set of client model parameters $\\{\mathbf{w}^k\\}_{k=1}^m$ are aggregated using a simple average, i.e., $\mathbf{w}=\frac{1}{m} \sum_\{k=1\}^m\mathbf{w}^k$. In a very heterogeneous case, each of these model parameters is the minimizer of a very different objective function over an entirely different optimization landscape, so it is completely expected that a linear averaging of these model parameters should lead to inferior performance, especially if the averaging operation is done only once.
- Following the above comment, I believe your primary competing baseline is FedONESHOT, so I suggest adding more details on this algorithm's main idea.
- Do you have any explanation as to why the standard deviation of the performance of the proposed methods is, on average, larger under SVHN in Table 2?

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence:

The claim states that FedCVAE-Ens and FedCVAE-KD "typically more than double the accuracy of baselines" at highest levels of heterogeneity. Looking at Table 1 (α=0.001, highest heterogeneity), FedCVAE-KD achieves ~1.75x accuracy for MNIST (82.24 vs 45.12), ~2x for FashionMNIST (69.53 vs 32.77), and ~2.75x for SVHN (55.48 vs 20.31) compared to FedAvg. FedCVAE-Ens shows even better improvements. While there is some variation, the claim's "more than double" characterization is generally supported by the empirical evidence across datasets.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In particular, at the highest levels of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of the baselines.

The review comment to be evaluated is:

### Strength

* The proposed method of one-shot FL is very interesting and novel.
* The proposed method doesn't require auxiliary public dataset.
* Both security and privacy extensions are considered in the paper.
* It can be applied to heterogeneous local models with similar generative capabilities.

### Weaknesses
* One potential issue of the method is training a CVAE locally. In the paper, the images used in the dataset are very small, and the amount of data each client has is large. However, in the real-world application of FL like medical imagings, with very large images and very limited number of data samples, training a CVAE locally can be difficult even just for one class of data.
* The training task of local models is different from the original task. This limits the re-use of pre-trained local model from original task. I think one of the reasons that people want to do one-shot FL is to prevent any re-training locally. But the proposed method still requires to train some new model.

The evidence to be evaluated is:We jointly propose FedCVAE-Ens and FedCVAE-KD, one-shot FL methods that do not require an auxiliary public dataset for server-side training (they are _data-free_). They address issues caused by high statistical heterogeneity by reframing the learning task using CVAEs and account for model heterogeneity by allowing different CVAE architectures across clients.

To demonstrate the efficacy of FedCVAE-Ens and FedCVAE-KD in the difficult setting of high statistical heterogeneity, we test on varying levels of \(\alpha\), from high (\(\alpha=0.05\)), to very high (\(\alpha=0.01\)), to extreme (\(\alpha=0.001\)) statistical heterogeneity. FedCVAE-Ens consistently outperforms all other methods, and FedCVAE-KD outperforms the baselines in nearly all datasets and levels of \(\alpha\) (the only exception is MNIST at the lowest level of statistical heterogeneity) as shown in Table 1. At \(\alpha=0.001\), FedCVAE-KD obtains more than \(1.75\times\) the accuracy of the best baseline method for MNIST, more than \(2\times\) the accuracy for FashionMNIST, and nearly \(2.75\times\) the accuracy for SVHN (Table 1). While the baselines FedAvg and FedOneShot are very sensitive to the level of statistical heterogeneity, both FedCVAE-Ens and FedCVAE-KD demonstrate consistent performance across levels of \(\alpha\) (Table 1).

While both FedAvg and FedOneShot struggle when many clients are present, Table 2 shows that FedCVAE-Ens and FedCVAE-KD perform consistently across the number of clients, with the only exception being FedCVAE-KD for SVHN. Even though it is typical to observe FL methods' accuracy degrade with increasing numbers of clients (Zhang et al., 2021), FedAvg and FedOneShot are unstable with no clear decrease in accuracy, which we ascribe to the highly variable partitions generated at high levels of statistical heterogeneity. Experiments varying the dataset partitions reveal high variation in accuracy for FedAvg and FedOneShot, whereas both FedCVAE-Ens and FedCVAE-KD exhibit consistent accuracy (Table 5 in Appendix C).

Both methods performed well given high statistical heterogeneity, demonstrated consistent performance with increasing numbers of clients, allow for model heterogeneity across clients, and can be extended to promote security. Extensive experimental results showed that FedCVAE-Ens and FedCVAE-KD substantially extended the state-of-the-art in one-shot FL under very high statistical heterogeneity, making way for more nuanced development in this difficult environment for distributed learning.

The KD approach we propose to aggregate client decoders generates substantially better samples (Figure 8) while also obtaining more than \(2\times\) classifier accuracy on MNIST, more than \(3.25\times\) accuracy on FashionMNIST, and nearly \(5\times\) accuracy for SVHN at \(\alpha=0.001\) (Table 6).

While accuracy for both FedCVAE-Ens and FedCVAE-KD degrade with less data per client, they both consistently perform better than FedAvg and FedOneShot across all tested percent subsets of the training data. \begin{table}
\begin{tabular}{l l|c c|c c} \hline \hline  & \# of Clients & FedAvg & FedOneShot & FedCVAE-KD (ours) & FedCVAE-Ens (ours) \\ \hline \multirow{3}{*}{MNIST} & \(m=5\) & \(42.34\pm 1.75\) & \(42.97\pm 1.54\) & \(80.19\pm 1.76\) & \(92.81\pm 2.16\) \\  & \(m=10\) & \(58.29\pm 2.83\) & \(41.15\pm 0.70\) & \(82.01\pm 1.61\) & \(93.83\pm 1.53\) \\  & \(m=20\) & \(37.14\pm 3.87\) & \(32.22\pm 0.71\) & \(81.93\pm 1.75\) & \(93.52\pm 0.86\) \\  & \(m=50\) & \(28.26\pm 7.98\) & \(35.62\pm 0.66\) & \(77.68\pm 1.39\) & \(87.56\pm 1.44\) \\ \hline \multirow{3}{*}{FashionMNIST} & \(m=5\) & \(38.51\pm 2.40\) & \(32.23\pm 1.68\) & \(68.01\pm 1.56\) & \(73.70\pm 1.93\) \\  & \(m=10\) & \(45.85\pm 2.95\) & \(37.63\pm 0.53\) & \(69.97\pm 1.63\) & \(76.62\pm 1.61\) \\  & \(m=20\) & \(27.49\pm 4.77\) & \(24.72\pm 1.20\) & \(71.28\pm 1.17\) & \(76.51\pm 1.38\) \\  & \(m=50\) & \(31.91\pm 2.20\) & \(41.01\pm 2.49\) & \(69.25\pm 1.13\) & \(76.13\pm 2.00\) \\ \hline \multirow{3}{*}{SVHN} & \(m=5\) & \(40.61\pm 0.64\) & \(42.05\pm 1.31\) & \(57.17\pm 2.56\) & \(65.72\pm 2.17\) \\  & \(m=10\) & \(25.12\pm 2.07\) & \(31.91\pm 1.26\) & \(55.97\pm 0.38\) & \(65.50\pm 2.28\) \\ \cline{1-1}  & \(m=20\) & \(26.99\pm 1.15\) & \(10.68\pm 0.70\) & \(51.80\pm 2.75\) & \(65.54\pm 1.89\) \\ \cline{1-1}  & \(m=50\) & \(19.59\pm 0.00\) & \(30.20\pm 0.64\) & \(40.03\pm 2.62\) & \(64.81\pm 2.44\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of clients \(m\). Best results for each dataset and each level of \(m\) are in purple, with second best results in yellow.

Input: Table 2: Performance of four one-shot FL methods over three datasets and across four numbers of

Input: Figure 8: Samples from the aggregated server decoder obtained through parameter averaging (top row) versus knowledge distillation (bottom row) at high statistical heterogeneity (α = 0.01).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "FedCVAE-Ens and FedCVAE-KD typically more than double the accuracy of baselines at highest levels of statistical heterogeneity" appears well-supported. At α=0.001 (highest heterogeneity), Table 1 shows:
- MNIST: 95.37% vs 45.12% (>2x)
- FashionMNIST: 76.04% vs 32.77% (>2x)
- SVHN: 65.52% vs 20.31% (>3x)

The performance improvements are consistently demonstrated across different numbers of clients (Table 2) and the claim acknowledges this is specifically for highest heterogeneity levels. The evidence directly supports the magnitude claimed without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First, we present two mechanisms towards equivariance-aware architectural optimization.

The evidence to be evaluated is:In more diverse approaches, Zhou et al. (2020) and Yeh et al. (2022) represent symmetry-inducing weight sharing via learnable matrices. Romero & Lohit (2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances for each layer.

We propose two mechanisms to enable search over subgroups: the equivariance relaxation morphism and a \([G]\)-mixed equivariant layer. The proposed morphism, depicted in Figure 1(A) and described in Section 4.1, changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The \([G]\)-mixed equivariant layer, shown in Figure 1(B) and presented in Section 4.2, allows for a single layer to represent equivariance to multiple subgroups through a weighted sum. The _equivariance relaxation morphism_ reparametrizes a \(G\)-equivariant group (or lifting) convolutional layer to operate over a subgroup of \(G\), partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.

Given a \(G\)-equivariant group convolutional layer with feature map \(f\) and filter \(\psi\), we define the relaxed feature map \(\tilde{f}\colon G^{\prime}\to\mathbb{R}^{(C_{l-1}\times|R|)}\) and relaxed filter \(\tilde{\psi}\colon G^{\prime}\to\mathbb{R}^{(C_{l}\times|R|)\times(C_{l-1} \times|R|)}\) as follows. For \(c\in[C_{l-1}]\), \(s,t\in R\), \(d\in[C_{l}]\):

We define the _equivariance relaxation morphism_ from \(G\) to \(G^{\prime}\) as the reparametrization of \(\psi\) as \(\tilde{\psi}\) (Eq. 4) and reshaping of \(f\) as \(\tilde{f}\) (Eq. 3). We will show that the new layer, \([\tilde{f}\star_{G^{\prime}}\tilde{\psi}]_{(d,t)}(g^{\prime})\), is equivalent to \([f\star_{G}\psi]_{d}(g^{\prime}t)\) down to reshaping.

\[[f\star_{G}\psi]_{d}(g^{\prime}t) =[f\star_{G}\psi]_{d}(g) \tag{5}\] \[=\sum_{h\in G}\sum_{c=1}^{C_{l-1}}f_{c}(h)\psi_{d,c}(g^{-1}h),\] (6) \[=\sum_{h^{\prime}\in G^{\prime}}\sum_{s\in R}\sum_{c=1}^{C_{l-1} }f_{c}(h^{\prime}s)\psi_{d,c}(t^{-1}g^{\prime-1}h^{\prime}s),\] (7) \[=\sum_{h^{\prime}\in G^{\prime}}\sum_{c=1}^{C_{l-1}}\sum_{s\in R }\tilde{f}_{(c,s)}(h^{\prime})\tilde{\psi}_{(d,t),(c,s)}(g^{\prime-1}h^{\prime }),\] (8) \[=\left[\tilde{f}\star_{G^{\prime}}\tilde{\psi}\right]_{(d,t)}(g^{ \prime}), \tag{9}\]

Thus, the convolution of \(\tilde{f}\) with \(\tilde{\psi}\) is equivariant to \(G\) but parametrized as a \(G^{\prime}\)-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing a \(G^{\prime}\)-equivariant layer with a pre-trained prior of equivariance to \(G\), maintaining any previous training.

Towards learning equivariance, we additionally propose partial equivariance via a mixture of layers, each constrained to equivariance to different groups, applied in parallel to the same input then combined via a weighted sum.

For a set of groups \([G]\), such as a subgroup lattice of some group \(G\), we define a \([G]\)_-mixed equivariant layer_ as: \[\left[f^{\hat{\pi}_{[G]}}[\psi]\right]_{(d,t)}(g) =\sum_{G\in[G]}z_{G}\left[f\star_{G^{\prime}}\widetilde{\psi^{G}} \right]_{(d,t)}(g) \tag{10}\] \[=\left[f\star_{G^{\prime}}\sum_{G\in[G]}z_{G}\widetilde{\psi^{G} }\right]_{(d,t)}(g), \tag{11}\] where each element \(z_{G}\) of \([z]\coloneqq\{z_{G}|G\in[G]\}\) is an architectural weighting parameter such that \(\sum_{G\in[G]}z_{G}=1\), \(G^{\prime}\) is a subgroup of all groups in \([G]\), each element \(\psi^{G}\) of \([\psi]\) is a filter with a domain of \(G\), and \(\widetilde{\psi^{G}}\) is the transformation of \(\psi^{G}\) from a domain of \(G\) to \(G^{\prime}\) as defined in Equation 4. Thus, the layer is parametrized by \([\psi]\) and \([z]\), computing a weighted sum of operations that are equivariant to different groups of \([G]\). The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown in Equation 11.

We present two NAS methods that utilize the presented mechanisms for discovering appropriate equivariance during training: Evolutionary Equivariance-Aware NAS (EquiNAS\({}_{E}\)) and Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)). Both methods optimize an architecture while learning weights, yielding a final trained network adapted to equivariances present in the training data.

EquiNAS\({}_{E}\), in Section 5.1, searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNAS\({}_{D}\), in Section 5.2, searches for smooth mixtures of equivariant layers.

Offspring are generated from each parent separately by mutation using the relaxation morphism. This preserves the weights of the parametrized equivariance during mutation, allowing for the continuous training of networks over evolution by inheritance from parent individuals. Specifically, mutation reduces a single layer's parametrized equivariance to a subgroup within the constraint that each layer has parametrized equivariance to a subgroup of all preceding layers. This constraint yields local equivariance properties for the network, as shown in Weiler and Cesa (2019) and Elsayed et al. (2020) to be empirically favorable in image classification tasks. The resulting individuals are each trained independently for a given training time, and then this process repeats.

In a contrasting paradigm, the \([G]\)-mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algorithm. Our Differentiable Equivariance-Aware NAS (EquiNAS\({}_{D}\)) algorithm, defined in Algorithm 2, is inspired by DARTS (Liu et al., 2018) with significant changes detailed in the following paragraphs.

These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance.

We focus on the regular representation of groups and show experiments with reflectional and up to 4-fold rotational symmetry groups applied to image classification tasks. Examples of symmetry groups acting on pixel space, which corresponds to \(\mathbb{Z}^{2}\), include \(T(2)\), which consists of discrete translations in both dimensions; the cyclical groups \(C_{n}\), which consist of \(n\)-fold rotations; and the dihedral groups \(D_{n}\), which consist of reflections with \(n\)-fold rotations, where \(n\in\{1,2,4\}\) for exact symmetry without interpolation.

As standard convolutional layers are already equivariant to \(T(2)\), we refer to layers also equivariant to \(n\)-fold rotations with or without reflections as \(D_{n}\) or \(C_{n}\)-equivariant, respectively. So, a \(C_{1}\) equivariant convolutional layer is a standard translation-equivariant convolutional layer. We use \((C_{1},D_{1},C_{2},D_{2},C_{4},D_{4})\) as the set of potential groups for mutation in EquiNAS\({}_{E}\) and as \([G]\) in EquiNAS\({}_{D}\).

These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints. Across all experiments, the architectures are designed to have consistent channel dimensions once expanded to a standard translation-equivariant convolutional layer for each layer across models. Thus, constrained equivariance to a larger symmetry group results in fewer learnable parameters. A layer constrained to \(C_{4}\) equivariance has \(|C_{4}\setminus D_{4}|=2\) times as many independent channels and as many parameters as a layer constrained to \(D_{4}\) equivariance. This is a notably different paradigm than other works that equate parameter counts across architectures with different equivariance properties.

In addition to the normally initialized static baselines, we also train and test baselines that are initialized with priors to larger symmetry groups. These are implemented by initializing all layers to be constrained to the prior symmetry group, then using the equivariance relaxation morphism on each layer. EquiNAS\({}_{E}\) searches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors.

To our knowledge, this is the first work which proposes search methods for networks with dynamically constrained equivariance. Many NAS approaches separately search for an architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in a single process, notably with dynamically constrained weights.

Our two equivariance-aware NAS approaches have distinct approaches: EquiNAS\({}_{E}\) searches for architectures composed of discretely equivariant layers, while EquiNAS\({}_{D}\) searches for continuous mixtures of equivariance within each layer.

We present two mechanisms towards equivariance-aware architectural optimization, the equivariance relaxation morphism and the \([G]\)-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS\({}_{E}\) and differentiably as EquiNAS\({}_{D}\).

* Yeh et al. (2022) Raymond A Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, and Alexander Schwing. Equivariance discovery by learned parameter-sharing.

* Zhou et al. (2020) Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In _International Conference on Learning Representations_, 2020.

Group convolutional layers The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism.

Equivariance relaxation morphism To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion.

The \([G]\)-mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.

For both EquiNAS\({}_{E}\) and EquiNAS\({}_{D}\) experiments, we use the same backbone architecture, such that the static baselines have the same architecture across experiments. The architectures have a lifting layer followed by 7 group convolutional layers, for a total of 8 convolutional layers.

This is enabled by the \([G]\)-mixed equivariant layer, which is a contribution of this work.

Our methods enable searching for both architecture and parameters concurrently in a single training process.

Input: Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equivariant layer, using the C4 group. In (A), the learnable parameters of a C4-equivariant convolutional layer are expanded using each group element action so the expanded filter can be used in a standard convolutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained toC2 equivariance, initialized to functionalC4 equivariance. In (B), convolutional operations equivariant to subgroups ofC4 are summed with learnable architectural weighting parameters.

Input: Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.

Input: Figure 12: Architecture weighting parameters by layer for all trials on ISIC.

Input: Figure 11: Architecture weighting parameters by layer for all trials on Galaxy10.

Input: Figure 3: Architecture weighting parameters by layer for one selected trial from RotMNIST, Galaxy10, and ISIC. For other trials, see Figures 10-12 in Appendix D.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I've carefully reviewed the claim "First, we present two mechanisms towards equivariance-aware architectural optimization" against the provided evidence.

The evidence strongly supports this claim through detailed mathematical formulations and clear explanations of both mechanisms:
1. The equivariance relaxation morphism (with full mathematical derivation in equations 3-9)
2. The [G]-mixed equivariant layer (defined in equations 10-11)

Figure 1 provides clear visual demonstrations of both mechanisms. The evidence shows rigorous theoretical development and practical implementation details. The claim is modest and accurately reflects what is presented, without overstating the contributions or their implications.

<score>1</score>