Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes.

The evidence to be evaluated is:In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence. A similar idea by Wu et al. (2020) was used to improve performance of training on video data, by balancing resolution with batch size. Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed.

In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

We will explore this notion by modifying the optimization method, enabling training with varying batch-sizes and limited hyper-parameter tuning.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. We continue to test our hypothesis using a novel mixed-size training regime.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

Formally, in the MixSize regime, these sizes can be described as random variables sharing a single discrete distribution \[(\hat{S},\hat{B},\hat{D})=\left\{(S,B,D)_{i}\;\;\text{w.p.}\;\;p_{i}\right\}, \tag{1}\] where \(\forall i:p_{i}\geq 0\) and \(\sum_{i}p_{i}=1\). As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\). Thus, in this regime, the computational and memory requirements for each step in remain fixed.

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

We denote our modified mixed-size regimes as \(B^{+}\) for an increased effective batch-size and \(D^{+}\) for an increased number of BA duplicates of the same ratio. MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size.

To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe three methods we found useful: **step-wise size sampling, gradient smoothing** and **batch-norm calibration**.

We therefore considered two alternatives for image size variations, where size is randomly sampled and changed throughout training: * Sampling image size per epoch

* Sampling image size per training step These sampling methods can be seen as a stochastic alternative to long and short cycle multigrid approach suggested by Wu et al. (2020). We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy. Considering these findings, we continue to perform our experiments using the third regime - sampling image size per training step.

This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training. This property was explored in previous works concerning large-batch regimes, in which a learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task or across models (Shallue et al., 2018), with solutions such as a square-root scaling (Hoffer et al., 2017), linear scaling (Goyal et al., 2017) or a fixed norm ratio (You et al., 2017). Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training.

Interestingly, we highlight the fact that although this process has little to no effect on models trained using a fixed-size input, it does improve our mixed-size models considerably on a wide range of image sizes. We used our sampling strategy to train and compare our regime to the baseline results of various models over the CIFAR10/100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) visual classification tasks.

As CIFAR datasets are limited in size, we consider the following balanced stochastic regime chosen: \[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant. For the ImageNet dataset, we use the following stochastic regime found by cross-validation on several alternatives over the ResNet-50 model (He et al., 2016): \[S^{(144)}:\ \ S=\begin{cases}256,&\text{w.p}\quad p=0.1\\ 224,&\text{w.p}\quad p=0.1\\ 128,&\text{w.p}\quad p=0.6\\ 96,&\text{w.p}\quad p=0.2\end{cases}\] While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\). This regime was again designed so that the reduced spatial size can be used to increase the corresponding batch size or the number of BA duplicates, as described in Section 3.

The ResNet-50 model benefits the most from our MixSize regime on the ImageNet model and so we chose it for further exploration.

We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F).

However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size. The model trained with the \(S^{(208)}\) regime offers a similar improvement in accuracy, but across a larger spatial size, as it observed an average size of \(208\) during training. Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation duplicates in the batch. Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases.

In order to adapt to varying input scales, we add an additional augmentation step, that resizes the images using bilinear interpolation to \(S\times S\), depending on a sampled size for each step. We note that this keeps the exact original augmentation procedure for \(S=32\).

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We show the impact of gradient smoothing on \(B^{+}\) MixSize regimes on CIFAR10 6 and ImageNet 7.

We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

* Increase image size from small to large (progressive resizing), where each image size is used for number of epochs \(E_{i}=p_{i}E_{\mathrm{total}}\), where \(E_{\mathrm{total}}\) is the total number training epochs required.

* Using a random image size for each epoch, keeping the epoch number for each size at \(E_{i}\)

* Sampling image size per training step at probability \(p_{i}\) We used a \(B^{+}\) regime and adapt learning rate for each batch-size according to best practices (Hoffer et al., 2017; Goyal et al., 2017).

We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 6: Impact of gradient smoothing on CIFAR10, ResNet-44. The training regime includes two image sizes: 32 × 32 and 16 × 16 (average size is S = 24). Using a B+ regime creates two batch sizes: 256 and 2, 048 respectively. Gradient norm values are shown with and without Grad-smoothing. Solid lines are gradients for S = 32 while dotted lines are for S = 16. Gradient smoothing helps to reduce gap between gradient norms at difference batch sizes.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Looking at the claim "we introduce a novel training regime, 'MixSize' for convolutional networks that uses stochastic image and batch sizes," I find this to be well-stated and accurately reflects the evidence provided. The evidence thoroughly documents:

1. The technical details of MixSize, including formal mathematical descriptions
2. Clear differentiation from previous work on progressive resizing
3. Comprehensive experimental validation across multiple datasets
4. Detailed implementation specifics for image size sampling and batch size adjustments
5. Results showing improved accuracy and training efficiency

The claim is modest in scope, simply stating what was introduced without overstating capabilities or results. The evidence strongly supports the novelty and basic functioning of the approach.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes.

The review comment to be evaluated is:

The paper proposes the use of mixed image sizes during training. They argue empirically that such an approach improves generalization for both fixed image size (e.g. 224 in ImageNet) as well as for variable image size. The proposed training algorithm maintains the same computational budget at each step by either changing the batch size or by using more/less augmentation. They show that adjusting the batch size leads to a faster training but using augmentation leads to a better test accuracy (hence a tradeoff).

However, in order for their proposed method to work, the authors also propose modifications to standard training procedures (i.e. smoothing the gradient, adjusting the batchnorm layers) but without carrying out an ablation study that shows the impact with/without each of these steps. My particular concern is in the use of "gradient smoothing". If I understand it correctly, this is not very different from using momentum, which is known to reduce the variance and improve generalization. However, the authors use gradient smoothing only in their proposed method and do not use it in the baseline method (why not?). It is possible that the reported improvements (e.g. for fixed image size) come solely from this step.

The other concern is when sampling the image size per step. The authors  propose distributions that seem odd in their experiments (e.g. why is p=0.6 for size 128 in ImageNet which is much larger than others, and why not uniform in CIFAR10). It is important to know if the results are sensitive to the choice of the distribution, to make sure that the benefit is not due to random chance. Also, if this distribution needs to be fine-tuned, then the discussion about improving the training time would be meaningless.

The last issue is the robustness to different image sizes. Figure 3(a) shows  that if the average image size during training is small, the network will perform better for small images but not for large images. Conversely, if the average image size during training is large, it will perform better for large images, but not for small images. If the concern here is around using mixed-image sizes at inference time, then the red curve in Figure 3(a) shows that a fixed image size is reasonably robust. If one knows in advance that the average image size would be smaller than 224, one can train with a fixed image size that is smaller.

One minor last remark (feel free to ignore) regarding the motivation: the authors study the correlation between the full gradients for the same image with different sizes, on the one hand, and for different images with the same size, on the other hand. They conclude that the first case (different sizes) shows a stronger correlation, which is true according to Table 2, but this statement omits the fact that most correlations were low anyway. For example, for partially trained network, it is 0.08 vs. 0.02. I do not think that one can use such figures to conclude that "smaller image gradients can be used as an estimate to the full image gradients".

The improvement in test accuracy is very promising but I believe, some ablation is needed to identify exactly where this improvement comes from and whether it can be obtained using simpler approach (e.g. smoothing the gradient alone or using augmentation alone, etc).

The evidence to be evaluated is:In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence. A similar idea by Wu et al. (2020) was used to improve performance of training on video data, by balancing resolution with batch size. Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed.

In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

We will explore this notion by modifying the optimization method, enabling training with varying batch-sizes and limited hyper-parameter tuning.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. We continue to test our hypothesis using a novel mixed-size training regime.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

Formally, in the MixSize regime, these sizes can be described as random variables sharing a single discrete distribution \[(\hat{S},\hat{B},\hat{D})=\left\{(S,B,D)_{i}\;\;\text{w.p.}\;\;p_{i}\right\}, \tag{1}\] where \(\forall i:p_{i}\geq 0\) and \(\sum_{i}p_{i}=1\). As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\). Thus, in this regime, the computational and memory requirements for each step in remain fixed.

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

We denote our modified mixed-size regimes as \(B^{+}\) for an increased effective batch-size and \(D^{+}\) for an increased number of BA duplicates of the same ratio. MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size.

To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe three methods we found useful: **step-wise size sampling, gradient smoothing** and **batch-norm calibration**.

We therefore considered two alternatives for image size variations, where size is randomly sampled and changed throughout training: * Sampling image size per epoch

* Sampling image size per training step These sampling methods can be seen as a stochastic alternative to long and short cycle multigrid approach suggested by Wu et al. (2020). We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy. Considering these findings, we continue to perform our experiments using the third regime - sampling image size per training step.

This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training. This property was explored in previous works concerning large-batch regimes, in which a learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task or across models (Shallue et al., 2018), with solutions such as a square-root scaling (Hoffer et al., 2017), linear scaling (Goyal et al., 2017) or a fixed norm ratio (You et al., 2017). Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training.

Interestingly, we highlight the fact that although this process has little to no effect on models trained using a fixed-size input, it does improve our mixed-size models considerably on a wide range of image sizes. We used our sampling strategy to train and compare our regime to the baseline results of various models over the CIFAR10/100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) visual classification tasks.

As CIFAR datasets are limited in size, we consider the following balanced stochastic regime chosen: \[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant. For the ImageNet dataset, we use the following stochastic regime found by cross-validation on several alternatives over the ResNet-50 model (He et al., 2016): \[S^{(144)}:\ \ S=\begin{cases}256,&\text{w.p}\quad p=0.1\\ 224,&\text{w.p}\quad p=0.1\\ 128,&\text{w.p}\quad p=0.6\\ 96,&\text{w.p}\quad p=0.2\end{cases}\] While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\). This regime was again designed so that the reduced spatial size can be used to increase the corresponding batch size or the number of BA duplicates, as described in Section 3.

The ResNet-50 model benefits the most from our MixSize regime on the ImageNet model and so we chose it for further exploration.

We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F).

However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size. The model trained with the \(S^{(208)}\) regime offers a similar improvement in accuracy, but across a larger spatial size, as it observed an average size of \(208\) during training. Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation duplicates in the batch. Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases.

In order to adapt to varying input scales, we add an additional augmentation step, that resizes the images using bilinear interpolation to \(S\times S\), depending on a sampled size for each step. We note that this keeps the exact original augmentation procedure for \(S=32\).

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We show the impact of gradient smoothing on \(B^{+}\) MixSize regimes on CIFAR10 6 and ImageNet 7.

We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

* Increase image size from small to large (progressive resizing), where each image size is used for number of epochs \(E_{i}=p_{i}E_{\mathrm{total}}\), where \(E_{\mathrm{total}}\) is the total number training epochs required.

* Using a random image size for each epoch, keeping the epoch number for each size at \(E_{i}\)

* Sampling image size per training step at probability \(p_{i}\) We used a \(B^{+}\) regime and adapt learning rate for each batch-size according to best practices (Hoffer et al., 2017; Goyal et al., 2017).

We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 6: Impact of gradient smoothing on CIFAR10, ResNet-44. The training regime includes two image sizes: 32 × 32 and 16 × 16 (average size is S = 24). Using a B+ regime creates two batch sizes: 256 and 2, 048 respectively. Gradient norm values are shown with and without Grad-smoothing. Solid lines are gradients for S = 32 while dotted lines are for S = 16. Gradient smoothing helps to reduce gap between gradient norms at difference batch sizes.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and reviewer comments, I've evaluated whether the claim "In this work, we introduce a novel training regime, 'MixSize' for convolutional networks that uses stochastic image and batch sizes" is well-stated or overstated.

The claim appears to be well-stated (1) for several reasons:
1. The evidence clearly demonstrates they developed and implemented a new training approach using stochastic image and batch sizes
2. The methodology is thoroughly detailed with specific formulations and distributions
3. The results show concrete improvements in some scenarios
4. The authors acknowledge limitations and tradeoffs
5. The reviewer, while raising some valid concerns about ablation studies, confirms the novelty and potential of the approach

The claim itself is factual and doesn't overstate the capabilities or results of the method.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes.

The review comment to be evaluated is:

This paper proposes to increase training costs to compensate for the reduced costs from multi-scale CNN training by either increasing batch size (and therefore lowering the number of iterations per epoch) or increasing the number of augmented versions (duplicates) of the same samples within a batch. The former allows for smaller total training costs than conventional single-scale training, while the latter maintains the total training costs but improves the final performance. Several training improvement methods are introduced to improve the multi-scale training.

Paper's strengths
- The paper is quite well-written.
- Code and models are provided for reproducibility.
- Gradient smoothing is a nice way to mitigate the variability of gradient statistics resulted from different input sizes. As far as I know, this is quite novel, particularly in the context of multi-scale training.

Paper's weaknesses
- Multi-scale training is a common practice in many computer vision tasks especially in object detection** (less common in image classification). This paper also does multi-scale training but only introduces some minor improvements that are neither breakthroughs nor that they provide any interesting insight.
> - Bag of Freebies for Training Object Detection Neural Networks. arXiv.
> - YOLO9000: Better, Faster, Stronger. CVPR 2017.
> - MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.

- For "step-wise size sampling", it seems like that conclusion to use this variant of sampling is heuristically chosen and totally ignores the existing practice in other computer vision tasks. One of the straightforward ways to do multi-scale training in object detection is to select different input sizes even for the images within the same batch (by padding zeros for the smaller images). Alternatively, one could sample different input sizes for different GPU batches (all images within a GPU share the same size) while doing multiple-GPU training.

- The three training improvements (step-wise size sampling, gradient smoothing and batch-norm calibration) are what separate this paper from prior work but they are not extensively evaluated. Some of them are briefly evaluated in the appendix and some of their effects are just briefly mentioned in the method section. They ought to appear in the experimental section of the main paper. Gradient smoothing seems like a nice idea but it is unclear how important it is given that there is only one figure (Fig.7) showing its impact on the performance.

- This paper strives to increase the number of batch samples given a fixed budget of computational and time resources for per iteration step. I wonder why this should be limited to the cost of an iteration step but not the entire training cost from all epochs/iterations. It makes more sense to measure the cost for the entire training process which accurately tells how much is spent to train the model until convergence.

- In Sec. 5.1, the size ratios for different datasets are carefully chosen based on cross-validation. This makes it hard to directly apply MixSize to other datasets or settings without going through this step. It also adds additional computations which defeat the purpose of increasing batch size to maintain the same training cost as conventional single-scale training.

- Using separate BatchNorm statistics for multi-scale inputs/features has been explored in the following papers (published at least few months before ICLR deadline). They should be cited and compared against MixSize:
> - Learning to Learn Parameterized Classification Networks for Scalable Input Images. ECCV 2020.
> - Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks. CVPR 2018.

Overall, this paper shows good performance and has some good ideas (e.g., gradient smoothing) for improving multi-scale training. But it fails to give more emphasis to or do a deeper dive into the potentially novel aspects of the work. The current performance improvements may come from doing just trivial multi-scale training which was already widely-explored in prior work.

The evidence to be evaluated is:In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence. A similar idea by Wu et al. (2020) was used to improve performance of training on video data, by balancing resolution with batch size. Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed.

In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

We will explore this notion by modifying the optimization method, enabling training with varying batch-sizes and limited hyper-parameter tuning.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. We continue to test our hypothesis using a novel mixed-size training regime.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

Formally, in the MixSize regime, these sizes can be described as random variables sharing a single discrete distribution \[(\hat{S},\hat{B},\hat{D})=\left\{(S,B,D)_{i}\;\;\text{w.p.}\;\;p_{i}\right\}, \tag{1}\] where \(\forall i:p_{i}\geq 0\) and \(\sum_{i}p_{i}=1\). As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\). Thus, in this regime, the computational and memory requirements for each step in remain fixed.

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

We denote our modified mixed-size regimes as \(B^{+}\) for an increased effective batch-size and \(D^{+}\) for an increased number of BA duplicates of the same ratio. MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size.

To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe three methods we found useful: **step-wise size sampling, gradient smoothing** and **batch-norm calibration**.

We therefore considered two alternatives for image size variations, where size is randomly sampled and changed throughout training: * Sampling image size per epoch

* Sampling image size per training step These sampling methods can be seen as a stochastic alternative to long and short cycle multigrid approach suggested by Wu et al. (2020). We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy. Considering these findings, we continue to perform our experiments using the third regime - sampling image size per training step.

This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training. This property was explored in previous works concerning large-batch regimes, in which a learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task or across models (Shallue et al., 2018), with solutions such as a square-root scaling (Hoffer et al., 2017), linear scaling (Goyal et al., 2017) or a fixed norm ratio (You et al., 2017). Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training.

Interestingly, we highlight the fact that although this process has little to no effect on models trained using a fixed-size input, it does improve our mixed-size models considerably on a wide range of image sizes. We used our sampling strategy to train and compare our regime to the baseline results of various models over the CIFAR10/100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) visual classification tasks.

As CIFAR datasets are limited in size, we consider the following balanced stochastic regime chosen: \[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant. For the ImageNet dataset, we use the following stochastic regime found by cross-validation on several alternatives over the ResNet-50 model (He et al., 2016): \[S^{(144)}:\ \ S=\begin{cases}256,&\text{w.p}\quad p=0.1\\ 224,&\text{w.p}\quad p=0.1\\ 128,&\text{w.p}\quad p=0.6\\ 96,&\text{w.p}\quad p=0.2\end{cases}\] While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\). This regime was again designed so that the reduced spatial size can be used to increase the corresponding batch size or the number of BA duplicates, as described in Section 3.

The ResNet-50 model benefits the most from our MixSize regime on the ImageNet model and so we chose it for further exploration.

We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F).

However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size. The model trained with the \(S^{(208)}\) regime offers a similar improvement in accuracy, but across a larger spatial size, as it observed an average size of \(208\) during training. Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation duplicates in the batch. Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases.

In order to adapt to varying input scales, we add an additional augmentation step, that resizes the images using bilinear interpolation to \(S\times S\), depending on a sampled size for each step. We note that this keeps the exact original augmentation procedure for \(S=32\).

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We show the impact of gradient smoothing on \(B^{+}\) MixSize regimes on CIFAR10 6 and ImageNet 7.

We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

* Increase image size from small to large (progressive resizing), where each image size is used for number of epochs \(E_{i}=p_{i}E_{\mathrm{total}}\), where \(E_{\mathrm{total}}\) is the total number training epochs required.

* Using a random image size for each epoch, keeping the epoch number for each size at \(E_{i}\)

* Sampling image size per training step at probability \(p_{i}\) We used a \(B^{+}\) regime and adapt learning rate for each batch-size according to best practices (Hoffer et al., 2017; Goyal et al., 2017).

We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 6: Impact of gradient smoothing on CIFAR10, ResNet-44. The training regime includes two image sizes: 32 × 32 and 16 × 16 (average size is S = 24). Using a B+ regime creates two batch sizes: 256 and 2, 048 respectively. Gradient norm values are shown with and without Grad-smoothing. Solid lines are gradients for S = 32 while dotted lines are for S = 16. Gradient smoothing helps to reduce gap between gradient norms at difference batch sizes.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, this appears to be a moderately well-stated claim with some minor overstatement. The evidence demonstrates that the authors did introduce a novel training regime called "MixSize" that uses stochastic image and batch sizes. The methodology, implementation details, and results are thoroughly documented. However, as noted in the review comments, multi-scale training was already common practice in computer vision, particularly object detection, and some of the improvements (like batch-norm calibration) had been explored in prior work. The novelty lies more in the specific combination and improvements rather than introducing a completely new concept.

<score>2</score>

Justification:
1. The claim is technically accurate - they did introduce a new training regime with stochastic image/batch sizes.
2. Evidence supports that the approach includes novel elements like gradient smoothing and specific sampling strategies.
3. Review comments highlight that while innovative, the work builds heavily on existing multi-scale training practices.
4. Results demonstrate improved performance and training efficiency, validating the practical value.
5. The term "novel" is slightly overstated given the existing work in multi-scale training, but the specific implementation and improvements do represent an original contribution.

Citations from evidence:
- "Multi-scale training is a common practice in many computer vision tasks especially in object detection"
- "Gradient smoothing is a nice way to mitigate the variability of gradient statistics...this is quite novel"
- Results show "B+ regime enables training with approximately 2.7× fewer training steps, while reaching an improved accuracy"


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes.

The review comment to be evaluated is:

== Summary ==

The paper proposes to use different image resolutions during the training of a deep neural network image classifier, and varying the batch size or number of data augmented versions of the images, keeping the computational cost per step roughly constant. The authors apply this approach to several architectures and three datasets, and show that they can achieve or improve the same accuracy as the baselines but much faster; or achieve better results with the same computational budget.

== Pros ==

+ The authors conducted their experiments using three different datasets (Cifar10, Cifar100 and ImageNet), and six different architectures (ResNet44, ResNet50, WideResNet, DenseNet and EfficientNet).

+ The proposed approach outperforms the baselines consistently across the 8 pairs of (dataset, architecture) that they have studied. In addition, MixSize can be easily implemented (the authors also provide a PyTorch implementation).

+ The authors investigate different "tricks" to apply during training when using MixSize to stabilize training or achieving better results. For instance, they compared randomly sampling the size from a distribution (as they propose) versus increasing the image resolution as training progresses, and showed that randomly sampling yields slightly better results.

+ Figure 3a seems to suggest that MixSize yields more robust classifiers under a wide range of image sizes. The area under the "Mixed S=144" curve seems to be larger than the area under the "Fixed S=224". However, further experimentation is needed to confirm this, since the area of the "Mixed S=208" seems closer to "Fixed S=224", and in any case the maximum image size was capped around 415.

== Cons ==

- On of the claimed contributions is: "Faster training vs. high accuracy. We show that reducing the average image size at train- ing leads to a trade-off between the time required to train the model and its final accuracy.". However, I would not consider this a novel contribution, since the trade-off between speed and accuracy is well-known. In fact, the authors cite Huang et al. (2018) and Szegedy et al. (2016) which already showed this. EfficientNet is another well-known architecture that takes advantage of this fact.

- In the intro, the authors claim "Touvron et al. (2019) demonstrated that networks trained on specific image resolutions perform poorly on other image sizes at evaluation time, as confirmed in Figure 1". This is inaccurate, since Touvron et al. (2019) actually show that slightly increasing the test resolution improves accuracy, due to the discrepancy in object sizes introduced by data augmentation (cropping). In fact, Figure 1 shows the same effect (model trained with 224 res, achieves best results with 284 eval image size). The statement in the introduction is again contradicted at the end of the first paragraph in Section 2.1.

- The authors do not report any statistical significance metric. Some datasets have very close results, so it's hard to tell whether the improvements are (statistically) significant or not.

- Poor captions in figures and tables. For instance, the difference between solid lines and dashed lines is only explained at the very last figure in the appending (Figure 7). Also, the caption of Figure 3 reads "Test accuracy on validation set" which is ambiguous: Is it a typo, or is it that the authors report the results on the 50k validation set of ImageNet (and use some smaller subset of the training set as validation)?

== Reasons for score ==

Although the proposed approach is simple and consistently improves the baseline results, I'm not convinced that the originality and significance of the work is enough for it to be accepted.

Regarding originality, there is a plethora of works exploring the trade-offs between image size and accuracy. The most similar works are Howard (2018) and Karras et al. (2017), which increase the image size through training. It's not clear that random sampling offers a much better result, judging from Figure 8 in Appending E, if one compares the accuracy of "Small->Large" strategy at 125k steps (possibly before the last increase in size).

Regarding significance, if one restricts the analysis to the best architectures in each dataset, the increase in accuracy does not seem to be very large. Cifar10: 98.16% -> 98.32% (AmoebaNet), ImageNet: 76.32% -> 76.53% (EfficientNet-B0). Cifar100 shows a larger improvement, but the authors did not use AmoebaNet (which worked best in Cifar10) for some unknown reason. The fact that no statistical significance metrics are reported, does not help to discern whether the improvements are meaningful or not.

The evidence to be evaluated is:In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence. A similar idea by Wu et al. (2020) was used to improve performance of training on video data, by balancing resolution with batch size. Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed.

In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

We will explore this notion by modifying the optimization method, enabling training with varying batch-sizes and limited hyper-parameter tuning.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. We continue to test our hypothesis using a novel mixed-size training regime.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

Formally, in the MixSize regime, these sizes can be described as random variables sharing a single discrete distribution \[(\hat{S},\hat{B},\hat{D})=\left\{(S,B,D)_{i}\;\;\text{w.p.}\;\;p_{i}\right\}, \tag{1}\] where \(\forall i:p_{i}\geq 0\) and \(\sum_{i}p_{i}=1\). As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\). Thus, in this regime, the computational and memory requirements for each step in remain fixed.

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

We denote our modified mixed-size regimes as \(B^{+}\) for an increased effective batch-size and \(D^{+}\) for an increased number of BA duplicates of the same ratio. MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size.

To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe three methods we found useful: **step-wise size sampling, gradient smoothing** and **batch-norm calibration**.

We therefore considered two alternatives for image size variations, where size is randomly sampled and changed throughout training: * Sampling image size per epoch

* Sampling image size per training step These sampling methods can be seen as a stochastic alternative to long and short cycle multigrid approach suggested by Wu et al. (2020). We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy. Considering these findings, we continue to perform our experiments using the third regime - sampling image size per training step.

This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training. This property was explored in previous works concerning large-batch regimes, in which a learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task or across models (Shallue et al., 2018), with solutions such as a square-root scaling (Hoffer et al., 2017), linear scaling (Goyal et al., 2017) or a fixed norm ratio (You et al., 2017). Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training.

Interestingly, we highlight the fact that although this process has little to no effect on models trained using a fixed-size input, it does improve our mixed-size models considerably on a wide range of image sizes. We used our sampling strategy to train and compare our regime to the baseline results of various models over the CIFAR10/100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) visual classification tasks.

As CIFAR datasets are limited in size, we consider the following balanced stochastic regime chosen: \[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant. For the ImageNet dataset, we use the following stochastic regime found by cross-validation on several alternatives over the ResNet-50 model (He et al., 2016): \[S^{(144)}:\ \ S=\begin{cases}256,&\text{w.p}\quad p=0.1\\ 224,&\text{w.p}\quad p=0.1\\ 128,&\text{w.p}\quad p=0.6\\ 96,&\text{w.p}\quad p=0.2\end{cases}\] While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\). This regime was again designed so that the reduced spatial size can be used to increase the corresponding batch size or the number of BA duplicates, as described in Section 3.

The ResNet-50 model benefits the most from our MixSize regime on the ImageNet model and so we chose it for further exploration.

We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F).

However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size. The model trained with the \(S^{(208)}\) regime offers a similar improvement in accuracy, but across a larger spatial size, as it observed an average size of \(208\) during training. Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation duplicates in the batch. Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases.

In order to adapt to varying input scales, we add an additional augmentation step, that resizes the images using bilinear interpolation to \(S\times S\), depending on a sampled size for each step. We note that this keeps the exact original augmentation procedure for \(S=32\).

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We show the impact of gradient smoothing on \(B^{+}\) MixSize regimes on CIFAR10 6 and ImageNet 7.

We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

* Increase image size from small to large (progressive resizing), where each image size is used for number of epochs \(E_{i}=p_{i}E_{\mathrm{total}}\), where \(E_{\mathrm{total}}\) is the total number training epochs required.

* Using a random image size for each epoch, keeping the epoch number for each size at \(E_{i}\)

* Sampling image size per training step at probability \(p_{i}\) We used a \(B^{+}\) regime and adapt learning rate for each batch-size according to best practices (Hoffer et al., 2017; Goyal et al., 2017).

We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 6: Impact of gradient smoothing on CIFAR10, ResNet-44. The training regime includes two image sizes: 32 × 32 and 16 × 16 (average size is S = 24). Using a B+ regime creates two batch sizes: 256 and 2, 048 respectively. Gradient norm values are shown with and without Grad-smoothing. Solid lines are gradients for S = 32 while dotted lines are for S = 16. Gradient smoothing helps to reduce gap between gradient norms at difference batch sizes.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and review comments, I'll assess the claim "In this work, we introduce a novel training regime, 'MixSize' for convolutional networks that uses stochastic image and batch sizes."

The evidence demonstrates that while MixSize does introduce a combination of techniques for varying image and batch sizes during training, several key aspects were previously explored:
- Progressive resizing was already established (Howard 2018, Karras 2017)
- Wu et al. (2020) previously balanced resolution with batch size for video training
- The reviewer correctly points out that the speed-accuracy tradeoff was already well-known

However, the specific implementation combining stochastic variation of both image and batch sizes, along with the gradient smoothing technique, appears to be novel. The claim is modestly stated without overstating the novelty or impact.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes.

The review comment to be evaluated is:

This paper presents a mixed-size CNN training scheme, using several different input image sizes for one single model training. The authors assume the training budget, represented as S_i^2*B_i*D_i (i.e., spatial sample size, the number of batched distinct samples and the duplicates for each distinct sample), to be a fixed constant during training step i.  Under such an assumption, two mixed-size training scenarios are considered, one for training acceleration and the other for improved model generalization ability. The authors additionally use step-wise image size sampling, gradient smoothing, and per-size BN calibration to enhance the model performance under the above two mixed-size training scenarios. Experimental validation is performed on CIFAR and ImageNet datasets using diverse CNN structures.

Mixed-size training is a critical problem and the methods proposed in this paper are interesting. My main concerns to this paper are as follows.

--- Critical related works and comparison are missing.

Mixed-size training of CNNs for image classification are not new. Here are some recent works, however they are missed by the authors.

“Resolution Adaptive Networks for Efficient Inference”, in CVPR 2020

 “Resolution Switchable Networks for Runtime Efficient Image Recognition”, in ECCV 2020.

“MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution”, in ECCV 2020.

Besides, as described by the authors, NeurIPS 2019 paper “Fixing the train-test resolution discrepancy” also considers how to enhance the performance of CNN models when applying them with different input image sizes. But a performance comparison is missing.

To show the advantages of this paper, thorough discussion and performance comparison with the above works are necessary. Taking ResNet-50 trained on ImageNet as an instance, I notice that the proposed method shows obviously worse accuracy compared to some of these works.

---Another critical baseline is also missing.

In page 8, “We note that for original fixed-size regimes this calibration procedure resulted, with degraded results and so we report accuracy without calibration for these models”. To my understanding, this is somewhat weird. Why BN calibration does not work on the other image sizes when the model is trained with a fixed image size? It is not clear. Furthermore, in NeurIPS 2019 paper “Fixing the train-test resolution discrepancy”, this line of methods work pretty well. Such a BN calibration should server as another baseline for more fair comparison.

---Regarding B+ design

How about the wall-clock training cost (in hours) instead of the number of iterations/epochs?
How about the performance of applying “scale the learning rate linearly” to train the baseline model?

---How about model transfer ability?

Only image classification tasks are considered in the experiments. How about the performance of the trained backbone models, when transferring them in the downstream tasks, such as object detection and semantic segmentation? Can performance gain be transferred?

---Others

Is there any principled way regarding the size sampling strategy? The current strategy is based on manual setting, which limits its use in real applications.

I suggest the authors to also provide precise accuracy numbers, etc. regarding some figures (e.g., figure 1, figure 4) shown in the paper.

Generally, I am on the fence, to this paper. I encourage the authors to address the questions I raised above.

The evidence to be evaluated is:In addition, previous works explored the notion of "progressive resizing" (Howard, 2018; Karras et al., 2017) -- increasing image size as training progresses to improve model performance and time to convergence. A similar idea by Wu et al. (2020) was used to improve performance of training on video data, by balancing resolution with batch size. Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size, with which evaluation will be performed.

In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient to test time changes of the image size.

We will explore this notion by modifying the optimization method, enabling training with varying batch-sizes and limited hyper-parameter tuning.

To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.

We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time.

This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. We continue to test our hypothesis using a novel mixed-size training regime.

We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions \(H,W\) (height and width) of the input image size2, as well as the batch size.

Formally, in the MixSize regime, these sizes can be described as random variables sharing a single discrete distribution \[(\hat{S},\hat{B},\hat{D})=\left\{(S,B,D)_{i}\;\;\text{w.p.}\;\;p_{i}\right\}, \tag{1}\] where \(\forall i:p_{i}\geq 0\) and \(\sum_{i}p_{i}=1\). As the computational cost of each training step is approximately proportional to \(S^{2}\cdot B\cdot D\), we choose these quantities to reflect a nearly fixed budget for any choice \(i\) such that \(S_{i}^{2}B_{i}D_{i}\approx Const\). Thus, in this regime, the computational and memory requirements for each step in remain fixed.

We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes.

We denote our modified mixed-size regimes as \(B^{+}\) for an increased effective batch-size and \(D^{+}\) for an increased number of BA duplicates of the same ratio. MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size.

To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe three methods we found useful: **step-wise size sampling, gradient smoothing** and **batch-norm calibration**.

We therefore considered two alternatives for image size variations, where size is randomly sampled and changed throughout training: * Sampling image size per epoch

* Sampling image size per training step These sampling methods can be seen as a stochastic alternative to long and short cycle multigrid approach suggested by Wu et al. (2020). We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E). While sampling both at epoch and step time frames performed similarly, replacing sizes on each step converged faster and exhibited less noise in measured test accuracy. Considering these findings, we continue to perform our experiments using the third regime - sampling image size per training step.

This further suggests that the optimization regime should be adapted to smaller spatial sizes, in a manner similar to learning-rate adaptations that are used for large-batch training. This property was explored in previous works concerning large-batch regimes, in which a learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task or across models (Shallue et al., 2018), with solutions such as a square-root scaling (Hoffer et al., 2017), linear scaling (Goyal et al., 2017) or a fixed norm ratio (You et al., 2017). Here we suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance of gradients within each step and further complicates the choice of optimal scaling.

Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training.

Interestingly, we highlight the fact that although this process has little to no effect on models trained using a fixed-size input, it does improve our mixed-size models considerably on a wide range of image sizes. We used our sampling strategy to train and compare our regime to the baseline results of various models over the CIFAR10/100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) visual classification tasks.

As CIFAR datasets are limited in size, we consider the following balanced stochastic regime chosen: \[S^{(28)}:\ S=\begin{cases}40,&\text{w.p.}\quad p=0.2\\ 32,&\text{w.p.}\quad p=0.3\\ 24,&\text{w.p.}\quad p=0.3\\ 16,&\text{w.p.}\quad p=0.2\end{cases}\] The regime was designed to be centered around the mean value of \(28\). As the original image size used for training is \(32\times 32\), we are now able to increase either the batch size or number of BA duplicates for each training step by a factor of \(\frac{32^{2}}{5^{2}}\) such that \(S^{2}\cdot B\cdot D\) is approximately constant. For the ImageNet dataset, we use the following stochastic regime found by cross-validation on several alternatives over the ResNet-50 model (He et al., 2016): \[S^{(144)}:\ \ S=\begin{cases}256,&\text{w.p}\quad p=0.1\\ 224,&\text{w.p}\quad p=0.1\\ 128,&\text{w.p}\quad p=0.6\\ 96,&\text{w.p}\quad p=0.2\end{cases}\] While the original training regime consisted of images of size \(224\times 224\), our proposed regime makes for an average image size of \(\hat{S}\times\hat{S}=144\times 144\). This regime was again designed so that the reduced spatial size can be used to increase the corresponding batch size or the number of BA duplicates, as described in Section 3.

The ResNet-50 model benefits the most from our MixSize regime on the ImageNet model and so we chose it for further exploration.

We enlarge the batch size used for each spatial size by a factor of \(\frac{224^{2}}{S^{2}}\) such that \(S^{2}\cdot B\) is kept approximately fixed. As the average batch size is larger than \(B_{o}\), which was used with the original optimization hyper-parameters, we scale the learning rate linearly as suggested by Goyal et al. (2017) by a factor of \(\frac{\tilde{B}}{B_{o}}\).

As can be seen in Figure 2, regime \(B^{+}\) enables training with approximately \(2.7\times\) fewer training steps, while reaching an improved accuracy of \(76.61\%\). As sizes were chosen to reflect in approximately equal computational cost per iteration, \(B^{+}\) regime offers a similar improvement in total wall-clock time.

We compare these results with \(D^{+}\) regimes, trained with an average size of \(\tilde{S}=144\) and \(\tilde{S}=208\) (see Appendix F).

However, due to the decreased average size, we were able to leverage more than \(1\) duplicates per patch on average, which improved the model's top-1 accuracy to \(77.14\%\) at size \(224\). Furthermore, we find that the model performs much more favorably at sizes smaller than \(224\), scoring an improved (over baseline) accuracy of \(76.43\%\) at only \(160\) spatial size. The model trained with the \(S^{(208)}\) regime offers a similar improvement in accuracy, but across a larger spatial size, as it observed an average size of \(208\) during training. Figure 2(a) demonstrates that while all three models (Fixed with \(S=224\), \(S^{(144)}\) and \(S^{(208)}\)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy over a wide range of evaluation sizes.

In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation duplicates in the batch. Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given a computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen to train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases.

In order to adapt to varying input scales, we add an additional augmentation step, that resizes the images using bilinear interpolation to \(S\times S\), depending on a sampled size for each step. We note that this keeps the exact original augmentation procedure for \(S=32\).

Interestingly, although designed only to reduce training time, we can see that our \(B^{+}\) regime also improves accuracy in some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by an increase in training error throughout its progress.

We show the impact of gradient smoothing on \(B^{+}\) MixSize regimes on CIFAR10 6 and ImageNet 7.

We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension \(H\), we wish to obtain an average size of \(\bar{H}=\sum_{i}p_{i}H_{i}\) such that \(\bar{H}<H_{o}\).

* Increase image size from small to large (progressive resizing), where each image size is used for number of epochs \(E_{i}=p_{i}E_{\mathrm{total}}\), where \(E_{\mathrm{total}}\) is the total number training epochs required.

* Using a random image size for each epoch, keeping the epoch number for each size at \(E_{i}\)

* Sampling image size per training step at probability \(p_{i}\) We used a \(B^{+}\) regime and adapt learning rate for each batch-size according to best practices (Hoffer et al., 2017; Goyal et al., 2017).

We used alternative size regimes balanced around \(224\), named \(S^{(208)}\) and \(S^{(224)}\). They can be described by the following distributions: \[S^{(208)}:\;\;S=\begin{cases}320,&\text{w.p}\;\;\;p=0.1\\ 288,&\text{w.p}\;\;\;p=0.1\\ 256,&\text{w.p}\;\;\;p=0.1\\ 224,&\text{w.p}\;\;\;p=0.2\\ 192,&\text{w.p}\;\;\;p=0.2\\ 160,&\text{w.p}\;\;\;p=0.1\\ 128,&\text{w.p}\;\;\;p=0.1\\ 96,&\text{w.p}\;\;\;p=0.1\end{cases}\]

Input: Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required ≈ 2.7× less steps per epoch.

Input: Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(144)) for progressive resizing (small to large) compared to per-step sampling.

Input: Figure 8: Test accuracy vs step (ResNet44 on CIFAR10,B+ with s(28)) for 3 size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.

Input: Figure 6: Impact of gradient smoothing on CIFAR10, ResNet-44. The training regime includes two image sizes: 32 × 32 and 16 × 16 (average size is S = 24). Using a B+ regime creates two batch sizes: 256 and 2, 048 respectively. Gradient norm values are shown with and without Grad-smoothing. Solid lines are gradients for S = 32 while dotted lines are for S = 16. Gradient smoothing helps to reduce gap between gradient norms at difference batch sizes.

Input: Figure 3: (a): Test accuracy on validation set per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim "we introduce a novel training regime, 'MixSize' for convolutional networks that uses stochastic image and batch sizes" is somewhat overstated. While the paper does present an interesting approach to mixed-size training, the reviewer points out several critical missing comparisons to related work, including "Resolution Adaptive Networks" (CVPR 2020) and other papers that previously explored similar concepts. The evidence shows the method builds upon existing ideas of progressive resizing and multi-size training rather than introducing a completely novel concept.

<score>3</score>

Justification:
1. The claim overstates novelty by not acknowledging significant prior work in mixed-size training, as noted by the reviewer
2. Evidence shows building upon existing concepts like "progressive resizing" (Howard, 2018) and Wu et al.'s (2020) work on balancing resolution with batch size
3. The methodology itself demonstrates innovation in specific implementation details and provides valuable empirical results
4. The reviewer's comments highlight missing comparisons and evaluations that would better contextualize the novelty
5. While not entirely novel, the work does present meaningful contributions to the field through its specific implementation and findings