Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(3) **Striking results of VA-RED\({}^{2}\) over baselines**, with a \(30\%\) reduction in computation in comparison to R(2+1)D (Tran et al., 2018), a \(40\%\) over I3D-InceptionV2 (Carreira and Zisserman, 2017), and about \(20\%\) over the recently proposed X3D-M (Feichtenhofer, 2020) without any performance loss, for video action recognition task.

The review comment to be evaluated is:

Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).

Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

\[Y_{l}[j+iR]=\left\{\begin{array}{ll}\Phi_{i,j}^{t}(Y_{l}^{\prime}[i])&\text{ if }j\in\{1,...,R-1\}\\ Y_{l}^{\prime}[i]&\text{if }j=0\end{array}\right.,i\in\{0,1,...,T_{o}/R-1\}, \tag{1}\]where \(Y_{l}[j+iR]\) represents the \((j+iR)^{th}\) feature map of \(Y_{l}\) along the temporal dimension, \(Y_{l}^{\prime}[i]\) denotes the \(i^{th}\) feature map of \(Y_{l}^{\prime}\), and \(\Phi_{i,j}^{t}\) is the cheap linear operation along the temporal dimension. The total computational cost of this process can be written as: \[\mathcal{C}(f_{l}^{t})=\frac{1}{R}\cdot\mathcal{C}(f_{l})+\sum_{i,j}\mathcal{C }(\Phi_{i,j}^{t})\approx\frac{1}{R}\cdot\mathcal{C}(f_{l}), \tag{2}\] where the function \(\mathcal{C}(\cdot)\) returns the computation cost for a specific operation, and \(f_{l}^{t}\) represents our dynamic convolution process along temporal dimension. Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference. The total computation cost of joint temporal-wise and channel-wise dynamic inference is: \[\mathcal{C}(f_{l}^{t,c})\approx\frac{r}{R}\cdot\mathcal{C}(f_{l}), \tag{3}\]

During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my analysis of the claim versus the evidence:

The claim states VA-RED2 achieves 30%, 40%, and 20% computation reductions compared to R(2+1)D, I3D, and X3D-M respectively "without any performance loss." Examining Table 5, the evidence shows:

- R(2+1)D: ~27% reduction (40.3 vs 55.2 GFLOPs) with slight accuracy gain
- I3D: ~43% reduction (32.1 vs 56.0 GFLOPs) with slight accuracy gain/loss depending on metric
- X3D: ~16% reduction (5.38 vs 6.42 GFLOPs) with small accuracy improvements

The percentage reductions are reasonably close to claimed values, and performance is indeed maintained or slightly improved. The claim is generally well-supported by the empirical evidence, with only minor discrepancies in the exact reduction percentages.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(4) A **generalization of our framework** to video action recognition, spatio-temporal localization, and semantic segmentation tasks, achieving promising results while offering significant reduction in computation over competing methods.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019). Feichtenhofer et al., (Feichtenhofer et al., 2018) propose a dedicated low-framerate pathway. Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

Fan et al. (Fan et al., 2020) incorporate a efficient learnable 3D-shift module into a 3D video network. Wang et al. (Wang et al., 2020) devise a correlation module to learn correlation along temporal dimension. Li et al. (Li et al., 2020) encode the clip-level ordered temporal information with a CIDC network.

Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019) while recognizing actions in long untrimmed videos.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution. Moreover, our method integrates all the inference routes into a single model which is in the almost same size to the original base model.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\).

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

To show the generalization ability to different task, we also conduct the **video spatio-temporal action localization** on J-HMDB-21 (Jhuang et al., 2013).

For **semantic segmentation** experiments, we use ADe20K dataset (Zhou et al., 2017, 2018b), containing 20k images for training and 2k images for validation. ADe20K is a densely labeled image dataset where objects and object parts are segmented down to pixel level.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model. In our implementation of X3D, we remove all the swish non-linearity (Ramachandran et al., 2017) except those in SE layer (Hu et al., 2018) to save training memory and speed up the inference speed on GPU. We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For video spatio-temporal action localization, we adopt YOWO architecture in (Kopuklu et al., 2019) and replace 2D branch with 3D backbone to directly compare them.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions.

**VA-RED\({}^{2}\) on Dense Visual Tasks.** Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018). We report computational cost of model encoder and the mean IoU (Intersection-Over-Union) in Table 11. As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation. This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

* Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pp. 305-321, 2018.

* Yang et al. (2020) Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition.

We evaluate the performance of our approach using three video action recognition datasets, namely Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira and Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB-21 (Jhuang et al., 2013). Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, this claim appears to be moderately overstated. While the framework does demonstrate successful application to video action recognition, spatio-temporal localization, and semantic segmentation tasks with reduced computation, there are several reasons for concern:

1. The semantic segmentation results are limited to just one dataset (ADE20K) and one architecture (PSPNet).

2. The spatio-temporal localization experiments are only shown on one dataset (J-HMDB-21).

3. The term "promising results" understates the variable performance - some models show minimal or no improvement in accuracy despite reduced computation.

4. The "significant reduction" claim isn't consistently supported across all tasks and models tested.

<score>4</score>

The evidence shows the framework has merit but the claim overstates its generalizability and impact without acknowledging key limitations and variability in results across different applications.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(4) A **generalization of our framework** to video action recognition, spatio-temporal localization, and semantic segmentation tasks, achieving promising results while offering significant reduction in computation over competing methods.

The review comment to be evaluated is:

This paper proposes a novel framework called $\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance.

The authors have done extensive experiments on video action recognition tasks and spatio-temporal action localization task in the area of video understanding. For the video action recognition task, experiments are carried out using Mini-Kinetics-200, Kinetics-400, and Moments-In-time datasets. For the action localization task, J-HMDB-21 dataset is used. Results show that this framework is promising, which reduces the computation but main the performance.

Question:
1. X3D-M in the original paper achieved at top-1 74.6 for Kinetics-400 dataset (but in table 4 reports clip-1: 61.8, video-1: 67.9) and FLOPs is 4.73 (6.20 reported in the paper). Can authors explain why there is a difference here?

Minor:
1. Reduce ratio fact for channel-wise dynamic convolution $r=\frac{1}{2}^{p_c}$ such as in Fig.2 on page 4 and equation 7, equation 8 in supplementary materials on page 13. I think it would make more sense representing it as $(\frac{1}{2})^{p_c}$.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019). Feichtenhofer et al., (Feichtenhofer et al., 2018) propose a dedicated low-framerate pathway. Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

Fan et al. (Fan et al., 2020) incorporate a efficient learnable 3D-shift module into a 3D video network. Wang et al. (Wang et al., 2020) devise a correlation module to learn correlation along temporal dimension. Li et al. (Li et al., 2020) encode the clip-level ordered temporal information with a CIDC network.

Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019) while recognizing actions in long untrimmed videos.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution. Moreover, our method integrates all the inference routes into a single model which is in the almost same size to the original base model.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\).

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

To show the generalization ability to different task, we also conduct the **video spatio-temporal action localization** on J-HMDB-21 (Jhuang et al., 2013).

For **semantic segmentation** experiments, we use ADe20K dataset (Zhou et al., 2017, 2018b), containing 20k images for training and 2k images for validation. ADe20K is a densely labeled image dataset where objects and object parts are segmented down to pixel level.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model. In our implementation of X3D, we remove all the swish non-linearity (Ramachandran et al., 2017) except those in SE layer (Hu et al., 2018) to save training memory and speed up the inference speed on GPU. We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For video spatio-temporal action localization, we adopt YOWO architecture in (Kopuklu et al., 2019) and replace 2D branch with 3D backbone to directly compare them.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions.

**VA-RED\({}^{2}\) on Dense Visual Tasks.** Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018). We report computational cost of model encoder and the mean IoU (Intersection-Over-Union) in Table 11. As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation. This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

* Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pp. 305-321, 2018.

* Yang et al. (2020) Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition.

We evaluate the performance of our approach using three video action recognition datasets, namely Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira and Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB-21 (Jhuang et al., 2013). Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find this claim to be moderately overstated. While the framework does show generalization to different tasks (action recognition, localization, and segmentation) with computational reductions, several factors make the claim somewhat inflated:

1. For semantic segmentation, only one dataset (ADE20K) was tested with modest improvements (31.2% to 31.3% mIoU)
2. The reviewer points out discrepancies in reported X3D-M performance numbers
3. While results are "promising," they vary significantly across tasks and models
4. The term "significant reduction" overstates the computational savings, which range from moderate to substantial depending on the specific case

A more measured claim would acknowledge these variations and limitations rather than broadly claiming significant improvements across all applications.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(4) A **generalization of our framework** to video action recognition, spatio-temporal localization, and semantic segmentation tasks, achieving promising results while offering significant reduction in computation over competing methods.

The review comment to be evaluated is:

Pros:
1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications.

2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks.

Cons:
1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a "speed" column in every table of this submission, to compare the actual inference speed among the methods.

2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?

3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models?

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019). Feichtenhofer et al., (Feichtenhofer et al., 2018) propose a dedicated low-framerate pathway. Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

Fan et al. (Fan et al., 2020) incorporate a efficient learnable 3D-shift module into a 3D video network. Wang et al. (Wang et al., 2020) devise a correlation module to learn correlation along temporal dimension. Li et al. (Li et al., 2020) encode the clip-level ordered temporal information with a CIDC network.

Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019) while recognizing actions in long untrimmed videos.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution. Moreover, our method integrates all the inference routes into a single model which is in the almost same size to the original base model.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\).

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

To show the generalization ability to different task, we also conduct the **video spatio-temporal action localization** on J-HMDB-21 (Jhuang et al., 2013).

For **semantic segmentation** experiments, we use ADe20K dataset (Zhou et al., 2017, 2018b), containing 20k images for training and 2k images for validation. ADe20K is a densely labeled image dataset where objects and object parts are segmented down to pixel level.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model. In our implementation of X3D, we remove all the swish non-linearity (Ramachandran et al., 2017) except those in SE layer (Hu et al., 2018) to save training memory and speed up the inference speed on GPU. We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For video spatio-temporal action localization, we adopt YOWO architecture in (Kopuklu et al., 2019) and replace 2D branch with 3D backbone to directly compare them.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions.

**VA-RED\({}^{2}\) on Dense Visual Tasks.** Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018). We report computational cost of model encoder and the mean IoU (Intersection-Over-Union) in Table 11. As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation. This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

* Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pp. 305-321, 2018.

* Yang et al. (2020) Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition.

We evaluate the performance of our approach using three video action recognition datasets, namely Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira and Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB-21 (Jhuang et al., 2013). Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, I find this claim to be moderately overstated. While the evidence shows the framework was successfully applied to video action recognition, localization, and semantic segmentation tasks, some key limitations exist:

1. The reviewer points out that only GFLOPs are reported as efficiency metrics, which is an incomplete measure of actual speed improvements
2. The semantic segmentation results are limited to one dataset (ADE20K) with modest improvements
3. The claim of "significant reduction" needs more context, as the reviewer questions comparisons to similar approaches like X3D
4. The term "promising results" is vague and doesn't fully reflect the mixed performance across different tasks/models shown in the tables

While the framework does demonstrate generalization ability, the language overstates the strength and breadth of the evidence.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(4) A **generalization of our framework** to video action recognition, spatio-temporal localization, and semantic segmentation tasks, achieving promising results while offering significant reduction in computation over competing methods.

The review comment to be evaluated is:

Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019). Feichtenhofer et al., (Feichtenhofer et al., 2018) propose a dedicated low-framerate pathway. Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

Fan et al. (Fan et al., 2020) incorporate a efficient learnable 3D-shift module into a 3D video network. Wang et al. (Wang et al., 2020) devise a correlation module to learn correlation along temporal dimension. Li et al. (Li et al., 2020) encode the clip-level ordered temporal information with a CIDC network.

Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019) while recognizing actions in long untrimmed videos.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution. Moreover, our method integrates all the inference routes into a single model which is in the almost same size to the original base model.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\).

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

To show the generalization ability to different task, we also conduct the **video spatio-temporal action localization** on J-HMDB-21 (Jhuang et al., 2013).

For **semantic segmentation** experiments, we use ADe20K dataset (Zhou et al., 2017, 2018b), containing 20k images for training and 2k images for validation. ADe20K is a densely labeled image dataset where objects and object parts are segmented down to pixel level.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model. In our implementation of X3D, we remove all the swish non-linearity (Ramachandran et al., 2017) except those in SE layer (Hu et al., 2018) to save training memory and speed up the inference speed on GPU. We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For video spatio-temporal action localization, we adopt YOWO architecture in (Kopuklu et al., 2019) and replace 2D branch with 3D backbone to directly compare them.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions.

**VA-RED\({}^{2}\) on Dense Visual Tasks.** Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018). We report computational cost of model encoder and the mean IoU (Intersection-Over-Union) in Table 11. As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation. This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

* Xie et al. (2018) Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pp. 305-321, 2018.

* Yang et al. (2020) Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition.

We evaluate the performance of our approach using three video action recognition datasets, namely Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira and Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB-21 (Jhuang et al., 2013). Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about generalization appears somewhat overstated. While the framework shows promising results across video action recognition, spatio-temporal localization, and semantic segmentation tasks (demonstrated in Tables 4, 5, and 11), the term "generalization" implies broader applicability than what's actually proven. The evidence shows testing on specific datasets and tasks, but doesn't fully demonstrate universal generalization. Additionally, while computation reduction is achieved, the reviewer points out that the core idea isn't novel, with similar approaches like AR-Net already existing. The improvement over competing methods isn't consistently demonstrated across all scenarios.

<score>4</score>

Justification:
1. The term "generalization" is too broad given the limited scope of testing
2. While positive results are shown, they are specific to certain datasets/tasks
3. Reviewer notes similar existing approaches like AR-Net
4. Computation reduction claims are supported but comparative advantages aren't comprehensively proven
5. Results show promise but don't fully support the broad generalization claim made


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Specifically, our VA-RED\({}^{2}\) framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions.

The evidence to be evaluated is:While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

We adopt an extremely lightweight policy layer \(p_{l}\) called soft modulation gate for each convolution layer \(f_{l}\) to modulate the ratio of features which need to be computed. Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively.

\[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.

\[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\). During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.

To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture. The channel-wise policy has also more variation than the temporal-wise policy among different categories.

Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

Specifically, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture. However, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy.

We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene configuration, like the first and second examples in Figure 12.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 5: Visualization of the first 9 filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and 3. As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists of a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim "VA-RED2 framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions" is well-stated and accurately reflects the evidence without exaggeration. The evidence strongly supports this through:

1. Detailed mathematical formulations showing how the framework computes input-dependent policies
2. Visualization evidence (Figures 8-9) demonstrating different computation patterns across video categories
3. Experimental results (Table 10) showing the effectiveness of dynamic modeling along both dimensions
4. Example cases (Figures 11-12) illustrating how different video content leads to different computational requirements

The claim straightforwardly describes the core functionality without overstating capabilities or results.

<score>1</score>