Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains.

The review comment to be evaluated is:

This paper proposes a novel framework called $\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance.

The authors have done extensive experiments on video action recognition tasks and spatio-temporal action localization task in the area of video understanding. For the video action recognition task, experiments are carried out using Mini-Kinetics-200, Kinetics-400, and Moments-In-time datasets. For the action localization task, J-HMDB-21 dataset is used. Results show that this framework is promising, which reduces the computation but main the performance.

Question:
1. X3D-M in the original paper achieved at top-1 74.6 for Kinetics-400 dataset (but in table 4 reports clip-1: 61.8, video-1: 67.9) and FLOPs is 4.73 (6.20 reported in the paper). Can authors explain why there is a difference here?

Minor:
1. Reduce ratio fact for channel-wise dynamic convolution $r=\frac{1}{2}^{p_c}$ such as in Fig.2 on page 4 and equation 7, equation 8 in supplementary materials on page 13. I think it would make more sense representing it as $(\frac{1}{2})^{p_c}$.

The evidence to be evaluated is:Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

We adopt an extremely lightweight policy layer \(p_{l}\) called soft modulation gate for each convolution layer \(f_{l}\) to modulate the ratio of features which need to be computed. Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively. The \(V_{t}^{l}\) and \(V_{c}^{t}\) are learned by: \[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution. During training, we obtain the final output of the dynamic convolution by weighted sum of all the feature maps which contains different ratio of fully-computed features as follows: \[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space.

We test performance of or approach by turning of dynamic modeling along temporal and channel dimensions on Mini-Kinetics-200. Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We demonstrate those in both temporal dimension (c.f. Figure 6) and channel dimension (c.f. Figure 7). In both cases we can see that the proposed cheap operation generates meaningful feature maps and some of them looks even no difference from the original feature maps.

Specifically, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate the claim: "Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains."

The evidence strongly supports this claim by demonstrating:
1. Successful application across multiple architectures (I3D, R(2+1)D, X3D)
2. Implementation on different tasks (action recognition, segmentation)
3. Empirical validation through comprehensive experiments on various datasets
4. Visualization evidence showing feature redundancy reduction in both temporal and channel dimensions
5. Quantitative improvements in efficiency while maintaining accuracy

However, while the approach shows broad applicability, calling it applicable to "any backbones" may be slightly overstated since it's primarily demonstrated on video understanding models and one segmentation model. The claim would be more precise if it specified "common video understanding architectures."

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains.

The review comment to be evaluated is:

Pros:
1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications.

2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks.

Cons:
1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a "speed" column in every table of this submission, to compare the actual inference speed among the methods.

2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?

3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models?

The evidence to be evaluated is:Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

We adopt an extremely lightweight policy layer \(p_{l}\) called soft modulation gate for each convolution layer \(f_{l}\) to modulate the ratio of features which need to be computed. Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively. The \(V_{t}^{l}\) and \(V_{c}^{t}\) are learned by: \[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution. During training, we obtain the final output of the dynamic convolution by weighted sum of all the feature maps which contains different ratio of fully-computed features as follows: \[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space.

We test performance of or approach by turning of dynamic modeling along temporal and channel dimensions on Mini-Kinetics-200. Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We demonstrate those in both temporal dimension (c.f. Figure 6) and channel dimension (c.f. Figure 7). In both cases we can see that the proposed cheap operation generates meaningful feature maps and some of them looks even no difference from the original feature maps.

Specifically, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the comprehensive evidence provided, I'll evaluate the claim "Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains."

The evidence shows the method was successfully applied to multiple architectures including I3D, R(2+1)D-18, X3D-M, and PSPNet with ResNet-18 backbone, demonstrating broad applicability. However, there are some limitations:

1. The approach is primarily tested on video models and one semantic segmentation model
2. It mainly focuses on 2D/3D CNN architectures
3. The review raises valid concerns about speed metrics and similarities to X3D
4. The method requires specific modifications to integrate with different architectures

While the approach shows good versatility, claiming it works with "any backbones" is somewhat overstated. A more precise claim would specify the types of architectures it's been validated on.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains.

The review comment to be evaluated is:

Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction.

The evidence to be evaluated is:Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

Without loss of generality, we start from a 3D convolutional network \(\mathcal{G}\), and denote its \(l^{th}\) 3D convolution layer as \(f_{l}\), and the corresponding input and output feature maps as \(X_{l}\) and \(Y_{l}\) respectively. For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

We adopt an extremely lightweight policy layer \(p_{l}\) called soft modulation gate for each convolution layer \(f_{l}\) to modulate the ratio of features which need to be computed. Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively. The \(V_{t}^{l}\) and \(V_{c}^{t}\) are learned by: \[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution. During training, we obtain the final output of the dynamic convolution by weighted sum of all the feature maps which contains different ratio of fully-computed features as follows: \[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017; He et al., 2016) as our backbone architecture.

We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space.

We test performance of or approach by turning of dynamic modeling along temporal and channel dimensions on Mini-Kinetics-200. Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation. In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output.

By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We demonstrate those in both temporal dimension (c.f. Figure 6) and channel dimension (c.f. Figure 7). In both cases we can see that the proposed cheap operation generates meaningful feature maps and some of them looks even no difference from the original feature maps.

Specifically, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 4: Comparison with CorrNet (Wang et al., 2020) and AR-Net (Meng et al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains" appears to be somewhat overstated, though not entirely unfounded.

The evidence shows successful application across several architectures (I3D, R(2+1)D, X3D) and tasks (recognition, segmentation), demonstrating broad applicability. However, there are important limitations:

1. Testing is primarily focused on video recognition models
2. The method requires specific architectural modifications (soft modulation gates, reconstruction operations)
3. Implementation details show it's mainly applied to 2D convolutions in R(2+1)D
4. The review points out similarity to existing approaches like AR-Net

While the approach shows promising generalizability, claiming it's "model-agnostic" and applicable to "any backbones" overstates its proven capabilities.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(3) **Striking results of VA-RED\({}^{2}\) over baselines**, with a \(30\%\) reduction in computation in comparison to R(2+1)D (Tran et al., 2018), a \(40\%\) over I3D-InceptionV2 (Carreira and Zisserman, 2017), and about \(20\%\) over the recently proposed X3D-M (Feichtenhofer, 2020) without any performance loss, for video action recognition task.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).

Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

\[Y_{l}[j+iR]=\left\{\begin{array}{ll}\Phi_{i,j}^{t}(Y_{l}^{\prime}[i])&\text{ if }j\in\{1,...,R-1\}\\ Y_{l}^{\prime}[i]&\text{if }j=0\end{array}\right.,i\in\{0,1,...,T_{o}/R-1\}, \tag{1}\]where \(Y_{l}[j+iR]\) represents the \((j+iR)^{th}\) feature map of \(Y_{l}\) along the temporal dimension, \(Y_{l}^{\prime}[i]\) denotes the \(i^{th}\) feature map of \(Y_{l}^{\prime}\), and \(\Phi_{i,j}^{t}\) is the cheap linear operation along the temporal dimension. The total computational cost of this process can be written as: \[\mathcal{C}(f_{l}^{t})=\frac{1}{R}\cdot\mathcal{C}(f_{l})+\sum_{i,j}\mathcal{C }(\Phi_{i,j}^{t})\approx\frac{1}{R}\cdot\mathcal{C}(f_{l}), \tag{2}\] where the function \(\mathcal{C}(\cdot)\) returns the computation cost for a specific operation, and \(f_{l}^{t}\) represents our dynamic convolution process along temporal dimension. Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference. The total computation cost of joint temporal-wise and channel-wise dynamic inference is: \[\mathcal{C}(f_{l}^{t,c})\approx\frac{r}{R}\cdot\mathcal{C}(f_{l}), \tag{3}\]

During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is somewhat overstated. While the evidence shows computational reductions using VA-RED², the specific percentage claims (30%, 40%, 20%) aren't directly supported by the data presented in the tables. From Table 5, we can calculate:

- R(2+1)D: ~27% reduction (40.3 vs 55.2 GFLOPs for 16-frame)
- I3D: ~43% reduction (32.1 vs 56.0 GFLOPs for 16-frame)
- X3D: ~16% reduction (5.38 vs 6.42 GFLOPs)

Additionally, there are some small performance variations (both positive and negative) rather than "without any performance loss" as claimed.

<score>3</score>

Justification:
1. The percentage reductions claimed don't exactly match the empirical results shown in tables
2. The claim of "no performance loss" isn't fully accurate - there are small variations in performance metrics
3. While the general direction of the claim (significant computation reduction) is supported, the specific numbers are overstated
4. The evidence from Table 5 demonstrates the method is effective but not precisely to the claimed degree


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(3) **Striking results of VA-RED\({}^{2}\) over baselines**, with a \(30\%\) reduction in computation in comparison to R(2+1)D (Tran et al., 2018), a \(40\%\) over I3D-InceptionV2 (Carreira and Zisserman, 2017), and about \(20\%\) over the recently proposed X3D-M (Feichtenhofer, 2020) without any performance loss, for video action recognition task.

The review comment to be evaluated is:

This paper proposes a novel framework called $\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance.

The authors have done extensive experiments on video action recognition tasks and spatio-temporal action localization task in the area of video understanding. For the video action recognition task, experiments are carried out using Mini-Kinetics-200, Kinetics-400, and Moments-In-time datasets. For the action localization task, J-HMDB-21 dataset is used. Results show that this framework is promising, which reduces the computation but main the performance.

Question:
1. X3D-M in the original paper achieved at top-1 74.6 for Kinetics-400 dataset (but in table 4 reports clip-1: 61.8, video-1: 67.9) and FLOPs is 4.73 (6.20 reported in the paper). Can authors explain why there is a difference here?

Minor:
1. Reduce ratio fact for channel-wise dynamic convolution $r=\frac{1}{2}^{p_c}$ such as in Fig.2 on page 4 and equation 7, equation 8 in supplementary materials on page 13. I think it would make more sense representing it as $(\frac{1}{2})^{p_c}$.

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).

Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

\[Y_{l}[j+iR]=\left\{\begin{array}{ll}\Phi_{i,j}^{t}(Y_{l}^{\prime}[i])&\text{ if }j\in\{1,...,R-1\}\\ Y_{l}^{\prime}[i]&\text{if }j=0\end{array}\right.,i\in\{0,1,...,T_{o}/R-1\}, \tag{1}\]where \(Y_{l}[j+iR]\) represents the \((j+iR)^{th}\) feature map of \(Y_{l}\) along the temporal dimension, \(Y_{l}^{\prime}[i]\) denotes the \(i^{th}\) feature map of \(Y_{l}^{\prime}\), and \(\Phi_{i,j}^{t}\) is the cheap linear operation along the temporal dimension. The total computational cost of this process can be written as: \[\mathcal{C}(f_{l}^{t})=\frac{1}{R}\cdot\mathcal{C}(f_{l})+\sum_{i,j}\mathcal{C }(\Phi_{i,j}^{t})\approx\frac{1}{R}\cdot\mathcal{C}(f_{l}), \tag{2}\] where the function \(\mathcal{C}(\cdot)\) returns the computation cost for a specific operation, and \(f_{l}^{t}\) represents our dynamic convolution process along temporal dimension. Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference. The total computation cost of joint temporal-wise and channel-wise dynamic inference is: \[\mathcal{C}(f_{l}^{t,c})\approx\frac{r}{R}\cdot\mathcal{C}(f_{l}), \tag{3}\]

During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I find this claim to be moderately overstated. Here's why:

1. For R(2+1)D: Table 5 shows reduction from 55.2 to 40.3 GFLOPs (27% reduction, not 30%) for 16-frame model.

2. For I3D: Table 5 shows reduction from 56.0 to 32.1 GFLOPs (43% reduction - this part is roughly accurate at ~40%).

3. For X3D-M: Reduction from 6.42 to 5.38 GFLOPs (16% reduction, not 20%).

Additionally, the review comment raises concerns about inconsistencies in the X3D-M baseline numbers compared to the original paper, which weakens the reliability of the comparison.

While the method does achieve significant computational reductions while maintaining performance, the specific percentage claims are slightly overstated compared to the actual results shown in the tables.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(3) **Striking results of VA-RED\({}^{2}\) over baselines**, with a \(30\%\) reduction in computation in comparison to R(2+1)D (Tran et al., 2018), a \(40\%\) over I3D-InceptionV2 (Carreira and Zisserman, 2017), and about \(20\%\) over the recently proposed X3D-M (Feichtenhofer, 2020) without any performance loss, for video action recognition task.

The review comment to be evaluated is:

Pros:
1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications.

2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks.

Cons:
1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a "speed" column in every table of this submission, to compare the actual inference speed among the methods.

2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?

3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models?

The evidence to be evaluated is:Despite promising results on common benchmarks, there is a significant interest in developing more efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).

Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (Feichtenhofer, 2020).

While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor \(R=2^{p_{l}(X_{l})[0]}\). Thus the shape of output \(Y_{l}^{\prime}\) becomes \(C_{out}\times T_{o}/R\times H_{o}\times W_{o}\).

\[Y_{l}[j+iR]=\left\{\begin{array}{ll}\Phi_{i,j}^{t}(Y_{l}^{\prime}[i])&\text{ if }j\in\{1,...,R-1\}\\ Y_{l}^{\prime}[i]&\text{if }j=0\end{array}\right.,i\in\{0,1,...,T_{o}/R-1\}, \tag{1}\]where \(Y_{l}[j+iR]\) represents the \((j+iR)^{th}\) feature map of \(Y_{l}\) along the temporal dimension, \(Y_{l}^{\prime}[i]\) denotes the \(i^{th}\) feature map of \(Y_{l}^{\prime}\), and \(\Phi_{i,j}^{t}\) is the cheap linear operation along the temporal dimension. The total computational cost of this process can be written as: \[\mathcal{C}(f_{l}^{t})=\frac{1}{R}\cdot\mathcal{C}(f_{l})+\sum_{i,j}\mathcal{C }(\Phi_{i,j}^{t})\approx\frac{1}{R}\cdot\mathcal{C}(f_{l}), \tag{2}\] where the function \(\mathcal{C}(\cdot)\) returns the computation cost for a specific operation, and \(f_{l}^{t}\) represents our dynamic convolution process along temporal dimension. Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference. The total computation cost of joint temporal-wise and channel-wise dynamic inference is: \[\mathcal{C}(f_{l}^{t,c})\approx\frac{r}{R}\cdot\mathcal{C}(f_{l}), \tag{3}\]

During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

We evaluate our method on three most widely-used model architectures: I3D (Carreira and Zisserman, 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer, 2020). We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D-18 (denoted as R(2+1)D below) as our base model.

We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.

**Results on Video Action Recognition.** We first evaluate our method by applying it to R(2+1)D-18 (Tran et al., 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with _K-LeftCenterRight_ strategy. Note that we report the FLOPs of a single video clips at the spatial resolution \(256\times 256\) (for I3D and X3D) or \(128\times 128\) (for R(2+1)D). In addition, we report the speed of each model with the metric of \(clip/second\), which denotes the number of video clips that are processed in one second.

Table 9 shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.

Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim about VA-RED2's computational reductions requires careful analysis:

1. For R(2+1)D: The claim states 30% reduction, and Table 5 shows reduction from 55.2 to 40.3 GFLOPs (16-frame) and 110.5 to 80.7 GFLOPs (32-frame), which is roughly ~27-30% reduction.

2. For I3D: Claim states 40% reduction. Table 5 shows reduction from 56.0 to 32.1 GFLOPs (16-frame) and 112.0 to 64.3 GFLOPs (32-frame), which is ~42-43% reduction.

3. For X3D-M: Claim states 20% reduction. Table 5 shows reduction from 6.42 to 5.38 GFLOPs, which is ~16% reduction.

However, the reviewer raises a valid concern that GFLOPs alone isn't a reliable metric for speed comparison. The claim doesn't acknowledge this limitation. While the numerical reductions are mostly accurate, the presentation could be more nuanced by addressing the relationship between GFLOPs and actual inference speed.

<score>2</score>