Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

(1) A novel **input-dependent adaptive framework** for efficient video recognition, VA-RED\({}^{2}\), that automatically decides what feature maps to compute per input instance.

The review comment to be evaluated is:

Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction.

The evidence to be evaluated is:While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Many adaptive computation methods have been recently proposed with the goal of improving efficiency (Bengio et al., 2015, 2013; Veit and Belongie, 2018; Wang et al., 2018; Graves, 2016; Meng et al., 2021). Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Yu et al., 2018; Figurnov et al., 2017; McGill and Perona, 2017; Teerapittayanon et al., 2016) Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training. Veit et al. (Veit and Belongie, 2018) propose a block skipping method controlled by samples from a Gumbel softmax, while Wu et al. (Wu et al., 2018) develop a reinforcement learning approach to achieve this goal. Adaptive computation time for recurrent neural networks is also presented in (Graves, 2016). SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition.

In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference. The total computation cost of joint temporal-wise and channel-wise dynamic inference is: \[\mathcal{C}(f_{l}^{t,c})\approx\frac{r}{R}\cdot\mathcal{C}(f_{l}), \tag{3}\] where \(f_{l}^{t,c}\) is the adjunct process of temporal-wise and channel-wise dynamic inference.

We adopt an extremely lightweight policy layer \(p_{l}\) called soft modulation gate for each convolution layer \(f_{l}\) to modulate the ratio of features which need to be computed. Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively. The \(V_{t}^{l}\) and \(V_{c}^{t}\) are learned by: \[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.

\[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\). During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

To encourage our network to output a computational efficient subgraph, we introduce the efficiency loss \(\mathcal{L}_{c}\) during the training process, which can be formulated as \[\mathcal{L}_{e}=(\mu_{0}\sum_{l=1}^{L}\frac{\mathcal{C}(f_{l})}{\sum_{k=1}^{L }\mathcal{C}(f_{k})}\cdot\frac{r_{l}^{s}}{R_{l}^{s}})^{2},\mu_{0}=\left\{ \begin{array}{ll}1&\text{if correct}\\ 0&\text{otherwise}\end{array}\right., \tag{6}\] where \(r_{l}^{s}\) is channel scaling factor of the largest filter in the series of channel-wise dynamic convolutions, and \(R_{l}^{s}\) is stride scaling factor of the largest filter of temporal-wise dynamic convolutions. Overall, the loss function of our whole framework can be written as \(\mathcal{L}=\mathcal{L}_{a}+\lambda_{e}\mathcal{L}_{e}\), where \(L_{a}\) is the accuracy loss of the whole network and \(\lambda_{e}\) is the weight of efficiency loss which can be used to balance the importance of the optimization of prediction accuracy and computational cost.

We conduct our **video action recognition** experiments on three standard benchmarks: Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.

To show the generalization ability to different task, we also conduct the **video spatio-temporal action localization** on J-HMDB-21 (Jhuang et al., 2013).

We test performance of or approach by turning of dynamic modeling along temporal and channel dimensions on Mini-Kinetics-200. Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture. The channel-wise policy has also more variation than the temporal-wise policy among different categories. Furthermore, we show few contrasting examples which are in the same category while requiring very different computation in Figure 4. Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions.

**VA-RED\({}^{2}\) on Dense Visual Tasks.** Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018). We report computational cost of model encoder and the mean IoU (Intersection-Over-Union) in Table 11. As can be seen from Table 11, our proposed VA-RED\({}^{2}\) has the absolute advantage in terms of efficiency while maintaining the precision of segmentation.

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. For example, the original channel-wise dynamic convolution at ratio \(r=(\frac{1}{2})^{i(i-1)}\) can be analogized to

where \([\cdot\ \cdot]\) is the index operation along the channel dimension, and \(i_{s}^{c}\) is the index of the largest channel-wise filter, during training phase, we have \(i_{s}^{c}=1\), while during inference phase, \(i_{s}^{c}\) is the smallest index for \(V_{c}^{l}\), \(s.t.V_{c}^{l}[i_{s}^{c}]=0\). By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to \((\frac{1}{2})^{i_{s}^{c}-1}\cdot\mathcal{C}(f_{l})\).

In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

Specifically, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture. However, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. Furthermore, we count the FLOPs of each instance on Mini-Kinetics-200, Kinetics-400, and Moments-In-Time and plot pie charts to visualize the the distribution of this instance-level computational cost. We analyze such distribution with two models: R(2+1)D-18 and X3D-M. All of the results are demonstrated in Figure 10.

We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene configuration, like the first and second examples in Figure 12.

Input: Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Table 11: VA-RED2 on semantic segmentation. We choose dilated ResNet-18 as our backbone architecture and set the search space as 2. Models are trained for 100K iterations with batch size of 8.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 5: Visualization of the first 9 filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and 3. As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists of a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I carefully evaluated the claim regarding VA-RED2 as an input-dependent adaptive framework. The evidence strongly supports this claim through:

- Detailed mathematical formulation showing how the framework adapts computation based on input characteristics
- Experimental results across multiple datasets demonstrating input-dependent behavior
- Visualizations (Figures 4,11,12) showing how different inputs require varying amounts of computation
- Ablation studies validating the importance of input-dependent policies
- Demonstrated generalization to different tasks (action recognition, localization, segmentation)
- Review comments acknowledge the empirical contributions while noting similarity to prior work

While the core adaptive concept isn't entirely novel (as noted in the review), the claim about it being "novel" is slightly overstated. However, the input-dependent adaptation mechanism itself is well-supported and accurately described.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Here we present a redundancy reduction framework, termed VA-RED\({}^{2}\), which is _input-dependent_.

The evidence to be evaluated is:While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training.

SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively.

\[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.

\[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

To better understand the policy decision process, we dissect the network layers and count the ratio of feature maps that are being computed during each convolution layers for each category.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We show additional input examples which consume different levels of computational cost on Kinetics-400 dataset (c.f. Figure 11) and Moments-In-Time dataset (c.f. Figure 12).

We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene configuration, like the first and second examples in Figure 12.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 5: Visualization of the first 9 filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and 3. As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists of a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim "VA-RED2 framework is input-dependent" is well-stated and supported by extensive evidence. The framework demonstrates clear input-dependent behavior through:

1. Dynamic computation based on video content complexity, shown in Figures 11-12 where videos with less motion/simpler scenes require less computation
2. Per-input adaptive feature map computation through soft modulation gates that determine temporal and channel-wise computation ratios
3. Detailed mathematical formulation showing how the framework makes decisions based on input features
4. Visualization evidence (Figures 6-7) demonstrating different computational patterns for different inputs

The claim accurately reflects the framework's demonstrated capability without exaggeration or overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Here we present a redundancy reduction framework, termed VA-RED\({}^{2}\), which is _input-dependent_.

The review comment to be evaluated is:

This paper proposes a novel framework called $\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance.

The authors have done extensive experiments on video action recognition tasks and spatio-temporal action localization task in the area of video understanding. For the video action recognition task, experiments are carried out using Mini-Kinetics-200, Kinetics-400, and Moments-In-time datasets. For the action localization task, J-HMDB-21 dataset is used. Results show that this framework is promising, which reduces the computation but main the performance.

Question:
1. X3D-M in the original paper achieved at top-1 74.6 for Kinetics-400 dataset (but in table 4 reports clip-1: 61.8, video-1: 67.9) and FLOPs is 4.73 (6.20 reported in the paper). Can authors explain why there is a difference here?

Minor:
1. Reduce ratio fact for channel-wise dynamic convolution $r=\frac{1}{2}^{p_c}$ such as in Fig.2 on page 4 and equation 7, equation 8 in supplementary materials on page 13. I think it would make more sense representing it as $(\frac{1}{2})^{p_c}$.

The evidence to be evaluated is:While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training.

SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively.

\[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.

\[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

To better understand the policy decision process, we dissect the network layers and count the ratio of feature maps that are being computed during each convolution layers for each category.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We show additional input examples which consume different levels of computational cost on Kinetics-400 dataset (c.f. Figure 11) and Moments-In-Time dataset (c.f. Figure 12).

We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene configuration, like the first and second examples in Figure 12.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 5: Visualization of the first 9 filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and 3. As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists of a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate whether the claim that VA-RED2 is "input-dependent" is well-stated or overstated.

Summary: The evidence strongly supports that the framework is truly input-dependent through multiple demonstrations:

1. Figures 11 and 12 show different computational costs for different video inputs within the same action categories
2. Figure 10 demonstrates varying computational distributions across different datasets
3. Figures 8-9 visualize how the framework adapts feature computation ratios per layer based on input content
4. The mechanism is explicitly input-dependent through the "soft modulation gate" that takes input features and learns probability vectors to determine computation ratios
5. Qualitative examples show the framework using less computation for videos with less motion/simpler scenes

The claim appears to be well-stated and accurately reflects the evidence without exaggeration. The framework demonstrably adapts its computation based on input characteristics.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Here we present a redundancy reduction framework, termed VA-RED\({}^{2}\), which is _input-dependent_.

The review comment to be evaluated is:

Pros:
1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications.

2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks.

Cons:
1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a "speed" column in every table of this submission, to compare the actual inference speed among the methods.

2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?

3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models?

The evidence to be evaluated is:While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training.

SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively.

\[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.

\[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

To better understand the policy decision process, we dissect the network layers and count the ratio of feature maps that are being computed during each convolution layers for each category.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We show additional input examples which consume different levels of computational cost on Kinetics-400 dataset (c.f. Figure 11) and Moments-In-Time dataset (c.f. Figure 12).

We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene configuration, like the first and second examples in Figure 12.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 5: Visualization of the first 9 filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and 3. As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists of a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim "Here we present a redundancy reduction framework, termed VA-RED2, which is input-dependent" based on the provided evidence:

Summary: The evidence strongly supports that VA-RED2 is truly input-dependent. Key supporting points include:
- The framework dynamically adjusts computations based on input video characteristics (shown in Figures 11-12 where simpler/static scenes use less computation)
- Detailed technical evidence showing how the framework uses soft modulation gates to determine feature map ratios for each input
- Visualization evidence (Figures 3-9) demonstrating different computational patterns for different input types
- Quantitative results showing varying computational costs across different input videos (Figure 10)

The claim appears to be well-stated (not overstated) as it simply states the framework is input-dependent, which is thoroughly demonstrated by multiple forms of evidence. The claim avoids exaggeration while accurately representing a core characteristic of the framework.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Here we present a redundancy reduction framework, termed VA-RED\({}^{2}\), which is _input-dependent_.

The review comment to be evaluated is:

Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction.

The evidence to be evaluated is:While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.

Wang et al. (Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training.

SpotTune (Guo et al., 2019) learns to adaptively route information through finetuned or pre-trained layers.

Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition.

In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy. Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.

For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.

For each 3D convolution layer, we use a very lightweight policy layer \(p_{l}\) denoted as _soft modulation gate_ to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.

Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with a factor \(r=(\frac{1}{2})^{p_{l}(X_{l})[1]}\). In this case, the shape of output \(Y_{l}^{\prime}\) is \(rC_{out}\times T_{o}\times H_{o}\times W_{o}\). Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as \(Y_{l}=[Y_{l}^{\prime},\Phi^{c}(Y_{l}^{\prime})]\), where \(\Phi^{c}(Y_{l}^{\prime})\in R^{(1-r)C_{out}\times T_{o}\times H_{o}\times W_{o}}\) represents the cheaply generated feature maps along the channel dimension, and \(Y_{l}\in R^{C_{out}\times T_{o}\times H_{o}\times W_{o}}\) is the output of the channel-wise dynamic inference.

Specifically, the soft modulation gate takes the input feature maps \(X_{l}\) as input and learns two probability vectors \(V_{t}^{l}\in R^{S_{t}}\) and \(V_{c}^{t}\in R^{S_{c}}\), where \(S_{t}\) and \(S_{c}\) are the temporal search space size and the channel search space size respectively.

\[[V_{t}^{l},V_{c}^{l}]=p_{l}(X_{l})=\phi(\mathcal{F}(\omega_{p,2},\delta( \mathcal{N}(\mathcal{F}(\omega_{p,1},G(X_{l})))))+\beta_{p}^{l}), \tag{4}\] where \(\mathcal{F}(\cdot,\cdot)\) denotes the fully-connected layer, \(\mathcal{N}\) is the batch normalization, \(\delta(\cdot)\) represents the \(\tanh(\cdot)\) function, \(G\) is the global pooling operation whose output shape is \(C_{in}\cdot T\times 1\times 1\), \(\phi(\cdot)\) is the output activation function, here we just use \(\max(\tanh(\cdot),0)\) whose output range is \([0,1)\), and \(\omega_{p,1}\in R^{(S_{t}+S_{c})\times D_{h}}\), \(\omega_{p,2}\in R^{D_{h}\times C_{in}\cdot T}\) are the weights of their corresponding layers, \(D_{h}\) is the hidden dimension number. \(V_{t}^{l}\) and \(V_{c}^{l}\) will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.

\[Y_{c}^{l}=\sum_{i=1}^{S_{c}}V_{c}^{l}[i]\cdot f_{l}^{c}(X_{l},r=(\frac{1}{2})^{(i -1)}),\quad Y_{l}=\sum_{j=1}^{S_{t}}V_{l}^{l}[j]\cdot f_{l}^{t}(Y_{c}^{l},R=2^{(j -1)}), \tag{5}\] where \(f_{l}^{c}(\cdot,r)\) is the channel-wise dynamic convolution with the channel scaling factor \(r\), and \(f_{l}^{t}(\cdot,R)\) it the temporal-wise dynamic convolution with the temporal stride scaling factor \(R\).

To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at a constant value invariant to the search space.

This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.

To better understand the policy decision process, we dissect the network layers and count the ratio of feature maps that are being computed during each convolution layers for each category.

Our VA-RED\({}^{2}\) framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018).

In this paper, we propose an input-dependent adaptive framework called VA-RED\({}^{2}\) for efficient inference which can be easily plugged into most of existing video understanding models to significantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained 13D-InceptionV2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, we first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.

We show additional input examples which consume different levels of computational cost on Kinetics-400 dataset (c.f. Figure 11) and Moments-In-Time dataset (c.f. Figure 12).

We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene configuration, like the first and second examples in Figure 12.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 1: Our VA-RED2 framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows a case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes a subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing a similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.

Input: Figure 12: Validation video clips from Moments-In-Time. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Moments-In-Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which is pre-trained on Kinetics-400. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 1 2 of them both on temporal and channel-wise convolutions.

Input: Figure 8: Ratio of computed feature per layer and class on Kinetics-400 dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporalwise features at the early stage and compute more channel-wise features at the late stage, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.

Input: Figure 9: Ratio of computed feature per layer and class on Moments-In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D-18 on all 339 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. Φt and Φs represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2pt to reduce computation, where pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of r = ( 1 2 )pc , where pc is the channel policy. Best viewed in color.

Input: Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost by different models on different datasets. For instance, for the upper-left one, we use the model backbone of R(2+1)D-18 on Mini-Kinetics-200. This sub-figure indicates that there are 87.7% of videos in Mini-Kinetics-200 (Dataset) consuming 38.6− 41.4 GFLOPs by using R(2+1)D-18 (Backbone), 8.8% of videos consuming 35.9− 38.6 GFLOPs, and 3.5% of videos consuming 41.4− 44.2 GFLOPs.

Input: Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200. These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBlock_1. We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach. Best viewed in color.

Input: Figure 7: Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes are cheaply generated. The analysis is performed on 8-frame dynamic R(2+1)D-18 model which is pretrained on Mini-Kinetics-200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock_1. Best viewed in color.

Input: Figure 4: Validation video clips from Mini-Kinetics-200. For each category, we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained on Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.

Input: Figure 3: Ratio of computed feature per layer and class on Mini-Kinetics-200 dataset. We pick the first 25 classes of Mini-Kinetics-200 and visualize the per-block policy of X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.

Input: Figure 5: Visualization of the first 9 filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and 3. As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists of a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim that the framework is "input-dependent" appears to be well-stated (score of 1) for these reasons:

1. The extensive empirical evidence demonstrates that the framework genuinely adapts computation based on input characteristics:
- Figures 11-12 show different computational costs for different video inputs based on their motion/complexity
- The distribution plots in Figure 10 confirm varying computational costs across inputs
- Figures 8-9 visualize how feature computation varies per layer and class
- Quantitative results in Table 5 show efficiency gains while maintaining accuracy

2. The technical details clearly describe the input-dependent policy mechanism through:
- A soft modulation gate that analyzes input features
- Dynamic adjustment of temporal and channel-wise computations
- Clear mathematical formulation of how the policy is derived from inputs

The evidence thoroughly validates that the framework's behavior is truly input-dependent, making this a well-supported and accurately stated claim.

<score>1</score>