Capturing spatiotemporal dynamics is an essential topic in video recognition. In this paper, we present learnable higher-order operation as a generic family of building blocks for capturing higher-order correlations from high dimensional input video space. We prove that several successful architectures for visual classification tasks are in the family of higher-order neural networks, theoretical and experimental analysis demonstrates their underlying mechanism is higher-order.   On the task of video recognition, even using RGB only without fine-tuning with other video datasets, our higher-order models can achieve results on par with or better than the existing state-of-the-art methods on both Something-Something (V1 and V2) and Charades datasets. Actions in videos arise from motions of objects with respect to other objects and/or the background. To understand an action, an effective architecture should recognize not only the appearance of the target object associated with the action, but also how it relates to other objects in the scene, in both space and time. Figure 1 shows four different categories of actions. Each column shown an action where, in temporal order, the figures above occur before the figures below. Recognizing the hand and the object is not enough. To distinguish left to right motion from right to left motion, the model must know how the hand moves against the background. It is more complicated to classify pull and push since it is an XOR operation on the relative positions of the hand and the object resulting from the hand's movements. Figure 1a , since the hand moves from left to right and the hand is on the right side of the iron, it is pull from left to right. Figure 1d has the same hand movement, but it is a different category since the hand is on the left of the pen. Figure 1b is a reverse action of Figure 1a , but it is not pull from right to left. The key point here is the need for recognizing patterns in spatiotemporal context. Even the same hand-iron-background combination has different meanings in different spatiotemporal contexts. The number of combinations increases sharply as scenes become more complicated and the number of objects involved increases. It would be difficult for conventional convolutions which recognize fixed patterns that are determined by the fixed filter parameters to capture the variety of variations that distinguish the action classes. To recognize every object-in-context pattern, the model needs to have more detailed filters, potentially leading to a blow up of the number parameters. On the other hand, although the object-in-context patterns can vary, they are related through a higherorder structure: pushing an iron, pushing a pen, pulling an iron, and so on affects the spatio-temporal relations of the involved structures to one another in similar ways. We hypothesize that the structure of object-in-context patterns can be learned, i.e., the model can learn to conclude object-in-context pattern given the context, and propose a corresponding feature extractor. Explicitly, let X and Y respectively represent the input and output of a convolution. Let y p and {x p } represent a specific position of Y and the set of positions of X from which y p is computed, respectively. Denote conventional convolution operation as Y = f (X; Θ) where Θ is the shared parameters at different positions. The parameters act as determined feature extractors as As we analyze, the visual pattern of the target object can vary in different contexts, and determined feature extractors (filters) that ignore this dependence are not optimal. We replace the fixed filters with context-dependent filters y p = f ({x p }; w p ) where the filters w p are in turn obtained as w p = g({x p }; Θ). The mapping g is the structure of object-in-context patterns and Θ are the learned parameters as we hypothesize. The entire relation between Y and X can be compactly represented through the higher-order function Y = f (X; g(X; Θ)). The proposed model is able to capture spatiotemporal contexts effectively. We test our method on four benchmark datasets for action recognition: Kinetics-400 (Carreira & Zisserman, 2017) , Something-Something V1 (Mahdisoltani et al., 2018) , Something-Something V2, and Charades datasets (Sigurdsson et al., 2016) . Specifically, we make comprehensive ablation studies on Something-Something V1 datasets and further evaluate on the other three datasets to demonstrate the generality of our proposed method. The experiments establish significant advantages of the proposed models over existing algorithms, achieving results on par with or better than the current state-of-the-art methods. In this paper, we have introduced higher-order networks to the task of action recognition. Higherorder networks are constructed by a general building block, termed as H-block, which aims to model position-varying contextual information. As demonstrated on the Something-Something (V1 and V2), Kinetics-400 and Charades datasets, the proposed higher-order networks are able to achieve state-of-the-art results, even using only RGB mobility inputs without fine-tuning with other image or video datasets. The good performance may be ascribed to the fact that higher-order networks are a natural for context modeling. The actual model itself is not restricted to visual tasks, but may be applied in any task where a context governs the interpretation of an input feature, such as cross-modal or multi-modal operations. In future work, we plan to investigate the benefits of our higher-order model and its extensions, in a variety of other visual, text and cross-modal tasks. M. Zolfaghari, K. Singh, and T. Brox. Eco: Efficient convolutional network for online video understanding. In European Conference on Computer Vision (ECCV), 2018. A APPENDIX Table 6 shows the factorization of different context fields. For example, we stack three convolutions with kernel size 3 × 3 × 3 to get a 7 × 7 × 7 context field. 3 × 3 × 3 1 × 3 × 3 3 × 1 × 1 1 × 1 × 1 3 × 5 × 5 1 × 3 × 3 3 × 3 × 3 1 × 1 × 1 5 × 5 × 5 1 × 3 × 3 3 × 3 × 3 3 × 1 × 1 5 × 7 × 7 1 × 3 × 3 3 × 3 × 3 3 × 3 × 3 7 × 7 × 7 3 × 3 × 3 3 × 3 × 3 3 × 3 × 3 Table 7 shows our backbone ResNet-50 I3D model. We use T×H×W to represent the dimensions of kernels and output feature maps. T = {8, 32}, and the corresponding input size is 8×224×224 and 32×224×224.
Proposed higher order operation for context learning Proposes a new 3D convolutional block which convolves video input with its context, based on the assumpton that relevant context is present around the image's object.