As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature. Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction. In these works, the global context information is incorporated into local features before they are fed into convolutional layers. However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs. Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context. As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features. Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation. Convolutional Neural Networks (CNNs) have achieved significant successes in various tasks, e.g., image classification (He et al., 2016a; Huang et al., 2017) , object detection (Girshick et al., 2014; , image translation , action recognition (Carreira & Zisserman, 2017) , sentence/text classification Kim, 2014) , machine translation (Gehring et al., 2017) , etc. However, the sliding window mechanism of convolution makes it only capable of capturing local patterns, limiting its ability of utilizing global context. Taking the 2D convolution on the image as one example, as Figure 1a shows, the standard convolution only operates on the local image patch and thereby composes local features. According to the recent research on neuroscience (Gilbert & Li, 2013) , neurons' awareness of global context is important for us to better interpret visual scenes, stably perceive objects and effectively process complex perceptual tasks. Many methods (Vaswani et al., 2017; Wang et al., 2017a; b; Hu et al., 2018; Chen et al., 2019; Cao et al., 2019; Bello et al., 2019) have been proposed recently to introduce global context modeling modules into CNN architectures. In this paper, such a family of works is named as global feature interaction methods. As Figure 1b shows, these methods modulate intermediate feature maps by incorporating the global context with the local feature representation. For example, in Non-local modules (Wang et al., 2017b) , local features are reassembled according to global correspondence, which augments CNNs with the global context modeling ability. As was discussed by Gilbert & Li (2013) , the global context influences neurons processing information in two distinct ways: "various forms of attention, including spatial, object oriented and feature oriented attention" and "rather than having a fixed functional role, neurons are adaptive processors, changing their function according to behavioral context". The previous work (Vaswani et al., 2017; Wang et al., 2017a; b; Hu et al., 2018; Chen et al., 2019; Cao et al., 2019; Bello et al., 2019) of global feature interaction methods, shown in Figure 1b , only modifies intermediate features, namely, inputs of neurons, which corresponds to the first way. However, to the best of our knowledge, the other efficient and intuitive way, i.e., explicitly modulating the convolution kernels according to context, has not been exploited yet. Motivated by this, we will model convolutional layers as "adaptive processors" and explore how to leverage global context to guide the composition of local features in convolution operations. In this paper, we propose Context-Gated Convolution (CGC), as Figure 1c shows, a new perspective of complementing CNNs with the awareness of the global context. Specifically, our proposed CGC learns a series of mappings to generate gates from the global context feature maps to modulate convolution kernels accordingly. With the modulated kernels, standard convolution is performed on input feature maps, which is enabled to dynamically capture representative local patterns and compose local features of interest under the guidance of global context. Our contributions are in three-fold. • To the best of our knowledge, we make the first attempt of introducing the contextawareness to convolutional layers by modulating the weights of them according to the global context. • We propose a novel lightweight CGC to effectively generate gates for convolution kernels to modify the weights with the guidance of global context. Our CGC consists of a Context Encoding Module that encodes context information into latent representations, a Channel Interacting Module that projects them into the space of output dimension, and a Gate Decoding Module that decodes the latent representations to produce the gate. • Our Context-Gated Convolution can better capture local patterns and compose discriminative features, and consistently improve the performance of standard convolution with a negligible complexity increment in various tasks including image classification, action recognition, and machine translation. 2 CONTEXT-GATED CONVOLUTION 2.1 PRELIMINARY Without loss of generality, we consider one sample of 2D case. The input to a convolutional layer is a feature map X ∈ R c×h×w , where c is the number of channels, and h, w are respectively the height and width of the feature map. In each convolution operation, a local patch of size c × k 1 × k 2 is collected by the sliding window to multiply with the kernel W ∈ R o×c×k1×k2 of this convolutional layer, where o is the number of output channels, and k 1 , k 2 are respectively the height and width of the kernel. Therefore, only local information within each patch is extracted in one convolution operation. Although in the training process, the convolution kernels are learned from all the patches from all the images in the training set, the kernels are not adaptive to the current context during inference time. We are aware of previous works on dynamically modifying the convolution operation (Dai et al., 2017; Wu et al., 2019; Jia et al., 2016; Jo et al., 2018; Mildenhall et al., 2018) . However, two key factors distinguish our approach from those works: whether the information guiding convolution is collected globally and how it changes the parameters of convolution. Dai et al. (2017) proposed to adaptively set the offset of each element in a convolution kernel, and Wu et al. (2019) proposed to dynamically generate the weights of convolution kernels. However, in their formulations, the dynamic mechanism for modifying convolution kernels only takes local patches or segments as inputs, so it is only adaptive to local inputs, which limits their ability of leveraging rich information in global context. According to experiments in Section 3.4, our proposed CGC significantly outperforms Dynamic Convolution (Wu et al., 2019) with the help of global context awareness. Another family of works on dynamic filters (Jia et al., 2016; Jo et al., 2018; Mildenhall et al., 2018) generates weights of convolution kernels using features extracted from input images by another CNN feature extractor. The expensive feature extraction process makes it more suitable for generating a few filters, e.g., in the case of low-level image processing. It is impractical to generate weights for all the layers in a deep CNN model in this manner. However, our CGC takes input feature maps of a convolutional layer and makes it possible to dynamically modulate the weight of each convolutional layer, which systematically improves CNNs' global context modeling ability. In this paper, motivated by neuroscience research on neurons as "adaptive processors", we proposed Context-Gated Convolution (CGC) to incorporate global context information into CNNs. Different from previous works which usually modifies input feature maps, our CGC directly modulates convolution kernels under the guidance of global context information. We proposed three modules to efficiently generate a gate to modify the kernel. As such, our CGC is able to extract representative local patterns according to global context. The extensive experiment results show consistent performance improvements on various tasks. There are still a lot of future works that can be done. For example, ew could design task-specific gating modules to fully uncover the potential of the proposed CGC. Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional network for online video understanding. For ImageNet, we use 224 × 224 random resized cropping and random horizontal flipping for data augmentation. Then we standardize the data with mean and variance per channel. We use a standard cross-entropy loss to train all the networks with a batch size of 256 on 8 GPUs by SGD with a weight decay of 0.0001 and a momentum of 0.9 for 100 epochs. We start from a learning rate of 0.1 and decrease it by a factor of 10 every 30 epochs. For CIFAR-10, we use 32 × 32 random cropping with a padding of 4 and random horizontal flipping. We use a batch size of 128 and train on 1 GPU. We decrease the learning rate at the 81st and 122nd epochs, and ends training after 164 epochs.
A novel Context-Gated Convolution which incorporates global context information into CNNs by explicitly modulating convolution kernels, and thus captures more representative local patterns and extract discriminative features. This paper uses global context to modulate the weights of convolutional layers and help CNNs capture more discriminative features with high performance and fewer parameters than feature map modulating.