Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks.
 The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed.
 In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF. 
 As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information.
 However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons.
 In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution).
 In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation.
 We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs. Convolutional Neural Networks (CNNs) and their convolution layers (Fukushima, 1980; Lecun et al., 1998) are inspired by the finding in cat visual cortex (Hubel & Wiesel, 1959) and they show the strong performance in various domains such as image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016) , natural language processing (Gehring et al., 2017) , and speech recognition (Abdel-Hamid et al., 2014; Zhang et al., 2016) . A notable characteristic of the convolution layer is the Receptive Field (RF), which is the particular input region where a convolutional output is affected by. The units (or neurons) in higher layers have larger RF by bundling the outputs of the units in lower layers with smaller RF. Thanks to the hierarchical architectures of CNNs, the units in high layers are able to capture the global context even though the units in low layers only see the local information. It is known that neurons in the primary visual cortex (i.e., V1 which is low layers) change the selfproperties (e.g., the RF size (Pettet & Gilbert, 1992) and the facilitation effect (Nelson & Frost, 1985) ) based on the information outside of the RF (D. Gilbert, 1992) . The mechanism is believed to originate from (1) feedbacks from the higher-order area (Iacaruso et al., 2017) and (2) intracortical horizontal connections (D. Gilbert, 1992) . The feedbacks from the higher-order area convey broader-contextual information than the neurons in V1, which allows the neurons in V1 to use the global context. For instance, Gilbert & Li (2013) argued that the feedback connections work as attention. Horizontal connections allow the distanced neurons in the layer to communicate with each other and are believed to play an important role in visual contour integration (Li & Gilbert, 2002) and object grouping (Schmidt et al., 2006) . Though both horizontal and feedback connections are believed to be important for visual processing in the visual cortex, the regular convolution ignores the properties of these connections. In this work, we particularly focus on algorithms to introduce the function of horizontal connections for the regular convolution in CNNs. We propose spatially shuffled convolution (ss convolution), where the information outside of the regular convolution's RF is incorporated by spatial shuffling, which is a simple and lightweight operation. Our ss convolution is the same operation as the regular convolution except for spatial shuffling and requires no extra learnable parameters. The design of ss convolution is highly inspired by the function of horizontal connections. To test the effectiveness of the information outside of the regular convolution's RF in CNNs, we perform experiments on CIFAR-10 (Krizhevsky, 2009) and ImageNet 2012 dataset (Russakovsky et al., 2015) and show that ss convolution improves the classification performance across various CNNs. These results indicate that the information outside of the RF is useful when processing local information. In addition, we conduct several analyses to examine why ss convolution improves the classification performance in CNNs and show that spatial shuffling allows the regular convolution to use the information outside of its RF. In this work, we propose spatially shuffled convolution (ss convolution) to incorporate the function of horizontal connections in the regular convolution. The spatial shuffling is simple, lightweight, and requires no extra learnable parameters. The experimental results demonstrate that ss convolution captures the information outside of the regular convolution's RF even in lower layers. The results and our analyses also suggest that using distant information (i.e., non-local) is effective for the regular convolution and improves classification performance across various CNNs. Figure 6: The receptive field of ImageNet-1k pre-trained ResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table 5 (a) conv2 1 Figure 7: The receptive field of ImageNet-1k pre-trained ResNet50 with ss convolutions. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table 5 (a) conv2 1 Figure 8: The receptive field of ImageNet-1k pre-trained SEResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table 5 l e n g t h = i n t ( c h n s / c h s )
We propose spatially shuffled convolution that the regular convolution incorporates the information from outside of its receptive field. Proposes SS convulation which uses information outside of its RF, showing improved results when tested on multiple CNN models. The authors proposed a shuffle strategy for convolution layers in convolution layers in convolutional neural networks.