In the visual system, neurons respond to a patch of the input known as their classical receptive field (RF), and can be modulated by stimuli in the surround. These interactions are often mediated by lateral connections, giving rise to extra-classical RFs. We use supervised learning via backpropagation to learn feedforward connections, combined with an unsupervised learning rule to learn lateral connections between units within a convolutional neural network. These connections allow each unit to integrate information from its surround, generating extra-classical receptive fields for the units in our new proposed model (CNNEx). We demonstrate that these connections make the network more robust and achieve better performance on noisy versions of the MNIST and CIFAR-10 datasets. Although the image statistics of MNIST and CIFAR-10 differ greatly, the same unsupervised learning rule generalized to both datasets. Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding the computations implemented by feedforward connections when the input is unreliable. While feedforward convolutional neural networks have resulted in many practical successes [1] , they are highly susceptible to adversarial attacks [2] . In contrast, the brain makes use of extensive recurrent connections, including lateral and feedback connections, which may provide some level of immunity to these attacks (for results on human adversarial examples, see [3] ). Additionally, the brain is able to build rich internal representations of information with little to no labeled data, which is a form of unsupervised learning, in contrast to the supervised learning required by most models. We present a model incorporating lateral connections (learned using a modified Hebbian rule) into convolutional neural networks, with feedforward connections trained in a supervised manner. When applying different noise perturbations to the MNIST [4] and CIFAR-10 [5] datasets, lateral connections in our model improve the overall performance and robustness of these networks. Our results suggest that integration of lateral connections into convolutional neural networks is an important area of future research. Orientation and distance dependence of lateral connections. A) Left: Connection probability as a function of difference in preferred orientation between excitatory neurons observed experimentally (from [6] ). Right: Normalized connection probability between excitatory neurons as a function of inter-somatic distance as reported experimentally in mouse auditory cortex [7] . B, C): Model predictions for orientation and distance dependence (k 1 represents the target neuron) of positive (B) and negative (C) lateral connection weights for filters constructed using estimates of spatial receptive field (RF) sizes from in-vivo recordings in mouse V1 [8] . Red (blue) bars/lines represent positive (negative) weights and dashed black lines represent Gaussian fits for distance dependence (standard deviations σ expt = 114µm, σ pos = 120 µm and σ neg = 143 µm for experiment, model positive and negative weights respectively). Predicted connections qualitatively match with experimental data. In our model, lateral connections capture structure in the statistics of the world via unsupervised learning. This structure allows for inference that can make use of the integration of information across space and features. By combining these lateral connections with features learned in a supervised manner using backpropagation, the network does not learn any arbitrary structure present in the world, but only the structure of features which is needed to solve a particular task. As a result, our method allows us to predict the structure of the world which is relevant to a given task. The vast majority of deep neural networks are feedforward in nature, although recurrent connections have been added to convolutional neural networks [24, 25] . Recurrent connections have also been used to implement different visual attention mechanisms [26, 27] . However, these networks are still largely trained in a supervised manner. An exception are ladder networks, which have been proposed as a means to combine supervised and unsupervised learning in deep neural networks [28] . However, different from our approach, ladder networks use noise injection to introduce an unsupervised cost function based on reconstruction of the internal activity of the network. Our model instead relies on a modified Hebbian learning rule which learns the optimal lateral connections between features within each layer based solely on the activations of units coding for these features. Neurons are inherently noisy, and their responses can vary even to the same stimulus. These neurons are embedded in cortical circuits that must perform computations in the absence of information, such as under visual occlusion. Optimal lateral connections can provide additional robustness to these networks by allowing integration of information from multiple sources (i.e. different features and spatial locations). This type of computation is also potentially useful for applications in which artificial neurons are not simulated with high fidelity, e.g. in neuromorphic computing. We chose a relatively simple network architecture as a proof-of-concept for our model. As such, we did not achieve state-of-the art performance on either image dataset. This accuracy could be further improved by either fine-tuning models after learning the optimal lateral connections or using deeper network architectures with more parameters. Future experiments will also have to test the scalability of learning optimal lateral connections on more complex network architectures and larger image datasets (e.g. ImageNet), and whether these connections provide any benefit against noise or other types of perturbations such as adversarial images.
CNNs with biologically-inspired lateral connections learned in an unsupervised manner are more robust to noisy inputs. 