We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training. Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than 0.5% classification layer connection density, or less than 5% overall network connection density. We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers. Based on our sparsifying technique, we introduce the `scatter' metric to characterize the quality of a particular connection pattern. As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns. Neural networks (NNs) in machine learning systems are critical drivers of new technologies such as image processing and speech recognition. Modern NNs are gigantic in size with millions of parameters, such as the ones described in Alexnet BID11 , Overfeat BID12 and ResNet BID9 . They therefore require an enormous amount of memory and silicon processing during usage. Optimizing a network to improve performance typically involves making it deeper and adding more parameters BID13 BID17 BID10 , which further exacerbates the problem of large storage complexity. While the convolutional (conv) layers in these networks do feature extraction, there are usually fully connected layers at the end performing classification. We shall henceforth refer to these layers as connected layers (CLs), of which fully connected layers (FCLs) are a special case. Owing to their high density of connections, the majority of network parameters are concentrated in FCLs. For example, the FCLs in Alexnet account for 95.7% of the network parameters .We shall refer to the spaces between CLs as CL junctions (or simply junctions), which are occupied by connections, or weights. Given the trend in modern NNs, we raise the question -"How necessary is it to have FCLs?" or, in other words, "What if most of the junction connections never existed? Would the resulting sparsely connected layers (SCLs), when trained and tested, still give competitive performance?" As an example, consider a network with 2 CLs of 100 neurons each and the junction between them has 1000 weights instead of the expected 10,000. Then this is a sparse network with connection density of 10%. Given such a sparse architecture, a natural question to ask is "How can the existing 1000 weights be best distributed so that network performance is maximized?"In this regard, the present work makes the following contributions. In Section 2, we formalize the concept of sparsity, or its opposite measure density, and explore its effects on different network types. We show that CL parameters are largely redundant and a network pre-defined to be sparse before starting training does not result in any performance degradation. For certain network architectures, this leads to CL parameter reduction by a factor of more than 450, or an overall parameter reduction by a factor of more than 20. In Section 2.4, we discuss techniques to distribute connections across junctions when given an overall network density. Finally, in Section 3, we formalize pre-defined sparse connectivity patterns using adjacency matrices and introduce the scatter metric. Our results show that scatter is a quick and useful indicator of how good a sparse network is. This paper discusses the merits of pre-defining sparsity in CLs of neural networks, which leads to significant reduction in parameters without performance loss. In general, the smaller the fraction of CLs in a network, the more redundancy there exists in their parameters. If we can achieve similar results (i.e., 0.2% density) on Alexnet for example, we would obtain 95% reduction in overall parameters. Coupled with hardware acceleration designed for pre-defined sparse networks, we believe our approach will lead to more aggressive exploration of network structure. Network connectivity can be guided by the scatter metric, which is closely related to performance, and by optimally distributing connections across junctions. Future work would involve extension to conv layers, since recent CNNs have lower values for the ratio of number of CLs to number of conv layers.
Neural networks can be pre-defined to have sparse connectivity without performance degradation. This paper examines sparse connection patterns in upper layers of convolutional image classification networks, and introduces heuristics for distributing connections among windows/groups and a measure called scatter to construct connectivity masks. Proposal to reduce the number of parameters learned by a deep network by setting up sparse connection weights in classiication layers, and introduction of a concept of "scatter."