Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network. But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively. Chen et al. (2015) proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression. We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weight-sharing, and use it to study the role of weight-sharing in neural networks. We show that common neural networks can be expressed as ArbNets with different hash functions. We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network. Most deep neural network architectures can be built using a combination of three primitive networks: the multi-layer perceptron (MLP), the convolutional neural network (CNN), and the recurrent neural network (RNN). These three networks differ in terms of where and how the weight-sharing takes place. We know that the weight-sharing structure is important, and in some cases essential, to the success of the neural network at a particular machine learning task.For example, a convolutional layer can be thought of as a sliding window algorithm that shares the same weights applied across different local segments in the input. This is useful for learning translation-invariant representations. BID10 showed that on a simple ten-class image classification problem like CIFAR10, applying a pre-processing step with 32, 000 random convolutional filters boosted test accuracy from 54% to 83% using an SVM with a vanilla Gaussian kernel. Additionally, although the ImageNet challenge only started in 2010, from 2012 onwards, all the winning models have been CNNs. This suggests the importance of convolutational layers for the task of image classification. We show later on that balanced and deterministic weight-sharing helps network performance, and indeed, the weights in convolutional layers are shared in a balanced and deterministic fashion.We also know that tying the weights of encoder and decoder networks can be helpful. In an autoencoder with one hidden layer and no non-linearities, tying the weights of the encoder and the decoder achieves the same effect as Principal Components Analysis BID8 . In language modeling tasks, tying the weights of the encoder and decoder for the word embeddings also results in increased performance as well as a reduction in the number of parameters used BID5 .Developing general intuitions about where and how weight-sharing can be leveraged effectively is going to be very useful for the machine learning practitioner. Understanding the role of weightsharing in a neural network from a quantitative perspective might also potentially lead us to discover novel neural network architectures. This paper is a first step towards understanding how weightsharing affects the performance of a neural network.We make four main contributions:• We propose a general weight-sharing framework called ArbNet that can be plugged into any existing neural network and enables efficient arbitrary weight-sharing between its parameters. (Section 1.1)• We show that deep networks can be formulated as ArbNets, and argue that the problem of studying weight-sharing in neural networks can be reduced to the problem of studying properties of the associated hash functions. (Section 2.4)• We show that balanced weight-sharing increases network performance. (Section 5.1)• We show that making an ArbNet hash function, which controls the weight-sharing, more deterministic increases network performance, but less so when it is sparse. (Section 5.2)1.1 ARBNET ArbNets are neural networks augmented with a hash table to allow for arbitrary weight-sharing. We can label every weight in a given neural network with a unique identifier, and each identifier maps to an entry in the hash table by computing a given hash function prior to the start of training.On the forward and backward passes, the network retrieves and updates weights respectively in the hash table using the identifiers. A hash collision between two different identifiers would then imply weight-sharing between two weights. This mechanism of forcing hard weight-sharing is also known as the 'hashing trick' in some machine learning literature. A simple example of a hash function is the modulus hash: DISPLAYFORM0 where the weight w i with identifier i maps to the (i mod n)th entry of a hash table of size n.An ArbNet is an efficient mechanism of forcing weight-sharing between any two arbitrarily selected weights, since the only overhead involves memory occupied by the hash BID0 . While the load factor is a variable controlling the capacity of the network, it is not necessarily the most important factor in determining network performance. A convolutional layer has a much higher load factor than a fully connected layer, and yet it is much more effective at increasing network performance in a range of tasks, most notably image classification.There are at least two other basic questions we can ask:• How does the balance of the hash table affect performance? -The balance of the hash table indicates the evenness of the weight sharing. We give a more precise definition in terms of Shannon entropy in the EXPERIMENTAL SETUP section, but intuitively, a perfectly balanced weight sharing scheme accesses each entry in the hash table the same number of times, while an unbalanced one would tend to favor using some entries more than others.• How does noise in the hash function affect performance?-For a fixed identifier scheme, if the hash function is a deterministic operation, it will map to a fixed entry in the hash table. If it is a noisy operation, we cannot predict a priori which entry it would map into. -We do not have a rigorous notion for 'noise', but we demonstrate in the EXPERIMEN-TAL SETUP section an appropriate hash function whose parameter can be tuned to tweak the amount of noise.We are interested in the answers to these question across different levels of sparsity, since as in the case of a convolutional layer, this might influence the effect of the variable we are studying on the performance of the neural network. We perform experiments on two image classification tasks, MNIST and CIFAR10, and demonstrate that balance helps while noise hurts neural network performance. MNIST is a simpler task than CIFAR10, and the two tasks show the difference, if any, when the neural network model has enough capacity to capture the complexity of the data versus when it does not.
Studied the role of weight sharing in neural networks using hash functions, found that a balanced and deterministic hash function helps network performance. Proposing ArbNets to study weight sharing in a more systematic way by defining the weight sharing function as a hash function.