Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent architectures. Specifically, in addition to sparsification of individual weights and neurons, we propose to sparsify preactivations of gates and information flow in LSTM. It makes some gates and information flow components constant, speeds up forward pass and improves compression. Moreover, the resulting structure of gate sparsity is interpretable and depends on the task. Recurrent neural networks (RNNs) yield high-quality results in many applications BID0 BID3 BID17 BID20 but often overfit due to overparametrization. In many practical problems, RNNs can be compressed orders of times with only slight quality drop or even with quality improvement BID1 BID14 BID19 . Methods for RNN compression can be divided into three groups: based on matrix factorization BID5 BID18 , quantization BID6 or sparsification BID1 BID14 BID19 .We focus on RNNs sparsification. Two main groups of approaches for sparsification are pruning and Bayesian sparsification. In pruning BID14 BID19 , weights with absolute values less than a predefined threshold are set to zero. Such methods imply a lot of hyperparameters (thresholds, pruning schedule etc). Bayesian sparsification techniques BID13 BID15 BID7 BID8 BID1 treat weights of an RNN as random variables and approximate posterior distribution over them given sparsity-inducing prior distribution. After training weights with low signal-to-noise ratio are set to zero. This allows eliminating the majority of weights from the model without time-consuming hyperparameters tuning. Also, Bayesian sparsification techniques can be easily extended to permanently set to zero intermediate variables in the network's computational graph BID15 BID7 (e.g. neurons in fully-connected networks or filters in convolutional networks). It is achieved by multiplying such a variable on a learnable weight, finding posterior over it and setting the weight to zero if the corresponding signal-to-noise ratio is small.In this work, we investigate the last mentioned property for gated architectures, particularly for LSTM. Following BID1 BID13 , we sparsify individual weights of the RNN. Following BID7 , we eliminate neurons from the RNN by introducing multiplicative variables on activations of neurons. Our main contribution is the introduction of multiplicative variables on preactivations of the gates and information flow in LSTM. This leads to several positive effects. Firstly , when some component of preactivations is permanently set to zero, the corresponding gate becomes constant. It simplifies LSTM structure and speeds up computations . Secondly, we obtain a three-level hierarchy of sparsification : sparsification of individual weights helps to sparsify gates and information flow (make their components constant), and sparsification of gates and information flow helps to sparsify neurons (remove them from the model). As a result, the overall compression of the model is higher.
We propose to sparsify preactivations of gates and information flow in LSTM to make them constant and boost the neuron sparsity level This paper proposed a sparsification method for recurrent neural networks by eliminating neurons with zero preactivations to obtain compact networks.