Recurrent convolution (RC) shares the same convolutional kernels and unrolls them multiple times, which is originally proposed to model time-space signals. We suggest that RC can be viewed as a model compression strategy for deep convolutional neural networks. RC reduces the redundancy across layers and is complementary to most existing model compression approaches. However, the performance of an RC network can't match the performance of its corresponding standard one, i.e. with the same depth but independent convolutional kernels.   This reduces the value of RC for model compression. In this paper, we propose a simple variant which improves RC networks: The batch normalization layers of an RC module are learned independently (not shared) for different unrolling steps. We provide insights on why this works. Experiments on CIFAR show that unrolling a convolutional layer several steps can improve the performance, thus indirectly plays a role in model compression. Deep convolution neural networks (DCNNs) have achieved ground-breaking results on a broad range of fields, such as computer vision BID0 and natural language processing BID1 . Unfortunately, DCNNs are both computation intensive and memory intensive for industrial applications. Many approaches have been proposed recently to obtain more compact DCNNs while keep their performance as much as possible. Conceptually, those approaches fall in two categories: 1) Reduce the computational cost or memory usage of big DCNNs by weights pruning, quantization and sharing BID2 BID3 BID4 BID5 ; 2) Improve the performance of small DCNNs by knowledge distillation BID6 or other techniques.In this paper, we explore a potential compression strategy which is complementary to most of the existing approaches: training a recurrent convolutional (RC) neural network. As the name suggests, the same convolutional kernels are unrolled multiple times on the computational graph. This is a weights sharing mechanism applied to the whole layer. Suppose there is a network with n RC layers each of which unrolls k times, then we say its depth is nk. If the performance of this network can match the performance of a standard DCNN with nk layers (Suppose other conditions are the same), we could say we compress the standard one with factor k. Then we can further compress the obtained RC network by applying other existing approaches, such as weight quantization. The key intuition is that RC can reduce the redundancy across layers by sharing weights of the whole layer. While most existing approaches work at a layer-wise manner and only remove part of a layer. This is why we say RC is a complementary strategy.However, we find the performance of RC networks can't match the performance of DCNNs with the same depth. This significantly reduces the value of RC for model compression. In this paper, we aim to improve the performance of RC in a simple way. Specifically, we learn the batch normalization layers (BN) BID7 independently at each unrolling step. We describe our insights in the next section. Experiments on CIFAR dataset demonstrate that such a simple variant works well. We also compare RC networks with their corresponding standard ones.The idea of RC is not new. Many works have used it to model time-space signals BID8 or to obtain a larger receptive field BID9 . However, to our knowledge, none of them view RC as a potential model compression strategy and none of them compare RC networks with their corresponding standard ones strictly. The intention of this paper is to show that RC is a considerable or at least a heuristic solution for model compression. We suggest recurrent convolution is a considerable strategy for model compression. RC reduces the redundancy across layers (which is ignored by most of the compression methods). We can train an RC network and then further compress it via existing approaches. We also suggest it is significantly better to learn independent BN parameters at each unrolling step when training RC networks. Experiments on CIFAR dataset demonstrate that unrolling the same convolutional layer several steps can improve the accuracy of the whole network, thus indirectly plays a role in model compression. We believe that the performance of RC could be further improved.
Recurrent convolution for model compression and a trick for training it, that is learning independent BN layres over steps. The author modifies the recurrent convolution neural network (RCNN) with independent batch normalization, with the experimental results on RCNN compatible with the ResNet neural network architecture when it contains the same number of layers.