Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting. We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods. A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.
It's possible to substitute the weight matrix in a convolutional layer to train it as a structured efficient layer; performing as well as low-rank decomposition. This work applies previous Structured Efficient Linear Layers to conv layers and proposes Structured Efficient Convolutional Layers as substitution of original conv layers.