We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.   This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer;  for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.  The paper is dedicated to computation of singular values of convolutional layers Derives exact formulas for computing singular values of convolutional layers of deep neural networks and show that computing the singular values can be done much faster than computing the full SVD of the convolution matrix by appealing to fast FFT transformations.