In a typical deep learning approach to a computer vision task, Convolutional Neural Networks (CNNs) are used to extract features at varying levels of abstraction from an image and compress a high dimensional input into a lower dimensional decision space through a series of transformations. In this paper, we investigate how a class of input images is eventually compressed over the course of these transformations. In particular, we use singular value decomposition to analyze the relevant variations in feature space. These variations are formalized as the effective dimension of the embedding. We consider how the effective dimension varies across layers within class. We show that across datasets and architectures, the effective dimension of a class increases before decreasing further into the network, suggesting some sort of initial whitening transformation. Further, the decrease rate of the effective dimension deeper in the network corresponds with training performance of the model. In this section, we analyze and discuss the implications of our findings. Further, we propose complementary analyses that would bolster our findings. In studied examples, neural networks initially spherize embeddings and then collapse dimensionality. The compression of the dimensionality of feature spaces via transformations on inputs is more dramatic in better-performing networks.6. Appendix is scaled by factor α, the spectral norm of α * Φ (l) is α * σ max (Φ (l) ). This is also true in a ReLU network with α > 0. Such a scaling is achieved while preserving φ (l+1) : DISPLAYFORM0 While Srebro et al. BID10 directly apply the trace-norm to bound the complexity of a completed matrix, we apply spectral normalization in Equation 1 to correct for this scale sensitivity. Hence, a small effective dimension corresponds to an eccentric feature space regardless of magnitude.
Neural networks that do a good job of classification project points into more spherical shapes before compressing them into fewer dimensions.