We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning. We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the ``good'' solution. Additionally, our attempts to find a decreasing objective path from ``bad'' solutions to the ``good'' ones in the sparse subspace fail. However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions. These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace. Reducing parameter footprint and inference latency of machine learning models is an active area of research, fostered by diverse applications like mobile vision and on-device intelligence. Sparse networks, that is, neural networks in which a large subset of the model parameters are zero, have emerged as one of the leading approaches for reducing model parameter count. It has been shown empirically that deep neural networks can achieve state-of-the-art results under high levels of sparsity BID12 BID20 BID7 , and this property has been leveraged to significantly reduce the parameter footprint and inference complexity BID15 uses the same, or even greater, computational resources compared to fully dense training, which imposes an upper limit on the size of sparse networks we can train.Training Sparse Networks. In the context of sparse networks, state-of-the-art results have been obtained through training densely connected networks and modifying their topology during training through a technique known as pruning BID29 BID24 BID12 . A different approach is to reuse the sparsity pattern found through pruning and train a sparse network from scratch. This can be done with a random initialization ("scratch") or the same initialization as the original training ("lottery"). Previous work BID7 BID18 demonstrated that both approaches achieve similar final accuracies, but lower than pruning 1 . The difference between pruning and both approaches to training while sparse can be seen in Figure 1 . Despite being in the same energy landscape, "scratch" and "lottery" solutions fail to match the performance of the solutions found by pruning. Given the utility of being able to train sparse from scratch, it is critical to understand the reasons behind the failure of current techniques at training sparse neural networks.There exists a line of work on training sparse networks BID0 BID21 BID19 BID23 which allows the connectivity pattern to change over time. These techniques generally achieve higher accuracy compared with fixed sparse connectivity, but generally worse than pruning. Though promising, the role of changing connections during the optimization is not clear. While we focus on fixed sparsity pattern in this work, our results give insight into why these other approaches are more successful.Motivated by the disparity in accuracy observed in Figure 1 , we perform a series of experiments to improve our understanding of the difficulties present in training sparse neural networks, and identify possible directions for future work.More precisely, our main contributions are:• A set of experiments showing that the objective function is monotonically decreasing along the straight lines that interpolate from:-the original dense initialization -the original dense initialization projected into the sparse subspace -a random initialization in the sparse subspace to the solution obtained by pruning 2 . This demonstrates that even when the optimization process fails, there was a monotonically decreasing path to the "good" solution.• In contrast, the linear path between the scratch and the pruned solutions depicts a high energy barrier between the two solutions. Our attempts to find quadratic and cubic Bézier curves BID8 with a decreasing objective between the two sparse solutions fails suggesting that the optimization process gets attracted into a "bad" local minima.• Finally , by removing the sparsity constraint from the path, we are consistently able to find decreasing objective Bézier curves between the two sparse solutions. This result suggests that allowing for dense connectivity might be necessary and sufficient to escape the stationary point converged in the sparse subspace.The rest of the paper is organized as follows: In §2, we describe the experimental setup. In §3 we present the results from these experiments, followed by a discussion in §4. Our experiments highlight a gap in our understanding of energy landscape of sparse deep networks. Why does training a sparse network from scratch gets stuck at a neighborhood of a stationary point with a significantly higher objective? This is in contrast with recent work that has proven that such a gap does not exist for certain kinds of over-parameterized dense networks BID26 BID2 . Since during pruning dimensions are slowly removed, we conjecture that this prevents the optimizer from getting stuck into "bad" local minima. The failure of the optimizer is even more surprising in the light of the linear interpolation experiments of Section 3.1, which show that sparse initial points are connected to the pruning solutions through a path in which the training loss is monotonically decreasing.In high dimensional energy landscapes, it is difficult to assess whether the training converges to a local minimum or to a higher order saddle point. BID27 shows that the Hessian of a convolutional network trained on MNIST is degenerate and most of its eigenvalues are very close to zero indicating an extremely flat landscape at solution. (2007) 's results and argues that critical points that are far from the global minima in Gaussian fields are most likely to be saddle points. In Section 3.2, we examine the linear interpolation between solutions and attempt to find a parametric curve between them with decreasing loss. This is because finding a decreasing path from the high loss solution ("scratch") to the low loss solution("pruned") would demonstrate that the former solution is at a saddle point. Our work provides insights into the dynamics of optimization in the sparse regime which we hope will guide progress towards better regularization techniques, initialization schema, and/or optimization algorithms for training sparse networks.Training of sparse neural networks is still not fully understood from an optimization perspective. In the sparse regime, we show that optimizers converge to stationary points with a sub-optimal generalization accuracy. This is despite monotonically decreasing paths existing from the initial point to the pruned solution. And despite nearly monotonically decreasing paths in the dense subspace from the "bad" local minimum to the pruned one.Optimizers sparse networks that reach pruned accuracy levels are yet to be found. We believe that understanding why popular optimizers used in deep learning fail in the sparse regime will yield important insights leading us towards more robust optimizers in general.
In this paper we highlight  the difficulty of training sparse neural networks by doing interpolation experiments in the energy landscape 