Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers. We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small (100k--1.2M parameter) image classification networks. Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts. Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth. Deep convolutional neural networks have become the dominant force for many image classification tasks; see BID10 ; BID17 ; BID19 . Their ability to assimilate low-, medium-, and high-level features in an end-to-end multi-layer fashion has led to myriad groundbreaking advances in the field. In recent years, residual networks (resnets) have emerged as one of the best performing neural network archetypes in the literature; see BID6 . Through the use of identity shortcut connections, resnets have overcome the challenging technical obstacles of vanishing gradients and the apparent degradation that otherwise comes with training very deep networks. Resnets have achieved state-of-the-art performance on several image classification datasets using very deep neural networks, sometimes with over 1000 layers.Although shortcut connections appeared in the early neural network literature, e.g., BID1 ; BID14 ; BID16 , their importance became more clear in 2015 with the emergence of the HighwayNets of BID18 and resnets. The former involved gated shortcut connections that regulate the flow of information across the network, while the latter used identity shortcut connections, which are parameterless. Resnets are also presumed to be easier to train and seem to perform better in practice. In their first resnet paper, He et al. argued that identity maps let gradients flow back, enabling the training of very deep networks, and that it's easier for a layer to learn when initialized near an identity map than near a zero map (with small random weights); see also .However , in a flurry of recent activity, most notably from BID25 ; BID4 ; BID22 ; BID13 and BID23 , arguments have emerged that the effectiveness of resnets is not due to their depth, where practitioners were training networks of hundreds or thousands of layers, but rather that deep resnets are effectively creating ensembles of shallower networks, and the layers are more likely to refine and reinforce existing features than engineer new ones. These arguments assert that the achievement of resnets is less about extreme depth and more about their ability to ease backpropagation with moderate depth. Indeed , in many cases wider residual networks that were only 10-50 layers deep were shown to perform better and train in less time than very deep ones (over 100 layers). See BID25 .More recently still, others have presented many clever and creative ways to train very deep networks using variations on the shortcut theme; see for example BID8 ; BID11 ; BID26 ; ; BID0 ; BID2 ; BID27 ; BID6 ; BID12 ; BID24 ; Savarese (2016); BID19 , and BID21 . In summary, shortcut connections clearly help in practice, but there are many different, and sometimes conflicting hypotheses as to why.In this paper we investigate a new hypothesis about shortcut connections, namely, that their power lies not in the identity mapping itself, but rather just in combining linear and nonlinear functions at each layer. The tests where identity shortcuts were observed to perform better than general linear connections were all done in very deep (100 or more layers) networks. The recent evidence that wider, shallower, resnet networks can outperform deeper ones suggests that it is worth investigating whether identity connections are better than general linear connections in such networks.We first describe some of the intuition about why this might be the case. We then investigate this idea with careful experiments using relatively small networks constructed of five different types of blocks. These blocks are all variations on the idea of residual blocks (resblocks), but where the identity shortcut is replaced with a more general linear function. We call these blocks, consisting of both a linear and a nonlinear part, tandem blocks and the resulting networks tandem networks. Residual networks and several similar architectures are special cases of tandem networks.The networks we use in our experiments are relatively small (100k-1.2M parameter) image classification networks constructed from these various tandem blocks. The small networks are appropriate because the goal of the experiments is not to challenge state-of-the-art results produced by much larger models, but rather to compare the five architectures in a variety of settings in order to gain insight into their relative strengths and weaknesses. Whereas many other authors pursue extreme network depth as a goal in itself, here we limit our focus to comparing performance (in this case, classification accuracy) of different architectures.Our experiments suggest that general linear layers, which have learnable parameters, perform at least as well as the identity shortcut of resnets. This is true even when some width is sacrificed to keep the total number of parameters the same. Our results further suggest that the best specific type of linear connection to use in the blocks of a tandem network depends on several factors, including both network width and depth. We generalized residual blocks (which use identity shortcut connections) to tandem blocks (which can learn any linear connection, not just the identity). We found that general linear connections with learnable weights, have the same benefits as the identity maps in residual blocks, and they actually increase performance compared to identity maps. We also showed that linear connections do not learn identity maps, even when initialized with identity weight matrices. These results seem to confirm that the success of residual networks and related architectures is not due to special properties of identity maps, but rather is simply a result of using linear maps to complement nonlinear ones.The additional flexibility gained by replacing identity maps with convolutions led to better results in every one of our experiments. This was not due to extra parameters, as we adjusted layer widths to keep parameter counts as close to equal as possible. Instead, general linear convolutions appear to do a better job than identity maps of working together with nonlinear convolutions.Our results further suggest that tandem blocks with a single nonlinear convolution tend to outperform those with two, but blocks that use 3 × 3 convolutions for their linear connections may be better in wide networks than those with 1 × 1s.Finally, we note that there are many more possible types of tandem block than those we have considered here, and many more applications in which to test them.
We generalize residual blocks to tandem blocks, which use arbitrary linear maps instead of shortcuts, and improve performance over ResNets. This paper performs an analysis of shortcut connections in ResNet-like architectures, and proposes to substitute the identity shortcuts with an alternative convolutional one referred to as tandem block. This paper investigates the effect of replacing identity skip connections with trainable convolutional skip connections in ResNet and finds that performance improves.