I implemented a DenseNet for class and wondered why it works well, when I noticed no one had posted on StackExchange specifically about DenseNet.

Why are DenseNets good?


Picture of a DenseNet:

Picture example of a denseNet

Much of this answer is taken from this paper, as well as my classmate's years of experience with Convolutional Neural Networks (CNNs).

DenseNets work because they allow information from early layers to be communicated directly to later layers. Something researchers found that in a vanilla CNN (no "skipping"), the later layers of CNNs actually relearn features learned in the earliest layers of the network because those features are important, but the later layers no longer have access to that information because of a lack of "skipping" .

The paper mentions these four advantages of DenseNets:

1. They alleviate the vanishing-gradient problem
2. Strengthen feature propagation
3. Encourage feature reuse, and
4. Substantially reduce the number of parameters
  1. Vanishing gradient

    a. Points 2 and 3 are intuitive enough and discussed in the above paragraph, but point 1 takes a little more technical understanding. Because the partial derivatives that we calculate during backprop are derivatives with respect to particular parameters, the only way a vanilla FeedForward Neural Network (FFNN) can update those parameters is by propogating information back through each and every intermediate layer. Not only is this very indirect, but the information communicated via the partial derivatives is literally muddled by the intermediate effects of layers between. But in a DenseNet, we get around this problem because the information from the Gradient can be communicated directly to that specific parameter rather than being mixed in with the information from other layers. Therefore, the gradient will be more directly correlated with the net's performance than it would if there were intermediate layers. More on vanishing gradient and its relationship to DenseNets here

  2. Strengthen feature propagation

    a. ie. features learned by layer 1 are directly accessible by layer 5

  3. Encourage feature reuse

    a. ie. Layer 5 doesn't have to relearn a feature learnt by layer 1 because it can access that information directly via concatenation (ResNets add all the inputs to layer 5 together, muddling them, whereas DenseNets do this better)

  4. Reduce number of parameters:

    a. The filter size (number of convolutions each layer has to do to pass to the next one) is reduced in DenseNet compared to architectures without skip because to communicate the same amount of information, we now have to allow each layer to "talk" more to the very next layer than we otherwise would have. When information "skips" intermediate layers, that filter depth is no longer required so we don't have to keep track of as many convolutional parameters.

Read more on skip architecture here


Your Answer

