Conversely, if $L_2$ is a fully-connected layer, then there are $n_1\times n_2$ parameters and only $n_2$ activations. We note that if $L_1$ is a convolution layer, then $n_1$ must be replaced by the total number of activations of that layer, namely $\frac{x_1}{s_1}\times \frac{x_2}{s_2}\times n_1$. This implies, in general, convolutional networks have lesser communication volume of weights than activations. However, for fully-connected networks the trend is reversed. In the case of AlexNet (shown in Figure \[fig:alexnet\_topology\]), a well-established DNN architecture, this can be seen directly. The first convolutional layer contains 34,848 parameters, but has 301,056 activations, a difference of an order of magnitude. The final fully-connected layer, however, has 4,096,000 parameters and 1,000 activations.

Lessons Learned
---------------

In the previous section, we provided an in-depth discussion on the possibilities of scaling out DL implementations. While model parallelism looks attractive, in practice the ratio of communication of activations to model parameters prohibits effective scaling. This is because CNNs are increasingly becoming commonplace – including the winners of last 5 years of ImageNet classification challenge [@imagenet]. At the same time, data parallelism provides scaling out possibilities, but has limitations regarding the growth of batch size [@Keuper:MLHPC2016]. However, solutions proposed by other researchers [@Krizhevsky14oneweird; 