asynchronous gradient descent (AGD)) are still useful, but do not provide strict equivalence to the default stochastic gradient descent (SGD) algorithm.

Scalability Analysis
--------------------

An advantage of model parallelism is its potential to scale-out the DL algorithms very well. For example, if there are 1,000 layers, then in an ideal situation each compute node may have one layer, resulting in scale-out. However, there are several reasons to not consider pure model parallelism based techniques for scaling out the DL implementations.

                     Method                     Symbol          
  --- ------------------------------------ ----------------- -- --
   1       Features in previous layer            $n_1$          
   2       Features in current layer             $n_2$          
   3   Activation shape in previous layer   $x_1\times x_2$     
   4      Window size in current layer      $w_1\times w_2$     
   5       Strides for current layer        $s_1\times s_2$     

  : Symbols Used For Computing Activations and Parameters[]{data-label="table:activations"}

\

For any DNN with several hidden layers, let us consider two consecutive layers: $L_1$ and $L_2$. We compute the number of parameters and activations for $L_2$ as follows: If $L_2$ is a convolution layer, then the array containing the parameters is $w_1\times w_2\times n_1\times n_2$. However, the activations are an array of size $\frac{x_1}{s_1}\times \frac{x_2}{s_2}\times n_2$. As $w_i<\frac{x_i}{s_i}$, the number of parameters is less than the number of activations. 