neural network topology[]{data-label="fig:alexnet_topology"}](./alexnet_topology.pdf){width="0.7\columnwidth"}

During the feedforward step, a batch of samples is executed on the first hidden layer. The output of the first hidden layer – which is typically referred to as activations – is forwarded to the next hidden layer, resulting in point-to-point communication between two compute nodes. This procedure is repeated untill the last layer of the DNN is reached, at which point the [*error*]{} is calculated. During the back-propagation step, the error is used to calculate the updates to the weights ([*gradients*]{}) which are communicated between compute nodes in the reverse order to the feedforward step.

Data Parallelism
----------------

A widely used option in scaling out DL implementations is [*data parallelism*]{} [@zheng:icpads16; @siegel:bigdata16; @siegel2016adaptive; @vishnu:cluster15-a; @vishnu:cluster15-b; @vishnu:ipdps16; @shohdy:icpp16; @shohdy:hipc16; @zheng:icpads16]. Under this type of parallelism, the model is replicated and the data is split among multiple compute nodes. A pictorial representation of the data parallelism is shown in the Figure \[fig:data\_parallelism\].

![A pictorial representation of data parallelism in DL algorithms using AlexNet neural network topology and four compute nodes. The model is synchronized at the end of each batch using `MPI_Allreduce` and other primitives such as NVIDIA Collective Communication Library (NCCL)[]{data-label="fig:data_parallelism"}](./data_parallelism.pdf){width="0.7\columnwidth"}

As shown in the figure, at 