the end of each batch each compute node (assuming that the implementation uses shared address space programming model such as OpenMP/pthread on a node) executes an `MPI_Allreduce`. By executing the all-to-all reduction primitive, the algorithm ensures that it is equivalent to the default sequential DL algorithm [@Keuper:MHLPC2015]. An important consideration for data parallelism is that strong scaling of work is essential to ensure the equivalence of the implementation to the sequential algorithm. Specifically, let us consider a batch size $b$, and let $n$ be the number of compute nodes. The overall expected complexity of the data parallelism based implementation is $\Theta\left(\frac{b}{n} + \log(n)\right)$. Naturally, the ratio of communication to computation increases with strong scaling – which is a potential downside to data parallelism. Several solutions have been proposed to handle this situation [@Krizhevsky14oneweird]. One possibility is to consider increasing the batch size and increasing the values of other hyper-parameters (such as learning rate) by Krizhevsky [@Krizhevsky14oneweird]. Recently proposed solutions such as S-Caffe [@scaffe] improve the scalability of data parallelism by leveraging the overlap of communication with computation. While a few of these approaches provide strict equivalence to the sequential algorithm, other approaches such as asynchronous variants (also referred to as 