“fault tolerance is a property of MPI programs and specification”. Hence, it is critical to consider these in conjunction. Our first element of discussion is the expected type of parallelism for scaling out DL algorithms.

Master-Slave Paradigm
---------------------

Over the last few years, several researchers have considered the possibility of scaling out DL algorithms [@NIPS2012_0598; @geeps; @projectadam; @mxnet; @caffeonspark]. The classical work in scaling out DL algorithms considered a [*master-slave*]{} paradigm, which was proposed under the DistBelief framework [@NIPS2012_0598]. It considered a hierarchical organization of [*parameter servers*]{} which would hold the latest copy of the model. The [*workers*]{} would periodically update the master with their updates and request the latest copy of the model. Several extensions to this fundamental paradigm have been proposed in the literature [@geeps; @mxnet; @firecaffe]. The limitations of the master-slave model have been well-studied in the distributed systems research [@das:arxiv16; @lian:nip15]. In addition to being a single point of failure, and a communication bottleneck, the limitation of this approach is that the convergence of master-slave paradigm worsens at scale-out. For extreme scale systems, this approach is infeasible. Hence, we disregard this approach which would be leveraged in practical deployments especially of HPC systems such as Leadership 