the increased frequency of faults, which are commonplace in large scale systems [@Schroeder:4775906]. Distributed DL implementations such as distributed TensorFlow, distributed memory implementations of Caffe and even recently proposed Caffe2  [@Caffe2] are primarily geared towards performance. As shown in Table \[table:newcomparisons\], we observe that state of the art HPC ready DL implementations are not fault tolerant. On the other hand, automatic fault tolerance is provided by MapReduce instantiations such as Hadoop, and Spark. However, the implementations are not HPC ready. At the same time, DL implementations are known to take days even on modest size datasets, significantly increasing the probability of observing a fault during the training phase. This raises two important questions: [*1) What are the elements of fault tolerant DL algorithms? and 2) What is needed from MPI for implementing these fault tolerant DL algorithms?*]{}

Contributions
-------------

In this paper, we address these questions and make the following contributions:

-   We present the case for several types of parallelism (model, data and hybrid) as motivated from common use-cases and DNN topologies. We use this discussion to derive the suitability of fault tolerance proposals in MPI.

-   We consider several design choices for implementing fault tolerant DL implementations. Specifically, 