we consider checkpoint-restart, Reinit (when a fault occurs, re-initialize the MPI automatically) and user-level fault mitigation.

-   We consider several approaches for recovery from faults. We primarily rely on “continued execution” – where the DL implementation continues to execute by using the remaining set of compute nodes.

-   We implement our design using MaTEx-Caffe and leverage the ULFM implementation available with OpenMPI. We provide an evaluation of fault tolerant MaTEx-Caffe using the ImageNet-1K dataset and widely studied neural network topologies such as AlexNet and GoogLeNet.

Our evaluation on a 16 node Intel Haswell system connected with InfiniBand indicates that the proposed fault tolerant MaTEx-Caffe is able to scale well and continues execution in the presence of actual permanent node faults. It incurs no observable overhead in the absence of faults, and provides expected performance after recovering from faults, since the overall number of compute nodes are reduced. We also observe that both Reinit [@Reinit] and ULFM [@Bland2013] proposals are suitable for addressing permanent node faults for DL algorithms. However, ULFM is simple – and versatile enough – since it obviates the need for any checkpoint/restart, re-reading of the entire dataset and allows continued execution in the presence of permanent node 