is minimal. Since the model is replicated across the compute nodes, the DL implementation requires no checkpointing. Hence, ULFM is potentially the right fit for implementing fault tolerant DL algorithms.

Suitability of Reinit Fault Tolerance Proposal
----------------------------------------------

Recently, Laguna [*et al.*]{} have proposed [*Reinit*]{} proposal for handling faults in MPI. The objective of Reinit is to address the limitations of ULFM, and is particularly suitable to applications where the code complexity of the recovery module is high. The salient features of the Reinit proposal are: 1) automatic re-initialization of MPI after a fault is detected, 2) semi-automatic recovery from the intermediate checkpoints, and 3) ability to handle shrinking/fixed process set.

We consider the suitability of the Reinit proposal to data parallelism based DL implementations. We observe that DL implementations would be required to check-point the model weights periodically, which would be used by the Reinit implementation during recovery. We also observe that the application would be re-started requiring the entire dataset to be read from the disk.

Lessons Learned
---------------

We observe that existing local fault notification in the MPI specification and implementations may be used for developing fault tolerant DL implementations. However, there is a significant amount of work 