needed within MPI and at the application level (such as intermediate checkpointing) to leverage the existing functionality. The Reinit proposal is suitable as well. However, there are two potential downsides that are readily observed: 1) Reinit would require DL implementations to consider intermediate check-pointing, when the DL algorithm does not mandate it, and 2) Reinit would require application to read the entire dataset from the disk, when reading the data could be fairly localized to the degree of failure.

ULFM has positive attributes which are definitely suitable for designing fault tolerant DL implementations. The primary functionality that is required is automatic fixing of the communicator, and reading the partial dataset from the disk, while continuing to execute with the existing set of compute nodes. The ULFM specification itself has a few implementation caveats, including the cost of fault detection (which is relatively lesser for Reinit), cost of global notification and cost of recovering the communicator. Yet, the overall cost of computation recovery is at most one batch – while in the case of Reinit, it is expected to be much higher depending upon the degree of checkpointing. Hence, we consider ULFM for implementing fault tolerant DL algorithms. In the next 