degree of failure. The reasoning behind this time complexity is due to the fact that the $n - 1$ compute nodes would need to read the entire dataset back from the disk (prohibitive data movement), in addition to the checkpointed model files. Hence, this approach may be considered as the baseline approach, but not necessarily as the optimal approach for handling permanent node faults in DL implementations.

Suitability of User-Level Fault Mitigation (ULFM) Proposal
----------------------------------------------------------

One of the fault tolerance proposal which has been considered for inclusion in the MPI specification for the last few years is ULFM. The salient features of ULFM are: 1) ability to provide non-collective global fault notification, 2) ability to recover from faults by fixing the [*broken*]{} communicator on the fly and 3) support for fixed/shrinking process set.

ULFM is particularly suited for applications which have small process-specific state information. Usually, resetting the global state information is non-trivial and requires writing a complex error handler. Naturally, for large-scale applications – which have been developed over decades – writing a correct error handler even for a subset of fault cases is non-trivial.

However, with data parallelism the overall state information that is required for DL implementations 