designing fault tolerant DL implementation.

Suitability of Existing MPI Features and Proposals {#sec:existing_proposals}
==================================================

Suitability of Existing Error Handling in MPI
---------------------------------------------

An important design consideration is the suitability of existing error handling in MPI for designing fault tolerant DL implementations. Specifically, by initializing with `MPI_ERRORS_RETURN`, it is possible for a DL implementation to prevent aborting when a fault is detected either during MPI communication or an external system software component (such as SLURM).

By setting an explicit error handler, it is possible for the DL implementation to checkpoint their critical data structures, exit the application and re-start the application from the recent saved checkpoint. In this specific case, the critical data structure is the model parameters of the DNN, since the dataset is read-only, and it can be readily recovered from the disk/file-system. The DL implementation may be re-started using $n$ compute nodes (if spare compute nodes are available) or $n - 1$ compute nodes, since DL implementations do not have specific requirements of a topology. This approach is definitely a suitable possibility. However, it may not be necessary, since this will result in a recovery complexity of $O(n)$, a function of number of compute nodes, instead of the 