step. However, at the back-propagation step, the model weights are updated while the input dataset remains untouched. Hence, the critical data structure for DL implementations is the model weights which are updated iteratively till convergence. It is worth noting that there are auxiliary data structures which are updated as well. As an example for DL implementations with [*momentum*]{}, data structures such as [*history*]{} are updated, which can be recalculated from the model weights. The gradients – iterative updates to the model weights – are calculated during the back-propagation step. However, they are accumulated iteratively in the model weights, and hence they are not critical.

Process Recovery Model
----------------------

An important design point is the process recovery model. Several scientific applications such as LULESH [@IPDPS13:LULESH] and NAS Parallel Benchmarks [@Bailey:SC91] typically require a fixed topology (such as a square/quadratic) in terms of number of MPI processes. However, there are no such requirements for DL implementations. Hence, it is possible to continue execution with the remaining set of compute nodes, without affecting the correctness of the DL implementation. A natural advantage of this approach is that it requires little support from process managers for practical deployments. Hence, we use this approach for 