@scaffe] still make data parallelism an amenable choice for scaling out DL implementations. It is worth noting that a possibility which combines DL model and data parallelism – hybrid parallelism – has been proposed in literature as well [@das:arxiv16]. However, usually model parallelism is applied on multiple GPUs/multiple sockets on each compute node and data parallelism is applied for multiple compute nodes. In essence, we already consider hybrid parallelism, while implicitly leveraging model parallelism within a node. [**Hence, we consider data parallelism for fault tolerance considerations in rest of the paper.**]{}

Solution Space {#sec:design}
==============

In this section, we present a solution space for designing fault tolerant DL implementations using data parallelism, as discussed in the previous section. An important consideration is exploring the suitability of existing features in MPI for this purpose, with detailed considerations for the primary proposals.

Critical Data Structures in DL Algorithms
-----------------------------------------

The first step is the identification of critical data structures in DL algorithms. Specifically, there are several data structures which are used during the feed-forward and the back-propagation phase of the DL implementations. During the feed-forward step, the input dataset and the model are used – both of which are read-only during the 