section, we present the implementation details for ULFM based DL algorithm.

Design and Implementation Details {#sec:implDetail}
=================================

In this section, we present design and implementation details for fault tolerant DL algorithms. We leverage the ULFM implementation provided by OpenMPI for this purpose and implement our changes in Caffe runtime. Figure \[fig:code\_snippet\] shows the overall interaction between different components.

![Overall Caffe architecture and code flow for the original code. We require changes to dataset reading and the callback for implementing fault tolerance.[]{data-label="fig:code_snippet"}](./code_snippet.pdf){width="0.8\columnwidth"}

As shown in figure, the Caffe architecture has layers such as for MLPs, CNNs and RNNs, which are defined in a prototxt file. Hence, these are already resident on disk. Our extensions of Caffe – also referred to as MaTEx-Caffe for rest of the paper – support parallel NetCDF format which requires changes for fault tolerance. Caffe runtime supports several types of solvers such as SGD, Adam and others. These solvers use a common substrate for data parallelism, where `ForwardBackward()` step computes the forward and back-propagation steps of the overall implementation. The resulting gradients are then synchronized using a callback, which is extended by MaTEx-Caffe for using `MPI_Allreduce`. The resulting synchronized gradients are then applied using the `ApplyGradients` function 