fault is detected by the `on_gradients_ready`.

![Code flow for original and fault tolerant implementations for data readback[]{data-label="fig:code_flow"}](./code_flow.pdf){width="\columnwidth"}

Performance Evaluation {#sec:exp}
======================

In this section, we present a detailed performance evaluation of the proposed fault tolerant approach using ULFM. Table \[table:arch\] shows the hardware and software details of our systems. Table \[table:heur\_desc\] shows a description of proposed approaches that we have implemented. Table \[table:datasets\] provides an overview of datasets, and the associated neural networks.

  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                                                     Name                                                     CPU (\#cores)   GPU   Baseline Caffe   Network   MPI   cuDNN   CUDA   Nodes   \#cores
  ---------------------------------------------------------------------------------------------------------- --------------- ----- ---------------- --------- ----- ------- ------ ------- ---------
   **[PUMA]{} & Haswell (20) & N/A & Intel-Caffe [@caffe-intel] & IB & ULFM-OpenMPI & N/A & N/A & 16 & 320\                                                                                
                                                      **                                                                                                                                   
  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

\

       Name                           Type                      Implemented  Description of DL Algorithm and Implementation
  --------------- -------------------------------------------- ------------- -----------------------------------------------------------------------
   SGD (Default)          Stochastic Gradient Descent               Yes      Implements strong scaling by dividing batch and all-to-all reduction.
      FT-SGD       Fault Tolerant Stochastic Gradient Descent       Yes      Implements strong scaling by dividing batch and all-to-all reduction.

\

[max width=]{}

                   Dataset                               Model                  Description       Training Samples   Validation Samples        Image Size         Classes  
  ------------------------------------------ ------------------------------ -------------------- ------------------ -------------------- ----------------------- --------- --
             MNIST [@mnistlecun]              LeNet-3 [@lecun1998gradient]   Handwritten Digits        60000               10000              $28 \times28$         10     
   CIFAR-10 [@Krizhevsky09learningmultiple]          Caffe-default              Small 