Images           60000               10000          $32 \times32\times3$      10     
             ImageNet [@ILSVRC15]               AlexNet [@NIPS2012_4824]       Diverse Images        1,281,167             50,000         $256\times256\times3$    1,000   
             ImageNet [@ILSVRC15]                      GoogLeNet               Diverse Images        1,281,167             50,000         $256\times256\times3$    1,000   

Objective
---------

The objective of our performance evaluation is to understand the performance overhead of using ULFM based implementation and correctness implications (if any) of the existing ULFM implementations.

Fault Injection Methodology
---------------------------

To emulate the process faults, we insert a fault in a process by using `SIGKILL`. For controlled experiment the fault is injected after 300 batches. This effectively emulates compute node faults, since we use one process on a compute node and multiple threads for each process.

Correctness Analysis
--------------------

For understanding the correctness, we compare the [*loss curves*]{} – a measure of the error as observed during the training phase. The curves are compared for SGD and FT-SGD implementations in Figures \[fig:alexnet\_loss\_n4\],  \[fig:alexnet\_loss\_n8\], and  \[fig:alexnet\_loss\_n16\] using 4, 8, and 16 compute nodes respectively. The FT-SGD evaluation consists of exactly one process fault – which is usually the case in real scenarios. We observe that the loss curves for both SGD and FT-SGD implementation track each other roughly. However, the FT-SGD is behind since with reduced number of available compute nodes, the overall 