batch size is reduced as well. The effect is diminished on 16 compute nodes since the overall effect of losing one compute node is reduced. Similar for other datasets as shown in Figures \[fig:googlenet\_loss\_n16\],  \[fig:cifar10\_loss\_n16\] and  \[fig:mnist\_loss\_n16\], the convergence of FT-SGD and SGD is similar.

Performance Analysis
--------------------

Figures \[fig:alexnet\_run\_time\],  \[fig:googlenet\_run\_time\],  \[fig:cifar10\_run\_time\] and \[fig:mnist\_run\_time\] shows the performance comparisons of SGD and FT-SGD using AlexNet, GoogLeNet, CifarNet (CIFAR10) and LeNet-3 (MNIST). The overall evaluation uses 1024 batches, which is a relatively small number of batches in comparison to the standard number of batches such as 60K for AlexNet. For the FT-SGD evaluation, exactly one process executes `SIGKILL` after 300 batch updates, resulting in $n - 1$ remaining number of compute nodes. Since the batch on each compute node remains constant, the overall computation time as observed on each compute node is similar for FT-SGD and SGD. Also, since we have one compute node failure, the overall difference in communication time is also negligible.

Figures \[fig:alexnet\_dataload\_time\], \[fig:googlenet\_dataload\_time\],  \[fig:cifar10\_dataload\_time\] and  \[fig:mnist\_dataload\_time\] shows the comparison of reading the overall dataset and partial dataset when a fault occurs for AlexNet, GoogLeNet, and other datasets. These charts are specifically useful to understand the cost of reading 