before the next batch of samples are ingested.

Pseudo-code Walk-through
------------------------

Figure \[fig:ulfm\_code\] shows the difference between the original non-fault-tolerant implementation of the callback `on_gradients_ready` and the fault tolerant version on the right. Figure \[fig:code\_flow\] presents the code changes for data readback in the fault tolerant version and compares it to the original code.

![Code snippet for ULFM based fault tolerant DL implementation[]{data-label="fig:ulfm_code"}](./ulfm_code.pdf){width="\columnwidth"}

### Original Callback

The original call-back receives the gradients from the `Forwardbackward` function and uses an all-to-all reduction to synchronize the gradients across all compute nodes. The resulting gradient is divided by the number of compute nodes and applied to the local model using `ApplyGradients` function.

### Callback with ULFM Changes

As shown in Figure \[fig:ulfm\_code\], the ULFM changes are handful and primarily restricted to a single callback. Specifically, when a fault is detected, we leveraget the `MPIX_Comm_shrink` function is used to shrink the communicator from original to the new communicator. Once the communicator is reset, then all to all reduction is retried till return code is `MPI_SUCCESS`.

### Data Readback for Fault Tolerant Version

Figure \[fig:code\_flow\] shows the original code flow and the code flow for fault tolerant DL. The data is read only when a 