Table \[table:notation\] and describe back-propagation in Algorithm \[alg:bp\].

\[alg:bp\] **input:** Data $X\in {\mathbb{R}}^{n\times p}$ and labels $Y\in {\mathbb{R}}^{n\times \ell}$ Compute all $z^{(\ell)}$ and $a^{(\ell)}$. $\delta^{(n_\ell)} = -(y-a^{n_{\ell}})\odot \sigma(z^{(n_\ell)})$ $\delta^{(\ell)}=W^{\ell} \delta^{(\ell+1)}\odot \sigma'(z^{(\ell)})$ $\nabla_{W^{(\ell)}}C = \delta^{(\ell+1)}{a^{(\ell)}}^T$ $\nabla_{b^{(\ell)}}C = \delta^{(\ell+1)}$

Algorithm \[alg:bp\] is most directly applicable to fully-connected neural networks. For structured data, however, convolutional neural networks (CNNs) are more useful. The fundamental unit of computation in a CNN are convolutions – which are stored as arrays of some dimension – unlike vectors in a fully-connected neural network as described above. Each neuron in a convolution layer considers input from a small window (such as `3x3,5x5`) in an image, applies a convolution and computes a value. The computation can be reduced to a matrix-vector multiplication with redundant weights, allowing the above algorithm to be applied.

Elements of Deep Learning Algorithms for Fault Tolerance Consideration {#sec:motivation}
======================================================================

In this section, we present the motivation of our work. Specifically, we consider the properties of DL algorithms, distinguishing between MLPs, CNNs and RNNs in terms of their expected execution on large scale systems. This distinction provides the necessary guidelines for requirements from MPI in terms of fault tolerance. As pointed out by Gropp and Lusk [@gropp:ijhpca02], 