approaches by extending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.'
author:
- Vinay Amatya
- Abhinav Vishnu
- Charles Siegel
- Jeff Daily
bibliography:
- 'bibTexts.bib'
title: 'What does fault tolerant Deep Learning need from MPI?'
---

Introduction {#sec:intro}
============

Deep Learning (DL) algorithms are a class of Machine Learning and Data Mining algorithms, which conduct model learning by emulating the computational structure of a mammalian brain. A deep neural network (DNN) – which stores the model of a DL algorithm – contains several [*layers*]{} of [*neurons*]{} inter-connected with [*synapses*]{}. By using deep layers, DL algorithms are able to conduct transformations on highly non-linear data, which is commonplace in many scientific datasets. DL algorithms have shown amazing results in many Computer Vision tasks [@NIPS2012_4824; @43022] and science domains such as High Energy Physics [@Baldi:2014kfa], Climate Modeling [@liu2016application] and Computational Chemistry [@goh2017deep]. DL implementations such as TensorFlow [@tensorflow2015-whitepaper], Caffe [@jia2014caffe], Theano [@bergstra+al:2010-scipy; @Bastien-Theano-2012], and Torch [@Collobert02torch:a] have become available. These implementations are primarily geared towards compute nodes that may contain a multi-core architecture (such as Intel 